\section{Wave runup over sinusoidal ridges}
This scenario simulates a wave flowing up a beach with skewed sinusoidal ridges. Following the initial wave runup, eventually the water elevation should become constant, and the velocities should approach zero.

\subsection{Results}
Figure~\ref{fig:vel_t1_centroid} shows the centroid velocities during the wave runup. The flow should be concentrating in the channels near the shore, and be free from major spikes.

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{vel_t1_centroid.png}
\caption{Velocity during the wave runup. Point color corresponds to the bed elevation}
\label{fig:vel_t1_centroid}
\end{center}
\end{figure}

Figure~\ref{fig:vel_t2_centroid} shows the velocities profile at time 200~s. They should be nearly zero (e.g. O($10^{-3}$) m$/s$). This case has been used to illustrate wet-dry artefacts in some versions of \anuga.

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{vel_t2_centroid.png}
\caption{Velocity at time 200~s after the wave runup. The flow speed should be nearly zero. Point color corresponds to the bed elevation.}
\label{fig:vel_t2_centroid}
\end{center}
\end{figure}

\endinput


\section{Circle.\-h File Reference}
\label{_circle_8h}\index{Circle.\-h@{Circle.\-h}}
\subsection*{Classes}
\begin{DoxyCompactItemize}
\item
class {\bf Circle}
\end{DoxyCompactItemize}
\subsection*{Macros}
\begin{DoxyCompactItemize}
\item
\#define {\bf P\-I}~3.\-1415926535898
\end{DoxyCompactItemize}

\subsection{Macro Definition Documentation}
\index{Circle.\-h@{Circle.\-h}!P\-I@{P\-I}}
\index{P\-I@{P\-I}!Circle.h@{Circle.\-h}}
\subsubsection[{P\-I}]{\setlength{\rightskip}{0pt plus 5cm}\#define P\-I~3.\-1415926535898}\label{_circle_8h_a598a3330b3c21701223ee0ca14316eca}

Definition at line 1 of file Circle.\-h.


\section{Experiments}
\label{sec:exp}

\begin{table*}[t] \footnotesize

\begin{tabular}{|l|l|l|}
\hline
Distributional Models & \mto & \vm \\
\hline
\cite{Lamar:2010:LCU:1870658.1870736} & .708 & -\\ %algorithm name LDC
\cite{Brown:1992:CNG:176313.176316}* & .678 & .630\\
%kcls(Och,1999) & .737 & .656\\
%\cite{goldwater-griffiths:2007:ACLMain} & .632 & .562\\
(Goldwater et al., 2007) & .632 & .562\\
\cite{Ganchev:2010:PRS:1859890.1859918}* & .625 & .548\\
\cite{maron2010sphere} & .688 (.0016)&-\\
Bigrams (Sec.~\ref{sec:bigram}) & \bgmto & \bgvm \\
Partitions (Sec.~\ref{sec:rpart}) & \rpmto & \rpvm \\
Substitutes (Sec.~\ref{sec:wordsub}) & \wsmto & \wsvm \\
\hline
\end{tabular}
\begin{tabular}{|l|l|l|}
\hline
Models with Additional Features & \mto & \vm \\
\hline
\cite{Clark:2003:CDM:1067807.1067817}* & .712 & .655 \\
\cite{christodoulopoulos-goldwater-steedman:2011:EMNLP} & .728 & .661\\
\cite{bergkirkpatrick-klein:2010:ACL} & .755 & -\\ % Interesting in  christo paper:73.9/67.7
\cite{Christodoulopoulos:2010:TDU:1870658.1870714} & .761 & .688\\
\cite{blunsom-cohn:2011:ACL-HLT2011} & .775 & .697\\
Substitutes and Features (Sec.~\ref{sec:feat}) & \ftmto & \ftvm \\
& & \\
& & \\
\hline
\end{tabular}

\caption{Summary of results in terms of the \mto and \vm scores.
  Standard errors are given in parentheses when available.  Starred
  entries have been reported in the review paper
  \cite{Christodoulopoulos:2010:TDU:1870658.1870714}.  Distributional
  models use only the identity of the target word and its context.
  The models on the right incorporate orthographic and
  morphological features.}
\label{tab:results}
\end{table*}

In this section we present experiments that evaluate substitute
vectors as representations of word context within the S-CODE
framework.  Section~\ref{sec:bigram} replicates the bigram based
S-CODE results from \cite{maron2010sphere} as a baseline.  The S-CODE
algorithm works with discrete inputs.  The substitute vectors as
described in Section~\ref{sec:lm} are high dimensional and continuous.
We experimented with two approaches to use substitute vectors in a
discrete setting.  Section~\ref{sec:rpart} presents an algorithm that
partitions the high dimensional space of substitute vectors into small
neighborhoods and uses the partition id as a discrete context
representation.  Section~\ref{sec:wordsub} presents an even simpler
model which pairs each word with a random substitute.  When the
left-word -- right-word pairs used in the bigram model are replaced
with word -- partition-id or word -- substitute pairs we see
significant gains in accuracy.  These results support our running
hypothesis that paradigmatic features, i.e. potential substitutes of a
word, are better determiners of syntactic category compared to left
and right neighbors.  Section~\ref{sec:feat} explores morphologic and
orthographic features as additional sources of information and its
results improve the state-of-the-art in the field of unsupervised
syntactic category acquisition.

Each experiment was repeated 10 times with different random seeds and
the results are reported with standard errors in parentheses or error
bars in graphs.  Table~\ref{tab:results} summarizes all the results
reported in this paper and the ones we cite from the literature.

\subsection{Bigram model}\label{sec:bigram}

In \cite{maron2010sphere} adjacent word pairs (bigrams) in the corpus
are fed into the S-CODE algorithm as $X, Y$ samples.  The algorithm
uses stochastic gradient ascent to find the $\phi_x, \psi_y$
embeddings for left and right words in these bigrams on a single
25-dimensional sphere.  At the end each word $w$ in the vocabulary
ends up with two points on the sphere, a $\phi_w$ point representing
the behavior of $w$ as the left word of a bigram and a $\psi_w$ point
representing it as the right word.  The two vectors for $w$ are
concatenated to create a 50-dimensional representation at the end.
These 50-dimensional vectors are clustered using an instance weighted
k-means algorithm and the resulting groups are compared to the correct
part-of-speech tags.  Maron et al. \shortcite{maron2010sphere} report
many-to-one scores of .6880 (.0016) for 45 clusters and .7150 (.0060)
for 50 clusters (on the full PTB45 tag-set).  If only $\phi_w$ vectors
are clustered without concatenation we found the performance drops
significantly to about .62.

To make a meaningful comparison we re-ran the bigram experiments using
our default settings and obtained a many-to-one score of \bgmto\
and the V-measure is \bgvm\  for 45 clusters.  The following
default settings were used: (i) each word was kept with its original
capitalization, (ii) the learning rate parameters were adjusted to
$\varphi_0=50$, $\eta_0=0.2$ for faster convergence in log likelihood,
(iii) the number of s-code iterations were increased from 12 to 50
million, (iv) k-means initialization was improved using
\cite{arthur2007k}, and (v) the number of k-means restarts were
increased to 128 to improve clustering and reduce variance.

\subsection{Random partitions}\label{sec:rpart}

Instead of using left-word -- right-word pairs as inputs to S-CODE we
wanted to pair each word with a paradigmatic representation of its
context to get a direct comparison of the two context representations.
To obtain a discrete representation of the context, the
random--partitions algorithm first designates a random subset of
substitute vectors as centroids to partition the space, and then
associates each context with the partition defined by the closest
centroid in cosine distance.  Each partition thus defined gets a
unique id, and word ($X$) -- partition-id ($Y$) pairs are given to
S-CODE as input.  The algorithm cycles through the data until we get
approximately 50 million updates.  The resulting $\phi_x$ vectors are
clustered using the k-means algorithm (no vector concatenation is
necessary).  Using default settings (64K random partitions, 25 s-code
dimensions, $Z=0.166$) the many-to-one accuracy is \rpmto\ and
the V-measure is \rpvm.

To analyze the sensitivity of this result to our specific parameter
settings we ran a number of experiments where each parameter was
varied over a range of values.

\begin{figure}[ht] \centering
\includegraphics[width=\linewidth]{plot-p.pdf}
\caption{\mto is not sensitive to the number of partitions used to
  discretize the substitute vector space within our experimental
  range.}
\label{plot-p}
\end{figure}

Figure~\ref{plot-p} gives results where the number of initial random
partitions is varied over a large range and shows the results to be
fairly stable across two orders of magnitude.

\begin{figure}[ht] \centering
\includegraphics[width=\linewidth]{plot-d.pdf}
\caption{\mto falls sharply for less than 10 S-CODE dimensions, but
  more than 25 do not help.}
\label{plot-d}
\end{figure}

Figure~\ref{plot-d} shows that at least 10 embedding dimensions are
necessary to get within 1\% of the best result, but there is no
significant gain from using more than 25 dimensions.

\begin{figure}[ht] \centering
\includegraphics[width=\linewidth]{plot-z.pdf}
\caption{\mto is fairly stable as long as the $\tilde{Z}$ constant is
  within an order of magnitude of the real $Z$ value.}
\label{plot-z}
\end{figure}

Figure~\ref{plot-z} shows that the constant $\tilde{Z}$ approximation
can be varied within two orders of magnitude without a significant
performance drop in the many-to-one score.  For uniformly distributed
points on a 25 dimensional sphere, the expected $Z\approx 0.146$.  In
the experiments where we tested we found the real $Z$ always to be in
the 0.140-0.170 range.  When the constant $\tilde{Z}$ estimate is too
small the attraction in Eq.~\ref{eq:attract} dominates the repulsion
in Eq.~\ref{eq:repulse} and all points tend to converge to the same
location.  When $\tilde{Z}$ is too high, it prevents meaningful
clusters from coalescing.
%%% I have seen the first, but the second is pure guess, need to
%%% look.  The distances seem to be decreasing on that end as well!

We find the random partition algorithm to be fairly robust to different
parameter settings and the resulting many-to-one score significantly
better than the bigram baseline.

\subsection{Random substitutes}\label{sec:wordsub}

Another way to use substitute vectors in a discrete setting is simply
to sample individual substitute words from them.  The
random-substitutes algorithm cycles through the test data and pairs
each word with a random substitute picked from the pre-computed
substitute vectors (see Section~\ref{sec:lm}).  We ran the
random-substitutes algorithm to generate 14 million word ($X$) --
random-substitute ($Y$) pairs (12 substitutes for each token) as input
to S-CODE.  Clustering the resulting $\phi_x$ vectors yields a
many-to-one score of \wsmto\ and a V-measure of \wsvm.

This result is close to the previous result by the random-partition
algorithm, \rpmto, demonstrating that two very different discrete
representations of context based on paradigmatic features give
consistent results.  Both results are significantly above the bigram
baseline, \bgmto.  Figure~\ref{plot-s} illustrates that the
random-substitute result is fairly robust as long as the training
algorithm can observe more than a few random substitutes per word.

\begin{figure}[ht] \centering
\includegraphics[width=\linewidth]{plot-s.pdf}
\caption{\mto is not sensitive to the number of random substitutes
  sampled per word token.}
\label{plot-s}
\end{figure}

\subsection{Morphological and orthographic features}\label{sec:feat}

Clark \shortcite{Clark:2003:CDM:1067807.1067817} demonstrates that
using morphological and orthographic features significantly improves
part-of-speech induction with an HMM based model.
Section~\ref{sec:related} describes a number other approaches that
show similar improvements.  This section describes one way to
integrate additional features to the random-substitute model.

%% In order to accommodate multiple feature types the CODE model needs to
%% be extended to handle more than two variables.
%% \cite{globerson2007euclidean} suggest the following likelihood
%% function:

%% \begin{eqnarray}
%% &\ell(\phi,& \psi^{(1)}, \ldots, \psi^{(K)}) = \label{eq:multicode}\\
%% &&\sum_k w_k \sum_{x,y^{(k)}} \bar{p}(x,y^{(k)}) \log p(x,y^{(k)}) \nonumber
%% \end{eqnarray}

%% \noindent where $Y^{(1)}, \ldots, Y^{(K)}$ are $K$ different variables
%% whose empirical joint distributions with $X$,
%% $\bar{p}(x,y^{(1)})\ldots\bar{p}(x,y^{(K)})$, are known.
%% Eq.~\ref{eq:multicode} then represents a set of CODE models
%% $p(x,y^{(k)})$ where each $Y^{(k)}$ has an embedding $\psi_y^{(k)}$
%% but all models share the same $\phi_x$ embedding.  The weights $w_k$
%% reflect the relative importance of each $Y^{(k)}$.

%% We adopt this likelihood function, set all $w_k=1$, let $X$
%% represent a word, $Y^{(1)}$ represent a random substitute and
%% $Y^{(2)}, \ldots, Y^{(K)}$ stand for various morphological and
%% orthographic features of the word.  With this setup, the training
%% procedure needs to change little: each time a word --
%% random-substitute pair is sampled, the relevant word -- feature pairs
%% are also generated and input to the gradient ascent algorithm.

The orthographic features we used are similar to the ones in
\cite{bergkirkpatrick-EtAl:2010:NAACLHLT} with small modifications:

\begin{itemize}
\item Initial-Capital: this feature is generated for capitalized words
  with the exception of sentence initial words.
\item Number: this feature is generated when the token starts with a
  digit.
\item Contains-Hyphen: this feature is generated for lowercase words
  with an internal hyphen.
\item Initial-Apostrophe: this feature is generated for tokens that
  start with an apostrophe.
\end{itemize}

%%% dy: which wsj?  1M?  why not 156M??
We generated morphological features using the unsupervised algorithm
Morfessor \cite{creutz05}.  Morfessor was trained on the WSJ section
of the Penn Treebank using default settings, and a perplexity
threshold of 300.  The program induced 5 suffix types that are present
in a total of 10,484 word types.  These suffixes were input to S-CODE
as morphological features whenever the associated word types were
sampled.

In order to incorporate morphological and orthographic features into
S-CODE we modified its input.  For each word -- random-substitute pair
generated as in the previous section, we added word -- feature pairs
to the input for each morphological and orthographic feature of the
word.  Words on average have 0.25 features associated with them.
This increased the number of pairs input to S-CODE from 14.1
million (12 substitutes per word) to 17.7 million (additional 0.25
features on average for each of the 14.1 million words).

Using similar training settings as the previous section, the addition
of morphological and orthographic features increased the many-to-one
score of the random-substitute model to \ftmto\ and V-measure to \ftvm.
Both these results improve the state-of-the-art in part-of-speech
induction significantly as seen in Table~\ref{tab:results}.

\begin{figure*}[ht] \centering
\vspace*{-25mm}
\includegraphics[width=\textwidth]{hinton.png}
\vspace*{-30mm}
\caption{Hinton diagram comparing most frequent tags and clusters.}
\label{plot-hinton}
\end{figure*}


\documentclass[main.tex]{subfiles}
\begin{document}

\section*{Thu Nov 07 2019}

We are trying to fix the last coefficient in the Einstein equations:
%
\begin{align}
  R_{\mu \nu } - \frac{R}{2} g_{ \mu \nu } = c_2 T_{\mu \nu }
\,.
\end{align}

The \(\frac[i]{1}{2} \) on the LHS is fixed by the conservation of the stress-energy tensor.

We look at a low-energy scenario: \(\rho \ll P\), \(u^{\mu } = (1, \vec{0})^\top\). Then the EFE become \(R_{00} = c_2 \rho /2\).

We consider a stationary metric \(g_{\mu \nu } (\vec{x})\), which does not depend on time. This is consistent with the stuff we usually see: planetary dynamics are quasi-static with respect to the speed of light.

In a LIF we have shown that the Riemann tensor is:
%
\begin{align}
  R_{\gamma \sigma \mu \nu } = \frac{1}{2} \qty(g_{\gamma \nu , \sigma \mu } - g_{\sigma \nu , \gamma \mu } - g_{\gamma \mu , \sigma \nu } + g_{\sigma \mu , \gamma \nu })
\,.
\end{align}

To get the Ricci tensor we need \(R^{\alpha }_{\sigma \mu \nu } = \eta^{\alpha \gamma }R_{\gamma \sigma \mu \nu }\) since we are in the LIF.

We get the Ricci tensor:
%
\begin{align}
  R_{\sigma \nu } &=  \eta^{\alpha \gamma } \qty(g_{\gamma \nu , \sigma \alpha } - g_{\sigma \nu , \gamma \alpha } - g_{\gamma \alpha , \sigma \nu } + g_{\sigma \alpha , \gamma \nu })
\,,
\end{align}
%
from which we can calculate \(R_{00}\):
%
\begin{subequations}
\begin{align}
    R_{0 0 } &= \frac{1}{2} \eta^{\alpha \gamma } \qty(g_{\gamma 0 , 0 \alpha } - g_{0 0 , \gamma \alpha } - g_{\gamma \alpha , 0 0 } + g_{0 \alpha , \gamma 0 })  \\
    &= -\frac{1}{2}\eta^{i j }g_{00, ij}
    = - \frac{1}{2} \sum _{i} g_{00, ii} = -\frac{1}{2} \nabla^2 g_{00}
    \,,
\end{align}
\end{subequations}
%
where only the second term survives since the metric is time-independent, its time derivatives all vanish.

Therefore, our equation becomes \(\nabla^2 g_{00} = c_2 \rho \). We just need to find out what the meaning of \(g_{00}\) is, how it is related to the gravitational potential.
We do it with gravitational redshift: we found that
%
\begin{align}
  \Delta \tau_{A} \approx \Delta \tau_{B} (1 - \Phi_{A} + \Phi_{B})
  \,,
\end{align}
%
if Alice, on the top of a building, is sending photons to Bob who is on the ground. To first order in the field, the expression is equivalent to
%
\begin{align}
    \Delta \tau_{A} \approx \Delta \tau_{B} (1 - \Phi_{A}) (1+ \Phi_{B}) \approx \Delta \tau_{B} \frac{1 + \Phi_{B}}{1 + \Phi_{A}}
\,,
\end{align}
%
which is equivalent to the constancy of
%
\begin{align}
  \frac{\Delta \tau}{1 + \Phi }
\,.
\end{align}

The \(\Delta \tau \) are proper times measured by the observers \(A\) and \(B\) at rest.
For an observer at rest, the spacetime interval is
%
\begin{align}
  -\dd{\tau }^2 = \dd{s}^2 = g_{00} \dd{t^2}
\,,
\end{align}
%
since the \(\dd{x^{i}} \) are null. Therefore, \(\dd{\tau } = \sqrt{-g_{00}}  \dd{t} \).

We have
%
\begin{align}
  \Delta \tau_{A} = \sqrt{-g_{00}(A)} \Delta t
    \qquad \text{and} \qquad
  \Delta \tau_{B} = \sqrt{-g_{00}(B)} \Delta t
\,,
\end{align}
%
but the \(\Delta t\) are the same, because the metric is constant with respect to time! so we get that \(\Delta t = \const\) and specifically it is equal to
%
\begin{align}
  \frac{\Delta \tau }{\sqrt{-g_{00}}}
\,,
\end{align}
%
so we can finally identify \(\sqrt{-g_{00}} \approx 1 + \Phi \), or \(g_{00} = - (1+\Phi )^2 = - (1+ 2 \Phi )\).

We can now bring the Laplacian inside the \(g_{00}\): we get
%
\begin{align}
  \nabla^2 \Phi = \frac{c_2}{2} \rho
\,,
\end{align}
%
Poisson's equation for the gravitational potential.

We know the gravitational potential to be defined by \(- \nabla \Phi = \vec{F_G}\).

We can find Gauss' law for the gravitational field just like we did for the electromagnetic field, substituting \(Q \rightarrow M\) and \(1 / (4 \pi \epsilon_{0}) \rightarrow -G\).
I will also denote the electric and gravitational charges with subscripts \(E\) and \(G\).

The integral form of this equation is
%
\begin{align}
  \int _{\partial V} \dd{\vec{A}} \cdot \vec{E} = \int_V \dd[3]{x} \rho_E
\,,
\end{align}
%
but it can be expressed differentially with the divergence theorem as
%
\begin{align}
  \vec{\nabla} \cdot \vec{E} = \frac{\rho_E}{\epsilon_0 }
\,.
\end{align}

For the gravitational field \(\vec{F_G}\) then we can just substitute:
%
\begin{align}
  \vec{\nabla} \cdot \vec{F_G} = - 4 \pi G \rho_G
\,,
\end{align}
%
and then we can substutite the gravitational potential:
%
\begin{align}
  \vec{\nabla} \cdot ( - \vec{\nabla} \Phi ) = - \nabla^2 \Phi  = - 4 \pi G \rho_G
\,,
\end{align}
%
or \(\nabla^2 \Phi = 4 \pi G \rho_G\).
So this is what we found before, with \(c_2/2 = 4 \pi G \).
So, our constant is \(c_2 = 8 \pi G\). So we get Einstein's equations:
%
\begin{align}
  R_{\mu \nu } - \frac{R}{2} g_{\mu \nu } = 8 \pi G T_{\mu \nu }
\,.
\end{align}

The LHS of these is called the Einstein tensor, \(G_{\mu \nu } \equiv R_{\mu \nu } - \frac[i]{R}{2} g_{\mu \nu }\).

Why do we not write an equation with more derivatives, more indices? We may, but we'd detect these only in the regime of very strong curvature: the relativistic effects are already hard to detect as is!
For now the vanilla EFE have always agreed with experiment.

\section{Geodesics}

In flat spacetime, we know that the curve with \(\dv*[2]{x^{\mu }}{\tau }=0\) stationarizes \(\tau = \int \dd{\tau }\).

Now we will do the exact same thing, except that \(\dd{\tau }\) will be calculated with \(g_{\mu \nu }\) instead of \(\eta_{\mu \nu } \).

We perturb our path \(x^{ \mu }(\lambda )\) as \(x^{\mu } + \delta x^{\mu }\), and we want to set to zero the first functional derivative \(\fdv*{\tau_{AB}}{x^{\mu }}\). This means that to first order
%
\begin{align}
  \delta \qty(\int _{0}^{1} \dd{\sigma }\sqrt{-g_{\alpha \beta } \dv{x^{\alpha }}{\sigma }\dv{x^{\beta }}{\sigma }} )=0
\,,
\end{align}
%
where we performed a reparametrization of the curve. Computing away, with the notation \(u^{\mu } = \dv*{x^{\mu }}{\sigma }\):
%
\begin{subequations}
\begin{align}
  \delta \tau_{AB} &= \int_0^{1} \dd{\sigma} \qty(
    \frac{{ -\delta g_{ \alpha \beta } u^{\alpha  } u^{\beta }}}{2 \sqrt{- u^2}} +
      \frac{-g_{\alpha \beta } u^{\alpha } \dv{ \delta x^{\beta }}{\sigma } }{\sqrt{- u^2}}
    )  \\
    &= - \frac{1}{2} \int_0^1  \dd{\sigma } \qty(
        \delta g_{\alpha \beta } \dv{x^{\alpha }}{\sigma }
        \dv{x^{\beta }}{\tau }
        + 2 g_{ \alpha \beta } \dv{x^{\alpha }}{\tau }
        \dv{ \delta x^{\beta }}{\sigma }
    )  \\
    &= - \frac{1}{2} \int_0^1 \dd{\sigma } \qty(
        \partial_{\gamma } g_{\alpha \beta } \delta x^{\gamma } \dv{x^{\alpha }}{\sigma }
        \dv{x^{\beta }}{\tau }
        -2 \dv{}{\sigma } \qty(g_{\alpha \beta } \dv{x^{\alpha }}{\tau }) \delta x^{\beta }
    )  \\
    &= - \frac{1}{2} \int_0^1 \dd{\tau } \qty(
        \partial_{\gamma } g_{\alpha \beta } \delta x^{\gamma } \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau } +
        \dv{}{\tau } \qty(g_{ \alpha \beta } \dv{x^{\alpha }}{\tau } \delta x^{ \beta } )
    )  \\
    & \int \dd{\tau } \qty(
        -\frac{1}{2} \partial_{\gamma }g_{\alpha \beta } \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau } +
        \dv{}{\tau } \qty(g_{\alpha \gamma } \dv{x^{\alpha }}{\tau }) \delta x^{\gamma }
    )
\,,
\end{align}
\end{subequations}
%
where we applied the product rule, identified two symmetric terms in the velocity, used the identity
%
\begin{align}
  \frac{1}{\sqrt{- u^2}} \dv{}{ \sigma } = \dv{}{\tau }
\,,
\end{align}
%
which follows from
%
\begin{align}
u^{\alpha } = \dv{x^{\alpha }}{\sigma }
\implies
u^2 = \frac{ \dd{x^{\alpha }} \dd{x_{\alpha }}}{ \dd{\sigma^2}} = - \frac{ \dd{\tau^2 }}{ \dd{\sigma^2}}
\,.
\end{align}

Also, we expanded the metric using:
%
\begin{align}
  \delta g_{\alpha \beta } = \partial_{\gamma } g_{ \alpha \beta } \delta x^{\gamma }
\,,
\end{align}
%
where we did not need to introduce a covariant derivative since we are just Taylor expanding.

Also, we integrated by parts (without boundary terms since the path variation vanishes at the path boundary), changed variables, and finally gotten an expression which must vanish for any path, therefore we get that
%
\begin{align}
    -\frac{1}{2} \partial_{\gamma }g_{\alpha \beta } \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau }
    + \dv{}{\tau } \qty(g_{\alpha \gamma } \dv{x^{\alpha }}{\tau }) = 0
\,,
\end{align}
%
so we can expand the derivative: we get
%
\begin{align}
  0= - \frac{1}{2} \partial_{\gamma }g_{\alpha \beta } \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau } +
  \partial_{\beta }g_{\alpha \gamma } \dv{x^{\beta }}{\tau } \dv{x^{\alpha }}{\tau } + g_{\alpha \gamma } \dv[2]{x^{\alpha }}{\tau }
\,,
\end{align}
%
where the second term can be symmetrized in \(\alpha \beta \):
%
\begin{align}
  0= g_{\alpha \gamma } \dv[2]{x^{\alpha }}{\tau } +
  \qty(\frac{1}{2} \partial_{\beta }g_{\alpha \gamma } + \frac{1}{2}\partial_{\alpha } g_{\beta \gamma }) - \frac{1}{2}\partial_{\gamma }g_{\alpha \beta } \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau }
\,,
\end{align}
%
so we get
%
\begin{align}
  0= g_{\alpha \gamma } \dv[2]{x^{\alpha }}{\tau } +
  \frac{1}{2}\qty(g_{\alpha \gamma , \beta } + g_{\beta \gamma , \alpha } - g_{\alpha \beta , \gamma })
  \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau }
\,,
\end{align}
%
which, raising an index and identifying the Christoffel symbols, gives us
%
\begin{align}
  0 = \dv[2]{x^{\gamma }}{\tau } + \Gamma^{\gamma }_{\alpha \beta } \dv{x^{\alpha }}{\tau } \dv{x^{\beta }}{\tau }
\,,
\end{align}
%
the \emph{geodesic equation}.

It can be written also as \(u^{\mu } \nabla_{\mu }u^{\nu } = a^{\nu } = 0\).

\end{document}


\section{Geometry optimization}\index{Geometry!BFGS optimizer}
The default geometry optimizer in MOPAC uses Baker's EigenFollowing method.  If
this is {\em not} wanted, for example, if there is a need to reduce memory
demands, then the Broyden Fletcher Goldfarb Shanno method can be used.

The most common use of MOPAC is for geometry optimization. This involves
starting with an approximation to the desired geometry and, by calculating the
forces acting on the system, changing the geometry so as to lower the total
energy. The objective of geometry optimization is to achieve a structure in
which all the atoms are at equilibrium, that is, one in which the forces acting
on every atom are very small, and in which the second derivatives are
everywhere positive.  Such a geometry is called a ground state stationary
point.

\input{t_ef}
\input{t_bfgs}

\subsection{Optimization of one unknown}
If a system has exactly one coordinate to be optimized, then obviously one
line-search will optimize the geometry.  Because of this, the geometry
optimization is done a little differently.  Given the initial geometry, the
$\Delta H_f$ is calculated, and the line-search started. Unlike the normal
line-search, however, the search is not stopped when the minimum is almost
reached, instead, the minimum is located with quite high precision.  After the
line-search is complete, the gradients are not \index{GRADIENTS} calculated
(unless requested by \comp{GRADIENTS}). Instead, it is assumed that the
gradient is small, and the results are output.  This saves some time. However,
if \comp{GRADIENTS} is {\em not} present, and the geometry is not at a
stationary point (because other coordinates are not optimized), then the
warning message that the geometry is not at a stationary point will not be
printed.

\index{Line search|)}

\input{t_geometry2}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}% http://ctan.org/pkg/amsmath

\title{CS2C Assignment 2}
\author{Angus Yip}
\date{April 24th, 2017}

\usepackage{natbib}
\usepackage{graphicx}

\begin{document}

\maketitle

\section*{2.1}

    The ascending order for the functions are:

    \[ \frac{2}{N}, 37, \sqrt{N}, N, N\log{\log{N}}, N\log{N}, N\log{N^2}, \]
    \[ N(\log{N})^2, N^{1.5}, N^2, N^2\log{N}, N^3, 2^\frac{N}{2}, 2^N \]

    \noindent Furthermore, the functions \( N\log{N} \) and \( N\log{N^2} \)
    grows at the same rate.

\section*{2.2}
    \renewcommand{\labelenumi}{\alph{enumi}}
    \begin{enumerate}
        \item true.
        \item false.
        \item false.
        \item false.
    \end{enumerate}

\section*{2.6}
    %\renewcommand{\labelenumi}{\alph{enumi}}
    \begin{enumerate}
        \item \( 2^{2^{N - 1}} \)
        \item
            \begin{equation*}
            \begin{split}
                2^{2^{N - 1}} & = D \\
                2^{N - 1}\log{2} & = \log{D}\\
                2^{N - 1} & = \frac{\log{D}}{\log{2}}\\
                (N - 1)\log{2} & = \log{\log{D}} - \log{\log{2}}\\
                N & = \frac{\log{\log{D}}}{\log{2}} - \frac{\log{\log{2}}}{\log{2}} + 1\\
                N & = O(\log{\log{D}})
            \end{split}
            \end{equation*}

    \end{enumerate}

\section*{2.7}
    \begin{enumerate}
        \renewcommand{\labelenumi}{\arabic{enumi}}
        \item
            \begin{enumerate}
                \item
                \item
                \item
            \end{enumerate}
        \item
            \begin{enumerate}
                \item
                \item
                \item
            \end{enumerate}
        \item
            \begin{enumerate}
                \item
                \item
                \item
            \end{enumerate}
        \item
            \begin{enumerate}
                \item
                \item
                \item
            \end{enumerate}
        \item
            \begin{enumerate}
                \item
                \item
                \item
            \end{enumerate}
        \item
            \begin{enumerate}
                \item
                \item
                \item
            \end{enumerate}

    \end{enumerate}

\section*{2.11}
    %\renewcommand{\labelenumi}{\alph{enumi}}
    \begin{enumerate}
        \item
            \begin{equation*}
            \begin{split}
                \frac{500}{100} & = \frac{N}{.5}\\
                N & = 2.50 ms
            \end{split}
            \end{equation*}
        \item
            \begin{equation*}
            \begin{split}
                \frac{500\log{500}}{100\log{100}} & = \frac{N}{.5}\\
                N & = 3.37 ms
            \end{split}
            \end{equation*}
        \item
            \begin{equation*}
            \begin{split}
                \frac{500^2}{100^2} & = \frac{N}{.5}\\
                N & = 12.5 ms
            \end{split}
            \end{equation*}
        \item
            \begin{equation*}
            \begin{split}
                \frac{500^3}{100^3} & = \frac{N}{.5}\\
                N & = 62.5 ms
            \end{split}
            \end{equation*}
    \end{enumerate}

\section*{2.13}
    %\renewcommand{\labelenumi}{\alph{enumi}}
    \begin{enumerate}
        \item
            \( \sum_{i=0}^{N} a_{i}x^i \) takes \(i\) operations to compute.
            \begin{equation*}
            \begin{split}
                & \sum_{i=0}^{N} i + N\\
                = &\, \frac{N(N + 1)}{2} + N\\
                = &\, O(N^2)
            \end{split}
            \end{equation*}
        \item
            From section 2.4.4, the new algorithm takes \( \log_2{i} \) operations
            \begin{equation*}
            \begin{split}
                & \sum_{i=0}{N} {\log_2{i}} + N\\
                = &\, \log_2{N!} + N\\
                = &\, O(\log{N!})
            \end{split}
            \end{equation*}
    \end{enumerate}

\section*{2.15}
    An efficient algorithm in the case would be a binary search, in which the runtime would be at the \( O(\log{N}) \) level.

\section*{2.16}
    Code attached as assignment2.cpp.

\end{document}


    \filetitle{nfitted}{Number of data points fitted in VAR estimation}{VAR/nfitted}

	\paragraph{Syntax}\label{syntax}

\begin{verbatim}
N = nfitted(V)
\end{verbatim}

\paragraph{Input arguments}\label{input-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{V} {[} VAR {]} - Estimated VAR object.
\end{itemize}

\paragraph{Output arguments}\label{output-arguments}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  \texttt{N} {[} numeric {]} - Number of data points (periods) fitted
  when estimating the VAR object.
\end{itemize}

\paragraph{Description}\label{description}

\paragraph{Example}\label{example}


\section{Thermodynamics constrains physical processes}
The first law of thermodynamics is the law of energy conservation, energy can neither be created or destroyed but that energy can only be converted from one type to another.

\begin{equation}
    \Delta Energy_surr + \Delta Energy_sys = 0
    \label{eq:Thermo_Sys_Surr}
\end{equation}

Due to the system being closed system only work and heat can be transferred between it and the surroundings.

\begin{equation}
    \Delta Energy_surr = - Q_surr - W_surr = - Q - W
    \label{eq:Thermo_Work_Heat_1}
\end{equation}
\begin{equation}
    \Delta Energy_sys = Q_sys + W_sys = Q + W
    \label{eq:Thermo_Work_Heat_2}
\end{equation}

Conventionally energy that flows into the system is taken as positive.
In a closed system there can also be processes that lead to changes in internal energy. These changes are the sum of the changes in heat and work occurring in the system.

\begin{equation}
    U_t \equiv Q + W
    \label{eq:Thermo_U}
\end{equation}

Where $U_t$ is the total change in internal energy. The differential form of equation \ref{eq:Thermo_U} is shown in \ref{eq:Thermo_U_extended}.

\begin{equation}
    dU_t = dQ + dW
    \label{eq:Thermo_U_small}
\end{equation}

Extensive variables $U_t$ and the volume (V) depend on the metabolites concentrations in the system. Intensive variables like temperature (T) and pressure (p) are concentration independent. Extensive and intensive variables are related through the mass (m) or the moles (n), as shown in \ref{eq:Thermo_nU} and \ref{eq:Thermo_nV}.

\begin{equation}
    U_t = mU or U_t = nU
    \label{eq:Thermo_nU}
\end{equation}
\begin{equation}
    V_t = mV or V_t = nV
    \label{eq:Thermo_nV}
\end{equation}

With this equation \ref{eq:Thermo_U_small} can be extended to \ref{eq:Thermo_U_extended}:

\begin{equation}
    d(nU_t) = dQ + dW
    \label{eq:Thermo_U_extended}
\end{equation}

Where Q and W always pertain to the total change of heat and work.

Equation \ref{eq:Thermo_U_extended} can be used to introduce ehnthalpy H. For a closed system under going a mechanically reversible process the mole of the system can be added by equation \ref{eq:Thermo_rev}.

\begin{equation}
    dW = - Pd(nV)
    \label{eq:Thermo_rev}
\end{equation}

Assuming constant pressure eq. \ref{eq:Thermo_rev} can then substitute into eq. \ref{eq:Thermo_U_extended} to yield eq. \ref{eq:Thermo_substituted}.

\begin{equation}
    d(nU_t) = dQ - Pd(nV)
    \label{eq:Thermo_substituted}
\end{equation}

This equation can be solved for dQ as in \ref{eq:Thermo_Q_solved_1} and \ref{eq:Thermo_Q_solved_2}.

\begin{equation}
    dQ = d(nU) + Pd(nV)
    \label{eq:Thermo_Q_solved_1}
\end{equation}
\begin{equation}
    dQ = d(n(U + PV))
    \label{eq:Thermo_Q_solved_2}
\end{equation}

Defining enthalpy as H $\equiv$ U + PV, the heat transferred to a system can be described by eq. \ref{eq:Thermo_Enthalpy} under the condition of constant pressure.

\begin{equation}
    \fbox{dQ = d(nH)}
    \label{eq:Thermo_Enthalpy}
\end{equation}

The second law of thermodynamics deals with entropy. In general the total change in entropy due to a process, as shown in \ref{eq:Thermo_Entropy} can never be negative.

\begin{equation}
    \fbox{\Delta S_total \geq 0}
    \label{eq:Thermo_Entropy}
\end{equation}

A consequence of \ref{eq:Thermo_Entropy} is that every process will only happen in the direction where the total entropy change is not negative. This is also limited by a value of 0 for reversible reactions.

\section{Gibbs free energy}
Gibbs free energy change of a reaction can be derived by combining the first and second law of thermodynamics. Here the reaction is assumed to be in thermal and mechanical equilibrium with it's surroundings. Hence, both temperature (T) and pressure (p) are constant.

The entropy change of the surroundings is shown in eq. \ref{eq:Gibbs_Entropy}.

\begin{equation}
    dS_surr = - dQ/T
    \label{eq:Gibbs_Entropy}
\end{equation}

The 2nd law dictates that the total change of entropy - the sum for the surroundings and the system - has to be non-negative:

\begin{equation}
    dS_t + dS_surr \geq 0
    \label{eq:Gibbs_Enthalpy}
\end{equation}

Substituting \ref{eq:Gibbs_Entropy} into \ref{eq:Gibbs_Enthalpy} and simplifying yields \ref{eq:Gibbs_Q}.

\begin{equation}
    dS_t - dQ/T \geq 0
    \label{eq:Gibbs_0}
\end{equation}
\begin{equation}
    dQ \leq TdS_t
    \label{eq:Gibbs_Q}
\end{equation}

The first law can be used to substitute dQ as ahown in \ref{eq:Gibbs_U} and \ref{eq:Gibbs_dQ}:

\begin{equation}
    U_t = dQ + dW = dQ - Pd(nV_t)
    \label{eq:Gibbs_U}
\end{equation}
\begin{equation}
    dQ = dU_t + PdV_t
    \label{eq:Gibbs_dQ}
\end{equation}

Combining the equations \ref{eq:Gibbs_Q}, \ref{eq:Gibbs_dQ} and rearranging of the equation:

\begin{equation}
    dU_t + PdV_t \leq TdS_t
    \label{eq:Gibbs_dU_t}
\end{equation}
\begin{equation}
    dU_t + PdV_t - TdS_t \leq 0
    \label{eq:Gibbs_leq_0}
\end{equation}

Substituting U + PV with H.

\begin{equation}
    dH - TdS_t \leq 0 (4.9)
    \label{eq:Gibbs_dH}
\end{equation}

Assuming constant T, \ref{eq:Gibbs_dH} can be written as \ref{eq:Gibbs_dG} . It can be substituted with G.

\begin{equation}
    \fbox{G = H - TS}
    \label{eq:Gibbs_G}
\end{equation}
\begin{equation}
    dG \leq 0
    \label{eq:Gibbs_dG}
\end{equation}

Equation \ref{eq:Gibbs_dG} shows that any spontaneous reaction will lead to a decrease of Gibbs free energy if they happen under constant pressure and temperature. Thus $\Delta$G can be used to predict the direction of a reaction.

\section{Transporters}
Transporters are a group of membrane proteins that facilitate the diffusion/transport of molecules across the lipid membrane of cells. These proteins can be broadly categorized into three groups, passive, primary active and secondary active (\cite{alva2020insights}). The difference between these is what process they utilize to power the substrate transport. Passively substrate can only flow from a high concentration to a lower one, while active transport can work against these gradients with the use of energy.

Small, lipophilic molecules can cross the lipid membrane by simple diffusion along the concentration gradient. For bigger and hydrophilic molecules this is not possible, thus cell have two types of passive transporter to allow those molecules to diffuse.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/Passive.pdf}
    \caption{A scheme of the three types of passive transport. from right to left: carrier protein, channel protein, simple diffusion. Also shown is the concentration gradient for a substrate.}
    \label{fig:Passive}
\end{figure}

One of those are channel proteins which are pores in the membrane. These pores have a hydrophilic inside, providing a substrate nonspecific way inside the cell for hydrophilic molecules. Another protein for passive transport are Carrier proteins. These are substrate specific and work by having that substrate bind to it. This leads to a change in the proteins confirmation pushing the substrate through and opening up on the other side.

The active transport is a way to work around the concentration gradients for molecules. Primary active transporters do that by using the energy released by hydrolysis of ATP to ADP and phosphate. For this the ATP binds to a transporter and when the substrate also binds this leads to a conformation change. Through hat change both the hydrolysis and the transport of the substrate to the other side are forced. The energy released is needed to enact the confirmation change.
This kind of transport generally happens either as a uniport where one substrate is transported or as a contraport. A contraport transports two different substrates against their respective gradients in opposite directions.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/PActive.pdf}
    \caption{Scheme of primary active transport. Both uniport and contraport as well as the corresponding concentration gradients are shown.}
    \label{fig:PActive}
\end{figure}

Secondary active transport is another way to transport a substrate against its concentration gradient. The difference between primary and secondary active transport is that the secondary active transport does not use energy in for of ATP. It uses the coupling of the transport two substrates. one is transported with its gradient the other one against it. For this both of the substrates need to bind to the protein which then leads to a change in conformation.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/SActive.pdf}
    \caption{Scheme of secondary active transporters. Shown are antiport and symport.}
    \label{fig:SActive}
\end{figure}

Secondary active transport can happen as an antiport where the two substrates are transported to opposite side of the membrane. It can also happen as a symport, where both of them are transported to the same side.

\section{Glycolysis}
Glycolysis is a cellular reaction chain in which the breakdown of glucose is coupled with the synthesis of the main energy metabolite ATP.
It consists of 2 part the upper glycolysis and the lower glycolysis. The upper glycolysis splits the C6 glucose into 2 C3 glyceraldehyd by using energy to bind phosphate to the
sugars. While the lower glycolysis turns the glyceraldehyd molecules into pyruvate, while recovering ATP.
The most commonly discussed glycolysis is the Embden-Meyerhof-Parnas (EMP) pathway, due to it being nearly ubiquitous in among eukaryotes \cite{flamholz2012glycotic}.
In this gycolisis pathway the glucose, a C6 sugar is broken down into 2 C3 units. The product of this reaction chain are 2 pyruvates, 4 ATP and 2 ADP.

In comparison the Entner–Doudoroff (ED) pathway is a natural alternative glycolysis pathway found in prokaryotes and together with the EMP pathway is the most common glycolysis pathway in bacteria. The general the process is similar to that of the EMP glycolysis, with it breaking down glucose into 2 pyruvates thus sharing the same net reaction (\ref{eq:Glycolysis_net_reaction}).

\begin{equation}
    D-glucose + n ADP + n P_i \rightarrow 2 Pyruvate + n ATP + n H_2O
    \label{eq:Glycolysis_net_reaction}
\end{equation}

The difference occurs in the upper glycolysis reactions and the energy yield of the lower glycolysis. The differing energy yields is a result of the upper glycolysis splitting glucose into 1 glyceraldehyd molecule and one pyruvate. This means only one molecule goes through the lower glycolysis leading to only half the ATP yield, see fig. \ref{fig:glycolysis_scheme}.
The two pathways can also differ in the specific amount of NAD/NADH and NADP/NADPH used.

According to \cite{flamholz2012glycotic} the EMP pathway is thermodynamically more constrained than the ED pathway, with some reactions (fructose 1,6-bisphosphate aldolase (fba), triose-phosphate isomerase (tpi), glyceraldehyde 3-phosphate dehydrogenase (gapdh), phosphoglycerate kinase (pgk), and phosphoglycerate mutase (pgm)) acting as thermodynamic bottlenecks. The ED pathway prevents these bottlenecks by replacing unfavorable reactions with more exergonic reactions.
In comparison the EMP reactions have higher enzyme concentrations to increase the catalytic rate and minimize the impact of the lower thermodynamic driving force. This means the EMP pathway is enzymatically more expensive, which leads to a trade off between cost and ATP yield between the two pathways.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/glycolysis.pdf}
    \caption{Schematic of the EMP and Ed pathways, showing upper and lower glycolysis.}
    \label{fig:glycolysis_scheme}
\end{figure}

\section{Self replicator model}
This self replicator model is a coarse grained representation of cellular metabolism based on the model by Molenaar, et. al 2009 (\cite{molenaar2009shifts}). It consists of passive transporters, ribosomes, metabolic enzymes and enzymes for lipid synthesis enzymes.

The model employs Mechaelis-Menten-kinetics for the reaction rates and is based on several physiological assumptions. The cell has a limited intracellular density, which constrains the amount of ribosomes, metabolic enzymes and lipid synthesis enzymes that can be present in the cell. The growth rate of the cell is limited by the membrane proportions, which consists of the surface to area ratio $\beta$ and the composition of the membrane namely the ratio of transporters and lipids, where the amount of transporters can not be higher than that of the lipids. This is so the structural integrity of the cell can be simulated because a real cell would generally not be able to survive with a membrane that consists of more transporters and other enzymes than lipids.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/Molenaar.pdf}
    \caption{Scheme of the basic seflreplicator model consisting of the most important parts.}
    \label{fig:Molenaar}
\end{figure}

In this model the precursor is used as the educt for both the lipid and enzyme synthesis. The lipid synthesising enzyme uses the precursor to create the lipids for the cell membrane while the ribosomes use it to create all the enzymes including the ribosomes them self. Due to the ribosomes producing not only the transporters, lipid synthesising enzymes, the enzymes used for metabolic reactions but also themselves this model is considered a self replicator model.

This basic model does not take overflow metabolism into account, it introduces the basic structure and boundaries for the model, to show the underlining relationships between the different parts. The model shows that there is a direct correlation between the substrate concentration and the growth rate. Specifically that an increase in substrate abundance leads to a higher growth rate, which occurs because at low substrate concentrations the cell has need of a lot more transporters to increase the chance of them coming into contact with a substrate molecule.

The connection between a growth rate and substrate concentration shown by this model in figure 2.1 can also be seen in the experimental data from Schulze, et. al. 1964 \cite{schulze1964relationship} in figure 2.2. This shows that the model behaves in a way that resembles in vivo data.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/fig2com.PNG}
    \caption{1. The results of the model, showing that with an increase of the substrate concentration the growth rate increases up to a certain point where it starts to plateau. 2. Experimental data from Schulze, et. al. 1964 \cite{schulze1964relationship} which shows a similar behaviour to the data from the model, meaning this is not just a random behaviour in the model.}
    \label{fig:fig2com}
\end{figure}

\ref{fig:Figure_3Cband} With a higher substrate concentration the chance of a transporter interacting with a substrate molecule rises, thus the cell can now spend more resources for other processes leading to a higher growth rate.
In the model these processes would be more synthesis of metabolic enzymes, ribosomes, lipid synthesis enzymes, as well as lipids. This shift in enzyme concentrations inside the cell can also be seen in the model.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/Figure_3Cband.pdf}
    \caption{}
    \label{fig:Figure_3Cband}
\end{figure}

The lipid synthesis enzymes plateaus fast due to the cell only being able to grow at a specific maximum rate, anything above that would be risking the integrity of the cells stability. The amount of metabolic enzyme increases as well so that the increasing concentration of substrate inside the cell can be used effectively. But the increase in the production for these enzymes forces the cell to drastically increase the amount of ribosomes, so that the total enzyme amounts can be upheld over a long period of time.
This increase in overall enzyme synthesis by the ribosomes - and thus the growth rate - also means that a higher rate in ribosome activity is observable.

\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/Figure_3E.pdf}
    \caption{An increase in growth rate is linked to an increase in ribosome synthesis rate.}
    \label{fig:Figure_3E}
\end{figure}

\section{Factorization of rate equations}
See \cite{noor2013note} for details. The flux of a reaction can be factorized into three
conceptually distinct terms: kinetic ($F_K$), thermodynamic ($F_T$) and regulatory ($F_R$),
as shown in Equation \ref{eq:flux_factorization}.
\begin{equation}
    v = F_K \times F_T \times F_R
    \label{eq:flux_factorization}
\end{equation}

Assuming that saturation and regulation effects can be ignored, i.e. the enzyme is substrate
saturated, and if $\Delta_r G' \leq 0$, the forward flux can be expressed as in Equation \ref{eq:thermokinetic_flux_forward}.
\begin{equation}
    v_\text{forward} \leq \underbrace{\text{kcat} \times E_i}_{F_K} \times \underbrace{\left(1 - \exp \left( \frac{\Delta_r G'}{RT} \right)\right)}_{F_T}
    \label{eq:thermokinetic_flux_forward}
\end{equation}

Likewise, if $\Delta_r G' \geq 0$, the backward flux can be expressed as Equation \ref{eq:thermokinetic_flux_backward}.
\begin{equation}
    v_\text{backward} \leq -\underbrace{\text{kcat}' \times E_i}_{F_K} \times \underbrace{\left(\exp \left( \frac{-\Delta_r G'}{RT}\right) - 1\right)}_{F_T}
    \label{eq:thermokinetic_flux_backward}
\end{equation}

Taken together, the flux of a reaction can be bounded by Equation \ref{eq:thermokinetic_flux_bound},
with the $\min$ and $\max$ ensuring that the thermodynamic factors are well behaved.
\begin{equation}
    \text{kcat}' \times E_i \times \min{\left(0, \exp \left( \frac{-\Delta_r G'}{RT}\right) - 1\right)} \leq v \leq \text{kcat} \times E_i \times \max \left(0, 1 - \exp \left( \frac{\Delta_r G'}{RT} \right)\right)
    \label{eq:thermokinetic_flux_bound}
\end{equation}

Assuming that the forward and reverse $\text{kcat}$'s are the same, and that flux is always
at its boundary (parsimonious enzyme usage), Equation \ref{eq:thermokinetic_flux_bound} can be
simplified to Equation \ref{eq:thermokinetic_flux_simplified}.
\begin{equation}
    v = \text{kcat} \times E_i \times \text{sign}\left(\Delta_r G' \right) \times \left(1 - \exp \left( \frac{-|\Delta_r G'|}{RT} \right)\right)
    \label{eq:thermokinetic_flux_simplified}
\end{equation}

Furthermore, to avoid the usage of indicator variables in the subsequent optimization
problems, the thermodynamic term will be approximated, as shown in Equation
\ref{eq:thermokinetic_tanh_approximation}, where $p$ is a fitting variable.
\begin{equation}
    \tanh \left(p \cdot x \right) \approx  \text{sign}\left( x \right) \times \left(1 - \exp \left( x \right)\right)
    \label{eq:thermokinetic_tanh_approximation}
\end{equation}

The fit is shown in Figure \ref{fig:tanh_approximation}.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{imgs/tanh_approximation.pdf}
    \caption{Fit of Equation \ref{eq:thermokinetic_tanh_approximation} where $p=-0.7$.}
    \label{fig:tanh_approximation}
\end{figure}

\section{Thermokinetic algorithm}
It is desirable to combine the MOMENT algorithm \cite{adadi2012prediction} with thermodynamic
constraints, since such a formulation will necessarily be more physiologically representative
of the underlying cellular metabolism. Equation \ref{eq:thermo_moment} shows this formulation,
including the thermodynamic approximation of Equation \ref{eq:thermokinetic_tanh_approximation}.
\begin{equation}
\begin{aligned}
& \underset{E, c, \Delta_r G^0}{\text{maximize}}
& & c^T v \\
& \text{subject to}
& & S v = 0 \\
& & & \Delta_r G_i = \Delta_r G_i^0 + R T \sum_j\nu_j\log\left(c_j \right) \\
& & & v_i = \text{kcat}_i \times E_i \times \tanh \left(\frac{\Delta_r G_i}{RT} \right)\\
& & & \sum_i E_i \leq \text{protein fraction}\\
& & & c_{j, \text{LB}} \leq c_j \leq c_{j, \text{UB}}\\
\end{aligned}
\label{eq:thermo_moment}
\end{equation}
Where $c$, $v$, and $S$ retain their usual meaning from FBA, $E_i$ is the enzyme
concentration of reaction $i$. When a constraint is indexed by $i$, it applies to each
reaction in the model. Each reaction $i$ is also associated with a change in standard Gibbs free
energy, $\Delta_r G_i^0$.


\subsection{F test for additional variables}


\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{url}
\newcommand{\vwi}{{\bf w}_i}
\newcommand{\vw}{{\bf w}}
\newcommand{\vx}{{\bf x}}
\newcommand{\vy}{{\bf y}}
\newcommand{\vxi}{{\bf x}_i}
\newcommand{\yi}{y_i}
\newcommand{\vxj}{{\bf x}_j}
\newcommand{\vxn}{{\bf x}_n}
\newcommand{\yj}{y_j}
\newcommand{\ai}{\alpha_i}
\newcommand{\aj}{\alpha_j}
\newcommand{\X}{{\bf X}}
\newcommand{\Y}{{\bf Y}}
\newcommand{\vz}{{\bf z}}
\newcommand{\msigma}{{\bf \Sigma}}
\newcommand{\vmu}{{\bf \mu}}
\newcommand{\vmuk}{{\bf \mu}_k}
\newcommand{\msigmak}{{\bf \Sigma}_k}
\newcommand{\vmuj}{{\bf \mu}_j}
\newcommand{\msigmaj}{{\bf \Sigma}_j}
\newcommand{\pij}{\pi_j}
\newcommand{\pik}{\pi_k}
\newcommand{\D}{\mathcal{D}}
\newcommand{\el}{\mathcal{L}}
\newcommand{\N}{\mathcal{N}}
\newcommand{\vxij}{{\bf x}_{ij}}
\newcommand{\vt}{{\bf t}}
\newcommand{\yh}{\hat{y}}
\newcommand{\code}[1]{{\footnotesize \tt #1}}
\newcommand{\alphai}{\alpha_i}

\title{Large-Scale Image Pattern Recognition\\
Parallel Machine Learning Implementation}

\author{(According to first name's initial's order in English vocabulary )\\
Haitang Hu, Huizhan Lv, Jian Jin, Tianyi Chen}

\date{\today}

\begin{document}
\maketitle

\begin{abstract}
Image recognition requires high workload because of its intrinsic complex feature space. When it comes to large amount of pictures, common machine learning algorithm shows obvious limitation in pattern recognition in terms of computation efficiency, resource occupation, etc. Thus techniques developed specifically to deal with large-scale data processing could be expected to have obviously superior performance. In this project we implement parallel SVM based on PCA in large-scale image pattern recognition and verify this scheme works in such kind of task with favourable performance as we have expected.
\end{abstract}

\section{Background and Motivation}
Pattern recognition concerning image has the mark of complex feature space. Consider a $32\times 32$ color picture, each picture will have 3072 features taking RGB scheme into account. For a dataset of 60,000 pictures, it is quite a huge workload for the classification task.\\
\\
However the ability to deal with classification in image recognition is of quite significance. For instance, when we  look for images by inputting keyword "car" through a search engine, clearly it should distinguish between distinct types of objects and we don't expect a picture of a boat in the search result.\\
\\
We aim to develop an algorithm which makes classification of large-scale image dataset in an efficient manner, that is to say, gives out result quickly with a high level of accuracy.
\section{Algorithm}
\subsection{Preprocessing}
We use PCA for preprocessing. In PCA, the goal is to project the data $\vx$ with dimensionality $D$ onto a space having dimensionality $M <D$ while  maximizing the variance of the projected data.\\
\\
Suppose $S$ is the data covariance matrix defined by\begin{equation}
S=Cov(\vx)
\end{equation}
Let $\{z_i\}$ represent $M$ linear combinations of our original $D$ predictors:
\begin{equation}
z_m=\vw_m^T\vx=\sum_{j=1}^D w_{jm} x_j,~~~~m=1,2,...,M
\end{equation}
We now maximize the projected variance \begin{equation}
Var(z_1)=\vw^T_1S\vw_1
\end{equation}
Constraint condition is $\vw^T_1\vw_1=1$. Make an unconstrained maximization of
\begin{equation}
w^T_1Sw_1-\lambda_1(w_1^T w_1-1).
\end{equation}
where $\lambda_1$ is a Lagrange multiplier. The optimization result is \begin{equation}
S\vw_1 = \lambda_1\vw_1
\end{equation}
So\begin{equation}
w_1^TSw_1 = \lambda_1
\end{equation}
which reveals that the variance will be a maximum when we set $\vw_1$ equal to the eigenvector having the largest eigenvalue $\lambda_1$. This eigenvector $\vw_1$ is the first principal component.\\
\\
The second principal component $\vw_2$ should also maximize variance, be of unit length, and be orthogonal to $w_1$. We could find that $\vw_2$ should be the eigenvector of $S$ with the second largest eigenvalue. Similarly, we can show that the other dimensions are given by the eigenvectors with decreasing eigenvalues.\\
\\
Since there are too many features in the task, we use PCA as preprocessing, which reduces 3072 features to 100 features. We projected all the features on the first 100 eigenvectors thus acquiring the data for classification. Also, to facilitate the computation of eigenvectors, we use SVD \begin{equation}
X=U\Sigma V
\end{equation}
where $X$ is data matrix. Then the eigenvectors of covariance matrix of $X$ will be the columns of the matrix $V$.
\subsection{Processing}
We use parallel SVM with Gaussian kernel for classification, which runs on four seperate machines. The standard SVMs are binary classifiers using a linear decision boundary with disciminant function \begin{equation}
g(\vx)=\vw^T\vx^t+w_0
\end{equation}
of which the decision function is \begin{equation}
y_{new}=\text{sign}(\vw^T\vx_i+w_0)
\end{equation}
In our tasks there is no guarantee that the dataset is linear separable, so we introduce kernel function which allow non-linear dicision boundaries.\\
\\
Kernel is based on transformation of original features, define the basis functions \begin{equation}
\vz =\phi(\vx)~\text{where}~z_j=\phi_j(\vx),~j = 1,...,k
\end{equation}
mapping from the $d$-dimensional $\vx$ space to the $k$-dimensional $\vz$ space where we write the discriminant as\begin{eqnarray}
&g(\vz)=\vw^T\vz\\
&g(\vx)=\vw^T\phi(\vx)=\sum_{j=1}^k w_j\phi_j(\vx)
\end{eqnarray}
The dual in standard SVM problem is now
\begin{equation}
L_d=\sum_i\lambda_i-\frac{1}{2}\sum\limits_i\sum\limits_j \lambda_i\lambda_j y_i y_j \phi(x_i)^T \phi(x_j)
\end{equation}
subject to \begin{equation}
\sum_i \lambda_iy_i=0,~~~~0\leq \lambda_i\leq C,~\forall i
\end{equation}
The idea in kernel machines is to replace the inner product of basis functions, $\phi(x_i)^T\phi(x_j)$, by a kernel function, $K(x_i,x_j)$, thus the kernel function also shows up in the discriminant
\begin{equation}
g(\vx)=\vw^T\phi(\vx)=[\sum_{i=1}^k \lambda_i y_i \phi(\vx_i)^T]\phi(\vx)=\sum_{j=1}^k \lambda_i y_i K(\vx_i,\vx)
\end{equation}
Use this kernel function, we do not need to map it to the new space at all. There are multiple types fo kernels and we use Guassian kernel  \begin{equation}
K(x,x')=exp(-\frac{1}{2}(x-x')^T\Sigma^{-1}(x-x'))
\end{equation}
Besides kernel, we make another extension from binary class to $K-$classes by applying 1-of-K encoding scheme, in which y is a vector of length $K$ containing a single 1 for the correct class and 0 elsewhere. For example, if we have $K=5$ classes, then an input that belongs to class $2$ would be given a target vector: \begin{equation}
y=(0,1,0,0,0)^T
\end{equation}

\subsection{K-means Clustering Algorithm}

Comparing with SVM, K-means is another clustering algorithm.In  we don't know the exact number of labels.  K-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster.
The MLlib on spark has an implementation of K-means in python. Therefore, we can directly use it by importing this package.

\subsection{Percolation Clustering Algorithm}

Besides SVM and k-means, we also realize another clustering algorithm which can be used in the case that we don't know the number of labels(SVM), and we don't have the expectation of number of labels(k-means). This algorithm derived from the percolation algorithm of Newman \cite{Percolation}.  The original algorithm is realized on lattice plane. We modified it to let it suitable for our continuable space. \\
In our derived algorithm, each data point i has other data points which are within the circle of radius  $R$ centred at data point i as its neighbors.\\
Radius $R$ is our parameter. After each data determining their neighbors, a directed graph will be formed. Then the derived percolation clustering algorithm will be realized on such directed graph to get the clusters.

\section{Parallelization of Algorithm}

\subsection{Parallelization of PCA}

On the python framework of spark, we can esaily use the map() function to implement the parallel computations of normalizing data, and covariance matrix. The key of implementation of PCA is how to parallel compute the SVD of the covariance matrix, since the svd function in numpy package cannot be used in rdd object. In order to solve the parallelization of SVD decomposition, we tried to implment it by one-side Jacobi SVD algorithm, and another SVD algorithm from freeman lab,thunder python package.\\

One-sided Jacobi Algorithm can be used for singular value decomposition\cite{Jacobi}. We implement it by python, it works well on matrix of array type. However, when our one-sided Jacobi Algorithm was run on rdd, it cannot load the dimension of seperate rdd correctly. The code of one-sided Jacobi is on our github. \\

Another way is to use the svd algorithm on spark based on thunder python package from freeman lab. The freeman's svd algorithm cannot work directly. We made some adjustments of their source codes, the modified codes can work well on spark.

\subsection{Parallelization of SVM}

The mechanism of parallelization of SVM is illustrated in the paper of \cite{}. But due to the limited time, we didn't successfully realize it.

\subsection{Parallelization of K-means Clustering}

In the MLlib of Apache Spark, they have provided the k-means algorithm in python. We can directly use it on RDD. The mechanism of parallelization is the whole dataset will be divided into several RDD parts. K-means method will be run on these different RDDs. Then, the results will be sum up to form the final conclusion.

\subsection{Parallelization of Percolation Clustering}

The parallelization of percolation clustering is very similar to Co-Clustering algorithm, and k-means, we can implement Percolation clustering on different RDDS seperately, and sum up the results to form the final clusters.

\section{Framework and Implementaion}

\subsection{Dataset}
The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

\begin{itemize}
\item Feature Size\\
Obviously, here we have a RGB picture with $32 \times 32 *3 = 3072$ features for each sample.
\item Feature Selection\\
As described, we here will first compressed this image from RGB mode to intensity mode, where each pixel just has one corresponding value stands for its intensity.\\
Then, we will apply PCA algorithm on this intensity picture($1024$ features), to shrink its size.\\
Later we will see the picture plot by eigenvectors.
\item Labels\\
10 lables stands from different object classes.\\
This is general supervised classification with multi classes. Here, we employ SVM with Gaussian Kernel to deal with this task.

\end{itemize}

\subsection{Framework-Apache Spark}
Apache Spark is a fast and general engine for large-scale data processing, which is could work in standalone cluster mode, on EC2, or run it on Hadoop YARN or Apache Mesos. Also, it can read from HDFS, HBase, Cassandra, and any Hadoop data source.\\
\\
Spark introduces an abstraction called resilient distributed datasets (RDDs). Spark can outperform Hadoop multiple times, which is the reason we choose it as the framework of our task. It supports both python and Java. We use python for the task.

\subsection{Implementations}

Since we have known the numbers of labels is ten from the original dataset, using SVM to train the dataset is the best comparing with k-means, and percolation algorithm in our clustering library.

\section{Result and evaluation}

\section{Conclusion}
Image recognition requires high workload because of its intrinsic complex feature space. When it comes to large amount of pictures, common machine learning algorithm shows obvious limitation in pattern recognition in terms of computation efficiency, resource occupation, etc. Thus techniques developed specifically to deal with large-scale data processing could be expected to have obviously superior performance. In this project we implement parallel SVM based on PCA in large-scale image pattern recognition and verify this scheme works in such kind of task with favourable performance as we have expected.

\begin{thebibliography}{}

\bibitem{Spark} Spark API, \emph{Spark Programming Guide}, \textbf{2014}
\url{http://spark.apache.org/docs/latest/programming-guide.html}.

\bibitem{CIFAR} Alex Krizhevsky, \emph{Learning Multiple Layers of Features from Tiny Images}, Master's Thesis, \textbf{2009}.

\bibitem{Percolation} M. E. J. Newman and R. M. Ziff, \emph{Fast Monte Carlo algorithm for site or bond percolation
}, \textbf{2001}.

\bibitem{Jacobi} B. B. Zhou and R. P. Brent, \emph{On Parallel Implementation of the One-sided Jacobi Algorithm for Singular Value Decompositions
}, \textbf{1995}.

\end{thebibliography}

\end{document}


\chapter{Results and Conclusions}
In this section we will describe our results and any conclusions we may have drawn from them as well as decisions that came from these results.

\section{Convolutional Neural Networks: Results}
All the neural network here are conv nx3x3 - conv nx3x3 - max pool 2x2 no strides - conv nx3x3 - conv nx3x3 - max pool 2x2 no strides - Dense 256 - Dense 128 - Dense Softmax 10 with Relu activations in the rest of the layers. Where this is not true we will note as such.

We are also using Stochastic Gradient Descent as an optimiser with Nesterov decay of 1e-6, a learning rate of 0.01, a momentum of 0.90 and categorical crossentropy as the loss function. A validation loss is calculated and is used for early stopping with a patience of 50 epochs, the set which is used is a 15 percent slice of the end of the data which is first shuffled to ensure validation data is uniform in class composition.

\subsection{Number of Filters}
For this we are evaluating the following models:
\begin{itemize}
	\item 8-8-14-14 with 3x3 filters
	\item 24-24-32-32 with 3x3 filters
	\item 32-32-64-64 with 3x3 filters
	\item 48-48-96-96 with 3x3 filters
	\item 128-128-256-256 with 3x3 filters
\end{itemize}

For the results we are presenting here we are using data augmentation in all cases to reduce overfitting as well as early stopping to get the best model possible from our training. We have results for both colour and black and white for this evaluation.

We will only be presenting the colour results in this part as we will be evaluating the impact of colour in a later part.

\subsubsection{Effects on Training Speed}
As we increase the amount of filters we see our training speed declining as the number of neurons increases, this makes sense as you increase the number of computations that need to be done each epoch somewhat.

We ommit actual data here we were unable to control running parameters exactly due to these tests having to be run over several nights at different system workloads which would affect the results but our observations showed that we can get a range of 12s per epoch on our smallest model to 57s per epoch on our most complicated model
\subsubsection{Effects on Performance}
We begin this investigation assuming we will get better results with more.
\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{8-8-14-14-3x3-15pct-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{24-24-48-48-3x3-15pct-color_confusion_matrix} \\
		(a) 8-8-14-14 & (b) 24-24-32-32 \\[6pt]
		\includegraphics[width=0.45\linewidth]{32-32-64-64-3x3-15pct-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-3x3-15pct-color_confusion_matrix} \\
		(c) 32-32-64-64 & (d) 48-48-96-96 \\[6pt]
		\multicolumn{2}{c}{\includegraphics[width=0.45\linewidth]{128-128-256-256-3x3-15pct-no-strides-color_confusion_matrix} }\\
		\multicolumn{2}{c}{(e) 128-128-256-256}
	\end{tabular}
	\caption{Confusion Matrices varied by filter number}
	\label{confnf}
\end{figure}

As we can see from the figure \ref{confnf}, a significant improvement between out smallest model and out largest one. However it appears we hit diminishing returns after a certain number of filters as we presumably hit the limit of the features we can draw from our small 32x32 data.

From the qualification report in fig. \ref{repnf} we also see an interesting pattern we hinted on earlier in this document. Dogs and Cats are the worst performing classes with confusing happening mostly between them but dogs get confused less with cats than the opposite.

This becomes clearer when we look at the classification reports as seen in figure \ref{repnf}
\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{8-8-14-14-3x3-15pct-color_report} &   \includegraphics[width=0.45\linewidth]{24-24-48-48-3x3-15pct-color_report} \\
		(a) 8-8-14-14 & (b) 24-24-32-32 \\[6pt]
		\includegraphics[width=0.45\linewidth]{32-32-64-64-3x3-15pct-color_report} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-3x3-15pct-color_report} \\
		(c) 32-32-64-64 & (d) 48-48-96-96 \\[6pt]
		\multicolumn{2}{c}{\includegraphics[width=0.45\linewidth]{128-128-256-256-3x3-15pct-no-strides-color_report} }\\
		\multicolumn{2}{c}{(e) 128-128-256-256}
	\end{tabular}
\caption{Classification Reports for experiments}
\label{repnf}
\end{figure}

As you might notice from the figures above we stop seeing overall improvements at 48-48-96-96 with small improvement in some classes, but the extra cost of the larger networks makes it a good compromise for further experiments.

\subsubsection{Effects on Size of Weights}
This investigation is much simpler. We look at the sizes of the weights we store for each network and try to correlate it the number of calculations and network complexity.

\begin{figure}[h!]
	\includegraphics[width=0.8\linewidth]{datasizes_filters}
	\caption{Bar Chart of Size of Stored Weights}
	\label{sownf}
\end{figure}

In figure \ref{sownf} we can see how size increases with  more features. This makes sense since our feature maps become larger, while we can also see the increase is not completely linear with the number of neurons. This is because in CNNs weights are shared.

\subsubsection{Conclusion}
As we increase the number of our filters, we see an increase in classification performance. This comes at the expense of the simplicity of our model which impacts the speed of training, the size of the model and works as a trade-off with our classification performance.  Because of the huge cost of models larger than the 48 and the small gains we will investigate using 48 filters in situations where we are testing other hyper parameters.

\subsection{Size of Filters}
For this series of experiments we will report on the results of varying our 48-48-96-96 and 8-8-14-14 with a set of filter sizes:

\begin{itemize}
	\item 3x3
	\item 4x4
	\item 5x5
\end{itemize}

As before we are keeping all other hyper parameters the same.

\subsubsection{Effects on Training Speed}
For this we found training speed was not really affected by using larger filters in fact it was faster in some cases. We originally run this study on most of our studies but for brevity we will only look at the two extremes combinations.

\subsubsection{Effects on Performance}
We expected to see better performance with larger filters that were less numerous on account of potentially capturing larger features. Instead what we saw in fig. \ref{conffs}

\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{8-8-14-14-3x3-15pct-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-3x3-15pct-color_confusion_matrix} \\
		(a) 8-8-14-14 3x3& (b) 48-48-96-96 3x3\\[6pt]
		\includegraphics[width=0.45\linewidth]{8-8-14-14-4x4-15pct-no-strides-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-4x4-15pct-no-strides-color_confusion_matrix} \\
		(a) 8-8-14-14 4x4& (b) 48-48-96-96 4x4\\[6pt]
		\includegraphics[width=0.45\linewidth]{8-8-14-14-5x5-15pct-no-strides-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-5x5-15pct-no-strides-color_confusion_matrix} \\
		(a) 8-8-14-14 5x5& (b) 48-48-96-96 5x5\\[6pt]
	\end{tabular}
	\caption{Confusion Matrices varied by filter size}
	\label{conffs}
\end{figure}

We can see from the confusion matrices above that larger filters affect results negatively. The difference is so significant it can clearly be seen on the confusion matrices for smaller numbers of filters which is the opposite than what we initially expected.

The other pattern we observe is filter numbers still matter with larger filters we see an improvement in the 48 model. However they never match the smaller filters though the results do converge somewhat.

We believe this might be worse due to the nature of our training images. Remember the images are quite small at 3x3. This also means that any feature in those images is also going to be very small. As a result many small filters seem to be the way to go here.

Due to how significant the difference is we will not be adding the classification report but we note a range of accuracy of 0.60 to 82 on the smaller network.
\subsubsection{Effects on Size of Weights}

\subsubsection{Conclusion}
Larger filters are not really advisable in this dataset as they give no real practical benefits over 3x3 filters in performance, and training speed. Size of weights does not mitigate the loss in classification performance.

\subsection{Augmentation}
For this section we will only use the 48 network again. As a consistent good performer, and having seen the performance of 3x3 networks with 2 layers between max pooling in this dataset we use it as our reference.

For our augmentation we decided to use a rotation range of 0.1, a horizontal and vertical shift range of 0.8 and a random horizontal flip here. From the intelligence we have in our data and smaller experiments these should work well.
\subsubsection{Effects on Training Speed}
Here we saw very significant drops in speed. This is largely because augmentations need to run  on each batch and run on a single CPU thread. This means that we are bottlenecked by the CPU as the GPU idles waiting for images to train on. One way we observe this is that models are restricted to a minimum of 12s per epoch with it on with no decrease in this time despite trying out parameters that helped in the past.

This could be alleviated if Keras used a parallel image augmentation mechanism since this problem can be solved in a divide a conquer algorithm. We lose about 50-80 percent per epoch to this, impact also depends on the number of channels. Because of this we'll be looking at the benefits both in colour and greyscale.

\subsubsection{Effects on Overfitting}
We would like to see how quickly our model overfits. If our Augmentation works we expect to see a significant reduction in overfitting that should in theory increase our classification performance
\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides-no_aug_learning_rate} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides_learning_rate} \\
		(a) 1 channel no augmentation & (b) 1 channel augmentation \\[6pt]
		\includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides-color-no_aug_learning_rate} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides-color_learning_rate} \\
		(c) 3 channels no augmentation & (d) 3 channel augmentation \\[6pt]
	\end{tabular}
	\caption{Plot of Training History in Terms of Loss}
\end{figure}

As we can see in the figure, we are seeing a surprisingly significant improvement in our cross-validation results. With a patience of 50, we end up terminating at close to 80 epochs without augmentation. This should reflect very negatively in our generalised classification performance.

An observant reader might also notice local minima for validation loss that is consistent with what we'd expect to see and the main reason we use Early Stopping.
\subsubsection{Effects on Performance}
Following our earlier results we are eager to see whether there is any impact and if so what is it with not using augmentation in terms of results. We've seen it leads to very fast overfitting but does this translate to results?
\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides-no_aug_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides_confusion_matrix} \\
		(a) 1 channel no augmentation & (b) 1 channel augmentation \\[6pt]
		\includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides-color-no_aug_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-15pct-no-strides-color_confusion_matrix} \\
		(c) 3 channels no augmentation & (d) 3 channel augmentation \\[6pt]
	\end{tabular}
	\caption{Confusion Matrices for Augmentation Experiment}
\end{figure}

We can see just from the confusion matrix that performance takes a big hit without our augmentation. Indeed looking at our f1 score in the reports we can see 0.76 and 0.86 between the two options in greyscale. A very significant difference.

Looking very briefly at older logs from past experiments we also notice a pattern that larger networks tend to benefit more from augmentation. To corroborate our results we re-run the experiment and produced \ref{8-8-14-14-noaug}

\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{8-8-14-14-15pct-no-strides-no_aug_learning_rate} &   \includegraphics[width=0.45\linewidth]{8-8-14-14-15pct-no-strides-no_aug_confusion_matrix} \\
	\end{tabular}
	\caption{Small model with 8-8-14-14 3x3 filters no augmentation}
	\label{8-8-14-14-noaug}
\end{figure}

\subsection{Conclusions}
Data augmentation helps significantly to reduce overfitting and improve performance in our models. The performance cost is very significant with the current implementation of it but results benefit enough that it would be foolish not to use it.

If also appears that the larger the network the more important good augmentation becomes. This makes some sense as with more features extracted, our network would be increasingly likely to overfit so avoiding this is important.

\subsection{Number of Colour Channels}
\subsubsection{Effects on Training Speed}
Going from our augmentation results we are eager to observer how colour affects the training speed. We use the 48 as usual as our reference. Conversion takes some extra time especially on our Mode 4 conversion method. What we see however is a big boost in speed. This makes sense since we reduce inputs to a third which means there is much less processing to do.

The effect is more pronounced when augmentation is used where the performance gap widens. Presumably this is because there is much less data to convert which reduces CPU cost for the augmentation code.
\subsubsection{Effects on Performance}
Here we see a small reduction in classification performance by  switching to greyscale. It's usually of the order of 2-3 percent. A noticeable decline but not to the extent to justify the extra performance hit to run with 4 channels and augmentation. We can see from \ref{colourcrt}

\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{48-48-96-96-4x4-15pct-no-strides_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{48-48-96-96-5x5-15pct-no-strides-color_confusion_matrix} \\
	\end{tabular}
	\caption{Colour to grey confusion matrix comparison}
	\label{colourcrt}
\end{figure}

Ultimately for fast experiments it should not matter as long as it is noted and then once parameters are optimised re-run with colour to observe the better result.
\subsubsection{Effects on Size of Weights}
Minimal increases in the size of the weights was observed here. This is because with a CNN weights are shared by neurons, for the colour that's especially handy as it makes a 3x increase in inputs have a very negligible impact in size.

\subsection{Multi Layer Perceptron: Figures in Matrices}
As an early test we obtained a number of results from MLPs. Here we have collated some of the most interesting ones showing the effects of depth of MLPs in terms of training and classification performance

\begin{figure}[h!]
	\begin{tabular}{cc}
		\includegraphics[width=0.45\linewidth]{mlp-256-128-no-aug-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{mlp-256-256-128-128-no-aug-color_confusion_matrix} \\
		(a) 256-128 & (b) 256-256-128-128 \\[6pt]
		\includegraphics[width=0.45\linewidth]{mlp-512-512-256-256-128-128-no-aug-color_confusion_matrix} &   \includegraphics[width=0.45\linewidth]{mlp-1024-1024-512-512-256-256-128-128color_confusion_matrix} \\
		(c) 512-512-256-256-128-128 & (d) 1024-1024-512-512-256-256-128-128 \\[6pt]
	\end{tabular}
	\caption{Confusion Matrices varied neurons}
	\label{confmlp}
\end{figure}

we can see this illustrated in figure \ref{confmlp} how deeper MLPs with no augmentation improve slighly but ultimately overfit too quickly this can be seen more clearly in \ref{mlptime} We couldn't get augmentation working when we ported MLP's back to the new test suite so we could not test augmentation with MLPs.

We could not exceed 0.54 f1 score with this solution.
\begin{figure}[h!]
	\begin{center}
			\includegraphics[width=0.45\linewidth]{mlp-512-512-256-256-128-128-no-aug-color_learning_rate}
	\end{center}
	\caption{512-512-256-256-12-128 MLP History}
	\label{mlptime}
\end{figure}

\section{Miscellaneous Discussion}
\subsection{Accounts of Early Experiments}
Here we will discuss the earliest parts of our experimentation. These were done before the experimental infrastructure was complete so we don't have complete figures recorded for them. These accounts are based on logs and diary entries taken at the time. Rapid prototyping was taking place at the time so results might not be consistent. As a result this section is included in a pure discussion format and is aimed to offer some insight to early decisions taken in this project.

\subsubsection{Early Experiments}
Early on we experimented with decay, momentum, activation functions and learning rate. We ended up finding that defaults or near defaults for these values worked well for our training solution so we left them as is for the final results in an effort to reduce the hyperparameters we reported on.

Towards the middle of our studies we implemented the greyscale mode. That took a lot of tweaking to get right and we noticed differences between the different modes however at the time we lacked the correct diagnostics to record the data properly.

\subsection{Conclusion}
From our experiments we learned a number of things about CNN design for this set. First lesson was that larger models can get higher results but suffer from diminishing returns and overfitting.

A successful model for this dataset needs a good augmentation strategy, more aggressive augmentation works to an effect but it is important to try to maintain as many features as possible to maintain performance. Features are small so one needs to moreso careful.

Colour gives a small bump but experimenting in greyscale can give significant speed improvements which makes optimisation go much quicker since we can get results faster. Ideally we can only switch to colour for final results.

Larger filters are generally not very good for this dataset. Features are quite fine so 3x3 seems to work quite well from all our experiments and benefits of other filters seem lacklustre from our experiences.

Deeper networks can do well in this dataset and deal better with more aggressive augmentation but the size of the CNN seems to be more important. Very deep networks are hard to train with random initialisation so require different strategies to take advantage of. We did not have time to properly test implications for this.

From looking at these results we can also conclude that for some problems, neural networks might be reasonable ready to be sold, with accuracies exceeding 88 percent for a small student project.


% \documentclass{article}
\usepackage[utf8]{inputenc}

% Packages Required
% \usepackage{pgfplots}
% \usepackage{pgfplotstable}
\pgfplotsset{compat=1.7}
% \usepackage{subcaption}

% PGFPlots Settings
\pgfplotsset{
SmallBarPlot/.style={
    font=\footnotesize,
    ybar,
    width=\linewidth,
    ymin=0,
    xtick=data,
    xticklabel style={text width=1.5cm, rotate=45, align=center}
},
BlueBars/.style={
    fill=blue!20, bar width=0.25
},
RedBars/.style={
    fill=red!20, bar width=0.25
}
}

\DeclareRobustCommand\legendbox[1]{(\textcolor{#1}{#1}~\begin{tikzpicture}[x=0.2cm, y=0.2cm] \draw [color=black, fill=#1!20] (0,0) -- (0,1) -- (0.6,1) -- (0.6,0) -- (0, 0); \end{tikzpicture})}

\pgfplotsset{select coords between index/.style 2 args={
    x filter/.code={
        \ifnum\coordindex<#1\def\pgfmathresult{}\fi
        \ifnum\coordindex>#2\def\pgfmathresult{}\fi
    }
}}

% % Documents Preamble
% \title{PGFPlots for Blog}
% \author{Shanto Roy}
% \date{December 2019}

% \begin{document}

% \maketitle

% \section{Bar plot}
% \begin{figure}
%     \centering
% \pgfplotstableread[col sep=comma,]{embodied_cost_model/contents/100SalesRecords.csv}\datatable
% \begin{tikzpicture}
% \begin{axis}[
%     ybar,
%     bar width=.1cm,
%     width=\textwidth,
%     height=10cm,
%     ymin = 0,
%     xtick=data,
%     xticklabels from table={\datatable}{Country},
%     x tick label style={font=\normalsize, rotate=90, anchor=east},
%     ylabel={Units Sold in Number}]
%     \addplot [select coords between index={1}{20}, BlueBars] table [x expr=\coordindex, y={Units Sold}]{\datatable};
% \end{axis}
% \end{tikzpicture}
% \caption{Units sold by Different countries}
% \label{fig:my_label}
% \end{figure}

% \section{Group Bar Plot}
% \begin{figure}[ht!]
% \pgfplotstableread[col sep = comma]{embodied_cost_model/contents/100SalesRecords.csv}\table
% \centering
%     \begin{tikzpicture}
%     \begin{axis}[
%             SmallBarPlot,
%             xticklabels from table={\table}{Country},
%             ylabel=Value in Number,
%             xlabel=Countries,
%         ]
%         \addplot [BlueBars] [select coords between index={1}{5}] table [x expr=\coordindex, y=Units Sold] {\table};
%         \addlegendentry{First}
%         \addplot [RedBars] [select coords between index={1}{5}] table [x expr=\coordindex, y=Unit Price] {\table};
%         \addlegendentry{Second}
%         \legend{}
%     \end{axis}
%     \end{tikzpicture}
% \caption{The first legend \legendbox{blue} and the second one \legendbox{red}}
% \label{fig:multiBar}
% \end{figure}

\section{Subplots with Group Bar}

\begin{figure}[ht!]
\pgfplotstableread[col sep = comma]{embodied_cost_model/contents/100SalesRecords.csv}\table
\centering
\begin{subfigure}[b]{.5\linewidth}
    \centering
    \begin{tikzpicture}
    \begin{axis}[
            SmallBarPlot,
            xticklabels from table={\table}{Country},
            ylabel=Value in Number,
            xlabel=Countries
        ]
        \addplot [BlueBars] [select coords between index={1}{5}] table [x expr=\coordindex, y=Units Sold] {\table};
        \addlegendentry{First}
        \addplot [RedBars] [select coords between index={1}{5}] table [x expr=\coordindex, y=Unit Price] {\table};
        \addlegendentry{Second}
        \legend{}
    \end{axis}
    \end{tikzpicture}
\end{subfigure}%
\begin{subfigure}[b]{.5\linewidth}
     \centering
    \begin{tikzpicture}
    \begin{axis}[
            SmallBarPlot,
            xticklabels from table={\table}{Country},
            ylabel= Value in Number,
            xlabel=Countries,
            ymode=log,
            log basis y={10}
        ]
        \addplot [BlueBars] [select coords between index={1}{5}] table [x expr=\coordindex, y=Total Revenue] {\table};
        \addlegendentry{First}
        \addplot [RedBars] [select coords between index={1}{5}] table [x expr=\coordindex, y=Total Cost] {\table};
        \addlegendentry{Second}
        \legend{}
    \end{axis}
    \end{tikzpicture}
\end{subfigure}
\caption{The first legend \legendbox{blue} and the second one \legendbox{red}.}
\label{fig:subBarPlot}
\end{figure}

% \end{document}


\section{Ensemble methods for causal trees}


\documentclass[a4paper,10pt]{article}
%\documentclass[a4paper,10pt]{scrartcl}

\usepackage[utf8]{inputenc}
\usepackage{color,soul}

\title{Solution of the TDGP equation using TSSM}
\author{Carlos Lopetegui}
\date{\today}

\pdfinfo{%
  /Title    ()
  /Author   ()
  /Creator  ()
  /Producer ()
  /Subject  ()
  /Keywords ()
}

\begin{document}
\maketitle
The objective we pursue in this first part is to implement a solution to the time dependent Gross-Pitaevskii (TDGP) equation based on a Time Splitting Spectral Method as proposed in \cite{BaoJack}. Let's recall the shape of the TDGP equation in the presence of an arbitrary potential $V(\vec{r},t)$.
\begin{equation}\label{GP_time_Dep}
 i\hbar \frac{\partial \psi(\vec{r},t)}{\partial t}=\left(\frac{-\hbar^2}{2m}\nabla^2 +V(\vec{r},t) +U_0 N |\psi(\vec{r},t)|^2 \right)\psi(\vec{r}),t
\end{equation}
The solution of this non linear Schrodinger equation must be normalized:
\begin{equation}
 \int_{\Re^3}|\psi(\vec{r},t)|^2 dx=1
\end{equation}
If a harmonic trap potential is considered the equation takes the particular form:
\begin{equation}\label{pot_trap_GPE}\label{GP_time_Dep}
 i\hbar \frac{\partial \psi(\vec{r},t)}{\partial t}=\left(\frac{-\hbar^2}{2m}\nabla^2 +\frac{m}{2}(\omega_x^2x^2+\omega_y^2y^2+\omega_z^2 z^2) +U_0 N |\psi(\vec{r},t)|^2 \right)\psi(\vec{r},t)
\end{equation}
\subsection{Dimensionless TDGPE}
The previous equation can be re-scaled and stated in a dimensionless manner. Let's introduce:
\begin{equation}
\tilde{t}=\omega_xt,      \tilde{\mathbf{r}}=\frac{\mathbf{r}}{l_s},\tilde{\psi}(\tilde{\mathbf{r}},\tilde{t})=l_s^{3/2}\psi(\mathbf{r},t)
\end{equation}
Where $l_s$ is  the characteristic length of the condensate. Inserting the re-scaling in \ref{pot_trap_GPE} we obtain as a first intermediate result:
\begin{equation}
 i \hbar \omega_x\frac{\partial \tilde{\psi}}{\partial \tilde{t}}=-\frac{\hbar^2}{2 m l_s^2}\nabla_{\tilde{\mathbf{r}}}^2 \tilde{\psi}+\frac{m\omega_x^2 \tilde{\psi}l_s^2}{2}(\tilde{x}^2+\gamma_y^2\tilde{y}^2+\gamma_z^2\tilde{z}^2)+U_0N l_s^{4/3}|\tilde{\psi}|^2\tilde{\psi}
\end{equation}
Where $\gamma_{j}=\omega_{j}/\omega_{x}$. Now, we perform multiply all the equation by $1/(m\omega_x^2l_s^{1/2})$. By doing this we obtain:
\begin{equation}\label{dim_less_GPE}
 i \epsilon \frac{\partial \psi}{\partial t}=-\frac{\epsilon^2}{2}\nabla^2\psi+V(\vec{r})\psi+\delta \epsilon^{5/2}|\psi|^2\psi
\end{equation}
where all tildes have been removed (both time and position are expressed in relative units), and:
\begin{center}
 $\epsilon=\frac{\hbar}{\omega_x m l_s^2}=\left(\frac{a_0}{l_s}\right)^2$\\
$ V(\vec{r})=\frac{x^2+\gamma_{y}^2y^2+\gamma_{z}^2z^2}{2}$\\
$\delta=\frac{U_0 N}{a_0^3\hbar \omega_x}$\\
$a_0=\sqrt{\frac{\hbar}{\omega_x m}}$
\end{center}
The term $\delta \epsilon^{5/2}$ is usually represented as $\kappa$ in the sake of clarity.
It is worth recalling the expression for $U_0=\frac{4 \pi \hbar^2a}{m}$, where $a$ is positive for repulsive interaction and negative for  attractive interactions. Recalling this, we can re express $\kappa$ as:
\begin{equation}
 \kappa=\delta \epsilon^{5/2}=\frac{4\pi a N}{a_0}\left(\frac{a_0}{x_s}\right)^2
\end{equation}
There are two extreme cases possible:
\begin{itemize}
 \item weakly interacting regimes, in which $\epsilon=\mathcal{O}(1)$ ($ \Leftrightarrow a_0=\mathcal{O}(l_s)$), and, $\kappa=o(1)$($\Leftrightarrow 4\pi|a|N\ll a_0$)
 \item strong interacting regimes $\epsilon=o(1)$ ($ \Leftrightarrow a_0\ll  l_s$), and $\kappa=o(1)$($\Leftrightarrow 4\pi|a|N\ll a_0$)
\end{itemize}
From now on we work with the dimensionless equation \ref{dim_less_GPE}. We present in the next section the numerical scheme in which we solve this nonlinear equation.
\section{Time Splitting Spectral Method}
 The implementation we perform follows the Time-splitting Spectral Method, as proposed in \cite{Bao_Jack}. The method proposed combines both time splitting and spectral methods based on Fourier Transform to particularly solve equation \ref{dim_less_GPE}. Several aspects that will be discussed bellow lead to this selection. Yet, an important feature of the method used is that it is unconditionally stable, time reversible, time-transverse invariant and conserves the total number of particles (\textcolor{red}{can we show this? On the other side what does it even means that it conserves the total number of particles if the equation they are solving does not actually deal with a multiparticle wave function, they are doing Mean Field. Is it just that it conserves the probability distribution, i mean $|\psi|^2$ }) \\
 Let's present the method first for the one dimensional version of the model. Then, the generalization is straightforward. The equation in 1D, defined using periodic boundary conditions is given by:
 \begin{equation}
 i \epsilon \frac{\partial \psi}{\partial t}=-\frac{\epsilon^2}{2}\frac{\partial^2}{\partial x^2}\psi+\frac{x^2}{2}\psi+\kappa_1|\psi|^2\psi \;\;\;\; a< x< b
 \end{equation}
\begin{equation}
 \psi(x,t=0)=\psi^{0}(x) \;\;\; a\leq x\leq b
\end{equation}
\begin{equation}
 \psi(a,t)=\psi(b,t), \;\;\;\; \psi_x(a,t)=\psi_x(b,t), \;\;\; t>0
\end{equation}
 To numerically solve the equation we discretize both in time and space. In particular we solve the problem in a spatial grid given by:
 \begin{equation}
  x_j=a+j h, \;\;\;\; h=(a-b)/M \;\;\;\; j=0,1,2,...,M
 \end{equation}
the time step being $k=\Delta t>0$, so that:
\begin{equation}
 t_n=n k.
\end{equation}
The approximation of the solution for time $t_n$ in position $x_j$ will be given by $\psi_j^n$
 The numerical update in time from $t_n$ to $t_{n+1}=t_n+k$, will be performed via a Strang splitting, which approimates the solution of the equation by splitting the solution of the whole equation into two steps, first:
\begin{equation}
i \epsilon \frac{\partial \psi}{\partial t}=\frac{x^2}{2}\psi(x,t)+\kappa_1 |\psi(x,t)|^2\psi(x,t),
\end{equation}
and then
\begin{equation}
i \epsilon \frac{\partial \psi}{\partial t}=-\frac{\epsilon^2}{2}\frac{\partial^2 \psi(x,t)}{\partial x^2}.
\end{equation}
The first equation can be easily shown to preserve the norm of $\psi$. We profit from this fact to solve it exactly from $t_n\rightarrow t_{n+1}$, as it can be written as:
\begin{equation}
i \epsilon \frac{\partial \psi}{\partial t}=\frac{x^2}{2}\psi(x,t)+\kappa_1 |\psi(x,t_n)|^2\psi(x,t),
\end{equation}
which can be easily integrated as the term $|\psi(x,t_n)|^2$
enter just as a constant.\\
The second equation can be integrated exactly in time after going to Fourier space for the position coordinate. \\
Finally, the actual update step from $t_n$ to $t_{n+1}=t_n+2k$ will be split in three parts in a way that we gain one order in the error of the solution:
\begin{equation}
\psi(x_j,t_{n}+\frac{k}{2})=\psi_j^*=\psi_j^n\exp(-i(x_j^2/2+\kappa_1|\psi_j^n|^2)\frac{k}{2 \epsilon})
\end{equation}
\begin{equation}
\psi(x_j,t_{n}+\frac{3 k}{2})=\psi_j^{**}=\sum_{l=-M/2}^{M/2+1}\exp(-i \frac{\epsilon k \mu_l^2}{2})\hat{\psi}_l^* \exp(i \mu_l (x_j-a))
\end{equation}
\begin{equation}
\psi_j^{n+1}=\psi_j^{**}\exp(-i(x_j^2/2+\kappa_1|\psi_j^{**}|^2)\frac{k}{2 \epsilon})
\end{equation}
In the second equation,
\begin{equation}
\hat{\psi_{l}}(x)^*=\sum_{j=0}^{M-1} \psi(x_j)^*\exp(-i \mu_l(x_j-a)) \;\;\;\;\mu_l=\frac{2\pi l}{b-a} \;\;\;\;
l=-\frac{M}{2},-\frac{M}{2}+1 ... \frac{M}{2}-1
\end{equation}
\textcolor{red}{Here we should present some discussion with respect to
\begin{itemize}
 \item The Strang Splitting, and why breaking into three steps really reduces the error.
 \item The effect of the boundary conditions and how it can be reduced (aliasing effects for example).
 \item Error introduced by the space discretization. Actually I think Kris left this to us as an exercise.
\end{itemize}
}
Finally, to analyze the solution of the equation we do, in one side, look directly into the shape of the solution for different time steps. On the other side, it is worth and brings quite a physical intuition on the properties of the condensate, to look at the evolution of the variance of the distribution function $|\psi(x,t)^2|$. It is defined as:
\begin{equation}
 \sigma_x^2=\sqrt{\langle(x-\langle x\rangle)\rangle}
\end{equation}
In the more general case off dimension $d$ we can calculate $\sigma_j$ in each coordinate axis $j={1,..,d}$. The mean values are calculated by:
\begin{equation}
 \langle f\rangle=\int_{\Re^d}f(\vec{r})|\psi(\vec{r},t)|^2
\end{equation}
\section{Numerical Results in 1D}
To test the performance of the implementation we firstly studied the 1D GPE in a harmonic potential, for a system with $\kappa_1>0$, which implies that the condensate have repulsive interaction, under exponentially decaying initial condition:
\begin{equation}
\psi(x,0)=\frac{1}{(\pi \epsilon)^{1/4}}\exp(-x^2/(2\epsilon))
\end{equation}
We work on a strong interaction scheme with $\epsilon=0.1$, $\kappa_1=1.2649$, on the ``box'' $x=[-16,16]$.

\section{2D defocusing condensate}
In this section we extend the analysis to a 2D condensate.
We rewrite the TDGPE, now for 2D:
\begin{equation}
 i \epsilon \frac{\partial \psi}{\partial t}=-\frac{\epsilon^2}{2}\nabla^2\psi+\frac{x^2+\gamma_{y}^2 y^2}{2}\psi+\kappa_2|\psi|^2\psi
\end{equation}
We will focus particularly in the isotropic trap, $\gamma_y=1$.\\
The implementation is exactly as for the 1D condensate, just that now, we solve the problem in a 2D grid which is the tensor product of the partition in the $y$ and $x$ direction. It implies that one performs a two dimensional Fourier analysis. Further than that no spatial complication arise neither remarkable numerical issues. In particular no modification is necessary in the Strang Splitting which remains unconditionally stable (\textcolor{red}{Need to prove it I think}). The set of parameters with which we worked were:
\begin{equation}
 \epsilon=1.0 \;\;\;\;\kappa_2=2.0
\end{equation}
Which corresponds to $\mathcal{O}(1)$ interactions (strong). We use gaussian shaped initial conditions to test the method.
\begin{equation}
 \psi(x,y,0)=\frac{1}{\sqrt{\pi \epsilon}}\exp(-(x^2+y^2)/(2\epsilon))
\end{equation}

\end{document}