\documentclass[a4paper,12pt,twoside]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{lmodern}
\usepackage[french]{babel}
\usepackage{url,csquotes}
\usepackage[hidelinks,hyperfootnotes=false]{hyperref}
\usepackage[titlepage,fancysections,pagenumber]{polytechnique}
\usepackage{multirow}
\usepackage{wrapfig}

\title{Shortest Paths on Surfaces\\ Geodesics in Heat}
\subtitle{INF555 Digital Representation\\ and Analysis of Shapes }
\author{Ruoqi \bsc{He} \& Chia-Man \bsc{Hung}}

\begin{document}

\maketitle

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

In this project we present the algorithm of a practical method for computing approximate shortest paths (geodesics) and our implementation on triangle meshes. This method of Heat Flow is based on \textit{Geodesics in Heat: A New Approach to Computing Distance Based on Heat Flow}, Crane [2013]. Results are mainly showed by pictures. The last part is dedicated to additional work, including the two boundary conditions, the case of multisource and a system of navigation represented by a walking man on surface.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Heat flow algorithm}

In this section we explain the Heat Flow algorithm used in this project. It determines the geodesic distance to a specified subset of a given domain. We interpret it on a triangle mesh.

Notations: heat field $u$, vector field $X$, distance function $\phi$.

\subsection{First Step}
Integrate the heat field $\dot{u} = \Delta u$.

The heat $u$ of a point on a considered surface is a value between 0 and 1. Source has 1 as its value.

In other words, we solve $(id - t\Delta)u_t = u_0$ where $u_t$ is the heat field after a time step $t$ and $u_0$ is the initial heat field.

On a triangle mesh and in matrix form, we have $(A - tLc)u = u_0$ where $A$ is a diagonal matrix containing the vertex areas, $A^{-1}Lc$ is the discrete Laplacian matrix of a triangle mesh, and $u_0$ is a vector which has value 1 on sources and 0 otherwise. Remark that here $u_0$ should be multiplied at left by $A$. However, in the case of one point source, this does not affect the second step.

\subsection{Second Step}
Evaluate the vector field $X = -\nabla u / \left | \nabla u \right |$.

We are only interested in the direction of $\nabla u$ and not in its value. $X$ points to the opposite direction of the source.

On a triangle mesh, by using the formula given in the article, we calculate the vector field on every triangle.

\subsection{Third Step}

Solve the Poisson equation $\Delta \phi = \nabla \cdot X$.

If a distance function $\phi$ exists, $\nabla \phi$ should give us a unit vector on every point, pointing to the opposite directin of the source. We approximate such a distance function $\phi$ by minimizing $\int \left | \nabla \phi - X \right |^{2}$, which is equivalent to solving the Poisson equation $\Delta \phi = \nabla \cdot X$.

On a triangle mesh and in matrix form, we solve $Lc\phi = b$ where $b$ is the vector of divergences of the vector field $X$.

%\subsection{Navigation}

%We also implemented a navigation system, represented by a walking man. He is positioned on the surface of the mesh and moves toward a given source point.

%A walking man's position is defined by barycentric coordinates of a triangle of the mesh. A walk function takes a distance as its argument and first applys on the triangle on which he is standing and then recursively calls itself at the next triangle he comes across, until walking the given distance or reaching the source.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Implementation}

\subsection{Environment}

We chose Unity to implement the heat method in order to better visualize the result. All codes are written in C\#, and the scene is built with Unity Editor. For the main problem, we use the library ALGLIB to do sparse matrix operations and to solve linear equations. In addition, we use the C5 Generic Collection Library for the priority queue implementation.

%We used the half-edge data structure to present triangle meshes.

\subsection{Mesh representation}

\begin{wrapfigure}{r}{0.32\textwidth}
	\centering
	\includegraphics[width=0.32\textwidth]{F2.png}
	\caption{Boundary face}
\end{wrapfigure}

We obtain ordinary triangle meshes via various ways. Those meshes are represented by a vertex array (array of Vector3) and a triangle array (sets of 3 indexes stored in an int array). We first wrote a method to convert them to half-edge representation, defined in Geometry.cs. The conversion can be done in time of $O(nd^2)$ where n is the number of vertices and d is the maximum degree of vertices. There are however two important things to consider:

\begin{wrapfigure}{l}{0.5\textwidth}
	\centering
	\includegraphics[width=0.5\textwidth]{F1.png}
	\caption{UV seams}
\end{wrapfigure}

1. The half-edge representation is not well defined when using meshes with boundaries. In order to incorporate with other methods built on this representation, we decided to add a new face to cap each boundary. Those faces are marked as ``boundary faces'', and all vertices around them are marked as ``boundary vertice''. This way we can easily implement the boundary conditions in the heat method.

2. Many 3D models obtained from the internet have UV mappings, and thus have UV seams. This means that at the same position there can be two separate points having different UV coordinates. So the geometry we built may have seam-like boundaries blocking the way. To cope with this, we implemented a method to weld all overlapping vertices with a complexity of $O(nlogn)$, based on kdTree range searching.\\

\subsection{Matrix precalculation}

For each mesh loaded, we calculate in the first place its discrete unweighted laplacian matrix $-Lc$, and the matrices $A-tLc$ adapted to two different boundary conditions (if there are boundaries).

\begin{itemize}
\item
Dirichlet condition:

We set all elements in the rows/columns of the boundary vertices to 0, except the diagonal elements. This way the heat value will always be 0 at the boundary.

\item
Neumann condition:

The original laplacian matrix described in the paper satisfies Neumann condition.
\end{itemize}

We also build a Vector3 array of size ($3 \times$ triangle count) keeping all the values of cot(angle) times opposite edge vector, in order to accelerate the calculation of divergence.

For the first time, we skipped the Cholesky decomposition step since the overall performance without it is still reasonable. However, because of the numerical problems that we will explain afterwards, we finally implemented the Cholesky decomposition. It is applied on all precalculated matrices.

\subsection{Main calculation}

For single source problem, we follow these steps:

\begin{enumerate}
\item
Calculate the heat field $u$ by solving the heat equation $(A-tLc)u = u_0$.
\item
Calculate $X$, the normalized gradient of the heat field, on every triangle.
\item
Calculate $DivX$ on every vertex using the value of $X$ on its surrounding triangles.
\item
Calculate the distance field $\phi$ by solving the Poisson equation $Lc \phi = DivX$
\item
We then calculate the gradient of the distance field $\nabla \phi$ on every triangle which can be used to calculate the shortest paths.
\end{enumerate}

At first, without Cholesky decomposition, we used LinCG (Linear Conjugate Gradient) solver to solve linear equations. This solver solves symmetric positive-definite problems, but it works fine even with our second semi-definite problem (Poisson equation). We just need to shift the result such that the distance value at the source equals to zero.

\begin{wrapfigure}{r}{0.5\textwidth}
	\centering
	\includegraphics[width=0.5\textwidth]{P1.png}
	\caption{Left: LinCG / Right: Cholesky}
\end{wrapfigure}

%\begin{figure}[htb]
%	\centering
%	\includegraphics[width=0.8\textwidth]{P1.png}
%	\caption{P1}
%\end{figure}

However, from the result we obtained, we observed that LinCG solver is sensible to numerical errors. The solution of the heat equation contains values ranging from 1 to $10^{-20}$ or smaller. The longer the distance to the source, the smaller the heat value is. LinCG returns values of 0 when smaller than $\sim 10^{-12}$, so the heat gradient of the farther area cannot be computed. We can only improve the solution by increasing the time step $t$, which increases the heat value but creates a smoothed distance field.

Therefore, we chose to implement the Cholesky decomposition of matrices. It consists of decomposing a symmetric positive matrix $M$ into $LL^T$, where $L$ is a lower triangular matrix. This means the linear equation $LL^Tx = y$ becomes two basic triangular systems, that can be solved by simple substitution. This improves greatly the calculation time when switching sources (The decomposition is only calculated once for every mesh). And since we calculate the exact solution this way, we managed to have much smaller numerical errors, so as to eliminate the problems above.
We overcame two difficulties regarding the Cholesky decomposition:

\begin{itemize}
\item
While $M$ is a sparse matrix, $L$ is not necessarily sparse, especially when the entries of $M$ are mostly far away from the diagonal. This comes to one of our model which has randomly ordered vertices – it costs more than 4G RAM space to store the matrix $L$. After we reorder the vertices of the mesh along a certain axis, $L$ becomes much sparser and takes much less time to compute.
\item
We need to add a small regularization term to the diagonal entries of the laplacian matrix to get strict positive-definiteness (needed for the Cholesky decomposition of ALGLIB).
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results}

\subsection{Main results}

We successfully obtained some very nice results on a variety of meshes. Below are some pictures of the calculated geodesics and gradient fields.

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{P4.png}
	\caption{Family photo}
\end{figure}

We observed the difference between the heat gradient and the distance gradient. The distance gradient smoothed out some non-uniform areas.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.5\textwidth]{P2.png}
	\caption{Left: Gradient of distance field / Right: Gradient of heat field}
\end{figure}

%\begin{wrapfigure}{r}{0.3\textwidth}
%	\centering
%	\includegraphics[width=0.3\textwidth]{P2.png}
%	\caption{P2}
%\end{wrapfigure}

\subsection{Time step t}

We usually set the time step t to the square of the maximum edge length. Using a much larger time step can create a smoothed distance. The smoothed distance is in general longer than the exact distance, because the trajectory is more curved. Below is an example of comparison.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.8\textwidth]{P8.png}
	\caption{Left: $t=h^2$ / Right: $t=100h^2$ ($h$ is the maximum edge length)}
\end{figure}

%numerical error when $t = 1$ (choice of $t$), tweak parameter

\subsection{Boundary conditions}

We also compared the results when using different boundary conditions. We observed that with high value of $t$ (smoothed distance), the paths obtained tend to avoid borders with Dirichlet condition, and to adhere to borders with Neumann condition. And by using the average heat field calculated from two different conditions we get more natural paths.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.8\textwidth]{P5.png}
	\caption{Left: Neumann / Middle: Dirichlet / Right: Average condition}
\end{figure}

\subsection{Broken mesh}

%\begin{wrapfigure}{r}{0.25\textwidth}
%	\centering
%	\includegraphics[width=0.25\textwidth]{P3.png}
%	\caption{P3}
%\end{wrapfigure}

The triceratops mesh that we tested has some bizarrely sharp obtuse triangles – they will break the positiveness of the heat matrix if we do not put a limit to the cotangent values of an angle. Even if we do, the result seems broken. We find a workaround by resetting the position of a vertex of the triangle to the average position of its neighbors.\\\\\\

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.34\textwidth]{P3.png}
	\caption{Mesh fixing}
\end{figure}

\subsection{Calculation time cost}

\begin{table}[htb]
\centering
\begin{tabular}{|c|c|c|c|c|c|c|}
  \hline

  %\multirow{2}{*}{Model} & \multirow{2}{*}{Triangles} & \multicolumn{2}{c|}{LinCG} & \multicolumn{2}{c|}{Cholesky} \\
  %& & Pre & Solve & Pre & Solve \\
  %\hline

  Model & Triangles & \multicolumn{2}{c|}{LinCG} & \multicolumn{2}{c|}{Cholesky} \\
  \cline{3-6}
  & & Pre & Solve & Pre & Solve \\
  \hline\hline

  High Genus & 3328 & 34ms & 27ms & 64ms & 8.9ms \\
  Bague & 5304 & 55ms & 60ms & 109ms & 17ms  \\
  Cow & 5804 & 62ms & 109ms & 335ms & 16ms \\
  Triceratops & 5660 & 59ms & 105ms & 306ms & 15ms \\
  Sphere 1 & 9024 & 94ms & 99ms & 236ms & 21ms \\
  Sphere 3 & 6912 & 65ms & 59ms & 327ms & 18ms \\
  Hemisphere & 3456 & 51ms & 34ms & 131ms & 10ms \\
  Skull & 12200 & 183ms & 338ms & 989ms & 38ms \\
  Maze & 5120 & 63ms & 94ms & 134ms & 12ms \\
  Horse & 39996 & 447ms & 1355ms & 2047ms & 113ms \\
  Dragon & 20388 & 188ms & 1291ms & 1787ms & 58ms \\
  \hline
\end{tabular}
\caption{Comparison of LinCG and Cholesky decomposition}
\end{table}

%\phantom \\

\section{Additional work}

\subsection{How we obtain meshes?}
Below are the four options we used:

\begin{enumerate}

\item
OFF format mesh from TD (High Genus, Bague, Cow, Triceratops, Horse) -> Unity mesh -> halfedge mesh

We wrote a OFF parser to transform OFF format meshes into Unity meshes.

\item
Internet resources (Skull, Dragon) -> Unity mesh -> processing (weld uv seams) -> halfedge mesh

\item
3Dmax handmade mesh (Maze, Sphere 2 \& 3, Hemisphere) -> Unity mesh -> halfedge mesh

\item
Code generated mesh (Sphere 1) -> halfedge mesh

\end{enumerate}

\subsection{Optimization}
Our solver ALGLIB solves sparse symetric positve-definite matrix system a lot faster than only symetric matrix system and Cholesky decomposition works only with symetric positve-definite matrix.
We noticed that $-Lc$ is a symetric positve-semidefinite matrix.
Therefore, $A-tLc$ (for $t>0$) in the first step is a symetric positve-definite matrix and $\epsilon I-Lc$ (for $\epsilon >0$) in the third step is also a symetric positve-definite matrix. Below is a math proof.

Prove that $-Lc$ is a symetric positve-semidefinite matrix.

We note $U_{ij}=E_{ii}+E_{jj}-E_{ij}-E_{ji}$ where $E_{ij}$ is an elementary matrix with only one nonnull value $1$ on position $(i,j)$. By definition, $-Lc$ is in form  $-Lc=\sum_{i,j}u_{ij}U_{ij}$ with all $u_{ij}>0$. Since the eigenvalues of $U_{ij}$ is $0$ and $2$, it is a symetric positve-semidefinite matrix. We conclude by saying that any positive combination of symetric positve-semidefinite matrices is also a symetric positve-semidefinite matrix. (Another proof is given in Lecture 9.)\\

\subsection{Multisource}

We also tried calculating the geodesics on surfaces when multiple vertices are marked as sources. However, by simply changing the initial heat vector to  $u_0$ (the vector which has value 1 on sources and 0 otherwise) we generate incorrect heat field and distance field – the solved heat field $u_t$ cannot guarantee equal values at each source vertex, and thus not every source vertex has a distance of zero. Typically, the vertex surrounded by more source vertices has a higher temperature and a negative distance.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.6\textwidth]{P6.png}
	\caption{Left: Simply adding source on $u_0$ / Right: Accurate multisource}
\end{figure}

We solved this by imposing the constraints $u_t = 1$ at every source vertex in the heat equation, and also the constraint $\phi = 0$ at every source vertex in the Poisson equation. (The first constraint cannot imply the second since the distance field we get is only the closest potential of the heat gradient field.)

One downside, however, is that the matrices need to be modified to contain these extra constraints related to the source. For exemple, to have $u_t = 1$ at the source we put $1$ at the diagonal entries of the sources, and $0$ everywhere else in the same lines and columns. We then compensate the terms we deleted by adding an additional vector at the right side of the equation. Because of this, the Cholesky decomposition need to be carried out each time we change the source. This slows down the calculation a lot.

\subsection{Navigation}

\begin{wrapfigure}{r}{0.36\textwidth}
	\centering
	\includegraphics[width=0.36\textwidth]{P7.png}
	\caption{Navigation}
\end{wrapfigure}

Now that we have the distance field, we would like to calculate the path from a given point to the source point. We achieved this by first calculating the gradient of the distance field (step 5 of the main calculation) which is considered uniform on every triangle, then tracing a trajectory by recursively following the gradient in every triangle and entering the next one.

We visualized this process by moving a walking man on the surface of the mesh. His position is defined by barycentric coordinates of the triangle he is standing on. The distance gradient on it is also converted to barycentric coordinates, but with a sum of 0. (In this way, we can add it to the position coordinates and still get a barycentric coordinate with a sum of 1.) A walk function takes a distance as its argument and first moves on the triangle on which he is standing. By finding the first coordinate that reaches zero we find the edge the man will come across. It then recursively calls itself at the next triangle he enters, until walking the given distance or reaching the source.

There are however two extra cases to consider:
\begin{itemize}
\item
The gradient is pointing to a boundary. In this case, when reaching a boundary, we walk alongside it towards the smaller-distance vertex.
\item
Two neighboring triangles have opposing gradient. In this case, when reaching the middle edge, we choose to follow the edge instead.
\end{itemize}

\subsection{Mapping}

In order to better visualize the distance field, we chose to map a striped texture onto the surface of the mesh. It is actually quite simple: We create some striped textures with color gradient that are only one pixel tall, assign it to the mesh, and for each vertex we set the U value of its UV coordinates to its distance to the source. And that’s it! When rendering the texture, each point on a triangle has its U coordinate interpolated from the three vertices of the triangle, and it is exactly the interpolated distance of this point – so it will be colored by the pixel of the striped texture representing this distance.
To make it even better, in addition to the main texture, we also apply generated normal map, specular map and emission map to each model, making it especially realistic. We just need to set the tangent vector of each vertex to the average distance gradient on it to make the normal map work.

\begin{figure}[htb]
	\centering
	\includegraphics[width=0.6\textwidth]{P9.png}
	\caption{An example of generated textures}
\end{figure}

%Albedo, Normal, Specular Smoothness, Emission

\subsection{Dijkstra}
We implemented a simple Dijkstra shortest path algorithm on the graph made by the vertices and edges of the mesh. This allows us to easily create a line of source vertices connecting two selected vertices. (In the demo, press shift and click on the surface of the mesh.)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion \& extensions}
Path finding is an important topic and has applications in several domains, such as video games and robotics. The heat method is a simple way that allows us to compute approximate geodesics to a point source or a specific domain of source (multisource) efficiently. In this project we succeeded to implement this method on surfaces represented by triangle meshes. A system of navigation is added to visualize the shortest path taken. Some extra work is done to improve the visualization, such as texture mappings.

Regarding improvement we could make, in the case of multisource repeated calculation of Cholesky decomposition is required due to the constraints of multisources. It might be possible to operate on the Cholesky decomposition of $Lc$ to cleverly avoid recalculating the decomposition each time we change the source.

One extension we can think of is adding different weight on different areas to represent the difficulty of passing through. Another is changing the form of the considered domain from surfaces represented by triangle meshes to bounded spaces represented by tetrehedron meshes.

\end{document}


\section{Instructions}
\label{sec:isa}

\subsection{Instruction Format}

PicoVersat features a very minimal set of instructions. There is only one
instruction type as illustrated in Table~\ref{tab:if}.

\begin{table}[!htbp]
  \centering
    \begin{tabular}{|c|p{7cm}|}
    \hline
    {\bf Bits} & {\bf Description} \\
    \hline \hline
     31-28 & Operation code (opcode)\\
    \hline
     27-0 & Immediate constant \\
    \hline

    \end{tabular}
  \caption{Instruction format.}
  \label{tab:if}
\end{table}

\subsection{Instruction Set Architecture}

The instruction set is given in Table~\ref{tab:isa}. The notation M[Imm]
represents the contents of position (address) Imm in the memory map.

\begin{table}[!htbp]
  \centering
    \begin{tabular}{|c|c|p{8cm}|}
    \hline
    {\bf Mnemonic} & {\bf Opcode} & {\bf Description} \\
    \hline \hline
    %nop & 0x0 & No operation; PC = PC+1\\
    %\hline
%
\multicolumn{3}{|c|}{{\bf Arithmetic / Logic}}\\
\hline \hline
    addi  & 0x0 & RA = RA + Imm; PC=PC+1\\
    \hline
    add   & 0x1 & RA = RA + M[Imm]; PC=PC+1\\
    \hline
    sub   & 0x2 & RA = RA - M[Imm]; PC=PC+1\\
    \hline
    shft  & 0x3 & RA = (Imm $<$ 0)? RA $<<$ 1: RA $>>$ 1; PC=PC+1\\
    \hline
    and   & 0x4 & RA = RA \& M[Imm]; PC=PC+1\\
    \hline
    xor   & 0x5 & RA = RA $\oplus$ M[Imm]; PC=PC+1\\
    \hline
\multicolumn{3}{|c|}{{\bf Load / Store}}\\
\hline \hline
    ldi   & 0x6 & RA = Imm; PC=PC+1\\
    \hline
    ldih  & 0x7 & RA[31:16] = Imm; PC=PC+1\\
    \hline
    rdw   & 0x8 & RA = M[Imm]; PC = PC+1\\
    \hline
    wrw   & 0x9 & M[Imm] = RA; PC = PC+1\\
    \hline
    rdwb  & 0xA & RA = M[RB+Imm]; PC = PC+1\\
    \hline
    wrwb  & 0xB & M[RB+Imm] = RA; PC = PC+1\\
    \hline
\multicolumn{3}{|c|}{{\bf Branch}}\\
\hline \hline
    beqi  & 0xC & RA == 0? PC = Imm: PC += 1; RA = RA-1\\
    \hline
    beq   & 0xD & RA == 0? PC = M[Imm]: PC += 1; RA = RA-1\\
    \hline
    bneqi & 0xE & RA != 0? PC = Imm: PC += 1; RA = RA-1\\
    \hline
    bneq  & 0xF & RA != 0? PC = M[Imm]: PC += 1; RA = RA-1\\
    \hline

    \end{tabular}
  \caption{Instruction Set.}
  \label{tab:isa}
\end{table}

\subsection{Aliased Instructions}

There are two instruction aliases:
\begin{description}
\item {\tt nop}: same as  {\tt addi 0}. It means No OPeration or do nothing.
\item {\tt wrc A,B}: same as {\tt wrw (A+B)}, where A and B are numeric and
  integer constants given with the {\tt wrc} alias.
\end{description}

\subsection{Delayed Branches}

The instruction following a branch instruction is always executed to the
instruction pipeline latency (delayed branch or slot). Make sure instructions
placed after branch instructions are useful and not harmful.

Insert {\tt nop} instructions after a branch if there is no suitable candidate
for the delay slot·


\section{Appendix :  reflection in more detail}\label{appendix:rho_details}

\subsection{Name equivalence}

We now come to one of the first real subtleties of this calculus as
compared to other process calculi. Both the calculation of the free
names of a process and the determination of structural congruence
between processes critically depend on being able to establish whether
two names are equal. In the case of the calculation of the free names
of an input-guarded process, for example, to remove the bound name we
must determine whether it is in the set of free names of the
continuation. Likewise, structural congruence includes
$\alpha$-equivalence. But, establishing $\alpha$-equivalence between
the processes $x?(z).\lift{w}{y!(z)}$ and $x?(v).\lift{w}{y!(v)}$, for
instance, requires calculating a substitution,
e.g. $x?(v).\lift{w}{y!(v)}\{z / v \}$. But this calculation requires,
in turn, being able to determine whether two names, in this case the
name in the object position of the output, and the name being
substituted for, are equal.

As will be seen, the equality on names involves structural equivalence
on processes, which in turn involves alpha equivalence, which involves
name equivalence. This is a subtle mutual recursion, but one that
turns out to be well-founded. Before presenting the technical details,
the reader may note that the grammar above enforces a strict
alternation between quotes and process constructors. Each question
about a process that involves a question about names may in turn
involve a question about processes, but the names in the processes the
next level down, as it were, are under fewer quotes. To put it another
way, each `recursive call' to name equivalence will involve one less
level of quoting, ultimately bottoming out in the quoted zero process.

Let us assume that we have an account of (syntactic) substitution and
$\alpha$-equivalence upon which we can rely to formulate a notion of
name equivalence, and then bootstrap our notions of substitution and
$\alpha$-equivalence from that. We take name equivalence, written
$\nameeq$, to be the smallest equivalence relation generated by the
following rules.

\begin{mathpar}
\inferrule*[left=Quote-drop]
{ }
{ \quotep{\dropn{x}} \nameeq x }

\inferrule*[right=Struct-equiv]
{ P \scong Q }
{ \quotep{P} \nameeq \quotep{Q} }
\end{mathpar}

\subsection{Syntactic substitution}

Now we build the substitution used by $\alpha$-equivalence. We use
$\Proc$ for the set of processes, $\QProc$ for the set of names, and
$\id{\{}\vec{y} / \vec{x} \id{\}}$ to denote partial maps, $s : \QProc
\rightarrow \QProc$. A map, $s$ lifts, uniquely, to a map on process
terms, $\widehat{s} : \Proc \rightarrow \Proc$ by the following
equations.

\begin{mathpar}
(0) \psubstp{Q}{P}         :=    0 \\
(R \juxtap S) \psubstp{Q}{P}
		:=
		(R)\psubstp{Q}{P} \juxtap (S) \psubstp{Q}{P} \\
(x?(y).R) \psubstp{Q}{P}
		:=
		(x)\substp{Q}{P} (z)\concat( (R \psubstn{z}{y}) \psubstp{Q}{P} ) \\
(\lift{x}{R}) \psubstp{Q}{P}
 		:=
 		\lift{(x)\substp{Q}{P}}{ R \psubstp{Q}{P} } \\
(\dropn{x})  \psubstp{Q}{P}
		:=
		\left\{
			\begin{array}{ccc}
				\dropn{\quotep{Q}} & & x \nameeq \quotep{P} \\
                              	\dropn{x} & & otherwise \\
			\end{array}
		\right.
\end{mathpar}

where

\begin{eqnarray}
(x)\id{\{} \lpquote Q \rpquote / \lpquote P \rpquote \id{\}}            =
		\left\{
			\begin{array}{ccc}
				\lpquote Q \rpquote & & x \nameeq \lpquote P \rpquote \\
                                x & & otherwise \\
			\end{array}
		\right. \nonumber
\end{eqnarray}

and $z$ is chosen distinct from $\quotep{P}$, $\quotep{Q}$, the free
names in $Q$, and all the names in $R$. Our $\alpha$-equivalence will
be built in the standard way from this substitution.

But, given these mutual recursions, the question is whether the calculation
of $\nameeq$ (respectively, $\scong$, $\alphaeq$) terminates. To
answer this question it suffices to formalize our intuitions regarding
level of quotes, or quote depth, $\#(x)$, of a name $x$ as
follows.

\begin{eqnarray}
	\#(\quotep{P}) & = & 1 + \#(P) \nonumber\\
    \#(P) & = & \left\{
			   \begin{array}{ccc}
				 max\{ \#(x) : x \in \names{P} \} & & \names{P} \neq \emptyset \\
				 0 & & otherwise \\
			   \end{array}
			\right. \nonumber
\end{eqnarray}

The grammar ensures that $\#(\quotep{P})$ is bounded. Then the
termination of $\nameeq$ (respectively, $\scong$, $\alphaeq$) is an
easy induction on quote depth.

\begin{remark}
  Note that by a related piece of reasoning we can see that $\forall
  P. \quotep{P} \not\in \freenames{P}$.
\end{remark}

\subsection{Semantic substitution}

The substitution used in $\alpha$-equivalence is really only a device
to formally recognize that binding occurrences do not depend on the
specific names. It is not the engine of computation. The proposal here
is that while synchronization is the driver of that engine, the real
engine of computation is a semantic notion of substitution that
recognizes that a dropped name is a request to run a process. Which
process? Why the one whose code has been bound to the name being
dropped. Formally, this amounts to a notion of substitution that
differs from syntactic substitution in its application to a dropped
name.

\begin{eqnarray}
(\dropn{x})  \psubstp{Q}{P}
		& = &
		\left\{
			\begin{array}{ccc}
				Q & & x \nameeq \quotep{P} \\
                              	\dropn{x} & & otherwise \\
			\end{array}
		\right. \nonumber
\end{eqnarray}

In the remainder of the paper we will refer to semantic and syntactic
substitutions simply as substitutions and rely on context to
distinguish which is meant. Similarly, we will abuse notation and
write $\substn{y}{x}$ for $\psubstn{y}{x}$.


\chapter{A sentence-level extension to LDA}
\label{chap:sentencelda}

\section{Introduction}

Generative models of text, such as latent Dirichlet allocation \citep{Blei:03}, seem to automatically capture the semantic content of collections of documents \citep{Griffiths:04}. These models typically assume documents are bags of words, with individual words only weakly correlating with one another, if at all. However, word meaning is dependent on context -- \textit{\textbf{support} vector machine} means something very different to \textit{generous \textbf{support}}. Therefore, modelling dependencies between words would allow these models to closer reflect the underlying text.

In this paper, we propose to use sentences, not words, as the basic exchangeable unit of topic models. We present two generative models of language which both generate whole sentences from the latent topics, replacing the topic-word categorical distribution with a more flexible topic-sentence distribution given by a recurrent neural network language model. We aim to capture intra-sentential dependencies such as syntax and compositional semantics using the RNN, and sentence-level semantics in the latent topics. Our models can be viewed as integrating language modelling into topic models.

One model, \textsc{sentVAE}, is a discrete hidden variable generative model of sentences. Here, for each sentence $s_i$ in a document $\mathcal{D}$, the model independently draws a topic $z$ from a uniform prior, and then draws a sentence from $z$. The other, \textsc{sentLDA}, is a direct generalisation of LDA. For each document, the model first draws a document-topic distribution $\theta$ from some prior. Then, for each sentence in the document, the model draws a topic $z$ from $\theta$, and then draws a sentence from $z$.

Traditional inference methods for generative models with latent variables have typically used iterative methods such as (variational) expectation maximisation or Gibbs sampling. However, calculating the data likelihood repeatedly with a computationally intensive function such as an RNN is prohibitively slow. We thus use neural variational inference \citep{Rezende:14,Mnih:14,Kingma:14} to train our models. This uses a neural network to parametrise an approximation to the  posterior distribution of the latent variables given the data, and then trains the generative model with samples from this approximate posterior. By using of reparametrisation to perform the sampling, gradient information can be backpropagated from the generative model to the inference model, which allows the model to be trained end-to-end. We make heavy use of such reparametrisation gradient estimators, using the recently introduced Gumbel-softmax estimator \citep{Jang:17,Maddison:17} to sample from categorical distributions, and replacing the Dirichlet prior of traditional LDA with a logistic normal prior \citep{Mackay:98,Lafferty:06,Hennig:12,Strivastava:17}.

We show that both our model achieves much lower perplexities than standard topic models and even a strong language model baseline on held-out documents, showing that integrating a more powerful data likelihood function with global topic information helps model documents much more accurately. We also manually inspect the latent topics, and show qualitatively that both models learn syntactically and semantically coherent clusters of sentences, and show some qualitative differences in the clusters learnt by each of our models. Finally, we show that document representations learnt via standard word-level LDA outperform sentence-level topic models at a standard shallow document classification task, and explore the reasons behind this.

\section{Related Work}

Unsupervised discovery of the latent topics of documents has been a long-standing goal in NLP, with early attempts including models such as latent semantic indexing and its probabilistic generalisation \citep{Deerwester:90,Hofmann:99}. However, these models are not true generative models, as they cannot properly assign probabilities to new documents outside the training set. Latent Dirichlet allocation (LDA; \citet{Blei:03}) was developed partly to overcome these issues; \citet{Griffiths:04} showed the utility of topic models for exploring large collections of documents.

Attempts have been previously made to relax the word independence assumptions of the traditional LDA model: \citet{Wallach:06} proposed a model that can also generate word n-grams, while \citet{Gruber:07} tie all the topic assignments for a single sentence together, but still generate each word of a sentence independently of the other words. Another previous attempt to integrate contextual information into topic models is the syntactic topic model of \citet{Boydgraber:09}.

Recently, \citet{Dieng:17} proposed \textsc{TopicRNN}, another model that integrates RNNs and topic models. However, their generative process is different to ours: it is effectively a mixture model of a traditional topic model and a recurrent neural network language model. In contrast, our model is a much more direct generalisation of LDA to the sentence-level, as it integrates the RNN directly into the generative process.

Variational inference is a common technique for inference in otherwise intractable graphical models; it was proposed as one of the earliest inference methods for LDA \citep{Blei:03}. The idea of using a neural network for the proposal distribution was first proposed in \citet{Dayan:95} in the context of Helmholtz machines. However, this trained the inference and generative portions of the model separately. End-to-end training of the entire ensemble as an autoencoder was introduced by \citet{Kingma:14}, \citet{Mnih:14} and \citet{Rezende:14}. \citet{Miao:16} applied these techniques to the task of document modelling, while \citet{Strivastava:17} used neural variational inference, as well as a diagonal logistic-normal approximation to the Dirichlet distribution, in the context of topic modelling. \citet{Bowman:16} combined neural variational inference with an LSTM as the likelihood function -- our \textsc{sentVAE} is an adaptation of their model with discrete rather than continuous latent variables. This architecture has subsequently been generalised for dialogue generation \citep{Cao:17,Serban:17} and text compression \citep{Miao:16b}.

\section{Model description}

\subsection{Motivation}

\begin{figure}
    \centering
    \includegraphics[width=0.6\columnwidth]{lda_plate}
    \caption{Plate diagram for (unsmoothed) LDA. Shaded variables are observed.}
    \label{fig:lda_plate}
    \vspace{-1em}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{sentlda_plate}
    \caption{Plate diagram for \textsc{sentLDA} (left) and \textsc{sentVAE} (right)}
    \label{fig:sentlda_plate}
\end{figure}

To motivate \textsc{sentLDA}, consider the classic LDA model, shown in Figure \ref{fig:lda_plate}. The model first draws, for each document, a document-topic distribution $\theta$ from a Dirichlet prior $\alpha$. Then, for each word in the document, the model samples a topic $z$ from $\theta$, and then samples a word $w$ from $P(w | z)$. Coherent semantic topics (i.e. clusters of words) emerge from the tradeoff between assigning as few topics per document as possible, and as few words per topic as possible. Here, the Dirichlet prior has a crucial role in promoting document-topic and topic-word sparsity.

However, the model assumes that words are generated independently once the underlying topics are known. This assumption means the model cannot capture compositional semantics, such as multi-word expressions. In addition, another side effect is that the model very strongly implements the \textit{one-sense-per-discourse} heuristic - multiple occurrences of the same word in a document will all be tagged with the same topic, regardless of the context of the word. Further, this model is completely syntax-blind -- the document is treated as a mere bag-of-words, and word order is completely ignored, in stark contrast to linguistic intuition.

\subsection{The Generative Story}

Motivated by these considerations, \textsc{sentLDA} makes the following change to the generative process of LDA: instead of assuming the topics correspond to distributions over the finite vocabulary of words, we assume topics correspond to a distribution over an infinite set of sentences. Recurrent neural networks are a natural fit to parametrise this distribution -- specifically LSTMs \citep{Hochreiter:97}, for their power and ability to capture long-range dependencies.

One way to incorporate topic information into the language model is to use one LSTM per topic. However, this would lead to an explosion in the number of parameters of the model. Instead, \textsc{sentLDA} shares the same LSTM for all topics, and passes in the topic information as the initial hidden state of the LSTM. We originally tried passing in the raw one-hot topic encoding as the initial hidden state, but this gave suboptimal results. Inspired by recent work on distributed representations of categorical data, and particularly motivated by the success of word embeddings, we instead look up the one-hot representation of a topic in a \textit{topic embedding} table, and pass this in to the decoder LSTM. This also has the benefit of divorcing the decoder LSTM hidden state size from the number of topics.

Finally, during model training, we would like to use the reparametrisation trick of \citet{Kingma:14} and \citet{Rezende:14} to jointly train the inference and reconstruction networks end-to-end. However, there is no convenient reparametrisation trick for the Dirichlet distribution, so \textsc{sentLDA} replaces the Dirichlet prior over the document-topic distributions by a diagonal logistic normal distribution \citep{Mackay:98,Hennig:12,Strivastava:17}. The logistic normal distribution in $d$-dimensions (hereafter denoted by $\sigma \mathcal{N}$) is parametrised by $\mu \in \mathbb{R}^{d-1}$ and $\Sigma \in \mathbb{R}^{(d-1)^2}$. Samples from this distribution are obtained by sampling from $\mathcal{N}(\mu, \Sigma)$, appending 0, and then taking the softmax of the resulting vector. One advantage of this distribution over the Dirichlet distribution is that it can capture correlations between topics.

The full generative story of \textsc{sentLDA} is shown in Figure \ref{fig:sentlda_plate}, and the log-likelihood of a document $\mathcal{D}$ with sentences $s_1, \dots, s_N$ has the following form (where $\alpha$ are the parameters of the logistic normal prior):
\vspace{-1em}
\begin{dmath}
P(\mathcal{D} | \alpha) = \int P(\theta | \alpha) \times \\ \left ( \prod_{i=1}^N \sum_{z_n} P(z_n | \theta) P(s_i|z_n) \right ) d\theta
\end{dmath}

In LDA, one motivation to introduce $\theta$ was to combat the weakness of the unigram and mixture-of-unigram models \citep{Nigam:00} of documents, which stems from the deficiencies of a multinomial data likelihood function. The unigram model just learns a single multinomial distribution from observing which words appear the most in the whole corpus (in effect, the entire corpus has a single topic). The mixture-of-unigram learns $N$ multinomials (corresponding to the number of latent topics), and for each document draws a topic, and then draws words from the multinomial corresponding to that topic. In contrast, the $\theta$ parameter of LDA allows each document to exhibit a mixture of topics.

RNNs are much more powerful data likelihood functions, so the role of $\theta$ in \textsc{sentLDA} is not immediately necessary. We thus introduce a simpler model, \textsc{sentVAE}, to investigate whether a document-level latent variable is needed to obtain good modelling performance (or indeed coherent sentence topics). The generative story for \textsc{sentVAE} is very similar to that of \textsc{sentLDA} -- the only difference is that \textsc{sentVAE} omits the $\theta$ latent variable. The log-likelihood of the same document has the following form in \textsc{sentVAE} (where now $\alpha$ corresponds to a uniform categorical prior over the latent variables):

\begin{dmath}
P(\mathcal{D}|\alpha) = \prod_{i=1}^N \sum_{z_n} P(z_n | \alpha) P(s_i|z_n)
\end{dmath}

\subsection{Model Inference and Training}

In LDA, the exact posterior $P(\mathbf{z}, \theta | \mathcal{D})$ is intractable to calculate exactly. The two main approximate inference methods have been variational expectation maximisation (EM) and collapsed Gibbs sampling. However, neither approach is suitable for our model:
\begin{itemize}
    \item Variational EM requires repeatedly calculating the data likelihood for each data point during the E-step, which is computationally expensive for RNN sequence scorers.

    \item The topic-sentence distribution is no longer a multinomial, and hence we no longer have a convenient Dirichlet-multinomial conjugacy to derive a Gibbs sample update.
\end{itemize}

We thus use neural variational inference \citep{Kingma:14,Mnih:14,Rezende:14} to solve our problem. Given a document $\mathcal{D}$ consisting of $N$ sentences $s_1, \dots, s_N$ is intractable to calculate, we make the mean field assumption that the posterior distribution over $(\theta, \mathbf{z})$ factorises into independent distributions $Q(\theta|\mathcal{D})$ and $Q(\mathbf{z} | \mathcal{D})$, and obtain an approximate posterior of the form:
\begin{equation}
Q(\mathbf{z}, \theta | \mathcal{D}) = Q(\theta|\mathcal{D})\prod_{i=1}^{N} Q(z_i|s_i)
\end{equation}

We assume that the approximate posterior over the document-topic distribution, $Q(\theta | \mathcal{D})$, is given by a diagonal logistic normal distribution. We calculate the parameters of $Q(\theta, \mathcal{D})$ as a function of the document (again, $\sigma \mathcal{N}$ refers to the logistic normal distribution):
\begin{equation}
Q(\theta | \mathcal{D}) = \sigma \mathcal{N}(\mu(\mathcal{D}), \mathrm{diag}(\Sigma(\mathcal{D})))
\label{eqn:theta_posterior}
\end{equation}

The posterior over the sentence-topic distribution $Q(\mathbf{z}| \mathcal{D})$ is by an independent product of the individual sentence topic distributions $Q(z_i | s_i)$ (we assume that the topic of each sentence depends only on that sentence).

Although the true data likelihood is intractable to optimize, we can obtain the following lower bound $\mathcal{L}$ (commonly refered to as the evidence lower bound or ELBO):
\begin{multline}
\label{eqn:sentlda_elbo}
\log P(\mathcal{D}) \geq \mathcal{L} =  - [KL(Q(\theta|\mathcal{D}) || P(\theta | \alpha)) \\ + \mathbb{E}_{\theta \sim Q}(KL(Q(\mathbf{z} | \mathcal{D}, \theta) || P(\mathbf{z} | \theta)))] \\ + \mathbb{E}_{\mathbf{z} \sim Q} \log P(\mathcal{D} | \mathbf{z})
\end{multline}

This lower bound differs from the true log-likelihood by exactly the KL divergence between the variational approximation and the true posterior:
\begin{equation}
\log P(\mathcal{D}) - \mathcal{L} = KL(Q(z, \theta | \mathcal{D}) || P(z, \theta | \mathcal{D}))
\label{eqn:ELBO_gap}
\end{equation}

To train our model, we optimize the lower bound in Equation \ref{eqn:sentlda_elbo}. We use the inference network to calculate the parameters of the posterior approximation $Q(z, \theta| \mathcal{D})$. Then, we use samples of $z$ from $Q$ to estimate the log-likelihood of the document. Finally, we calculate the ELBO (given above), and then take an update step to maximise this quantity.

Gradients of the ELBO with respect to the generative model are easy to calculate. However, gradients of the ELBO with respect to the inference model require taking the derivative of an expectation of samples from the discrete distribution $Q(\mathbf{z}|\mathcal{D})$. One potential solution is to use the score function estimator, otherwise known as REINFORCE \citep{Williams:92}. However, this estimator has high variance, although variance reduction techniques can be employed \citep{Greensmith:04,Mnih:14}

We instead use the Gumbel-softmax estimator, as proposed independently in \citet{Maddison:17} and \citet{Jang:17}. This makes use of the following fact: if $a_i$ are the logits corresponding to a discrete distribution, and $\epsilon_i$ are i.i.d. Gumbel variables, then $\argmax_i (a_i + \epsilon_i)$ simulates samples from that distribution. By replacing the argmax with a softmax, and scaling the individual terms by a temperature parameter $\tau$, we have a continuous approximation to the discrete sample, which in the zero temperature limit approaches a discrete sample. In particular, this gives us a reparametrisation trick for a multinomial distribution, and lets us backpropagate gradients to the inference network.

Inference for \textsc{sentVAE} is exactly the same, except we do not need to calculate $Q(\theta|\mathcal{D})$. The ELBO for \textsc{sentVAE} is given by:

\begin{multline}
\log P(\mathcal{D} | \alpha) \geq -KL(Q(\mathbf{z}|s) || P(\mathbf{z}|\alpha)) + \\ \mathbb{E}_{\mathbf{z} ~ Q} \log P(\mathcal{D} | \mathbf{z})
\end{multline}

\subsection{Model Implementation}

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth]{model_architecture}
    \caption{The model architecture of \textsc{sentLDA}. The document consists of two sentences: \textit{I do not like} and \textit{green eggs and ham}. Arrows represent the flow of information, and dashed arrows indicate sampling. Stochastic variables are represented as circles, and deterministic variables are represented in rounded rectangles. The red dashed line indicates that the KL between the two variables is minimised during training. The architecture of \textsc{sentVAE} is very similar, except there is no document level RNN and no document-level $\theta$.}
    \label{fig:model_architecture}
\end{figure*}

We calculate $Q(\mathbf{z}|\mathcal{D})$ and $Q(\theta | \mathcal{D})$ in both models above using neural networks (see Figure \ref{fig:model_architecture} for an illustration of \textsc{sentLDA}).

Given a document $\mathcal{D}$ of $N$ sentences $s_1, \dots, s_N$, we first run a bidirectional LSTM over each $s_i$, concatenate the final states of the LSTM in each direction and pass them through a single feedforward layer to obtain our representation of each sentence, $h_i$. We form a document representation by further running a second bidirectional LSTM over the $h_i$'s, concatenating the final states of each and passing them through a second feedforward layer to obtain $d$, our representation of the entire document (cf. the hierarchical document models of \citet{Yang:16}, \citet{Li:2015} and \citet{Lin:15}). We next predict the posterior mean and variance of the document-topic distribution as:
\begin{equation}
\begin{split}
Q(\theta | \mathcal{D}) &= \sigma \mathcal{N}(\mu_{pos}, \Sigma_{pos}) \\
\mu_{pos} &= W_{\mu}d + b_{\mu} \\
\log(\Sigma_{pos}) &= diag(W_{\Sigma}d + b_{\Sigma})
\end{split}
\end{equation}

To calculate $Q(z|s)$, we pass $h_i$, the representation of the sentence $s_i$ through an affine transformation and then apply a softmax to obtain the logits corresponding to $Q(z|s)$:

\begin{equation}
Q(z | \theta) = \sigma(W_z h_i + b_z)
\end{equation}

We estimate all expectations in the ELBO using Monte Carlo sampling. For the expectation of the KL between the posterior sentence-topic distribution and the prior, we use samples of $\theta$ from the posterior document-topic distribution and calculate the KL analytically. As the KL between two categorical distributions has an easy to compute closed form, and sampling from a normal is easy, we can take many samples of $\theta$ to obtain a good estimate (in the experiments below, we use 10000).

Unfortunately, the Gumbel-softmax estimator does not give true one-hot samples. While we could marginalise out the non-zero values in the sample by computing the data likelihood under each topic, weighted by the probability of that topic in the Gumbel-softmax sample, this is computationally inefficient. We originally tried the straight-through Gumbel estimator proposed in \citet{Jang:17}, but found this gave poor results. We found that the samples were sufficiently peaked to just combine the topic embeddings directly, weighted by the value of that topic under the Gumbel-softmax sample, and pass this as the initial hidden state of the LSTM. As observed by \citet{Maddison:17}, a fixed Gumbel-softmax sampling temperature of 1 worked well, due to the large dimensionality of the latent variables.

We also encountered the problems reported in \citet{Bowman:16}, whereby our model would not make use of the latent variables, as LSTMs are powerful sequence models. We overcome these difficulties through the techniques suggested in that paper, by  annealing the weight of the KL term in the ELBO from 0.05 (initialising the KL weight to 0 led to divergence during training) to 1 over the course of model training and by dropping out input words with probability $0.8$.

Another issue for latent variable models trained through neural variational inference is mode collapse. This is when latent variables are stuck in a local minimum during training, and do not learn to encode useful information. As reported in \citet{Sonderby:16} and \citet{Strivastava:17}, batch normalisation (BN) \citep{Ioffe:15} helps alleviate this issue, and indeed every topic was used to label some sentence in the development set after training with BN (if BN was not used, only one topic was consistently used to label sentences).

\section{Experiments}

We evaluate the above models on two tasks. The first is document modelling, which is a measure of how well generative models of text can predict unseen documents. For this, we are given a document, and have to infer the likelihood of that document. The second is using the document-level features that each model infers to perform document classification. Finally, we qualitatively inspect the topics that the models learn, to investigate whether the model has learnt coherent clusters of sentences.

\subsection{Document Modelling}

\begin{table}[t]
    \centering
    \begin{tabular}{c|cc}
        \toprule
        Model & Dev Perplexity & Test Perplexity \\
        \midrule
        \textsc{sentVAE} & \textbf{105} & \textbf{92} \\
        \textsc{sentLDA} & 108 & 95 \\
        LSTM LM & 124 & 116 \\
        LDA & 782 & 571 \\
        \bottomrule
    \end{tabular}
    \caption{Perplexity on held-out documents}
    \label{tab:perplexity}
\end{table}

Here, we test how well \textsc{sentLDA} can fit to the data and evaluate the likelihood of unseen documents. We use data from 100 randomly selected days of the RCV1 dataset to train our model, which corresponds to 211k documents and 52 million words. To evaluate our model, we used data from a disjoint randomly-selected set of 36 days, corresponding to 92k documents and 22 million words. We also used files from a disjoint set of 5 days as validation data, corresponding to 9k documents and 2 million words. We lowercased all words in the dataset, replaced all numbers with 0's and collapsed any string of more than one 0 to a single 0. We filtered out any words that appeared fewer than 20 times in the training set as our vocabulary, which left 41341 words. We also truncated all documents to 20 sentences, and each sentence to 40 words.

As baselines, we compare our \textsc{sentLDA} model to standard LDA\footnote{We used the Gensim implementation \citep{gensim}}, and a self-implemented single-layer LSTM language model. As much as possible, we kept the same hyperparameters across all our models (see the appendix for full hyperparameter details). All the latent variable models used 50 latent topics. For \textsc{sentVAE} and \textsc{sentLDA}, we use 1 sample from the posterior to estimate the data log-likelihood during training time, and use 10 samples during test time. For \textsc{sentLDA}, we used a standard logistic normal prior ($\mu = 0$, $\Sigma=I_d$). We trained all of our models for 5 epochs, and picked the model with the lowest validation set perplexity to evaluate on the test set. The results are shown in Table \ref{tab:perplexity}.

Both sentence-level models do considerably better on perplexity than LDA on unseen documents, showing the utility of using a more powerful data likelihood function. \textsc{sentLDA} and \textsc{sentVAE} also do better than the LSTM language model, as these models can perform inference to work out global topics before predicting individual words.

The discrepancy between \textsc{sentVAE} and \textsc{sentLDA} can be explained by the fact that the ELBO for \textsc{sentVAE} is tight. Recall Equation \ref{eqn:ELBO_gap} -- the bound on the log-likelihood given by variational inference differs from the true log-likelihood by the KL divergence between the approximate posterior and the true posterior.For \textsc{sentVAE}, the true posterior $P(z | s)$ is also categorical, as any distribution over a finite set of categorical. Hence, it is possible for the approximate posterior $Q(z | s)$ to learn to match the true posterior, and so the ELBO can reach the true log-likelihood. For \textsc{sentLDA}, the approximate posterior is just that, and so there is always a gap between the ELBO and the true log-likelihood.

\subsection{Document Classification}

\begin{table}[t]
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{c|cc|cc}
        \toprule
        Model & Dev Accuracy & Dev F1 & Test Accuracy & Test F1 \\
        \midrule
        \textsc{sentVAE} & 0.9787 & 0.5676 & 0.9724 & 0.3886 \\
        \textsc{sentLDA} & 0.9780 & 0.5553 & 0.9777 & 0.5563 \\
        LDA & \textbf{0.9812} & \textbf{0.6578} & \textbf{0.9809} & \textbf{0.6596} \\
        \bottomrule
    \end{tabular}
    }
    \caption{Binary classification accuracy and (microaveraged) F1 on dev and test sets}
    \label{tab:classification}
\end{table}

We next experimented with document classification using the document representations learnt by our models. For each model type, we used the model with the lowest perplexity from the document modelling task to extract document representations (to be described later). We then used these representations to perform document tagging, using the topic tags in the RCV1 corpus.

The documents in RCV1 are all tagged with topic tags, indicating the semantic domain of the document \citep{Lewis:04}. Each document can have multiple tags (usually up to 4), and there are 103 in total, making this a sparse multiclass classification problem. We thus train 103 independent logistic classifiers, one for each tag, each taking as input the document representations. We use the same documents for training, validation and testing as above, and do not do any fine-tuning of our models (i.e. the only parameters we train are those of the classifier). We train each classification model for 5 epochs, and evaluate the model with the best F1 score on the development set.

For LDA, we used $\gamma$, the parameter of the inferred posterior Dirichlet distribution over the $\theta$ latent variable, as the document representation. This has a natural interpretation: $\gamma$ is obtained by incrementing each value of the parameter of the Dirichlet prior $\alpha$ by the number of times that topic appears in the document.

For \textsc{sentLDA}, $Q(\theta | \mathcal{D})$ is controlled by two parameters $\mu$ and $\Sigma$. We combine the two parameters by estimating the mean of the distribution $\hat{\theta} = \mathbb{E}_{\theta \sim Q} \theta$, and use the log of this as the document representation.

\textsc{sentVAE} does not infer a simple document-level statistic, but instead infers independent sentence-level ones $Q(z_i | s_i)$. To aggregate these into a document level representation, we take the average of these distributions, and then take the log of the resulting distribution. This has the natural interpretation that to draw a topic for $\mathcal{D}$, we first draw a sentence randomly from $\mathcal{D}$, and then draw a topic from the sentence-topic posterior.

As can be seen, there is still a large gap in document classification performance between \textsc{sentLDA}, \textsc{sentVAE}, and LDA, with traditional LDA outperforming its sentence-level counterparts. \textsc{sentLDA} also seems to be more stable across unseen data than \textsc{sentVAE}, which exhibits a dramatic drop in F1 score on the test set.

The superiority of LDA could be due to the nature of the classification task: the RCV1 topics tagset are very topically focussed (as the name suggests), and it is often enough to identify one or two keywords to tag a document. Indeed, \citet{Lewis:04} report microaveraged F1 scores of 0.8 using an SVM with word-level features, using a different training/test split and different data preprocessing (they perform hierarchical expansion of document tags, which makes the task easier by reducing sparsity). The LDA $\gamma$ parameter is precisely a summary of the words which appear in a document, which suggests why it does so well at this task. In contrast, the topic information learnt by \textsc{sentVAE} and \textsc{sentLDA} is more on the sentence-level, and hence it is harder to spot these keywords from the document representations inferred by these models.

\subsection{Topic Coherence}

\begin{table*}[t]
\centering{
\begin{tabular}{c c p{30em}}
\toprule
    \multirow{9}{*}{\textsc{sentLDA}}
        & \multirow{3}{*}{Topic 39}
            & \tabitem \resizebox{29em}{0.5em}{diplomats said the talks had been good to the talks with the united states , the united states are trying} \\
        &   & \tabitem \resizebox{29em}{0.5em}{he said the government of the peace talks in the united states ' visit to the united states and the} \\
        &   & \tabitem \resizebox{29em}{0.5em}{he said he was a meeting for the talks with an official with the country 's government to the united} \\
    \cmidrule(lr){2-3}
        & \multirow{3}{*}{Topic 2}
            & \tabitem \resizebox{29em}{0.4em}{traders said the results were expected to be released by the following were a few percent from the current crop} \\
        &   & \tabitem \resizebox{29em}{0.5em}{it said the market is expected to be around the end of the week 's main states and the market} \\
        &   & \tabitem \resizebox{29em}{0.5em}{traders said the market , the current food prices have been a 0 percent in 0 and the u.s. import} \\
    \cmidrule(lr){2-3}
        & \multirow{3}{*}{Topic 47}
            & \tabitem usa : u.s. police end on friday \\
        &   & \tabitem south korea : s.africa 's largest producer to 0 pct \\
        &   & \tabitem usa : u.s. dollar end lower on friday \\
\midrule
    \multirow{9}{*}{\textsc{sentVAE}}
        & \multirow{3}{*}{Topic 27}
            & \tabitem france : india 's favourite pension funds to 0 pct \\
        &   & \tabitem france : press digest - march 0 - feb \\
        &   & \tabitem usa : texas , california , '' on thursday \\
    \cmidrule(lr){2-3}
        & \multirow{3}{*}{Topic 15}
            & \tabitem \resizebox{29em}{0.5em}{north korea is the biggest of the collapse of the country 's efforts to have to reduce the peace process } \\
        &   & \tabitem \resizebox{29em}{0.5em}{european union 's efforts to join the united states for the united states , which is a new york to} \\
        &   & \tabitem \resizebox{29em}{0.4em}{european arab countries have been held in the wake of the united states in a dispute with israeli government and} \\
    \cmidrule(lr){2-3}
        & \multirow{3}{*}{Topic 31}
            & \tabitem \resizebox{29em}{0.6em}{czech republic : czech bond ended march 0 0 to 0 on friday after a march 0 march 0 0} \\
        &   & \tabitem \resizebox{29em}{0.5em}{czech republic : 0 percent of the previous week ended march 0 0 to a year earlier in the first } \\
        &   & \tabitem \resizebox{29em}{0.5em}{czech republic : czech republic : 0 points in a tonne on friday , the previous session of the previous} \\
\bottomrule
\end{tabular}

}
\caption{Sampled sentences (truncated to 20 words) from the 3 most common topics of each model}
\label{tab:samples}
\vspace{-1em}
\end{table*}

Finally, we inspect the content of the latent variables $z$. We select three most common topics used to label sentences in our dev set, and generated 3 sentences from each topic by initialising the hidden state of the decoder with the topic embedding corresponding to that topic and then sampling from the decoder. To ensure that our generated sentences are grammatical, we divide the logits output by the decoder by a temperature parameter $\tau$, which serves to sharpen the per-word probability distribution. We found that $\tau=0.5$ gave reasonably varied but coherent output. Table \ref{tab:samples} shows the samples. For more samples from other topics, please see the appendix.

The topics seem to be coherent clusters of sentences, with sentences sampled from each topic having similar syntactic and semantic properties. Qualitatively, the topics learnt appear to be different for each model, with \textsc{sentVAE} learning topics which seem to have a loose semantic topic, while \textsc{sentLDA} learns more focussed syntactically and semantically coherent topics. In addition, output from \textsc{sentLDA} seems more grammatical in general. It seems that adding a document-level latent variable $\theta$ does qualitatively change the behaviour of the resulting model, and forces the topics to be more tightly clustered.

We found that passing in one-hot topics resulted in incoherent output, and that we needed to take Gumbel-softmax samples of the one-hot topic vectors before our decoder generated coherent text. We suspect this is due to the non-one-hotness of the Gumbel-softmax samples used during, which must be replicated during test time for grammatical output. An interesting extension would be to see whether exact sampling from $Q(z|s)$ during training time would result in more grammatical output and more coherent topics during test time.

We also experimented with autoencoding sentences from the development set using both models - we performed inference on the sentences of the first document in the development set, and then sampled from the resulting sentence-topic posteriors. The resulting autoencoded sentences are similar in syntax and general semantic content, although differ slightly in their exact meanings. Full results are shown in the appendix.

\section{Conclusion}

We present two sentence-level generative models of documents. One is a categorical sentence autoencoder, while the other is a direct generalisation of LDA with a sentence-level likelihood function. We show that both models outperform LDA and traditional language models at the task of document modelling, and that both models learn coherent sentence-level topics. We also show that the document-level representations can be used to perform document classification, but are outperformed at a shallow document classification task by traditional LDA.


\documentclass{article}
\usepackage{fullpage}
\usepackage{amsmath,amsthm,amssymb}
\usepackage{todonotes}

\newcommand{\mbf}[1]{{\bf #1}}

\title{A geometric lemma for complex polynomial curves with applications in harmonic analysis.}
\author{Jaume de Dios Pont}

\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{defi}{Definition}
\newtheorem*{remark}{Remark}

\begin{document}
	\maketitle
	\abstract{ABSTRACT TO BE DONE
	}

	\tableofcontents

	\todo[inline]{Check what's the most natural way to write a ball of radius r}

	\newpage
	\section{Introduction} % (fold)
	\label{sec:introduction}
	\input{Parts/intro}
	%section introduction (end)

	\newpage
	\section{The partitioning lemma} % (fold)
	\label{sec:the_geometry}
	\input{Parts/geometric_lemma}
	% section the_geometry (end)
	\newpage
	\section{Uniform restriction for polynomial curves} % (fold)
	\label{sec:the_analysis}
	\input{Parts/complex_restriction}

	% section the_harmonic_analysis (end)
	\newpage
	\section{Convolution estimates for measures on complex curves} % (fold)
	\label{sec:convolution_estimates_for_measures_on_complex_curves}

	% section convolution_estimates_for_measures_on_complex_curves (end)

\end{document}


\documentclass{../Common/Structure/doc_pdf}
%
\usepackage{pgfgantt}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}

\titleSubtitle{{\Huge\textit{Power EnJoy}}\\{\LARGE Project Plan Document}}{Version 1.0.0}
\pageHeader{Project Plan Document}

\newcommand{\fp}[4]{\itemBold{#1}\par#2\begin{center}$#3 \cdot #4$\end{center}}

\begin{document}
	\titleToc{}
  \chapter{Introduction}
  \input{Introduction/introduction.tex}
  \newpage

  \chapter{Function Points}
  \input{FP/fp.tex}

	\chapter{COCOMO}
	\input{Cocomo/cocomo.tex}
  \newpage

	\chapter{Task and Schedule}
	\input{Task/task.tex}

	\chapter{Resources Allocation}
	\input{Res/resources.tex}

	\chapter{Risk}
	\input{Risk/risk.tex}

  \appendix
  \chapter{Appendix}

  \section{Tools}
  \begin{itemize}
  	\itemBold{TeXstudio} \LaTeX{} editor used to write the document.
		\itemBold{COCOMO Tool II} at \url{http://csse.usc.edu/tools/COCOMOII.php} to estimate the project dimension.
  	\itemBold{StarUML} To draw Gant diagram and Resources allocation.\end{itemize}
  \section{Hours of work}
  In the following are listed the hours of work that each member of the group did:
  \begin{enumerate}
  	\item Marco Redaelli: 16 \emph{hours}
  	\item Francesco Zanoli: 16 \emph{hours}
  \end{enumerate}
  \section{Version History}
  In the following are listed the differences between versions:
  \begin{enumerate}
  	\itemBold{ 22/01/2017} First version
  \end{enumerate}

  \end{document}


\section{Wheel Profile Measurement Systems}
\label{sec:sc-wpms}
The steady increasing in traffic and high-speed transit in railway lines, coupled with the increase of safety standards, requires frequent and precise monitoring of risk factors. The rolling stock wheels and their interaction with the rail, represent a key risk factor and therefore their parameters have to be measured with high accuracy and frequency. Furthermore, the high number of trains make difficult to control them periodically one by one. For these reasons the presence of autonomous measurement systems is of primary importance. System like these are called \textit{Wheel Profile Measurement Systems} (\acs{WPMS}). \\

To fully understand the importance of constantly observing the state of the wheels of a train, it is necessary to understand their geometry. Reference \cite{european-norm} is the current European standard that defines railway wheels profiles and parameters for wheels with diameter greater than or equal to $330 \, mm$. In Figure \ref{fig:uni-profile} a wheel tread profile is shown, including all the guidelines needed to produce a wheel that complies with the European regulations.
  \begin{figure}[t!]
    \centering
      \centering
      \includegraphics[width=\textwidth]{./images/wpms/uni_profile.png}
    \caption{Tread profile}
    \label{fig:uni-profile}
  \end{figure}

In the railway engineering is used to refer to wheels as \textit{wheelset}, that is a pair of wheels, fixed each other by an axle, obtaining a rotating solid (Figure \ref{fig:wheelset}). When a train is running, the two wheels belonging to the same wheelset rotate at the same speed: this is the reason why the flange is put in the inner side and the wheel is conical. During train's motion, it may happen that the wheelset is not perfectly centered with the rail, causing the wheel to hit the rail. In this case, the total motion of the wheelset is called ``winding motion''. Thanks to wheels geometry, this motion is damped gradually and rapidly, but the repeated hits wear out the profile of the wheel. The wider will be the oscillating motion, the stronger the damage to the wheel. \\
Another advantage of the conicity is when the train turns. In this case, centrifugal force pushes the wheelset to the outer side of the curve, allowing the inner wheel to rotate along a smaller diameter with respect to the outer wheel. Wheel measures reduce the number of crashing against the rail, which are, however, not null. Another way to decrease the number of these bumps is the rotation of the rails inwards. \\
All these effects greatly reduce safety during train travel, this is why an accurate and constant monitoring of the wheel status is fundamental. \\

Anyway, the great amount of detail makes the wheel reconstruction (from laser profile) a remarkably complex operation. To simplify the problem, some points of interest are usually identified (see Figures \ref{fig:uni-profile}, \ref{fig:keypoints} and \ref{fig:measures}), such as:
  \begin{itemize}
    \item The rolling point $L_0$
    \item The flange top $S$
    \item The finishing point $H_0$ of the flange, on internal face of the wheel, and its symmetrical $Q_2$
    \item The finishing points of inner and outer faces
  \end{itemize}
Despite their simplicity, these points can provide all the measures reported in Table \ref{tab:measures}. In the same table will be also indicated some definition of interest, useful for our analysis. \\

  ~\\

  \input{./src/chapters/ch3-WPMS/tab-measures.tex}

  %\medskip
  \vfill

  \begin{figure}[h!]
    \centering
    \begin{minipage}[c]{.50\textwidth}
      \centering
      \includegraphics[width=0.95\textwidth]{./images/wpms/wheel-section.png}
        \caption{Wheel section keypoints}
    \label{fig:keypoints}
    \end{minipage}%
    \begin{minipage}[c]{.50\textwidth}
      \centering
      \includegraphics[width=1.05\textwidth]{./images/wpms/wheel_parameters.png}
      \caption{Wheel section measures}
      \label{fig:measures}
    \end{minipage}
  \end{figure}

  %\medskip
  \vfill

  \begin{figure}[h!]
    \centering
    \includegraphics[width=0.7\textwidth]{./images/wpms/assile.jpg}
    \caption{Example of a wheelset}
    \label{fig:wheelset}
  \end{figure}

%  \begin{figure}[h!]
%    \centering
%    \includegraphics[width=0.6\textwidth]{./images/wpms/wheel-section.png}
%    \caption{Wheel section keypoints}
%    \label{fig:keypoints}
%  \end{figure}
%  \begin{figure}[h!]
%    \centering
%    \includegraphics[width=0.6\textwidth]{./images/wpms/wheel_parameters.png}
%    \caption{Wheel section measures}
%    \label{fig:measures}
%  \end{figure}

% Needed to force next section to begin after figures
\clearpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%

Wheel profile measurement system can be grouped into two categories, depending on the type of measurement, such as: \textit{static} measurement and \textit{dynamic} measurement.

In general, static measurement is performed in storage areas, during the maintenance of the train or of the wheel, when the train is stopped. This type of measures are done using mechanical calipers or optical methods: if we consider manual systems, the skill of the operator can influence seriously the results. Vice versa autonomous systems (typically based on optical sensors) are more precise and don't require any action of the operator. In both cases the main disadvantage is the fact that we have to disassemble the wheelset before performing the measures of interest, and as we said above, to achieve them periodically for each train is an impossible task.

Concerning dynamic systems, they allow to evaluate all the measures of interest while the train is running. These systems are typically based on contactless devices, such as electromagnetic or structured light sensors. Unlike mechanical sensors, these ones may also have a low resolution, but require a very good mathematical model that can process the collected data. In particular, structured light devices use at least two projector-camera pairs to avoid the problem of the occlusions. Furthermore, they need to collect at least three rolling points in different section of the profile to correctly evaluate the wheel diameter\cite{wpms-giuseppe}. Note that systems like these are more complex than the previous type: the high speeds of the trains and the serial acquisition of each axis need the presence of auxiliary technologies in order to detect the presence of a train, determine which axis is being analysed and pre-process collected data to reduce the transmission overhead. All this devices have to be synchronized if we want an high performance system.


\chapter{Introduction}
\label{sec:intro}

\kant[4] % Dummy text

\begin{equation}
    \iint_D \diff x \diff y
    =
    \int_0^{2\pi} \int_0^t \rho \diff \rho \diff t
    =
    \frac{4}{3} \pi^3.
\end{equation}

\section{Figures and Tables}

% Standalone with \input:
\begin{figure}[htbp]
    \centering
    \input{figures/ball}
    \caption[One ball]{One ball.}
\end{figure}

% Standalone with \includegraphics:
\begin{figure}[thbp]
    \centering
    \includegraphics{balls}
    \caption[Two balls]{Two balls.}
\end{figure}

% Todonotes:
\begin{figure}[hbp]
    \centering
    \missingfigure{Three balls.}
    \caption[Three balls]{Three balls.}
\end{figure}

\kant[5-6] % Dummy text

% Booktabs:
\begin{table}[htbp]
    \centering
    \begin{tabular}{@{}ll@{}}
        \toprule
        \textbf{Correct}               & \textbf{Incorrect}      \\
        \midrule
        \( \varphi \colon X \to Y \)   & \( \varphi : X \to Y \) \\[0.5ex]
        \( \varphi(x) \coloneqq x^2 \) & \( \varphi(x) := x^2 \) \\
        \bottomrule
    \end{tabular}
    \caption[Colons]{Proper colon usage.}
\end{table}

\begin{table}[htbp]
    \centering
    \begin{tabular}{@{}ll@{}}
        \toprule
        \textbf{Correct}     & \textbf{Incorrect}         \\
        \midrule
        \( A \implies B \)   & \( A \Rightarrow B \)      \\
        \( A \impliedby B \) & \( A \Leftarrow B \)       \\
        \( A \iff B \)       & \( A \Leftrightarrow B \)  \\
        \bottomrule
    \end{tabular}
    \caption[Arrows]{Proper arrow usage.}
\end{table}

% Tablefootnote and multirow:
\begin{table}[htbp]
    \centering
    \begin{tabular}{@{}ll@{}}
        \toprule
        \textbf{Correct}
        &
        \textbf{Incorrect}
        \\
        \midrule
        \( -1 \)
        &
        -1
        \\[0.3ex]
        1--10
        &
        1-10
        \\[0.3ex]
        Birch--Swinnerton-Dyer\tablefootnote{It is now easy to tell that Birch and Swinnerton-Dyer are two people.} conjecture
        &
        Birch-Swinnerton-Dyer conjecture
        \\[0.3ex]
        The ball \dash which is blue \dash is round.
        &
        \multirow{ 2}{*}{The ball - which is blue - is round.}
        \\[0.3ex]
        The ball---which is blue---is round.
        &
        \\
        \bottomrule
    \end{tabular}
    \caption[Dashes]{Proper dash usage.}
\end{table}

\begin{table}[hbtp]
    \centering
    \begin{tabular}{@{}*{2}{p{0.5\textwidth}}@{}}
        \toprule
        \textbf{Correct} &  \textbf{Incorrect}
        \\
        \midrule
        \enquote{This is an \enquote{inner quote} inside an outer quote}
        &
        'This is an "inner quote" inside an outer quote'
        \\
        \bottomrule
    \end{tabular}
    \caption[Quotation marks]
    {Proper quotation mark usage.
    The \texttt{\textbackslash enquote} command chooses the correct
    quotation marks for the specified language.}
\end{table}

\section{Outline}

The rest of the text is organised as follows:
\begin{description}
    \item[\cref{sec:second}]
    is second to none, with the notable exception of \cref{sec:intro}.
    The main tool introduced here is the employment of unintelligible sentences.

    \item[\cref{sec:third}]
    asserts the basic properties of being the third chapter of a text.
    This section reveals the shocking truth of filler content.

    \item[\cref{sec:fourth}]
    demonstrates how easily one can get to four chapters by simply using the \texttt{kantlipsum} package to generate dummy words.

    \item[\cref{sec:first-app}]
    features additional material for the specially interested.

    \item[\cref{sec:second-app}]
    consists of results best relegated to the back of the document,
    ensuring that nobody will ever read it.
\end{description}


\pagestyle{plain}

\addcontentsline{toc}{section}{Appendix A: Calibration}

{\Large Appendix A: Calibration}
\vskip 5mm
\hrule
\vskip 5mm

calibration of cameras and smartphone

\section*{Vision Calibration}
Before calibrating the stereo cameras and their singular quantifications it is important to take a step back and determine the camera matrix for each individual camera. This allows us to model the cameras mathematically and individually. This will allow us to accurately model elements such as lens distortion, principal ray offset and uncertainties accurately. The pinhole camera model is used to quantify camera parameters. In this model each point of

In order to obtain accurate positional information from the video cameras calibration was performed. MATLAB has a built in stereo camera calibration application that can calibrate a set of cameras and create an object containing all the essential parameters, including the individual camera intrinsics.

To calibrate the cameras pictures of a white and black chequerboard is fed in to the calibration application that then mathematically determines the camera parameters. In order to make this process more efficient a video of the the moving chequerboard was taken and various frames of importance extracted using a simple MATLAB script.

\section*{Smartphone Calibration}
calibrated out of the factory

\subsection*{Accelerometer}
Using the Smartphone IMU comes with several advantages and disadvantages. The phone sensors are calibrated inhouse during production of most smart phones, but as with all accelerometers the the element of sensor drift is inescapable. Due to the physical nature of smartphone accelerometers there will be a drifting bias. This is usually influenced by the temperature of the of the smartphone.

An easy way to quantify and eliminate this drift is to use a stationary logging test. With the smartphone body being stationary data we would expect the only acceleration to be measured that of gravity, yet when the experiment is performed we can see non zero acceleration artefacts after the gravitational acceleration has been accounted for.

\subsection*{Gyroscope}
Although the above mentioned inaccuracies are also present in the gyroscope, they are treated differently. This is mainly due to the fact that a gyroscope measure a rate of change whereas an accelerometer measure the rate of change of the rate of change.

Naturally...


\subsubsection*{Symmetric Positive Definite Linear System Solver\hspace*{\fill} \hyperlink{linearAlgebraFunctions}{(up)}\hypertarget{symmetricPositiveDefiniteSolver}{}} \addcontentsline{toc}{subsubsection}{Symmetric Positive Definite Linear System Solver}
\begin{table}[H]
\caption{Symmetric Positive Definite (SPD) Linear System Solver  \ref{tab:linearAlgebraFunctions}}
\label{tab:symmetricPositiveDefiniteSolvers}
\begin{center}
\begin{tabular}{|l|l|}
\multicolumn{2}{c}{\hyperlink{choldFunc}{\rmfamily \bfseries Cholesky Decomposition Function set}}\\
\hline
chold & Cholesky Decomposition\\
chold\_create & Create Cholesky Decomposition Object\\
chold\_destroy & Destroy CHOLD Object\\
chold\_getattr & CHOLD Get Attributes\\
cholsol & Solve SPD Linear System\\
\hline\end{tabular}
\end{center}
%\label{default}
\end{table}%


\subsection{2\texorpdfstring{\textsuperscript{k}}{k}r
analysis}\label{subsec:rect2kr}

Analysis performed with \(k\!=\!4\) and \(r\!=\!10\), for a total of \(2^4 \cdot
10 \!=\!160\) experiments.

The simulation configuration for this analysis, named ``Rectangular2kr'' can be
found in the \code{simulations.ini} file. The simulations have been run using
our \code{simulate.sh} script with the following command:
\begin{verbatim}
$ ./simulate.sh -s Rectangular -c Rectangular2kr
\end{verbatim}

We note that the user density of this configuration is the same as the one of
the high density scenario. So we will compare the results obtained mainly with
the scenario analyzed in \secref{sec:high-density}.

First, we have verified that the range of values for the \code{maxCopies}
parameter (2--6) was fine. The analysis can be found in \code{histograms.ipynb}.

Then, the 2\textsuperscript{k}r analysis can be found in \code{2kr.ipynb}. We
will verify the assumptions of normality, independence and finite variance for
the residuals in \secref{subsec:rectassumptions}. Here we will discuss the
results.

\input{scenarios/rectangular/2kr/coverage}
\input{scenarios/rectangular/2kr/collisions}
\input{scenarios/rectangular/2kr/messages}
\input{scenarios/rectangular/2kr/time}


\documentclass[12pt]{cdblatex}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% ============================================================================================
\section*{Example 6 Evaluating components}

\CdbSetup{action=hide}

\begin{cadabra}
   {\theta, \varphi}::Coordinate.
   {a,b,c,d,e,f,g,h#}::Indices(values={\theta, \varphi}, position=independent).

   \partial{#}::PartialDerivative.

   def check (objA,objB):

       tmp := @(objA) - @(objB).

       distribute     (tmp)
       sort_product   (tmp)
       rename_dummies (tmp)
       canonicalise   (tmp)

       return tmp

   import cdblib

   # ---------------------------------------------
   check000A = cdblib.get ('check000','expected/example-06.json')
   check001A = cdblib.get ('check001','expected/example-06.json')
   check002A = cdblib.get ('check002','expected/example-06.json')

   # ---------------------------------------------
   check000B = cdblib.get ('check000','output/example-06.json')
   check001B = cdblib.get ('check001','output/example-06.json')
   check002B = cdblib.get ('check002','output/example-06.json')

   # ---------------------------------------------
   diff000 = check (check000A,check000B)   # cdb (ex-06.diff000,diff000)
   diff001 = check (check001A,check001B)   # cdb (ex-06.diff001,diff001)
   diff002 = check (check002A,check002B)   # cdb (ex-06.diff002,diff002)

\end{cadabra}

% \clearpage

\begin{dgroup*}
   \Dmath*{\cdb*{ex-06.diff000}}
   \Dmath*{\cdb*{ex-06.diff001}}
   \Dmath*{\cdb*{ex-06.diff002}}
\end{dgroup*}

\end{document}


%!TEX root =  ../main.tex

\subsection{STATS in the TI}

\objective{Create appropriate regressions of data in a grapher.}

In order to proceed from numerical to algebraic and graphical models of
real-world situations, we need to perform \textbf{regression analysis}, a thoughtful
process of establishing the strength and kind of relationship between two
variables.  The entirety of Chapter 13 is about regression.
For now, we simply seek to make equations from sets of ordered pairs.
\index{regression}

\index{TI-8*!data entry}
Your TI-8* has \emph{some} capacity as a spreadsheet.  You may enter tabular data,
graph, and analyze the results on this little computer!  Most things you will need start by
pressing the \Touche[style=function,principal={stat},raise=-5pt] button.
This information is stored in lists, like \texttt{L$_1$}, \texttt{L$_2$}, \texttt{L$_3$}, etc.
These names are not letters found via the \Touche[style=alpha]
button, but are whole characters.  They can be accessed via \Touche[style=second]
\Touche[style=number, principal=1,second=L1], \Touche[style=second]
\Touche[style=number, principal=2,second=L2], etc.  The most normal way to proceed is to put $x$
data in \texttt{L$_1$} and $y$'s in \texttt{L$_2$}.

Next, in order to make data you've entered visible, one must turn on a \texttt{STAT PLOT}.
This is done via \Touche[style=second]
\Touche[style=graph,principal={y=},position = 0.9,second={stat plot},raise=-7pt].
Here you will see three different \texttt{STAT PLOT}s you can turn on or off.  Press
\Touche[style=enter,principal=enter,raise=-5pt] to select one, and then again to turn it on.
Be sure to turn \texttt{STAT PLOT}s off when you're not using them, so as not to obfuscate
other graphs.

\reminder{\lefthand}{
It is always faster to press the number than the \Touche[style=arrows, raise=-0.15cm,scalearrows=0.2]
and scroll through the menu:

\Ecran[width=4,height=3]{{\renewcommand\tabcolsep{-7pt}
\begin{tabular}{ll}
\Menu[size=10,select=true]{ZOOM} & \Menu[colourbox={ForestGreen!15}, size=10]{MEMORY} \\[-8pt]
\multicolumn{2}{l}{\Menu[colourbox={ForestGreen!15}, size=9, text={ZBox}]{1 :}} \\[-8pt]
\multicolumn{2}{l}{\Menu[colourbox={ForestGreen!15}, size=9, text={Zoom In}]{2 :}} \\[-8pt]
\multicolumn{2}{l}{\Menu[colourbox={ForestGreen!15}, size=9, text={Zoom Out}]{3 :}} \\[-8pt]
\multicolumn{2}{l}{\Menu[colourbox={ForestGreen!15}, size=9, text={ZDecimal}]{4 :}} \\[-8pt]
\multicolumn{2}{l}{\Menu[colourbox={ForestGreen!15}, size=9, text={ZSquare}]{5 :}} \\[-8pt]
\multicolumn{2}{l}{\Menu[select=true, size=9, text={ZStandard}]{6 :}} \\[-8pt]
\multicolumn{2}{l}{\Menu[colourbox={ForestGreen!15}, size=9, text={ZTrig}]{7 $\downarrow$}}
\end{tabular}
}/
}
}

\subsubsection{Windows}\index{TI-8*!window}
One of the trickier things to manage on the TI-8* is the \texttt{WINDOW}.  The Cartesian plane is
a big place, and finding your place in it can be harder still.  Having a grasp
on concepts like \gls{domain} and \gls{range} is once again worth its weight in gold.

First of all, the TI-8* does try to help you out by offering several automatic options, which
may or may not be  useful.  The most commonly used is
\Touche[style=graph,principal=zoom,position=0.9,raise=-5pt]
\Touche[style=number, principal=6,raise=-5pt]: \texttt{ZStandard}.  This is the common window,
$-10\le x\le 10$ and $-10\le y\le 10$, with tick-marks every unit.  When in doubt, start here.

If, however, the data is given to you, make the \texttt{Xmin} the smallest element in the domain,
and \texttt{Xmax} the biggest (press WINDOW).  Similarly, the range should dictate your \texttt{Ymin} and
\texttt{Ymax}.  For truly useful graphs, determine the breadth of both the domain and range,
by subtracting the smallest element from the biggest.  Divide these by 20 and you will get
the \texttt{Xscl} and \texttt{Yscl} respectively (the little tick marks).

Of course, the procedure outlined in the last paragraph will not work for most function definitions,
since they will go on forever, often in both domain and range.  This means you just have to know
what are the salient features of a given kind of graph, and how to bring them into view.  For example,
a quadratic function's most important point is the vertex, the place where it changes from increasing
to decreasing or visa versa.  Most of the time, this is involves a process of guess-and-check,
combined with instinct, which you will improve over time.

\subsubsection{Table}\index{TI-8*!table}
\Touche[style=second] \Touche[style=graph,principal={graph},position=0.9,fontsize=7pt,raise=-5pt] is
the very helpful \texttt{TABLE} of data.  This is especially useful for comparing multiple functions on
the same inputs.  The default is to start at $x=1$ and step by one from there on up.
To customize the table, you must press \Touche[style=second]
\Touche[style=graph,principal=window,position=0.9,fontsize=7pt,second=tblset,raise=-10pt].
You can either make the table
automatically increment through the domain, starting where you wish and skipping by the
same amount every time (\texttt{AUTO}), or you can manually enter values for the input (
\texttt{Ask}).

\paragraph{Linear Regression}
If you are given only two points and told to make a line through them both, then the problem is
a very easy system of two equations, and can be immediately solved via the RREF of a 2x3 matrix.
However, most scientific and real-life data is ``fuzzy,'' and no line can go through all the data points.
In such situations, it is best to enter all the data and let the machine find the ``line of best fit''.  In
simple cases, it is simply a matter of constructing the line where half the data is below and half is
above.  But as data multiplies, this becomes harder and harder.\index{regression!linear}

\begin{example}{Linear Regression}
\exProblem
Find the line of best fit for the data in this table.

\begin{tabular}{c|c}
   \textbf{x} & \textbf{y} \\ \hline
   1 & 1 \\
   2 & 2 \\
   3 & 1.3 \\
   4 & 3.75 \\
   5 & 2.25 \\
\end{tabular}

\exSolution
Using a TI-8*, press \Touche[style=function,principal=stat], \texttt{EDIT}, and enter the $x$ data in
$L_1$ and the $y$ data in $L_2$.  If
there is already data clogging up these lists, move up to their names at the top of the window
and press \Touche[style=function,principal=clear]  \Touche[style=enter,principal=enter,raise=-5pt].  Assuming
the appropriate STATPLOT is on, press \Touche[style=graph,principal={zoom},position=0.9,fontsize=7pt]
\Touche[style=number,principal=9] and inspect the graph.  It is not a simple linear graph, where
any line passing through any two of the points is perfect, so we proceed with the calculator's regression
program.

Pressing \Touche[style=function,principal=stat], moving over to \texttt{CALC}, we scroll down to
\texttt{LINREG(AX+B)}.  Depending upon your version,
you may need to type $L_1, L_2, Y_1$ or select all these setting for a menu.  ($Y_1$ is located under
\Touche[style=function,principal={vars},raise=-5pt], \texttt{Y-VARS}, \texttt{FUNCTION}, etc. and should be
entered under \texttt{Store EQ}.) Press \Touche[style=enter,principal=enter] or \texttt{CALCULATE}
and the answer of $y=.425x+.785$ appears, as well as the coefficient of determination, $r^2$.
\reminder{\lefthand}{If your
calculator doesn't show you $r^2$, you might need to press \Touche[style=second]
\Touche[style=number,principal=0,second=catalog], scroll down to \texttt{DiagnosticOn}
and turn it on.} We will learn about $r$ and $r^2$ in §13.3, but for now it is enough to say it is helpful
measure of how closely the equation ``hugs'' the data.

Finally, press \Touche[style=graph,principal=graph,position=0.9,fontsize=7pt,raise=-5pt] to see the equation
against the data.
\end{example}

\paragraph{Quadratic Regression}\index{regression!quadratic}
When the data rises and falls (or falls and rises), it is possible that a
quadratic equation would be a better fit.
Visualize the data and then consider running QuadReg and generating an equation of the form $ax^2+bx+c$.
Perfect data can be solved simply with a 3x4 matrix.

\paragraph{Exponential and Logarithmic Regression}\index{regression!exponential}
ExpReg and LnReg are great tools, if you know how to choose between them.

\includegraphics[scale=0.8]{\chapdir/pics/expln.png}\index{regression!logarithmic}

Exponential functions have a horizontal asymptote at $y=0$ if un-translated.  Logarithmic functions
has a vertical asymptote at $x=0$, if un-translated.  In most practical situations,
it is possible to tell if
there is a limiting $x$ or $y$ value.  If there are multiple asymptotes, it is probably a logistic function
(something we will study much later).

\paragraph{Sinusoidal Regression}\index{regression!sinusoidal}
While we have not formally introduced period functions yet, it is relatively easy to run a SinReg in the calculator
and get an equation from data.  The only additional questions you must answer are the number of iterations ---
how many times should the calculator pour over the data (the max is 16, which you should use for all ``fuzzy''
data) --- and the \textbf{period} --- the distance from peak to peak, or trough to trough.  For now, the period will
always be given to you.