%#!pdflatex Naruse_Esurf_2020.tex
\section{Discussion}

\subsection{Performance of inverse model}
The performance of the inverse model for turbidity currents is evaluated using the test data set, implying that this model can accurately reconstruct the flow characteristics of the turbidity currents from the spatial distribution of the thickness and grain size of turbidites (Figs. \ref{fig:test_scatter_plot} and \ref{fig:test_histogram_deviation}). The biases in the values reconstructed from the true input parameters are also very small and thus should not pose a serious issue when the method is applied to actual field data.

The inverse model not only reconstructed the initial conditions of turbidity currents accurately, but also the predicted time evolution of the flow behavior was sufficiently accurately and precisely. In the results of the forward model calculations using the predicted model input parameters that are relatively deviate from the true values (Table \ref{table:example_time_evolution}), the time evolution of the velocity and the thickness of the flow does not deviate significantly from the results using the true values (Fig. \ref{fig:test_example_time_evolution}).

Turbidity currents have a mechanism called the self-acceleration, which is caused by erosion and associated increase of the flow density \citep{parker1986self,Naruse2007,Sequeiros2009}. Therefore, even slight differences in the initial conditions of the flow can lead to very different results of the time evolution of the flow parameters. However, the results of this test imply that the accuracy of the inverse analysis in this study is enough to prevent to cause such a drastic change in the flow behavior.

The relationship between turbidity currents and characteristics of turbidites is nonlinear. Especially when the flow is self-accelerating, a small difference in the initial conditions can result in very different sedimentary characteristics. This means that it is easy to find the initial conditions of the flow by inverse analysis, because even if the characteristics of the deposits are very different, the initial conditions of the flow should not be so different. Thus, the inverse results in this case are expected to be robust even if there are some measurement errors in characteristics of deposits. In other words, there is a tradeoff between the robustness of the forward and inverse modeling.

This property of the inversion can be understood when we consider the opposite case. If the initial conditions of the flow are different but the characteristics of the turbidites are exactly the same, it is impossible to estimate the flow conditions from the turbidites. The inverse analysis of hydraulic conditions is possible because the depositional characteristics are sensitive to conditions of turbidity currents. The self-acceleration of turbidity flow is an extreme example of the sensitivity of turbidites to the flow initial conditions.

\subsection{Applicability to field-scale problems}
To apply this method to outcrops, the extent of the area that should be surveyed to collect data and the interval between outcrops should be determined. The tests with different sizes of sampling windows suggest that the survey region should be located more than 10 km from the proximal region (Fig. \ref{fig:training_different_number_length}). The loss function (i.e., the MSE of the estimates of the parameters) decreases as the length of the sampling window increases, and the best result is obtained at the 10 km-long window. Regarding the interval of the outcrops, the test results of sampling rates of more than 1.0\% with interpolation for data at non-sampled grids are not inferior to the full sample. Since the training data used in this study are computed on 5 m-spaced grids, extracting data from these grids with a 1.0\% probability is equivalent to conducting an inverse analysis from outcrop data that are distributed at 0.5 km intervals on average. Although the RMSEs of the model prediction certainly increase when the sampling rate decreases below 1.0 \%, the RMSE values does not drastically worsen until 0.5 \%. Therefore, even if the outcrop spacing is about 1 km, it should be possible to obtain a reasonable estimates of the flow characteristics.

These requirements for accurate inversion are attainable in the actual field. For example, \citet{Hirayama1977} correlated individual turbidites of the Pleistocene Otadai Formation distributed in the Boso Peninsula, Japan, on the basis of the key tuff beds. Their correlation covered a region over 30 km long with 33 outcrops. Thus, the average interval between outcrops was approximately 1 km. \citet{Amy2006} correlated individual beds in the Miocene Marnoso Arenacea Formation, Italy, using the Contessa MegaBed and an overlying ``columbine'' marker bed as the key beds. Their correlation covers 109 sections of approximately 30 m thick succession and extends over 120 km in a direction parallel to flow. Other studies in various regions (e.g., the Arnott Sandstone in France) also reported the correlation of individual turbidites in similar scale and frequency \citep{HESSE1974,Tokuhashi1979,Tokuhashi1989,Amy2000,Amy2004}. Furthermore, \citet{Bartolini1972} surveyed the Western Alboran Basin Plain, Mediterranean Sea, and discovered an individual turbidite on the sea floor at 49 cores over approximately 30 km. The records of cores in similar scale and intervals have also been reported by other studies of the modern submarine fans in different areas \citep{BORNHOLD1971, Pilkey1980}. In summary, although the method proposed in this study requires fairly high resolution data of turbidite individual beds correlated over a long distance, such conditions in ancient geological records as well as modern seafloor surveys can be achieved.

Besides these outcrop conditions, measurement errors in the field are another important factor for application. The test results suggest that the proposed inverse model of this study is very robust against random noise; random errors in the measured data have little effect on the results (Fig. \ref{fig:test_noise}). Therefore, even if localized and small-scale scouring and sedimentation occur due to some processes such as bottom currents after the deposition of a turbidite, results of inverse analysis will not be seriously affected. However, if deposits of multiple events are amalgamated to form a single thick massive sandstone, the hydraulic conditions reconstructed from the bed should be considerably different from the actual conditions. To avoid this situation, it is important to identify the erosional surface inside the bed carefully at the actual outcrop. In addition, it is safer not to analyze massive sandstones that are more than several meters thick, because they are likely to be amalgamated deposits.

Perhaps the most significant drawback to analyze actual turbidites is the assumption about the topography of the upstream submarine canyon. In this study, we tested doubling the length of the upstream slope and found that the predicted values for the concentration that were different from the original values (Fig. \ref{fig:test_slope_length}). In case of actual analysis, the upstream topography can be set correctly if the modern submarine fan is analyzed. Regarding ancient turbidites, however, some assumptions about the length and scale of the submarine canyons are necessary without measurements. In this case, it is recommended to set up various lengths of submarine canyons within a reasonable range, and to examine the degree to which these assumptions affect the inverse analysis results carefully. Nevertheless, it is worth noting that the test results were reasonable for velocity (Fig. \ref{fig:test_slope_length}), even if the assumption about the length of upstream slope was substantially different. This suggests that the inverse model proposed in this study can generally reconstruct the behavior of turbidity currents in sedimentary basins, even if the development process of turbidity currents upstream is different.

\subsection{Comparison with previous methodologies}
In existing inverse analysis methods of turbidity currents, the difference in depositional characteristics between the outputs of the forward model and the field observation is quantified as the objective function, and the initial and the boundary conditions of the forward model are determined by conducting optimization calculations to minimize the objective function \cite[e.g.,]{Nakao2017}. This is because models of turbidity currents are generally nonlinear and are difficult to linearize, especially when considering the entrainment of the basal sediment \citep{parker1986self}. Although the actual computational load depends on the choice of algorithm, this type of optimization calculation generally consists of multiple steps, and each step depends on the results of the previous calculation. Thus, the entire optimization procedure is difficult to parallelize. For instance, the kriging-based surrogate management method \citep{lesshafft2011towards} or the genetic algorithm \citep{Nakao2017} have been used to optimize the objective function for inversion of turbidity currents. In these methods, multiple calculations are conducted in each calculation step (generation), and the distribution of the objective function in the parametric space is iteratively estimated. Although the computations within each generation can be parallelized in this kind of algorithms, the next generation's computation depends on the results of the previous generation's computation, and therefore, the entire computation process cannot be parallelized. Thus, if the computational load of the forward model is high, the inverse analysis takes an unrealistic amount of time.

\citet{Parkinson2017} applied the adjoint method with the gradient-based optimization algorithm. Although the differentiation of the layer-averaged model by the adjoint method greatly reduces the load of the gradient calculation, this approach still requires an iterative calculation for optimization. Thus, the sediment entrainment process is omitted from their model. Their model does not consider resuspension (entrainment) process of sediment, whereas suspended sand in turbidity currents is maintained by balancing the effects of particle settling and diffusion from the bottom (i.e. entrainment). Their model only considers advection and settling of particles, so that the suspended sediment quickly settles and be lost over short distances at realistic flow thicknesses and concentrations. The only way to transport large amounts of suspended sediment for long distance and to deposit thick turbidites without resuspension is to make the flow extremely thick or to suppose unusually high velocity or concentration. This is the reason for that the extremely thick flow depth (more than 3000 m) was obtained in their results. Their inversion method requires iterations that cannot be parallelized, so that the forward model needs to be simplified for this purpose. In addition, gradient-based optimization tends to have problems with initial value dependency and escaping from local optimal solutions. For this reason, the results of their inverse analysis of turbidites were quite unrealistic. In contrast, we were able to adopt "full model" that incorporate the entrainment process of suspended sand into our model without any problems. As a result, our inversion did not produce any anomalous reconstructions even though most of our test data exhibit thickness and grain size distributions similar to realistic turbidites. This strongly suggests the robustness of our inverse model and its applicability to real turbidites.

Another potential approach to optimization is the Markov Chain Monte Carlo (MCMC) method, but even with this method, repetition of the forward model calculation is unavoidable, since MCMC usually requires repetition of calculations of objective function, which cannot be parallelized, more than the order of $10^4$ time. The layer-averaged model of unsteady turbidity currents is probably not suitable for the forward models due to their computational load.

The approach proposed in this study is obviously superior to existing methods in terms of applicability to the field, as it allows computationally demanding models to be applied as forward models. The general relationship between the bed and the input parameters is learned by NN rather than adjusting the input parameters of the numerical model to reproduce the characteristics of specific individual beds. The objective function used in the training of this NN is not the difference between the features of the sediment, but the precision of the inverse analysis results themselves. The most computationally demanding part of the inverse analysis method proposed here is the generation of the training data for the NN. However, since the computations of the forward models are completely independent of each other, the generation of the training data can be conducted in parallel. Thus, our method enables us to easily prepare a large number of training data by using PC clusters, even for very computationally demanding forward models. In addition, the number of calculations required for training is not as high as other methods, specifically only approximately 3,000. It is also advantageous that the proposed method enables us to perform various tests for robustness or precision of inversion before application to field examples, because the NN outputs results of inverse analysis extremely fast. For these reasons, we consider that this study successfully generated an inverse model using the layer-averaged model for unsteady turbidity currents that can be applied to the field.

\subsection{Limitations and future tasks}

The inverse model proposed in this study has several limitations. Inevitably, the accuracy of the inverse analysis is governed by the validity of the forward model that generates the training data. The present implementation of the inverse model uses the one dimensional layer-averaged model as the forward model, but this model is likely to be applicable only to sedimentary basins that are laterally constrained or to the inside of the submarine channels. The layer-averaged model of \citet{parker1986self} used in this study has been widely accepted, but various doubts have been recently raised such as the formulation of entrainment rates of basal sediment \citep{Dorrell2018} and ambient seawater \citep{Luchi2018}. The assumption of a lock exchange condition for the occurrence of turbidity currents may not be appropriate in some situations.

Although \citet{Luchi2018} suggested that the a single layer model may not be sufficient for considering behavior of turbidity currents maintained over long distances, it is expected that such turbidity currents do not leave turbidites and create a bypassing zone. Otherwise, the concentration in the lower layers of turbidity currents decrease, and therefore the currents stop within a relatively short distance. Thus, a two-layer model of turbidity currents is not always necessary for inversion of bed-scale turbidites. However, modeling of continuos sustained turbidity currents is necessary for inverse analysis of the development of submarine fans and channel-levee systems in a larger scale.

It is relatively easy to solve these problems described above. Without changing the framework of the proposed method, we can adapt to any situation by changing the forward model to generate the training data. For processes such as sediment transport, it is easy to revise the model to incorporate the state-of-the-art knowledge. By adopting computationally demanding models, inverse analysis using 2-D and 3-D forward models may be possible. In Future research, these issues should be addressed, and the methodology to actual field examples should be applied.

The analysis of ancient turbidites is an important issue in the future. However, even if ancient turbidites are analyzed, it is not possible to verify that the results obtained are correct, because the hydraulic conditions for ancient turbidity currents are unknown. Another way to verify the validity of the method is to reconstruct the hydraulic conditions of experimental turbidity currents from the turbidites deposited in the flume, and compare them with the measured values. The turbidity currents measured in the modern submarine canyons and their deposits would be another candidate to be used for the model verification.


\documentclass[a4paper,11pt]{article}
%\documentclass[a4paper,11pt]{scrartcl}

\input{../preambles/preamble}
\input{../preambles/unicode}

\setmainlanguage{english}
\setotherlanguages{german,greek,russian}

\input{../preambles/math-single}
\input{../preambles/math-brac}
\input{../preambles/math-thm}
\input{../preambles/phys-chem}

\setromanfont[Mapping=tex-text]{Linux Libertine O}
% \setsansfont[Mapping=tex-text]{DejaVu Sans}
% \setmonofont[Mapping=tex-text]{DejaVu Sans Mono}

\usepackage[style=authoryear-icomp,
			backend=biber]{biblatex}
\addbibresource{../singular-dynamics.bib}

\title{Notes on Lagrangian Singular Dynamics}
\author{Yi-Fan Wang (王\ 一帆)}
%\date{}

\begin{document}
\maketitle

\section{Classical formalism}

Lagrangian with velocity
\begin{equation}
L^\text{v} \coloneqq \fat{L}{\dot{q} = v}
\end{equation}
Equations of motion
\begin{equation}
\sum_j M_{ij}\dot{v}_j = K^\text{v}_i,\quad
\dot{q}_i = v_i.
\end{equation}
where
\begin{equation}
\rfun{M_{ij}}{q,v} \coloneqq \frpa{^2 L}{v_i\,\partial v_j}.
\end{equation}

% Adding
% \begin{equation}
% p_i \coloneqq \frpa{L^\text{v}}{v_i}.% \eqqcolon \rfun{\ol{p}_i}{q,v}.
% \end{equation}
% Variation of
% \begin{equation}
% \sfun{S}{q, p; v} \coloneqq \int\dd t\,\sbr{L^\text{v} + \sum_i
% p_i\rbr{\dot{q}_i - v_i}}.
% \end{equation}
% gives the \emph{extended Euler--Lagrange equations}
% \begin{equation}
% \dot{q}_i = v_i,\quad
% \dot{p}_i = \frpa{L^\text{v}}{q_i},\quad
% p_i = \frpa{L^\text{v}}{v_i}.
% \end{equation}
%
% Extended Hamiltonian
% \begin{equation}
% \rfun{H^\text{v}}{q, p; v} \coloneqq \sum_i p_i v_i - L^\text{v}.
% \end{equation}
% Identities
% \begin{equation}
% \frpa{H^\text{v}}{q_i} \equiv - \frpa{L^\text{v}}{q_i},\quad
% \frpa{H^\text{v}}{p_i} \equiv v_i,\quad
% \frpa{H^\text{v}}{v_i} \equiv p_i - \frpa{L^\text{v}}{v_i}.
% \end{equation}
% Variation of
% \begin{equation}
% \sfun{S}{q, p; v} \coloneqq \int\dd t\,\sbr{\sum_i
% p_i \dot{q}_i - H^\text{v}}
% \end{equation}
% gives the \emph{extended canonical equations}
% \begin{equation}
% \dot{q}_i = \sbr{q_i, H^\text{v}}_\text{P},\quad
% \dot{p}_i = \sbr{p_i, H^\text{v}}_\text{P},\quad
% \frpa{H^\text{v}}{v_i} = 0,
% \end{equation}
% where the \emph{Poisson bracket} is defined as
% \begin{equation}
% \sbr{f^\text{v}, g^\text{v}}_\text{P} \coloneqq
% \sum_i\rbr{\frpa{f^\text{v}}{q_i}\frpa{g^\text{v}}{p_i} -
% \frpa{f^\text{v}}{p_i}\frpa{g^\text{v}}{q_i}}.
% \end{equation}
%
% $v_a = \rfun{\ol{v}_a}{q,p;\cbr{v_\alpha}}$ can be solved, $a = 1, 2, \ldots,
% r_M$; $v_\alpha$
% cannot be solved, $\alpha = r_M + 1, \ldots, n$, where $r_M = \rank M$.
%
% (need to show $v_a = \rfun{\ol{v}_a}{q,p_a}$)
%
% \emph{Primary constraints in the standard form}
% \begin{equation}
% \rfun{\Phi_\alpha}{q, p} \coloneqq
% \fat{\frpa{H^\text{v}}{v_\alpha}}{\cbr{v_\alpha = \ol{v}_\alpha}} \equiv
% p_\alpha - \rfun{\ol{p}_\alpha}{q, \cbr{p_a}},
% \end{equation}
% where
% \begin{equation}
% \rfun{\ol{p}_\alpha}{q, \cbr{p_a}}
% \coloneqq \fat{\frpa{L^\text{v}}{v_\alpha}}{\cbr{v_a = \ol{v}_a}}.
% \end{equation}
%
%
% \emph{Total Hamiltonian}
% \begin{equation}
% H^\text{t} \coloneqq \fat{H^\text{v}}{\cbr{v_a = \ol{v}_a}} \equiv
% \rfun{H^\text{v}}{q, p; \cbr{\rfun{\ol{v}^a}{q, p_a; \cbr{v_\alpha}},
% v_\alpha}}.
% \end{equation}
%
% \emph{Subspace of primary constraints}
% \begin{equation}
% \Gamma_\text{P} = \cbr{ \rbr{q, p}\, |\, \rfun{\Phi_\alpha}{q, p} = 0,
% \forall \alpha}
% \end{equation}
%
% Since
% \begin{equation}
% \frpa{H^\text{t}}{v_\alpha} =
% \fat{\frpa{H^\text{v}}{v_\alpha}}{\cbr{v_a = \ol{v}_a}} = \Phi_\alpha
% \equiv p_\alpha - \rfun{\ol{p}_\alpha}{q, \cbr{p_a}},
% \end{equation}
% $H^\text{t}$ is linear in $v_\alpha$. One writes
% \begin{equation}
% \rfun{H^\text{t}}{q, \cbr{p_a}; \cbr{p_\alpha}, \cbr{v_\alpha}} = \rfun{H}{q,
% \cbr{p_a}} + \sum_\alpha v_\alpha \Phi_\alpha,
% \end{equation}
% where $H$ is the \emph{canonical Hamiltonian} or simply \emph{Hamiltonian}.
%
% \paragraph{Proposition}
% $H$ is independent of $\cbr{p_\alpha}$.
%
% \paragraph{Proposition}
% Canonical equations with primary constraints
% \begin{align}
% \dot{q}_i &= \sbr{q_i, H}_\text{P} + \sum_\beta v_\beta
% \sbr{q_i, \phi_\beta}_\text{P},
% \label{eq:q-i-primary}\\
% \dot{p}_i &= \sbr{p_i, H}_\text{P} + \sum_\beta v_\beta
% \sbr{p_i, \phi_\beta}_\text{P}, \\
% \rfun{\Phi_\alpha}{q, p} &= 0,
% \end{align}
% where $v_\beta$'s are undetermined. Note that \cref{eq:q-i-primary} for $i =
% \alpha$ holds identically: $\dot{q}_\alpha = \dot{q}_\alpha$.
%
% Weak equality: $f_1 \approx f_2$ iff $\fat{f_1}{\Gamma_\text{P}} =
% \fat{f_2}{\Gamma_\text{P}}$.
%
% \paragraph{Proposition} if $f$ and $g$ are two functions over the phase space
% $\Gamma$, and $f \approx h$, then
% \begin{align}
% \frpa{}{q_i} \rbr{f-\sum_\beta \phi_\beta \frpa{f}{p_\beta}} &\approx
% \frpa{}{q_i} \rbr{h-\sum_\beta \phi_\beta \frpa{h}{p_\beta}}, \\
% \frpa{}{p_i} \rbr{f-\sum_\beta \phi_\beta \frpa{f}{p_\beta}} &\approx
% \frpa{}{p_i} \rbr{h-\sum_\beta \phi_\beta \frpa{h}{p_\beta}}.
% \end{align}
%
% \paragraph{Corollary}
% $\forall H_1 \approx H$,
% \begin{equation}
% \dot{q}_i \approx \sbr{q_i, H}_\text{P},\qquad
% \dot{p}_i \approx \sbr{p_i, H}_\text{P}.
% \end{equation}
%
% Primary and second constraints $\phi^{(1,)}_\mu$, $\phi^{(2,)}_\omega$; first
% and second class constraints $\phi^{(,1)}_u$, $\phi^{(,2)}_w$.

\section{Examples}

%\begin{equation}
%L^\text{v} = \frac{1}{2} \sum_{i,j}\rfun{W_{ij}}{q} v_i v_j + \sum_i
%\rfun{\eta_i}{q}
%v_i - \rfun{V}{q}.
%\end{equation}

%\begin{equation}
%p_i = \frpa{L^\text{v}}{v_i} = \sum_{i,j} W_{ij} v_j + \eta_i.
%\end{equation}

%Let
%\begin{align}
%\sum_j W_{ij} e_j^{(a)} &= \lambda^{(a)} e_i \neq 0, \\
%\sum_j W_{ij} e_j^{(\alpha)} &= 0.
%\end{align}

\subsection{Toy examples}

\subsubsection*{Example 0}
\cite[sec.\ 1.2]{Gitman1990}
\begin{equation}
L = \frac{1}{2}\rbr{\dot{x}-y}^2
\end{equation}

\subsubsection*{Example 1}
\begin{equation}
L = \frac{1}{2} \dot{x}^2 + \dot{x} y - \frac{1}{2}\rbr{x-y}^2.
\end{equation}

% One has
% \begin{equation}
% L^\text{v} = \frac{1}{2} v_x^2 + v_x y - \frac{1}{2} \rbr{x-y}^2,
% \end{equation}
% so that
% \begin{equation}
% p_x = \frpa{L^\text{v}}{v_x} = v_x + y, \qquad p_y = 0,
% \end{equation}
% thus
% \begin{equation}
% \ol{v}_x = p_x - y.
% \end{equation}
% So that $v_y$ is the primary inexpressible velocity.
%
% The extended Hamiltonian reads
% \begin{equation}
% \rfun{H^\text{v}}{q, p; v} = v_x p_x + v_y p_y - \frac{1}{2} v_x^2 - v_x y
% + \frac{1}{2}\rbr{x-y}^2,
% \end{equation}
% whilst the total Hamiltonian is
% \begin{equation}
% \rfun{H^\text{t}}{q, p; \ol{v}_x, v_y} = \frac{1}{2}\rbr{p_x - y}^2 +
% \frac{1}{2} \rbr{x-y}^2 + v_y p_y.
% \end{equation}

\subsubsection*{Example 2}

\begin{equation}
L = \frac{1}{2}\dot{x}^2 + \dot{x} y + \frac{1}{2}\rbr{x-y}^2
\end{equation}

% Primary constraint
% \begin{equation}
% p_y = 0;
% \end{equation}
% total Hamiltonian
% \begin{equation}
% H^\text{t} = \frac{1}{2}p_x^2 - p_x y - \frac{1}{2} x^2 + xy + v_y p_y.
% \end{equation}

\subsection*{Example 3}

\begin{equation}
L = \frac{1}{2} \rbr{\dot{q}_2 - \ee^{q_1}}^2 + \frac{1}{2} \rbr{\dot{q}_3 -
q_2}^2.
\end{equation}

\subsection{Parametrised systems}

\subsubsection*{Non-relativistic point particle}

\cite[sec.\ 3.1.1]{Kiefer2012}
\begin{equation}
\sfun{S}{\rfun{q}{t}} \coloneqq \int_{t_0}^{t_1}\dd t\,\rfun{L}{q, \frde{q}{t}}
\end{equation}

\subsubsection*{Relativistic charged point particle}

\cite[sec.\ 16]{Landau1975},
\cite[sec.\ 3.1.2]{Kiefer2012}
\begin{equation}
S \coloneqq \int_\gamma -m\,\dd s + e \rfun{A_\mu}{x} \,\dd x^\mu
\eqqcolon \int_{\tau_0}^{\tau_1} \dd \tau\, L,\\
\label{eq:point-charged-action}
\end{equation}
where the Lagrangian reads
\begin{equation}
L = -m \sqrt{-\eta_{\mu\nu} \dot{x}^\mu \dot{x}^\nu } + q \dot{x}^\mu
\rfun{A_\mu}{x}.
\end{equation}

\begin{equation}
M_{\mu\nu} \coloneqq \frpa{^2 L}{\dot{x}^\mu\,\partial \dot{x}^\nu} =
m\frac{-\eta_{\mu\nu}\eta_{\alpha\beta} + \eta_{\mu\alpha}\eta_{\nu\beta}}%
{\rbr{-\eta_{\rho\sigma}\dot{x}^\rho \dot{x}^\sigma}^{3/2}} \dot{x}^\alpha
\dot{x}^\beta,
\end{equation}
which has one and only one zero eigenvector
\begin{equation}
\dot{x}^\mu M_{\mu\nu} = 0.
\end{equation}

Euler--Lagrange derivatives
\begin{equation}
E_\mu = \rbr{\frpa{}{x^\mu}-\frde{}{\tau}\frpa{}{\dot{x}^\mu}} L
\equiv K_\mu - M_{\mu\nu}\ddot{x}^\nu,
\label{eq:point-charged-eld}
\end{equation}
where
\begin{equation}
K_\mu \coloneqq -q F_{\mu\nu} \dot{x}^\nu,
\end{equation}
and
\begin{equation}
F_{\mu\nu} \coloneqq \partial_\mu A_\nu - \partial_\nu A_\mu.
\end{equation}

Contracting the zero eigenvector with \cref{eq:point-charged-eld} yields
\begin{equation}
\dot{x}^\mu E_\mu = \dot{x}^\mu K_\mu - \dot{x}^\mu M_{\mu\nu} \ddot{x}^\nu
\equiv 0,
\end{equation}
so that it generates a gauge identity, and no further constraint exists. Thus
the system has a symmetry
\begin{equation}
\dva x^\mu = \dot{x}^\mu \dva \lambda.
\end{equation}

\subsubsection*{Relativistic point particle with einbein}

\cite[sec.\ 2.1]{Blumenhagen2013}
\begin{equation}
L \coloneqq \frac{1}{2} \rbr{e^{-1}\eta_{\mu\nu}\dot{x}^\mu \dot{x}^\nu - m^2 e}
\label{eq:point-aux-lagrangian}
\end{equation}

Euler--Lagrange derivatives
\begin{align}
E_\mu &\coloneqq \rbr{\frpa{}{x^\mu}-\frde{}{\tau}\frpa{}{\dot{x}^\mu}}L
= e^{-1}\eta_{\mu\nu} \rbr{\frac{\dot{e}}{e}\dot{x}^\nu - \ddot{x}^\nu}, \\
E_e &\coloneqq \rbr{\frpa{}{e}-\frde{}{\tau}\frpa{}{\dot{e}}}L
= -\frac{1}{2}\rbr{\frac{1}{e^2}\eta_{\mu\nu}\dot{x}^\mu\dot{x}^\nu + m^2},
\end{align}
and collectively $E^{(0)} = \begin{pmatrix} E_\mu & E_e
\end{pmatrix}^\intercal$.

\begin{equation}
M^{(0)} \coloneqq M =
\begin{pmatrix}
e^{-1}\eta_{\mu\nu} & 0^\mu \\
0^\nu & 0
\end{pmatrix},
\end{equation}
so that the system is singular, with $w^{(0)} = \rbr{0^\mu; 1}$.

One can choose $u^{(0)} = \rbr{0^\mu; e^2}$, so that
\begin{align}
\phi^{(0)} &\coloneqq u^{(0)}\cdot E^{(0)} = e^2 E_e \\
&= -\frac{1}{2}\rbr{\eta_{\mu\nu}\dot{x}^\mu\dot{x}^\nu + m^2 e^2},
\end{align}
and thus
\begin{align}
E^{(1)}_1 &\coloneqq \dot{\phi}^{(0)} = e\rbr{2\dot{e} E_e + e\dot{E}_e} \\
&= - m^2 e\dot{e}-\eta_{\mu\nu}\dot{x}^\mu\ddot{x}^\nu.
\end{align}
Collectively, $E^{(1)} = \begin{pmatrix} \rbr{E^{(0)}}^\intercal & E^{(1)}_1
\end{pmatrix}^\intercal$.

Straightforwardly,
\begin{equation}
M^{(1)} =
\begin{pmatrix}
e^{-1}\eta_{\mu\nu} & 0^\mu \\
0^\nu & 0 \\
\eta_{\mu\nu}\dot{x}^\mu & 0
\end{pmatrix},
\end{equation}
and the new zero eigenvector $w^{(1)} = \rbr{e\dot{x}^\mu; 0, -1}$.

One finds that
\begin{align}
w^{(1)} \cdot E^{(1)} &= e\dot{x}^\mu E_\mu - E^{(1)}_1 =
e\rbr{\dot{x}^\mu E_\mu - 2 \dot{e} E_e - e\dot{E}_e} \\
&= \eta_{\mu\nu} \frac{\dot{e}}{e} \dot{x}^\mu \dot{x}^\nu  + m^2 e \dot{e}
= -2e \dot{e} E_e,
\end{align}
so that a gauge identity
\begin{equation}
G \coloneqq \dot{x}^\mu E_\mu - e \dot{E}_e \equiv 0
\end{equation}
is obtained.

\begin{equation}
G\epsilon = E_\mu \dot{x}^\mu \epsilon + E_e \rbr{\dot{e} \epsilon + e
\dot{\epsilon}} - \frde{}{\tau}\rbr{e E_e \epsilon},
\end{equation}
so that
\begin{align}
\dva x^\mu &= \dot{x}^\mu \epsilon, \\
\dva e &= \dot{e} \epsilon + e \dot{\epsilon}.
\end{align}

\subsubsection{Neutral scalar field}
\cite[sec.\ 3.3]{Kiefer2012}

\subsection{Maxwell--Proca theory}

\begin{equation}
\Ld = -\frac{1}{4} F_{\mu\nu} F^{\mu\nu} - \frac{1}{2}m^2 A_\mu A^\mu + A_\mu
J^\mu
\end{equation}
where $m > 0$ corresponds to the Proca theory \cite[sec.\ 2.3]{Gitman1990}, and
$m = 0$ the Maxwell theory \cite[sec.\ 3.3.3]{Rothe2010}, \cite[sec.\
2.4]{Gitman1990}.

\begin{equation}
\Ld \equiv \frac{1}{2}\rbr{-\eta^{\alpha\beta}\eta^{\mu\nu} + \eta^{\alpha\nu}
\eta^{\beta^\mu}}\rbr{\partial_\mu A_\alpha}\rbr{\partial_\nu A_\beta}
- \frac{1}{2} m^2 \eta^{\alpha\beta} A_\alpha A_\beta + A_\alpha J^\alpha
\end{equation}

\begin{align}
E^\alpha &= \rbr{\frpa{}{A_\alpha} - \partial_\mu \frpa{}{\rbr{\partial_\mu
A_\alpha}}}\Ld \nonumber \\
&= -m^2 A_\beta \eta^{\alpha\beta} + J^\alpha -\rbr{-\eta^{\alpha\beta}
\eta^{\mu\nu} + \eta^{\alpha\nu} \eta^{\beta^\mu}}
\partial_\mu \partial_\nu A_\beta.
\end{align}

%\subsection{Dirac field}

%\subsection{Gauge theories}

%\subsubsection{Spinor electrodynamics}

%\subsubsection{Yang--Mills theory}

%\subsubsection{Yang--Mills--Higgs theory}

\subsection{String theories}

\subsubsection*{Nambu--Gotō action}

Generalising the kinetic part of \eqref{eq:point-charged-action}, one has
\begin{equation}
S_\text{NG} \coloneqq -T \int_\Sigma \dd A
\eqqcolon -T \int_\Sigma\dd^2\sigma \Ld,
\end{equation}
where the Lagrangian density
\begin{equation}
\Ld = \sqrt{-\Gamma},\quad
\Gamma \coloneqq \det \Gamma_{\alpha\beta},\quad
\Gamma_{\alpha\beta} \coloneqq \frpa{X^\nu}{\sigma^\alpha}
\frpa{X_\nu}{\sigma^\alpha}.
\end{equation}

Historically \cite{Nambu1970,Goto1971}; Reference e.g.\
\cite{Blumenhagen2013}
\cite[sec.\ 3.2]{Kiefer2012}

\subsubsection*{Polyakov action}

Generalising \eqref{eq:point-aux-lagrangian}
\begin{equation}
\sfun{S_\text{P}}{X^\mu, h_{\alpha\beta}} = -\frac{T}{2}\int_\Sigma \Ld,
\end{equation}
where
\begin{equation}
\Ld \coloneqq \sqrt{-h} h^{\alpha\beta}\Gamma_{\alpha\beta}.
\end{equation}

Historically \cite{Brink1976,Deser1976,Polyakov1981};
Reference
\cite[sec.\ 3.2]{Kiefer2012}

\begin{equation}
\sqrt{}
\end{equation}

\subsection{Gravitation theories}

\subsubsection*{Closed Friedmann universe}
This part adapts \cite[sec.\ 8.1.2]{Kiefer2012}.

The total action reads
\begin{equation}
S \coloneqq S_\text{EG} + S_\phi,
\end{equation}
where $S_\text{EG}$ follows \eqref{eq:action-einstein-gravity}, and
\begin{equation}
S_\phi \coloneqq \int_\mscrM\dd^4 x\, \sqrt{-g}\,
\rbr{-\frac{1}{2} g^{\mu\nu} \rbr{\nabla_\mu\phi} \rbr{\nabla_\nu\phi}
-m^2\phi^2}.
\end{equation}

Adapting
\begin{equation}
\dd s^2 = -\rfun{N^2}{t}\,\dd t^2 + \rfun{a^2}{t}\,\dd\Omega_3^2,
\end{equation}
where
\begin{equation}
\d\Omega_3^2 = \dd\chi^2+\sin^2\chi\,\rbr{\dd\theta^2+\sin^2\theta\,\dd\phi^2}.
\end{equation}
One has
\begin{equation}
\sqrt{-g} = N a^3 \sin^2\chi\,\sin\theta,\qquad
\sqrt{h} = a^3\sin^2\chi\,\sin\theta;
\end{equation}
whereas
\begin{equation}
R = \frac{6}{N^2}\rbr{-\frac{\dot{N}\dot{a}}{Na} + \frac{\ddot{a}}{a} +
\rbr{\frac{\dot{a}}{a}}^2} + \frac{6}{a^2},\qquad
K = \frac{3\dot{a}}{Na}.
\end{equation}

\begin{equation}
S_\text{EG} = \frac{A_3}{16\pp\nG} \rbr{\int_{t_1}^{t_2} \dd t
Na^3\rbr{R - 2\Lambda} - \sbr{\frac{6\dot{a}a^2}{N}}_{t_1}^{t_2}},
\end{equation}
where
\begin{equation}
A_3 = \int \sin^2\chi\,\sin\theta\,\dd\chi\,\dd\theta\,\dd\phi = 2\pp^2.
\end{equation}
The term proportional to $\ddot{a}/a$ in the integrand can be integrated by
parts
\begin{equation}
\int_{t_1}^{t_2} \dd t\,Na^3 \frac{6}{N^2} \frac{\ddot{a}}{a}
= 6\rbr{\sbr{\frac{\dot{a}a^2}{N}}_{t_1}^{t_2} - \int_{t_1}^{t_2}\dd t
\,\dot{a}\frde{}{t}\frac{a^2}{N^2}},
\end{equation}
in which the first term cancels the Gibbons--Hawking--York term. One has
\begin{equation}
S_\text{EG} = \frac{3\pp}{4\nG}\int_{t_1}^{t_2} \dd t\,
\rbr{-\frac{a}{N}\dot{a}^2 + N a - \frac{\Lambda}{3}N a^3}.
\end{equation}

The matter part of the action reads
\begin{equation}
S_\phi = \pp^2 \int_{t_1}^{t_2}\dd t\,
a^3 \rbr{\frac{1}{N}\dot{\phi}^2 - m^2N\phi^2}.
\end{equation}

One derives the Euler--Lagrange derivatives
\begin{align}
E_N &= \frac{3\pp}{4\nG}\rbr{\frac{a\dot{a}^2}{N^2}+a-\frac{\Lambda a^3}{3}}
-\pp^2 a^3 \rbr{\frac{\dot{\phi}^2}{N^2} - m^2 N \phi^2} ,\\
E_a &= \frac{3\pp}{4\nG}\rbr{-\frac{\dot{a}^2}{N}+N-\Lambda Na^2
+\frac{2a\ddot{a}}{N} - \frac{2a\dot{a}\dot{N}}{N^2}}
+3\pp^2a^2\rbr{\frac{\dot{\phi}^2}{N} - m^2 N \phi^2},\\
E_\phi &= 2\pp^2 \rbr{-m^2 N a^3 \phi - \frac{3 a^2 \dot{a}\dot{\phi}}{N}
-\frac{a^3\ddot{\phi}}{N} + \frac{a^3\dot{\phi}\dot{N}}{N^2}},
\end{align}
and the primary mass matrix reads
\begin{equation}
\mbfM^{(0)} = \begin{pmatrix}
0 & 0 & 0 \\ 0 & -\frac{3\pp}{2\nG}\frac{a}{N} & 0 \\
0 & 0 & 2\pp^2\frac{a^3}{N},
\end{pmatrix}
\end{equation}
so that the system is singular, with $w^{(0)} = \begin{pmatrix}
1 & 0 & 0\end{pmatrix} \eqqcolon u^{(0)}$.

Therefore, the only primary constraint
\begin{equation}
\phi^{(0)} \coloneqq u^{(0)}\cdot E^{(0)} = E_N,
\end{equation}
and
\begin{align}
E^{(1)} &\coloneqq \dot{\phi}^{(0)} = \dot{E}_N \\
&= \frac{3\pp}{4\nG}\dot{a}
\rbr{\frac{\dot{a}^2}{N^2}+2\frac{a\ddot{a}}{N^2}
-\frac{2a\dot{a}\dot{N}}{N^3}+1-\Lambda a^2}\nonumber \\
&- \pp^2 a^2
\rbr{3\frac{\dot{a}\dot{\phi}^2}{N^2} +2\frac{a\dot{\phi}\ddot{\phi}}{N^2}
-2\frac{a\dot{N}\dot{\phi}^2}{N^3}-m^2\phi
\rbr{3N\phi\dot{a}+a\phi\dot{N}+2aN\dot{\phi}}}.
\end{align}
The secondary mass matrix
\begin{equation}
\mbfM^{(1)} = \begin{pmatrix}
0 & 0 & 0 \\ 0 & -\frac{3\pp}{2\nG}\frac{a}{N} & 0 \\
0 & 0 & 2\pp^2\frac{a^3}{N} \\
0 & -\frac{3\pp}{2\nG}\frac{a\dot{a}}{N^2} & 2\pp^2\frac{a^3\dot{\phi}}{N^2},
\end{pmatrix}
\end{equation}
and an additional left zero
\begin{equation}
w^{(1)} \coloneqq \begin{pmatrix}
0 & \dot{a} & \dot{\phi} & -N
\end{pmatrix}
\eqqcolon v^{(1)}
\end{equation}
is obtained, resulting in a gauge identity
\begin{equation}
0 \equiv G = \dot{a} E_a + \dot{\phi} E_\phi - N\dot{E}_N,
\end{equation}
which terminates the algorithm. The gauge transformation reads
\begin{equation}
\dva N = -\dot{N}\epsilon - N\dot{\epsilon},\qquad
\dva a = \dot{a}\epsilon,\qquad \dva \phi = \dot{\phi}\epsilon.
\end{equation}

\subsubsection{Einstein gravity}

\begin{equation}
S_\text{EG} = S_\text{EH} + S_\text{GHY},
\label{eq:action-einstein-gravity}
\end{equation}
where the Einstein--Hilbert action
\begin{equation}
S_\text{EH} = \frac{1}{16\pp\nG}\int_\mscrM\dd^4 x\,\sqrt{-g}\,
\rbr{R-2\mitLambda},
\end{equation}
and the Gibbons--Hawking--York action
\begin{equation}
S_\text{GHY} = -\frac{1}{8\pp\nG}\int_{\partial\mscrM}\dd^3 x\,\sqrt{h}\, K,
\end{equation}
which is named after \cite{Gibbons1977,York1972} but actually already
mentioned in \cite{Einstein1916}. See \cite{Dyer2009} for a brief review.

\printbibliography

\end{document}


\RequirePackage{amsmath}
\documentclass{jfp1}
\synctex=1

%
% the following standard packages may be helpful, but are not required
%
%\usepackage{amsmath}
\usepackage{amssymb}
% \usepackage{SIunits}            % typset units correctly
% \usepackage{courier}            % standard fixed width font
% \usepackage[scaled]{helvet} % see www.ctan.org/get/macros/latex/required/psnfss/psnfss2e.pdf
\usepackage{url}                  % format URLs
\usepackage{listings}          % format code
% \usepackage{enumitem}      % adjust spacing in enums
% \usepackage[colorlinks=false,allcolors=blue,breaklinks,draft=false]{hyperref}   % hyperlinks, including DOIs and URLs in bibliography
% % known bug: http://tex.stackexchange.com/questions/1522/pdfendlink-ended-up-in-different-nesting-level-than-pdfstartlink
% \newcommand{\doi}[1]{doi:~\href{http://dx.doi.org/#1}{\Hurl{#1}}}   % print a hyperlinked DOI
\usepackage{xspace}
\usepackage{graphicx}
\usepackage{pdfpages}
%\usepackage{pdfpages}
%\usepackage[doi]{natbib}

% \usepackage{amsthm}
% \renewcommand{\qedsymbol}{\ensuremath{\blacksquare}}
% \newtheorem{lemma}{Lemma}
% \newtheorem{corollary}{Corollary}
% \newtheorem{theorem}{Theorem}
% \newtheorem{definition}{Definition}
% \theoremstyle{plain}% default
\newtheorem{thm}{\bf Theorem} % [section]
\newtheorem{lem}[thm]{\bf Lemma}
% \newtheorem{prop}[thm]{Proposition}
\newtheorem{cor}{\bf Corollary}
% \theoremstyle{definition}
% \newtheorem{defn}{Definition} % [section]

\usepackage[inference]{semantic}

\usepackage{ifthen}
\newcommand{\isTechReport}{true} % true or false
\newcommand\includeTechReport[1]{%
  \ifthenelse{\equal{\isTechReport}{true}}
    {{#1}}
    {\ignorespaces}
\xspace}

%% avoid orphan text and headings
% \clubpenalty=10000
% \widowpenalty=10000
% \displaywidowpenalty=10000

%% bibliography
%% \usepackage[backend=biber,isbn=false,firstinits=true,style=trad-abbrv]{biblatex}
%% \addbibresource{main.bib}

\title[Dynamic Witnesses for Static Type Errors]
      {Dynamic Witnesses for Static Type Errors\thanks{This work was supported by NSF grants CCF-1422471, CCF-1223850,
CCF-1218344, CCF-1116289, CCF-0954024, Air Force grant FA8750-15-2-0075,
and a gift from Microsoft Research.}\\
      (or, Ill-Typed Programs Usually Go Wrong)}

\author[E. L. Seidel, R. Jhala, and W. Weimer]
       {ERIC L. SEIDEL, RANJIT JHALA\\
         University of California, San Diego, USA\\
         % \email{\{eseidel,jhala\}@cs.ucsd.edu}
        WESTLEY WEIMER\\
         University of Virginia, USA
         \email{eseidel@cs.ucsd.edu, jhala@cs.ucsd.edu, weimer@virginia.edu}
       }

\usepackage{tcolorbox}

% colors from http://colorbrewer2.org/?type=qualitative&scheme=Set2&n=3
% \definecolor{tree}{HTML}{66C2A5}
% \definecolor{sherrloc}{HTML}{FC8D62}
% \definecolor{fix}{HTML}{8DA0CB}
% \definecolor{sherrloc}{HTML}{FC8D62}
\definecolor{ocaml}{HTML}{8dd3c7}
\definecolor{sherrloc}{HTML}{FFFFB3}
%\definecolor{fix}{HTML}{bebada}

\tcbset{every box/.style={on line,colframe=black,size=fbox,boxrule=1pt}}
\newtcbox{\hlOcaml}{colback=red!25,toprule=0pt,bottomrule=0pt}
\newtcbox{\hlSherrloc}{colback=red!25,leftrule=0pt,rightrule=0pt}
%\newtcbox{\hlFix}{colback=fix!50}

\usepackage{commands}
\input{haskellListings}

\begin{document}
\maketitle

\input{abstract}

\input{introduction}
\input{overview}
\input{semantics}
\input{interactive-short}
\input{evaluation}
% \input{gallery}
% \input{gallery2}
\input{related}
% \input{conclusion}

\section*{Acknowledgments}

We thank Ethan Chan, Matthew Chan and Timothy Nguyen for assisting
with our user study, and we thank the anonymous reviewers and
Matthias Felleisen for their insightful feedback on earlier drafts
of this paper.
%

{
%% \printbibliography
\bibliographystyle{jfp}
%% \bibliography{paperpile}
\bibliography{main}
}

\includeTechReport
{

\clearpage
\appendix
\input{proofs}
%\input{interactive}
\section{User Study}
\label{sec:user-study-exams}
% \includepdf[pages={-},pagecommand={},scale=0.65,frame,fitpaper]{user-study.pdf}
\subsection{Version A}
\fbox{\includegraphics[width=\textwidth,page=4]{user-study.pdf}}
\newpage
\fbox{\includegraphics[width=\linewidth,page=5]{user-study.pdf}}
\newpage
\fbox{\includegraphics[width=\linewidth,page=6]{user-study.pdf}}
\newpage
\subsection{Version B}
\fbox{\includegraphics[width=\linewidth,page=1]{user-study.pdf}}
\newpage
\fbox{\includegraphics[width=\linewidth,page=2]{user-study.pdf}}
\newpage
\fbox{\includegraphics[width=\linewidth,page=3]{user-study.pdf}}
}

\end{document}


\documentclass[../main.tex]{subfiles}
\begin{document}
Up till now, we have learned both DFS and BFS in graph search. From the last chapter, we have seen how the backtracking method utilized DFS to  solve enumeration and CSP based problems. The type of problem we should solve with backtracking is the one that requires enumeration all possibilities. We have seen that most of the time, backtracking comes with high complexity. If the problem only requires us to find if
\section{Bidirectional BFS}
\url{http://theoryofprogramming.com/2018/01/21/bidirectional-search/}
\section{}
\end{document}


\chapter{Define the problem}

\section{The problem}
Given data from different sensors of a mobile device, the problem is to classify the activity of the person as one of the following: walking, walking upstairs, walking downstairs, sitting, standing, laying)

\section{History of the dataset}
The experiments have been carried out with a group of 30
volunteers within an age bracket of 19-48 years. Each
person performed six activities (walking, walking upstairs, walking downstairs, sitting, standing, laying) wearing a smartphone on the
waist. Using its embedded accelerometer and gyroscope, we
captured 3-axial linear acceleration and 3-axial angular
velocity at a constant rate of 50Hz. The experiments have
been video-recorded to label the data manually. The
obtained dataset has been randomly partitioned into two
sets, where 70\% of the volunteers was selected for
generating the training data and 30\% the test data.

The sensor signals (accelerometer and gyroscope) were
pre-processed by applying noise filters and then sampled
in fixed-width sliding windows of 2.56 sec and 50\%
overlap (128 readings/window). The sensor acceleration
signal, which has gravitational and body motion
components, was separated using a Butterworth low-pass
filter into body acceleration and gravity. The
gravitational force is assumed to have only low frequency
components, therefore a filter with 0.3 Hz cutoff
frequency was used. From each window, a vector of features
was obtained by calculating variables from the time and
frequency domain. \cite{uci-link}

\section{Attribute information}
For each record in the dataset it is provided:
\begin{enumerate}
    \item Triaxial acceleration from the accelerometer (total acceleration) and the estimated body acceleration.
    \item Triaxial Angular velocity from the gyroscope.
    \item A 561-feature vector with time and frequency domain variables.
    \item A 561-feature vector with time and frequency domain variables.
    \item Its activity label.
    \item An identifier of the subject who carried out the experiment.
\end{enumerate}

\section{First look at the dataset}

\chapter{Summarize data}

\section{Descriptive statistics}
\section{Visualize the data}
\section{Handling missing values}
\section{Standardization and normalization}
\section{Feature selection}

\chapter{Prepare data}

\section{Raw view of the dataset}
\section{Normalized view of the dataset}
\section{Standardized view of the dataset}
\section{View after feature selection}

\chapter{Evaluate algorithms}

\section{Experiment design 1}
\section{Experiment design 2}
\section{Experiment design 3}
\section{Experiment design 4}

\chapter{Improve results}

\section{Ensemble algorithms}
\section{Parameter tuning}

\chapter{Present results}
\section{Final model location}
\section{Possible extensions}


 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Faceted Embedding Model}\label{chap:faceted}
To further understand the effect of each entity type on the meaning of a word and as an effort to make the dimension of entity embeddings more interpretable, we experiment with creating embeddings with separable parts. By studying these models we gain some insights about the significance of each entity type for vector representations of words and explore if an embedding can be efficiently divided based on these types. In order to create these faceted models, we modify the well-established word- and graph-embedding techniques. In Section~\ref{sec:faceted_overview}, an overview of the objectives is given. In Section~\ref{sec:faceted_glove}, the first model based on the GloVe model is proposed. For the faceted GloVe, we experiment with different cost functions, to obtain good performance. In Section~\ref{sec:faceted_word2vec}, a faceted model based on word2vec is introduced, which takes the edges of a co-occurrence graph as input. Since the graph-based embeddings have the best performance for entity-based embeddings, we experiment with DeepWalk to generate similar embeddings for the faceted model in Section~\ref{sec:faceted_deepwalk}. For all models, the modifications to the original techniques and training procedures are discussed.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Overview and Objectives}\label{sec:faceted_overview}
Word embedding are in general very ambiguous, it is unknown what each value in the vector representation indicate, if each value corresponds to a distinct attribute of the word or shows to what degree each word belongs to a certain group. We introduce faceted embedding models to address this problems, where we divide the word embedding into components that show a pre-defined attribute of each word, namely its relation to entities with specific type.
\begin{figure}
\centering
\resizebox{0.97\textwidth}{0.16\textwidth}{
\input{images/facetted_pipeline.tex}
}
\caption{Pipeline for generating faceted embeddings. The first step is the annotation of the raw text with POS tagging, entity recognition, and disambiguation. A co-occurrence graph is extracted, which is used as input for node embedding or variation of word embedding methods.   }
\label{fig:facetted_pipeline}
\end{figure}
Since we are looking at entity-based relations, the corpus should be annotated with named entities. A general pipeline of the approach is shown in Figure~\ref{fig:facetted_pipeline}, where the entity recognition and disambiguation is applied to the input text that  is used to extract a co-occurrence graph. The graph representation of the corpus is then used by graph-based embeddings or variations of common word embedding techniques to generate a typed embedding, in which instead of capturing the semantic of a word in relation to the entire vocabulary (as normal word embeddings do), we capture the semantic in relation to entities of a specific type. The entity-entity relations are very important for the generation of faceted models, hence, unlike the entity-based embedding in Chapter~\ref{chap:entity}, applying the normal word embeddings with a word-based window would not capture the relations that these models need to reflect and extraction of a co-occurrence is graph is a mandatory step. This process is repeated for all entity types available for the model, which results in multiple embeddings, each specific to a type. These embeddings can be concatenated to generate a final embedding or used separately based on the use-case scenario. \\
Ideally, a faceted model is able to capture surrounding relations for each type separately in different components of the vector, where the relation to entities that co-occur with a word would be captured by the corresponding component. An illustration of a faceted embedding for the entity \emph{``Donald Trump''} is shown in Figure~\ref{fig:faceted_emb}. The center word is the entity \emph{``Donald Trump''} marked in red and each type is annotated with a different color in the text. Any word that is not an entity or a date is considered a term. Each component is responsible for mapping the entity \emph{``Donald Trump''} in the neighbourhood of entities of a specific type that co-occur with it. \\
\begin{figure}
\centering
\resizebox{0.97\textwidth}{0.3\textwidth}{
\input{images/facetted_emb.tex}
}
\caption{Faceted embedding of the entity \emph{``Donald Trump''} for a short paragraph. Each part of the embedding corresponds to the relation of the center word to that specific type. Each entity type is illustrated with matching color in text and embedding. }
\label{fig:faceted_emb}
\end{figure}
\noindent
Some of the potential advantages and motivations for analysing each part separately are as follows:
\begin{compactitem}
\item \emph{Interpretability of relations:} Since each part of the embedding vector encodes relations to one of the entity types, e.g., the actors part only encodes the actors in the context of a word. Through these separate components, the relations between two words or entities become more interpretable. Any similarity measure to compare two word vectors can be applied to each component separately. Thus, the similarity between words can be broken down into their different attributes. If two entities are closer in location space and farther in actor space, it is possible that they are situated close locally, but are not related to the same people. \\

\item \emph{Change in meaning:} Exploring and visualizing different components of an embedding over time gives us insights into how that entity has evolved over time, with respect to that component. When applied to a corpus with a temporal aspect, like news articles, faceted models can illustrate how a word changes its meaning in relation to different types. For example, \emph{``Donald Trump''} is mentioned frequently in location context with \emph{``Iran''} during the discussion for the nuclear deal, but some time after the abandonment of the deal the topic fades away and \emph{``Donald Trump''}  is mentioned more often in relation to other countries. This information can be extracted from the location subspace of the word embedding. Consequently, the values for the location part of the embedding will bring the word vector closer to any other entity that is also frequently mentioned with the Iran deal during the nuclear discussions, e.g., \emph{``Khamenei''} (Iran's supreme leader). Even if \emph{``Khamenei''} is not directly mentioned with \emph{``Donald Trump''}, they are mentioned in the same context location-wise, and, therefore, are mapped to points close to each other in the location domain. If embeddings are reconstructed for a time span after the nuclear abandonment, the two mentioned entities should become less similar in location aspect. Although the same experiment can be done on normal embeddings, where all the words are treated equally, the distinguishable components give us additional insights as to what aspect has invoked the change.\\

\item \emph{Interpretability of evaluation tasks:} Separable parts also play a role in interpretability of evaluation tasks. We experiment with using only a specific component in our evaluations, in Chapter~\ref{chap:eval}, to find out which type of entity surrounding a word is more influential for a certain task.\\

\item \emph{Flexible neighbourhood search:} With faceted models, the search for a similar word or entity becomes more flexible. Same as the embedding space, the search space can be divided into various types, where one can look for neighbours in a specific context. The type-specific information can be used in the search, where one can query for entities that are closer to a certain word in the temporal or location aspect. For example, \emph{``Washington''} as a political person should appear more often with the same actors and has a different actor dimension than the city of \emph{``Washington DC''}. The standard embedding treats all words equally and therefore, cannot reflect this type of similarity. Since the component for different types are independent and can be combined in an arbitrary way, a combination of different components can tailor search results further and create a type-specific search.
\end{compactitem}
To achieve this type of embeddings, we define faceted embeddings as word embeddings, where each dimension represents the relation of the embedded word to a specific type of entities surrounding it. Which types are considered during training is arbitrary, but for the purpose of this work, we choose to contain actors, locations, organisations, dates, and terms, since these types are available by the LOAD model. An example of such an embedding vector ($w$) for a corpus containing types of actor (ACT), location (LOC), organisation (ORG), date (DAT) and term (TER) is shown below: \\
\mathleft
\begin{equation}
w=\left[ \underbrace { \begin{matrix}{ a }_{ 1,1 } ... { a }_{ 1,M } \end{matrix} } |\underbrace { \begin{matrix}{ a }_{ 1,M+1 } ... { a }_{ 1,2M } \end{matrix} } |\underbrace { \begin{matrix}{ a }_{ 1,2M+1 } ... { a }_{ 1,3M } \end{matrix} } |\underbrace { \begin{matrix}{ a }_{ 1,3M+1 } ... { a }_{ 1,4M } \end{matrix} } |\underbrace { \begin{matrix}{ a }_{ 14M+1 } ... { a }_{ 1,5M } \end{matrix} }  \right]
\label{eq:concat_vec}
\end{equation}
$$ \quad  \mathrm{ACT} \quad  \qquad  \mathrm{LOC}\qquad \qquad \mathrm{ORG}\qquad \quad \qquad \mathrm{DAT}\qquad \quad  \qquad  \mathrm{TER}\qquad \qquad$$
\mathcenter
Provided that the most common dimension for word embeddings are between $100$ to $300$, to maintain the same order of dimension in faceted embeddings, all the different parts have the same dimension but in a lower magnitude ($20$ to $50$). These small embeddings are concatenated to create the final embedding, where each part is independent of the rest and is usually trained separately.\\
With these independent components, the embeddings stand to be more interpretable. Although a model that divides the embedding space into entity types has not been studied, some work has been done to make the vector representations more interpretable. In the work of Frauqui et al. transformation of word vectors into sparse (and optionally binary) vectors is proposed~\brackettext{\cite{DBLP:journals/corr/FaruquiTYDS15}}. Each vector is projected into an over-complete binary vector, where each dimension represents a feature similar to ones used in traditional NLP systems but found automatically during training. However, since their dimensions are binary valued, there is no notion of the extent to which a word participates in a particular dimension. Later, a model based on rotating the word vectors was introduced in order to improve the interpretability.~\brackettext{\cite{DBLP:conf/emnlp/ParkBO17}}. Recently, a neural network-based approach to the problem was introduced by using sparse auto-encoders~\brackettext{\cite{DBLP:conf/aaai/SubramanianPJBH18}}. Although these works shine some light on the potential meaning of each dimension, they focus purely on terms and share the problems of all term-based models. whereas, we generate embeddings that each part reflects the word relation to entities of a certain type and is based not only on terms but entities of different types. \\
As explained in Chapter~\ref{chap:background}, the aim of the word embedding is to reduce the high dimensional space of text and embed words into a meaningful space, where words with similar meaning are mapped to points close to each other. In the case of faceted embeddings, it is as if we are dividing the textual space into all possible entity types, where each subspace contains only entities of a specific type. To put it differently, the space of all words $V$ is represented as $V=A\bigcup  L\bigcup  O\bigcup  D\bigcup \overline {N} $, where $A$,$L$,$O$,$D$ are the sets of all actors, locations, organisations and, dates, respectively. The set $ T=\overline{N}$ denotes all the words which are not a named entity (terms). The goal is to map each word in the vocabulary into all the subspaces, where words that co-occur with the same entities of a specific type are mapped closer together in that particular subspace. Since all the subspaces are independent of each other (e.g., the space of all actors does not have any of the location entities inside it), each embedding learned on the different subspace is also independent. Hence, it is perfectly reasonable that two words are close in one subspace and far in another. The independence of type also allows for arbitrary combination and independent analysis of learned components after training. \\
In the remainder of this chapter, we aim to introduce faceted models as a general framework for learning embeddings with known types and separable components. To create such embeddings, we propose three approaches, two of which are based on word embedding models and one use the graph embedding techniques. It is worth noting that while we used word embedding methods, the models are modified to take a graph structure as input rather than raw text. Our three models are constructed as follows:
\begin{compactenum}
\item By modifying the cost function of the GloVe model to train a separate embedding for each entity type on an annotated corpus. Although the original GloVe uses the word co-occurrence matrix as input, we transform the model to use the adjacency matrix of a co-occurrence graph.
\item By using a variation of the word2vec model, which supports an arbitrary definition of context for each word to train an embedding for each entity type.
\item By Modifying the graph embedding model DeepWalk to generate embeddings based on neighbours of a specific type.
\end{compactenum}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Faceted GloVe}\label{sec:faceted_glove}
In the previous chapter, we indicated that the LOAD edge weight captures the corpus relations similar to the co-occurrence matrix and that meaning components can be extracted from them. In this chapter, we use this knowledge to create faceted embeddings from the weighted adjacency matrix of the LOAD graph. In the first section, the construction of a weighted adjacency matrix is explained followed by the model definition. We denote the faceted GloVe model by $f$GLV, where \emph{``f''} indicates the faceted model. The proposed model contains two type of cost functions, that are discussed separately in Sections~\ref{sec:normal_cost} and~\ref{sec:unified_cost}. \\
The complete training task can be formulated in four steps:
 \begin{enumerate}
 \item Extraction of co-occurrence matrices for different types, explained in Section~\ref{sec:adj_matrix}.
 \item For each type, a GloVe based embedding is learned using either a separate or unified cost function, discussed separately in Section~\ref{sec:normal_cost} and~\ref{sec:unified_cost}.
 \item The context embedding is kept and the focal embedding is either disregarded or added with focal addition, discussed in Section~\ref{sec:faceted_embeddings}.
 \item The results of the third step are different embeddings for each type, which can be concatenated to generate the final faceted embedding.
 \end{enumerate}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Weighted adjacency matrix}\label{sec:adj_matrix}
To learn embeddings, a weighted adjacency matrix for each type of entity and term is needed (ACT, LOC, TER, ORG, DAT). Matrices are constructed based on the weighted edge list of the co-occurrence graph, where the weights can be co-occurrence counts (identical to the GloVe model) or, any other distance measure defined by the graph. For example, the weighted adjacency matrix for actors contains the edges that have an actor as start nodes and any other type as the end node. Since in this thesis we use undirected co-occurrence graphs, the same edge exists in the opposite direction. For instance, an edge between an actor and a term is repeated twice and will be used once to create the actor matrix and once to create the term matrix. A visual construction of the matrices is shown in Figure~\ref{fig:co-matrix}, where the entries of the matrices are edge weights between words.
\begin{figure}
%\[\mathrm{ACT}=
%\begin{blockarray}{cccccccc}
% &\mathrm{TER}_1  & \color{myblue}{ \mathrm{ORG}_1 } &  \mathrm{TER}_2  & \mathrm{ACT}_1 & \mathrm{DAT}_1 & \mathrm{LOC}_1 & ... \\
%\begin{block}{c(ccccccc)}
%  \mathrm{ACT}_1 &m_{00} & \color{myblue}{ m_{01}}  &  m_{02} & m_{03} & m_{04} &m_{00}& ... \\
%  \mathrm{ACT}_2 & m_{10} & \color{myblue}{ m_{11} } &  m_{12} & m_{13} & m_{14} &m_{15}& ... \\
%  \mathrm{ACT}_3 & m_{20} &  \color{myblue}{m_{21} } &  m_{22} & m_{23} & m_{24} &m_{25}& ... \\
%  .. & ... &  \color{myblue}{...}  &  ... & ... & ... &...& ... \\
%\end{block}
%\end{blockarray}
% \]
%\[\mathrm{ORG}=
%\begin{blockarray}{cccccccc}
% &\mathrm{TER}_1  & \color{myblue}{ \mathrm{ORG}_1} &  \mathrm{TER}_2  & \mathrm{ACT}_1 & \mathrm{DAT}_1 & \mathrm{LOC}_1 & ... \\
%\begin{block}{c(ccccccc)}
%  \mathrm{ORG}_1 &m_{00} & \color{myblue}{ m_{01}}  &  m_{02} & m_{03} & m_{04} &m_{00}& ... \\
%  \mathrm{ORG}_2 & m_{10} & \color{myblue}{ m_{11} } &  m_{12} & m_{13} & m_{14} &m_{15}& ... \\
%  \mathrm{ORG} _3& m_{20} &  \color{myblue}{m_{21} } &  m_{22} & m_{23} & m_{24} &m_{25}& ... \\
%  .. & ... &  \color{myblue}{...}  &  ... & ... & ... &...& ... \\
%\end{block}
%\end{blockarray}
% \]
% \[\mathrm{DAT}=
%\begin{blockarray}{cccccccc}
% &TER1  & \color{myblue}{ \mathrm{ORG}_1} &  \mathrm{TER}_2  & \mathrm{ACT}_1 & \mathrm{DAT}_1 & \mathrm{LOC}_1 & ... \\
%\begin{block}{c(ccccccc)}
%  \mathrm{DAT}_1 &m_{00} & \color{myblue}{ m_{01}}  &  m_{02} & m_{03} & m_{04} &m_{00}& ... \\
%  \mathrm{DAT}_2 & m_{10} & \color{myblue}{ m_{11} } &  m_{12} & m_{13} & m_{14} &m_{15}& ... \\
%  \mathrm{DAT}_3 & m_{20} &  \color{myblue}{m_{21} } &  m_{22} & m_{23} & m_{24} &m_{25}& ... \\
%  .. & ... &  \color{myblue}{...}  &  ... & ... & ... &...& ... \\
%\end{block}
%\end{blockarray}
% \]
\centering
\resizebox{0.97\textwidth}{0.26\textwidth}{
\input{images/adj_matrix.tex}
}
 \caption{Faceted weighted adjacency matrix. For each type of entity a weighted adjacency matrix is created, which contains all entities of that type on rows and all words in the vocabulary in columns.}
 \label{fig:co-matrix}
\end{figure}
The columns marked in blue in matrices $\mathrm{ACT}\in R^{|V_{\mathrm{ACT}}|\times |V|}$ , $ORG\in R^{|V_{\mathrm{ORG}}|\times |V|}$  and $\mathrm{DAT}\in R^{|V_{\mathrm{DAT}}|\times |V|}$ are used to learn the actor, organisation, and date part of the ORG$1$embedding, respectively. The complete vocabulary is divided into words belonging into each entity type. As a result, if $V$ is the complete vocabulary and $A$,$L$ $O$, $D$, and $T$ are the set of all actors, location, organization, dates, and terms in the graph, we will have $V=A \bigcup L \bigcup O \bigcup D \bigcup T$. The model will learn embeddings for all entities in the row and all entities in the column of the matrix. In case of a symmetric co-occurrence matrix, the model learns the same embedding twice. Nevertheless, in our case the matrices are not symmetric, because the size of the vocabulary for each entity type differs. This asymmetry leads to a definition of a new cost functions that are explained in following sections.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Cost function of faceted embeddings}
\label{sec:faceted_embeddings}
For each different adjacency matrix, different embeddings need to be learned. To train all embeddings, two methods are proposed. First, training a network for each type separately and concatenating the outputs to generate the faceted embeddings. Second, train all the embeddings at once in a single network with a unified cost function. We refer to separate and unified cost function as $f$GLV$_{sep}$ and $f$GLV$_{uni}$, respectively.  the second method, despite the compactness, is slower than optimising separate cost functions. It is worth noting that the switching between the training method has no effect on the quality of embeddings, only the training time. For the sake of completeness however, we present both methods. Below we consider some alteration to the cost function of the original GloVe, which are considered to make faceted embedding possible. Below we list the modifications needed for the both cost functions:
\begin{figure}
{\small
\tikzset{every picture/.style={line width=0.75pt}} %set default line width to 0.75pt
\input{images/focal_addtion.tex}
}
\caption{Selectional addition of focal and context embedding in case of actor embeddings. ACT$_1$ is shown as an example, where embeddings in focal and context matrices are combined to make the final embedding. Focal embeddings are only added to the corresponding actors in the context matrix.} \label{fig:focal_addition}
\end{figure}
\begin{compactitem}
\item \emph{Asymmetry of context and focal embeddings:} Similar to the GloVe model, embeddings for both context and focal words are learned, but the focal embedding can no longer be naively summed up with the context. Instead, the focal embeddings can either be disregarded completely or added in a new way. For this purpose, we propose \emph{selectional addition}, where the focal embedding of a certain type is only added to the corresponding embedding in the context. As an example, take the adjacency matrices in Section~\ref{sec:adj_matrix} as our input, then for the matrix ACT,  focal embedding will learn to encode each row, while the context will encode the columns. The rows of the matrix will represent embeddings of all the actors with respect to the whole vocabulary. We are interested in column embeddings that encode the actor part for all the words in our vocabulary, which is the purpose of this model. Focal embedding also contains information about the whole vocabulary, but only for the focal words (actors), adding back this information in terms of selectional addition is beneficial to the model. The selectional addition adds the focal embedding of a type only to the respective entities of that type in the context matrix. The rest of embeddings remain intact, because a corresponding embedding column for them in the focal matrix does not exist. A visualization of the focal addition for type actor can be seen in Figure~\ref{fig:focal_addition}.\\

\item \emph{Final concatenation:} The output of the model is separate embeddings for each entity type. These vectors can later be combined to generate the complete faceted embedding. As the embeddings are independent, the combination can be arbitrary with only the desired components kept. \\

\item \emph{Weighting function:} The weighting function is the weighting function for the GloVe model. The faceted model is valid with or without the use of a weighting function, but since the weights of a co-occurrence graph are highly unbalanced, without a weighting function, very large weights will overpower the smaller ones and reduce performance. Therefore cutting off the weights at a certain threshold can to some extent balance out this effect. The hyperparameters of the weighting function should be tuned based on the dataset.\\

\item \emph{Normalization:} In an attempt to boost performance, weights of the graph were normalized using logarithmic normalization. With normalization, we reduce the range of values, in which the weights fluctuate. Moreover, the gradient descent algorithm can converge more smoothly to the functions minimum. Unfortunately, adding the normalization for the GloVe model made the problem ill-conditioned, so the algorithm could not converge to a solution and, therefore, is dismissed. The reason for this could be that the logarithmic normalization  might result in negative values for values between $0$ and $1$. In the cost function, a second $\log$ is applied to the inputs, and since the $\log$ of a negative number is undefined, it would result in undefined gradient updates and untrainable weights. Other than log normalization, a linear transformation like \emph{min-max normalization} was also experimented. Min-max normalization forces all weights to be between $0$ and $1$ and is calculated with Equation~\ref{eq:minmaxNom}, where $e=(e_1,...,e_n)$ is the set of input weights and $z_i$ are the $i$th normalized outputs. Unfortunately, adding this form of normalization does not improve the results and is also removed from the final model.
\begin{equation}
z_{ i }=\frac { e_{ i }-min(e) }{ max(e)-min(e) }
\label{eq:minmaxNom}
\end{equation}

\item \emph{Adding $1$ to the logarithm:} The cost function applies a logarithm to the weights. For an edge weight of $1$ the $\log(1)=0$, but the weight of $1$ should not indicate that there is no relationship between them. Therefore, we tested the model with the addition of $1$ to the $\log$ of weights, to adjust for this effect. With this change, the cost function of the GloVe model is changed to Equation~\ref{eq:log_plus}. Although in some cases this change improved the model performance, in other cases it generated poor results and was also removed from the model.
\begin{equation}
J_e=\sum _{ j=1 }^{ |V| }{}\sum _{ i=1 }^{ |V_f| }{ f({ X }_{ ij } } )(w_{ i }^{ T }\tilde{  w_{ j } } +b_{ i }+\tilde{  b_{ j } } -(log{ X }_{ ij }+1))^2
\label{eq:log_plus}
\end{equation}
\end{compactitem}
All the discussed alterations to the original cost function of the GloVe can be applied to both unified and separate cost function. In the following, we discuss the two cost functions for the faceted GloVe and explain their difference.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Separate cost functions}
\label{sec:normal_cost}

The general cost function for a single network is the one used for the GloVe model with minor modifications.
As noted in the previous section, context and focal embeddings are no longer symmetric, since the size of context vocabulary is different from the focal.
By limiting the focal words to a certain type we try to infer the impact of that type on all the words and generate type-specific embeddings. Accordingly, the context embedding in each case reflects these type-specific properties for all words. Biases for focal and context embeddings are longer be symmetric as well. Therefore, the GloVe cost function is re-written to Equation~\ref{eq:sep_cost}, where $V_f$ is the size of the focal vocabulary. The function $f$ is the weighting function, which is  applied to the edge weight $e_{ij}$ between nodes $i$ and $j$. $w_i$ is focal embedding for the start node and $w_j$ is the context embedding for the end node.
\begin{equation}
J_e=\sum _{ j=1 }^{ |V| }{}\sum _{ i=1 }^{ |V_f| }{ f({ e }_{ ij } } )(w_{ i }^{ \top }\tilde{  w_{ j } } +b_{ i }+\tilde{  b_{ j } } -log{ e }_{ ij })^2
\label{eq:sep_cost}
\end{equation}
Equation~\ref{eq:sep_cost}, minimizes the distance between the dot product of embeddings for start and end node of an edge with log of their edge weight. This cost function has to be minimized separately for each entity type, where the focal vocabulary changes to reflect the type being learned. Thus, the training will contain a separate training phases for each type and a final concatenation. \\
The Figure~\ref{fig:separate_cost}, shows visual illustration of training with separate cost function. Two networks are shown as an example, where a single edge affects embeddings of the type associated with the focal (start node), while the edge in the opposite direction affects the embedding associated with the context (end node). Each network is trained separately and a final embedding concatenation of all context embeddings of different types.
\begin{figure}
{\small
\tikzset{every picture/.style={line width=0.75pt}} %set default line width to 0.75pt
\input{images/separate_cost.tex}
}
\caption{Faceted embedding with separate cost functions. Contains training of five different networks (two are shown in the figure) for a single input. The target word is TER$_1$, which is being trained against all actors, organisations, locations, dates and terms. The final embedding is created by concatenating the columns in context embedding matrices that are associated with TER$_1$.} \label{fig:separate_cost}
\end{figure}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Unified cost function  }
\label{sec:unified_cost}
The unified cost function is simply the summation of all the separate cost functions in the previous section, where $K$ is the set of all possible types, shown in Equation~\ref{eq:unified_cost}.
\begin{equation}
J=\sum _{ k=1 }^{ |K| }{J_k}=\sum _{ k=1 }^{ |K| }{}\sum _{ j=1 }^{ |V| }{}\sum _{ i=1 }^{ |V_f| }{ f({ e }_{ ij } } )(w_{ i }^{ \top }\tilde{  w_{ j } } +b_{ i }+\tilde{  b_{ j } } -log{ e }_{ ij })^2
\label{eq:unified_cost}
\end{equation}
As a result, all embeddings can be generated with a single network. For a better understanding of the difference between the methods a visual comparison is shown in Figures~\ref{fig:separate_cost} and~\ref{fig:unified_cost} for a loss on a single input. Each input is an edge with its corresponding weight, if the input has the starting node of type actor, the rest of the layers related to other types of embeddings will be frozen during training and only the weights for the actor network will be updated. Hence, it is as if all the networks were trained separately.  \\
In Figure~\ref{fig:unified_cost}, we show an illustration of training the unified cost function. As a result of unification of the five different cost functions, one network with five times more parameters is trained, where each input edge only affects weights of the network associated with the focal word (start node). Therefore, an edge between an actor and term does not affect location related embeddings. Since edges are undirected, the edge in the opposite direction will be fed as an input to the term network and hence, no information is lost.
\begin{figure}
{\small
\tikzset{every picture/.style={line width=0.75pt}} %set default line width to 0.75pt
\input{images/unified_cost.tex}
\caption{Faceted embedding with unified cost function. Training for a single input is shown using only one network. The target word is TER$_1$, which is being trained against all actors, organisations, locations, dates and terms. While training, weights related to actors are only updated if the input edge weight has the start node type of an actor. Meanwhile, rest of the weights related to other entity-types are freezed. The final embedding is created by concatenating columns in context embedding matrices that are associated with TER$_1$.} \label{fig:unified_cost}
}
\end{figure}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Vectorized faceted embeddings}
In order to speed up the model and use batch gradient descent we need to vectorize the cost function. In the following, we go through the vectorization process for embeddings with separate costs. The same idea can also be applied during the training of the unified cost function.
If we consider $W$ as the matrix of all focal embeddings and $\tilde{ W }$ as the matrix of all context embeddings stacked row-wise, then we can vectorize the model according to Equation~\ref{eq:vectorized}.
\begin{equation}
J=(W \hat{  W }^{ \top } +B+\tilde{ B } -log{ X })^{ 2 }
\label{eq:vectorized}
\end{equation}
%The original GloVe model contains:
%\begin{itemize}
%\item $W \in R^{V \times M} $ for focal embeddings.
% \item $\tilde{ W }  \in R^{V \times M} $ for context embeddings.
%\item $B,\tilde{B} \in R^{V \times 1}$ biases for focal and context embeddings.
%\item $X \in R^{V\times V} $ co-occurrence matrix of the whole vocabulary.
%\end{itemize}
Unlike the original GloVe model, dimensions of  $W$ and $\tilde{ W } $ are no longer equal as our weighted adjacency matrix is no longer symmetric $X \in R^{|V_f|\times |V|} $.  $V_f$ is the set of all focal words, which is defined by the input. If actors are the focal words then $V_f$ is the set of all actors in the vocabulary. Parameters of the model are as follows:
\begin{itemize}
\item $W \in R^{|V_f| \times M} $ for focal embeddings.
 \item $\tilde{ W }  \in R^{|V| \times M} $ for context embeddings.
\item $B,\tilde{B} \in R^{|V_f| \times 1}$ biases for focal and context embeddings.
\item $X \in R^{|V_f|\times |V|} $ co-occurrence matrix of the whole vocabulary.
\end{itemize}
Biases have to be added row-wise to matrices to preserve the original cost function~\ref{eq:glove_cost}. Consequently, biases are transformed to column vectors in the size of the focal vocabulary. A comparison between parameter matrices of the original GloVe model ($J_g$) and faceted cost function ($J_f$) is shown below, where the size of the vocabulary is indicated by $|V|=v$ and size of the focal vocabulary by $|V_f|=v_f$.\\
\[J_{g}=\stackrel{\mbox{$W(v\times M)$ }}{%
    \begin{bmatrix}
    a_{11} & a_{12}  \cdots  a_{1M} \\
    a_{21} & a_{22}  \cdots  a_{2M} \\
    \vdots & \vdots  \ddots  \vdots \\
    a_{v1} & a_{v2}  \cdots  a_{vM}
    \end{bmatrix}%
  } .
  \stackrel{\mbox{$\tilde{W}^T(M \times v)$ }}{%
    \begin{bmatrix}
    a_{11} & a_{12}  \cdots  a_{1v} \\
    a_{21} & a_{22}  \cdots  a_{2v} \\
    \vdots & \vdots  \ddots  \vdots \\
    a_{M1} & a_{M2}  \cdots  a_{Mv}
    \end{bmatrix}%
  } +
  \stackrel{B( v \times 1)}{%
    \begin{bmatrix}
    e_1 \\
    e_2 \\
    \vdots \\
    e_{v}
    \end{bmatrix}%
   }
   +
  \stackrel{\tilde{B |V|\times 1)}}{%
    \begin{bmatrix}
    e_1 \\
    e_2 \\
    \vdots \\
    e_{v}
    \end{bmatrix}%
   }-
   \stackrel{\mbox{$logX(v \times v)$ }}{%
    \begin{bmatrix}
    a_{11} & a_{12}  \cdots  a_{1v} \\
    a_{21} & a_{22}  \cdots  a_{2v} \\
    \vdots & \vdots  \ddots  \vdots \\
    a_{v1} & a_{v2}  \cdots  a_{vv}
    \end{bmatrix}%
  }
\]
\\
\[J_{f} =\stackrel{\mbox{$W( v_{f}\times M)$ }}{%
    \begin{bmatrix}
    a_{11} & a_{12}  \cdots  a_{1M} \\
    a_{21} & a_{22}  \cdots  a_{2M} \\
    \vdots & \vdots  \ddots  \vdots \\
    a_{v1} & a_{v2}  \cdots  a_{v
_{e}M}
    \end{bmatrix}%
  } .
  \stackrel{\mbox{$\tilde{W}^T(M \times |V|)$ }}{%
    \begin{bmatrix}
    a_{11} & a_{12}  \cdots  a_{1v} \\
    a_{21} & a_{22} \cdots  a_{2v} \\
    \vdots & \vdots \ddots \vdots \\
    a_{M1} & a_{M2}  \cdots  a_{Mv}
    \end{bmatrix}%
  } +
  \stackrel{B( v_{f}\times 1)}{%
    \begin{bmatrix}
    e_1 \\
    e_2 \\
    \vdots \\
    e_{v_{f}}
    \end{bmatrix}%
   }
   +
  \stackrel{\tilde{B}(v_{f}\times 1)}{%
    \begin{bmatrix}
    e_1 \\
    e_2 \\
    \vdots \\
    e_{v_{f}}
    \end{bmatrix}%
   }-
   \stackrel{\mbox{$logX( v_{f}\ \times v)$ }}{%
    \begin{bmatrix}
    a_{11} & a_{12}  \cdots  a_{1v} \\
    a_{21} & a_{22}  \cdots  a_{2v} \\
    \vdots & \vdots  \ddots  \vdots \\
    a_{v1} & a_{v2}  \cdots  a_{v_{f}v}
    \end{bmatrix}%
  }
\]

\section{Faceted Word2vec}\label{sec:faceted_word2vec}
Word embeddings are used as input features in many NLP tasks. Although methods like word2vec capture semantic features well, they often lack task-specific features. Thus, many studies focus on modifying and tweaking the existing methods for certain tasks, such as text classification~\brackettext{\cite{DBLP:conf/coling/LiuHGWTL18}}, semantic relation classification~\brackettext{\cite{DBLP:conf/conll/HashimotoSMT15}} and dependency parsing~\brackettext{\cite{DBLP:conf/acl/BansalGL14}}. Because with faceted embedding we also aim to modify existing methods to match our specific task, we looked at a wide range of litreture on modifications of word embedding methods to match a specific context definition. In $2014$, Levy and Goldberg proposed a method to generalize the skip-gram achitecure of word2vec to include arbitrary contexts and used it to create dependency based embeddings~\brackettext{\cite{DBLP:conf/acl/LevyG14}}. We use the idea behind this model to define our own context for the skip-gram model, which allows us to learn separate components of the faceted embedding. In this section, first, we give a brief description of embeddings with arbitrary contexts by Levy and Goldberg. Then, we look at our definition of context and modifications to the original model. Finally, we use this knowledge to generate the algorithm for faceted word2vec. The model proposed by Levy and Goldberg uses the skip-gram architecture, thus, we also generate the faceted word2vec based on skip-gram and do not use the CBOW architecture.\\
For creating the faceted model using embeddings with arbitrary contexts, we follow these four steps:
\begin{enumerate}
 \item Extraction of type-specific edge lists from the co-occurrence graph, discussed in Section~\ref{sec:edges_as_pairs}.
 \item Training a low dimensional embedding for each type-specific edge list using the method proposed by  Levy and Goldberg, where focal context pairs are edges of the graph~\brackettext{\cite{DBLP:conf/acl/LevyG14}}, discussed in Section~\ref{sec:arbitarty_context}.
 \item Disregarding focal embeddings and keeping only context embeddings, as it represents the relation of a word to entities of a specific type.
 \item Concatenating context embedding of all types to create the final embedding.
 \end{enumerate}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Embeddings with arbitrary contexts}\label{sec:arbitarty_context}
The general principle behind skip-gram is to bring words that appear in similar contexts in the text, closer in embedding space, where the contexts of a word are the words surrounding it. Therefore, the context vocabulary $C$ is identical to the complete word vocabulary $V$. In other words, the focal words and their context share the same vocabulary. Nonetheless, for embeddings with arbitrary contexts, this restriction is not required. The context does not need to correspond to words and can be defined based on the use-case, which is an important attribute for the faceted model. The negative sampling objective of word2vec, shown in Equation~\ref{eq:w2v_negative}, assumes a dataset $Q$ of focal and context pairs $(f,c)$ from a large body of text and samples negative examples form $\overline{Q}$ (the noise distribution). The model assigns a low score to the random negative samples and a high score to the real focal and context pairs. Since the input of the model is word pairs, it can be constructed in an arbitrary way. Levy and Goldberg specifically look at contexts, based on the syntactic relations, where the context is defined by the type of the dependency relation between the head and the modifier. They scan the text once and create focal and context pairs based on specific dependency relations and then use the generated pairs as positive inputs. For each $(f,c) \in Q$, $n$ samples $(f,c_1),\dots,(f,c_n)$ are constructed as negative examples, where $c_j$ is drawn according to its unigram distribution raised to the power of $\frac{3}{4}$~\brackettext{\cite{DBLP:conf/acl/LevyG14}}. Their framework is not only limited to dependency relations but also allows for various context definitions~\brackettext{\cite{SCHOL:website/Levy2014}}. Following their example, we can define a type-specific context to generate faceted embeddings. We denoted the faceted word2vec model by $f$W2V, where $f$ indicates the faceted model. In the following section, we show how the edges of co-occurrence graphs are used as focal and context pairs for our study.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Edges of co-occurrence graphs as focal and context pairs}\label{sec:edges_as_pairs}
We take advantage of the definition of arbitrary context and define our context based on the types of entities surrounding a word. Our goal is to create embeddings, such that each part corresponds to the relation of the focal word to the specific type of entities as context. A co-occurrence graph extracted from annotated text can help us achieve just that. To obtain each componnent of the embedding we define a type-specific edge list, in a similar fashion as type-specific adjacency matrices for faceted GloVe, where the start nodes are entities of a specific type and end nodes can be any term or entity from the vocabulary. More formally, in the heterogeneous graph $G=(V,E)$, where $V=A\bigcup  L\bigcup  \bigcup  O\bigcup  D\bigcup T $ is the set of all nodes with types actors, locations, organisations, dates, and any non-entity word (term) and $E$ is the set of all weighted edges, we extract five edge lists where the start node is one of the types in set $V$. An example of edge lists is shown in Figure~\ref{fig:facettedword2vec}. As demonstrated in Figure~\ref{fig:facettedword2vec}, we have a different edge list for each type that are treated as a focal and context pairs $(f,c)$ for the input of skip-gram with arbitrary contexts. Context word can be any node $c \in V$ and focal word $f$ has specific type, e.g, $f \in A$ for generating actor component of the embedding. Thus, the focal and context vocabulary are not equal, as the focal vocabulary is always a subset of the context ($A \subset  V$).
\begin{figure}
\centering
\resizebox{0.80\textwidth}{0.28\textwidth}{
\input{images/facettedword2vec.tex}
}
\caption{An example of creating type specific edge lists for a graph with node type: actor (A), location (L), organisation (L), date (D), and term (T).}
\label{fig:facettedword2vec}
\end{figure}
Based on each edge list the model learns a separate embedding, with entities of a certain type as focal words and the rest of the vocabulary as context. As a result, each context embedding encodes the relation of the complete vocabulary with a certain type. For example, if we look at the context embedding for actors, it encodes the relation of the words in the vocabulary to all the actors. Naturally, if a word does not have an edge to any actor it will not be present in the context embeddings. For such cases, it makes sense to set the actor component of that word equal to a vector of zeros. For example, in Figure~\ref{fig:facettedword2vec}, $T_2$ will only have a non-zero date component and the rest are set to zero. This approach has some drawbacks, as the cosine similarity between a vector of zero and any other vector is undefined. Therefore, if a component of a vector is set to zero, that word becomes incomparable to all other words in that sub-space. It also causes  problems for clustering algorithms that require a distance measure between the points, if the cosine similarity is undefined, then we are unable to compare the words and form clusters. To solve this issue, we add a small number $0.0001$ to the vectors of all zero, mapping all the words to the same point, while keeping the component small. Moreover, clustering algorithms are able to compute the distance between the word vectors in the sub-spaces. This approach also has some unwanted implications,  if components of two words are identical, their cosine similarity is always one. Thus, any two words that do not have relations to entities of a particular type are considered similar in that subspace, which is not always the case. For example, in Figure~\ref{fig:facettedword2vec}, $T_2$ and $L_2$ both have the same organisation component, as they are not connected to any organisation node. The absence of an edge to an organisation does not necessary mean that $T_2$ and $L_2$ should be similar in organisation space, which is what the model would imply.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Faceted DeepWalk}\label{sec:faceted_deepwalk}

Another way to achieve the faceted model is by embedding the nodes of a co-occurrence matrix, where node embedding algorithm preserves the neighbourhood of a specific For this purpose, we alter the DeepWalk model to meet our needs, and introduce a \emph{type-restricted random walk}.\\
In $2017$, \emph{metapath2vec} was proposed by Dong et al. to study representation learning in heterogeneous graphs with multiple edge and node types, such as coauthor relationships graphs~\brackettext{\cite{DBLP:conf/kdd/DongCS17}}. Metapath2vec learns a low dimensional representation of the graph using random walks that are restricted to the only transition between particular types of nodes and edges. They also introduce a heterogeneous skip-gram which maximizes the probability of having the heterogeneous context for a given node~\brackettext{\cite{DBLP:journals/tkde/CaiZC18}}. For learning the faceted embeddings, however, we take a more straightforward approach. We take advantage of the weighted random walks used in Chapter~\ref{chap:entity} to generate entity embeddings, to introduce a new type-restricted random walk. With the help of this customized random walk,  we restrict the context of a word to certain entity type.
In general, faceted DeepWalk model consists of three steps:
\begin{enumerate}
\item Generating type specific corpus based on type-restricted fixed length random walks starting from all the nodes in the graphs.
\item Applying the skip-gram model on the random walk corpora.
\item Concatenating the resulting embedding for each type to achieve the final faceted model.
\end{enumerate}
\emph{Type-restricted random walk:} Formally, a type-restricted random walk in graph $G=(V,E)$ is denoted in the form of $v_1 \rightarrow v_2 \rightarrow v_3\rightarrow \dots \rightarrow v_n$, where the nodes $v_i$ belong to a certain node type. In each step, the random walker is only allowed to visit the next node if it satisfies the type restriction. For example, in a graph, where $V=A\bigcup  L\bigcup  O\bigcup  D\bigcup  T$, to sample only locations related to a node, we perform a location-restricted random walk. Consequently, the transition probability for the restricted type $L$ between node $i$ and $j$, in a weighted graph, is defined in Equation~\ref{eq:transition_type_restricted}, where $e_{i,j}$ is the edge weight between the two nodes $i$ and $j$. Given that we are removing the possibility of transitioning to any type other than the pre-defined one, the probabilities are normalized by the summation of edge weights that have that specific type as their end node. If such random walk is repeated for each node, the generated random walk corpus would reflect the separation of context needed to train the faceted models.
\begin{equation}
P_{ i,j }=\left\{
\begin{matrix}
 \frac { f(e_{ i,j }) }{ \sum _{ k\in L,f(e_{ k })\in E_{ i } }^{  }{ f(e_{ k }) }  }  & \mathrm{if}\quad j\in L \\
0 & \mathrm{if}\quad j\notin L
\end{matrix}
\right.
\label{eq:transition_type_restricted}
\end{equation}
\noindent
Function $f$, in Equation~\ref{eq:transition_type_restricted}, is the edge normalization function. Similar to our approach for entity embeddings, we use two normalization functions: \\
\begin{enumerate}
\item  $f=\mathrm{id}$, identity function or no normalization. We denote these models by $f$DW$_{id}$, where $id$ shows the identity mapping. \\
\item  $f=\log$, logarithmic normalization. These models are denoted by  $f$DW$_{log}$.\\
\end{enumerate}
\noindent
An example of the type-restricted random walk for type location is illustrated in Figure~\ref{fig:facetteddeepwalk}. Any nodes in the random walk have to be a location to be visited, otherwise, it is disregarded. Hence, by creating a type specific corpus with the type-restricted random walk, we can train a skip-gram model that learns the representation of all nodes with regards to a single entity type. By repeating this process for all node types and concatenating the results we achieve the faceted model.
\begin{figure}
\centering
\resizebox{0.60\textwidth}{0.27\textwidth}{
\input{images/facted_deepwalk.tex}
}
\caption{An example of type restricted random walk of length $2$ for node types location and starting node $A_1$.}
\label{fig:facetteddeepwalk}
\end{figure}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Similarity Between Embeddings}
In the previous chapter, we presented cosine similarity as the main similarity measure between word vectors. To take advantage of the separation in dimensions for faceted embedding, an additional vector multiplication for cosine similarity is defined, resulting into two types of similarity:
\begin{itemize}
\item \emph{Full similarity:} Using the full dimensions of vectors for computing the cosine similarity or the dot product between two words. The full embedding vector contains the concatenation of entity types available in the data.
\item \emph{Partial similarity:} Using components based on query words. For example, if the similarity between an actor and a term is requested, only components related to actors and terms in both embeddings are kept to compute the cosine similarity. This method is used to test whether a particular dimensions keeps the most informative information about a type or not. Partial similarity can also be viewed as looking at the similarity between two word vectors in a chosen subspace. For example, looking only at cosine similarity between the organisation component of two embeddings will show how close they are in the organisation subspace.
\end{itemize}

 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Summary of Faceted Embeddings}\label{sec:faceted_summary}
In this chapter, we introduced three models for learning faceted embeddings with separable components, where each component defines the relation of a word to entities of a specific type. We modified the existing word embedding and graph embedding techniques to obtain a more flexible definition of context for a word, where for each component the context is limited to entities of a certain type. As input for all models, we used the co-occurrence graph extracted on annotated text, which contains the entity-entity relations and captures their co-occurrence with terms. From the word embedding methods, we modify GloVe and word2vec and from the graph-embedding models, we chose DeepWalk with type-restricted random walks. In Chapter~\ref{chap:eval}, we evaluate the faceted models against the well-established word embedding methods on various term-based evaluation tasks and discuss their advantages and drawbacks.