\chapter[The Module System]{The Module System\label{chapter:Modules}}

The module system extends the Calculus of Inductive Constructions
providing a convenient way to structure large developments as well as
a means of massive abstraction.
%It is described in details in Judicael's thesis and Jacek's thesis

\section{Modules and module types}

\paragraph{Access path.} It is denoted by $p$, it can be either a module
variable $X$ or, if $p'$ is an access path and $id$ an identifier, then
$p'.id$ is an access path.

\paragraph{Structure element.} It is denoted by \elem\ and is either a
definition of a constant, an assumption, a definition of an inductive,
 a definition of a module, an alias of module or a module type abbreviation.

\paragraph{Structure expression.} It is denoted by $S$ and can be:
\begin{itemize}
\item an access path $p$
\item a plain structure $\struct{\nelist{\elem}{;}}$
\item a functor $\functor{X}{S}{S'}$, where $X$ is a module variable,
  $S$ and $S'$ are structure expression
\item an application $S\,p$, where $S$ is a structure expression and $p$
an access path
\item a refined structure $\with{S}{p}{p'}$ or $\with{S}{p}{t:T}$ where $S$
is a structure expression, $p$ and $p'$ are access paths, $t$ is a term
and $T$ is the type of $t$.
\end{itemize}

\paragraph{Module definition,} is written $\Mod{X}{S}{S'}$ and
 consists of a module variable $X$, a module type
$S$ which can be any structure expression and optionally a module implementation $S'$
 which can be any structure expression except a refined structure.

\paragraph{Module alias,} is written $\ModA{X}{p}$ and
 consists of a module variable $X$ and a module path $p$.

\paragraph{Module type abbreviation,} is written $\ModType{Y}{S}$, where
$Y$ is an identifier and $S$ is any structure expression .

\section{Typing Modules}

In order to introduce the typing system we first slightly extend
the syntactic class of terms and environments given in
section~\ref{Terms}. The environments, apart from definitions of
constants and inductive types now also hold any other structure elements.
Terms, apart from variables, constants and complex terms,
include also access paths.

We also need additional typing judgments:
\begin{itemize}
\item \WFT{E}{S}, denoting that a structure $S$ is well-formed,

\item \WTM{E}{p}{S}, denoting that the module pointed by $p$ has type $S$ in
environment $E$.

\item \WEV{E}{S}{\overline{S}}, denoting that a structure $S$ is evaluated to
a structure $\overline{S}$ in weak head normal form.

\item \WS{E}{S_1}{S_2}, denoting that a structure $S_1$ is a subtype of a
structure $S_2$.

\item \WS{E}{\elem_1}{\elem_2}, denoting that a structure element
  $\elem_1$ is more precise that a structure element $\elem_2$.
\end{itemize}
The rules for forming structures are the following:
\begin{description}
\item[WF-STR]
\inference{%
  \frac{
    \WF{E;E'}{}
  }{%%%%%%%%%%%%%%%%%%%%%
    \WFT{E}{\struct{E'}}
  }
}
\item[WF-FUN]
\inference{%
  \frac{
    \WFT{E;\ModS{X}{S}}{\overline{S'}}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WFT{E}{\functor{X}{S}{S'}}
  }
}
\end{description}
Evaluation of structures to weak head normal form:
\begin{description}
\item[WEVAL-APP]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{S}{\functor{X}{S_1}{S_2}}~~~~~\WEV{E}{S_1}{\overline{S_1}}\\
    \WTM{E}{p}{S_3}\qquad \WS{E}{S_3}{\overline{S_1}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{S\,p}{S_2\{p/X,t_1/p_1.c_1,\ldots,t_n/p_n.c_n\}}
  }
}
\end{description}
In the last rule, $\{t_1/p_1.c_1,\ldots,t_n/p_n.c_n\}$ is the resulting
 substitution from the inlining mechanism. We substitute in $S$ the
 inlined fields $p_i.c_i$ form $\ModS{X}{S_1}$ by the corresponding delta-reduced term $t_i$ in $p$.
\begin{description}
\item[WEVAL-WITH-MOD]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{S}{\structe{\ModS{X}{S_1}}}~~~~~\WEV{E;\elem_1;\ldots;\elem_i}{S_1}{\overline{S_1}}\\
    \WTM{E}{p}{S_2}\qquad \WS{E;\elem_1;\ldots;\elem_i}{S_2}{\overline{S_1}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \begin{array}{c}
    \WEVT{E}{\with{S}{x}{p}}{\structes{\ModA{X}{p}}{p/X}}
    \end{array}
  }
}
\item[WEVAL-WITH-MOD-REC]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{S}{\structe{\ModS{X_1}{S_1}}}\\
    \WEV{E;\elem_1;\ldots;\elem_i}{\with{S_1}{p}{p_1}}{\overline{S_2}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \begin{array}{c}
    \WEVT{E}{\with{S}{X_1.p}{p_1}}{\structes{\ModS{X}{\overline{S_2}}}{p_1/X_1.p}}
    \end{array}
  }
}
\item[WEVAL-WITH-DEF]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{S}{\structe{\Assum{}{c}{T_1}}}\\
    \WS{E;\elem_1;\ldots;\elem_i}{\Def{}{c}{t}{T}}{\Assum{}{c}{T_1}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \begin{array}{c}
    \WEVT{E}{\with{S}{c}{t:T}}{\structe{\Def{}{c}{t}{T}}}
    \end{array}
  }
}
\item[WEVAL-WITH-DEF-REC]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{S}{\structe{\ModS{X_1}{S_1}}}\\
    \WEV{E;\elem_1;\ldots;\elem_i}{\with{S_1}{p}{p_1}}{\overline{S_2}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \begin{array}{c}
    \WEVT{E}{\with{S}{X_1.p}{t:T}}{\structe{\ModS{X}{\overline{S_2}}}}
    \end{array}
  }
}

\item[WEVAL-PATH-MOD]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{p}{\structe{ \Mod{X}{S}{S_1}}}\\
    \WEV{E;\elem_1;\ldots;\elem_i}{S}{\overline{S}}
   \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{p.X}{\overline{S}}
  }
}
\inference{%
  \frac{
    \begin{array}{c}
    \WF{E}{}~~~~~~\Mod{X}{S}{S_1}\in E\\
    \WEV{E}{S}{\overline{S}}
   \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{X}{\overline{S}}
  }
}
\item[WEVAL-PATH-ALIAS]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{p}{\structe{\ModA{X}{p_1}}}\\
    \WEV{E;\elem_1;\ldots;\elem_i}{p_1}{\overline{S}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{p.X}{\overline{S}}
  }
}
\inference{%
  \frac{
    \begin{array}{c}
      \WF{E}{}~~~~~~~\ModA{X}{p_1}\in E\\
      \WEV{E}{p_1}{\overline{S}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{X}{\overline{S}}
  }
}
\item[WEVAL-PATH-TYPE]
\inference{%
  \frac{
    \begin{array}{c}
    \WEV{E}{p}{\structe{\ModType{Y}{S}}}\\
    \WEV{E;\elem_1;\ldots;\elem_i}{S}{\overline{S}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{p.Y}{\overline{S}}
  }
}
\item[WEVAL-PATH-TYPE]
\inference{%
  \frac{
    \begin{array}{c}
    \WF{E}{}~~~~~~~\ModType{Y}{S}\in E\\
    \WEV{E}{S}{\overline{S}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WEV{E}{Y}{\overline{S}}
  }
}
\end{description}
 Rules for typing module:
\begin{description}
\item[MT-EVAL]
\inference{%
  \frac{
 \WEV{E}{p}{\overline{S}}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WTM{E}{p}{\overline{S}}
  }
}
\item[MT-STR]
\inference{%
  \frac{
    \WTM{E}{p}{S}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WTM{E}{p}{S/p}
  }
}
\end{description}
The last rule, called strengthening is used to make all module fields
manifestly equal to themselves. The notation $S/p$ has the following
meaning:
\begin{itemize}
\item if $S\lra\struct{\elem_1;\dots;\elem_n}$ then
  $S/p=\struct{\elem_1/p;\dots;\elem_n/p}$ where $\elem/p$ is defined as
  follows:
  \begin{itemize}
  \item $\Def{}{c}{t}{T}/p\footnote{Opaque definitions are processed as assumptions.} ~=~ \Def{}{c}{t}{T}$
  \item $\Assum{}{c}{U}/p ~=~ \Def{}{c}{p.c}{U}$
  \item $\ModS{X}{S}/p ~=~ \ModA{X}{p.X}$
  \item $\ModA{X}{p'}/p ~=~ \ModA{X}{p'}$
  \item $\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I}/p ~=~ \Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}$
  \item $\Indpstr{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p'}{p} ~=~ \Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p'}$
  \end{itemize}
\item if $S\lra\functor{X}{S'}{S''}$ then $S/p=S$
\end{itemize}
The notation $\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}$ denotes an
inductive definition that is definitionally equal to the inductive
definition in the module denoted by the path $p$. All rules which have
$\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I}$ as premises are also valid for
$\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}$. We give the formation rule
for $\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}$ below as well as
the equality rules on inductive types and constructors. \\

The module subtyping rules:
\begin{description}
\item[MSUB-STR]
\inference{%
  \frac{
    \begin{array}{c}
      \WS{E;\elem_1;\dots;\elem_n}{\elem_{\sigma(i)}}{\elem'_i}
                                  \textrm{ \ for } i=1..m \\
      \sigma : \{1\dots m\} \ra \{1\dots n\} \textrm{ \ injective}
    \end{array}
  }{
    \WS{E}{\struct{\elem_1;\dots;\elem_n}}{\struct{\elem'_1;\dots;\elem'_m}}
  }
}
\item[MSUB-FUN]
\inference{%       T_1 -> T_2 <: T_1' -> T_2'
  \frac{
    \WS{E}{\overline{S_1'}}{\overline{S_1}}~~~~~~~~~~\WS{E;\ModS{X}{S_1'}}{\overline{S_2}}{\overline{S_2'}}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WS{E}{\functor{X}{S_1}{S_2}}{\functor{X}{S_1'}{S_2'}}
  }
}
% these are derived rules
% \item[MSUB-EQ]
% \inference{%
%   \frac{
%     \WS{E}{T_1}{T_2}~~~~~~~~~~\WTERED{}{T_1}{=}{T_1'}~~~~~~~~~~\WTERED{}{T_2}{=}{T_2'}
%   }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%     \WS{E}{T_1'}{T_2'}
%   }
% }
% \item[MSUB-REFL]
% \inference{%
%   \frac{
%     \WFT{E}{T}
%   }{
%     \WS{E}{T}{T}
%   }
% }
\end{description}
Structure element subtyping rules:
\begin{description}
\item[ASSUM-ASSUM]
\inference{%
  \frac{
    \WTELECONV{}{T_1}{T_2}
  }{
    \WSE{\Assum{}{c}{T_1}}{\Assum{}{c}{T_2}}
  }
}
\item[DEF-ASSUM]
\inference{%
  \frac{
    \WTELECONV{}{T_1}{T_2}
  }{
    \WSE{\Def{}{c}{t}{T_1}}{\Assum{}{c}{T_2}}
  }
}
\item[ASSUM-DEF]
\inference{%
  \frac{
    \WTELECONV{}{T_1}{T_2}~~~~~~~~\WTECONV{}{c}{t_2}
  }{
    \WSE{\Assum{}{c}{T_1}}{\Def{}{c}{t_2}{T_2}}
  }
}
\item[DEF-DEF]
\inference{%
  \frac{
    \WTELECONV{}{T_1}{T_2}~~~~~~~~\WTECONV{}{t_1}{t_2}
  }{
    \WSE{\Def{}{c}{t_1}{T_1}}{\Def{}{c}{t_2}{T_2}}
  }
}
\item[IND-IND]
\inference{%
  \frac{
    \WTECONV{}{\Gamma_P}{\Gamma_P'}%
    ~~~~~~~~\WTECONV{\Gamma_P}{\Gamma_C}{\Gamma_C'}%
    ~~~~~~~~\WTECONV{\Gamma_P;\Gamma_C}{\Gamma_I}{\Gamma_I'}%
  }{
    \WSE{\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I}}%
        {\Ind{}{\Gamma_P'}{\Gamma_C'}{\Gamma_I'}}
  }
}
\item[INDP-IND]
\inference{%
  \frac{
    \WTECONV{}{\Gamma_P}{\Gamma_P'}%
    ~~~~~~~~\WTECONV{\Gamma_P}{\Gamma_C}{\Gamma_C'}%
    ~~~~~~~~\WTECONV{\Gamma_P;\Gamma_C}{\Gamma_I}{\Gamma_I'}%
  }{
    \WSE{\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}}%
        {\Ind{}{\Gamma_P'}{\Gamma_C'}{\Gamma_I'}}
  }
}
\item[INDP-INDP]
\inference{%
  \frac{
    \WTECONV{}{\Gamma_P}{\Gamma_P'}%
    ~~~~~~\WTECONV{\Gamma_P}{\Gamma_C}{\Gamma_C'}%
    ~~~~~~\WTECONV{\Gamma_P;\Gamma_C}{\Gamma_I}{\Gamma_I'}%
    ~~~~~~\WTECONV{}{p}{p'}
  }{
    \WSE{\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}}%
        {\Indp{}{\Gamma_P'}{\Gamma_C'}{\Gamma_I'}{p'}}
  }
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\item[MOD-MOD]
\inference{%
  \frac{
    \WSE{S_1}{S_2}
  }{
    \WSE{\ModS{X}{S_1}}{\ModS{X}{S_2}}
  }
}
\item[ALIAS-MOD]
\inference{%
  \frac{
    \WTM{E}{p}{S_1}~~~~~~~~\WSE{S_1}{S_2}
  }{
    \WSE{\ModA{X}{p}}{\ModS{X}{S_2}}
  }
}
\item[MOD-ALIAS]
\inference{%
  \frac{
      \WTM{E}{p}{S_2}~~~~~~~~
      \WSE{S_1}{S_2}~~~~~~~~\WTECONV{}{X}{p}
  }{
    \WSE{\ModS{X}{S_1}}{\ModA{X}{p}}
  }
}
\item[ALIAS-ALIAS]
\inference{%
  \frac{
    \WTECONV{}{p_1}{p_2}
  }{
    \WSE{\ModA{X}{p_1}}{\ModA{X}{p_2}}
  }
}
\item[MODTYPE-MODTYPE]
\inference{%
  \frac{
    \WSE{S_1}{S_2}~~~~~~~~\WSE{S_2}{S_1}
  }{
    \WSE{\ModType{Y}{S_1}}{\ModType{Y}{S_2}}
  }
}
\end{description}
New environment formation rules
\begin{description}
\item[WF-MOD]
\inference{%
  \frac{
    \WF{E}{}~~~~~~~~\WFT{E}{S}
  }{
    \WF{E;\ModS{X}{S}}{}
  }
}
\item[WF-MOD]
\inference{%
  \frac{
\begin{array}{c}
  \WS{E}{S_2}{S_1}\\
  \WF{E}{}~~~~~\WFT{E}{S_1}~~~~~\WFT{E}{S_2}
\end{array}
  }{
    \WF{E;\Mod{X}{S_1}{S_2}}{}
  }
}

\item[WF-ALIAS]
\inference{%
  \frac{
    \WF{E}{}~~~~~~~~~~~\WTE{}{p}{S}
  }{
    \WF{E,\ModA{X}{p}}{}
  }
}
\item[WF-MODTYPE]
\inference{%
  \frac{
    \WF{E}{}~~~~~~~~~~~\WFT{E}{S}
  }{
    \WF{E,\ModType{Y}{S}}{}
  }
}
\item[WF-IND]
\inference{%
  \frac{
    \begin{array}{c}
      \WF{E;\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I}}{}\\
      \WT{E}{}{p:\struct{\elem_1;\dots;\elem_n;\Ind{}{\Gamma_P'}{\Gamma_C'}{\Gamma_I'};\dots}}\\
      \WS{E}{\Ind{}{\Gamma_P'}{\Gamma_C'}{\Gamma_I'}}{\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I}}
    \end{array}
  }{%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    \WF{E;\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p}}{}
  }
}
\end{description}
Component access rules
\begin{description}
\item[ACC-TYPE]
\inference{%
  \frac{
    \WTEG{p}{\struct{\elem_1;\dots;\elem_i;\Assum{}{c}{T};\dots}}
  }{
    \WTEG{p.c}{T}
  }
}
\\
\inference{%
  \frac{
    \WTEG{p}{\struct{\elem_1;\dots;\elem_i;\Def{}{c}{t}{T};\dots}}
  }{
    \WTEG{p.c}{T}
  }
}
\item[ACC-DELTA]
Notice that the following rule extends the delta rule defined in
section~\ref{delta}
\inference{%
  \frac{
    \WTEG{p}{\struct{\elem_1;\dots;\elem_i;\Def{}{c}{t}{U};\dots}}
  }{
    \WTEGRED{p.c}{\triangleright_\delta}{t}
  }
}
\\
In the rules below we assume $\Gamma_P$ is $[p_1:P_1;\ldots;p_r:P_r]$,
  $\Gamma_I$ is $[I_1:A_1;\ldots;I_k:A_k]$, and $\Gamma_C$ is
  $[c_1:C_1;\ldots;c_n:C_n]$
\item[ACC-IND]
\inference{%
  \frac{
    \WTEG{p}{\struct{\elem_1;\dots;\elem_i;\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I};\dots}}
  }{
    \WTEG{p.I_j}{(p_1:P_1)\ldots(p_r:P_r)A_j}
  }
}
\inference{%
  \frac{
    \WTEG{p}{\struct{\elem_1;\dots;\elem_i;\Ind{}{\Gamma_P}{\Gamma_C}{\Gamma_I};\dots}}
  }{
    \WTEG{p.c_m}{(p_1:P_1)\ldots(p_r:P_r){C_m}{I_j}{(I_j~p_1\ldots
       p_r)}_{j=1\ldots k}}
  }
}
\item[ACC-INDP]
\inference{%
  \frac{
    \WT{E}{}{p}{\struct{\elem_1;\dots;\elem_i;\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p'};\dots}}
  }{
    \WTRED{E}{}{p.I_i}{\triangleright_\delta}{p'.I_i}
  }
}
\inference{%
  \frac{
    \WT{E}{}{p}{\struct{\elem_1;\dots;\elem_i;\Indp{}{\Gamma_P}{\Gamma_C}{\Gamma_I}{p'};\dots}}
  }{
    \WTRED{E}{}{p.c_i}{\triangleright_\delta}{p'.c_i}
  }
}

\end{description}

% %%% replaced by \triangle_\delta
% Module path equality is a transitive and reflexive closure of the
% relation generated by ACC-MODEQ and ENV-MODEQ.
% \begin{itemize}
% \item []MP-EQ-REFL
% \inference{%
%   \frac{
%     \WTEG{p}{T}
%   }{
%     \WTEG{p}{p}
%   }
% }
% \item []MP-EQ-TRANS
% \inference{%
%   \frac{
%     \WTEGRED{p}{=}{p'}~~~~~~\WTEGRED{p'}{=}{p''}
%   }{
%     \WTEGRED{p'}{=}{p''}
%   }
% }

% \end{itemize}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "Reference-Manual"
%%% End:


\section{Polysemous words prediction}
\label{sec:polysemous-words-prediction}
In this section, we look at various methods for estimating the number of word meanings and predicting whether or not a word is polysemous. In particular, we will investigate the notion of topological polysemy in \cref{sec:analysis-of-embeddings-topological-polysemy} and Geometric Anomaly Detection (GAD) in \cref{sec:analysis-of-embeddings-geometric-anomaly-detection}. We will use topological polysemy to estimate the number of word meanings of word embeddings. Particularly, we would like to see if the $\text{TPS}_n(w)$ score measures polysemy. Following this, we would like to see if singular word embeddings identified by GAD are polysemous. Next, we will compute the estimated intrinsic dimension of word embeddings and compare the results with the number of word meanings in \cref{sec:analysis-of-embeddings-intrinsic-dimension-estimation}. Finally, we end the section by proposing supervised models for predicting the number of word meanings and whether or not a word is polysemous in \cref{sec:analysis-of-embeddings-supervised-polysemy-prediction}.

\subsection{Topological polysemy}
\label{sec:analysis-of-embeddings-topological-polysemy}
In this subsection, we will apply topological polysemy (\cref{sec:topological-polysemy}) to the word embeddings of the SGNS-enwiki model. Additionally, we will also train a word2vec model using the same training data as in \cite{jakubowski2020topology}, which we refer to as the \textit{SGNS-semeval} model. We will apply topological polysemy to its word embeddings. Furthermore, we will compare the results to topological polysemy applied to word embeddings from pre-trained models, namely the fastText model (\textit{fastText.TPS.300d}) used in experiments of \cite{jakubowski2020topology}, the \textit{GoogleNews-vectors-negative300} (shortened to \textit{GoogleNews300}) word embeddings from \cite{GoogleCodeArchiveWord2vec}, the \textit{glove.840B.300d} word embeddings from \cite{GloVeProject2014} and the English (\textit{fastText.en.300d}) word embeddings from \cite{grave2018learning}. The fastText.TPS.300d model was kindly given in private communication with one of the authors of topological polysemy \cite{ZibrowiusPrivComs2021}.

The authors of topological polysemy, \cite{jakubowski2020topology}, trained a fastText model on training data from the \textit{SemEval-2010 Task 14: Evaluation Setting for Word Sense Induction \& Disambiguation Systems} \cite{manandhar-klapaftis-2009-semeval}. The training data from the SemEval task consists of several sentences related to 100 polysemous words (50 nouns and 50 verbs). The SemEval data set also includes the number of true meanings (also called \textit{gold standard} or \textit{GS}) for each of the 100 polysemous words, as perceived by humans. In private communication with one of the authors of the topological polysemy measure \cite{ZibrowiusPrivComs2021}, we received some additional information regarding their training and data preprocessing choices. In particular, they stated that they used a fastText model with vector dimensionality of 300, that they removed all punctuation from words and replaced capital letters with the corresponding small letters. To compare with the $\text{TPS}_n(w)$, the authors use the 100 polysemous words, words from the SemEval training data that has a \textit{WordNet} \cite{fellbaum1998} entry and all words in SemEval training data. WordNet is a lexical database of the English language. In particular, it allows for querying nearly any word from the English language and returns the \textit{synsets} of the word. The synset of a word $w$ is a word that shares a similar meaning to the word $w$. In other words, by querying a word in WordNet, we can get the number of meanings of a word, as perceived by WordNet. Furthermore, the Pearson correlation coefficient \cite{James2013} is computed between $\text{TPS}_n(w)$ and GS, the number of synsets for WordNet words and the word frequency as they appear in the SemEval training data, respectively. \cite{jakubowski2020topology} shows that there is a moderate (positive) correlation between $\text{TPS}_n(w)$ and GS at $n \in \enclc{40, 50, 60}$, a decreasing correlation between $\text{TPS}_n(w)$ and the number of synsets for WordNet words and no correlation between $\text{TPS}_n(w)$ and word frequencies.

Furthermore, we will describe how we applied topological polysemy to our word embeddings. We first implemented topological polysemy using the steps we described in \cref{sec:topological-polysemy}, utilizing multiprocessing and the ScaNN \cite{scann2020} approximate nearest neighbour algorithm to speed up the computation. We chose ScaNN because it performs well when applied to word embeddings, as shown in \cite{AnnBenchmarks2021}. We used the \path{ripser} \cite{ctralie2018ripser} Python package to compute Vietoris–Rips complexes. Next, we trained the SGNS-semeval model using the training data from the SemEval task and the hyperparameters used to train the SGNS-enwiki model from \cref{sec:word2vec-hyperparameter-choices}. From the training of SGNS-semeval, we got a vocabulary of size $\sim$122K words and a corpus of size $\sim$67 million. Following, we will compare the results from the experiments of \cite{jakubowski2020topology} by computing topological polysemy at varying levels of $n$ using the word embeddings of SGNS-enwiki and SGNS-semeval. Finally, we will compare the results using the SGNS-enwiki and SGNS-semeval word embeddings to the word embeddings of the fastText.TPS.300d, GoogleNews300, glove.840B.300d and fastText.en.300d models.

We computed topological polysemy at varying levels of $n$ using the word embeddings of the SGNS-enwiki and SGNS-semeval models, and show the results in \cref{table:tps-n-correlation-sgns-enwiki,table:tps-n-correlation-sgns-semeval}. In \cref{table:tps-n-correlation-sgns-enwiki}, we see that the correlation between $\text{TPS}_n$ and GS is rather stable with respect to $n$. In particular, we notice that the correlation between $\text{TPS}_n$ and GS is negative, suggesting a relationship in the opposite direction of the results from \cite[Table 1]{jakubowski2020topology}. Nonetheless, we see a decreasing correlation when comparing $\text{TPS}_n$ versus the number of WordNet synsets for each word, and a negligible correlation between $\text{TPS}_n$ and word frequencies of the top 10000 most common words. Furthermore, in \cref{table:tps-n-correlation-sgns-semeval}, we observe a decreasing negative correlation going towards zero between $\text{TPS}_n$ and GS, meaning that the SGNS-semeval model performs worse than the SGNS-enwiki model on this particular task. This result indicates that by training SGNS-semeval on a smaller vocabulary than the vocabulary of SGNS-enwiki, we get worse results. Furthermore, we see a decreasing correlation between $\text{TPS}_n$ and the number of WordNet synsets and a negligible correlation between $\text{TPS}_n$ and word frequencies of the top 10000 most common words. Although the negative correlation between $\text{TPS}_n$ and the number of WordNet synsets is larger for the SGNS-semeval model than the SGNS-enwiki model, it is still not particularly large. In addition to this, we are considering a lot fewer words when computing the correlation in the SGNS-semeval model than the SGNS-enwiki model (see sample size).
\begin{table}[H]
    \centering
    \begin{tabular}{@{}rrrr@{}}
    \toprule
    $n$ & $\text{TPS}_n$ vs. GS & $\text{TPS}_n$ vs. synsets & $\text{TPS}_n$ vs. frequency \\
    \midrule
    \trcolor 10  & -0.353        & -0.077             & \textbf{-0.043}               \\
    40  & \textbf{-0.383}        & -0.181             & -0.041               \\
    \trcolor 50  & -0.380        & -0.190             & -0.041               \\
    60  & -0.381        & -0.196             & -0.040               \\
    \trcolor 100 & -0.380        & \textbf{-0.205}             & -0.033               \\
    \midrule
    \textit{sample size} & 98 & 144412 & 10000 \\
    \bottomrule
    \end{tabular}
    \caption{Correlations between $\text{TPS}_n$ and the number of word meanings as perceived by the SemEval gold standard (GS), the number of WordNet synsets and the word frequencies of the top 10000 most common words from the SGNS-enwiki model. \textbf{Bold} values indicate the largest (absolute) correlation.}
    \label{table:tps-n-correlation-sgns-enwiki}
\end{table}
\begin{table}[H]
    \centering
    \begin{tabular}{@{}rrrr@{}}
    \toprule
    $n$ & $\text{TPS}_n$ vs. GS & $\text{TPS}_n$ vs. synsets & $\text{TPS}_n$ vs. frequency \\
    \midrule
    \trcolor 10  & \textbf{-0.300}        & -0.248             & 0.102                \\
    40  & -0.201        & -0.300             & \textbf{0.120}                \\
    \trcolor 50  & -0.194        & -0.304             & 0.116                \\
    60  & -0.169        & -0.306             & 0.110                \\
    \trcolor 100 & -0.130        & \textbf{-0.310}             & 0.098                \\
    \midrule
    \textit{sample size} & 100 & 62111 & 10000 \\
    \bottomrule
    \end{tabular}
    \caption{Correlations between $\text{TPS}_n$ and the number of word meanings as perceived by the SemEval gold standard (GS), the number of WordNet synsets and the word frequencies of the top 10000 most common words from the SGNS-semeval model. \textbf{Bold} values indicate the largest (absolute) correlation.}
    \label{table:tps-n-correlation-sgns-semeval}
\end{table}

To further broaden our understanding of the results we got from computing topological polysemy of the word embeddings of the SGNS-enwiki and the SGNS-semeval models, we plot $\text{TPS}_n(w)$ against the SemEval gold standard (GS), the number of WordNet synsets and word frequencies. We visualize the results in \cref{fig:tps-n-correlation-sgns-enwiki,fig:tps-n-correlation-sgns-semeval}, and for each plot, we let $n$ be equal to the most optimal value for each column in \cref{table:tps-n-correlation-sgns-enwiki,table:tps-n-correlation-sgns-semeval}. In \cref{fig:tps-n-correlation-sgns-enwiki}, we see a similar situation to the results from \cite[Figures 8 and 9]{jakubowski2020topology}, namely that we see an indication of a linear relationship between $\text{TPS}_n(w)$ and the SemEval gold standard and that we see a clear trend between $\text{TPS}_n(w)$ and the number of synsets in WordNet. In \cref{fig:tps-n-correlation-sgns-enwiki} (c) it is clear that there is no apparent relationship between $\text{TPS}_n(w)$ and the word frequencies. Following, we see a similar situation appearing in \cref{fig:tps-n-correlation-sgns-semeval}. These results suggest that, by computing $\text{TPS}_n(w)$ of the SGNS-enwiki word embeddings, which has a vocabulary much larger than in the experiments of \cite{jakubowski2020topology}, we are unable to use $\text{TPS}_n(w)$ alone for predicting the number of word meanings, as given by the number of WordNet synsets.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/tps-n-correlation-sgns-enwiki.pdf}
    \caption{$\text{TPS}_n(w)$ of the word embeddings of the SGNS-enwiki model plotted against the SemEval gold standard (GS) (a), the number of WordNet synsets (b) and word frequencies (c). The plots are inspired by \cite[Figures 8 and 9]{jakubowski2020topology}.}
    \label{fig:tps-n-correlation-sgns-enwiki}
\end{figure}
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/tps-n-correlation-sgns-semeval_2010_task_14.pdf}
    \caption{$\text{TPS}_n(w)$ of the word embeddings of the SGNS-semeval model plotted against the SemEval gold standard (GS) (a), the number of WordNet synsets (b) and word frequencies (c). The plots are inspired by \cite[Figures 8 and 9]{jakubowski2020topology}.}
    \label{fig:tps-n-correlation-sgns-semeval}
\end{figure}

Following, we compared the results of computing $\text{TPS}_n(w)$ of the word embeddings of the SGNS-enwiki and SGNS-semeval models to the word embeddings of the fastText.TPS.300d, GoogleNews300, glove.840B.300d and fastText.en.300d models. We show the $\text{TPS}_n(w)$ results of using the fastText.TPS.300d model in \cref{table:tps-n-correlation-fasttext-tps-word-embeddings}, and using the GoogleNews300, glove.840B.300d and fastText.en.300d in \cref{table:tps-n-correlation-external-word-embeddings}. We did not compute the correlation between $\text{TPS}_n(w)$ and word frequencies in \cref{table:tps-n-correlation-fasttext-tps-word-embeddings,table:tps-n-correlation-external-word-embeddings}, since we did not have the data available. In addition to this, it is unlikely that $\text{TPS}_n(w)$ and word frequencies have anything in common, as shown in the previous results using the SGNS-enwiki and SGNS-semeval models, as well as by the experiments of \cite{jakubowski2020topology}.

\begin{table}[H]
    \centering
    \begin{tabular}{@{}rrr@{}}
    \toprule
    $n$ & $\text{TPS}_n$ vs. GS & $\text{TPS}_n$ vs. synsets \\
    \midrule
    \trcolor 10 & 0.131	& \textbf{0.135} \\
    40 & 0.395 & 0.066 \\
    \trcolor 50 & \textbf{0.416} & 0.053 \\
    60 & 0.363 & 0.043 \\
    \trcolor 100 & 0.301 & 0.020 \\
    \midrule
    \textit{sample size} & 100 & 62049 \\
    \bottomrule
    \end{tabular}
    \caption{Correlations between $\text{TPS}_n$ and the number of word meanings as perceived by the SemEval gold standard (GS) and the number of WordNet synsets from the fastText.TPS.300d model. \textbf{Bold} values indicate the largest (absolute) correlation.}
    \label{table:tps-n-correlation-fasttext-tps-word-embeddings}
\end{table}
In \cref{table:tps-n-correlation-fasttext-tps-word-embeddings}, we see similar results to the experiments of \cite{jakubowski2020topology}, namely that we get a modest, positive correlation when comparing $\text{TPS}_n(w)$ to the SemEval gold standard, and that we get a decreasing correlation when comparing $\text{TPS}_n(w)$ to the number of WordNet synsets. We note, however, that we did not get exactly the same correlation results as \cite{jakubowski2020topology}, which could be affected by the use of ScaNN, which approximates the nearest neighbours of words.

\begin{table}[H]
    \centering
    \begin{tabular}{ccccccc}
    \toprule
    \multicolumn{1}{c}{\multirow{2}{*}{$n$}} & \multicolumn{2}{c}{GoogleNews300}    & \multicolumn{2}{c}{glove.840B.300d}              & \multicolumn{2}{c}{fastText.en.300d}             \\
    \cmidrule(l){2-7}
    \multicolumn{1}{c}{}                   & \makecell[tc]{$\text{TPS}_n$ vs.\\GS} & \makecell[tc]{$\text{TPS}_n$ vs.\\synsets} & \makecell[tc]{$\text{TPS}_n$ vs.\\GS} & \makecell[tc]{$\text{TPS}_n$ vs.\\synsets} & \makecell[tc]{$\text{TPS}_n$ vs.\\GS} & \makecell[tc]{$\text{TPS}_n$ vs.\\synsets} \\ \midrule
    \trcolor 10           & \textbf{-0.446}  & -0.095       & -0.103  & 0.008        & -0.240  & \textbf{0.114}        \\
    40           & \textbf{-0.446}  & -0.166       & \textbf{-0.125}  & -0.039       & \textbf{-0.289}  & 0.110        \\
    \trcolor 50           & -0.436  & -0.174       & -0.053  & -0.044       & -0.199  & 0.108        \\
    60           & -0.428  & -0.180       & -0.023  & -0.048       & -0.150  & 0.105        \\
    \trcolor 100          & -0.417  & \textbf{-0.193}       & -0.053  & \textbf{-0.058}       & -0.105  & 0.099        \\
    \midrule
    \makecell[tc]{\textit{sample}\\\textit{size}} & 100 & 207119 & 100 & 249352 & 100 & 230175 \\
    \bottomrule
    \end{tabular}
    \caption{Correlations between $\text{TPS}_n$ and the number of word meanings as perceived by the SemEval gold standard (GS), and the number of WordNet synsets from the GoogleNews300, glove.840B.300d and fastText.en.300d models. \textbf{Bold} values indicate the largest (absolute) correlation.}
    \label{table:tps-n-correlation-external-word-embeddings}
\end{table}
Furthermore, in \cref{table:tps-n-correlation-external-word-embeddings} we see that the GoogleNews300 model yield particularly high values when comparing $\text{TPS}_n(w)$ to the SemEval gold standard, while the remaining models are modest at best. We also observe that when comparing $\text{TPS}_n(w)$ to the number of WordNet synsets, we do not get high correlation scores. These results further suggest that by only increasing the vocabulary of the word embedding model, we are not able to model the number of WordNet synsets efficiently, only using the $\text{TPS}_n(w)$ scores. Additionally, the correlation results in \cref{table:tps-n-correlation-external-word-embeddings} indicate that the $\text{TPS}_n(w)$ scores are behaving rather inconsistent across the data sets, and it is not clear if $\text{TPS}_n(w)$ measures polysemy of words.

To compare how well the various word embedding models agree on the $\text{TPS}_n(w)$, we created a correlation matrix comparing the $\text{TPS}_n(w)$ scores and the SemEval gold standard. Using a correlation matrix, we summarize the results nicely and further deepen our understanding of the results. By majority vote, we let $n=40$ when we compared the $\text{TPS}_n(w)$ scores to the SemEval gold standard. We show the correlation matrix in \cref{fig:correlation-matrix-tps-vs-gs}, where we see that the SGNS-enwiki, SGNS-semeval and GoogleNews300 models yield similar $\text{TPS}_{40}(w)$ scores. We also note that the fastText.TPS.300d model either yield no correlation or negative correlations when compared to the other models.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{thesis/figures/correlation-matrix-tps-vs-gs.pdf}
    \caption{Correlation matrix comparing word embedding models on correlations between the $\text{TPS}_{40}(w)$ scores and the SemEval gold standard. High (absolute) values indicate that the two models are similar in terms of scoring using $\text{TPS}_{40}(w)$.}
    \label{fig:correlation-matrix-tps-vs-gs}
\end{figure}

To deepen the understanding, we visualize the similarity of the SGNS-enwiki, SGNS-semeval and GoogleNews300 models in \cref{fig:tps-vs-gs-top-3-correlation-word-embedding-models}, where we can see linear relationships appearing. These results suggest that the SGNS-enwiki, SGNS-semeval and GoogleNews300 models agree on how to score using $\text{TPS}_{40}(w)$.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/tps-vs-gs-top-3-correlation-word-embedding-models.pdf}
    \caption{$\text{TPS}_{40}(w)$ scores plotted against each other using the SGNS-enwiki, SGNS-semeval and GoogleNews300 models.}
    \label{fig:tps-vs-gs-top-3-correlation-word-embedding-models}
\end{figure}

Following, we looked at the three negative correlations which we show in \cref{fig:correlation-matrix-tps-vs-gs} and visualize the negative relationships in \cref{fig:tps-vs-gs-top-3-negative-correlation-word-embedding-models}, where we see negative relationships appearing, although it is less significant than the positive relationships seen in \cref{fig:tps-vs-gs-top-3-correlation-word-embedding-models}.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/tps-vs-gs-top-3-negative-correlation-word-embedding-models.pdf}
    \caption{$\text{TPS}_{40}(w)$ scores plotted against each other using the fastText.TPS.300d, GoogleNews300, fastText.en.300d and SGNS-enwiki models.}
    \label{fig:tps-vs-gs-top-3-negative-correlation-word-embedding-models}
\end{figure}

We have now looked at the effect of computing $\text{TPS}_n(w)$ at varying levels of $n$ using various word embeddings. We saw that, even by decreasing/increasing the vocabulary size of the word embedding models, the $\text{TPS}_n(w)$ score did not improve significantly. In all our experiments, except using the fastText.TPS.300d model, the correlation between $\text{TPS}_n(w)$ and the SemEval gold standard were always negative, while in the experiments of \cite{jakubowski2020topology}, they got a moderate, positive correlation. These results suggest that the topological polysemy scoring could be affected by the choice of word embedding model, i.e. choosing fastText over word2vec, and the fact that the model used in \cite{jakubowski2020topology} was trained on a data set that is strongly related to the 100 polysemous words from the SemEval task. In other words, it could seem that the measure of topological polysemy does not work well for a general word embedding model.

To deepen our understanding of how the $\text{TPS}_n(w)$ score is computed, we will perform an experiment by computing $\text{TPS}_n(w)$ of a custom data set. The custom data set consists of sampled data points of two spheres that share one intersection point. We denote this data set as \textit{2Spheres-$d$}, where $d$ represents the dimensionality of the spheres. In particular, we let $d \in \enclc{2, 3, 4, 5, 10, 20, 50, 300}$. To ensure that the dimensionality of the \textit{2Spheres-$d$} data set was similar to the dimensionality of word embeddings, we let the dimensionality of the space be equal to 300, i.e. \textit{2Spheres-$d$} $\in \R^{300}$. In other words, if $d$ was less than 300, we add zeros to the remaining dimensions to fill up to 300. For each sphere in \textit{2Spheres-$d$}, we generate 1000000 points on the sphere in $\R^d$. We sort the points by distance to the intersection point and split the points into 20 intervals, i.e. chunks of 100000 data points for each sphere. Next, we sample 1000 points from each interval, leading to 20000 points for each sphere. The motivation for sampling from intervals sorted by distance was to reduce the effect of the curse of dimensionality, namely that it becomes harder to measure the distance between points in high (e.g. 300) dimension. For the sake of simplicity, we let $n=50$ when computing the topological polysemy. We illustrate the result of computing $\text{TPS}_{50}$ of 2Spheres-$2$ and 2Spheres-$3$ in \cref{fig:two-spheres-2d-3d-tps-scores}, where we see that, for both 2Spheres-$2$ and 2Spheres-$3$, the $\text{TPS}_{50}$ is at its highest (yellow color) around the intersection point between the two spheres (see \cref{fig:two-spheres-2d-3d-tps-scores} (b) and (d)). In addition to this, at the intersection point between the two spheres, the $\text{TPS}_{50}$ score is low. These two observations suggest that for low values of $d$, the $\text{TPS}_{50}$ score fails to identify the singular point and instead manages to identify the area around it. We will now look at how the $\text{TPS}_{50}$ score behaves for $d \in \enclc{4, 5, 10, 20, 50, 300}$.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{thesis/figures/two-spheres-2d-3d-tps-scores.pdf}
    \caption{Plots of the 2Spheres-$2$ and 2Spheres-$3$ data sets, with $\text{TPS}_{50}$ as labels. The intersection point between the spheres is at $(2, 2)$ in 2 dimensions and at $(2, 2, 2)$ in 3 dimensions.}
    \label{fig:two-spheres-2d-3d-tps-scores}
\end{figure}

We visualize the result of computing $\text{TPS}_{50}$ of 2Spheres-$d$ for $d \in \enclc{4, 5, 10, 20, 50, 300}$ in \cref{fig:two-spheres-distance-to-int-point-vs-tps-scores}, by plotting the distance to the intersection point between the spheres against the $\text{TPS}_{50}$ scores. In \cref{fig:two-spheres-distance-to-int-point-vs-tps-scores}, we see that as the dimension of the spheres increases, the "peak" of $\text{TPS}_{50}$ scores close to the intersection point diminishes. The diminishing effect comes due to the curse of dimensionality, namely that in high dimensional space, all distances become similar, as seen in \cref{fig:two-spheres-distance-to-int-point-vs-tps-scores} (f). In other words, for high dimensional spheres, it becomes difficult to identify the intersection point between the spheres, using the $\text{TPS}_{50}$ scores, as the distances become similar, and $\text{TPS}_{50}$ is unable to identify areas around the intersection point, which we saw happening in lower dimensions in \cref{fig:two-spheres-2d-3d-tps-scores}. We note, however, however, that for high values of $d$, the intersection point has a $\text{TPS}_{50}$ score which generally is higher than all other values of $\text{TPS}_{50}$. Finally, we argue that the results in \cref{fig:two-spheres-distance-to-int-point-vs-tps-scores} indicate that the topological measure of polysemy may suffer when applied to high-dimensional (e.g. 300) data.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/two-spheres-distance-to-int-point-vs-tps-scores.pdf}
    \caption{Distance to the intersection point between spheres plotted against $\text{TPS}_{50}$ scores for 2Spheres-$d$, $d \in \enclc{4, 5, 10, 20, 50, 300}$.}
    \label{fig:two-spheres-distance-to-int-point-vs-tps-scores}
\end{figure}

Following, we repeated the experiment where we computed the topological polysemy of two spheres. In particular, we used a noisy version of the 2Spheres-$d$ data set, which we denoted as the \textit{2SpheresNoisy-$d$} data set. The motivation for adding some noise to the spheres data set was to emulate some real-world effect since real-world data sets are usually not uniformly distributed. In particular, the 2SpheresNoisy-$d$ data set was created by perturbing the 2Spheres-$d$ data set by adding Gaussian noise at every data point. In particular, we used Gaussians with zero mean and a variance of 0.1. We first computed the $\text{TPS}_{50}$ score of the 2SpheresNoisy-$2$ and 2SpheresNoisy-$3$ data sets. We show the results in \cref{fig:two-spheres-noisy-2d-3d-tps-scores}, where we see that in the 2-dimensional case, the $\text{TPS}_{50}$ scores are high (i.e. yellow colour) and we are unable to identify the intersection point between the spheres. Moreover, in \cref{fig:two-spheres-noisy-2d-3d-tps-scores} (c) and (d) we see that the $\text{TPS}_{50}$ scores are mediocre at best, and we are unable to identify the intersection point between the spheres. The results in \cref{fig:two-spheres-noisy-2d-3d-tps-scores} tells us that if we perturb the data set by adding noise, it becomes harder to identify singular points in lower dimensions ($d \in \enclc{2, 3}$), when compared to the $\text{TPS}_{50}$ results using 2Spheres-$d$, which we show in \cref{fig:two-spheres-2d-3d-tps-scores}.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{thesis/figures/two-spheres-noisy-2d-3d-tps-scores.pdf}
    \caption{Plots of the 2SpheresNoisy-$2$ and 2SpheresNoisy-$3$ data sets, with $\text{TPS}_{50}$ as labels. The intersection point between the spheres is at $(2, 2)$ in 2 dimensions and at $(2, 2, 2)$ in 3 dimensions.}
    \label{fig:two-spheres-noisy-2d-3d-tps-scores}
\end{figure}

Finally, we visualize the result of computing $\text{TPS}_{50}$ for 2SpheresNoisy-$d$ for $d \in$ \{{4, 5, 10, 20, 50, 300}\} in \cref{fig:two-spheres-noisy-distance-to-int-point-vs-tps-scores}, by plotting the distance to the intersection point between the spheres against the $\text{TPS}_{50}$ scores. In \cref{fig:two-spheres-noisy-distance-to-int-point-vs-tps-scores}, we see a similar situation appearing to the results using 2Spheres-$d$ in \cref{fig:two-spheres-distance-to-int-point-vs-tps-scores}. In particular, we observe that as we increase the dimensionality of the spheres towards 300, the distances to the intersection point becomes more or less the same, and it is difficult to differentiate between the intersection point and regular points on the two spheres. We note, however, that in \cref{fig:two-spheres-noisy-distance-to-int-point-vs-tps-scores} (f) we see that the $\text{TPS}_{50}$ score is significantly larger than the rest of the $\text{TPS}_{50}$ scores and it could be possible to identify the intersection point using its $\text{TPS}_{50}$ score. Although this result might seem significant in \cref{fig:two-spheres-noisy-distance-to-int-point-vs-tps-scores} (f), the 2SpheresNoisy-$d$ is still rather simple when compared to real-world word embeddings, even when we added the noise. Additionally, the significance we show in \cref{fig:two-spheres-noisy-distance-to-int-point-vs-tps-scores} (f) can be due to a random effect of the 2SpheresNoisy-300 data set.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/two-spheres-noisy-distance-to-int-point-vs-tps-scores.pdf}
    \caption{Distance to the intersection point between spheres plotted against $\text{TPS}_{50}$ scores for 2SpheresNoisy-$d$, $d \in \enclc{4, 5, 10, 20, 50, 300}$.}
    \label{fig:two-spheres-noisy-distance-to-int-point-vs-tps-scores}
\end{figure}

Furthermore, we will use the measure of topological polysemy when we create supervised models for polysemy prediction in \cref{sec:analysis-of-embeddings-supervised-polysemy-prediction}. Next, we will look at Geometric Anomaly Detection, and in particular, how it performs when applied to word embeddings.

\subsection{Geometric Anomaly Detection}
\label{sec:analysis-of-embeddings-geometric-anomaly-detection}
In this subsection, we will apply the Geometric Anomaly Detection (GAD) algorithm (see \cref{sec:geometric-anomaly-detection}) to the word embeddings from the SGNS-enwiki model. In particular, we will show the relationship between how GAD categorizes word embeddings into groups and whether words are polysemous. We implemented GAD as explained in \cref{sec:geometric-anomaly-detection}, using similar packages to the ones we used to implement topological polysemy in \cref{sec:analysis-of-embeddings-topological-polysemy}. In particular, we used the ScaNN \cite{scann2020} approximate nearest neighbour algorithm, to speed up the nearest-neighbour computation, and \path{ripser} \cite{ctralie2018ripser} Python package, to compute Vietoris–Rips complexes. We also included an option to use the Ripser++ \cite{zhang2020ripserplusplus} Python package instead of \path{ripser}, which is a GPU accelerated version. However, we quickly found that the GPU overhead was too big and it was faster just to use the regular \path{ripser} Python package.

Before applying GAD to word embeddings, we will motivate the use of GAD by visualizing GAD applied to the 3-dimensional \textit{Henneberg surface} data set, as used in the experiments of \cite{stolz2020geometric}. To compute the GAD of the Henneberg surface data set, we used the same hyperparameters as in \cite{stolz2020geometric}, i.e. we let the inner annulus radius equal 1.5, outer annulus radius equal 2 and the manifold dimension $k$ equal 2. We visualize the result in \cref{fig:gad-henneberg-3d}, where we see how GAD groups data points to the manifold, boundary and singular groups. In the 3-dimensional Henneberg surface data set, four 2-dimensional surfaces intersect, and as we see in \cref{fig:gad-henneberg-3d} (b), where the GAD algorithm managed to identify the intersecting points as singular points. In addition to the singular points, we see the boundary points quite clearly in \cref{fig:gad-henneberg-3d} (a) and (b).
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/gad-henneberg-3d.pdf}
    \caption{2D and 3D projections of the Henneberg surface data set. The data points are labelled using their groups from the GAD algorithm. This figure is inspired by \cite[Figure 3]{stolz2020geometric}.}
    \label{fig:gad-henneberg-3d}
\end{figure}

Following, we visualize the Henneberg surface data set with the $\text{TPS}_{50}$ score computed for each point. In \cref{fig:gad-henneberg-3d-tps-50}, we see that the $\text{TPS}_{50}$ scores fails to identify the singular data points if the Henneberg surface data set,which we expected to have relatively high $\text{TPS}_{50}$ scores. In particular, the $\text{TPS}_{50}$ scores are relatively high for points on the manifold, and lower for the boundary and singular points.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/gad-henneberg-3d-tps-50.pdf}
    \caption{2D and 3D projections of the Henneberg surface data set. The data points are labelled using their $\text{TPS}_{50}$ scores. This figure is inspired by \cite[Figure 3]{stolz2020geometric}.}
    \label{fig:gad-henneberg-3d-tps-50}
\end{figure}

We will now apply GAD to the word embeddings from the SGNS-enwiki model. In particular, we will apply GAD to the words that have a WordNet entry, similar to how we performed the topological polysemy experiments on SGNS-enwiki word embeddings in \cref{sec:analysis-of-embeddings-topological-polysemy}. Furthermore, the standard GAD algorithm uses annulus radii parameters to do its computation, and because it is unknown which radii parameters to use for a general data set, we will instead default to a $k$-nearest neighbour approach when computing GAD of the WordNet SGNS-enwiki word embeddings. Using the $k$-nearest neighbour approach, we let the inner annulus radius equal the distance to the $s$ nearest neighbour of each word, and similarly for the outer annulus radius, which we set equal to the distance to the $t$ nearest neighbour of each word. To apply GAD to the WordNet SGNS-enwiki word embeddings, we used the parameters $s=25$ and $t=500$. We show the number of words in each GAD group in \cref{table:number-of-words-gad-polysemous-sgns-enwiki-wordnet}, where we see that the number of polysemous WordNet words that fall into the singular group is particularly low, i.e. only 344 of 48880. In addition to this, we see that the number of words falling into the boundary group words is high. These two observations suggest that our inner and outer annulus radius, as well as the manifold dimension $k$, were not set correctly for the data set we are applying GAD to. Keep in mind that the intrinsic dimensionalities of word embeddings are most likely higher than $k=2$, but due to the computational cost of setting $k > 2$ (creation of the Vietoris–Rips complex), we will not set $k$ greater than 2 in this thesis.
\begin{table}[H]
    \centering
    \begin{tabular}{@{}lcccl@{}}
    \toprule
    \multicolumn{1}{c}{}       & \multicolumn{3}{c}{GAD group}  & \multicolumn{1}{l}{} \\ \cmidrule(lr){2-4}
    \multicolumn{1}{c}{}       & Manifold & Boundary & Singular & \textit{Sum}                  \\ \midrule
    \trcolor Number of monosemous words            & 4640     & 86731    & 4161     & 95532                \\
    Number of polysemous words & 634      & 47902    & 344      & 48880                \\ \midrule
    \trcolor \textit{Sum}                        & 5274     & 134633   & 4505     & 144412 \\ \bottomrule
    \end{tabular}
    \caption{Comparison of the number of monosemous and polysemous words that belong to the various GAD groups, when we compute GAD of the WordNet SGNS-enwiki word embeddings.}
    \label{table:number-of-words-gad-polysemous-sgns-enwiki-wordnet}
\end{table}

Finally, we visualize the result using a 2-dimensional UMAP embedding of the 10000 most common words of the WordNet SGNS-enwiki word embeddings in \cref{fig:gad-umap-2d-10k-most-common-wordnet-enwiki-words}, which we labelled using the GAD groups. We created the UMAP embedding using the default hyperparameterization of the \path{umap-learn} Python package. In \cref{fig:gad-umap-2d-10k-most-common-wordnet-enwiki-words}, we see that only a single word is categorized to be singular (the word "branch"), some words are categorized to be on the manifold, and the rest are categorized to be on the boundary. These results indicate that the hyperparameters we used to compute GAD are not suitable for the data set at hand. We also observe that \cref{fig:gad-umap-2d-10k-most-common-wordnet-enwiki-words} differs a whole lot from GAD applied to the Henneberg surface data set (see \cref{fig:gad-henneberg-3d}), which could indicate bad hyperparameterization.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/gad-umap-2d-10k-most-common-wordnet-enwiki-words.pdf}
    \caption{2-dimensional UMAP embedding of the 10000 most common words of the WordNet SGNS-enwiki word embeddings. The points are labelled using their respective GAD groups.}
    \label{fig:gad-umap-2d-10k-most-common-wordnet-enwiki-words}
\end{figure}

Furthermore, we will investigate the effect of using different sets of hyperparameters when applying GAD to word embeddings in \cref{sec:analysis-of-embeddings-supervised-polysemy-prediction}, where we create supervised models for polysemy prediction. Next, we will investigate algorithms for estimating the intrinsic dimensionality of word embeddings, and show how it correlates with the actual number of word meanings.

\subsection{Intrinsic dimension estimation}
\label{sec:analysis-of-embeddings-intrinsic-dimension-estimation}
In this subsection, we will look at intrinsic dimension (ID) estimation algorithms (see \cref{sec:intrinsic-dimension-estimation}) and apply them to word embeddings. In particular, we will apply ID estimation algorithms to the WordNet SGNS-enwiki word embeddings, used in experiments in \cref{sec:analysis-of-embeddings-topological-polysemy} and \cref{sec:analysis-of-embeddings-geometric-anomaly-detection}. We will show the relationship between the estimated ID and the number of WordNet word meanings. To demonstrate the relationship between estimated ID and number of WordNet word meanings, we will use the LPCA (\cref{sec:id-estimation-lpca}), TWO-NN (\cref{sec:id-estimation-twonn}) and TLE (\cref{sec:id-estimation-tle}) algorithms. For each of the ID estimation algorithms, we used the 200 nearest neighbours of each word to estimate their local IDs, using the \path{scikit-dimension} Python package \cite{scikitdimension2020}. Following, we plot the estimated IDs versus the number of WordNet word meanings in \cref{fig:intrinsic-dimension-estimation-vs-wordnet-synsets}, where we observe a similar situation appearing to the ones we see in \cref{fig:tps-n-correlation-sgns-enwiki} and \cref{fig:tps-n-correlation-sgns-semeval}. Particularly, we see a clear trend when plotting the estimated IDs to the number of WordNet word meanings. We also see that the different ID estimation algorithms yield different results: LPCA estimates ID up to 120, while TWO-NN and TLE estimate ID up to 50 and 60. These results suggest that we can not simply rely on a single estimate of the ID, and it could be useful to use multiple ID estimates since they are measured differently (see \cref{sec:intrinsic-dimension-estimation} for more details).
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/intrinsic-dimension-estimation-vs-wordnet-synsets.pdf}
    \caption{Estimated IDs plotted against the number of word meanings, using LPCA, TWO-NN and TLE ID estimation algorithms.}
    \label{fig:intrinsic-dimension-estimation-vs-wordnet-synsets}
\end{figure}

We have now shown the relationship between estimated IDs and the number of word meanings. In the next section, we will create supervised models for estimating the number of word meanings and predicting whether or not a word is polysemous. We will use multiple sets of hyperparameters and all ID estimation algorithms specified in \cref{sec:intrinsic-dimension-estimation}, as well the topological polysemy (\cref{sec:analysis-of-embeddings-topological-polysemy}) and Geometric Anomaly Detection (\cref{sec:analysis-of-embeddings-geometric-anomaly-detection}) algorithms.

\subsection{Supervised polysemy prediction}
\label{sec:analysis-of-embeddings-supervised-polysemy-prediction}
In this subsection, we will propose two supervised models to predict the number of word meanings. As we have seen in the previous subsections (\cref{sec:analysis-of-embeddings-topological-polysemy}, \cref{sec:analysis-of-embeddings-geometric-anomaly-detection} and \cref{sec:analysis-of-embeddings-intrinsic-dimension-estimation}), the number of word meanings seem to be more or less correlated with topological polysemy, Geometric Anomaly Detection (GAD) and intrinsic dimension (ID) estimation. For this reason, we will propose two supervised model using lasso regression (\cref{sec:lasso-regression}) and logistic regression (\cref{sec:logistic-regression}), incorporating the results from topological polysemy, GAD and ID estimation. The goal of these models is to use the results from the topological polysemy, GAD and ID estimation methods to improve on the prediction of the number of word meanings. We chose to use lasso regression because it has feature importance packed in the model. The feature importance part is important for us because we would like to try multiple configurations of hyperparameters for each algorithm used to create the training data. The logistic regression model is trained using $\ell_1$-penalty, allowing the model to perform feature importance. The lasso regression model tries to predict the number of word meanings, while the logistic regression model performs binary prediction of whether or not a word is polysemous. We also attempted to create a multi-class (e.g. one meaning, two meanings, etc.) model using multinomial logistic regression, but it became apparent that the problem was too hard and we decided not to follow up with those experiments. Furthermore, we denote the lasso regression model as \textit{WME-enwiki} (short for \textbf{W}ord \textbf{M}eaning \textbf{E}stimation-enwiki) and the logistic regression model as \textit{BWME-enwiki} (short for \textbf{B}inary \textbf{W}ord \textbf{M}eaning \textbf{E}stimation-enwiki). Next, we will describe the creation of training data used for both supervised models, before going into detail about the training and evaluation process.

To create the training data used in the WME- and BWME-enwiki models, we used the word embeddings from the SGNS-enwiki model. In particular, we used the word embeddings that have a WordNet entry, resulting in 144412 words. We denote these word embeddings as the WordNet SGNS-enwiki word embeddings. The number of word meanings (i.e. the number of WordNet synsets) is used as labels $y$ for the WME-enwiki model. For the BWME-enwiki model, we used binary labels, i.e. $y=0$ if the word had exactly one word meaning, and $y=1$ if the word had two or more meanings.

To create the features of the training data, we first computed topological polysemy $\text{TPS}_n(w)$ of the WordNet SGNS-enwiki word embeddings. We computed $\text{TPS}_n(w)$ at for varying $n=10, 20, 30, \ldots, 250$ (step size of 10, leading to 25 values of $n$) and used them as features in the data. In addition to this, we computed the maximum, average and standard deviation of the birth values of the zero-degree persistence diagram computed by $\text{TPS}_n(w)$, leading to 3 additional features for each $\text{TPS}_n(w)$. In total, this resulted in 25 (values of $n$) $\times$ 4 = 100 features from topological polysemy.

Following, we applied GAD to the WordNet SGNS-enwiki word embeddings. To compute GAD, we used the $k$-nearest neighbour version, similar to the experiments of \cref{sec:analysis-of-embeddings-geometric-anomaly-detection}; we let the inner annulus radius equal the distance to the $s$-nearest neighbour and the outer annulus radius equal the distance to the $t$-nearest neighbour. Since we used the $k$-nearest neighbour version of GAD we were more in control of the computation time, because setting the radius manually can lead to large and difficult computations of the Vietoris–Rips complex, as some areas are denser than others. We show the different choices of $s$ and $t$ in \cref{table:supervised-polysemy-prediction-gad-configurations}, which leads to 23 different configurations of the inner and outer annulus $k$-nearest neighbours. We let the manifold dimension $k$ equal 2 for all words, even though the local intrinsic dimension for each word is likely higher than 2. This was done to make the GAD computation feasible within the computational resources at hand; we will revisit the manifold dimension choice when discussing future work in \cref{chap:future-work}. For each of the $(s, t)$ configurations used to parameterize GAD, we created one feature for each GAD group (i.e. manifold, boundary and singular) as 3-dimensional one-hot encodings. For example, if a word is categorized as being on the manifold, then its value is equal to 1 and the rest are set to zero. In other words, we are left with 23 (configurations) $\times$ 3 (GAD groups) = 69 features from GAD. We also attempted to vectorize the persistence diagrams created by GAD using persistence images (\cref{sec:persistence-image}), but it quickly led to far too many features as we used each pixel in the images as a separate feature, and we were unable to train the WME- and BWME-enwiki models efficiently. We will revisit the use of persistence images when discussing future work in \cref{chap:future-work}.
\begin{table}[H]
    \centering
    \begin{tabular}{@{}cc@{}}
    \toprule
    \multicolumn{1}{l}{Inner annulus, $s$-nearest neighbour} & \multicolumn{1}{l}{Outer annulus, $t$-nearest neighbour} \\
    \midrule
    \trcolor 25 & 250 \\
    25 & 500 \\
    \trcolor 25 & 750 \\
    25 & 1000 \\
    \midrule
    \trcolor 50 & 250 \\
    50 & 500 \\
    \trcolor 50 & 750 \\
    50 & 1000 \\
    \midrule
    \trcolor 100 & 1000 \\
    100 & 1250 \\
    \trcolor 100 & 1500 \\
    100 & 1750 \\
    \trcolor 100 & 2000 \\
    \midrule
    150 & 1000 \\
    \trcolor 150 & 1250 \\
    150 & 1500 \\
    \trcolor 150 & 1750 \\
    150 & 2000 \\
    \midrule
    \trcolor 200 & 1000 \\
    200 & 1250 \\
    \trcolor 200 & 1500 \\
    200 & 1750 \\
    \trcolor 200 & 2000 \\
    \bottomrule
    \end{tabular}
    \caption{Our configurations of $s$, i.e. inner annulus nearest neighbour, and $t$, i.e. outer annulus nearest neighbour, for computing GAD of the WordNet SGNS-enwiki word embeddings.}
    \label{table:supervised-polysemy-prediction-gad-configurations}
\end{table}

Furthermore, we estimated the local ID of the WordNet SGNS-enwiki word embeddings using the ID estimation algorithms from \cref{sec:intrinsic-dimension-estimation}. More precisely, we used the LPCA (\cref{sec:id-estimation-lpca}), KNN (\cref{sec:id-estimation-knn}), TWO-NN (\cref{sec:id-estimation-twonn}), MLE (\cref{sec:id-estimation-mle}) and TLE (\cref{sec:id-estimation-tle}) algorithms. For each of the ID estimation algorithms, we used the $k$-nearest neighbours of each word to estimate their local IDs. We used the following values for $k$: 25, 50, 100, 150 and 200. The estimated local ID of each word is used as a feature in the training data, leading to 5 (algorithms) $\times$ 5 (values for $k$) = 25 features from ID estimation. We used the \path{scikit-dimension} Python package \cite{scikitdimension2020} to estimate the local IDs.

In total, the training data had 100 (from topological polysemy) + 69 (from GAD) + 25 (from ID estimation) = 194 features. Following, we split the training data into three new distinct data sets (as motivated by \cref{sec:train-val-test-splits}): training, test and SemEval test data sets. The new training data set consisted of 95\% random words of the original training data set, excluding the 100 SemEval-2010 Task 14 target words (as used in \cref{sec:analysis-of-embeddings-topological-polysemy}). The test data set consisted of 5\% random words of the original training data set, excluding the 100 SemEval-2010 Task 14 target words. We will use the test data to evaluate the performance of the trained WME- and BWME-enwiki models. Furthermore, the SemEval test data set consisted of the 100 SemEval-2010 Task 14 target words and will be used to evaluate the performance using the WME-enwiki model. We emphasize that the training, test and SemEval test data sets do not have overlapping words, as we do not want to be training on words from the test data sets. The training data set consisted of 137098 words, the test data set consisted of 7216 words, and the SemEval test data set consisted of 98 words (as 2 of the words were out of the SGNS-enwiki vocabulary). For each data set, we transformed the features by removing the mean and scaling to unit variance, as we did not want the WME- and BWME-enwiki models to be affected by different means and variances across the features. For the SemEval test data set, we used the SemEval gold standard as the number of word meanings, while for the training and test data set we used the number of WordNet synsets as the number of word meanings.

Following, we trained the WME- and BWME-enwiki models using $k$-fold cross-validation (\cref{sec:cross-validation}). We found $k=20$ to work well with our data, meaning that we used 6855 random words for each fold in the cross-validation. For the WME-enwiki model, we cross-validated over 10000 values of $\lambda$, starting from $\lambda=0.0000001$ to $\lambda=0.01$. We found the most optimal value of $\lambda$ for the WME-enwiki model to be 0.0000291. For the BWME-enwiki model, we cross-validated over 10000 values of $\lambda$, starting from $\lambda=0.00001$ to $\lambda=0.01$. We found the most optimal value of $\lambda$ for the BWME-enwiki model to be 0.000692. To perform the cross-validation we used the \path{LassoCV} and \path{LogisticRegressionCV} classes from \path{scikit-learn} for the WME- and BWME-enwiki models, respectively. For the WME-enwiki model, we used the default scoring of the \path{LassoCV} class. On the other hand, for the BWME-enwiki model, our goal was to maximize the ability of the model to predict polysemous words accurately. As such, we used the sensitivity metric (\cref{sec:sensitivity}) to score the folds from the BWME-enwiki cross-validation. Furthermore, we show the results from training the WME-enwiki model in \cref{fig:wme-enwiki-correlation-result}, where we see a weak correlation between the predicted number of word meanings and the number of WordNet synsets for both the training and test data sets. In \cref{fig:wme-enwiki-correlation-result} (c), we see that the model is unable to predict the number of word meanings for the SemEval data set, which is not surprising, as we have trained using the number of WordNet synsets. We note, however, that we see clear trends in all plots of \cref{fig:wme-enwiki-correlation-result}.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/wme-enwiki-correlation-result.pdf}
    \caption{The predicted number of word meanings plotted against the number of WordNet synsets and SemEval gold standard, using the WME-enwiki model.}
    \label{fig:wme-enwiki-correlation-result}
\end{figure}

By looking at the values of the feature coefficients of the WME-enwiki model, we saw how the model prioritized certain features over others. We show the top 10 most important features in \cref{fig:wme-enwiki-feature-importances}, where we observe that the features from topological polysemy, for large values of $n$, are the most relevant for the model. The MLE and TLE intrinsic dimension estimators are also relatively relevant for high values of the $k$-nearest neighbour. The features from GAD are not in the top 10 most important features.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/wme-enwiki-top-10-feature-importances.pdf}
    \caption{Feature importances (i.e. coefficients) for the top 10 most important features of the WME-enwiki model. We sort the feature importances by their absolute values in descending order.}
    \label{fig:wme-enwiki-feature-importances}
\end{figure}

To investigate the feature importances for the TPS, GAD and ID estimator features, we visualize the top 10 most important features in \cref{fig:wme-enwiki-feature-importances-tps-gad-estimated-ids}, where we see that the $\text{TPS}_{250}$ features are especially relevant. From the GAD features in \cref{fig:wme-enwiki-feature-importances-tps-gad-estimated-ids} (b), we see that whether or not a word is classified as boundary or singular is important, while being classified as a manifold is not as relevant. Finally, we see that the MLE and TLE ID estimator methods yield important features for various values of $n$. We note, however, that the features importances in \cref{fig:wme-enwiki-feature-importances-tps-gad-estimated-ids} (a) and (c) are more important than the feature importances in \cref{fig:wme-enwiki-feature-importances-tps-gad-estimated-ids} (b), as noted by the x-axis scales.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/wme-enwiki-top-10-feature-importances-tps-gad-estimated-ids.pdf}
    \caption{Feature importances (i.e. coefficients) of the top 10 TPS, GAD and ID estimator features, using the WME-enwiki model. We sort the feature importances by their absolute values in descending order.}
    \label{fig:wme-enwiki-feature-importances-tps-gad-estimated-ids}
\end{figure}

From the training of the WME-enwiki model, the lasso set some of the features to zero, essentially removing them from the model. In particular, 48 of 194 features were set to zero, and most of them were various configurations of GAD which did not yield any interesting result (e.g. all words classified as boundary words). We have now looked at the results from training the WME-enwiki model, and in particular, looked at its performance for predicting the number of word meanings and which features were important to the model.

Next, we will look at the results from the training of the BWME-enwiki model. We show the results from the training of the BWME-enwiki model in \cref{fig:bwme-enwiki-confusion-matrices}, where we see the result of predicting the number of word meanings on the training and test data sets using confusion matrices (\cref{sec:confusion-matrix}). As we show in \cref{fig:bwme-enwiki-confusion-matrices}, we get a sensitivity of 0.393 on the train data sets, meaning that the model identifies 39.3\% of the polysemous words. The test sensitivity shows that the model identifies 39.4\% of all the unseen polysemous words. These results indicate that the model can not efficiently predict whether or not a word is polysemous, as we ideally would like the sensitivity on both the training and test sets to be at least 0.5 (or 50\%).
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/bwme-enwiki-confusion-matrices.pdf}
    \caption{Confusion matrices for predicting the number of word meanings, using the BWME-enwiki model. The first confusion matrix (a) shows the result on the training data set, while the second confusion matrix (b) shows the result on the test data set.}
    \label{fig:bwme-enwiki-confusion-matrices}
\end{figure}

To deepen our understanding of which words the BWME-enwiki model had a harder time with, we looked at the misclassified monosemous and polysemous test words. Of the 1484 words the BWME-enwiki model incorrectly predicted to be monosemous, we report the top 10 most common misclassified test words, namely the following words: "time", "age", "returned", "Italian", "Chicago", "gold", "tower", "jones", "unable" and "opposition". Furthermore, of the 553 words the BWME-enwiki model incorrectly predicted to be polysemous, we report the top 10 most common misclassified test words, namely the following words: "january", "ninety-six", "ninety-one", "seventy-one", "sixty-three", "fifty-four", "fifty-eight", "non", "citizens" and "additionally". From these sets of words, we do not see any particular pattern. Furthermore, we visualize the correctly and incorrectly classified words from the test data set in \cref{fig:bwme-enwiki-umap-classified-words}, using a 2-dimensional UMAP embedding. In \cref{fig:bwme-enwiki-umap-classified-words} (b) and (c), we emphasize the misclassified words, and we do not see any particular pattern here either, as the words are spread throughout the UMAP embedding.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/bwme-enwiki-umap-classified-words.pdf}
    \caption{2-dimensional UMAP embedding of the test data set evaluated on the BWME-enwiki model. We emphasize the correctly and incorrectly classified words in a "correlation matrix" fashion.}
    \label{fig:bwme-enwiki-umap-classified-words}
\end{figure}

Next, we will investigate the feature importance in the BWME-enwiki model by looking at the coefficient values of the features. We show the top 10 most important features of the BWME-enwiki model in \cref{fig:bwme-enwiki-feature-importances}, where we see a similar pattern to the top 10 feature importances of the WME-enwiki model, i.e. that the TPS features (for varying $n$) are most important, followed by the features from the ID estimator models.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/bwme-enwiki-top-10-feature-importances.pdf}
    \caption{Feature importances (i.e. coefficients) for the top 10 most important features of the BWME-enwiki model. We sort the feature importances by their absolute values in descending order.}
    \label{fig:bwme-enwiki-feature-importances}
\end{figure}

Furthermore, we show the top 10 feature importances for the TPS, GAD and ID estimator features separately in \cref{fig:bwme-enwiki-feature-importances-tps-gad-estimated-ids}, where we see that the TPS features with high values of $n$ are generally more important than the ones with low values of $n$. From the GAD features in \cref{fig:bwme-enwiki-feature-importances-tps-gad-estimated-ids} (b), we see a different situation to the top 10 features importances for GAD using the WME-enwiki model. In particular, we see that whether or not a point is categorized as singular is important for predicting whether or not a word is polysemous, and the rest of the GAD categories are less relevant. One interesting finding is that the GAD singular features importances were negative; we expected them to be positive, as it would make sense for them to be a positive contribution to whether or not a word is polysemous. Finally, we see that the TLE and TLE ID estimator models yield important features for high neighbourhood values. Similar to the feature importances in \cref{fig:wme-enwiki-feature-importances-tps-gad-estimated-ids}, we note the fact that the feature importances in \cref{fig:bwme-enwiki-feature-importances-tps-gad-estimated-ids} (a) and (c) are more important than the feature importances in \cref{fig:bwme-enwiki-feature-importances-tps-gad-estimated-ids} (b), as noted by the x-axis scales.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/bwme-enwiki-top-10-feature-importances-tps-gad-estimated-ids.pdf}
    \caption{Feature importances (i.e. coefficients) of the top 10 TPS, GAD and ID estimator features, using the BWME-enwiki model. We sort the feature importances by their absolute values in descending order.}
    \label{fig:bwme-enwiki-feature-importances-tps-gad-estimated-ids}
\end{figure}

We have now explained how we trained and evaluated two supervised models for predicting the number of word meanings and whether or not a word is polysemous. Next, we will summarize and conclude the thesis and discuss ideas for future work.


\subsection{Euro Converter}
{\centering\itshape
  [Warning: Image ignored] % Unhandled or unsupported graphics:
%\includegraphics[width=4.671cm,height=2cm]{images/rockbox-manual-img64.png}
 \newline
Euro converter
\par}

This plugin converts euros back into pre{}-euro currency.  The country for which is does this is selectable by pressing the MENU key.  The MINUS and
PLUS keys move the cursor between the digits and the PLAY and STOP keys
increase and decrease the current digit.  The amount in the old
currency is displayed on the second line.