\documentclass{article}

% Packages
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{ismir,cite,url}
\usepackage{graphicx}
\usepackage{color}
\usepackage[british]{babel}
\usepackage{csquotes}
\usepackage{microtype}
\usepackage{balance}
\usepackage{enumitem}
\usepackage{amsmath,amsthm,amssymb}
\usepackage{newtxmath}
\usepackage{nicefrac}
\usepackage{upgreek}
\usepackage{graphicx}

% Custom commands
\newcommand{\vect}[1]{\mathrm{\mathbf{#1}}}
\newcommand{\R}{\mathbb R}
\newcommand{\E}{\mathbb E}
\newcommand{\vx}{\vect x}
\newcommand{\vr}{\vect r}
\newcommand{\vu}{\vect u}
\newcommand{\vS}{\vect S}
\newcommand{\vX}{\vect X}
\newcommand{\vC}{\vect C}
\newcommand{\vR}{\vect R}
\newcommand{\vW}{\vect W}
\newcommand{\vomega}{\boldsymbol{\upomega}}
\newcommand{\Real}{\text{Re}}
\newcommand{\hvx}{\hat \vx}
\newcommand{\hvX}{\hat \vX}
\DeclareMathOperator{\proj}{proj}
\DeclareMathOperator{\Cov}{Cov}
\newcommand{\subfiglabel}[1]{\textbf{\textsc{#1}}}

% Metadata
\title{Cosine Contours:\\ A  Multipurpose Representation for Melodies}
\oneauthor%
  {Bas Cornelissen \hfil Willem Zuidema \hfil John Ashley Burgoyne}
  {Institute for Logic, Language and Computation, University of Amsterdam
  \\{\url{b.j.m.cornelissen@uva.nl}, \url{zuidema@uva.nl}, \url{j.a.burgoyne@uva.nl}}
}
\def\authorname{B.~Cornelissen, W.~Zuidema, and J.A.~Burgoyne}

% Some final
\usepackage[bookmarks=false, pdfauthor={\authorname}, pdfsubject={\papersubject}, hidelinks]{hyperref}
\sloppy
\begin{document}

%=========
\maketitle
%=========

\begin{abstract}
Melodic contour is central to our ability to perceive and produce music.
We propose to represent melodic contours as a combination of cosine functions, using the discrete cosine transform.
The motivation for this approach is twofold: (1) it approximates a maximally informative contour representation (capturing most of the variation in as few dimensions as possible), but (2) it is nevertheless independent of the specifics of the data sets for which it is used.
We consider the relation with principal component analysis, which only meets the first of these requirements.
Theoretically, the principal components of a repertoire of random walks are known to be cosines.
We find, empirically, that the principal components of melodies also closely approximate cosines in multiple musical traditions.
We demonstrate the usefulness of the proposed representation by analyzing contours at three levels (complete songs, melodic phrases and melodic motifs) across multiple traditions in three small case studies.
\end{abstract}

%—————————————————————
\section{Introduction}
%—————————————————————

Humans are born with a remarkable sensitivity to melodic contour.
This is dramatically illustrated when newborns cry: the cries of German babies tend to go down in pitch, but those of French babies go up, even if falling contours are physiologically easier to produce \cite{Mampe2009}.
By imitating the intonation patterns of their mothers' language, babies take the first steps towards a spoken language---helped by exaggerated pitch contours of infant directed speech \cite{Wermke2021}.
Contour perception remains central to speech, for intonation or even word distinctions, but is also a key ingredient of human musicality \cite{Honing2015}.
Dowling famously argued that melodies are remembered as two independent parts, a scale and a contour \cite{Dowling1978}.
A scale then functions as a ladder ``on which the ups and downs of the contour where hung.''
Indeed, when listening to novel melodies, contours appear to stand out more than the exact intervals and influence the perceived similarity of melodies \cite{Schmuckler2016}.
That has also motivated studies of contour in \textsc{mir}, in particular for measuring melodic similarity \cite{Mullensiefen2004b}.
As we briefly review below, many representations of contour have been proposed in answer to the recurring question: how can one best describe melodic contour?

We propose representing melodies as combinations of cosine functions.
This is motivated by the need for a concise, maximally informative representation: how can we capture as much of the variability in contour data in as few dimensions as possible?
The easiest solution would be to use a \emph{principal component analysis} (\textsc{pca}).
In section~\ref{sec:pc}, we show empirically that the principal components of melodies do not take arbitrary shapes, but in fact closely approximate cosines.
We then relate this observation to theoretical results showing that the principal components of certain random walks are sinusoidal, as a result of a particular covariance structure.
The proposed `cosine contour' space thus closely approximates the optimal solution provided by \textsc{pca}, but offers several benefits.
The key argument for this representation is theoretical and
we leave a systematic comparison of contour representations for future work.
Instead we discuss three case studies that demonstrate the usefulness of cosine contours.

Cosine contours meet several desiderata for contour representations.
First, a good representation respects the linear structure of melody and is \emph{invariant to transposition and tempo changes}.
Second, the representation should be \emph{interpretable} and \emph{intuitive} (and,
in particular, avoid some of the shortcomings of polynomial coefficients).
Third, the representation should support \emph{variable levels of abstraction}, so that one can interpolate between a broad summary of the shape, and the exact pitch curve.
Fourth, we look for a \emph{broadly applicable} and \emph{culturally neutral} representation: it should be able to describe contours from different cultures, or even from different domains (e.g.,~speech). It should also be able to handle both audio and symbolic data, although we only analyze symbolic data here.

\begin{figure}[t]
    \centering
    \includegraphics{figs/fig01-cosine-contours.pdf}
    \caption{
        \textbf{Cosine contours} represent a melodic contour as a combination of cosine functions.
        %
        \subfiglabel{(a)}~%
        This is illustrated for a short melodic phrase.
        %
        \subfiglabel{(b)}~%
        A piano roll is interpolated to obtain fixed-length vector of \textsc{midi} pitches (black curve).
        This vector is approximated using a discrete cosine transform (coloured curves).
        Increasing the dimensionality, from, e.g.,~1 (blue) to 3 (green) improves the approximation.
        %
        \subfiglabel{(c)}~%
        The basis functions correspond to simple shapes.
        %
        % \subfiglabel{(d)}~%
        This makes the cosine contour space interpretable, as illustrated in \subfiglabel{(d)} for the first two dimensions.
        Every point in this space defines a contour shape, varying in what we call the \emph{descendingness} and \emph{archedness}.
        The orange dot represents the orange contour from \subfiglabel{(b)}.
    }
    \label{fig:representation}
\end{figure}

%—————————————————————————————————
\section{What is melodic contour?}
%—————————————————————————————————

Melodic contour is a general description of a melody's shape that abstracts away from the particular pitches and precise rhythms.
It has been characterised in many different ways.
Ethnomusicologists (and composers) have used \emph{contour typologies}: small sets of contour types \cite{Adams1976}.
David Huron, for example, distinguished nine types of contours by comparing the initial and final pitches to the average pitch on the middle part of a melody \cite{Huron1996}.
When, say, the initial is above the middle, which in turn equals the final, the melody has a `descending-horizontal' contour.
Such a formal typology can be used in \textsc{mir} \cite{Mullensiefen2009}, but typologies have also been defined using verbal descriptions or even drawings \cite{Adams1976,Kelkar2018}.
CantoCore, for example, instructs an annotator to look for six types: ascending, descending, arched, U-shaped, undulating and horizontal \cite{Savage2012}.
Even though the types are less sharply defined, such typologies have inspired cross-cultural generalizations such as the \emph{melodic arch hypothesis}: the claim that melodic phrases tend to be arch-shaped or descending \cite{Huron1996,Savage2015,Tierney2011,Savage2017a}.

In melody extraction from audio, contours are usually represented by sequences of pitches ordered in time.
Various contour features derived from this, such as the range or pitch deviation, have been used in classification tasks \cite{Bittner2017,Panteli2017a,Bittner2015,Salamon2012}.
Contours in symbolic data can be similarly represented as \emph{step curves} (figure \ref{fig:representation}B, black line) \cite{Steinbeck1982,Mullensiefen2012}.
\emph{Parsons code} drastically simplifies a step curve \cite{Parsons1975}.
It describes the direction of movement from one note to the next (up, down, or level) and discards interval size and note durations.
Variants between these two extremes have also been used, by distinguishing various classes of jump sizes \cite{Mullensiefen2004b}.
Another strategy is to focus on salient notes, typically turning points (maxima and minima), and to discard other notes \cite{Adams1976,Steinbeck1982,Salamon2012}.
This often requires special handling of ornaments \cite{Mullensiefen2012}, possibly tailored to the repertoire.
Yet another approach considers the relative ordering of all pairs of notes in a melody, summarized in a matrix.
Such combinatorial models in way expand rather than reduce the representation, break the linearity of the melody and are sensitive to local changes \cite{Mullensiefen2012}.

Finally, one can describe melodies using continuous functions.
Müllensiefen and Wiggins fit a polynomial function to a step curve and use the coefficients to represent the contour \cite{Mullensiefen2012}.
The degree of the polynomial is chosen per phrase, using the Bayesian information criterion (\textsc{bic}) to avoid overfitting.
Polynomial coefficients are quite difficult to interpret, however: they change drastically when the degree changes, and can also be sensitive to changes in the data, especially when the polynomials are not orthogonal and introduce correlations between the coefficients (collinearity).
Instead of fitting a function to the contour, one can also \emph{decompose} the contour and express it as a sum of (orthogonal) basis functions.
Velarde and colleagues have for example used \emph{Haar wavelets} as basis functions in musical pattern discovery \cite{Velarde2016}.
The step-like shapes of those wavelets are well suited to describe particular melodic patterns, but make them less suited for describing the overall contour.
An alternative basis of sinusoidal functions is implicit in Schmuckler's use of a Fourier analyses to represent melodic contour \cite{Schmuckler1999}.
This has been interpreted as measuring the `periodic information' in a melody, and was reported to correlate with perceived similarity.

%—————————————
\section{Data}
%—————————————

With the broad applicability in mind, we analyze music from several independent traditions.
The choice of traditions was partly motivated by our aim to analyze contours at multiple levels of description: we expect (different) regularities at different levels.
At the highest level, complete \emph{songs} can have characteristic shapes, and those shapes may differ between traditions.
At the smaller level \emph{phrases} may be subject to the melodic arch hypothesis cited above.
Finally, at the smallest level, \emph{melodic motifs} could exhibit sequential structure, for example when melodies in a repertoire are formed by stringing together melodic motifs (sometimes called \emph{centonization} \cite{Nuttall2019}).
We also analyze \emph{random segments} obtained by slicing a melody at random in approximately phrase-length segments, so that their boundaries usually do not overlap with actual phrase boundaries \cite{Cornelissen2020DLfM}.

One tradition for which all of these levels are directly available is Gregorian chant, thanks to two recently released corpora: the CantusCorpus and the GregoBaseCorpus \cite{Cornelissen2020DLfM}.
Gregorian chant has been sung in Roman Catholic churches for well over a thousand years.
The close connection between music and text in chant suggests a natural subdivision of the music into motifs corresponding to words or syllables.
The notation suggests even smaller motifs: it is based on small figures, called \emph{neumes}, that represent short groups of notes \cite{Kelly2018}.
To analyse motif contours, we use chants from the CantusCorpus (v0.2) with transcriptions of medieval manuscripts, which include neume boundaries.
We focus on the two largest chant genres: \emph{antiphons} and \emph{responsories}.
Phrase boundaries are not available in the CantusCorpus, however, and so for that, we turn to the GregoBaseCorpus (v0.3) of modern chant transcriptions.
Modern chant notation includes explicit breathing marks (\emph{pausas}), which have been used to extract phrases \cite{Cornelissen2020DLfM}.

Phrase markings are also included in the Essen Folksong Collection \cite{Schaffrath1995}, from which we analyse phrases from German and Chinese folksongs.
We focus on the two largest subsets, `Erk' \cite{Erk1893} (9782 contours) and `Han' (7601 contours).\footnote{%
    %———
    Much is unclear about the exact (bibliographic) origins of the Chinese subset of \emph{Essen}.
    This is problematic given its wide use in computational musicology and deserves further attention from the community.
    }%———
At the level of complete songs, we also add music from the Sioux people made available in the \emph{Densmore Collection} \cite{Densmore1918,Shanahan2014}.
In the supplementary material, we include some further analyses of several other traditions from the \emph{Essen} and \emph{Densmore} collections.

We convert all melodies (be it songs, phrases or motifs) to step contours by extracting note onsets (in quarter notes) and pitches (in \textsc{midi} semitones).
We then interpolate a step function through these points, from which we sample $N=100$ equally spaced pitches.
Those pitches are collected in vectors $\vect x = (x_0, \dots, x_{N-1})$ (black curve in figure \ref{fig:representation}\textsc{a}), which are the basic data analysed in this paper.\footnote{%
    %——
    See \href{https://github.com/bacor/cosine-contours}{github.com/bacor/cosine-contours}
    for data, code and supplements.
    }%——

\begin{figure}[t]
    \centering
    \includegraphics{figs/fig02-pca.pdf}
    \caption{%
        \textbf{Principal components of contours}~%
        %———————————————————————————————————————
        (solid lines) are roughly cosine shaped (dashed) across different levels \textsc{\textbf{(a)}}.
        This is a result of the particular structure of the covariance matrix \textsc{\textbf{(b)}}: matrices of this type have Fourier basis functions as their eigenvectors.
        This is clearest for phrases \textsc{\textbf{(2)}} or random segments from melodies \textsc{\textbf{(3)}}, here of similar length as phrases.
        Crucially, we see the same effect for simulated, contour-like random walks \textsc{\textbf{(4)}}.
        For complete songs \subfiglabel{(5)} the effect is less clear, probably due to differences in typical length \textsc{\textbf{(c)}} and data size.
        Contours in \subfiglabel{1--4} are from Gregorian chant.
    }
    \label{fig:pca}
\end{figure}

Our starting representation makes several assumptions that seem reasonable (and common: \cite{Savage2017a,Tierney2011,Velarde2016}) when only interested in contour.
First, we ignored all rests.
Second, we normalize the duration of all contours.
Both 3-note motifs and 30-note songs are represented by vectors of 100 pitches.
The relative durations within that melody are of course retained, so we would still see that contours of short motives are probably simpler than those of long melodies.
Third, we assume Euclidean distances between melodies.
This is usually problematic, but less so when we are only interested in contour similarity.
Our analyses require that all contours are embedded in a vector space.
Using more sophisticated measures such as dynamic time warping distance, would require us to reconstruct a space (e.g., using multidimensional scaling), and make the analyses less transparent.
Finally, note that we do \emph{not} center the contours to have mean pitch 0.
This is sometimes done to make contours transposition invariant and more directly comparable \cite{Savage2017a,Velarde2016,Cornelissen2020DLfM}.
We will soon see that our proposed representation elegantly resolves this problem without requiring centring.

%———————————————————————————————————————————————————————
\section{Principal components of contours}\label{sec:pc}
%———————————————————————————————————————————————————————

In this section, we explore principal component analysis applied to contours.
The goal of \textsc{pca} is to find a set of orthogonal axes, the \emph{principal components}, that contain most of the variance in the dataset. Note that the principal components, like the original contours from our data, are $N$-dimensional vectors, such that the contours and components can be interpreted and plotted in the same space.

In figure~\ref{fig:pca}\textsc{a}~, we show results from applying \textsc{pca} on a large dataset of Gregorian chant (similar results with German and Chinese folksongs can be found in supplement S2). We plot the first four principal components of several types of melodies: short motifs (syllables), phrases, random segments of melodies, and complete songs.
We show responsory syllables from CantusCorpus for the motifs,
antiphon phrases from the GregoBaseCorpus
and finally all song contours from GregoBaseCorpus.

Surprisingly, we find that the principal components are highly similar across most of those data sets, and correspond to well-known contour shapes: descending, convex, and—perhaps—undulating.
This is clearest for the phrases and random segments.
For complete songs the effect is weaker, especially for even smaller datasets (see the supplement S2).
Besides small data sizes, the fact that songs are
much longer also plays a role (see fig.~\ref{fig:pca}\textsc{c}).
We also applied the analysis on simulated random walks approximating phrases: we draw the number of notes from a similar length distribution, normalize the duration and then sample $N=100$ pitches as before (see supplement S1 for details).
Interestingly, the pattern is now even clearer, suggesting there must be a mathematical explanation.

To give that explanation, we need to first describe \textsc{pca} more formally. We consider a collection of $M$ contour vectors $\vx_m$ of length $N$.
Denote the sample mean by $\bar \vx = \frac{1}{M} \sum_{m} \vx_m$ and the centered data by $\hat \vx_m = \vx_m - \bar \vx$.
The first principal component of the dataset is then defined as a normalized vector $\vu_1 \in \R^D$ for which the projected data $\{\vu_1^T \vx_m: 1 \le m \le M \}$ has maximal variance.
It can be shown (e.g.,~\cite{Jolliffe2002}) that this is the case when $\vu_1$ is an eigenvector corresponding to the largest eigenvalue $\lambda_1$ of the covariance matrix
%———
\begin{align}
    \label{eq:covariance}
    \vS = \frac{1}{M} \sum_{m=1}^M (\vx_m- \bar\vx)(\vx_m - \bar\vx)^T,
\end{align}
%———
so that $\vS \vu_1 = \lambda_1 \vu_1$.
It follows that the projected variance is given by $\lambda_1$, the largest eigenvalue.
The other principal components similarly emerge as the other eigenvectors of the covariance matrix.

The covariance matrices (figure \ref{fig:pca}\textsc{b})
for both random walks and our empirical data have a particular structure: they \emph{roughly} resemble \emph{Toeplitz matrices}, which have fixed values along each of their diagonals.
Such covariance structures are frequently encountered in spatial or temporal data, when the covariance decreases with the distance between the points\cite{Gray2006,Novembre2008,Antognini2018}.
With the empirical contours that appears to be the case (and for random walks it is there by design): there is higher correlation between successive pitches and lower correlation between distant pitches.
As a result, the higher covariances are concentrated along the diagonal.
Again, this clearest for the phrases and random segments.
For motifs we see some deviations: two `blocks' in the covariance matrix, and corresponding jumps half way through the principal components.
This is easily explained by the fact that motifs often span only two notes.
In that case, all pitches in the first half of the contour are then perfectly correlated, as are pitches in the final half.
Crucially, despite such deviations from a perfect Toeplitz structure, the principal components are still well-approximated by cosines.

If you let a Toeplitz matrix grow in size, it asymptotically tends towards a \emph{circulant} matrix, preserving properties such as eigenvalues and eigenvectors along the way \cite{Gray2006}.
Circulant matrices have exactly the same values in every row, but rotated one step to the right with respect to the previous row.
This has the surprising result that all circulant matrices have the same eigenvectors: basis vectors of the discrete Fourier transform.
For a real and symmetric matrices, like covariance matrices, this results in cosine-shaped eigenvectors of increasing frequency---exactly what we see in figure \ref{fig:pca}.
We discuss all of this in more detail in the supplement S2.
In sum, because of a Toeplitz-like covariance structure, the principal components of melodic contours will tend to look like cosine functions.

%————————————————————————
\section{Cosine contours}
%————————————————————————

Next we turn this observation, and its explanation, into a proposal for a new contour representation.
The idea is to approximate the principal components by cosine functions and then project the contours on those first few cosines to obtain a low-dimensional representation.
This is exactly equivalent to taking a \emph{discrete cosine transform} (\textsc{dct}) of the contour \cite{Ahmed1974}.

Formally, consider a collection of contours of length $N$ as before.
We approximate the $k$-th principal component $\vu_k$ by a vector $\vect v_k = \bigl(v_k(0), \dots, v_k(N-1)\bigr)$ whose entries are given by the cosine function\footnote{%
    %———
    These basis functions correspond to the most popular version of the discrete cosine transform, \textsc{dct-ii}, for which fast implementations are widely available; others would have been possible \cite{Strang1999}.
    }%———
%———
\begin{align}
    v_k(n)
        &= \alpha_k \cdot \cos \frac{\pi(2n + 1)k}{2N}.
\end{align}
%———
Here $\alpha_0 = 1/\sqrt{N}$ and $\alpha_k = \sqrt{2/N}$ for $k \ge 1$ are normalizing constants ensuring that $\vect v_k$ has unit norm.
The projection of a contour $\vx = (x_0, \dots, x_{N-1})$ on $\vect v_k$ is then given by the inner product $c_k =  \vect v_k^T \vx$.
Expanding this gives the usual definition of the discrete cosine transform (\textsc{dct-ii}):
%———
\begin{align}
    c_k
    = \sum_{n=0}^{N-1}
        x_n  \alpha_k \cos \frac{\pi(2n + 1)k}{2N}.
\end{align}
%———
Conversely, the contour can be reconstructed from the coefficients $c_k$ using the inverse transform $x_n = \sum_{k=0}^{N-1} c_k v_k(n)$.
Using only $D<N$ coefficients, we define our low-dimensional \emph{cosine contour representation} as $C_D(\vx) = (c_1, \dots, c_D)$.
Note that we deliberately discard $c_0$.
This coefficient corresponds to a flat line and describes the overall pitch height of a contour: exactly what we need to get rid of to make the contour transposition invariant.
In this way we resolve the centering of contours discussed above.

\begin{figure}[t]
    \centering
    \includegraphics{figs/fig03-evaluation.pdf}
    \caption{%
        \textbf{\textsc{dct} approximates \textsc{pca}},~%
        %——————————————————————————————————————————————
        the optimal transform,
        in terms of the reconstruction error \subfiglabel{(a)} and the explained variance ratio \subfiglabel{(b)}.
        The reconstruction error is the mean squared error between an contour and a lower dimensional reconstruction.
        Note that data corresponds to figure \ref{fig:pca}, and that we did \emph{not} discard the first component $c_0$ of the \textsc{dct} in this figure.
    }
    \label{fig:evaluation}
\end{figure}

Why use this representation instead of principal components?
Indeed, a principal component projection (also known, in this context, as the \emph{Karhunen-Loève transform}), is optimal in several ways \cite{Rao1990,Ahmed1974}.
Not only does it decorrelate the data, it also packs most variance in the first few transform coefficients (sometimes called \emph{energy compaction}), and minimizes the reconstruction error when using only a few coefficients.
However, the transformation depends on the data.
Concretely, the principal components of German phrase contours differ from Chinese ones.
Any choice for using one of the two is arbitrary.
In contrast,  the \textsc{dct} is a principled, neutral solution---that approximates the optimal transform.
In fact, the \textsc{dct} was originally introduced for similar reasons \cite{Ahmed1974}, and was then found to empirically approximates \textsc{pca} well in domains ranging from image to audio \cite{Rao1990}.
The current results suggest that the same applies for melodies.

%————————————————————————————————————
\section{Evaluation and case studies}
%————————————————————————————————————

We evaluate proposed contour representation by comparing it to a principal component transformation, to demonstrate that representation is close to the optimum. We further designed three case studies to illustrate its usefulness at the levels of (1) song, (2) phrases and (3) motifs.
The case studies show that the representation is musicologically meaningful, as it allows visualization of variation (1), a quantitative evaluation of constraints on variation (2), and accurate classification into traditional categories (3).
For simplicity, we only look at two dimensional representations in these case studies, but higher dimensions may be useful in practice.

\subsection{Optimality}
%----------------------

To empirically verify the claim that the \textsc{dct} approximates the optimal \textsc{pca} transform, we compute the reconstruction error and the explained variance ratio using the same data as before.
The reconstruction error is measured as the mean square error between a contour and its $D$-dimensional reconstruction, using either the principal components (\textsc{pca}) or cosines (\textsc{dct}) as basis functions (so for $D=N$, the reconstruction is guaranteed to be perfect).
Figure \ref{fig:evaluation}\textsc{a} shows that the reconstruction errors of \textsc{dct} closely approximate that of \textsc{pca}.
For the shorter contours (motifs and phrases), the error very rapidly decreases, indicating that low-dimensional representations are already effective.
Indeed, to explain 95\% of the variance using cosine contours, you need 1 dimension for motifs, 9 for phrases and 61 for songs (this is sometimes called the \emph{effective dimensionality} \cite{Moore2018}).\footnote{%
    %——
    However, note that Moore et al \cite{Moore2018} show that high-dimensional random walks can falsely appear to have a low effective dimensionality.%
    %——
    }

\begin{figure}[t]
    \centering
    \includegraphics{figs/fig04-songs.pdf}
    \caption{
        \textbf{Songs of three cultures} represented in the cosine contour space \subfiglabel{(a)} show substantial variability.
        The average of all contours in a tradition \subfiglabel{(b--d)} also illustrates this
        (thick black lines; dashed lines highlight one contour).
        }
    \label{fig:songs}
\end{figure}

\subsection{Case Study 1: Visualizing different traditions}
%----------------------------------------------------------

Low dimensional representations of song contours are not likely to be very informative, yet we find that some traditions can be somewhat distinguished in just two dimensions.
Figure \ref{fig:songs} shows song contours from German, Chinese and Sioux songs.
Sioux songs have a striking overall shape (subplot \textsc{d}), often strongly descending, which is reflected in the distribution of contour shapes.
Similarly, German songs appear to be more arch-like than songs from the other traditions.

\subsection{Case Study 2: The melodic arch hypothesis}
%-----------------------------------------------------

In a second case study, we look at the melodic arch hypothesis, which states that \emph{phrases} tend to be arch-shaped or descending \cite{Huron1996} (see figure \ref{fig:melodic-arch}\textsc{a, b}) in a way that it becomes much easier to test (cf.~\cite{Savage2017a}).
We observe that the first component $c_1$ of a cosine representation roughly measures the \emph{descendingness} of the contour, and, similarly, that $-1\cdot c_2$ measures the \emph{archedness}.
The melodic arch hypothesis can thus be reformulated as stating that $c_1$ and $-c_2$ are larger for phrases than %expected by chance.
for random segments of the melodies (cf.~\cite{Cornelissen2020DLfM}).
Comparing Chinese and German phrases, we find that all are significantly ($p \ll 0.001$) more descending and arched than the corresponding random segments (see figure \ref{fig:melodic-arch}\textsc{c, d}).
This demonstrates that the coefficients of the cosine contour representation are musicologically meaningful.

\begin{figure}[t]
    \centering
    \includegraphics{figs/fig05-melodic-arch.pdf}
    \caption{\textbf{Phrases} of German \subfiglabel{(a)} and Chinese \subfiglabel{(b)} songs tend to be more descending and arched compared to random segments from the same melodies, as visible from their average contours.
    This can be quantified by comparing the first \subfiglabel{(c)} and second \subfiglabel{(d)} coefficients of their cosine representations.}
    \label{fig:melodic-arch}
\end{figure}

\subsection{Case Study 3: Mode classification}
%---------------------------------------------

In the final case study, we evaluate the performance of this contour representation on a task: mode classification in plainchant.
Gregorian chant uses a system of eight \emph{modes}: Dorian, Phrygian, Lydian and Mixolydian, each in the two flavours plagal and authentic.
Modes differ not only in their scales, but also in their melodic movement.
Plagal melodies tend to move lower than authentic ones, closer around the tonal center.
In a recent paper we suggest that the mode of Gregorian chant can be predicted from contours alone, in that case using a Parsons code contour representation \cite{Cornelissen2020ISMIR}.
We sliced up chants in sequences of motifs corresponding to the notational units (so called \emph{neumes}) or textual units: all notes set to one \emph{syllable} of the text would form a unit, and similarly for \emph{words}.
Next, we represented chants as vectors of motif or \emph{term frequencies} (tf), where each entry was weighted by the \emph{inverse document frequency} (df; the number of chants or documents containing that motif).
A linear support vector machine was then trained on these \emph{tf--idf vectors} to predict the mode.

\begin{figure}[t]
    \centering
    \includegraphics{figs/fig06-mode-classification.pdf}
    \caption{
        \textbf{Motifs used for mode classification} in Gregorian chant.
        \subfiglabel{(a)}
        A chant is segmented into motifs derived from the notation (neumes) or lyrics (syllables, words).
        The blue curves show the two-dimensional cosine contours for those motifs.
        \subfiglabel{(b)}
        We discretize the contour space and represent the chant as a vector of tf--idf weighed motif frequencies (`grid cell frequencies').
        Dots illustrate the nonzero entries of this vector for the chant shown above.
        \subfiglabel{(c)}
        The chant is now a walk through contour space, but our `bag of motifs` ignores order.
        \subfiglabel{(d)}
        Using these vectors to classify mode, we outperform a previous study using a Parsons code for the smaller motifs neumes and syllables.
        }
    \label{fig:chant}
\end{figure}

We repeat these experiments using a two-dimensional cosine representation for the motifs rather than a Parsons code.
There is one technical problem: whereas cosine contours are continuous, the tf--idf model requires a discrete vocabulary of motifs.
We therefore discretize the cosine contour space to a grid, and effectively treat every chant as a sequence of grid-cells (fig.~\ref{fig:chant}\textsc{c}).
All in all, this introduces two new parameters to the experiment: the dimensionality of the cosine contour and the resolution of the grid.
In this case study, we do not tune these parameters and focus on two dimensional contours, discretized to a grid between $-20$ and $20$ with a grid size of 1.
For ease of reading, the figure~\ref{fig:chant}\textsc{b} shows the grid only from $-10$ to $10$.

The results are summarized in figure \ref{fig:chant}\textsc{d}.
We see an interesting pattern: the cosine contours outperform the original results for small motifs such as neumes and syllables, but not for words, which are much longer motifs.
This seems to makes sense: two dimensional cosine contours are a fairly crude approximation of those longer contours, but may reasonably approximate short motifs.

%———————————————————————————————————
\section{Discussion and conclusions}
%———————————————————————————————————

This paper proposed a novel representation for melodies using the discrete cosine transform.
Observing that the principal components of melodies tend to be shaped like cosines, this representation approximates the optimal representation in the sense that it packs most variance in a few dimensions.
First, the cosine representation is easily interpretable, since it presents contours as a linear combination of cosine functions with intuitive shapes.
Second, by changing the dimensionality, the level of abstraction of the contour can be varied, allowing arbitrary small reconstruction error by including more and more dimensions.
Third, this representation allows one to map contours at multiple levels, from motifs to songs, to one common space.
The cosine representation thus creates a common ground for comparing contours across traditions and levels.
That is possible as, fourth, the representation is independent of the data, and in that sense culturally neutral.

The observation that principal components of spatial and temporal data can have sinusoidal shapes is not novel, but does not appear to be widely known.
Indeed, the sinusoidal shapes have been interpreted as genuine effects, rather than mathematical artefacts.
For example, one study interpreted gradients in the principal components of human genetic variation across the world as evidence for certain migration events in human history \cite{Cavalli-Sforza1993}.
Closer inspection revealed that those gradients were sinusoidal `artefacts' analogous to those reported in the present paper \cite{Novembre2008}.
Closer to \textsc{mir}, it has been observed that the training trajectories of deep neural networks have sinusoidal principal components \cite{Lorch2016}, for the same reason.
Again, a detailed analysis \cite{Antognini2018} revealed these were artefacts, but accurately reflecting the behaviour of high-dimensional random walks \cite{Antognini2018,Moore2018}.
We hope this paper helps increasing the awareness of this phenomenon.

The present work only begins to explore this new contour representation and raises many further questions.
One particularly promising possibility is the application to audio data.
We only explored symbolic data, but the proposed representation lends itself well for applications on acoustic data.
One application we hope to explore further is the analysis of speech intonation using the cosine contour representation.
A possible other avenue would be the analysis of folk song recordings, of which vast collections have been collected.
Folk song researchers have often used contour in some way to organize repertoires \cite{Adams1976}, and this representation may contribute to that.
Contour typologies have also be used in cross-cultural comparisons (see e.g.~\cite{Savage2015}).
Many typologies have been proposed \cite{Adams1976,Huron1996,Kelkar2018,Savage2012}, but they have not been systematically evaluated, and we think the proposed representation will be valuable there.

%—————————————————————————
\section{Acknowledgements}
%—————————————————————————

We would like to thank Henkjan Honing and Marianne de Heer Kloots for their feedback on the manuscript.
We would also like to thank the four anonymous reviewers for their careful reviews; we have tried to address all your comments.

%——————————————————————————
\bibliography{bibliography}
%——————————————————————————

\end{document}


\documentclass{beamer}

\usepackage{verbatim}
\usepackage{fancyvrb}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{booktabs}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{calc}
\usepackage{color}
\usepackage{multicol}
\usepackage{wrapfig}
\usepackage{natbib}
\usepackage[ruled,vlined,linesnumbered]{algorithm2e}
\usepackage{animate}
\usepackage{mathtools}
\usepackage{listings}

\usepackage{cmbright}
\fontencoding{OT1}\fontfamily{cmbr}\selectfont %to load ot1cmbr.fd
\DeclareFontShape{OT1}{cmbr}{bx}{n}{% change bx definition
<->cmbrbx10%
}{}
\normalfont % back to normalfont

% two col: two columns
\newenvironment{twocol}[4]{
\begin{columns}[c]
\column{#1\textwidth}
#3
\column{#2\textwidth}
#4
\end{columns}
}

\makeatletter
\setbeamertemplate{theorem begin}
{%
\begin{\inserttheoremblockenv}
  {}{\usebeamerfont*{block title}\usebeamercolor[fg]{block title}%
  \inserttheoremname
  %\inserttheoremnumber
  \ifx \inserttheoremaddition \empty \else\ (\inserttheoremaddition)\fi
  \inserttheorempunctuation}
  \normalfont
  }
  \setbeamertemplate{theorem end}{\end{\inserttheoremblockenv}}
\makeatother

\newcommand{\E}{\mathrm{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\sd}{\mathrm{sd}}
\newcommand{\s}{\mathrm{s}}
\newcommand{\Corr}{\mathrm{Corr}}
\newcommand{\rank}{\mathrm{rank}}
\newcommand{\trace}{\mathrm{trace}}
\newcommand{\nullspace}{\mathrm{null}}
\newcommand{\myspan}{\mathrm{span}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\softmax}{softmax}
\DeclareMathOperator{\diag}{diag}

\definecolor{darkgreen}{rgb}{0,0.5,0}

\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{exe}{Exercise}
\newtheorem{notation}{Notation}
\newtheorem{remark}{Remark}

\definecolor{darkgreen}{rgb}{0,0.5,0}

\title{Remedial Measures}
\author{Zhenisbek Assylbekov}
\institute{Department of Mathematics}
\date{Regression Analysis}

\AtBeginSection[]
{
  \begin{frame}<beamer>
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Weighted Least Squares}
\begin{frame}{Non-homogeneous variance}
\begin{columns}
\begin{column}{.4\textwidth}
\begin{figure}
    \centering
    \includegraphics[width=.9\textwidth]{plots/heteroscedastic.pdf}
\end{figure}
\end{column}
\begin{column}{.6\textwidth}
\begin{itemize}
    \item Chapters 3 and 6 discuss transformations of $x_1, \ldots, x_k$ and/or $Y$.
    \item<2-> More advanced remedy: \textbf{weighted least squares} (WLS) regression.
    \item<3-> Model is as before
    $$
    Y_i = \beta_00 + \beta_1\cdot x_{i1} + \ldots + \beta_k\cdot x_{ik} + \epsilon_i,
    $$
    \item<4-> Except that
    $$
    \epsilon_i\,\,{\stackrel{\text{ind}}{\sim}}\,\,\mathcal{N}(0,\sigma^2_i)
    $$
\end{itemize}
\end{column}
\end{columns}
\end{frame}

\begin{frame}{Weighted least squares}
\begin{itemize}
\item We have $\Var[Y_i] = \sigma_i^2$.
\item<2-> Idea: Give observations with higher variance less
weight in the regression fitting.
\item<3-> Let $\omega_i=\frac{1}{\sigma^2_i}$. WLS solves
$$
\min_{\boldsymbol\beta}\sum_{i=1}^n\omega_i[Y_i-(\beta_0+\beta_1 x_{i1}+\ldots+\beta_k x_{ik})]^2
$$
\item<4-> Or in matrix notation:
$$\min_{\boldsymbol\beta}(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)^\top\mathbf{\Omega}(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)$$
where $\mathbf{\Omega}=\diag[\omega_1,\ldots,\omega_n]\in\mathbb{R}^{n\times n}$.
\end{itemize}
\end{frame}

\begin{frame}{Solving WLS}
    Let us rewrite the WLS objective as:
    \begin{align*}
        \hat{\boldsymbol{\beta}}_{\text{WLS}}&=\argmin_{\boldsymbol{\beta}}(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)^\top\mathbf{\Omega}(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)\\
        \onslide<2->{&=\argmin_{\boldsymbol{\beta}}(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)^\top\mathbf{\Omega}^{1/2}\mathbf{\Omega}^{1/2}(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)\\}
        \onslide<3->{&=\argmin_{\boldsymbol{\beta}}(\mathbf{\Omega}^{1/2}\mathbf{Y}-\mathbf{\Omega}^{1/2}\mathbf{X}\boldsymbol\beta)^\top(\mathbf{\Omega}^{1/2}\mathbf{Y}-\mathbf{\Omega}^{1/2}\mathbf{X}\boldsymbol\beta)\\}
        \onslide<4->{&=\argmin_{\boldsymbol{\beta}}\|\mathbf{\Omega}^{1/2}\mathbf{Y}-\mathbf{\Omega}^{1/2}\mathbf{X}\boldsymbol\beta\|^2\qquad\text{(This is OLS!)}}
    \end{align*}
    \onslide<5->{Hence}
    \begin{multline*}
    \onslide<5->{\hat{\boldsymbol\beta}_\text{WLS}=\left((\mathbf{\Omega}^{1/2}\mathbf{X})^\top(\mathbf{\Omega}^{1/2}\mathbf{X})\right)^{-1}\left(\mathbf{\Omega}^{1/2}\mathbf{X}\right)^\top\mathbf{\Omega}^{1/2}\mathbf{Y}}\\\onslide<6->{=(\mathbf{X}^\top\mathbf{\Omega}\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{\Omega}\mathbf{Y}}
    \end{multline*}
\end{frame}

\begin{frame}{But $\sigma_i$'s are unknown!}
\begin{itemize}
    \item However, $\sigma_1,\ldots,\sigma_n$ are usually unknown!
    \item<2-> Notice, that in OLS
    $$
    \E[e_i^2] = \Var[e_i]+(\E[e_i])^2=\Var[Y_i-\hat{Y}_i]=\sigma^2(1-h_{ii})
    $$
    \item<3-> So $e_i^2$ in OLS estimates $\sigma^2$
and $|e_i|$ estimates $\sigma$ if $h_{ii} \approx 0$.
    \item<4-> Look at plots of $|e_i|$ from an OLS fit against $x_i$'s and $\hat{Y}_i$'s to see how $\sigma$ changes with predictors or fitted values.
    \item<5-> For example, if $|ei|$ increases linearly with $\hat{Y}_i$, then we'll
    fit
    $$
    |e_i| = \alpha_0 + \alpha_1\cdot x_{i1} + \ldots + \alpha_k\cdot x_{ik} + \delta_i
    $$
    and obtain the fitted values $\widehat{|e_i|}$.
    \item<6-> Or, e.g., if $|\epsilon_i|$ increases linearly w.r.t. $x_{i4}$ only, then we'll fit $|e_i|=\alpha_0+\alpha_1\cdot x_{i4}+\delta_i$
\end{itemize}
\end{frame}

\begin{frame}[fragile]{Putting it all together}
\begin{enumerate}
    \item Regress $Y$ against predictor variable(s) as usual (OLS), and obtain $e_1,\ldots, e_n$ \& $\hat{Y}_1, \ldots, \hat{Y}_n$.
    \item<2-> Regress $|e_i|$ against (all or some) predictors $x_1, \ldots, x_k$ or fitted values $\hat{Y}$.
    \item<3-> Let $\widehat{|e_i|}$ be the fitted values for the regression in 2.
    \item<4-> Define $\omega_i = 1/\widehat{|e_i|}^2$ and feed them into \verb|lm| command using the \verb|weights| parameter.
\end{enumerate}
\end{frame}

\begin{frame}{Example: diastolic blood pressure vs age}
We are interested in studying the relationship between diastolic blood pressure and age among healthy adult women 20 to 60 years old.\\~\\

\pause Fitting an OLS $\text{dbp}_i=\beta_0+\beta_1\cdot\text{age}_i+\epsilon_i$ gives:
\begin{figure}
    \includegraphics[width=.49\textwidth]{plots/res_age.pdf}\includegraphics[width=.49\textwidth]{plots/abs_res_age.pdf}
\end{figure}
\end{frame}

\begin{frame}[fragile]{OLS vs WLS}{\url{https://github.com/zh3nis/MATH440/blob/main/chp11/dbp.R}}
\begin{footnotesize}
\begin{verbatim}
> summary(ols)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) 56.15693    3.99367  14.061  < 2e-16 ***
age          0.58003    0.09695   5.983 2.05e-07 ***
---
Residual standard error: 8.146 on 52 degrees of freedom
Multiple R-squared:  0.4077,	Adjusted R-squared:  0.3963
\end{verbatim}
\pause\begin{verbatim}
> summary(wls)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) 55.56577    2.52092  22.042  < 2e-16 ***
age          0.59634    0.07924   7.526 7.19e-10 ***
---
Residual standard error: 1.213 on 52 degrees of freedom
Multiple R-squared:  0.5214,	Adjusted R-squared:  0.5122
\end{verbatim}
\end{footnotesize}
\end{frame}

\begin{frame}{Comments}
\begin{itemize}
    \item $\s[b_1]$ reduced from 0.097 (OLS) to 0.079 (WLS)
    \item<2-> $R^2$ is no longer interpreted the same way in terms of amount of total variability explained by model.
    \item<3-> In WLS, standard inferences about coefficients may not be valid for small sample sizes when weights are estimated from the data.
    \item<4-> If MSE of the WLS regression is near 1, then our estimation of the $\sigma_i$ function is okay. Here it's 1.21.
\end{itemize}
\end{frame}

\section{Ridge Regression}
\begin{frame}{Ridge Regression}
If some predictors are collinear, then columns of $\mathbf{X}$ become linearly dependent, and $\mathbf{X}$ loses its rank.\\~\\
\onslide<2->{\structure{Exercise}: Show that for $\mathbf{X}\in\mathbb{R}^{n\times d}$ with $n\ge d$
$$
\mathrm{rank}(\mathbf{X})<d\quad\Rightarrow\quad\nexists\,{(\mathbf{X}^\top\mathbf{X})^{-1}}.
$$}%
\onslide<3->{This is bad for OLS, because $\mathbf{X}^\top \mathbf{X}$ will not be invertible.\\~\\

Simple solution: add penalty term into the cost function:
$$
Q(\boldsymbol{\beta})=\|\mathbf{Y}-\mathbf{X}\boldsymbol\beta\|^2+\lambda\|\boldsymbol{\beta}\|^2,
$$
where $\lambda$ is a \textit{hyperparameter}, to be chosen through a criterion like PRESS or training/validation approach.}
\end{frame}

\begin{frame}{Solving Ridge Regression}
\begin{theorem} The function $Q(\boldsymbol{\beta})=\|\mathbf{Y}-\mathbf{X}\boldsymbol\beta\|^2+\lambda\|\boldsymbol{\beta}\|^2$
reaches its minimum at
$$
\hat{\boldsymbol{\beta}}_{\text{R}}=(\mathbf{X}^\top\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{Y}
$$%
\end{theorem}
\begin{proof}
\vspace{-15pt}
\begin{align*}
\onslide<2->{Q(\boldsymbol\beta)&=}\onslide<3->{(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)^\top(\mathbf{Y}-\mathbf{X}\boldsymbol\beta)+\lambda\boldsymbol\beta^\top\boldsymbol\beta\\}
\onslide<4->{&=\mathbf{Y}^\top\mathbf{Y}-\boldsymbol\beta^\top\mathbf{X}^\top\mathbf{Y}-\mathbf{Y}^\top\mathbf{X}\boldsymbol\beta+\boldsymbol\beta^\top\mathbf{X}^\top\mathbf{X}\boldsymbol\beta+\lambda\boldsymbol\beta^\top\mathbf{I}\boldsymbol\beta\\}
\onslide<5->{\nabla_{\boldsymbol\beta}Q&=-\mathbf{X}^\top\mathbf{Y}-\mathbf{X}^\top\mathbf{Y}+2\mathbf{X}^\top\mathbf{X}\boldsymbol\beta+2\lambda\mathbf{I}\boldsymbol\beta}\onslide<6->{\\&=2(\mathbf{X}^\top\mathbf{X}+\lambda\mathbf{I})\boldsymbol\beta-2\mathbf{X}^\top\mathbf{Y}}\onslide<7->{=\mathbf{0}\\}
\onslide<8->{&(\mathbf{X}^\top\mathbf{X}+\lambda\mathbf{I})\boldsymbol\beta=\mathbf{X}^\top\mathbf{Y}\quad\Rightarrow\quad}\onslide<9->{\boxed{\hat{\boldsymbol\beta}_\text{R}=(\mathbf{X}^\top\mathbf{X}+\lambda\mathbf{I})^{-1}\mathbf{X}^\top\mathbf{Y}}}
\end{align*}
\end{proof}

\onslide<10->{\structure{Exercise.} Show that $(\mathbf{X}^\top\mathbf{X}+\lambda\mathbf{I})$ is \textit{always} invertible for $\lambda>0$.}
\end{frame}

\begin{frame}{Chapter 7 example: Body fat}
$n=20$ healthy females 25--34 years old.
\begin{itemize}
\item $x_1=$ triceps skinfold thickness (mm)
\item $x_2=$ thigh circumference (cm)
\item $x_3=$ midarm circumference (cm)
\item $Y=$ body fat (\%)
\end{itemize}

Obtaining $Y_i$, the percent of the body that is purly fat, requires
immersing a person in water. Want to develop model based on simple body measurements that avoids people getting wet.
\end{frame}

\begin{frame}{Scatterplot}
\centering\includegraphics[scale=0.45]{plots/scatterplot}
\end{frame}

\begin{frame}{Correlation coefficients}
\begin{center}
\includegraphics[scale=0.3]{plots/corr-matrix}
\end{center}
\pause There is high correlation among the predictors. \pause For example
$r = 0.92$ for triceps and thigh. These two variables are \textit{essentially
carrying the same information}. \pause Maybe only one or the other is really needed.
\end{frame}

\begin{frame}[fragile]{Effects of multicolinearity}
\begin{small}
\begin{verbatim}
lm(formula = bodyfat ~ triceps + thigh + midarm,
   data = bodyfat_data)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  117.085     99.782   1.173    0.258
triceps        4.334      3.016   1.437    0.170
thigh         -2.857      2.582  -1.106    0.285
midarm        -2.186      1.595  -1.370    0.190
\end{verbatim}
\end{small}

\begin{itemize}
    \item\pause Two of the three regression effects are {\it negative}.
    \item\pause Holding midarm
and triceps constant, increasing the thigh circumference {\it decreases} bodyfat.
    \item\pause This may not make sense!
\end{itemize}
\end{frame}

\begin{frame}[fragile]{OLS vs Ridge on bodyfat data}{\url{https://github.com/zh3nis/MATH440/blob/main/chp11/ridge.R}}
\begin{columns}
\begin{column}{.6\textwidth}
\includegraphics[height=.4\textheight]{plots/ridge_lambda.pdf}
\end{column}
\begin{column}{.4\textwidth}
\begin{footnotesize}
\begin{verbatim}
> mean(ols$residuals^2)
[1] 4.920244

> mean((y-ridge_yhat)^2)
[1] 5.340952
\end{verbatim}
\end{footnotesize}
\end{column}
\end{columns}
\begin{footnotesize}
\begin{verbatim}
> coef(ols)
(Intercept)     triceps       thigh      midarm
 117.084695    4.334092   -2.856848   -2.186060

> t(coef(ridge))
   (Intercept)   triceps     thigh     midarm
s0   0.6555753 0.8074414 0.1588796 -0.3266744
\end{verbatim}
\end{footnotesize}
\end{frame}

\section{Robust Regression}
\begin{frame}{Back to outliers}
\begin{itemize}
    \item Leverages $h_{ii}$ and deleted residuals $t_i$ useful for finding outlying $\mathbf{x}_i$ and $Y_i$ cases.
    \item\pause Cook's $D_i$ and $\text{DFFIT}_i$ indicate which cases are highly influencing the fit of the model.
    \item\pause What to do with influential and/or outlying cases? Are they transcription errors or somehow (un)representative of the target population?
    \item\pause Outliers are often interesting in their own right and can help in building a better model.
    \item\pause \textbf{Robust regression} weakens the effect of outlying cases on estimation to provide a better fit to the majority of cases.
    \item\pause Useful in situations when there's no time for ``influence diagnostics'' or a more careful analysis.
\end{itemize}
\end{frame}

\begin{frame}{M-estimation}
\begin{itemize}
    \item Robust regression is effective when the error distribution is not normal, but heavy-tailed.
    \item\pause \textbf{M-estimation} is a general class of estimation methods. \pause
    $$
    \min_{\boldsymbol\beta}\sum_{i=1}^n\rho(Y_i-\mathbf{x}_i^\top\boldsymbol\beta)
    $$
    \pause where $\rho(\cdot)$ is some function.
    \item\pause $\rho(u)=u^2$ gives OLS
    \item\pause $\rho(u)=|u|$ gives \textbf{least absolute residual} (LAR) regression
    \item\pause Huber's method is a compromise between OLS and LAR. \pause It looks like $u^2$ for $u$ around zero, and like $|u|$ for $u$ further away from zero.
\end{itemize}
\end{frame}

\begin{frame}{Iteratively reweighted least squares (IRLS)}
\begin{small}
Outlying residuals are (iteratively) given less weight in the estimation process.\\~\\

\begin{algorithm}[H]
\DontPrintSemicolon
\textbf{Input}: $\{(\mathbf{x}_i, {y}_i)\}_{i=1}^n$\;
Fit OLS. Let $\mathbf{e}=\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}_\text{OLS}$\;
Initialize weights: $\omega_i\leftarrow\frac{1}{e_i^2}$, $\mathbf{\Omega}=\diag[\omega_1,\ldots,\omega_n]$\;
Fit WLS: $\hat{\boldsymbol\beta}_{\text{WLS}}\leftarrow(\mathbf{X}^\top\mathbf{\Omega X})^{-1}\mathbf{X}^\top\mathbf{\Omega}\mathbf{Y}$\;
Estimate: $\hat{\sigma}\leftarrow\mathrm{median}_i\left\{\frac{|y_i-\mathbf{x}_i^\top\hat{\boldsymbol\beta}_{\text{WLS}}|}{\Phi^{-1}(0.75)}\right\}$\;
Update weights: $\omega_i\leftarrow w\left(\frac{y_i-\mathbf{x}_i^\top\hat{\boldsymbol\beta}_{\text{WLS}}}{\hat{\sigma}}\right)$, where
$$
w(u)=\begin{cases}
1,\quad &|u|<1.345\\
\frac{1.345}{|u|},&|u|>1.345
\end{cases}
$$\;\vspace{-10pt}
Repeat steps 4--6 until $\hat{\sigma}$ and $\hat{\boldsymbol\beta}_{\text{WLS}}$ stabilize.\;
\textbf{Output}: $\hat{\boldsymbol\beta}_{\text{WLS}}$
\caption{IRLS}
\end{algorithm}
\end{small}
\end{frame}

\begin{frame}[fragile]{IRLS example}{\url{https://github.com/zh3nis/MATH440/blob/main/chp11/irls.R}}
\begin{verbatim}
require(foreign)
require(MASS)

cdata <- read.dta(
    "https://stats.idre.ucla.edu/stat/data/crime.dta")

plot(crime ~ poverty, data=cdata)
ols <- lm(crime ~ poverty, data = cdata)
abline(ols)

irls <- rlm(crime ~ poverty, data=cdata)
abline(irls, col='blue')

summary(ols)
summary(irls)
\end{verbatim}
\end{frame}

\begin{frame}{IRLS example}
\begin{figure}
    \centering
    \includegraphics[width=.9\textwidth]{plots/ols_irls.pdf}
\end{figure}
\end{frame}

\end{document}


\documentclass[main.tex]{subfiles}
\begin{document}

% \marginpar{Friday\\ 2019-12-13, \\ compiled \\ \today}
% \section*{Fri Dec 13 2019}

% We discuss the maximum mass of stars.

% We were able to give meaning to the parameter \(a\): the pressure is given by
% %
% \begin{align}
%   P(r) =
%   \frac{2 \pi}{3} G \rho_c^2 a^2 \qty(\exp(- \frac{r^2}{a^2}) - \exp(- \frac{R^2}{a^2}))
% \,,
% \end{align}
% %
% so we can see that
% %
% \begin{align}
% a = \qty(\frac{3M}{4 \pi \rho_c \sqrt{6}})^{1/3}
% \,,
% \end{align}
% %
% and then we got an expression for the central pressure \(P_c\): the parameter multiplying it is approximately \(\num{.44}\), while more accurate models give: if \(\gamma = 5/3\) (ideal gas) we get \(\num{.48}\) whiile if \(\gamma = 4/3\) (ultrarelativistic) we get \(\num{.36}\).

% We have the relation
% %
% \begin{align}
%   P_c = \frac{\rho _c }{\overline{m}} k_B T_c
% \,,
% \end{align}
% %
% which we use to get the last relation from last time.

% This allows us to get some figures for main sequence (hydrogen burning) stars.

% What is the maximum mass for Main Sequence stars?

In the core of the star, we have both nonrelativistic and relativistic material in equilibrium: electrons and protons are nonrelativistic, while photons are relativistic.
We have discussed earlier that if most of the material in a star were relativistic it would become unstable (since, as \(\gamma \to 4/3\), the binding energy approaches 0); let us then discuss the composition of the star.

% A star becomes unstabel when most of its material becomes ultrarelativistic: then, its total energy goes from a negative value to 0 and the adiabatic index approaches \(4/3\).

% Suppose that the central energy is partly given by radiation and partly by matter.
% We write
We can decompose the pressure at the core into the fractions due to nonrelativistic matter and to radiation:
%
\begin{align}
  P_c = P_m  + P_r = \beta P_c + (1 - \beta )P_c
\,,
\end{align}
%
where we define \(\beta \in (0,1)\) as the fraction of the core pressure which is due to matter: \(\beta = P_m / P_c\).

% where the terms of the two sums exactly correspond to each other, and
The two contributions can be separately expressed as:
%
\begin{align}
  \beta P_c = P_m &= \frac{\rho _c k_B T_c}{\overline{m}} \\
  (1 - \beta ) P_c = P_r &= \frac{1}{3} a T^4
\,,
\end{align}
%
where \(a\) is the radiation constant, related to the Stefan-Boltzmann constant \(\sigma \):
%
\begin{align}
  a = \frac{\pi^2   k_B^2}{15 \hbar^3 c^3}
\,.
\end{align}

We can then relate \(\beta \) to the mass of the star: in order to simplify the core temperature, we start by computing
%
\begin{align}
  \frac{\qty(\beta P_c )^{4}}{(1 - \beta ) P_c} &= \frac{\rho _c^{4}}{\overline{m}^{4}} \qty(k_B T_c)^{4} \frac{3}{a T_c^{4}}  \\
  \frac{\beta^{4}}{1 -\beta } P_c^3 &= \frac{3}{a} \qty(\frac{k_B \rho _c}{\overline{m}})^{4}
\,,
\end{align}
%
which we can invert to find an expression for the core pressure \(P_c \) in terms of \(\beta \), which we then compare to the expression we found for the core pressure as a result of the Clayton model:
% %
% \begin{align}
%   \frac{1 - \beta }{\beta^{4}} P_c^{-3} = \frac{a}{3}
%   \qty(\frac{k_B \rho _c }{\overline{m}})^{-4}
% \,.
% \end{align}
% Inverting this we can eliminate the temperature dependence:
%
\begin{align}
  P_c = \qty(\frac{3}{a} \frac{1 - \beta  }{\beta^{4}})^{1/3} \qty(\frac{k_B \rho _c }{\overline{m}})^{4/3}
  &= \qty(\frac{\pi }{36})^{1/3} G M^{2/3} \rho _c^{4/3} \\
  \qty(\frac{\pi }{36})^{1/3} G M^{2/3}
  &= \qty(\frac{3}{a} \frac{(1-\beta )}{\beta^{4}})^{1/3}
  \qty(\frac{k_B}{\overline{m}})^{4/3}
  \label{eq:maximum-mass-main-sequence}
\,,
\end{align}
%
the core density simplifies!

So, if we compare stars at the same stage of fusion so that \(\overline{m}\) is constant, we have \(M \propto f(\beta ) = (1 - \beta )^{1/2} / \beta^2\).
% so as \(\beta \) decreases, \(M\) increases.

\(f(\beta )\) decreases as \(\beta \) increases, and it diverges to \(+ \infty \) for \(\beta \to 0\).

Looking at the plot the other way, the heavier the star, the larger the contribution of radiation to the core pressure, which is what \(1-\beta \) quantifies.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/beta_star_core_pressure.pdf}
\caption{A plot of \(M\) in terms of \(\beta \).}
\label{fig:beta-core-pressure}
\end{figure}
% \todo[inline]{Figure made quickly, to improve by scaling it correctly, making it vector, setting the text in the right font. The plot is ready, but the units don't seem to work! }

This makes sense intuitively: heavier stars reach higher temperatures and densities, so they have more radiation in the core.

We know that for \(\beta \to 0\) the star is surely unstable, but the instability is actually reached earlier, since even before the gravitational binding energy being exactly zero large parts of the star can be flung out as stellar winds.
Proper considerations about what an appropriate critical value of \(\beta \) should be allow us to bound the stellar mass from above, at around \(50 M_{\odot}\).

% [Plot of \(1 - \beta \) versus \(M / M_{\odot} \), showing this.]

\subsection{Degenerate electron gas}

Now, we will deal with the degenerate electron gas in stars, and see what is its effect on the minimum and maximum mass of a star.

The distribution function of the electrons, which are fermions, is given by
%
\begin{align}
  f(p) = \qty[\exp(\frac{\epsilon _p - \mu }{k_B T})+1]^{-1}
\,,
\end{align}
%
where \(\epsilon _p = \sqrt{m^2 c^{4} + p^2 c^2}\). With it, we can calculate the number density of electrons:
%
\begin{align}
n_e = \frac{g_s}{h^3} \int \dd[3]{p} f(p)
\,,
\end{align}
%
where \(g_s\), the number of helicity states of the electron, is equal to 2.

We want to consider the degenerate case for this distribution, which corresponds to the saturation of all the low-energy configurations in phase space: this is known as a Fermi gas.
As the temperature approaches zero, the phase space distribution approaches the configuration
%
\begin{align}
f(p) = \lim_{T \to 0} \qty[\exp(\frac{\epsilon_p - \mu }{k_B T})+1]^{-1}=
\begin{cases}
  1 \qquad \epsilon _p < \mu  \\
  0 \qquad \epsilon _p > \mu
\,.
\end{cases}
\end{align}

The chemical potential yields a critical energy, known as the Fermi energy, \(\epsilon _F = \mu \), which is also tied to a Fermi momentum:
%
% Then, as \(T \rightarrow 0\) we get: \(f(\epsilon _p) = 1\) if \(\epsilon _p \leq \epsilon _F\) and \(f(\epsilon _p) = 0 \) if \(\epsilon _p > \epsilon _F\); we can express this energy in terms of the momentum:
%
\begin{align}
  \epsilon_F^2 = c^2p_F^2 + m^2 c^{4}
\,.
\end{align}

The number density of electrons in this configuration is given by the integral mentioned before: since the distribution is spherically symmetric we have
%
\begin{align}
  n_e = 2 \int_{0}^{p_F} \dd{p} p^2 4 \pi \frac{1}{h^3}
  = \frac{8 \pi }{3} \qty(\frac{p_F}{h})^{3}
\,,
\end{align}
%
% where we have a factor of \(2\) to account for the spin-\(1/2\) nature of the electrons.
% This means that
which allows us to express the Fermi momentum in terms of the number density of electrons:
%
\begin{align}
  p_F = \qty(\frac{3n_e}{8 \pi })^{1/3} h
\,.
\end{align}

In natural units, this is roughly \(p_F \approx \num{6.6} \sqrt[3]{n_e}\).

The energy density is given by the expression
%
\begin{align}
  \rho = \frac{2}{h^3} \int_{0}^{p_F} 4 \pi p^2\dd{p} \epsilon _p
\,,
\end{align}
%
which we can consider in either the nonrelativistic or the ultrarelativistic limit --- the analytic integral is complicated and not very enlightening.

\paragraph{Nonrelativistic limit}

% In the nonrelativistic limit we find
In this limit the energy is approximately
%
\begin{align}
  \epsilon _p = mc^2 + \frac{p^2}{2m}
\,,
\end{align}
%
so in the computation we need to integrate a polynomial: the result is
%
\begin{align}
  \rho = n \qty(mc^2 + \frac{3}{10} \frac{p_F^2}{m})
\,,
\end{align}
%
where the first term corresponds to the rest-energy of the electrons, while the second gives their kinetic energy.
We have derived earlier the following expression for the pressure of a nonrelativistic gas:
%
\begin{align}
  P  = \frac{2}{3} \frac{E_k}{V}
\,,
\end{align}
%
% where \(E/V\) is the kinetic energy density.
and \(E_k / V\) is precisely the kinetic energy density, the second term of the expression for the total energy density \(\rho \);
so for our nonrelativistic Fermi gas we will have:
%
\begin{align}
  P = n \frac{p_F^2}{5m}
\,.
\end{align}

Since the Fermi momentum \(p_F\) can be written as a function of the number density \(n_e\), so can the pressure \(P\): we find
%
\begin{align}
P = \underbrace{\frac{h^2}{5m} \qty(\frac{3}{8 \pi })^{2/3}}_{K_{NR}} n_e^{5/3}
\,.
\end{align}

\paragraph{Ultra relativistic limit}

% In
% %
% \begin{align}
%   P = k_{NR} n^{5/3}
% \,,
% \end{align}
% %
% where
% %
% \begin{align}
%   k_{NR} = \frac{h^2}{5m} \qty(\frac{3}{8 \pi })^{2/3}
% \,.
% \end{align}
%

In the relativistic case, on the other hand, we can approximate the energy as \(\epsilon _p \approx cp\), so the energy density will be given by
%
\begin{align}
\rho = \frac{3}{4} n \rho _F c
\,.
\end{align}

In this case, we also know that the pressure becomes
%
\begin{align}
  P = \frac{1}{3} \frac{E_k}{V}
\,,
\end{align}
%
so we find
%
\begin{align}
P = \underbrace{\frac{hc}{4} \qty(\frac{3}{8 \pi })^{1/3}}_{K_{UR}} n^{4/3}
\,.
\end{align}

\paragraph{Fermion gas classification}

We have discussed some expressions describing a non-relativistic or ultrarelativistic degenerate fermion gas.

We have derived our results with the assumption \(T \to 0\), but a gas can behave very similarly with nonzero temperatures as well.
What is the temperature threshold under which the gas behaves in a degenerate-like way?
We will not discuss how the transition region looks, but if \(k_B T \ll \epsilon _F\) then the gas behaves like a degenerate one, while if \(k_B T \gg \epsilon _F\) then there will be many unfilled gaps in the phase space distribution, so the gas will not be degenerate.

Recall that \(p_F \propto n_e^{1/3}\): therefore, in a log-log plot of temperature \(T\) versus density of possible electron densities \(n_e\) we can draw a line distinguishing the degenerate and nondegenerate cases, with the critical temperature becoming higher for higher \(n_e\).

% \todo[inline]{Is it really a straight line though? If the criterion is indeed to compare \(k_B T\) and \(\epsilon _p\) then I'd expect a curve, since \(\epsilon _F\) is not a polynomial function of \(p_F\)\dots}

% We make a plot: on the \(x\) axis we have the number density in \(\SI{}{m^{-3}}\), on the \(y\) axis we have the temperature in \(\SI{}{K}\).
Having distinguished the degenerate and nondegenerate regions, we can distinguish the relativistic and nonrelativistic ones: for the nondegenerate case, as is usual, we reach the relativistic condition if we increase the temperature.

In the degenerate case this is not really the case: as long as the gas is degenerate, the temperature does not really matter, and the gas becomes relativistic when the Fermi energy \(\epsilon _F\) becomes larger than the mass of the fermion.
Since \(\epsilon _F\) is only a function of the number density, this means that the gas can become relativistic at arbitrarily low temperatures as long as it is dense enough.

% \todo[inline]{Add plot --- maybe we can do a chromatic region plot, integrating numerically the distribution and coloring the region based on the fraction of relativistic particles, and for the degeneracy measure in some way how ``sharp'' the boundary is between the filled and unfilled regions?}
\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/relativisticity_degeneracy.pdf}
\caption{Regions in which an electron gas is degenerate (red) and/or relativistic (blue), depending on its density \(n_e\) and on its temperature \(T\). The colors are decided defining ``degenerate'' as having \(T < T_F = \epsilon _F - m_e c^2\) and ``relativistic'' as having the average kinetic energy of an electron (\(E_k = (\gamma - 1) m_e c^2\)), be larger than \(m_e c^2\). The transition region for both spans an order of magnitude, symmetrically around the equality condition.
Orders of magnitude are also given regarding common objects; ``iron core'' refers to the core of a massive star in the Silicon burning phase.}
\label{fig:relativisticity_degeneracy}
\end{figure}

% For more details, see the code at \url{https://github.com/jacopok/notes/blob/master/ap_first_semester/astrophysics_cosmology/figures/degenerate_relativistic.ipynb}.}

% We divide the plot into:
% \begin{enumerate}
%     \item Classical UR: \(P \propto n k_B T\);
%     \item classical NR (like the Sun);
%     \item degenerate NR: \(P = K_{NR} n^{4/3}\)
%     \item degenerate UR.
% \end{enumerate}

% Classical vs degenerate is marked by a line similar to \(T \sim n\), while we have NR for both \(T\) and \(n\) lower than certain critical values (since a degenerate gas can become ultrarelativistic even at low temperatures! this is the point).

\paragraph{Application to the Sun}

As we have discussed earlier, the core temperature, pressure and density of the Sun are related by the following relation:
%
\begin{align}
  P_c = \frac{\rho _c}{\overline{m}} k_B T_c
\,.
\end{align}

% and it can be (easily?) shown that
The average mass which appears here is a function of the chemical composition of the interior: we can neglect all the metals and only consider the mass fractions of hydrogen (\(x_1 \)) and of helium (\(x_4 \)):
%
\begin{align}
  \overline{m} = 2 m_H \times \frac{1}{1 + 3x_1 + 0.5 x_4}
\,.
\end{align}
%
% where \(x_{1, 4}\) are the concentrations of hydrogen and helium respectively.

\todo[inline]{This expression works well in the limits of \(x_1 =1\) and \(x_4 = 1\), but where does it come from? I would have expected \(\overline{m} = (x_1 m_H  + x_4 m_{He} ) /2\)\dots}

The Clayton model gave us an expression for the central pressure \(P_c\), which we turned into one for the central temperature \(T_c\):
%
\begin{align}
P_c &=\approx \qty(\frac{\pi }{36})^{1/3} G M^{2/3} \rho _c^{4/3} \\
 k_B T_c &\approx \qty(\frac{\pi }{36})^{1/3} G \overline{m}
  M^{2/3} \rho _c^{1/3}
\,,
\end{align}
%
however when deriving it we not consider the effect of the fact that the gas there may be at least party degenerate.

Let us consider a different approximation: suppose that the electrons in the core are fully degenerate and nonrelativistic, while the ions (whose density is \(n_i\), which by local neutrality is also equal to \(n_e = \rho _c / \overline{m}\)) are completely classical.\footnote{In order to see why it makes sense to consider them as classical while the electrons are degenerate, let us look at the fact that in the nonrelativistic approximation the Fermi energy is given by \(\epsilon _F = p_F^2 / 2m \sim m^{-1} n^{2/3}\), so the critical temperature needed for a Fermi gas to become degenerate depends on the number density as well as the mass of the particle: for a higher-mass particle, the Fermi energy is lower.

The electrons being degenerate means that the temperature of the core is (roughly speaking) lower than their Fermi energy; the Fermi energy of the ions however is at least three orders of magnitude lower, so it makes sense that it is not as low as the Fermi temperature of the ions.

In order to have some numbers at hand, with a number density like that of the core of the Sun the Fermi temperature for electrons is \(\sim \SI{11}{MK}\), while the Fermi temperature for protons is a measly \(\sim \SI{6000}{K}\). The actual temperature of the core is \(T_c \sim \SI{15}{MK}\), slightly above the Fermi temperature of electrons. It is close enough that modelling them as degenerate works, while the assumption of the ions being nondegenerate is completely valid.}
Our estimate for the central pressure will need to account for both electrons and ions:
% To estimate the maximum achievable central temperature we do:
%
\begin{align}
  P_c = k_{NR} n_{e}^{5/3} + n_i k_B T_c
\,.
\end{align}
%

Let us equate this expression with the one given by the Clayton model for the central pressure: we find
%
\begin{align}
  \qty(\frac{\pi}{36})^{1/3} G M^{2/3} \rho_{c}^{4/3}
  &= k_{NR} \qty(\frac{\rho_c}{m_{H}})^{5/3} + \frac{\rho_{c}}{m_H} k_B T_c \\
  k_B T_c &= \underbrace{\qty(\frac{\pi}{36})^{1/3} G m_{H} M^{2/3}}_{A} \rho_{c}^{1/3} - \underbrace{k_{NR} m_H^{-2/3}}_{B} \rho _c^{2/3}
\,.
\end{align}
%

We can then ask what is the maximum temperature \(T_c\) we can reach for a given mass \(M\) if we vary the core density \(\rho _c\):
this can be calculated to be
% (((np.pi/36) * ac.G**3 * ac.m_p**3 )/ (2 * ac.h**6 / 5**3 / ac.m_e**3 * (3/8 /np.pi)**2 /ac.m_p**2)).to(u.kg/u.m**3 / u.M_sun**2)
%
\begin{align}
\rho_{c}^{\text{max}} = (A/ 2B)^{3} \approx \SI{5e7}{kg/m^3} \qty( \frac{M}{M_{\odot}})^2
\,,
\end{align}
%
where we have
%
\begin{align}
  k_B T_c = \frac{A^2}{4B} = \qty(\frac{\pi }{36})^{2/3} \frac{G^2m_H^{8/3}}{4 k_{NR}} M^{4/3}
  \approx \SI{5.7}{keV} \qty( \frac{M}{M_{\odot}})^{4/3}
\,.
\end{align}

% Now, we can set this temperature to be larger than the ignition temperature for any process we want, to see whether it will happen.
This allows us to estimate the minimum mass a star needs to have in order to fuse hydrogen: we just need to set \(T_c\) to be equal to the ignition temperature \(T_c = T _{\text{ign}} \approx \SI{1}{keV}\) and we find
%
\begin{align}
  M _{\text{min}} =
  \qty(\frac{36}{\pi })^{1/2}
  \qty(\frac{4 k_{NR}}{G^2m_H^{8/3}})^{3/4}
  \qty(k_B T _{\text{ign}})^{3/4}
  \approx \SI{.27}{M_{\odot}}
\,.
\end{align}

This is a much better estimate than the one we found earlier since we are now accounting for the degenerate Fermi gas nature of the electrons in the core (this lowers the estimate, since it means that even at relatively low temperatures there will be electrons with high energy) and since we are computing the core density \(\rho _c\) instead of the average density \(\overline{\rho}\).

\todo[inline]{The estimate is\dots still not great really, right? it is still 3 times larger than the correct value of \(\num{.08} M_{\odot}\)! How do we account for such a discrepancy?}

\paragraph{Expressing the result with coupling constants}

The gravitational potential energy between two hydrogen nuclei separated by a distance equal to their (reduced!) Compton wavelength \(r = \hbar / m_H c\) is
%
\begin{align}
E_g = - \frac{G m_H^2}{r} = - \frac{G m_H^{3}c }{\hbar}
\,.
\end{align}

Comparing this to the rest energy of an electron, \(E = m_H c^2\), is the way to calculate the \emph{gravitational coupling constant} \(\alpha _G\), a dimensionless parameter quantifying the ``strength'' of the gravitational interaction between hydrogen nuclei:
%
\begin{align}
  \alpha_{G} = \frac{E_g}{E} =  \frac{G m_H^2}{\hbar c} \sim \num{5.9e-39}
\,.
\end{align}

In natural units, \(\alpha _G = m_H^2 / m_P^2\).

By a similar line of reasoning we find the electromagnetic coupling constant:
%
\begin{align}
  \alpha_{EM} = \frac{e^2}{4 \pi \epsilon_{0} \hbar c} \approx \frac{1}{137}
\,,
\end{align}
%
which is \emph{enormously} greater.

In terms of the gravitational coupling constant the minimum mass we found can be written as
%
\begin{align}
  M _{\text{min}} \approx
  16 \qty(\frac{k_B T _{\text{ign}}}{m_e c^2})^{3/4}
  \alpha_{G}^{-3/2} m_H
\,.
\end{align}

If \(T _{\text{ign}} \sim \SI{1.5e6}{K}\), one tenth of the temperature of the Sun, we find
\todo[inline]{Is \SI{0.1}{keV} really enough to reach ignition? This seems to contradict what was said earlier\dots}
%
\begin{align}
  M _{\text{min}} \sim \num{.03} \alpha_{G}^{-3/2} m_H
\,.
\end{align}

We can apply a similar line of reasoning to the formula we found for the maximum mass: taking equation \eqref{eq:maximum-mass-main-sequence}  with a critical fraction of nonrelativistic matter of \(\beta = \num{.5}\) and \(\overline{m} = \num{.61} m_H\) we get a result which, once again, scales with \(\alpha _G^{-3/2} m_H\):
%
\begin{align}
  M _{\text{max}} \approx 56 \alpha_{G}^{-3/2} m_H
\,.
\end{align}

This hints to the fact that \(m_{*} = \alpha_{G}^{-3/2} m_H \) is an important characteristic mass for all of stellar evolution.

This is around \(\num{1.85} M_{\odot}\), and it corresponds to a number of nucleons of
%
\begin{align}
  N_{*} = \frac{m_{*}}{m_H} \approx \num{2e57}
\,.
\end{align}

\section{Stellar remnants}

\subsection{Full degeneracy and white dwarfs}

% Let us now suppose that the core of a star is held together by the pressure of degenerate electrons alone: we will discuss \emph{white dwarfs}.
White dwarfs are the remnants of low-mass stars who have exhausted the elements they are able to fuse in their core. They glow, emitting thermal radiation, which causes their temperature to slowly decrease until they become brown dwarfs.
They are dim but observable; the closest one to the Solar System is Sirius B, a companion to the brightest star in the night sky.

They are of interest to us since they allow us to apply the theory of degenerate Fermi gasses once more: they are very dense objects, since there is no fusion-induced pressure gradient inside them to balance gravity, and the electrons inside them form a degenerate gas.

% We define
The number density of electrons inside a white dwarf is given by
%
\begin{align}
  n_e = Y_{e} \frac{\rho_{c}}{m_H}
\,,
\end{align}
%
where \(Y_e = (1 + x_1) /2\) quantifies the number of electrons per baryon (\(x_1 \) is the hydrogen mass fraction).

\todo[inline]{Why would \(Y_e\) be given by that expression? hydrogen has one electron per each baryon, but helium also has half an electron per baryon\dots If the white dwarf was exclusively hydrogen, would we not expect \(Y_e = 1/2\)?}

Let us start by assuming that the matter is nonrelativistic: then
the pressure is given by
%
\begin{align}
  P = k_{NR} n_e^{5/3} = k_{NR} \qty(\frac{Y_e \rho_{c}}{m_H})^{5/3}
\,,
\end{align}
%
which as usual we compare to the results of the Clayton model:
%
\begin{align}
  P_c = \qty(\frac{\pi }{36})^{1/3} G M^{2/3} \rho_{c}^{4/3}
\,.
\end{align}

Equating these two we find
%
\begin{align}
  \rho_{c} \approx \frac{\num{3.1}}{Y_e^{5}} \qty(\frac{M}{m_{*}})^2 \frac{m_H}{(h / m_e c^2)^{3}}
\,.
\end{align}

% \todo[inline]{
%   Missing square on the \(M / m_*\)?
% }

If, on the other hand, we were to assume that the matter is ultrarelativistic the pressure would be given by
%
\begin{align}
P = k_{UR} n_e^{4/3}
= k_{UR}\qty(\frac{Y_e \rho_{c}}{m_H})^{4/3}
\,,
\end{align}
%
so, instead of getting an expression for the central density \(\rho _c\), we would find
%
\begin{align}
  k_{UR} \qty(\frac{Y_e \rho_{c}}{m_H})^{4/3} \approx
  \qty(\frac{\pi }{36})^{1/3} G M^{2/3} \rho_{c}^{4/3}
\,:
\end{align}
%
in this limit the expression becomes independent of \(\rho_{c}\)!

This gives us a limit mass, since as we increase the mass of a white dwarf which is not relativistic we increase its density and thus its temperature, making it closer to being relativistic, and this is the mass we get for the fully relativistic configuration (which is unstable because of the usual binding energy considerations).

The limit is known as the Chandrasekhar mass, the largest mass at which a fully degenerate white dwarf can support itself:
%
\begin{align}
  M_{CH} =
  \qty(\frac{36}{\pi } )^{1/2} \qty(\frac{Y_e}{m_H})^2
  \qty(\frac{k_{UR}}{G})^{3/2} \approx 2.3 Y_e^2 m_{*} \approx 4.3 Y_e^2 M_{\odot} \approx 1.4 M_{\odot}
\,.
\end{align}

% This is the maximum mass of a white dwarf to remain stable, held together by the degeneracy pressure of electrons.
% Above this, it becomes a neutron star.

% \todo[inline]{Pacciani includes some more detailed considerations about the Chandrasekhar mass.}

% \todo[inline]{Something peculiar about the Chandrasekhar mass is that its derivation does not address the actual mechanism of the collapse (which is electron capture by nuclei, I think). If I understand correctly, this is because as \(M \to M _{\text{Ch}}\) the core density diverges, so something or other is \emph{bound} to happen, be it electron capture, the formation of an event horizon\dots}

\end{document}


\section{Problem and Approach}
\label{sec:description}

Here we will give a specification of the problem, and a detailed approach (i.e. the design of solution).
In order to simplify the problem and keep MLPs as simple as possible, one MLP will be used to check only the graphs (models) with a certain number of nodes and certain types of edges.
In other words, each type of graph will have its own MLP.
Notice that all the MLPs will have the same overall architecture.
The only difference will be the number of nodes at each layer.
That is only the bisimulation between graphs with the same scale will be studied.
The problem of the project become that if an MLP is possible to compute the bisimulation equivalence of two same scale models and how well it can do.
The basic steps of approach are as follows.
\begin{itemize}
    \item Generate a standard distinguishing algorithm.
    \item Based on the standard algorithm, develop a dataset generator.
    \item Construct MLPs that can accept graphs pairs and output judgement.
    \item Do experiments on the performance of the MLP with different training sets. \footnote{N.B. this step will be stated independently in Section \ref{sec:experiment}}
\end{itemize}

\subsection{Standard Bisimulation Algorithm}
For the first step, we will directly use the algorithm that solves the \emph{relational coarsest partition problem} \cite{Paige1987}, i.e. given the relation $E$ and initial partition $P$ over a set $U$ find the partition $P$ that \textquotedblleft every other stable partition is a refinement of it \textquotedblright.
From the perspective of set-theory, it is the same as solving bisimulation equivalence \cite{Dovier2004}.
Namely, if each block of the relational coarsest partition of the union of two graphs has nodes from both multi-directed graphs, these two graphs are bisimilar (see the set-perspective definition in Section \ref{sec:background}).
Moreover, compared with the algorithm given by Kanellakis and Smolka \cite{Milner1980}, their algorithm is more efficient in time (see Section \ref{sec:bac:bis}).
Yet, the algorithm given by Paige and Tarjan is not designed for multi-directed graphs.
It only processes the directed graph with only one kind of edges.
So by replacing every operation on one relation by a group of operations on each relation (see Step \ref{each_relation} in Algorithm \ref{alg:sbs}), the algorithm can be used for multi-directed graphs.
Here we give the description of the algorithm based on \cite{Paige1987}.

\begin{algor}\label{alg:sbs}
Standard Bisimulation Algorithm
\begin{enumerate}
    \item Get the union of two given graphs $U$.
    \item Initialise the partition $X$ and block set $C$, where $X=C=\{U\}$.
    \item Get the initial partition $Q$, by refine the only block $U$ in $Q$ with respect to $U$ itself. In other words, split $U$ on the preimage of itself, i.e. $Q = \{E^{-1}(U), U-E^{-1}(U)\}$, where $E$ is the any type of binary relation of nodes (also edges).
    \item Loop until block set $C$ is empty.
    \begin{enumerate}
        \item \label{loop} Pop the first block $s$ from block set $C$
        \item Check first two blocks of partition $Q$, that is contained in block $s$, make block $b$ be the smaller one.
        \item In partition $X$ split the block $s$ into block $b$ and block $s'=s-b$. if $s'$ is compound with respect to partition $Q$, add it back to block set $C$
        \item \label{each_relation} For each type of relation $E_i$, i.e. $i=1, i=2$ if there are two types of edges.
        Refine every blocks in partition $Q$ with respect to block $b$ and block $s'$.
        \item Add all blocks $x\in X$ that are compound with respect to partition $Q$ to block set $C$
    \end{enumerate}
    \item If there are nodes of both graphs in each block of partition $Q$ return true, else return false.
\end{enumerate}

\end{algor}

% \begin{defin}
% Concepts used in the algorithm. \footnote{All the definition here directly from \cite{Paige1987}}

% \begin{itemize}
%     \item $U$ is the union of two given graphs.
%     \item $S$ is the subset of $U$, i.e. $S \subseteq U$
%     \item $E$ is the links of $U$, which means edge $\langle x, y \rangle \in E $ (also denoted $xEy$).
%     \item For any subset $S$, $E(S) = \{y|\exists x \in S \text{ such that } xEy\}$ and $E^{-1}(S)=\{x|\exists y\in S \text{ such that } xEy \}$
%     \item If $B \subseteq U$, $B$ is \emph{stable} with respect to $S$ if either $S\subseteq E^{-1}(S)$ or $B\cap E^{-1}(S)=\emptyset$
%     \item If $P$ is a partition of $U$, $P$ is \emph{stable} with the respect to $S$ if all of the blocks belonging to $P$ are stable with respect to $S$.
%     \item $P$ is \emph{stable} if it is stable with respect to each of its own block.
% \end{itemize}
% \end{defin}
% This algorithm keep refine the a partition $Q$ (with the initial state that only contain $U$) until it is \emph{stable}.

\subsection{Test Case Generator}\label{sec:generator ds}
All the data used in this project is self-generated random and abstract without any human participation.
Here the model (multi-directed graph) generated will be represented as an expend adjacency matrix (see Figure \ref{fig:exp:model_represent}), where 1 stands for existence.
\begin{figure}[h]
    \centering
    \subfigure[Model]{
        \label{fig:example1_model}
        \includegraphics[width=2.0cm]{img/graph_example.pdf}}
    \subfigure[Adjacency Matrix]{
        \label{fig:example1_matrix}
        \includegraphics[width=0.6\textwidth]{img/example1_matrix.pdf}}
    \caption{Example model represent}
    \label{fig:exp:model_represent}
\end{figure}

There are two generators implemented.
However, only one is used in the final experiments (to be explained in Section \ref{sec:relisation})
The first idea is to randomly generate graphs.
Depends on the rate given, generate a bisimilar graph of another random non-bisimilar graph.
Then record them into the appointed file with flags.
Description of this algorithm is given as follow.
\begin{algor}
\label{alg:R_tcg}
Random Test Case Generator
\begin{enumerate}
    \item \label{item:gen_ran}Generate a random graph with certain scale, using the given density times a random number ($< 1$, $> 0$) as a probability of each possible edge.
    \item According given rate of positive cases and negative cases, decide to generate a bisimilar graph (jump to Step \ref{item:bi}) or non-bisimilar graph (jump to Step \ref{item:non_bi}).
    \item \label{item:non_bi} Keep generate random graph like Step \ref{item:gen_ran} and check the bisimulation equivence, until they are not (then jump to Step \ref{item:convert}).

    \item \label{item:bi}Get the \emph{relational coarsest partition} $P_\text{origin}$ of given graph.
    \item Randomly divides the nodes of the new graph into the same partition $P_\text{similar}$, i.e. the partition that has the same number of blocks (N.B. the length of each block not necessary to be same).
    \item Assemble the minimum bisimilar graph, where each block of its partition contains only one node.
    \item \label{item:bi_end}For each edge of the minimum graph, generate a random number ($\geq 1$) of edges between the random nodes from two corresponding blocks of $P_\text{similar}$ respectively.

    \item \label{item:convert}Convert the new generated graph into adjacency matrix.
    \item Write two graphs with flag into file (jump to Step \ref{item:gen_ran} if there are not enough test cases).
\end{enumerate}
\end{algor}

However, the test cases generated may not actually random, which may have some superficial features that are easy to be caught by the later MLP (to be described in Section \ref{sec:des:ml}).
For example the heuristic algorithm for generate bisimilar graph (see Algorithm \ref{alg:R_tcg}, Step \ref{item:bi} to Step \ref{item:bi_end}) may have some kind of inclination.
To get rid of these potential bias, a full case generator is designed.
This generator can generate all possible graphs in given scale and record all possible combinations with flags.
The basic idea of iteration is that since a graph can be seen as an adjacency matrix, it can be represented as a binary string.
By counting in binary, all possible graphs on the same scale can be traversed.
Here gives the description of this algorithm.
\begin{algor}
\label{alg:F_tcg}
Full Test Case Generator
\begin{enumerate}
    \item For every graph, calculate its minimum bisimilar graph. Meanwhile, construct a dictionary where the key is the minimum bisimilar graph and the value is a set of all graphs that are bisimilar, i.e. have the same minimum bisimilar graph which is their key.
    \item According to the dictionary, catalogue all the graph (graphs with the same key will have the same label).
    \item Go through all graphs again, generate all combinations with flag base on their label.
\end{enumerate}
\end{algor}

\subsection{MLP Structure}
\label{sec:des:ml}
After the test case generator produces data, the network should be trained.
The structure is designed to be as simple as possible.
Combine with the process step of the standard algorithm, the network is structured to meet the process (see Figure \ref{fig:mlp_model}).
The first layer is the input layer, which will accept fixed length binary (i.e. adjacency matrix).
In order to push the network to get the abstraction, the second layer called re-represent layer has fewer neurons than the input layer.
And the first two layers are divided into two independent parts that share parameters (see Part 1 and Part 2 in Figure \ref{fig:mlp_model}).
Each part will accept one graph.
The distinguishing part will accept input from two graphs and get the conclusion about bisimulation equivalence, i.e. output the probability of positive and negative.
There are three reasons to design a network like that.
First is by sharing the parameters, the overall training will be master.
The second reason is that it solves the problem of sequence, i.e. the output will not be affected by the priority of two graphs.
The third reason is that this kind of modular design will be much easier for reuse, e.g. used as part of the graph simplify the network.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.65\textwidth]{img/mlp.pdf}
    \caption{MLP model}
    \label{fig:mlp_model}
\end{figure}

\subsection{General Structure}
The design of the general structure is aimed at developing a reusable tool.
Thus the structure is very simple and highly modularised (see Figure \ref{fig:general_str}).
Dataset generator and ML-algorithm is packed independently.
And each module can be called in a python program or used as a command line tool directly by the user.
\begin{figure}[h]
    \centering
    \includegraphics[width=\textwidth]{img/architecture.pdf}
    \caption{General structure}
    \label{fig:general_str}
\end{figure}


\par
\section{Data Structure}
\par
The {\tt Coords} object has four fields.
\begin{itemize}
\item {\tt int type} : coordinate type.
When {\tt type = 1}, coordinates are stored by tuples,
$(x_0,y_0,\ldots)$ first,
$(x_1,y_1,\ldots)$ next, etc.
% \begin{verbatim}
% x(icoor, idim) = coors[idim + icoor*ndim]
% \end{verbatim}
When {\tt type = 2}, coordinates are stored by $x$-coordinates
first, $y$-coordinates next, etc.
% \begin{verbatim}
% x(icoor, idim) = coors[icoor + idim*ncoor]
% \end{verbatim}
\item {\tt int ndim} :
number of dimensions for the coordinates,
e.g., for $(x,y)$ coordinates {\tt ndim = 2},
for $(x,y,z)$ coordinates {\tt ndim = 3}.
\item {\tt int ncoor} :
number of coordinates (i.e., number of grid points).
\item {\tt float *coors} :
pointer to a {\tt float} vector that holds the coordinates
\end{itemize}
A correctly initialized and nontrivial {\tt Coords} object
will have {\tt type} be {\tt 1} or {\tt 2},
positive {\tt ndim} and {\tt ncoor} values,
and a non-{\tt NULL} {\tt coors} field.


\documentstyle[11pt,reduce]{article}
\title{{\tt meijerG}, a package for simplification\\
of Meijer's G function}
\date{}
\author{Victor S. Adamchik\\
	Wolfram Research Inc. \\
	former address : \\
	Byelorussian University, Minsk, Byelorussia\\
\\
\\
	Present \REDUCE{} form by \\
	Winfried Neun \\
	ZIB Berlin \\
        Email: {\tt Neun@sc.ZIB-Berlin.de}}
\begin{document}
\maketitle

This note describes the {\tt meijerG} package of \REDUCE{}, which is able
to do simplification of several cases of Meijer's G function.
The simplifications are performed towards polynomials, elementary or
special functions or (generalized) hypergeometric functions.
Therefore this package should be used together with the \REDUCE{}
special function and hypergeometric (ghyper) package.

\section{Introduction}

The function

\begin{displaymath}
G_{p q}^{m n} \left( z \  \Bigg\vert \  {(a_p) \atop (b_q)} \right)
\end{displaymath}

has been studied by C.~S.~Meijer beginning in 1936 and has been
called Meijer's G function later on. The complete definition of Meijer's
G function can be found in \cite{Prudnikov:90}.
Many well-known functions can be written as G functions,
e.g. exponentials, logarithms, trigonometric functions, Bessel functions
and hypergeometric functions.

Several hundreds of particular values can be found in \cite{Prudnikov:90}.

\section{\REDUCE{} operator {\tt meijerg}}

The operator {\tt meijerg} expects 3 arguments, namely the
list of upper parameters (which may be empty), the list of lower
parameters (which may be empty too), and the argument.

The first element of the lists has to be the list of the
first n or m respective parameters, e.g. to describe
\begin{displaymath}
G_{1 1}^{1 0} \left( x \  \Bigg\vert \  {1 \atop 0} \right)
\end{displaymath}

one has to write
\begin{verbatim}

MeijerG({{},1},{{0}},x); % and the result is:

 HEAVISIDE( - X + 1)
---------------------
      GAMMA(1)

\end{verbatim}
and for
\begin{displaymath}
G_{0 2}^{1 0} \left( \frac{x^2}{4} \  \Bigg\vert \ {} \atop  {1+ \frac{1}{4} }
{1-\frac{1}{4}} \right)
\end{displaymath}
\begin{verbatim}

MeijerG({{}},{{1+1/4},1-1/4},(x^2)/4) * sqrt pi;

                   1      2
 SQRT(PI)*BESSELJ(---,X)*X
                   2
----------------------------
             4

\end{verbatim}

Note: Using the special function package these results will be
simplified further.

\begin{thebibliography}{9}

\bibitem{Prudnikov:90} A.~P.~Prudnikov, Yu.~A.~Brychkov, O.~I.~Marichev,
{\em Integrals and Series, Volume 3: More special functions},
Gordon and Breach Science Publishers (1990).

\end{thebibliography}
\end{document}