\chapter{Method}\label{ch:method}

As stated in the introduction, this thesis examines a way to train a machine learning classifier to predict the nominal stock price change.
This chapter will now introduce the classifiers that were trained, as well as the approach to represent the business reports as feature vectors using \acs{BERT}.

\section{Overview of machine learning algorithms}
All these algorithms have in common, that they take a $n$-dimensional input vector to predict $m$ classes.
Mathematically this can be seen as a function $f(x)=y$ with $x\in\mathbb{R}^n$ and $y\in\{0, 1\}$, since it is a binary classification problem.
The algorithms are able to find patterns in the data and can autonomously improve their prediction performance.
This learning process requires a training phase.
During the research a Naive Bayes classifier, a \acl{KNN} classifier, a \acl{SVM} and a \acl{BLSTM} was trained.
\subsection{Naive Bayes classifiers}
The idea behind the Naives Bayes classifier is, that certain input features correlate with a certain class, which means that these input features result with a high probability from this certain class.
The Naive Bayes classifier makes use of the assumption, that each feature of the input vector is independent to one another.
Using the Bayes theorem (equation \ref{equation:bayes}), the Naive Bayes classifier can compute the probability of a class given the features of the input.
\begin{equation}
    P(Y|X) = \frac{P(X|Y)P(Y)}{P(X)}
    \label{equation:bayes}
\end{equation}
The probability $P(Y)$ is given by the proportion of a class in the whole training dataset.
For a balanced dataset with two classes this gives the probabilities $P(y_1, y_2)=\langle0.5,0.5\rangle$.
Determining $P(X|Y)$ is not as trivial:
Since $X$ can be a vector $(x_1, x_2, ..., x_n)$ of variables, it would be necessary to determine $P(x_1 \land x_2 \land ... \land x_n|Y)$ for all combinations of $x_i$, which would require $O(2^n)$ space to save the values and make it impossible to use the algorithm with high dimensional input \cite[p. 493]{Russel2016}.
The assumption that $x_1, ..., x_n$ are independent, however, simplifies this problem.
$P(x_1 \land x_2 \land ... \land x_n|Y)$ can now be decomposed into $P(x_1|Y)\cdot P(x_2|Y)\cdot...\cdot P(x_n|Y)$ which only requires $O(n)$ to save the conditional probabilities and allows for practical usage of the algorithm \cite[p. 499]{Russel2016}.
So during training the algorithm has to determine $P(Y)$ and $P(x_1|Y), P(x_2|Y) ... P(x_n|Y)$.
The probability $P(X)$ is a normalization factor which is chosen so that $P(y_1|X)+P(y_2|X)=1$ \cite[p. 493]{Russel2016}.
During prediction phase $P(Y|X)$ is computed for all classes and the class with the highest probability given $X$ will be the classification result:
\begin{equation}
    y_{pred} = arg max_i P(y_i|X)
\end{equation}
Even though the assumption, that the features are independent to one another is generally not true, the Naive Bayes classifier often performs equally or even better than more complex methods \cite[p. 211]{Hastie2009}.
For this research, the Naive Bayes implementation from scikit-learn was used.

\subsection{\acl{KNN} classifiers}
As the name of this method implies, a nearest neighbor classifier makes its predictions based on the surrounding data points.
Because every input to this classifier can be seen as a point in a n-dimensional space, the \ac{KNN} can compute the distances to other points.
The distance can be measured as Euclidean distance, however other distance metrics are also possible.
Scikit-learn's \texttt{KNeighborsClassifier} for example, measures the distance with the Minkowski distance, which is a generalisation of the Euclidean distance\footnote{\url{https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html}}:
\begin{equation}
    d(x_1, x_2) = \left(\sum_{i=1}^n \abs{x_{1,i} - x_{2,i}} ^p \right)^{\frac{1}{p}}
\end{equation}
The Minkowski distance collapses to the Euclidean distance for $p=2$ and to the Manhatten distance for $p=1$ \cite[p. 738]{Russel2016}.
The predicted class is then the most common class of all $k$ neighbors.
Because the \ac{KNN} classifier makes use of a geometric property of the input data, it is important to scale each feature to have a mean of zero and a variance of one \cite[p. 465]{Hastie2009}.

In this thesis the \ac{KNN} implementation from scikit-learn was used as well as the \texttt{StandardScaler} to scale the features.
$k$ was determined by creating a validation curve (section \ref{subsec:learning_validation_curve}), which shows how the accuracy changes for different values of $k$.
It is implemented with the code in listing \ref{py:knn_validation_curve}.
The results of this finding are presented in chapter \ref{ch:experiments}.
\lstinputlisting[language=Python,caption={Creates data for a validation curve},captionpos=b, label={py:knn_validation_curve}]{listings/knn_validation.py}
The \texttt{knn\_\_n\_neighbors} argument tells the \texttt{validation\_curve} function, which parameter it should test for the provided estimator.
In this case the estimator is a pipeline, which wraps the \texttt{StandardScaler} and the \texttt{KNeighborsClassifier} and executes them sequentially.
The \texttt{validation\_curve} also allows to use cross-validation, where the number of folds can be set with the \texttt{cv} argument.
Cross-validation is explained in detail in section \ref{subsec:cross_validation}.

\subsection{\acl{SVM}s}
A \ac{SVM} tries to fit a decision boundary between the points of two classes, so that the surrounding margin is maximised.
This decision boundary can mathematically be seen as a hyperplane in the n dimensional space of the data points.
For a two dimensional space that results in a line:
\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/svm_decision_boundary.png}
    \caption{A decision boundary with its support vectors.\protect\footnotemark}
    \label{figure:svm_decision_boundary}
\end{figure}
\footnotetext{A modified version of the example on scikit-learn: \url{https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane.html}}
The line in figure \ref{figure:svm_decision_boundary} represents the decision boundary and the dashed lines mark the borders of the margin with maximum width, respecting some constraints which will be discussed in this section.
The circled points mark the \textit{support vectors} which are used to determine the hyperplane.
The decision boundary is mathematically a hyperplane defined by:
\begin{equation}
    f(x)=x^T\beta + \beta_0=0
    \label{equation:svm_hyperplane}
\end{equation}
where $\beta$ is a unit vector, $x_i \in \mathbb{R}^n$ are the sample points and $y_i \in \{-1, 1\}$ are the labels to these points.
Therefore the aim is to maximise the margin between the training points and the hyperplane, which can be re-expressed as
\begin{align}
    & min_{\beta, \beta_0}\frac{1}{2}\abs{\abs{\beta}}^2 + C\sum_{i=1}^N\xi_i \\
\text{subject to }
    & \xi_i \geq 0, y_i(x_i^T\beta + \beta_0)\geq 1 - \xi_i \forall i \notag
\end{align}
where $\xi_i$ is a penalty term, that increases with the distance a data point $x_i$ lies on the wrong side of its margin \cite[p. 7]{Fletcher2008}.
Since this penalty terms are weighted by the constant $C$, the width of the margin varies with choosing the value of $C$.
If $C$ is set really low, a broad margin with many support vectors is used.
If it is set high, only points near the decision boundary are used as support vectors \cite[p.421]{Hastie2009}.
This optimization problem can be solved using Lagrange multipliers and gives for $\beta$
\begin{equation}
    \beta = \sum_{i=1}^{N} \alpha_i y_i x_i
\end{equation}
and for $\beta_0$
\begin{equation}
    \beta_0 = \frac{1}{N_S} \sum_{i=1}^{N_S}(y_i - \sum_{j=1}^{N_S} \alpha_j y_J x_i^T x_j)
\end{equation}
$\alpha$ is the Lagrange multiplier which is $0 < \alpha \leq C$ for all \textit{support vectors} and $0$ for all other points \cite[p. 746]{Russel2016}.
Its exact value can be determined with the dual Lagrangian function of equation \ref{equation:svm_hyperplane} \cite[p. 9]{Fletcher2008}.

The \ac{SVM} as defined so far, has the drawback, that it only creates linear decision boundaries.
Often the classes are not linearly seperable, so a mathematical trick is used, that the \ac{SVM} can still perform on such data.
This is done with a kernel function, which maps the data points to a enlarged feature space.
% $\xi_i$ are the \textit{support vectors}.
% The support vectors are the points that are used to build up the decision boundary, so that it has a maximum distance to them.
% If the classes are linearly seperable, these are the closest points of one class to the other class.
% If the data points of both classes overlap, the support vectors are constructed from the points of one class that lie in the decision region of the other class.
% $C$ is the parameter which sets the width of the margin, if it is set really low, a broad margin with many support vectors is used.
% If it is set high, only points near the decision boundary are used as support vectors.
A way to find decision boundaries for data which is not linearly seperable is to enlarge the feature space with a kernel function.
Thus the data becomes linearly seperable in the enlarged feature spaces, which has the effect of nonlinear boundaries in the original space \cite[p.423]{Hastie2009}.
One typical kernel function is the \textit{Radial Basis Kernel} which has the following form:
\begin{equation}
    K(x_i, x_j) = e^{-\frac{\abs{\abs{x_i - x_j}}^2}{2\sigma^2}}
\end{equation}
The kernel function is used in the dual Lagrangian function of equation \ref{equation:svm_hyperplane}, which is then again optimised.
This has the implicit effect, that the datapoints are mapped to a higher dimensional feature space.
Figure \ref{figure:kernel_3d} shows this mapping to a three-dimensional space with the hyperplane, that seperates the classes, figure \ref{figure:kernel_decision_boundary} shows the base feature space with the resulting circular decision boundary.
\begin{figure}[h]
    % \centering
    \subcaptionbox{Datapoints with a circular decision boundary, resulting from the kernel trick.\label{figure:kernel_decision_boundary}}[0.5\linewidth]{\includegraphics[width=0.5\textwidth]{figures/svm_circular_decision_1.png}}
    \subcaptionbox{When mapped to a higher dimensional space, the datapoints become linearly seperable.\label{figure:kernel_3d}}[0.5\linewidth]{\includegraphics[width=0.5\textwidth]{figures/svm_circular_decision_2.png}}
    % \begin{subfigure}{0.5\textwidth}
    %     \centering
    %     \includegraphics[width=\textwidth]{figures/svm_circular_decision_1.png}
    %     \caption{Datapoints with a circular decision boundary, resulting from the kernel trick.}
    %     \label{figure:kernel_decision_boundary}
    % \end{subfigure}
    % \begin{subfigure}{0.5\textwidth}
    %     \centering
    %     \includegraphics[width=\textwidth]{figures/svm_circular_decision_2.png}
    %     \caption{When mapped to a higher dimensional space, the datapoints become linearly seperable.}
    %     \label{figure:kernel_3d}
    % \end{subfigure}
    \caption{The kernel trick as visualised in \cite[p. 747]{Russel2016}}
    \label{figure:svm_kernel_trick}
\end{figure}

\subsection{Artificial Neural Networks}
Artificial neural networks imitate the structure of a brain, where each neuron receives input from other neurons, processes this information and passes it on to other neurons \cite[p. 727]{Russel2016}.
In artificial neural networks these neurons are represented as \textit{units}, which perform a mathematical operation on their input data.
This consists of first computing a weighted sum of its inputs and then applying a activation function $g$ on that sum, which determines whether the unit "fires" \cite[pp. 727-728]{Russel2016}:
\begin{equation}
    a_j=g\left(\sum_{i=0}^{n}w_{i,j}a_{i}\right)
    \label{equation:nn_unit_operation}
\end{equation}
Where $w_{i,j}$ is the weight, assigned to the input $a_i$.
A typical activation function is the sigmoid function defined by $\sigma(x)=\frac{1}{1+e^{-x}}$.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/sigmoid.png}
    \caption{The sigmoid function.}
    \label{figure:sigmoid}
\end{figure}
As it can be seen in figure \ref{figure:sigmoid}, the sigmoid function results in values in the range from 0 to 1, which can be interpreted as the probability that the input belongs to one class.
The units are then structured into layers, with the simplest possible network having only two layers, one input and one output layer.
These simple networks are limited in their learning capabilities, that is why usually several hidden layers are added \cite[pp. 730-731]{Russel2016}.
% The input $a_j$ to a unit in such a hidden layer is itself the result of the computation with the function \ref{equation:nn_unit_operation}, therefore the whole network can mathematically be seen as a function applied to the input vector.
The network network predicts values by computing the output of each unit with function \ref{equation:nn_unit_operation} starting from the input layer.
The nodes in the subsequent layers use the same function to compute their outputs, however the input $a_i$ is the result of the computations on the previous layer.
The results of this computation on the output layer is the prediction of the network.
At training time, the units of the output layer then compute an error between the prediction it made and the actual value it should have predicted.
A common error metric is the squared-error loss $Loss_k$ on an output node $k$, which is defined in equation \ref{equation:squared_error_loss} \cite[p. 735]{Russel2016}.
\begin{equation}
    Loss_k=(y_k - a_k)^2
    \label{equation:squared_error_loss}
\end{equation}
With $a_k$ being the predicted value on output $k$ and $y_k$ being the actual value that should be predicted.
The aim is to minimise this loss with respect to all weights $w_{i,j}$ in the network:
\begin{equation}
    \frac{\partial Loss_k}{\partial w_{i,j}}
\end{equation}
This gives the gradient for the output node $k$.
The gradient can be computed with the \textit{back-propagation} algorithm.
The last layer computes an error term, based on the derivative of the loss function.
This error term is back-propagated to the previous layer, where each node again computes its own error, by summing the weighted errors of the subsequent nodes.
This is again repeated until the input layer is reached, then the weights between all nodes are updated based on the determined error terms \cite[p. 734]{Russel2016}.
% Auf nichtlinearität eingehen

A special class of neural networks are the \acp{RNN}, which have the ability to gather information from the order of the input items.
Therefore each input example consists of several \textit{timesteps}.
In case of report classification, like it is done in this thesis, the timesteps are the sentences of the report, which have a clear order to each other.
To process this sequential data, the units in a \acs{RNN} (figure \ref{figure:rnn_unfolded}) are different to the units of a traditional feed-forward neural network.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/rnn_unfolded.png}
    \caption{The flow of the activations between two timesteps in an RNN as depicted in \cite[p. 12]{Lipton2015}.}
    \label{figure:rnn_unfolded}
\end{figure}
They have one input from the previous layer and a second one from the previous timestep.
On the output side they have an output for the next timestep and optionally also an output to the next layer \cite[p. 373]{Goodfellow2016}.
The training is done by computing the loss in the last timestep and propagating the loss back to the previous timestep.
A problem that occurs in standard \acp{RNN} is that the gradient decayes or increases exponentially while propagated back in time \cite[p. 13]{Lipton2015}.
A concept to avoid this problem is the \acs{LSTM}, which was proposed by \cite{Hochreiter1997}.
Figure \ref{figure:lstm_cell} shows the structure of a LSTM cell.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/lstm_cell.png}
    \caption{A LSTM cell as depicted in \cite[p. 17]{Lipton2015}.}
    \label{figure:lstm_cell}
\end{figure}
Each of these units has an input gate which protects the unit against irrelevant input ($i_c$ in figure \ref{figure:lstm_cell}) and an output gate $o_c$ which protects other units from possibly irrelevant content in the respective unit \cite[p. 6]{Hochreiter1997}.
Both, the \ac{RNN} as well as the \ac{LSTM}, have forms that also process the input sequence in the reverse order; the \ac{LSTM} is then called \ac{BLSTM}.

In this research the \ac{LSTM} implementation from Keras was used.
Although different network architectures were used, the general implementation with all used layer types is shown in listing \ref{py:lstm_implementation}.
\lstinputlisting[language=Python,caption={Implementation of a \acs{LSTM} in Keras.},label={py:lstm_implementation}]{listings/lstm.py}
The \texttt{LSTM} layers in listing \ref{py:lstm_implementation} consist of the just introduced \ac{LSTM} cells.
With the \texttt{return\_sequences} parameter the hidden states of all \ac{LSTM} cells on the respective layer can be used and the \texttt{go\_backwards} parameter reverses the order in which the sequential data is processed\footnote{\url{https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/layers/LSTM}}.
\texttt{Bidirectional} is a wrapper for a forward processing \ac{LSTM} layer and a backward processing \ac{LSTM} layer to create a \ac{BLSTM}\footnote{\url{https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/layers/Bidirectional}}.
With the \texttt{GlobalAveragePooling1D} layer the hidden states from the previous \ac{BLSTM} layer are averaged across the timesteps\footnote{\url{https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/layers/GlobalAveragePooling1D}}.
The \texttt{Dense} layer then is a layer with the regular feed forward units described at the beginning of this section \footnote{\url{https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/layers/Dense}}.

A key aspect is the usage of \texttt{None} in the first dimension of the \texttt{input\_shape} on line 11 in listing \ref{py:lstm_implementation}.
This dimension defines the number of time steps each input sample has.
Since each report has a different length, this value was set to \texttt{None}.
Even though the \ac{LSTM} can handle input with a different number of time steps, on batch level it still requires the input to have a fixed number of time steps.
Therefore the reports need to be grouped by their lengths, which was accomplished with the \texttt{ReportSequence} class (Listing \ref{py:lstm_report_sequence}).
\lstinputlisting[language=Python,caption={The \texttt{ReportSequence} class creates batches with reports of same length.},label={py:lstm_report_sequence}]{listings/report_sequence.py}
This grouping is then used by calling the \texttt{fit\_generator} method on the model with an instance of the \texttt{ReportSequence} class in listing \ref{py:lstm_implementation} on line 26.
Similar to the previously described classifiers, a 5 fold cross-validation was used to select the best model architecture.

Now that the classification algorithms are introduced, one problem still remains:
The classifiers require the input to be a numerical vector.
Therefore the reports have to be converted to feature vectors.
This embedding step is described in the next section.

\section{Creating document embeddings}
\label{sec:creating_document_embeddings}
\subsection{Methods to create a document representation}
\label{subsec:methods_for_document_representation}
The task of the embedding step is to map a text of variable length to a vector of fixed length, which is necessary to train a classification algorithm.
The most basic approach for creating a text representation is the bag-of-words model.
For that, a vocabulary is created which contains each word of a text corpus.
Each word in this vocabulary is represented by a vector with the dimensionality of the vocabulary size, having a $1$ on the entry at which the word is stored in the vocabulary and zeros on all other positions.
To create the representation of a document, the vectors of the words in the document are summed \cite[p. 13]{Grzegorczyk2018}.
Since a single document typically does not contain all words of the vocabulary this vector is sparse and an inefficient representation of the text.
This method also does not take into account that some words occur more frequently than others, which can be solved by mutiplying the term frequency with the inverse document frequency.
The resulting metric is called \textit{TF-IDF}:
\begin{equation}
    TF-IDF(t,d) = f_{t,d} \cdot log \frac{N}{n(t)}
    \label{equation:tf_idf}
\end{equation}
Where $f_{t,d}$ is the number of occurences of a term $t$ in a document $d$, $N$ the total number of documents and $n(t)$ the number of documents that contain the term $t$ \cite[p. 13]{Grzegorczyk2018}.

Another drawback of the word vectors in the bag-of-words model is, that it is not possible to determine similarity between different words.
Since they are all orthogonal to each other, "powerful", "strong" and "Paris" share the same similarity with each other \cite[p. 1]{Mikolov2014}.
This problem can be solved by using distributed vector representations, which can be created with the popular \textit{Word2Vec} implementation.
Word2Vec is a simple neural network with one hidden layer \cite[p. 22]{Grzegorczyk2018}.
It offers two algorithms to learn the word representations.
With the \ac{CBOW} algorithm the network predicts a word based on its surrounding words and with the \textit{skip-gram} algorithm the network tries to predict context words based on a center word \cite[pp. 4-5]{Mikolov2013}.
The word embeddings are then a result of maximizing the probabilities of these predictions \cite[p. 23]{Grzegorczyk2018}.
A simple approach for creating a document representation of these word embeddings would be to average all word vectors of the document.
Similar to the bag-of-words model with the term frequencies, this approach also does not respect the order of the words.
An extension to Word2Vec that creates a document representation is the \textit{Paragraph Vector} model \cite[p. 1]{Mikolov2014}, which is better known as the implementation \textit{Doc2Vec}.
For this model an additional paragraph representation is trained in parallel to the word representations \cite[p. 3]{Mikolov2014}.
The paragraph vector is incorporated in all word prediction contexts, therefore it captures the meaning of the whole document \cite[p. 3]{Mikolov2014}.

At the time of publication, \textit{Paragraph Vector} created state-of-the-art results in different sentiment tasks \cite[pp. 4-7]{Mikolov2014}, this thesis however, will use the modern language representation model \ac{BERT} to create the document embeddings.
In comparison to \textit{Paragraph Vector}, \ac{BERT} can use distant context words to the same extend as close words to create an embedding, whereas for the \textit{Paragraph Vector} model, distant words contribute less to the embedding of a center word \cite[p. 4]{Mikolov2013}.
The ability to incorporate these long range relations is a result of the Transformer architecture, which was proposed in \cite{Vaswani2017} and is the basis of \ac{BERT}.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/transformer_architecture.png}
    \caption{The Transformer architecture as proposed in \cite[p. 3]{Vaswani2017}.}
    \label{figure:transformer_architecture}
\end{figure}
From the pure application area, Transformer is really similar to \acp{RNN} or \acp{LSTM}, as it is also optimised for handling sequential data.
It uses an encoder-decoder structure with a self-attention mechanism.
The encoder maps an input sequence to a vector representation, which the decoder uses to create an output sequence.
This can be used for example for translation tasks, where a sentence from the original language is passed to the encoder and the decoder returns the translated sentence \cite[p. 392]{Goodfellow2016}.
Traditional encoder-decoder models use \acp{RNN} or \acp{LSTM} for the encoder and decoder, with the vector representation being the final hidden state of the encoder \cite[p. 392]{Goodfellow2016}.
Enhancing these models with Attention, the hidden states of all encoder units can be used, with the focus on the most relevant states.
The Transformer architecture now relies purely on Attention and the sequential information is manually encoded in the data, which is labeled as \textit{Positional Encoding} in figure \ref{figure:transformer_architecture}.
Attention is accomplished by computing a weighted sum of the hidden states.
The weights are given by the similarity between the hidden states of the encoder, which are called the \textit{keys} and the most recent hidden state of the decoder, called the \textit{queries}.
Both, the \textit{keys} and the \textit{queries}, are matrices and the similarity between them is calculated as a multiplication, on which a softmax function is applied afterwards \cite[p. 4]{Vaswani2017}.
Between these weights and the \textit{values}, which are the hidden states of the encoder as well, another matrix multiplication is calculated, giving the weighted sum.
In the Transformer model, this Attention mechanism is combined to a \textit{multi-head attention}.
The multi-head attention consists of multiple attention layers, each performing the attention on different linear projections of the query, key and value inputs and concatenating the results in the end \cite[pp. 4-5]{Vaswani2017}.
The advantage of the Transformer model is a high parallelization of the computation and the already mentioned capability to capture long range relations \cite[p. 2]{Vaswani2017}.
For \ac{BERT} the Transformer blocks depicted in figure \ref{figure:transformer_architecture} are stacked in multiple layers, with 12 layers in case of BERT\textsubscript{BASE} and 24 layers in case of BERT\textsubscript{LARGE} \cite[p. 3]{Devlin2018}.
Because of resource limitations, the BERT\textsubscript{BASE} model is used in this research.

Another important property of \ac{BERT} that can't be ignored, is that it can only process input text with a maximum sequence length of 512 tokens.
\textit{Tokens} are the entries in the vocabulary of \ac{BERT}.
Before \ac{BERT} can process the input text, it has to be tokenized.
In this step, each word from the input sequence is replaced with the longest token in the \ac{BERT} vocabulary, that matches the word.
If no token matches the word exactly, the word is split in multiple tokens.
For example, the word \textit{discounted} is not in the vocab and the tokenizer therefore splits the word into the sub tokens \textit{discount} and \textit{\#\#ed}.
The restriction to a maximum sequence length of 512 tokens implies, that it is not possible to pass a whole business report to \ac{BERT}.
The segmentation into sentences with spaCy, that was described in section \ref{sec:data_preprocessing}, is part of solving this problem, however in some cases the resulting sentences are way shorter than the sequence length of 512 tokens.
Therefore the sentences are again concatenated until the maximum sequence length is reached.
For most applications working with \ac{BERT} involves two steps, a pre-training and a fine-tuning step.
Since \ac{BERT} is used to create the text embeddings, an additional feature extracting step was added which is described in section \ref{subsec:extracting_feature_vectors}.

\subsection{Pre-training}
In the \textit{pre-training} step, \ac{BERT} learns the general language structure which includes the positions of different words within a sentence and the relationship between successive sentences.
Since BERT is a language model with a deep architecture, bidirectional training of the word positions is not possible, because the network can trivially predict the target word \cite[p. 4]{Devlin2018}. %(<- Elaborate on here)
Therefore \cite{Devlin2018} decided to randomly mask out a number of words in the sentence and then predict the masked words.
\begin{figure}[h]
    \centering
    \includegraphics[width=1.0\textwidth]{figures/illustration_masked_lm.png}
    \caption{Illustration of the \textit{masked language model} pre-training task.}
    \label{figure:masked_lm}
\end{figure}
The masking of the words is also done by the already introduced tokenizer.
For representing the masked words, the special token \texttt{[MASK]} is used.
Another special token is the \texttt{[CLS]} token, which is prepended to each input sequence and holds the meaning of this sequence after the \textit{fine-tuning} procedure \cite[p. 4]{Devlin2018}.
The whole task of predicting masked words based on their contexts is called \textit{masked language model} \cite[p. 4]{Devlin2018}.
The second task during pre-training is the binary \textit{next sentence prediction}.
For that, \ac{BERT} has to predict whether a sentence \textit{B} is the actual successive sentence to a sentence \textit{A} or whether it s a randomly chosen sentence from the training corpus.

Because \ac{BERT} is using Tensorflow as a backend, it expects the input data to be in the TFRecord file format.
TFRecord is a binary file format, which contains the tokenized text and the tokens that are masked.
Because the creation of each TFRecord file takes a few seconds, the dataset is again split to optimise the processing with multiple cores, as it can be seen in Listing \ref{py:split_data_and_spawn_processes}.
\lstinputlisting[language=Python,caption={The dataset is split to optimise the TFRecord creation on multicore systems.},label={py:split_data_and_spawn_processes}]{listings/split_data_and_spawn_processes.py}
The \texttt{\_handlePretrainingDataCreation()} function, which is provided as as a callback to the \texttt{Process}, is mostly similar to the \texttt{main()} function of \texttt{create\_pretraining\_data.py} from the official \ac{BERT} repository\footnote{The official \ac{BERT} repository on GitHub: \url{https://github.com/google-research/bert}}.
It was modified, so it creates the TFRecord files for multiple input files at once.

The actual training of \ac{BERT} was done with the script in listing \ref{py:run_pretraining}.
\lstinputlisting[language=Python,caption={\ac{BERT} pre-training.},label={py:run_pretraining}]{listings/run_pretraining.py}
Again most of the source code is similar to the \texttt{main()} function of \texttt{run\_pretraining.py} from the official \ac{BERT} repository.
One change that was made, was to set the number of checkpoints, that are created during training, to 30 as opposed to 5 as default.
These checkpoints were later used to evaluate the model.
The validation step was also moved to a seperate script to evaluate the model for every training checkpoint.
This is accomplished by iterating over the created checkpoints and calling the evaluation code from \ac{BERT} for each of them.
In the original source code only the final model was evaluated.
The results are appended to a Pandas DataFrame, which is saved as a \ac{CSV} file in the end.

\subsection{Fine-Tuning}
\label{subsec:fine_tuning}
The \textit{fine-tuning} step is used to optimise \ac{BERT} for a specific task.
This can be a task like question answering or sentence classification \cite[pp. 5-7]{Devlin2018}.
Even though the final goal is to classify the business reports and not every sentence on its own, the fine-tuning is essential so that the \texttt{[CLS]} token holds a good representation of the sentence\footnote{This hint was mentioned in \texttt{extract\_features.py} on lines 237-239 \url{https://github.com/google-research/bert/blob/fe354751d7de010f60d362ae8d9343849ec39456/extract_features.py}}.
Since the final aim is to classify each business report, the most similar task was selected for fine-tuning, which is the sentence classification.
In difference to the pre-training, the sequences for fine-tuning need a label.
Since each input sequence to \ac{BERT} is a part of a larger report, the sequences "inherit" their label from this report.
Therefore every sentence of a \textit{positive} report is also labeled \textit{positive} and every sentence of a \textit{negative} report is also labeled as \textit{negative}.
Of course this approach seems wrong on the first glance, since a positive report can also contain negative sentences and vice versa.
However one could also re-interpret the classification task as follows:
Instead of saying that the label indicates whether a sentence is a \textit{positive} or \textit{negative} sentence, it can also be interpreted as indicating whether the sentence belongs to a \textit{positive} or \textit{negative} report.
The experimental results in section \ref{sec:fine_tuning_results} show how this approach affects fine-tuning.

As opposed to the pre-training, the creation of the TFRecord files was part of the training script for fine-tuning.
Therefore every training run was delayed, because the TFRecord files were generated first.
The call to the \texttt{file\_based\_convert\_examples\_to\_features} function, which creates the TFRecord files was therefore moved to another script (Listing \ref{py:convert_to_tfrecord}).
\lstinputlisting[language=Python,caption={Creates the TFRecord files for fine-tuning.},label={py:convert_to_tfrecord}]{listings/convert_to_tfrecord.py}
This allows to train the model without waiting for the creation of the TFRecord files.
Similar to the changes made for pre-training, the number of checkpoints to save was also increased and the validation of the model was moved to a second script.

\subsection{Extracting feature vectors}
\label{subsec:extracting_feature_vectors}
After the fine-tuning is accomplished, the \texttt{[CLS]} token at the start of each sequence holds information about the class of the sequence.
This information can be retrieved by accessing the hidden states of the Transformer blocks in the last layer of the model.
Listing \ref{py:extract_features} shows a snippet of the \texttt{extract\_features.py} script.
\lstinputlisting[language=Python,caption={Accesses the hidden states from the transformer layers.},label={py:extract_features}]{listings/extract_features.py}
The official implementation allows to access the hidden states from all layers of the model, however for this thesis only the last layer is used.
For that, the \texttt{layer\_indexes} variable which is passed to the \texttt{model\_fn\_builder} is set to -1.
Therefore, the hidden states of this layer can be accessed by \texttt{result["layer\_output\_0"]} on line 21.
In theory, the \texttt{result} dictionary also includes \texttt{layer\_output\_1, layer\_output\_2, ...}, but the \texttt{layer\_indexes} limits the results.
In the list comprehension on line 23 \texttt{layer\_output[0]} only the hidden states for the first token, the \texttt{[CLS]} token, is accessed.
The \texttt{for} loop is necessary because the \texttt{predict} method returns an iterator, where each element represents the hidden states for one line of the input file.
As a summary it can be said, that this implementation only takes the hidden states from the last layer for the first token, while the official implementation takes the hidden states for all tokens from possibly all layers.

Because of the limitation of \ac{BERT} that it can only process sequences with a maximum length of 512 tokens, many embedding vectors result for each report.
For the \ac{BLSTM} the single sequence embeddings give the timesteps for each sample.
The Naive Bayes, \ac{KNN} and \ac{SVM} however, need a single vector representation for each report, therefore the sequence embeddings are averaged per report.

\section{Other used techniques}
\subsection{Cross-Validation}
\label{subsec:cross_validation}
Cross-validation is a method to estimate the prediction accuracy of a model for a unknown dataset \cite[p. 241]{Hastie2009}.
For that a training dataset is split in $K$ equally sized parts.
$K-1$ of these parts are then used to train the model and the $k$th part is used to evaluate the model.
This is repeated for $k=1,2,...,K$ and the resulting accuracies are averaged \cite[p. 242]{Hastie2009}.
The averaged accuracy is used to optimise the model:
The hyperparameters of the model are changed so that the averaged accuracy is maximised \cite[p. 242]{Hastie2009}.
One drawback of the method is that the time to train and evaluate the model increases with a larger $K$ \cite[p. 242]{Hastie2009}.

For this research 5-fold cross-validation was used for the Naive Bayes, \ac{KNN}, \ac{SVM} and the \ac{BLSTM}.
The cross-validation dataset covered 80\% of the whole dataset, therefore for the test dataset remained 20\% of the data.
Both datasets were balanced so that they include the same number of positive and negative reports.
The hyperparameters for the \ac{BERT} pre-training and fine-tuning were determined without cross-validation, because \ac{BERT} itself is quite computational expensive.
Since for \ac{BERT} no cross-validation was used the training dataset covered 60\%, the validation and test dataset both covered 20\% of the data.
Again all datasets were balanced.

\subsection{Learning and validation curve}
\label{subsec:learning_validation_curve}
This thesis uses learning curves to visualise the learning progress of the machine learning models.
A learning curve is a line chart that shows the accuracy of a machine learning model depending on a second variable, like the training dataset size or the number of epochs a neural network trained \cite[pp. 702-703]{Russel2016}.
It can give evidence about whether the model would improve its performance if more data was available or the number of epochs was increased.
Figure \ref{figure:sample_learning_curve} shows a possible learning curve.
In this thesis, the accuracies of the Naive Bayes, \ac{KNN} and \ac{SVM} are indicated depending on the size of the training dataset and for the \ac{BLSTM} the accuracies are indicated depending on the number of epochs.
For \ac{BERT} the horizontal axis shows the number of training steps, which is a comparable measurement with the number of epochs (Chapter \ref{ch:experiments}).

Validation curves are used to find the best hyperparameters of a model.
They show the accuracy of the model depending on different values of this hyperparameter, that can be for example the number of $K$ in the case of the \ac{KNN} or the value of $C$ for a linear \ac{SVM}.
The best value is then selected by choosing the model that reached the highest validation accuracy \cite[p. 711]{Russel2016}.
Figure \ref{figure:sample_validation_curve} shows a possible validation curve for a \ac{KNN} classifier,

In this thesis both of the curves are created in combination with cross-validation.
\begin{figure}[h]
    % \begin{subfigure}{0.5\textwidth}
    %     \includegraphics[width=\textwidth]{figures/knn_sample_learning_curve.png}
    %     \caption{The learning curve shows the accuracy depending on the training dataset size.}
    %     \label{figure:sample_learning_curve}
    % \end{subfigure}
    % \begin{subfigure}{0.5\textwidth}
    %     \includegraphics[width=\textwidth]{figures/knn_sample_validation_curve.png}
    %     \caption{The validation curve shows how a change in the value of a hyperparameter affects the accuracy of the model.}
    %     \label{figure:sample_validation_curve}
    % \end{subfigure}
    \subcaptionbox{The learning curve shows the accuracy depending on the training dataset size.\label{figure:sample_learning_curve}}[0.5\linewidth]{\includegraphics[width=0.5\textwidth]{figures/knn_sample_learning_curve.png}}
    \subcaptionbox{The validation curve shows how a change in the value of a hyperparameter affects the accuracy of the model.\label{figure:sample_validation_curve}}[0.5\linewidth]{\includegraphics[width=0.5\textwidth]{figures/knn_sample_validation_curve.png}}
    \caption{Sample learning and validation curves of a \ac{KNN} classifier.}
    \label{figure:sample_learning_validation_curves}
\end{figure}

\FloatBarrier
\subsection{\acl{PCA}}
\acl{PCA} is a method to reduce the dimensionality of a dataset with loosing as little information as possible.
In this thesis \ac{PCA} is used to obtain a two dimensional representation of the data, which can then be visualised.
Mathematically the \ac{PCA} can be seen as the projection of a data matrix $X$ to a target matrix $Y$ with other basis vectors (equation \ref{equation:pca_projection}) \cite[p. 3]{Shlens2014}.
\begin{equation}
    Y=PX
    \label{equation:pca_projection}
\end{equation}
For the application of this thesis $X$ is a $m\times n$ matrix where each column contains the hidden states for the business report.
$Y$ is a $m\times n$ matrix as well, with the special property, that the rows are sorted by their variances, with the highest variance in the first row \cite[p. 5]{Shlens2014}.
To accomplish this, the covariance matrix $C_X$ of $X$ is computed.
If the row entries of $X$ have a zero mean, the covariance matrix is given by $C_X=\frac{1}{n}XX^T$ \cite[p. 5]{Shlens2014}.
If the mean is not zero, each value has to be subtracted by the corresponding row mean first.
Then the eigenvectors of $C_X$ are computed, which form the rows of the projection matrix $P$ \cite[p. 6]{Shlens2014}.
Another way of solving the \ac{PCA} is to use singular value decomposition, this is described in detail in \cite[p. 7]{Shlens2014}.

% This is done by fitting a hyperplane with dimension $q \leq p$ into the featurespace with dimension $p$ that minimises the squared error to the data points.
% For practical usage the dimension $q$ of the hyperplane is typically 2 or 3.
% For $q=2$, the hyperplane is a twodimensional plane on which the values are projected and can then be visualised.


\documentclass[letterpaper]{article}
\title{\textbf{Tokio university's entrance exam 2016}}
\author{Acosta Domínguez Jorge}
\date{}

\usepackage{amsmath}
\usepackage{tabto}
\usepackage{tasks}
\usepackage{geometry} %this reduces the amount of margin
\newcommand{\itab}[0]{\hspace{2em}}

\begin{document}
\pagenumbering{gobble}
\maketitle
\newpage
\pagenumbering{arabic}

\section{Mathmatics exam}
\subsection{Lineal algebra}
The tribonacci numbers ${T_n}$ are defined for non-negative integers n as follows.
\begin{displaymath}
\begin{cases}
T_0 &= T_1 = 0\\
T_2 &= 1\\
T_{n+3} &= T_{n+2} + T_{n+1} + T_{n} \itab(n\geq 0)
\end{cases}
\end{displaymath}
Answer the following questions.
\begin{tasks}
\task{
Find the matrix $A$ that satisfies Eq. (1) for all non-negative integers $n$.
\begin{equation}
\left(\begin{matrix}
T_{n+3}\\
T_{n+2}\\
T_{n+1}
\end{matrix}\right)
=A
\left(\begin{matrix}
T_{n+2}\\
T_{n+1}\\
T_{n}
\end{matrix}\right)
\end{equation}
}
\end{tasks}
\newpage
\subsubsection{Resolution}
\begin{tasks}
\task{
The resolution for this exercise would be the below matrix $A$, since it accomplishes the equation:
\begin{displaymath}
A = \left(\begin{matrix}
1 & 1 & 1\\
1 & 0 & 0\\
0 & 0 & 0
\end{matrix}\right)
\end{displaymath}
Because resolving $1$ we have:
\begin{displaymath}
\left(\begin{matrix}
T_{n+3}\\
T_{n+2}\\
T_{n+1}
\end{matrix}\right) =
\left(\begin{matrix}
T_{n+2} + T_{n+1} + T_{n}\\
T_{n+2}\\
0
\end{matrix}\right)
\end{displaymath}
}
\end{tasks}
\end{document}


%
% 235
%
\chapter{The Gamma Function}

\Section{12}{1}{Definitions of the Gamma- function. The Weierstrassian product.}

Historically, the Gamma-function* V z) was first defined by Euler as
the limit of a product \hardsubsectionref{12}{1}{1}) from which can be derived the
infinite integral

1 i ~ e~ dt; but in developing the theory of the function, it is more
con- Jo

venient to define it by means of an infinite product of Weierstrass'
canonical

form.

Consider the product ze fl -I ( 1 -H -

e

where 7= lim ]-+--f-...-l- log ?/i[ =0-5772157....

[The constant y is known as Euler's or Mascheroni's constant; to
prove that it exists we observe tliat, if

u = I -. .a<= - log - -,

[ dt \ "

u IS positive and less than / -, = -,; therefore 2 w converges, and Jo
n 'i n=l

,. n 1 1, 1,. ( "', m + l\ *

hm J- + -+...+ logwl= hm < 2 ?< + log \= 2 .

The value of y has been calculated by J. C. Adams to 260 places of
decimals.]

The product under consideration represents an analytic function of z,
for

all values of z; for, if N be an integer such that \ z\ \ iV, we
havef, if n > N,

1 2- 1 Z'

1

2n'-' Sn' '"

z z z n- 1 n n-

-

IN \ \ 1 1 1 1 '

 4 r' 2 + 2' +---) 2 n '

00

Since the series 2 [N' l 2n?)] converges, it follows that, when \ z\ \
N,

n = N+l

* The notation V (z) was introduced by Legendre iu 1814. t Taking the
principal value of log (1 + z/?().

%
% 236
%

S ]log(l + - is an absolutely and uniformly convergent series

of analytic functions, and so it is an analytic function \hardsectionref{5}{3});
con- sequently its exponential 11 ( l+-]e~nl is an analytic function,
and

so zey n ! ( 1 + - J e ~ [ is an analytic function when \ z\ \ N,
where N is any integer; that is to say, the product is analytic for
all finite values of z. The Gamma-function was defined by Weierstrass*
by the equation

,= -T- = zey U ( l+-]e ~n[-

from this equation it is apparent that T (z) is analytic except at the
points z=0, - 1, - 2, . . ., where it has simple poles.

Proofs have been published by Holder t, Moore J, and Bai-nes§ of a
theorem known to Weierstrass that the Gamma-function does not satisfy
any differential equation with rational coefficients.

Example 1. Prove that

r(i) = i, r'(i)=-y,

where y is Euler's constant.

[Justify differentiating logarithmically the equation

~ = zey''n ( l+ -)e~ri\ Viz) 1 IV nJ I

by \hardsectionref{4}{7}, and put z = l after the differentiations have been
performed.]

Example 2. Shew that

,11 1 P 1

2 3 n Jo

t and hence that Euler's constant y is given by||

-(1-""*,

lim

[/: '-04)" ?-/:(-r.)"T']'

Example 3. Shew that

e> r(g + l)

T z-x + ) '

n = \ \ \ Z + n) J

* Journal filr Math. li. (1856). This formula for F(z) had been
obtained from Euler's formula \hardsubsectionref{12}{1}{1}) in 1848 by F. W. Newman,
Cambridge and Dublin Math. Journal, iii. (1848), p. 60. t Math. Ann.
sxvm. (1887), pp. 1-13. + Math. Ann. xlviii. (1897), pp. 70-74. §
Messenger of Math. xxix. (1900), pp. 122-128. li The reader will see
later \hardsectionref{12}{2} example 4) that this limit may be written

%
% 237
%

\Subsection{12}{1}{1}{Euler's TODO formula for the Gamma-function.}

By the definition of an
infinite product we have

TODO

Hence T (.) = - H j(l + -j (l +,;j | .

This formula is due to Euler*; it is valid except when = 0, - 1, - 2,
....

Example. Prove that

,,,. 1.2...(n-l)

r (2) = hm, .,. 7- - - . n\

\addexamplecitation{Euler.}

VlVl. The difference equation satisfied by the Gamma-function.

We shall now shew that the function F (z) satisfies the difference
equation

T z -l) = zT z). For, by Euler's formula, if z is not a negative
integer,

FU + 1)/F(.) = -

lim n

;h -* X = 1

(1 +

V n

ly+r

1 +

z+ 1

1 lim n ii

m-*---c ? =1

1 +

1 J

= - lim II,

  + 1, -*x = 1 I + ?i +

, . m + 1

= z hm = z.

 i x 2: + Wi + 1

This is one of the most important properties of the Gamma-function.
Since F (1) = 1, it follows that, if 2: is a positive integer, F (2 )
= (2 - 1) !.

It was given in 1729 in a letter to Goldbacb, printed in Fuss'
Corresp. Math.

%
% 238
%

Example. Prove that

[Consider the expression

111 1

z z z- ) 2(s + l)(2 + 2) z z + l)... z+m)

in d

It can be expressed in partial fractions in the form 2 -, where

\ (-) / \ \ 1 1 ]\ (-)" f °° 1

"=1 e "'( -)"!( =" 11

Noting that 2 -,<-, itt-t, prove that 2 - .- \ 2 -;h- 0 as

r=m-ri+\ r\ m-n + l)\ n=o n\ z + n [r=m-n+irl]

m- <X) when z is not a negative integer.]

\Subsection{12}{1}{3}{The evaluation of a general class of infinite products.}

By means of the Gamma-function, it is possible to evaluate the general
class of infinite products of the form

n Un,

7t = l

where Un is any rational function of the index n.

For, resolving m into its factors, we can Avrite the product in the
form

* f (n -a )(n-a2) ...(n- ak) ]

n=i[ (n-b,).: n-bi) I'

and it is supposed that no factor in the denominator vanishes.

In order that this product may converge, the number of factors in the
numerator must clearly be the same as the number of factors in the
denominator, and also A 1; for, otherwise, the general factor of the
product would not tend to the value unity as n tends to infinity.

We have therefore k= I, and, denoting the product by P, we may \ vrite

 =i ] n-b ) ... n-bk)i ' The general term in this product can be
written

(- (- )(-r-(-r

a + a + ... +ak-b,- ...-bk . = 1 - - h Jin

n

where 4 is (h~-) when n is large.

In order that the infinite product may be absolutely convergent, it is
therefore necessary further \hardsectionref{2}{7}) that

tti + . . . + Ojk - 1 -   - /fc = 0.

%
% 239
%

We can therefore introduce the factor

exp /i-i ( ! + ... +ak-b,- ... - hk)

into the general factor of the product, without altering its value;
and thus we have

p= n

(i\ ), (i\ )e"...(i- *)."

"=N (- )--(- )-

But it is obvious from the Weierstrassian definition of the Gamma-
function that

,"|('" )''"]"- r(- )e-T''

and so P = r(- 6,)6,r(-6 (- .) TJ l-K) . a T -CH)...akV -ak), =
ir(l-a )

a formula which expresses the general infinite product P in terms of
the Gamma-function.

Example 1. Prove that

 " (a-i-6 + ) \ r(a + l)r (6-1-1) s=i (a+ ) (6-l-s) T a + b + ) '

Example 2. Shew that, if a = cos (27r/n) + i sin 27r jn\ then

/ \ / \ - -

 (i- j(i-|;)... = -r(-x )r(-aa- )...r(-a"-i:r ) -'.

\Subsection{12}{1}{4}{Connexion between the Gamma-function and the circular functions.}

We now proceed to establish another most important property of the
Gamma-function, expressed by the equation

r( )r(i-. )= .- .

sm TTZ We have, by the definition of Weierstrass \hardsectionref{12}{1}),

r(.)r(-.)=-in (i+i)r rnj(i-i);"

Z Sin TTZ

by \hardsectionref{7}{5} example 1. Since, by \hardsubsectionref{12}{1}{2},

r(i-z) = -zr(-z)

we have the result stated.

%
% 240
%

Corollary 1. If we assign to z the value \, this formula gives r( ) 2
= 7r; since, by the formula of Weierstrass, r ( ) is positive, we
have

Corollary "l. If r/.(s) = r'(s)/r(2), then i/ (1 -z)-- /r (0) = 7r cot
tts.

\Subsection{12}{1}{5}{The multiplication-theorem of Gauss* and Legendre.}
We shall next obtain the result

r(.)r(. + )r(. + )...r(. + 'i ) = (2.)n -. - -rM.

n) \ n For let ( (.) = iT;; \ .

Then we have, by Euler's formnla \hardsubsectionref{12}{1}{1} example),

 - 1 .2...(m- l).m - /'*

n'"- n lim

 (Z)

,. 1 . 2 ... (m - 1) . (nm)'*

n lim

\ im - 1)! m

nz (nz + 1 ) . . . (71Z + n7n - 1)

 n'ln - 1) ! nm)'" i(m- 1) !j wi" 'n

m oo (wm-1)!

It is evident from this last equation that <p (z) is independent of z.
Thus (j) (z) is equal to the value which it has when z = -; and so

Therefore < (z)Y = n| | T ( -) T (l -;:)|

tt' -i (27r)"-

. TT . 27r . (?i- l)7r

sm - sm - ... sm

n n n

Thus, since cj) (n~ ) is positive,

<,W = (2,r)4<"-'' -*,

i.e. r (.) r (. + ) . . . r (. + '-i i) = - (2,r)4 < - '> r ( .).

Corollary. Taking n = 2, we have

2 -ir(.)r( +i)=,r4r(24

This is called the duplication formula.

* Werke, iii. p. 149. The case in which n - 2 was given by Legendre.

%
% 241
%

Example. If (, o) = EMlM

shew that

D(p,,)b p + \ q)...B(p ',q) B (np, no) = a ~ i \ n /

\Subsection{12}{1}{6}{Expansions for the logarithmic derivates of the Gamma-function.}

We have r (2+l) -J = '>' n /("l + -") e~ ]- .

DifiFerentiating logarithmically \hardsectionref{4}{7}), this gives

l gJ iiiLi)= \ + !\ + --,,

rfs ''' 1 (2 + 1) '*"2(2 + 2) " 3(2 + 3) ""

Therefore, since log r (s + 1 ) = log 2 + r (2), we have

Iogr(2)=-y--.+ 2 2

dz ° ' z n=in(2 + n)'

d-. \,,. (f

Differentiating again,,logr(.+ l) = + 2 (2 + 2) + -

1 1

 (2+l>' (2 + 2)2 ""

These expansions are occasionally used in ai)plications of the theory.
\Section{12}{2}{Elder s expression of T z) as an injinite integral.}

The infinite integral I e~H ~ dt represents an analytic function of z
when*

Jo

the real part of z is positive \hardsubsectionref{5}{3}{2}); it is called the Eulerian
Integral of the Second Kindf. It will now be shewn that, when li z)>0,
the integral is equal to F (z). Denoting the real part of z by x, we
have x > 0. Now, if J

n(z,n)=j'Ul-yJ\'- dt,

we have U (z, n) = n l (1 - TyW-'hlr,

J )

if we write t= nr; it is easily shewn by repeated integrations by
parts that, when a; > and n is a positive integer,

f n 1 n; !

 1 - t)"t - dr = - T- (1 - Tf + - (I - Tf-'T'dr Jo \ J Jo S'.

n(n-l)...l fi,

z(5 + l)...( + n-l)Jo '* '

J n / \ 1 . 2 . . . ?i

and so 11 (2, n) = -. - n

z z ) ... z -n)

Hence, by the example of \hardsubsectionref{12}{1}{1}, 11 z, n) r( ) as /i -* x .

* If the real part of z is not positive the integral does not converge
on account of the singu- larity of the integrand at f = 0.

t The name was given by Legendre; see \hardsectionref{12}{4} for the Eulerian
Integral of the First Kind.

t The many-valued function < ~i is made precise by the equation *~i =
c(*-i)'°s', log r being purely real.

W. M. A. 16'

%
% 242
%

Consequently F ( ) = lim (1 ) t-~ dt.

And so, if T, (2) = I e-H'-'dt,

J (\ we have

ri( )-r( )= lim r ie-'-(l- y\ t'-'dt+j e-H'-'dt .

Now lim I e'U'-'dt = 0,

since I e~H ~ dt converges.

.'0

To shew that zero is the limit of the first of the two integrals in
the formula for Fj z) - F(z) we observe that

[To establish these inequalities, we proceed as follows : when < y <
1, from the series for e and (1 - 3/)" Writing tjn for y, we have

1 + - e-'

n

HJ-

and so 0 e- -fl--

71

= e-Ml-eM 1

<e-Ml- 1

-1)1

Now, if O-Sa l, (1 - a)" l - na by induction when ?ia<l and obviously
when 7ia 1; and, writing t /n for a, we get

and so* 0 e-*-(l--) %e-H ln,

which is the required result.]

From the inequalities, it follows at once that

r' - f 1 - - j ) t'-'dt ' I n-'e-ff'- 'dt

<n-'j e-H' - 'dt O, Jo

as n - X, since the last integral converges.

* This analysis is a modification of that given by Schloinilch,
Compendium der hoheren Analysis, ii. p. 243. A simple method of
obtaining a less precise inequality (which is sufficient for the
object required) is given by Bromwich, Infinite Series, p. 459.

%
% 243
%

Consequently Fi 2)= V (z) when the integral, by which Tj (z) is
defined, converges; that is to say that, when the real part of z is
positive,

J n And so, when the real part of z is positive, F (z) may be defined
either by this integral or by the Weierstrassian product. Example 1,
Prove that, when R (z) is positive,

Example 2. Prove that, if li z) > and R (s) > 0,

/ e-"=x -'cla.-= . Jo z

Example 3. Prove that, if R z)>Q and R is) > 1,

Example 4. From :; 12'1 example 2, by using the inequality

0%e- -( X fie- ln,

deduce that

l\ e- \ e-i/<

-at.

<

t

\Subsection{12}{2}{1}{Extension of the infinite integral to the case
  in which the argument of the Gamma-function is negative.}

The formula of the last article is ni> longer applicable when the real
part of z is negative. Cauchy* and Saalschiitzt have shewn, however,
that, for negative arguments, an analogous theorem exists. This can be
obtained in the following way.

Consider the function

r, z)=j\'- (e-'-\ + t-, + ... + -)>'- dt,

where k is the integer so chosen that -k>x>-k-l, x being the real part
of z. By partial integration we have, when s < - 1,

r,(.)=[f(.-.-n.<-il + ...+(-r.,)];

+.'j7''( -'-i+'--+(-''(j )*

The integrated part tends to zero at each limit, since x+k is negative
and x + k + l is positive : so we have

r, z) = lT, z+i).

The same proof applies when x lies between and -1, and leads to the
result

r z+i)=zr2 z) (0>.r>-i).

The last equation shews that, between the values and -1 of a*,

To z) = r(z).

* Exercices de Math. u. (1827), pp. 01-92.

t Zeitschrift fur Math, und Phys. xxxii. (1887), xxxiii. (1888).

16-2

%
% 244
%

The preceding equation then shews that To z) is the same as T (2) for
all negative values of Ji z) less than -1. Thus, for all negative
values of R(z), we have the result of Cauchy and Saalschiitz

where k is the integer next less than - B (z).

Example. If a function P (ft) be such that for positive values of /x
we have

J

- 1 -x

e~'-' dx,

and if for negative values of /x we define Pj (fi) by the equation

dx.

Pi(;u)=|\ i'--i(e- -l + .r-... + (-)*- + i,)

where k is the integer next less than - ju, shew that

/',W./'W-l + -... + (-)'--.-,. \addexamplecitation{Saalschiitz.}

\Subsection{12}{2}{2}{Hankers expression of TODO as a contour integral.}

The integrals obtained for T z) in
\hardsectionref{12}{2}, \hardsubsectionref{12}{2}{1} %TODO:cite multiple
are members of a
large class of definite integrals by which the Gamma-function can be
defined. The most general integral of the class in question is due to
Hankel*; this integral will now be investigated.

Let D be a contour which starts from a point p on the real axis,
encircles the origin once counter-clockwise and returns to p.

Consider 1 (-t)~~ e~hlt, when the real part of z is positive and z is
not an integer.

The many-valued function (- ty~ is to be made definite by the
convention that (- ty~ = e' ~ ' ° '"*' and log (- t) is purely real
when t is on the negative part of the real axis, so that, on D, - ir %
arg (- t) % it.

The integrand is not analytic inside D, but, by \hardsectionref{5}{2} corollary 1, the
path of integration may be deformed (without affecting the value of
the integral) into the path of integration which starts from p,
proceeds along the real axis to h, describes a circle of radius h
counter-clockwise round the origin and returns to p along the real
axis.

On the real axis in the first part of this new path we have arg (- ) =
- tt, so that - ty~' - e~ '' ~ H ~ (where log i is purely real); and
on the last part of the new path (- ty- = e " ' - ' t ' .

On the circle we write - i = 8e*; then we get

f -ty-'e-Ht-= \ e-'-(--i) '-ie-'f? +f"(8e' )--ie (cose+;sm< )gg,ej 7

+ ''e' ' -' H'-' e' dt = - 2i sin -jTz) j t'-' e-hit + ih' I " e<>e+
(cose+tsine) \

* Zeitschrift filr Math, und Phys. ix. (1864), p. 7.

%
% 245
%

This is true for all positive values of S p; now make S; then 8' and
I gize+s (COS 0+isin e 0 j ize iQ [ \ \ q integrand tends to its limit

J - TT J -IT

uniformly.

We consequently infer that

[ (-ty-'e-'dt = -21 sin 7r2)[''t'-'e-*dt.

J B Jo

This is true for all positive values of p; make p x, and let C be
the limit of the contour D.

Then f (- ty-'e- dt = - 2i sin ttz) f t'-'e- dt.

J c Jo

Therefore T ( ) = - r 4 I (- tf-'e-'dt.

'2i sm TTzJ c

Now, since the contour C does not pass through the point t = 0, there
is no need longer to stipulate that the real part of 2 is positive;
and

I (-ty~ e~ dt is a one-valued analytic function of z for all values of
2.

J c

Hence, by \hardsectionref{5}{5}, the equation, just proved when the real part 0/2 is
positive, persists for all values of z with the exception of the
values 0, ±1, +2

Consequently, for all except integer values of,

r(0) = -\ J - I -ty-'e-'dt.

This is Hankel's formula; if we write 1 - 2 for z and make use of
\hardsubsectionref{12}{1}{4}, we get the further result that

We shall write / for \, meaning thereby that the path of inte-

1 re C

gration starts at 'infinity' on the real axis, encircles the origin in
the positive direction and returns to the starting point.

Example 1. Shew that, if the real part of z be positive and if a be
any positive

constant, l - t)~ e~ dt tends to zero as p -ao, when the path of
integration is either of

the quadrants of circles of radius p + a with centres at - a, the end
points of one quadrant being p and - a + 1 (p -|- a), and of the other
p and -a - i p- a).

V

' (0 + )

%
% 246
%

Deduce that lim I " ' t)-' e- dt= lim I -t)-'e- dt,

p-*xy-a + 'P p- y J C

and hence, by writing t= -a-iu, shew that

\ \ ['

--- = -- I e "*- " (a + <)"* f M- r (s) ztt y \ x

[This formula was given by Laplace, Theorie Analytique des
Prohahilites (1812), p. 134, and it is substantially equivalent to
Hankel's formula involving a contour integral.]

Example 2, By taking a = 1, and putting t= -\ + i tan B in example 1,
shew that -J- = - ( "" cos (tan (9 - 2(9) cos - 2 (9c?<9.

r (2) 77 Jo

Example 3. By taking as contour of integration a parabola whose focus
is the origin, shew that, if a > 0, then

Y z)=-. - - e- ''(l + -) -Jcos 2a + (22-l)arctani; (/i!. sin nz J

\addexamplecitation{Bourguet, Acta Math, i.}

Example 4. Investigate the values of x for which the integral

2 r*

 - sin tdt

TT J

converges; for such values of x express it in terms of
Gamma-functions, and thence shew that it is equal to

\addexamplecitation{St John's, 1902.}

Example 5. Prove that I (log t)" ' dt converges when m > 0, and, bv
means

Jot

of example 4, evaluate it when m=l and when m = 2.
\addexamplecitation{St John's, 1902.}

\Section{12}{3}{Gauss' expression for the logarithmic derivate of the
  Gamma-function as an infinite integral*TODO.}

We shall now express the function -7- log V (z) = as an infinite

integral when the real part of z is positive; the function in
question is frequently written yjr (z). We first need a new formula
for 7.

Take the formula \hardsectionref{12}{2} example 4)

  Jo i J\ i 5 0 Us J t / a olJA J\& t J'

where A = l-e- since | - = log .- O as 8-*-0.

./a l-e"

 Yriting = 1 -e~" in the first of these integrals and then replacing u
by t we have

y= lim I r,dt- r '- dt] = r \ j~,-]] e- dt.

5-*o Us l-e Js t J Jo U-e ' t)

This is the formula for y which was required.

* Wtr'ke, HI, p. 159.

%
% 247
%

To get Gauss' formula, take the equation \hardsubsectionref{12}{1}{6})

r'( ) 1 .. /I 1 \

T z) ' z acm=iVm z-vm)

1 r

and write = e-'( +'">rf;

z - m Jo

this is permissible when m = 0, 1, 2, ... if the real part of z is
positive.

It follows that

 '=-7- I e- ? + lim S (e-""-e-(' + )')f <

.0 rt\rightarrow \inftyJowi = l

r( )

-7+ lim = -I dt

1-

V 1 - e-'

W - lim 1 -e-(''+ '(/t

1-e-

Now, when < < 1, i is a bounded function of t whose limit as - [0 is
6nite;

I 1 - e 'I

and when t 1, - | < - J- < - .

Therefore we can find a number A' independent of t such that, on the
path of integration,

I 1 - ' I

andso I r\ zl~,-K ' Vdt\ < K[ e-i"*')' rf = /r(H + l)-' 0 as /i- -x .

I jo 1-e-' ./o

We have thus proved the formula

t< ) = s'" ' *'->=.C(T-r ')'"-

which is Gauss' expression of - z) as an infinite integral. It may be
remarked that this is the first integral which we have encountered
connected with the Gamma-function in which the integrand is a
single-valued function.

Writing <=log (1 -f-.i') in Gauss' result, we get, if A=(J - 1,

 -if=limf t'-,,U.

since < I - dt < j y =log - g- 0 as 8 0.

T'(z),. / " f 11 rf.r

Hence \ = uaj |,-.\ \ \ \ |\,

sothat ''<=)=rM/J e-'-(, §',

an equation due to Dirichlet*.

Werke, i. p. 275.

%
% 248
%

Example 1 . Prove that, if the real part of z is positive,

Example ± Shew that y=\ l -t)-' -e- ]t-' dt. \addexamplecitation{Dirichlet.}

\Subsection{12}{3}{1}{Binet's first expression for TODO in terms of an infinite integral.}

Binet* has given two expressions for logr( ) which are of
great

importance as shewing the way in which log V z) behaves as ] 2 j -* oo
. To

obtain the first of these expressions, we observe that, when the real
part of

z is positive,

r' /'-r \ L 1 \ r* (o-t o-tz ]

dt,

r( + i) Jo [ e -i

writing z + \ for 2 in \hardsectionref{12}{3}.

Now, by \hardsubsubsectionref{6}{2}{2}{2} example 6, we have

log2=) - r~ '

f" 1 and so, since (22 )" = e~ dt,

'Jo

we have

dz

logr(. + l) = l + log.-/"g-;+ - Je-*.

The integrand in the last integral is continuous as i -; and since

- - - + -f - z. is bounded as - 00, it follows without difficulty
that the

integral converges uniformly when the real part of z is posijiive; we
may consequently integrate from 1 to under the sign of integration
(\hardsubsectionref{4}{4}{4}) and we get-f-

iogr(.+i) = (. + |)iog.-.H-i+/;g- + p-- -

dt.

Since - - + - - i 7 continuous as i - by \hardsectionref{7}{2}, and since

log r (z + 1) = log + log r (z),

have

/ 1\ (" CI 1 1 ) e-*

iogr(.) = (.-2)iog.-. + i.-j |--- + - \ -- |- .

J (2 t e' - l)t

* Journal de I'Ecole Poly technique, xvi. (1839), pp. 123-143.

t Logr(2 + l) meaus the sum of the principal values of the logarithms
in the factors of the Weierstrassian product.

%
% 249
%

To evaluate the second of these integrals, let*

so that, taking z = in the last expression for logr(2:), we get

i log 7r = i + t/-/.

( + - ) - - dt, we have

t

r ' 1 \ dt

-jo V t e'- ) t

=/:r- '-HT

~jo 1 c/ V t ) t ' '2tj

* - rf<

t

= 2 + Uogi Consequently /=1- log(27r).

We therefore have Binet's result that, when the real part of z is
positive,

3-tZ

 -dt.

log r( ) = [z- \ ogz-z + \ log(27r) + J G ~ 7 + "'-t)

li z = x- iy, we see that, if the upper bound of ( ~ 7 " t \ i ) 7 ' *
' values of is K, then

logr( )-( -i)log + -|log(27r)'<i(:| "'"

so that, when x is large, the terms (z - \ og z - z \ o (2'jr) furnish
an

approximate expression for log V (z).

Example I. Prove that, when A' (a) > 0,

logr(.-)= r *i4.1-r+( -l) "j 7- \addexamplecitation{Malmsten.}

Example 2. Prove that, when R (z) > 0,

* This artifice is due to Priagsheim, Math. Ann. xxxi. (1888), p. 473.

%
% 250
%

Example 3. From the formula of \hardsubsectionref{12}{1}{4}, shew that, if <;r < 1,

2logrW-log.+logsin..-=/; |-' Ui=fl-'-(l-2.).-. f .

\addexamplecitation{Kummer.}

Example 4. By expanding sinh (| - .v) and 1 - 2x' in Fourier sine
series, shew from example 3 that, if < .v < 1,

00

log r (.p) = i log TT - - log sin -nx + 2 2 a,i sin 2?i7r,

, / " r 2?i7r e-n dt

" jo V + 4wV2 27rJ i! Deduce from example 2 of \hardsubsectionref{12}{3} that

"*" "" 2 " "*" 2"" + * g ''* )

(Kummer, Journal fiir Math. xxxv. (1847), p. 1.)

\Subsection{12}{3}{2}{Binet's second expression for log V z) in terms of an infinite integral.}

Consider the application of example 7 of Chapter vii (p. 145) to the
equation \hardsubsectionref{12}{1}{6})

The conditions there stated as sufficient for the transformation of a
series into integrals are obviously satisfied by the function ( )= -
r-, if the real part of z be positive; and we have

Since !5'(, + /i) I is easily seen to be less than K t/n, where Ki is
inde- pendent of t and n, it follows that the limit of the last
integral is 'zero.

Hence V;: log F ( ) = 2,- -I i - -, - -.

dz 2z- z Jo z + t'f e- - 1

I 2 I Since ~ - - does not exceed K (where K depends only on 8) when
the

I I*

real part of z exceeds 8, the integral converges uniformly and we may
integrate under the integral sign \hardsubsectionref{4}{4}{4}) from 1 to z.

We get

-I log r (.) = -;+ log . + - 2 /; . JL,

where C is a constant. Integrating again,

log r (.) = (. - 1) log . + (c - 1) . + c + 2/; '- dt.

where 6" is a constant.

%
% 251
%

Now, if z is real, arc tan tjz \$ tlz,

and so

logrW-( -i)log -(C-l).-C"|<?/" d(.

But it has been shewn in \hardsubsectionref{12}{3}{1} that

\ \ % z)-[z-- \ ogz- z- \ og 1'K) -0,

as 2 -* X through real values. Comparing these results we see that C =
0, 6"=ilog(27r).

Hence for all values of z whose real part is positive,

logrW = (.-l)log.-. + llog(2.) + 2|; -I rf*.

where arc tan u is defined by the equation

arc tan xi = |,

.'o l+<-

in which the path of integration is a straight line.

This is Binet's second expression for log V z).

Example. Justify differentiating with regard to z under the sign of
integration, so as to get the equation

12 33. The asymptotic expansion of the logarithm of the Gamma-
function (Stirling's series).

We can now obtain an expansion which represents the function log F z)
asymptotically \hardsectionref{8}{2}) for large values of \ z\, and which is used in
the calculation of the Gamma-function.

Let us assume that, if = x -f iy, then a-' S >; and we have, by
Binet's second formula,

log r ( ) = ( - ) log - + - log (27r) + </) ( ),

where (.) = 2/; c/.

Now

(\ )n-i 2,1-1 (\ y rt u n

arc

,,,,, t It' It' (\ )n-i 2n-i (\ )H rt

u + z' Substituting and remembering \hardsectionref{7}{2}) that

Jo e' ' -l ~4w'

%
% 252
%

where Bi, Bo, ... are Bernoulli's numbers, we have

  / X V (-)' "' Br 2 (-)'* r " f f U' ' du \ dt

.=1 2r (2?- - 1) '--l 271-1 j \ [ ! () 2 2 + 2 j g2,r< \ I

Let the upper bound * of - 1 for positive values of u be K,

j w + j

Then

ftl

f u' 'du] dt 1 r, I, r" 1 r, ) (

uow + ' j e-' '-li" ' ' Jo [Jo \ e'-'-l

Ji-z n+1

 4(n+l)(2M+l)| |2" Hence

2(-y r ( f* u l \ d KA+

z "-' Jo Vou' + z e -l 2 n + l) 2n + l)\ z p +i '

and it is obvious that this tends to zero uniformly as | | - oo if I
arg | tt - A, where tt > A > 0, so that K cosec 2 A.

Also it is clear that if ] arg r | Itt (so that Kz = l) the error in
taking the first 71 terms of the series

I (-y-'Br 1

rti 2r (2r - 1) z -' as an approximation to <f) (z) is numerically
less than the (n + l)th term. Since, if | arg z\ \ 7r - A,

-,211-1

!</> ( ) - i Z ' \ I < cosec 2 A .,

r r=i 2r(2r-l)l; 2(n + l)(2,

2(n + l)(2n + l)

-0,

as 00, it is clear that

B, B, B,

1.2.2 3. 4. 2=* ' 5.6.2 "* is the asymptotic expansion f \hardsectionref{8}{2}) of
(2). We see therefore that the series

is the asymptotic expansion of log V z) when | arg | tt - A.

-Kj 18 the lower bound of -!, .j and is consequentlj' equal to

4 2y2,

, 9, .n o or 1 as x2<m2 or x >y . t The development is asymptotic;
for if it converged when | 2 | p, by \hardsectionref{2}{6} we could find K, such that
B <(2;i-l)2nA>2"; and then the series 2 ~ "~' " " would define an
integral function; this is contrary to \hardsectionref{7}{2}.

%
% 253
%

This is generally known as Stirling s series. In \hardsectionref{13}{6} it will be
estab- lished over the extended range | arg tt - A.

In particular when z is positive (= x), we have

r u? du'] dt Bn+.

Jo Uo w + j e ' '-l

w + je ' '-l 2 n + l) 2n + l)af

Hence, when x>0, the value of < )(oc) always lies between the sm7i of
n terms and the sum ofn + 1 terms of the series for all values of n.

D g

In particular < (f> (x) < - -, so that <f> (./) = y where < < 1.

Hence T x) = x'- - e'' i2'rrf e ' 'l

Also, taking the exponential of Stirling's series, we get

\ x .r-, a f 1 1 139 571 /'I

r ix) = e X ~ (27r) -,1 - 288 ~ 51840 ~ 2488320 "*" V

This is an asymptotic formula for the Gamma-function. In conjunction
with the formula T x + l) = xr x), it is very useful for the purpose
of com- puting the numerical value of the function for real values of
x.

Tables of the function logioT (. ), correct to 12 decimal places, for
values of .v between 1 and 2, were constructed in this way by
Legendre, and published in his Exercices de Calcul Integral, ii. p.
85, in 1817, and his Traite des fonctions elliptiques (1826), p. 489.

It may be observed that V (x) has one minimum for positive values of
.>;, when .r= 1-4616321..., the value of log,or(.r) then being
1-9472391....

Example. Obtain the expansion, convergent when R z) > 0,

\ og,T z) = z-l)\ og, z- z+ \ og, 27r) + J (z),

where in which

- W- 5 |, + | + 2 (- +1) (. + 2) 3 (.-+1) (.- + 2) z + S) -

and generally

c = r (x+l) x + 2) ... x + n- ) 2x - ) xdx. \addexamplecitation{Binet.}

J

\Section{12}{4}{The Eulenan Integral of the First Kind.}

The name Eulerian Integral of the First Kind was given by Legendre to
the integral

B (p, q) = f xP-' (1 - x)'i-' dec,

J

which was first studied by Euler and Legendre*. In this integral, the
real parts of p and q are supposed to be positive; and xp~, (1 - x) ~
are to be understood to mean those values of e( ~ ) °°* and
e('v-i)'°e(i- =) which correspond to the real determinations of the
logarithms.

* Euler, Nov. Comvi. Petrop. xvi. (1772); Legendre, Exercices, i. p.
221.

%
% 254
%

With these stipulations, it is easily seen that B (p, q) exists, as a
(possibly improper) integral \hardsectionref{4}{5} example 2).

We have, on writing (1 - *) for w,

B p,q) = B q,p).

Also, integrating by parts,

Jo L i Jo PJo

so that B p,q + l) = B p+l,q).

Example 1. Shew that

B p,q)= £ip + l,q) + B p,q + l).

Example 2. Deduce from example 1 that

B P,<l + )= B p,q).

Example 3. Prove that if n is a positive integei*,

,,, 1 . 2 ... M

B p,n + ) = -

p p + l)... p + n)\ Example 4. Prove that

Example 5. Prove that

r (2) = lim n B z, n).

\Subsection{12}{4}{1}{Expression of the Eulerian Integral of the First
  Kind in terms of the Gamma function.}

We shall now establish the important theorem that

R/.,, x\ r(m)r(n)

First let the real parts of m and n exceed |; then

r (m) r (n) = e-* '"-i c a; x g-?' if'- dy. Jo Jo

On writing a;'- for .t, and y for y, this gives

fR rR

V (m) r (n) = 4 lim e"*' x'-""-' dx x g-?/' /-"-i dy

= 4 lim e-( '+2'') 2m-iy2n-i( ( \

Now, for the values of m and n under consideration, the integrand is
continuous over the range of integration, and so the integral may be
con- sidered as a double integral taken over a square Sji. Calling the
integrand

%
% 255
%

f x, y), and calling Qg the quadrant with centre at the origin and
radius R, we have, if Tji be the part of S, outside Qb,,

I f(x, y) dxdy - f oc,y) dxdy JJsr JJqr I

= 1 f(a:,y)dxdy

 JJ Tr

i \ f >y)ida dy-\ \ \ f,y)dxdy\

JJSr JJ Sir

  . ., .,

- 0 as R X,

since 1 1 |/(;, y) \ dxdy converges to a limit, namely

JJ Sr

Therefore

Imi

2 I e- '-, x""-' idxx2\ e' \ y-''-' \ dy. Jo' Jo

I f(x,y)dxdy= lim // f x,y)dxdy.

Changing to polar* coordinates x = rcos 0, y = r sin 6), we have f x,
y) dxdy = |

Oh

Hence

1 1 f x, y) dxdy = ( \ e-"" (r cos 0) '"-' (r sin 0)-''-' rdrd0.

J J Or J J

r (w) r ( ) = 4 e-r' r-('"+")-i dr cos=" -i sin ' - c? Jo Jo

= 2r ( i + ?z) cos '"-' sin ''- <9c? .

Jo

Writing cos- = u we at once get

r (m) r (?i) = r (?u + ) . z (w, ?i).

This has only been proved when the real parts of ni and n exceed |;
but it can obviously be deduced when these are less than | by TODO
example 2.

This result, discovered by Euler, connects the Eulerian Integral of
the First Kind with the Gamma-function.

Example 1. Shew that

[' (l-f-A-)' -i(l-.r) -i( =2P + 9-i Mli2).

  -I ' ' ip+q)

  * It is easily proved by the methods of \hardsubsectionref{4}{1}{1} that the areas A /x of
  \hardsectionref{4}{3} need not be rect- angles provided only that their greatest
diameters can be made arbitrarily small by taking the number of areas
sufficiently large; so the areas may be taken to be the regions
bounded by radii vectores and circular arcs.

%
% 256
%

Example 2. Shew that, if

j\ isj .v x + 1 2! .r + 2 3! .r + d

then

f x,y)=f y + \,x-\ \

where and y have such vakies that the series are convergent. (Jesus,
1901.)

Example 3. Prove that

j'J'j xy) l-xT- y> i yr- dxdy = - > 'j

\addexamplecitation{Math. Trip. 1894.}

\Subsection{12}{4}{2}{Evaluation of trigonometrical integrals in terms of the Gamma-function.}

We can now evaluate the integral cos" ~ x sm' ~ xdx, where m and n

Jo

are not restricted to be integers, but have their real parts positive.
For, writing cos o; = t, we have, as in \hardsubsectionref{12}{4}{1},

I'*" .,, IF am) ran)

cos' - cc sin" -1 xdx = f. ' - .

The well-known elementary formulae for the cases in which m and n are
integers can be at once derived from this result.

Example. Prove that, when \ k\ < .\,

fh CDS'" e sin' edd \ T (lm+ )T /i+ ) fh cos'" -'"Odd J \ l ksm 6)i ~
Tlh-a + hi+l) 7r J (l - k sin2 )i + *

\addexamplecitation{Trinity, 1898.}

\Subsection{12}{4}{3}{Pochhammers* extension of the Ealerian Integral of the First Kind.}

We have seen in \hardsubsectionref{12}{2}{2} that it is possible to replace the second
Eulerian integral for F(z) by a contour integral which converges for
all values of z. A similar process has been carried out by Pochhammer
for Eulerian integrals of the first kind.

Let P be any point on the real axis between and 1; consider the

integral

r(i+,o+, 1-, 0-) e-'"'<' + ) t - (1 - 0 -' dt = e a, /3).

J F

The notation employed is that introduced at the end of \hardsubsectionref{12}{2}{2} and
means that the path of integration starts from P, encircles the point
1 in the positive (counter-clockwise) direction and returns to P, then
encircles the origin in the positive direction and returns to P, and
so on.

, * Math. Ann. xxxv. (1890), p. 495.

%
% 257
%

At the starting-point the arguments of t and - t are both zero; after
the circuit (1 +) they are and l-ir; after the circuit (0 +) they are
Itt and 27r; after the circuit (1 - ) they are lir and and after the
circuit (0 - ) they are both zero, so that the final vahie of the
integrand is the same as the initial value.

It is easily seen that, since the path of integration may be deformed
in any way so long as it does not pass over the branch points 0, 1 of
the integrand, the path may be taken to be that shewn in the figure,
wherein the four parallel lines are supposed to coincide with the real
axis.

>

// the real parts of a. and y9 are positive the integrals round the
circles tend to zero as the radii of the circles tend to zero*; the
integrands on the paths marked a, h, c, d are

 a-Ig-'W(a-l) (1 \ )3-lg2.r<(8-l) a-l g2,r.(a-]) ( \ )fl-i

respectively, the arguments of t and 1 - nuiu being zero in each case.

Hence we may write e (a, ) as the sum of four (possibly improper)
integrals, thus :

e (a, /3) = e- '(''+ )

I t -' (1 - 0 "' + I t"-' (1 - 0 ~'e-''

 dt

C 1 /""

+ t"-' (1 - tf-' e -'t"* ) dt + -' (1 - tf-'e- ' dt . .' 1

Hence

€ (a, /3) = e- '(' - ) (1 - e---) (1 - e ' ) f -' (1 - tf-' dt

JO

, ., ., l (a)r(/9) = - -isin (a7r)sin (ott) ~

- 47r* "r(l-a)r(l-/: )r(a + )-

Now e (a, /S) and this last expression are analytic functions of a and
of /3 for all values of a and /S. So, by the theory of analytic
continuation, this equality, proved when the real parts of a and are
positive, holds for all values of a and l3. Hence for all values of a
and /3 lue have proved that

/ J -47r-

r i-cc)l\ l-id)l\ a+id)- ' The reader ought to have uo difficulty in
proving this.

AV. M. A.

17

%
% 258
%

\Section{12}{5}{Dirichlet's integral*TODO.}

We shall now shew how the repeated integral

/ = f f . . . I f t, + t,+ ...+ tn) i"'- o" - . . . tn' n-idt dt, . .
. dt

may be reduced to a simple integral, where/is continuous, a > (?* = 1,
2, ... n) and the integration is extended over all positive values of
the variables such that 1 + 4+ ... + tn \$1.

To simplify p'M f\ t + T+ ) t' -'T -'dtdT

Jo Jo

(where we have written t, T, a, j3 for t, t, cui, Og and X for 3 + 4
+ ... +tn), put t = T \ - v)/v; the integral becomes (if X. 0)

r~U' /(x + Tlv) l - vy-' V-''-' r-+3-i dvdT.

Jo J T/(l- )

Changing the order of integration \hardsubsectionref{4}{5}{1}), the integral becomes

( [ V( + T/v) l- vy-'v''-' T- -'dTdv. J J

Putting T = VTo, the integral becomes

I i /( + " 2) (1 - vy-' v -' r. + -' dr., dv J J

r (a)r(8) r -

r( + /3) Jo

Hence

  J|... j/(T. + 3+ ... +Qt/.+ -i 3" -i ... tn ' -'dr.dt, ... dtn,

r(,)r(,) r(e

the integration being extended over all positive values of the
variables such that T2 + ts + ... +tn- .

Continually reducing in this way we get

r((x, + a,+ ...+an) Jo- ' which is Dirichlet's result. Example 1.
Reduce

to a simple integral; the range of integration being extended over
all positive values of the variables such that

it being assumed that a, 6, c. a,, y, p, q, r are positive.
\addexamplecitation{Dirichlet.}

* Werkt, I. pp. 375, 391.

%
% 259
%

Example 2. Evaluate / / x yi dxdy,

m and n being positive and

' >0, 2/ 0, x>" + y " 1 . ( Pembroke, 1 907 . )

Example 3. Shew that the moment of inertia of a homogeneous ellipsoid
of unit density, taken about the axis of z, is

where a, 6, c are the semi-axes.

Example 4. Shew that the area of the epicycloid x-' +y =P is fn-r-.

REFERENCES. N. Nielsen, Handhnch der Theorie aer (Jamma-funktion*.
(Leipzig, 1906.) 0. ScHLOMiLCH, Compendium der ho her en Analysis, ll.
(Brun.swick, 1874,) E. L. LiNDELOF, Le Calcul des Residus, Ch. iv.
(Paris, 1905.) A. Pringsheim, Math. Ann. xxxi. (1888), pp. 455-481.

Miscellaneous Examples.

1. Shew that

\addexamplecitation{Trinity, 1897.}

2. Shew that

.,'i"i vr. rrp rTr.-i7i;""='"<"+"- '™'"' ' ' '>

3. Prove that

r'(i) T' (is)

f(i) ~ r(|) = S - (' ® " ' 2-)

4. Shew that

 r(i)! 32 52-1 72 92-1 IP . .

"16;; " = 3  5  r T  92  i -ZTi  \addexamplecitation{Trinity, 1891.}

5. Shew that

- f ( -a)(M+ + y ) a \ \ 1 .,, .,\,,

n \ - - --7-7 -; - r 1+, H- = - - sm (an) B (/3, y).

 =o I ( +/:<) ( +y) \ + l/j TT ' " '

\addexamplecitation{Trinity, 1905.}

6. Shew that ( ) ('Isf ' \addexamplecitation{Peterhouse, 1906.}

7. Shew that, if z = iC where is real, then

l l=\/(fSVc)- \addexamplecitation{Trinity, 1904.}

8. When x is positive, shew thatt

r (x) r (h) °° 2n ! 1

 . ., Tx = 2 -; - - - . \addexamplecitation{Math. Trip. 1897.}

* This work contains a complete bibliography.

t This and some other examples are most easily proved by the result of \hardsubsectionref{14}{1}{1}.

17-2

%
% 260
%

9. If a is positive, shew that

r( )r( +i) i ( -)" ( - 1) ((f - 2) ...(g-H) 1 r(2+ ) =o ! s+ '

10. If .V > and

JO

shew that

and

11. Shew that if X > 0,,c> 0, - tt <a <\ tt, then
$$
TODO
$$
\addexamplecitation{Euler.}

12. Prove that, if 6 > 0, then, when < s < 2,

/ - 7" dx = \ u\ f- cosec (J7rs)/r (s), y - and, when < s < 1,

/ * cosfto; \ J (a. .)/p (5), \addexamplecitation{Euler.}

\ / .

13. If < /i < 1, prove that

jj(l+.t'-cos -rf.r=r(,0 cos (f -1) - +,

\addexamplecitation{Peterhouse, 1895.}

14. By taking as contour of integration a parabola with its vertex at
the origin, derive from the formula

1 r** "")

2 sin an \ the result

1 f

r(a)= - -. / e- ''x"--' l+:c''-) [Ziim x + a?(.YCcot(-Jc)

2 sui utt J )

+ sin x + a - 2) arc cot ( - x) ] dx, the arc cot denoting an obtuse
angle.

(Bourguet, Acta Math. i. p. 367.)

15. Shew that, if the real part of a-n is positive and 2 l/a is
convergent, then

n = l

d' is convergent when m > 2, where \//-( ) (2) = -r log r (z). (Math.
Trip. 1907.)

16. Prove that

"jo 1'

d\ ogr z)\ /"e- g-e-g'

< 2

= r (l+a)-i-(l+a)- ---), Jo a

/ 1 .2 - 1 \ ] = / dx-y. \addexamplecitation{Legendre.}

Jo x-l

%
% 261
%

17. Prove that, when R z)>0,

\ ogr z)=r i- -xiz-l)] f- . \addexamplecitation{Bmet.}

18. Prove that, for all values of z except negative real values,

log r (2) = (3 - 1) log 2 - 2 + Uog (27r)

J 1 1 2 J; \ 3\ = J ]

19. Prove that, when (2) > 0,

-rlogr(2) = log2- / r j l-a.-+log.r .

dz ° ° Jo (l-.r)log.r

20. Prove that, when R (2) > 0,

c/2' °

21. If

shew that

/2+1 logr(Oo? = ?',

and deduce from \hardsubsectionref{12}{3}{3} that, for all values of 2 except negative real
value.s,

u = z log z - z + i log (2n-).

\addexamplecitation{Raabe, Jovnial fur Math, xxv.}

22. Prove that, for all wilues of 2 except negative real values,

sin 2nTrx

00 f dx' logr(2) = (2-i)log2-2 + Uog(27r)+ 2 / --

n=lj *+'

23. Prove that

(Bourguet*.

Bip,p)B p+h,p+h)= y \addexamplecitation{Binet.}

24. Prove that, when -(</<(,

r,,,1 /""" cosh (2n*) c?M

25. Prove that, when q>\,

B p, q) + Bip + \, q) + B p + 2, q) + ...=B p, q- ) .

26. Prove that, when p-a>0,

B p-ci,q) aq a(a + l) g(g + l)

 (i, ?) ' it;4-?'*' 1. 2. (/> + (/) (io + ? + l)

27. Prove that

B p, q)B p q, r) = B q, r)B q+r, p). \addexamplecitation{Euler.}

28. Shew that

n,,, .,, d.r r(a)r(b) 1

Jo " '' (.r+?>) + '' r(a + 6) (l+jo) jo"'

if a > 0, 6 > 0, p > 0. \addexamplecitation{Trinity, 1908.}

* This result is attributed to Bourguet by Stieltjes, Journal de Math,
(i), v. p. 432.

%
% 262
%

29. Shew that, if m > 0, n>0, then

n ( i+. )2 >-i (1 - -y n-i, \ r(m)r n)

and deduce that, when a is real and not an integer multiple of ijr,

'i /cos 6 + sin \ cos 2a

/'

\ i,r veos C - SUl

d6=.

and

30. Shew that, if a > 0, > 0,

/:

2 sin (tt cos a) '

\addexamplecitation{St John's, 1904.}

\addexamplecitation{Kummer.}

31 . Shew that, if a > 0, + 6 > 0,

fr(a)r(S) r(a + b)r 8)

r..r--(i- ) \,, lrw

Jo I-''*'' 5-0 I (a

l= / (a+6)- /'(a).

' + 8) T a + b + 8) Deduce that, if in addition a + c>0, a- b + c>0,

/l a-l (l\ b)(l\ . .c) r (t ).! ( ± \ +\ ' )

jo "(i-.r)(-log.r) * ~ *r(a + 6)r(a+c)" 32. Shew that, if a, b, c be
such that the integral converges,

fUl-x")(l-afi)(l-x'),, T b + c+l)T c + a + l)T a+b + l) ' a.v=los

JO (i-A-)(-log.r) '" '' °r(a + ' ) r b + l)r c + l)r a + h + c+l)' 33.
By the substitution cos = 1-2 tan <, shew that

(3-cos )5 4v/7r

\addexamplecitation{St John's, 1896.}

f sinP.v

34. Evaluate in terms of Gamma-functions the integral / ' dx, when /)
is a

J '*-"

fraction greater than unity whose numerator and denominator are both
odd integers.

[Shew that the integral is h I sin'' x\~+ 2 ( - )" ( 1 ) [ d.v.]

'- Jo l n=i \ x+mr x-mrj)

35. Shew that

\addexamplecitation{Clare, 1898.}

/:

93?-

2" + 2 r=o2/'!(n-r)

-a-mi-

36. Prove that

log Bip,g) log ( +i) 4- I ' \: '"\ \ l '" dv. \addexamplecitation{Euler.}

s v/', -/; °\ pq J j(, (1-V)l0g 7

37. Prove that, if p>0, p + s>0, then

B(p,p)

 '

s s~l), s(s-l)(s-2)(s-3)

 (P>P + )=- f7 ' l+ ( ) + 2.4.(2 +

fd i,-- - <--

38. The curve r"*=2'"~i a'"cos ??i is composed of ?n equal closed
loops. Shew that the length of the arc of half of one of the loops is

i~ a I (i cos x) ' dx, Jo

and hence that the total perimeter of the curve is

a iV

lm)\

%
% 263
%

39. Draw the straight line joiniug the points ±i, and the semicircle
of \ z\ = \ which lies on the right of this line. Let C be the contour
formed by indenting this figure at

- ?', 0, i. By considering / 2P- -i z + z- )p "-- dz, shew that, if p
+ q>l, q <l,

I " cosP*i--6 cos ip-q) 6 d6 =,, "!,, r.

Jo ip + q-l)2P i- B(p,q)

Prove that the result is true for all values of p and q such that p +
q>'l.

\addexamplecitation{Cauchy.}

40. If s is positive (not necessarily integral), and - in .r hn, shew
that

Mild draw graphs of the series and of the function cos*.i'.

41. Obtain the expansion

cos .f-2, r (*+i)[r(| +ia+i)r(i5- a+i) " r(h+?,a+i)raj- a+iy-]'

and find the values of x for which it is applicable. \addexamplecitation{Cauchy.}

42. Prove that, if /> > A,

22p-i r 2/>"- f 12 12.32 1 "|2

'"'''"=-A ""Wi'U + l i' 2(2yT3-) + 2.4.(2p+3)(2f+5)+ . '

\addexamplecitation{Binet.}

43. Shew that, if .c < 0, ./ + -- > 0, then

r(-.f) [-.f,(-..-)(l-.r) (-.r)(l- )(2-f ) 1 r(5) 1 5 - s(l+2) " z
l+z) 2 + z) j

and deduce that, when x + r> 0,

  Ina m +* ) \ f \ 4   -1), X '( -l)(- -2) \ f/-' r>) z z(z +
l) - z z + l) z + 2)

44. Using the result of example 43, prove that

logr(2 + a) = logr(0 + rtlogs

2z

fa

dt

a [\ \ { t)(2-t) ...(n-t)dt- f" t l-t) 2-t) ...(n-t)'

\ 5 .' .'o

nti n + l)z z+l) z + 2) ... z + n)

investigating the region of convergence of the .series.

(Binet, Journal de V Ecole polytechnique, xvi. (1839), p. 256.)

45. Prove that, if /> > 0, > 0, then

-. P-

B ip, q) = V- i (2 )* * "' '

%
% 264
%

where

II (p, g) = 2p - - T arc tan - - -. - -, -,

and p2 = 2 + q +pq-

46. If 6 =2*-'7r(l-i'r), F=2 - 7r(i-iA'),

and if the function F (,r) be defined by the equation

shew (1) that F o:) satisfies the equation

F x+ ) =xF x) +

r(l-A-)'

(2) that, for all positive integral values of x,

F x) = rix\

(3) that F(x) is analytic for all finite values of x,

1 (7 \ 2

(4) that "- "* '

47. Expand

F(x)= ~- r -r- log- 7

as a series of ascending powei's of a.

(Various evaluations of the coefficients in this expansion have been
given by Bourguet, Bull des Set. Math. v. (1881), p. 43; Bourguet,
Acta Math. ll. (1883), p. 261; Schlomilch,' Zeitschrift fiir Math,
und Phys. xxv. (1880), pp. 35, 351.)

48. Prove that the G-function, defined by the equation

G(z + ) = (2nf'e- ' + - n |(i+i)%-'+-'- /(2")|,

is an integral function which satisfies the relations

0 z + ) = V z)G z), (?(1) = 1,

(n !)'V6-' (n + 1) = 11 . 22 . 33 ... w". \addexamplecitation{Alexeiewsky.}

(The most important properties of the G-function are discussed in
Barnes' memoir, Quarterly Journal, xxxi.)

49. Shew that and deduce that

log Q J = / s cot Tvzdz-z log (27r).

50. Shew that

logrri + l)c <=i3log(2rr)-|3(2+i;+3logr(s + l)-logG'(s + l).


\chapter{Some Properties of Trees}
\label{sct_defining_clades}

\begin{figure}[t]
\centering
\includegraphics{oriented_graph.png}
 % oriented_graph.png: 485x393 pixel, 300dpi, 4.11x3.33 cm, bb=0 0 116 94
 \caption{An oriented graph (not a tree!), with edge orientation shown as arrows. Two paths are highlighted (cyan and blue). The cyan path starts at node A and ends at node E. Nodes B, C, D, and E are reachable from A. Node C is the direct successor of B in the cyan path; in the blue path the direct successor of B is F. The segment highlighted in red is \emph{not} a path.}
 \label{fig_oriented_graph}
\end{figure}

In the following, $T$ will be a rooted tree. We'll recall that a tree (rooted or not) is a connected, acyclic graph. By choosing one node to be the root (which we'll note $R$), and orienting all edges away from the root, we get an oriented graph. A \textit{path} along such a graph is a sequence of nodes such that there is an edge between two successive nodes, and that these nodes are ordered in the sequence as they are in the edge. The first node in the path is called the \textit{start} of the path, and the last node is its \textit{end}. We speak of a path \textit{from} the start \textit{to} the end, equivalently we say that the end is \textit{reachable} from the start. The nodes in a path are ordered: if $a$ and $b$ are in a path, then either $a$ is reachable from $b$, or $b$ is reachable from $a$, or $a = b$; furthermore if $a$ is reachable from $b$, then $b$ is \textit{not} reachable from $a$. If an edge connects $a$ to $b$, then $b$ is the \textit{direct successor} of $a$, and $a$ is the \textit{direct predecessor} of $b$
See figure {\ref{fig_oriented_graph}.

Although paths are not sets, we will (somewhat abusively, perhaps) use set notation with paths, unless there is a risk of confusion. Thus is $P$ is a path and $n$ is a node, then $n \in P$ means that $n$ is ``in'' $P$  or ``belongs to'' $P$ (strictly speaking, $n$ is reachable from the start of $P$, and the end of $P$ is reachable from $n$).

Now a few definitions - these are mostly renamings of graph-theoretical concepts, to make them more intuitive for phylogenetic trees.

\begin{dfn}
\label{def_ancestor}
Let $a$ and $b$ be two different nodes of $T$. We say that $a$ is an \textit{ancestor} of $b$, or (equivalently) that $b$ is a \textit{descendant} of $a$, if and only if there is a path from $a$ to $b$.
\end{dfn}

\begin{dfn}
\label{def_parent}
Let $P$ be a path in $T$, and $a, b$ two nodes in $P$ such that $b$ is the direct successor of $a$. We will call $a$ the \textit{parent} of $b$. We can say \textit{the} parent (rather than \textit{a} parent) of $b$ because i) $a$ lies on the path from $R$ to $b$ (which is unique), and ii) only one node on any path can be another node's direct predecessor. We also see that any node's parent is also its ancestor (which shouldn't be too surprising\ldots), since $b$ is reachable from $a$
\end{dfn}

\begin{dfn}
\label{def_child}
Let $P$ be a path in $T$, and $a, b$ two nodes in $P$ such that $b$ is the direct successor of $a$. We say that $b$ is a \textit{child} of $a$.
\end{dfn}

Note that a node has exactly one parent, except the root which has none. A node can have zero or more children, but we rarely ever encounter nodes with a single child. A node with no children is called a \textit{leaf}.

\begin{dfn}
\label{def_lineage}
A path that starts from the root we call a \emph{lineage}. By the definition of a rooted tree, there is always a lineage to $n$ for any node $n$, and this lineage is unique. So we can unambiguously talk of \textit{the} lineage of $n$ to mean the path from $R$ to $n$. We say that a set $L$ of nodes form a lineage if there exists $n \in L$ such that the path from $R$ to $n$ contains all and only elements of $L$.
\end{dfn}

\begin{figure}[b]
 \centering
 \includegraphics{prop_tree.png}
  \caption{A rooted tree}
 \label{fig_app_tree_prop}
\end{figure}

\begin{prop}
\label{prop_lineage_ancestors_equivalence}
Let $n$ be a node of $T$, and $L$ its lineage. Then all ancestors of $n$ belong to $L$, and all nodes of $L$, except $n$ itself, are ancestors of $n$.
\end{prop}

\begin{proof}

$\Rightarrow$ Let $a$ be an ancestor of $n$. By definition \ref{def_ancestor}, there is a path from $a$ to $n$. By tree properties, there is a path from $R$ to $a$. Hence $a$ belongs to the path from $R$ to $n$, which is the lineage of $n$.
\end{proof}

\noindent{}$\Leftarrow$ Let $l \neq n$ be a node in $L$. Since $L$ is the path from $R$ to $n$, $n$ is reachable from $a$, and hence $a$ is an ancestor of $n$ by definition \ref{def_ancestor}.

\begin{dfn}
If $L$ and $M$ are lineages of $T$, then the \textit{intersection}
of $L$ and $M$, noted $L \cap M$, is the set of nodes that belong to both $L$ and $M$.
\end{dfn}

\begin{lemma}
\label{lem_ancestors_in_lineage}
Let $n$ be a node of $T$, $L$ its lineage, and $l$ a node of $L$. Then $l$'s ancestors also belong to $L$.
\end{lemma}
\begin{proof}
If $l = n$ then the ancestors of $l$ are those of $n$, which belong to $L$ by proposition \ref{prop_lineage_ancestors_equivalence}. If $l \neq n$ then let $a$ be an ancestor of $l$. By definition \ref{def_ancestor}, $l$ is reachable from $a$. By proposition \ref{prop_lineage_ancestors_equivalence}, $l$ is an ancestor of $n$ and again $n$ is reachable from $l$. Therefore, $n$ is reachable from $a$, so $a$ is an ancestor of $n$, and so belongs in $L$.
\end{proof}

\begin{lemma}
Let $L$ and $M$ be lineages of $T$, and $q \in L \cap M, q \neq R$. Then $q$'s parent also belongs to $M \cap L$.
\end{lemma}
\begin{proof}
Call $p$ the parent of $q$. $p$ is an ancestor of $q$, and since $q$ belongs to $L$ (by hypothesis), $p$ belongs to $L$ by lemma \ref{lem_ancestors_in_lineage}. By the same reasoning, $p$ belongs to $M$, therefore $p \in L \cap M$.

\end{proof}

\begin{prop}
\label{prop_lineage_intersection}
Let $L$ and $M$ be lineages of $T$. Then $L \cap M$ also forms a lineage.
\end{prop}
\begin{proof}
Let $N = L \cap M$. Because $N$ is a subset of a lineage (in fact it is a subset of at least two lineages, $L$ and $M$), its elements are ordered, so we can pick the last one and call it $n$. Now all other elements of $N$ are ancestors of $n$, so by lemma \ref{lem_ancestors_in_lineage} they belong in $L$ and in $M$. So far we have proven
\end{proof}

\begin{prop}
In a rooted tree, you can always define a clade by specifying a single node: the clade consists of the specified node and all its descendants.
\end{prop}

\begin{prop}
In a rooted tree $T$ with unique labels and a node $n$ in $T$, it is always possible to choose two nodes $n_1$ and $n_2$ such that lca($n_1$,$n_2$) = $n$.
\end{prop}