\subsection{Line Integrals of Scalar Functions}
\noindent
Let's say we have a a simple (non-self-intersecting) curve in the xy-plane $C$ parameterized by $\vec{r}(t) = \langle x(t), y(t)\rangle$ and a surface $z = f(x,y)$ above $C$. We can extrude $C$ up to $f$, forming a "curtain" with an area $A$ that can be found through integration.
\begin{align*}
	\mathrm{d}A &= f(x,y) \cdot \mathrm{d}s	\\
	s &= \int{\norm{\vec{r^\prime}(t)}\mathrm{d}t} \\
	\mathrm{d}s &= \norm{\vec{r'}(t)}\mathrm{d}t \\
	\implies A &= \int\limits_{C}{f(x,y)\mathrm{d}s} = \int{(f\circ\vec{r})(t) \cdot \norm{\vec{r^\prime}(t)}\mathrm{d}t}
\end{align*}
This is the line integral of $\vec{r}$ on $f$. $(f\circ\vec{r})$ is called the "pullback."

\noindent
For example, let's find the line integral of $y = x^2$ for $x \leq x \leq \sqrt{2}$ in $f(x,y) = 2x$.
\begin{equation*}
	\vec{r}(t) = \langle t, t^2 \rangle, 0 \leq t \leq \sqrt{2}
\end{equation*}
\begin{equation*}
	\int\limits_{C}{f(x,y)\mathrm{d}s} = \int_{0}^{\sqrt{2}}{(2x \circ \langle t, t^2\rangle) \cdot \norm{\vec{r^\prime}(t)}\mathrm{d}t} = \int_{0}^{\sqrt{2}}{2t\sqrt{1 + 4t^2}\mathrm{d}t}
\end{equation*}
Let $u = 1+4t^2$, $\mathrm{d}u = 8t\mathrm{d}t$.
\begin{equation*}
	= \frac{1}{4}\int_{1}^{9}{\sqrt{u}\mathrm{d}u} = \frac{13}{3}
\end{equation*}


\subsection{Existence of an infinite number of prime numbers}

\subsubsection{Existence of an infinite number of prime numbers}

If there are a finite number of primes, we can call the set of primes \(P\).

We identify a new natural number \(a\) by taking the product of existing primes and adding \(1\).

\(a=1+\prod_{p\in P} p\)

From the fundamental theorem of arithmetic we know all numbers are primes or the products of primes.

If \(a\) is not a prime then it can be divided by one of the existing primes to form number \(n\):

\(\dfrac{\prod^n p_i +1}{p_j}=n\)

\(\dfrac{p_j \prod^n_{i\ne j} p_i +1}{p_j}=n\)

\(\prod^n_{i\ne j} p_i +\dfrac{1}{p_j}=n\)

As this is not a whole number, \(n\) must prime.

We can do this process for any finite number of primes, so there are an infinite number.


\subsection{Fields}

A field is a ring where the multiplication function has an inverse.

The integers, addition and multiplication form a ring, but not a group.

The rational numbers (except \(0\)), addition and multiplication form a field (and a ring).

The real numbers and complex numbers also form fields.


\section{Matrices}

\subsection{Exercise 1}
$(P + T)_{ij} = ((P + T)e_j)_i = (Pe_j + Te_j)_i = P_{ij} + T_{ij} $.

\subsection{Exercise 2}
Represent $A$ as a column of row vectors $A_i$ and $B$ as a row of column
vectors $B_i$. Denote blocks by parenthesized subscripts.
Then the first block of $AB$ looks like:
\begin{align*}
        (AB)_{(11)} &=
    \begin{pmatrix}
            A_1B_1 & & \\
            & \ddots & \\
            & & A_kB_k
    \end{pmatrix}
    \\
    &=
    \begin{pmatrix}
            A_{1, :(k+1)} B_{1, :(k+1)} & & \\
            & \ddots & \\
            & & A_{k, :(k+1)} B_{k, :(k+1)}
    \end{pmatrix}
    \\
    &+
    \begin{pmatrix}
            A_{1, (k+1):} B_{1, (k+1):} & & \\
            & \ddots & \\
            & & A_{k, (k+1):} B_{k, (k+1):}
    \end{pmatrix}
    \\
    &=
    A_{(11)}B_{(11)} + A_{(12)}B_{(21)}
\end{align*}

Where:
\begin{align*}
        A_{i, :(k+1)} B_{i, :(k+1)} &= \sum_{j=1}^{k} A_{i, j} B_{i, j} \\
        A_{i, (k+1):} B_{i, (k+1):} &= \sum_{j=k+1}^{n} A_{i, j} B_{i, j}
\end{align*}
The rest follow similarly.


\section{Non Compact Projection}%
\label{sec:ncp}
Recall the expression for the \(x\) update in the augmented L2HMC sampler,
%
\begin{align}
  x^{\prime} &= m^{t}\odot x + \bar{m}^{t}\odot\big[x\odot\exp(\varepsilon
           S_{x}(\zeta^{t}_{2})) + \varepsilon * \big(v \odot \exp(\varepsilon
           Q_x(\zeta^{t}_{2})) + T_{x}(\zeta^{t}_{2})\big)\big]\\
        &= m^{t}\odot x + \bar{m}^{t}\odot\left[\alpha x + \beta\right]
        % &= m^{t}\odot x + \bar{m}^{t}\odot g(x)
\end{align}
%
where \(\alpha = \exp(\varepsilon S_{x}), \beta =
\varepsilon\cdot\left(v\odot\exp(\varepsilon Q_{x}) + T_{x}\right)\), depend on
\(\zeta_{2} = (m^{t}\odot x, v, t)\), but are independent of \(\bar{m}^{t}\odot
x\).

Let \(z = h(x): [-\pi, \pi] \rightarrow \mathbb{R}\) be given by
%
\begin{equation}
  z = \tan\left(\frac{x}{2}\right).
\end{equation}
%
We can perform the affine transformation in \(\mathbb{R}\) and then project
back to \([-\pi, \pi]\) via \(x = h^{-1}(z)\) to complete the
update:
%
% &= m^{t}\odot x + \bar{m}^{t}\odot h^{-1}\left(g\left(h(x)\right)\right) \\
\begin{align}
  x^{\prime} &= m^{t}\odot x + \bar{m}^{t}\odot \left[2\tan^{-1}\left(\alpha
           \tan\left(\frac{x}{2}\right)\right) + \beta\right]
           %   &= m^{t}\odot x +
           %   \bar{m}^{t}\odot\big[2\tan^{-1}\left(\tan{\left(\frac{x}{2}\right)}\odot\exp(\varepsilon
           %       S_{x}(\zeta^{t}_{2}))\right) + \varepsilon * \big(v \odot \exp(\varepsilon
           % Q_x(\zeta^{t}_{2})) + T_{x}(\zeta^{t}_{2})\big)\big]
\end{align}
%
With Jacobian factor
%
\begin{align}
  \frac{\partial x^{\prime}}{\partial x}
  &= \frac{\alpha}{\cos^{2}(\frac{x}{2})+\alpha^{2}\sin^{2}(\frac{x}{2})} \\
  &= \frac{\exp{(\eps S_{x})}}{\cos^{2}{(\frac{x}{2})}
  + \exp{(2\cdot\eps S_{x})} \sin{(\frac{x}{2})}}
\end{align}


\subsection{Leontief production function}

\(Q=\sum_i X_i^{\alpha_i} \)


% -*- root: Main.tex -*-
\section{Essentials}
\subsection*{Matrix/Vector}
\begin{compactdesc}
	\item[Orthogonal:] (i.e. columns are orthonormal!) $\mathbf{A}^{-1} = \mathbf{A}^\top$, $\mathbf{A} \mathbf{A}^\top = \mathbf{A}^\top \mathbf{A} = \mathbf{I}$, $\operatorname{det}(\mathbf{A}) \in \{+1, -1\}$, $\operatorname{det}(\mathbf{A}^\top \mathbf{A}) = 1$
	\item[Inner Product:] $\langle \mathbf{x}, \mathbf{y} \rangle = \mathbf{x}^\top \mathbf{y} = \sum_{i=1}^{N} \mathbf{x}_i \mathbf{y}_i$.
	\begin{inparaitem}
		\item $\langle \mathbf{x} \pm \mathbf{y}, \mathbf{x} \pm \mathbf{y} \rangle = \langle \mathbf{x}, \mathbf{x} \rangle \pm 2 \langle \mathbf{x}, \mathbf{y} \rangle + \langle \mathbf{y}, \mathbf{y} \rangle$
		\item $\langle \mathbf{x}, \mathbf{y} + \mathbf{z} \rangle = \langle \mathbf{x}, \mathbf{y} \rangle + \langle \mathbf{x}, \mathbf{z} \rangle$
		\item $\langle \mathbf{x} + \mathbf{y}, \mathbf{z} \rangle = \langle \mathbf{x}, \mathbf{z} \rangle + \langle \mathbf{y}, \mathbf{z} \rangle$
		\item $\langle \mathbf{x}, \mathbf{y} \rangle = \|\mathbf{x}\|_2 \cdot \|\mathbf{y}\|_2 \cdot \cos(\theta)$
		\item If $\mathbf{y}$ is a unit vector then $\langle \mathbf{x}, \mathbf{y} \rangle$ projects $\mathbf{x}$ onto $\mathbf{y}$
	\end{inparaitem}
	\item[Outer Product:] $\mathbf{u} \mathbf{v}^\top$, $(\mathbf{u} \mathbf{v}^\top)_{i, j} = \mathbf{u}_i \mathbf{v}_j$
	\item[Transpose:] $(\mathbf{A}^\top)^{-1} = (\mathbf{A}^{-1})^\top$,  $(\mathbf{A}\mathbf{B})^\top= \mathbf{B}^\top\mathbf{A}^\top$, $(\mathbf{A}+\mathbf{B})^\top= \mathbf{A}^\top + \mathbf{B}^\top$
	\item[Cross product:] $\vec{a}\times\vec{b}=(a_2b_3-a_3b_2, a_3b_1-a_1b_3, a_1b_2-a_2b_1)^\top$
\end{compactdesc}

\subsection*{Norms}
\begin{inparaitem}
	\item $\|\mathbf{x}\|_0 = |\{i | x_i \neq 0\}|$
	\item $\|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^{N} \mathbf{x}_i^2} = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle}$
	\item $\|\mathbf{u}-\mathbf{v}\|_2 = \sqrt{(\mathbf{u}-\mathbf{v})^\top(\mathbf{u}-\mathbf{v})}$
	\item $\|\mathbf{x}\|_p = \left( \sum_{i=1}^{N} |x_i|^p \right)^{\frac{1}{p}}$
	\item $\mathbf{A} \in \mathbb{R}^{N \times N}, \|\mathbf{A}\|_F =\allowbreak \sqrt{trace(\mathbf{A}\mathbf{A}^\top)}$
	\item
	$\|\mathbf{M}\|_F =\allowbreak \sqrt{\sum_{i=1}^{m} \sum_{j=1}^{n}\mathbf{m}_{i,j}^2} =\allowbreak \sqrt{\sum_{i=1}^{\min\{m, n\}} \sigma_i^2}$
	\item
	$\|\mathbf{M}\|_1 = \sum_{i,j} | m_{i,j}|$
	\item $\|\mathbf{M}\|_2 = \sigma_{\text{max}}(\mathbf{M})$\\
	$\|\mathbf{M}\|_p = \max_{\mathbf{v} \neq 0} \frac{\|\mathbf{M}\mathbf{v}\|_p}{\|\mathbf{v}\|_p}$
	\item $\|\mathbf{M}\|_\star = \sum_{i=1}^{\min(m, n)} \sigma_i$
\end{inparaitem}

\subsection*{Derivatives}
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{b}^\top \mathbf{x}) = \frac{\partial}{\partial \mathbf{x}}(\mathbf{x}^\top \mathbf{b}) = \mathbf{b}$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}^\top \mathbf{x}) = 2\mathbf{x}$\\
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}^\top \mathbf{A}\mathbf{x}) = (\mathbf{A}^\top + \mathbf{A})\mathbf{x}$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{b}^\top \mathbf{A}\mathbf{x}) = \mathbf{A}^\top \mathbf{b}$\\
$\frac{\partial}{\partial \mathbf{X}}(\mathbf{c}^\top \mathbf{X} \mathbf{b}) = \mathbf{c}\mathbf{b}^\top$ \quad
$\frac{\partial}{\partial \mathbf{X}}(\mathbf{c}^\top \mathbf{X}^\top \mathbf{b}) = \mathbf{b}\mathbf{c}^\top$\\
$\frac{\partial}{\partial \mathbf{x}}(\| \mathbf{x}-\mathbf{b} \|_2) = \frac{\mathbf{x}-\mathbf{b}}{\|\mathbf{x}-\mathbf{b}\|_2}$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\|\mathbf{x}\|^2_2) = \frac{\partial}{\partial \mathbf{x}} (\mathbf{x}^\top \mathbf{x}) = 2\mathbf{x}$\\
$\frac{\partial}{\partial \mathbf{X}}(\|\mathbf{X}\|_F^2) = 2\mathbf{X}$ \quad
$\frac{\partial}{\partial \mathbf{x}}\log(x) = \frac{1}{x}$

\subsection*{Eigenvalue / -vectors}
Eigenvalue Problem: $\mathbf{Ax} = \lambda \mathbf{x}$\\
1. solve $\operatorname{det}(\mathbf{A} - \lambda \mathbf{I}) \overset{!}{=} 0$ resulting in $\{\lambda_i\}_i$\\
2. $\forall \lambda_i$:
solve $(\mathbf{A} - \lambda_i \mathbf{I}) \mathbf{x}_i = \mathbf{0}$, for $\mathbf{x}_i$.

\subsection*{Eigendecomposition}
$\mathbf{A} \in \mathbb{R}^{N \times N}$ then $\mathbf{A} = \mathbf{Q} \boldsymbol{\Lambda} \mathbf{Q}^{-1}$ with $\mathbf{Q} \in \mathbb{R}^{N \times N}$.\\
if fullrank: $\mathbf{A}^{-1} = \mathbf{Q} \boldsymbol{\Lambda}^{-1} \mathbf{Q}^{-1}$ and $(\boldsymbol{\Lambda}^{-1})_{i,i} = \frac{1}{\lambda_i}$.\\
if $\mathbf{A}$ symmetric: $A = \mathbf{Q} \boldsymbol{\Lambda} \mathbf{Q^\top}$ ($\mathbf{Q}$ orthogonal).

\subsection*{Probability / Statistics}
\begin{inparaitem}
	\item $P(x) := Pr[X = x] := \sum_{y \in Y} P(x, y)$
	\item $P(x|y) := Pr[X = x | Y = y] := \frac{P(x,y)}{P(y)},\quad \text{if } P(y) > 0$
	\item $\forall y \in Y: \sum_{x \in X} P(x|y) = 1$ (property for any fixed $y$)
	\item $P(x, y) = P(x|y) P(y)$
	\item $P(x|y) = \frac{P(y|x)P(x)}{P(y)}$ (Bayes' rule)
	\item $P(x|y) = P(x) \Leftrightarrow P(y|x) = P(y)$ (iff $X$, $Y$ independent)
	\item $P(x_1, \ldots, x_n) = \prod_{i=1}^n P(x_i)$ (iff IID)
	\item Variance $Var[X]:= E[(X-\mu_x)^2]:=\sum_{x \in X}(x-\mu_x)^2P(x)$
	\item expectation $\mu_x := E[X]:=\sum_{x \in X}xP(x)$
	\item standard deviation $\sigma_x := \sqrt{Var[X]}$
\end{inparaitem}


\section*{Singular Value Decomposition}
$\mathbf{A} = \mathbf{U} \mathbf{D} \mathbf{V}^\top = \sum_{k=1}^{\operatorname{rank}(\mathbf{A})} d_{k,k} u_k (v_k)^\top$\\
$\mathbf{A} \in \mathbb{R}^{N \times P}, \mathbf{U} \in \mathbb{R}^{N \times N}, \mathbf{D} \in \mathbb{R}^{N \times P}, \mathbf{V} \in \mathbb{R}^{P \times P}$\\
$\mathbf{U}^\top \mathbf{U} = I = \mathbf{V}^\top \mathbf{V}$ ($\mathbf{U}, \mathbf{V}$ orthonormal)\\
$\mathbf{U}$: cols are eigenvectors of $\mathbf{A} \mathbf{A}^\top$, $\mathbf{V}$: cols are eigenvectors of $\mathbf{A}^\top \mathbf{A}$, $\mathbf{D}$ diag. el. are singular values.

1. calculate $\mathbf{A}^\top \mathbf{A}$.\\
2. calculate eigenvalues of $\mathbf{A}^\top \mathbf{A}$, the square root of them, in desc. order, are the diagonal elements of $\mathbf{D}$.\\
3. calculate eigenvectors of $\mathbf{A}^\top \mathbf{A}$ using the eigenvalues resulting in the columns of $\mathbf{V}$.\\
4. calculate the missing matrix: $\mathbf{U} = \mathbf{A} \mathbf{V} \mathbf{D}^{-1}$.\\
5. normalize each column of $\mathbf{U}$ and $\mathbf{V}$.\\
Complexity: $O(\min(mn^2, nm^3))$

\subsection*{Eckart-Young Theorem}
$\min_{rank(B)=K} ||A-B||_F^2 = ||A-A_k||_F^2 = \sum_{r=k+1}^{rank(A)} \sigma_r^2$
$\min_{rank(B)=K} ||A-B||_2 = ||A-A_k||_2 = \sigma_{k+1}$


\subsection{K-Means Clustering}

\noindent{\bf Description}
\smallskip

Given a collection of $n$ records with a pairwise similarity measure,
the goal of clustering is to assign a category label to each record so that
similar records tend to get the same label.  In contrast to multinomial
logistic regression, clustering is an \emph{unsupervised}\/ learning problem
with neither category assignments nor label interpretations given in advance.
In $k$-means clustering, the records $x_1, x_2, \ldots, x_n$ are numerical
feature vectors of $\dim x_i = m$ with the squared Euclidean distance
$\|x_i - x_{i'}\|_2^2$ as the similarity measure.  We want to partition
$\{x_1, \ldots, x_n\}$ into $k$ clusters $\{S_1, \ldots, S_k\}$ so that
the aggregated squared distance from records to their cluster means is
minimized:
\begin{equation}
\textrm{WCSS}\,\,=\,\, \sum_{i=1}^n \,\big\|x_i - \mean(S_j: x_i\in S_j)\big\|_2^2 \,\,\to\,\,\min
\label{eqn:WCSS}
\end{equation}
The aggregated distance measure in~(\ref{eqn:WCSS}) is called the
\emph{within-cluster sum of squares}~(WCSS).  It can be viewed as a measure
of residual variance that remains in the data after the clustering assignment,
conceptually similar to the residual sum of squares~(RSS) in linear regression.
However, unlike for the RSS, the minimization of~(\ref{eqn:WCSS}) is an NP-hard
problem~\cite{AloiseDHP2009:kmeans}.

Rather than searching for the global optimum in~(\ref{eqn:WCSS}), a heuristic algorithm
called Lloyd's algorithm is typically used.  This iterative algorithm maintains
and updates a set of $k$~\emph{centroids} $\{c_1, \ldots, c_k\}$, one centroid per cluster.
It defines each cluster $S_j$ as the set of all records closer to~$c_j$ than
to any other centroid.  Each iteration of the algorithm reduces the WCSS in two steps:
\begin{Enumerate}
\item Assign each record to the closest centroid, making $\mean(S_j)\neq c_j$;
\label{step:kmeans:recluster}
\item Reset each centroid to its cluster's mean: $c_j := \mean(S_j)$.
\label{step:kmeans:recenter}
\end{Enumerate}
After Step~\ref{step:kmeans:recluster} the centroids are generally different from the cluster
means, so we can compute another ``within-cluster sum of squares'' based on the centroids:
\begin{equation}
\textrm{WCSS\_C}\,\,=\,\, \sum_{i=1}^n \,\big\|x_i - \mathop{\textrm{centroid}}(S_j: x_i\in S_j)\big\|_2^2
\label{eqn:WCSS:C}
\end{equation}
This WCSS\_C after Step~\ref{step:kmeans:recluster} is less than the means-based WCSS
before Step~\ref{step:kmeans:recluster} (or equal if convergence achieved), and in
Step~\ref{step:kmeans:recenter} the WCSS cannot exceed the WCSS\_C for \emph{the same}
clustering; hence the WCSS reduction.

Exact convergence is reached when each record becomes closer to its
cluster's mean than to any other cluster's mean, so there are no more re-assignments
and the centroids coincide with the means.  In practice, iterations may be stopped
when the reduction in WCSS (or in WCSS\_C) falls below a minimum threshold, or upon
reaching the maximum number of iterations.  The initialization of the centroids is also
an important part of the algorithm.  The smallest WCSS obtained by the algorithm is not
the global minimum and varies depending on the initial centroids.  We implement multiple
parallel runs with different initial centroids and report the best result.

\Paragraph{Scoring}
Our scoring script evaluates the clustering output by comparing it with a known category
assignment.  Since cluster labels have no prior correspondence to the categories, we
cannot count ``correct'' and ``wrong'' cluster assignments.  Instead, we quantify them in
two ways:
\begin{Enumerate}
\item Count how many same-category and different-category pairs of records end up in the
same cluster or in different clusters;
\item For each category, count the prevalence of its most common cluster; for each
cluster, count the prevalence of its most common category.
\end{Enumerate}
The number of categories and the number of clusters ($k$) do not have to be equal.
A same-category pair of records clustered into the same cluster is viewed as a
``true positive,'' a different-category pair clustered together is a ``false positive,''
a same-category pair clustered apart is a ``false negative''~etc.

\smallskip
\noindent{\bf Usage: K-means Script}
\smallskip

{\hangindent=\parindent\noindent\it%
{\tt{}-f }path/\/{\tt{}Kmeans.dml}
{\tt{} -nvargs}
{\tt{} X=}path/file
{\tt{} C=}path/file
{\tt{} k=}int
{\tt{} runs=}int
{\tt{} maxi=}int
{\tt{} tol=}double
{\tt{} samp=}int
{\tt{} isY=}int
{\tt{} Y=}path/file
{\tt{} fmt=}format
{\tt{} verb=}int

}

\smallskip
\noindent{\bf Usage: K-means Scoring/Prediction}
\smallskip

{\hangindent=\parindent\noindent\it%
{\tt{}-f }path/\/{\tt{}Kmeans-predict.dml}
{\tt{} -nvargs}
{\tt{} X=}path/file
{\tt{} C=}path/file
{\tt{} spY=}path/file
{\tt{} prY=}path/file
{\tt{} fmt=}format
{\tt{} O=}path/file

}

\smallskip
\noindent{\bf Arguments}
\begin{Description}
\item[{\tt X}:]
Location to read matrix $X$ with the input data records as rows
\item[{\tt C}:] (default:\mbox{ }{\tt "C.mtx"})
Location to store the output matrix with the best available cluster centroids as rows
\item[{\tt k}:]
Number of clusters (and centroids)
\item[{\tt runs}:] (default:\mbox{ }{\tt 10})
Number of parallel runs, each run with different initial centroids
\item[{\tt maxi}:] (default:\mbox{ }{\tt 1000})
Maximum number of iterations per run
\item[{\tt tol}:] (default:\mbox{ }{\tt 0.000001})
Tolerance (epsilon) for single-iteration WCSS\_C change ratio
\item[{\tt samp}:] (default:\mbox{ }{\tt 50})
Average number of records per centroid in data samples used in the centroid
initialization procedure
\item[{\tt Y}:] (default:\mbox{ }{\tt "Y.mtx"})
Location to store the one-column matrix $Y$ with the best available mapping of
records to clusters (defined by the output centroids)
\item[{\tt isY}:] (default:\mbox{ }{\tt 0})
{\tt 0} = do not write matrix~$Y$,  {\tt 1} = write~$Y$
\item[{\tt fmt}:] (default:\mbox{ }{\tt "text"})
Matrix file output format, such as {\tt text}, {\tt mm}, or {\tt csv};
see read/write functions in SystemML Language Reference for details.
\item[{\tt verb}:] (default:\mbox{ }{\tt 0})
{\tt 0} = do not print per-iteration statistics for each run, {\tt 1} = print them
(the ``verbose'' option)
\end{Description}
\smallskip
\noindent{\bf Arguments --- Scoring/Prediction}
\begin{Description}
\item[{\tt X}:] (default:\mbox{ }{\tt " "})
Location to read matrix $X$ with the input data records as rows,
optional when {\tt prY} input is provided
\item[{\tt C}:] (default:\mbox{ }{\tt " "})
Location to read matrix $C$ with cluster centroids as rows, optional
when {\tt prY} input is provided; NOTE: if both {\tt X} and {\tt C} are
provided, {\tt prY} is an output, not input
\item[{\tt spY}:] (default:\mbox{ }{\tt " "})
Location to read a one-column matrix with the externally specified ``true''
assignment of records (rows) to categories, optional for prediction without
scoring
\item[{\tt prY}:] (default:\mbox{ }{\tt " "})
Location to read (or write, if {\tt X} and {\tt C} are present) a
column-vector with the predicted assignment of rows to clusters;
NOTE: No prior correspondence is assumed between the predicted
cluster labels and the externally specified categories
\item[{\tt fmt}:] (default:\mbox{ }{\tt "text"})
Matrix file output format for {\tt prY}, such as {\tt text}, {\tt mm},
or {\tt csv}; see read/write functions in SystemML Language Reference
for details
\item[{\tt O}:] (default:\mbox{ }{\tt " "})
Location to write the output statistics defined in
Table~\ref{table:kmeans:predict:stats}, by default print them to the
standard output
\end{Description}

\begin{table}[t]\small\centerline{%
\begin{tabular}{|lcl|}
\hline
Name & CID & Meaning \\
\hline
{\tt TSS}             &     & Total Sum of Squares (from the total mean) \\
{\tt WCSS\_M}         &     & Within-Cluster  Sum of Squares (means as centers) \\
{\tt WCSS\_M\_PC}     &     & Within-Cluster  Sum of Squares (means), in \% of TSS \\
{\tt BCSS\_M}         &     & Between-Cluster Sum of Squares (means as centers) \\
{\tt BCSS\_M\_PC}     &     & Between-Cluster Sum of Squares (means), in \% of TSS \\
\hline
{\tt WCSS\_C}         &     & Within-Cluster  Sum of Squares (centroids as centers) \\
{\tt WCSS\_C\_PC}     &     & Within-Cluster  Sum of Squares (centroids), \% of TSS \\
{\tt BCSS\_C}         &     & Between-Cluster Sum of Squares (centroids as centers) \\
{\tt BCSS\_C\_PC}     &     & Between-Cluster Sum of Squares (centroids), \% of TSS \\
\hline
{\tt TRUE\_SAME\_CT}  &     & Same-category pairs predicted as Same-cluster, count \\
{\tt TRUE\_SAME\_PC}  &     & Same-category pairs predicted as Same-cluster, \% \\
{\tt TRUE\_DIFF\_CT}  &     & Diff-category pairs predicted as Diff-cluster, count \\
{\tt TRUE\_DIFF\_PC}  &     & Diff-category pairs predicted as Diff-cluster, \% \\
{\tt FALSE\_SAME\_CT} &     & Diff-category pairs predicted as Same-cluster, count \\
{\tt FALSE\_SAME\_PC} &     & Diff-category pairs predicted as Same-cluster, \% \\
{\tt FALSE\_DIFF\_CT} &     & Same-category pairs predicted as Diff-cluster, count \\
{\tt FALSE\_DIFF\_PC} &     & Same-category pairs predicted as Diff-cluster, \% \\
\hline
{\tt SPEC\_TO\_PRED}  & $+$ & For specified category, the best predicted cluster id \\
{\tt SPEC\_FULL\_CT}  & $+$ & For specified category, its full count \\
{\tt SPEC\_MATCH\_CT} & $+$ & For specified category, best-cluster matching count \\
{\tt SPEC\_MATCH\_PC} & $+$ & For specified category, \% of matching to full count \\
{\tt PRED\_TO\_SPEC}  & $+$ & For predicted cluster, the best specified category id \\
{\tt PRED\_FULL\_CT}  & $+$ & For predicted cluster, its full count \\
{\tt PRED\_MATCH\_CT} & $+$ & For predicted cluster, best-category matching count \\
{\tt PRED\_MATCH\_PC} & $+$ & For predicted cluster, \% of matching to full count \\
\hline
\end{tabular}}
\caption{The {\tt O}-file for {\tt Kmeans-predict} provides the output statistics
in CSV format, one per line, in the following format: (NAME, [CID], VALUE).  Note:
the 1st group statistics are given if {\tt X} input is available;
the 2nd group statistics are given if {\tt X} and {\tt C} inputs are available;
the 3rd and 4th group statistics are given if {\tt spY} input is available;
only the 4th group statistics contain a nonempty CID value;
when present, CID contains either the specified category label or the
predicted cluster label.}
\label{table:kmeans:predict:stats}
\end{table}

\noindent{\bf Details}
\smallskip

Our clustering script proceeds in 3~stages: centroid initialization,
parallel $k$-means iterations, and the best-available output generation.
Centroids are initialized at random from the input records (the rows of~$X$),
biased towards being chosen far apart from each other.  The initialization
method is based on the {\tt k-means++} heuristic from~\cite{ArthurVassilvitskii2007:kmeans},
with one important difference: to reduce the number of passes through~$X$,
we take a small sample of $X$ and run the {\tt k-means++} heuristic over
this sample.  Here is, conceptually, our centroid initialization algorithm
for one clustering run:
\begin{Enumerate}
\item Sample the rows of~$X$ uniformly at random, picking each row with probability
$p = ks / n$ where
\begin{Itemize}
\item $k$~is the number of centroids,
\item $n$~is the number of records, and
\item $s$~is the {\tt samp} input parameter.
\end{Itemize}
If $ks \geq n$, the entire $X$ is used in place of its sample.
\item Choose the first centroid uniformly at random from the sampled rows.
\item Choose each subsequent centroid from the sampled rows, at random, with
probability proportional to the squared Euclidean distance between the row and
the nearest already-chosen centroid.
\end{Enumerate}
The sampling of $X$ and the selection of centroids are performed independently
and in parallel for each run of the $k$-means algorithm.  When we sample the
rows of~$X$, rather than tossing a random coin for each row, we compute the
number of rows to skip until the next sampled row as $\lceil \log(u) / \log(1 - p) \rceil$
where $u\in (0, 1)$ is uniformly random.  This time-saving trick works because
\begin{equation*}
\Prob [k-1 < \log_{1-p}(u) < k] \,\,=\,\, p(1-p)^{k-1} \,\,=\,\,
\Prob [\textrm{skip $k-1$ rows}]
\end{equation*}
However, it requires us to estimate the maximum sample size, which we set
near~$ks + 10\sqrt{ks}$ to make it generous enough.

Once we selected the initial centroid sets, we start the $k$-means iterations
independently in parallel for all clustering runs.  The number of clustering runs
is given as the {\tt runs} input parameter.  Each iteration of each clustering run
performs the following steps:
\begin{Itemize}
\item Compute the centroid-dependent part of squared Euclidean distances from
all records (rows of~$X$) to each of the $k$~centroids using matrix product;
\item Take the minimum of the above for each record;
\item Update the current within-cluster sum of squares (WCSS) value, with centroids
substituted instead of the means for efficiency;
\item Check the convergence criterion:\hfil
$\textrm{WCSS}_{\mathrm{old}} - \textrm{WCSS}_{\mathrm{new}} < \eps\cdot\textrm{WCSS}_{\mathrm{new}}$\linebreak
as well as the number of iterations limit;
\item Find the closest centroid for each record, sharing equally any records with multiple
closest centroids;
\item Compute the number of records closest to each centroid, checking for ``runaway''
centroids with no records left (in which case the run fails);
\item Compute the new centroids by averaging the records in their clusters.
\end{Itemize}
When a termination condition is satisfied, we store the centroids and the WCSS value
and exit this run.  A run has to satisfy the WCSS convergence criterion to be considered
successful.  Upon the termination of all runs, we select the smallest WCSS value among
the successful runs, and write out this run's centroids.  If requested, we also compute
the cluster assignment of all records in~$X$, using integers from 1 to~$k$ as the cluster
labels.  The scoring script can then be used to compare the cluster assignment with
an externally specified category assignment.

\smallskip
\noindent{\bf Returns}
\smallskip

We output the $k$ centroids for the best available clustering, i.~e.\ whose WCSS
is the smallest of all successful runs.
The centroids are written as the rows of the $k\,{\times}\,m$-matrix into the output
file whose path/name was provided as the ``{\tt C}'' input argument.  If the input
parameter ``{\tt isY}'' was set to~{\tt 1}, we also output the one-column matrix with
the cluster assignment for all the records.  This assignment is written into the
file whose path/name was provided as the ``{\tt Y}'' input argument.
The best WCSS value, as well as some information about the performance of the other
runs, is printed during the script execution.  The scoring script {\tt Kmeans-predict}
prints all its results in a self-explanatory manner, as defined in
Table~\ref{table:kmeans:predict:stats}.

\smallskip
\noindent{\bf Examples}
\smallskip

{\hangindent=\parindent\noindent\tt
\hml -f Kmeans.dml -nvargs X=/user/biadmin/X.mtx k=5 C=/user/biadmin/centroids.mtx fmt=csv

}

{\hangindent=\parindent\noindent\tt
\hml -f Kmeans.dml -nvargs X=/user/biadmin/X.mtx k=5 runs=100 maxi=5000
tol=0.00000001 samp=20 C=/user/biadmin/centroids.mtx isY=1 Y=/user/biadmin/Yout.mtx verb=1

}
\noindent To predict {\tt Y} given {\tt X} and {\tt C}:

{\hangindent=\parindent\noindent\tt
\hml -f Kmeans-predict.dml -nvargs X=/user/biadmin/X.mtx
         C=/user/biadmin/C.mtx prY=/user/biadmin/PredY.mtx O=/user/biadmin/stats.csv

}
\noindent To compare ``actual'' labels {\tt spY} with ``predicted'' labels given {\tt X} and {\tt C}:

{\hangindent=\parindent\noindent\tt
\hml -f Kmeans-predict.dml -nvargs X=/user/biadmin/X.mtx
         C=/user/biadmin/C.mtx spY=/user/biadmin/Y.mtx O=/user/biadmin/stats.csv

}
\noindent To compare ``actual'' labels {\tt spY} with given ``predicted'' labels {\tt prY}:

{\hangindent=\parindent\noindent\tt
\hml -f Kmeans-predict.dml -nvargs spY=/user/biadmin/Y.mtx prY=/user/biadmin/PredY.mtx O=/user/biadmin/stats.csv

}

\smallskip
\noindent{\bf References}
\begin{itemize}
\item
D.~Aloise, A.~Deshpande, P.~Hansen, and P.~Popat.
\newblock {NP}-hardness of {E}uclidean sum-of-squares clustering.
\newblock {\em Machine Learning}, 75(2):245--248, May 2009.
\item
D.~Arthur and S.~Vassilvitskii.
\newblock {\tt k-means++}: The advantages of careful seeding.
\newblock In {\em Proceedings of the 18th Annual {ACM-SIAM} Symposium on
  Discrete Algorithms ({SODA}~2007)}, pages 1027--1035, New Orleans~{LA},
  {USA}, January 7--9 2007.
\end{itemize}


\subsection{Domains and ranges}

\subsubsection{Domain}

All values on which the function can be called

\(\forall x(f(x)=y)\rightarrow P(y))\)

\subsubsection{Image}

\(\forall x((\exists y f(x)=y)\rightarrow P(y))\)

Outputs of a function.

AKA: Range

The image of \(x\) is \(f(x)\).

\subsubsection{Preimage}

The preimage of \(y\) is all \(x\) where \(f(x)=y\).

\subsubsection{Codomain}

Sometimes the image is a subset of another set. For example a function may map onto natural numbers above \(0\). Natural numbers above \(0\) would be the image, and the natural numbers would be the codomain.

\subsubsection{Example}

\(f(n)=s(n)\)

Domain is: \(\mathbb{N}\)

Codomain is also: \(\mathbb{N}\)

Image is \(\mathbb{N}\land n\ne 0\)

\subsubsection{Describing  functions}

If function \(f\) maps from set \(X\) to set \(Y\) we can write this as:

\(f:X\rightarrow Y\)


\pagebreak
\null
\newpage
\section{Appendix}

In this section we provide the necessary definitions and mathematical concepts necessary to properly introduce the weak formulation of a differential problem, the Galerkin method and finally the Finite Element Method.\\

\paragraph{Linear operators, functionals and bilinear forms} A \textit{linear operator} $L: X\to X$ from the Hilbert space $(X, \norm \cdot)$ to itself is a map such that $L(\alpha x + \beta y) = \alpha L x + \beta L y$. A linear operator on a Hilbert space is \textit{continuous} (or {bounded}) if $\exists M:\ |Lx|\leq Mx$. $(x, \lambda)$ are called respectively \textit{eigenvector} and \textit{eigenvalue} of the linear operator $L$ if the image of $x$ through $L$ is a rescaling of $x$ of factor $\lambda$ i.e. $Lx = \lambda x$. The operator $L$ is called \textit{self-adjoint} if $\langle Lx, y\rangle = \langle x,Ly\rangle\ \  \forall x,y \in X$. Self-adjoint operators have two important properties: their eigenvalues are real, and two eigenvectors $x, y$ associated to different eigenvalues $\lambda, \mu$ are orthogonal. Indeed
\begin{align*}
	\langle Lx, y\rangle &= \langle x,Ly\rangle,\\
	\langle \lambda x, y\rangle &= \langle x,\mu y\rangle,\\
	\lambda \langle  x, y\rangle &= \mu \langle x, y\rangle,
\end{align*}
that implies $ \langle  x, y\rangle = 0$. If the eigenvectors of a self-adjoint operator $L$ span the whole space $X$, then $L$ is called \textit{diagonalisable}.  A linear operator from a Hilbert space $X$ to $\mathbb R$ is called a linear \textit{functional} on $X$. A functional is bounded if $\exists M\geq 0: \ |Lx|\leq M\norm x$. The normed vector space made by the set of all linear bounded functionals on $X$ endowed with the norm $\norm L = sup_{v\in X}|Lv|/\norm v$ is called the \textit{dual} space of $X$ and is indicated with $X^\star$. An important property of linear functionals is that for every functional $L\in X^\star$, there exists a unique vector $u\in X$ such that
$$Lv = \langle u, v\rangle\quad \forall v\in X$$

A \textit{bilinear form} on the vector space $X$ is a map $a(\cdot, \cdot): X\times X\to F$ that is linear with respect to both arguments. It is said to be \textit{strongly coercive} if \\
$\exists\alpha>0:\ |a(x, x)|\geq \alpha \norm {x}^2$, and \textit{bounded} if $\exists M>0:\ |a(x, y)|\leq M \norm x \norm y$.

\paragraph{The Lax-Milgram theorem}
Before introducing the Galerkin method the last thing to do is to state the Lax-Milgram theorem, theorem that is the foundation of the FEM formulation.
\vspace{0.5cm}
\begin{theorem}[Lax-Milgram]
	If \(a(\cdot, \cdot)\) is a bounded and strongly coercive bilinear form on the Hilbert space \(X\),  $L\in X^\star$ is a linear bounded functional on $X$, there exist a unique solution $f$ to the following problem:
	\begin{equation}\label{eq:variational abstract problem}
	\text{Find }f\in X\text{ such that } a(f, v)=Lv \text{ for all } v \in X
	\end{equation}
	For such \(f\) one has \(\|f\| \leq \frac{1}{\alpha}\norm L\) where \(\alpha>0\) is the coercive constant \\\(a(v, v) \geq \alpha\norm v^{2} \forall v \in X.\)
\end{theorem}
\vspace{0.5cm}
The Finite Element Method takes a PDE problem in strong form (\ref{eq:strong form}), reformulates it in the equivalent \textit{weak form} (\ref{eq:variational abstract problem}) through suitable definitions of $X, a(\cdot, \cdot), L$ and finally solves it through polynomial interpolation.
\subsubsection{Weak formulation of a PDE and Galerkin Method}

Galerkin's method is a method to approximate the solution $f$ of an infinite dimensional problem of the form (\ref{eq:variational abstract problem}) with the solution $f_h$ of a finite dimensional problem. Our goal is now to explain how to write a differential problem like
\begin{align}\label{eq:strong form}
\begin{split}
&\text{Given a regular domain }\Omega\subset\mathbb R^2\text{ and } u\in C(\Omega)\text{, find }f\in C^2(\Omega) \text{ such that}\\
&\begin{cases}
-\partial_{x_1x_1}f(\mathbf x) - \partial_{x_2x_2}f(\mathbf x) = u(\mathbf x) & \mathbf x \in \Omega\\
f(\mathbf x) =  0& \mathbf x \in \partial \Omega
\end{cases}
\end{split}
\end{align}
in the form \ref{eq:variational abstract problem}. Let's multiply the differential equation times a sufficiently regular function $v$ that vanishes on $\partial \Omega$ and integrate on $\Omega$. We obtain
\begin{align}
\begin{cases}
-\int_{\mathbb S^2}\Delta f(\mathbf x)v(\mathbf x)d\mathbf x =\int_{\mathbb S^2} u(\mathbf x)v(\mathbf x)d\mathbf x &\quad \mathbf x \in \Omega\\
f(\mathbf x) =  0&\quad \mathbf x \in \partial \Omega
\end{cases}
\end{align}
 Since the contribution of both $f$ and $v$ on the border $\partial \Omega$ is zero, \\$\int_{\partial \Omega}\nabla f \cdot \mathbf nv d\sigma=0$ and integrating by parts we get

\begin{equation}\label{eq:int by parts}
	\int_\Omega \nabla f(\mathbf x)\cdot\nabla v(\mathbf x) d\mathbf x = \int_\Omega  u(\mathbf x)\cdot v(\mathbf x)d\mathbf x
\end{equation}

By defining $a(f, v):=	\int_\Omega \nabla f(\mathbf x)\cdot\nabla v(\mathbf x) d\mathbf x $ and $Lv := \int_\Omega  u(\mathbf x)\cdot v(\mathbf x)d\mathbf x$ problem \ref{eq:int by parts} can be written in the form of equation \ref{eq:variational abstract problem}. It remains to choose a Hilbert space $(X, \langle\cdot,\cdot\rangle_X)$ such that (i) $X$ is "big enough" to include those functions such that all the integrals and derivatives in the problem \ref{eq:int by parts} is well defined. This means that $X$ must include those functions $f$ such that $f, \nabla f$ are in $L^2(\Omega)$ and has to be \textit{complete} with respect to the norm induced by the scalar product $\langle\cdot,\cdot\rangle_X$. At the same time $X$ must be (ii) "small enough" to include only those functions that vanish on the boundary of $\Omega$. and (iii) the hypothesis of the Lax-Milgram theorem are satisfied, i.e. $a(\cdot, \cdot)$ is actually bounded and coercive and $L$ is bounded and linear. It turns out that such an Hilbert space exists: it is called $H^1_0(\Omega)$, it contains all the functions $v$ such that $v\in L^2(\Omega), \nabla v\in L^2(\Omega)$ and it is endowed with the scalar product
$$
\langle u, v\rangle_{H^1_0(\Omega)} = \int_\Omega u(\mathbf x)v(\mathbf x)d\mathbf x + \int_\Omega\nabla u(\mathbf x)\cdot \nabla v(\mathbf x) d\mathbf x
$$
$H^1_0$ contains only those functions that vanish on $\delta \Omega$ i.e. \\$f\in H^1_0(\Omega)\cap C(\Omega) \implies \left.f(\mathbf x)\right|_{\partial\Omega}=0$. This means that thanks to Lax-Milgram theorem the problem

\begin{equation}\label{eq:final variational form}
\begin{split}
	&\text{Given }u\in L^2(\Omega)\text{ find }f\in H^1_0(\Omega)\text{ such that }\\
	&\int_\Omega \nabla f(\mathbf x)\cdot\nabla v(\mathbf x) d\mathbf x = \int_\Omega  u(\mathbf x)\cdot v(\mathbf x)d\mathbf x\quad \forall v\in H^1_0(\Omega)
\end{split}
\end{equation}

\textit{has one and only one solution in} $H^1_0(\Omega)$. However, to have this result (existence and uniqueness of the solution) we had to pay the price of looking for the solution $f$ in $H^1_0(\Omega)$, a much bigger space than  $C^2(\Omega)$ that we had in the original, strong form of the problem. This means that our solution $f\in H^1_0(\Omega)$ to the problem (\ref{eq:final variational form}) could not be a solution to problem (\ref{eq:strong form}), because it could be not regular enough and the second derivative $\Delta f$ could not exist! Fortunately there are regularity results - that we omit here - that assure that if the forcing term $u$ is regular enough, then also the solution $f$ will be regular and thus the two formulations - strong and weak - of the problem are actually equivalent, and thus solving \ref{eq:final variational form} eventually leads to solving \ref{eq:strong form}.

Now that we know that a solution exists, we need to compute it! Computing it analytically is often impossible; Galerkin's method in a mathematical tool provide us a way to compute an approximation of the solution $f$. Take the weak problem \ref{eq:variational abstract problem}, but restrict the ambient space to be a finite dimensional subspace of $X$, say $V_h = \text{span}\{\phi_0, ..., \phi_{n-1}\}$. We write thus the Galerkin problem
\vspace{0.5cm}

\begin{equation}\label{eq:Galerkin problem}
		\text{Find }f_h\in V_h\text{ such that } a(f_h, v_h)=\langle u_h, v_h\rangle \text{ for all } u_h \in V_h
\end{equation}
\vspace{0.5cm}

The key property of the Galerkin method is that the error $f-f_h$ is orthogonal to $V_h$; thus by choosing a sequence of finite dimensional spaces that fill the original space $X$, we can get as close as we want to the continuous solution $f$.
To solve equation \ref{eq:Galerkin problem} we write $f_h, u_h, v$ as linear combinations of the basis $\{\phi_0, ..., \phi_{n-1}\}$
\begin{equation}\label{eq:basis functions}
	\begin{cases}
	f_h = f_0\phi_0 +  f_1\phi_1 + ...  f_{n-1}\phi_{n-1}\\
	u_h = u_0\phi_0 +  u_1\phi_1 + ...  u_{n-1}\phi_{n-1}\\
	v_h = v_0\phi_0 +  v_1\phi_1 + ...  v_{n-1}\phi_{n-1}
	\end{cases}
\end{equation}
 thus obtaining by linearity of the bilinear form and of the scalar product a linear system of equations in the $n$ coordinates of $f_h$
\begin{equation}\label{eq:Galerkin problem in the basis functions}
\text{Find }\mathbf f\in\mathbb R^n\text{ such that } \sum_{j=0}^{n-1}a(\phi_i, \phi_j)f_j=\sum_{j=0}^{n-1}\langle \phi_i, \phi_j\rangle u_j \text{ for all } i=0, ... n-1
\end{equation}

Defining the \textit{stiffness matrix} $(A)_{ij} = a(\phi_i, \phi_j)$ and the \textit{mass matrix} \\$(B)_{ij} = \langle \phi_i, \phi_j\rangle$ we can rewrite problem \ref{eq:Galerkin problem in the basis functions} in the following algebraic form
\begin{equation}\label{eq:Galerkin algebraic}
\text{Find }\mathbf f\in\mathbb R^n\text{ such that } A\mathbf f = B \mathbf u
\end{equation}
where $\mathbf f, \mathbf u$ are the vectors of coordinates of $f_h, u_h$ with respect to the basis $(\phi_i)$.
\vspace{0.5cm}
\begin{remark}
	 If the basis functions $\phi_i$ are orthonormal, then the mass matrix $B$ is the identity matrix. Furthermore, the scalar product in the space $V_h$ of two functions $u_h, v_h$ is equal to the dot product defined by the mass matrix $B$ in $\mathbb R^n$ of the coordinate vectors
	\begin{equation}\label{eq:dot product}
	\langle u_h, v_h\rangle = \mathbf u^\intercal B \mathbf v
	\end{equation}
	The Galerkin method \ref{eq:Galerkin problem} is well posed by a straight-forward application of the Lax-Milgram theorem, and thus also the system \ref{eq:Galerkin algebraic} admits one and only one solution.
\end{remark}

 \subsubsection{The Finite Element Method}

  The Finite Element Method is a technique that let us construct a particular subspace $V_h$ in (\ref{eq:Galerkin problem}) through polynomial interpolation. Let's refer again to the problem \ref{eq:strong form} defined on the domain $\Omega\subset\mathbb R^2$ in figure \ref{fig:omega and mesh}.
 \paragraph{Approximation of the domain $\Omega$}
\begin{figure}[h]
	\centering
	\includegraphics[width=0.4\textwidth]{figs/Chapter3/omega.png}
	\includegraphics[width=0.4\textwidth]{figs/Chapter3/mesh.png}
	\caption{\label{fig:omega and mesh}The domain $\Omega$ and its approximation $\mathcal T_h$}
\end{figure}
First we need to construct a discretization of the continuous domain $\Omega$. In this example we take the triangulation in figure \ref{fig:omega and mesh} $\mathcal T_h = \left\{\tau_{k} : 1 \leq k \leq q\right\}$, where $h\in\mathbb R$ is a parameter such that every edge of the triangles $\tau_k\in\mathcal T_h$ is smaller than $h$. $\mathcal T_h$ is such that
\begin{itemize}
	\item The elements \(\tau \in \mathcal{T}_h\) are closed subsets of \(\Omega\) with pairwise disjoint interior and \({\Omega}_h=\bigcup_{\tau \in \mathcal{T}_h} \tau\)
	\item The triangulation \(\mathcal{T}_h\) has no hanging vertices.
\end{itemize}

It's clear that by discretizing the continuous domain $\Omega$ we introduce a first source of errors in the method; however for simplicity we won't take this into account in the next discussion, and we will identify the domain $\Omega$ with the domain $\Omega_h$ covered by $\mathcal T_h$. The quality of the \textit{mesh} $\mathcal T_h$ is important for a good solution; a good quality mesh should avoid triangles with extreme angles and every triangle of $\mathcal T_h$ should look like as much as possible to an equilateral triangle. In figure \ref{fig:bad mesh} we see an example of a bad quality mesh: its triangles look very stretched; there are vertices that are shared by many triangles and vertices that are shared by very few. In figure \ref{fig:omega and mesh} we see a better mesh: there are no stretched triangles, and every vertex is shared by an almost constant number of triangles.
\begin{wrapfigure}{r}{0.3\textwidth}
	\begin{center}
		\includegraphics[width=0.4\textwidth]{figs/Chapter3/badmesh.png}
		\caption{\label{fig:bad mesh}A bad quality mesh}
	\end{center}

\end{wrapfigure}
\paragraph{Choosing $V_h$ and the basis functions $\phi_i$}

Beyond $X_h^1$, explained in section \ref{sec:Chapter3:FEM on the sphere}, other choices for $V_h$ are possible. A common choice is the space $X_h^{2}$ that is the space of all the piecewise second-order polynomials. In this case, since every second order polynomial in $\mathbb R^2$ has 6 degrees of freedom per each triangle $\tau_k$, meaning that we need to know its values in at least 6 different points on the triangle $\tau_k$ to uniquely identify it, the dimension of the space will grow (figure \ref{fig:elements}). A bigger space means that the approximation $f_h$ will be better, but we'll need more basis functions to define and thus it will result in a bigger linear system $Af = Bu$ and higher computational costs.
\begin{figure}[h!]
	\begin{center}
		\includegraphics[width=0.4\textwidth]{figs/Chapter3/elemntsP1P2.png}
	\end{center}
	\caption{\label{fig:elements}The degrees of freedom (DOF) for $X_h^{1}$ and $X_h^{2}$ on a reference triangle}
\end{figure}
\paragraph{Assembling the stiffness and mass matrices}
Once defined the basis functions $\phi_i$, the FEM method constructs the stiffness matrix and the mass matrix $(A)_{ij} = a(\phi_i, \phi_j),\  (B)_{ij}=\langle\phi_i,\phi_j\rangle$ and solves the linear system \ref{eq:Galerkin algebraic} for the coefficients $f_i$ of the FEM solution $f_h$. An important fact to notice is both matrices $A$ and $B$ are \textit{sparse} and share the same sparsity pattern. Due to the form of the basis function $\phi_i$, the element $(i, j)$ of these matrices is different from zero only if the supports of the corresponding basis functions $(\phi_i, \phi_j)$ overlap, meaning that the vertices $(x_i, x_j)$ are connected by an edge of the mesh $\mathcal T_h$. In other words, the number of non-null entries of the $i-th$ row of $A$ and $B$ is equal to number of triangles of the mesh $\mathcal T_h$ that share the $i$th vertex i.e., the degree of the $i-th$ vertex.

\paragraph{About the boundary conditions}
	Note that the Dirichlet boundary conditions in the strong formulation of the differential problem \ref{eq:strong form} got transformed in the weak formulation \ref{eq:final variational form} as a condition on the ambient space $H_0^1(\Omega)$. Other kind of boundary conditions (Neumann, Robin for example) would not translate into the condition on the ambient space that functions must vanish on the border, but instead they would impose a different formulation of the bilinear form $a(\cdot, \cdot)$ or of the functional $L$. For this reason, Dirichlet boundary conditions are called \textit{essential} since translated into a condition on the ambient space and thus automatically satisfied from the FEM formulation; Neumann boundary conditions are called \textit{natural} since they transform into a different weak formulation through a modification of the bilinear form $a(\cdot, \cdot)$ and/or the functional $L$.


\subsection{Tensor product}

We have spaces \(V\) and \(W\) over field \(F\). If we have a linear operation which takes a vector from each space and returns a scalar from the underlying field, it is an element of the tensor product of the two spaces.

For example if we have two vectors:

\(v=e_iv^i\)

\(w=e_jw^j\)

A tensor product would take these and return a scalar.

There are three types of tensor products:

\begin{itemize}
\item Both are from the vector space
\item \(T_{ij}v^iw^j\)
\item \(T_{ij} \in V\otimes W\)
\item Both are from the dual space
\item \(T^{ij}v_iw_j\)
\item \(T_{ij} \in V^*\otimes W^*\)
\item One is from each space
\item \(T_i^jv^iw_j\)
\item \(T_{ij} \in V\otimes W^*\)
\end{itemize}

As a vector space, we can add together tensor products, and do scalar multiplication.

\subsubsection{Basis of a tensor product}

Not all elements spanned by a basis of a tensor product are themselves tensor products.

\subsubsection{Eigenvalues and Eigenvectors of a tensor product}

\subsubsection{Homomorphisms}

We can define homomorphisms in terms of tensor products.

\(Hom (V) =V \otimes V^*\)

\(T_j^i\)

We use the dual space for the second argument. This is because it ensures that changes to the bases do not affect the maps.

\(w^j=T_i^j v^i\)