\chapter{Fit of histograms}
\label{ch:fit}

This chapter illustrates how the fits were done in \autoref{ch:anal}. They will
turn out to be almost exactly a standard least squares fit of a histogram, but
we will give a Bayesian interpretation to all the procedures. If you do not
favor Bayesian methods, just do not think about it. To actually do the fits, we
used the Python package \texttt{lsqfit} \cite{lepage2021}, which we warmly
recommend.

\section{Bayesian least squares}

Bayesian inference is based on applying Bayes' theorem to get the probability
distribution of the parameters conditional on the observed data. Call $\theta$
the parameters and $y$ the data, then it reads
%
\begin{equation}
    p(\theta|y) = \frac {p(y|\theta) p(\theta)} {p(y)}.
\end{equation}

The term $p(y|\theta)$ is the likelihood, how the data depends on the
parameters. $p(y)$ is the unconditional probability of the data; whatever that
means, we can compute it with $\int\mathrm d\theta\,p(y|\theta)p(\theta)$.
$p(\theta)$ is the unconditional probability distribution of the parameters. It
is called \emph{prior} because it encodes what one knows about the parameters
before having the data. By the same logic, $p(\theta|y)$ is called
\emph{posterior}.

Least squares is a general procedure to produce an estimator, but a standard
interpretation is also as a maximum likelihood fit when the likelihood is
Gaussian. We can do the same in the Bayesian case if the prior is Gaussian too.
We switch to logarithms and drop addends which do not depend on the parameters:
%
\begin{align}
    \log p(\mathbf y|\boldsymbol\theta) &=
    -\sum_{i=1}^n \log\sigma_i
    -\frac 12 \sum_{i=1}^n
    \left( \frac {y_i - \mu_i(\boldsymbol\theta)} {\sigma_i} \right)^2, \\
    %
    \log p(\boldsymbol\theta) &=
    -\sum_{j=1}^k \log\sigma^\theta_j
    -\frac 12 \sum_{j=1}^k
    \left( \frac {\mu^\theta_j - \theta_j} {\sigma^\theta_j} \right)^2.
\end{align}

Now these two terms have to be summed to obtain the log posterior. If we extend
the $\mathbf y$, $\boldsymbol\mu$ and~$\boldsymbol\sigma$ vectors, we can see
the two sums together as a single sum of squares:
%
\begin{align}
    y_{n+j} &\equiv \mu^\theta_j, \\
    %
    \mu_{n+j}(\boldsymbol\theta) &\equiv \theta_j, \\
    %
    \sigma_{n+j} &\equiv \sigma^\theta_j, \qquad j = 1, \ldots, k, \\
    %
    \log p(\boldsymbol\theta|\mathbf y) &=
    -\sum_{i=1}^{n+k} \log\sigma_i
    - \frac 12 \sum_{i=1}^{n+k}
    \left( \frac {y_i - \mu_i(\boldsymbol\theta)} {\sigma_i} \right)^2 \equiv \\
    %
    &\equiv -\sum_{i=1}^{n+k} \log\sigma_i - \frac12 \chi^2. \label{eq:chi2}
\end{align}

So formally the prior is equivalent to additional datapoints, one for each
parameter. We have not dropped the sum with the $\log\sigma$ although it does
not depend on $\boldsymbol\theta$; it will be needed later. Note that the
degrees of freedom of the $\chi^2$ now are $(n+k)-k = n$, instead of the usual
$n-k$ without prior.

In the analysis we said we put uniform priors over the parameters defined in
the interval $(0,1)$. This is implemented by fitting a transformed parameter
with a standard Gaussian prior with zero mean and unitary variance, and
applying as inverse transformation the Gaussian cumulative density function
(cdf). Say, if we want to put a uniform prior on $\theta_j$, we fit the
transformed parameter $\theta_j'$, with
%
\begin{align}
    \mu^{\theta'}_j &= 0, \\
    %
    \sigma^{\theta'}_j &= 1, \\
    %
    \theta_j(\theta'_j) &= \Phi(\theta'_j)
    \equiv \int_{-\infty}^{\theta'_j} \mathrm du\,
    \frac {e^{-u^2/2}} {\sqrt{2\pi}}.
\end{align}

Bayesianly, this is equivalent to omitting the squared term for the prior and
putting bounds on the parameter value, without transforming. However it is
convenient for the fitting routine. By the same logic, when we use the
logarithm to map a positive parameter to the real line, the prior on the
untransformed parameter is a log-Gaussian distribution.

Now the least squares estimator would be obtained by minimizing the $\chi^2$,
while a Bayesian cares about the posterior as a distribution. If we approximate
the posterior as a Gaussian, we can do formally the same calculations. To
compute the Gaussian approximation we expand the logarithm of the distribution
to second order around the maximum:
%
\begin{align}
    \log p(\boldsymbol\theta|\mathbf y) &\approx
    -\frac 12 (\boldsymbol\theta-\boldsymbol\theta_0)^\top
    V^{-1} (\boldsymbol\theta-\boldsymbol\theta_0), \\
    %
    \boldsymbol\theta_0
    &= \operatorname*{arg\,min}_{\boldsymbol\theta} \chi^2, \\
    %
    V^{-1}_{jk} &= \frac12 \left.
    \frac {\partial^2 \chi^2} {\partial\theta_j \partial\theta_k}
    \right|_{\boldsymbol\theta_0}. \label{eq:lsqpm}
\end{align}

Now let us carry further the calculation of the precision matrix, to discuss
an additional issue:
%
\begin{align}
    \frac{\partial\chi^2}{\partial\theta_j} &=
    2 \sum_i \frac {y_i - \mu_i(\boldsymbol\theta)} {\sigma_i^2}
    \frac {\partial\mu_i} {\partial\theta_j}, \\
    %
    \frac {\partial^2 \chi^2} {\partial\theta_j \partial\theta_k} &=
    2 \sum_i \frac1{\sigma_i^2} \left(
    -\frac {\partial\mu_i} {\partial\theta_j}
    \frac {\partial\mu_i} {\partial\theta_k}
    + (y_i - \mu_i(\boldsymbol\theta))
    \frac {\partial^2 \mu_i} {\partial\theta_j \partial\theta_k}
    \right). \label{eq:lsqhess}
\end{align}

In standard least squares, the precision matrix of the estimator is estimated
with the above expression, but with the term with the second derivative
dropped. The reason is for convenience, but also because the expectation of the
residuals $y_i - \mu_i(\boldsymbol\theta_0(\mathbf y))$ is approximately zero.
A frequentist would like to compute this with the true value of the parameters,
but has to content herself with the least squares estimate, getting something
which on average is the true precision matrix. So any term which is zero on
average can be removed.

For a Bayesian, instead, there is not some ideal error corresponding to an
ideal parameter value; there is only the covariance of the posterior obtained
with the particular values at hand. Different data, different posterior.
However in our fits the residuals term is missing, as we said just because this
is convenient for the implementation. In practice, the difference should be
small.

\section{Least squares fit of a histogram}

The standard model of a histogram is Poisson distributions for the bin counts.
Let $c_i$ be the count in bin~$i$, $f_i(\boldsymbol\theta)$ the model
normalized distribution of the bins, and $M$ a parameter for the mean of the
total (Poisson) number of samples. Then the likelihood is
%
\begin{equation}
    P(\mathbf c|M,\boldsymbol\theta) =
    \prod_i P(c_i|M,\boldsymbol\theta) =
    \prod_i \operatorname{poisson}(c_i;Mf_i(\boldsymbol\theta)),
\end{equation}
%
where $\operatorname{poisson}(\cdot;\mu)$ is a Poisson probability mass
function (pmf) with mean~$\mu$.

We do not care about $M$, only about $\boldsymbol\theta$, so we marginalize
the posterior over $M$:
%
\begin{align}
    p(\boldsymbol\theta|\mathbf c)
    &= \int \mathrm dM\, p(M,\boldsymbol\theta|\mathbf c) = \\
    %
    &= \int \mathrm dM\,
    \frac {P(\mathbf c|M,\boldsymbol\theta) p(M,\boldsymbol\theta)}
    {P(\mathbf c)}.
\end{align}

Now we assume that a priori $M$ is independent from $\boldsymbol\theta$, i.e.,
$p(M,\boldsymbol\theta) = p(M)p(\boldsymbol\theta)$. Then, continuing the
calculation:
%
\begin{align}
    p(\boldsymbol\theta|\mathbf c)
    &= \frac {p(\boldsymbol\theta)} {P(\mathbf c)}
    \int \mathrm dM\, p(M)
    \prod_i \operatorname{poisson}(c_i;Mf_i(\boldsymbol\theta)) = \\
    %
    &= \frac {p(\boldsymbol\theta)} {P(\mathbf c)}
    \int \mathrm dM\, p(M)
    \prod_i
    e^{-Mf_i(\boldsymbol\theta)}
    \frac {(Mf_i(\boldsymbol\theta))^{c_i}} {c_i!} = \\
    %
    &= \frac {p(\boldsymbol\theta)} {P(\mathbf c)}
    \int \mathrm dM\, p(M)
    e^{-M \sum_i f_i(\boldsymbol\theta)}
    M^{\sum_i c_i}
    \prod_i
    \frac {f_i(\boldsymbol\theta)^{c_i}} {c_i!}
\end{align}
%
Since $f_i$ is normalized the sum over it yields~1, while we call $N$ the total
count, so
%
\begin{align}
    p(\boldsymbol\theta|\mathbf c)
    &= \frac {p(\boldsymbol\theta)} {P(\mathbf c)}
    \prod_i
    \frac {f_i(\boldsymbol\theta)^c_i} {c_i!}
    \int \mathrm dM\, p(M) e^{-M} M^N.
\end{align}
%
The integral does not depend on $\boldsymbol\theta$, so we call it $I(N)$ and
do not care about its value. Now we take the logarithm:
%
\begin{align}
    \log p(\boldsymbol\theta|\mathbf c) &=
    -\log P(\mathbf c) + \log I(N) - \sum_i \log c_i!
    + \log p(\boldsymbol\theta) + \sum_i c_i \log f_i(\boldsymbol\theta).
\end{align}

This is the exact expression for the posterior, which we want to compare to
least squares, which yields
%
\begin{align}
    Q(\boldsymbol\theta) &=
    \log p(\boldsymbol\theta)
    -\frac 12
    \sum_i \frac {(c_i - N f_i(\boldsymbol\theta))^2} {c_i}.
\end{align}

Note the particular flavor of chisquare we have picked: there is no parameter
for the normalization, which is fixed to the actual total count $N$, and the
denominator is the observed count $c_i$.

To do the comparison, we expand both expressions to second order, ignoring the
shared prior term which is already matched:
%
\begin{align}
    L(\boldsymbol\theta) &\equiv \log p(\boldsymbol\theta|\mathbf c), \\
    %
    \frac {\partial L} {\partial\theta_j} &=
    \sum_i \frac {c_i} {f_i(\boldsymbol\theta)}
    \frac {\partial f_i} {\partial \theta_j}, \\
    %
    \frac {\partial^2 L} {\partial\theta_j \partial\theta_k} &=
    \sum_i c_i \left[
    -\frac 1 {f_i(\boldsymbol\theta)^2}
    \frac {\partial f_i} {\partial \theta_j}
    \frac {\partial f_i} {\partial \theta_k}
    + \frac 1 {f_i(\boldsymbol\theta)}
    \frac {\partial^2 f_i} {\partial\theta_j \partial\theta_k}
    \right].
\end{align}

For $Q$ we start from \eqref{eq:lsqhess}, obtaining
%
\begin{align}
    \frac {\partial^2 Q} {\partial\theta_j \partial\theta_k} &=
    -N^2 \sum_i
    \frac 1 {c_i} \left[
    \frac {\partial f_i} {\partial \theta_j}
    \frac {\partial f_i} {\partial \theta_k}
    - \left( \frac{c_i} N - f_i(\boldsymbol\theta) \right)
    \frac {\partial^2 f_i} {\partial\theta_j \partial\theta_k}
    \right]. \label{eq:Qjk}
\end{align}

We do not discuss the different results for the maximum because it is well
known that when there at least five counts per bin the least squares
approximation is good enough. The covariance instead needs a discussion. Now we
manipulate $\partial_{jk} L$ to show it is similar to $\partial_{jk} Q$. First,
we approximate the observed counts with the model, $c_i \approx N
f_i(\boldsymbol\theta)$:
%
\begin{align}
    \frac {\partial^2 L} {\partial\theta_j \partial\theta_k} &=
    -N^2 \sum_i \frac {c_i} {N^2 f_i(\boldsymbol\theta)^2} \left[
    \frac {\partial f_i} {\partial \theta_j}
    \frac {\partial f_i} {\partial \theta_k}
    - f_i(\boldsymbol\theta)
    \frac {\partial^2 f_i} {\partial\theta_j \partial\theta_k}
    \right] \approx \\
    %
    &\approx
    -N^2 \sum_i \frac 1 {c_i} \left[
    \frac {\partial f_i} {\partial \theta_j}
    \frac {\partial f_i} {\partial \theta_k}
    - f_i(\boldsymbol\theta)
    \frac {\partial^2 f_i} {\partial\theta_j \partial\theta_k}
    \right].
\end{align}

Next, we observe that since $f$ is normalized, $\sum_i \partial_{jk} f_i = 0$,
so we can add the missing term to obtain the residuals:
%
\begin{align}
    \frac {\partial^2 L} {\partial\theta_j \partial\theta_k} &\approx
    -N^2 \sum_i \frac 1 {c_i} \left[
    \frac {\partial f_i} {\partial \theta_j}
    \frac {\partial f_i} {\partial \theta_k}
    + \left( \frac {c_i} N - f_i(\boldsymbol\theta) \right)
    \frac {\partial^2 f_i} {\partial\theta_j \partial\theta_k}
    \right]. \label{eq:Ljk}
\end{align}

Comparing \eqref{eq:Qjk} with \eqref{eq:Ljk}, we see that the difference is the
sign of the residuals term. As we explained above, in the least squares fit
that term is dropped altogether, so notwithstanding the Poisson model, we end
up doing the same kind of approximation we did starting from Gaussians.

\section{The $\chi^2/\mathrm{dof}$ correction}

Starting from the Gaussian model posterior \eqref{eq:chi2}, we add a parameter
$s$ that rescales uniformly all the errors:
%
\begin{align}
    \log p(\boldsymbol\theta|\mathbf y)
    &= -\sum_{i=1}^{n+k} \log\sigma_i - \frac12 \chi^2 \mapsto \\
    %
    \mapsto \log p(\boldsymbol\theta,s|\mathbf y)
    &= \log p(s) -\sum_{i=1}^{n+k} \log(s \sigma_i)
    - \frac 12 \sum_{i=1}^{n+k}
    \left( \frac {y_i - \mu_i(\boldsymbol\theta)} {s \sigma_i} \right)^2 = \\
    %
    &= \log p(s) - (n+k) \log s - \sum_{i=1}^{n+k} \log\sigma_i
    - \frac 12 \frac{\chi^2}{s^2}.
\end{align}

Since $s$ is a scale parameter, it makes sense to use a uniform prior over
$\log s$, i.e., $p(s) = 1/s$, because it is scale-invariant. Now that we have
let errors vary, we marginalize away $s$, freely dropping normalizations as
usual:
%
\begin{align}
    p(\boldsymbol\theta|\mathbf y)
    &= \int \mathrm ds\,
    p(\boldsymbol\theta,s|\mathbf y) = \\
    %
    &= \int \frac {\mathrm ds} {s^{n+k+1}}
    \exp\left(-\frac12 \frac {\chi^2} {s^2}\right) = \\
    %
    &= \frac 1 {(\chi^2)^{(n+k)/2}}.
\end{align}

Now again we take the logarithm and expand to second order:
%
\begin{align}
    P(\boldsymbol\theta) &\equiv
    \log p(\boldsymbol\theta|\mathbf y)
    = -\frac{n+k}2 \log \chi^2, \\
    %
    \frac {\partial P} {\partial \theta_j}
    &= -\frac{n+k}2 \frac 1{\chi^2}
    \frac {\partial \chi^2} {\partial \theta_j}, \\
    %
    \frac {\partial^2 P} {\partial\theta_j \partial\theta_k}
    &= -\frac{n+k}2 \left[
    -\frac 1 {(\chi^2)^2}
    \frac {\partial \chi^2} {\partial \theta_j}
    \frac {\partial \chi^2} {\partial \theta_k}
    + \frac 1{\chi^2}
    \frac {\partial^2 \chi^2} {\partial\theta_j \partial\theta_k}
    \right]. \label{eq:Pjk}
\end{align}

In the maximum $\partial_j P = 0$, so the first term in \eqref{eq:Pjk}
vanishes, leaving
%
\begin{align}
    \frac {\partial^2 P} {\partial\theta_j \partial\theta_k}
    &= -\frac12 \frac 1 {\chi^2/(n+k)}
    \frac {\partial^2 \chi^2} {\partial\theta_j \partial\theta_k}.
    \label{eq:chi2dof}
\end{align}
%
This is to be compared with the precision matrix in \eqref{eq:lsqpm}. The
only difference is that the matrix has been divided by $\chi^2/(n+k)$. Note
that $\mathrm{dof} = n$, so we obtained a different prescription than the
conventional one. Nevertheless, in the analysis we used $\chi^2/\mathrm{dof}$
just because it is what people expect; compared to \eqref{eq:chi2dof}, this
gives errors up to \SI{25}\% larger on our fits, see \autoref{tab:ct}.

Similarly, we used the pvalue; since we always obtained a stark contrast
between good and bad fits, we did not bother giving a precise Bayesian
equivalent.


\documentclass[11pt]{article}

\usepackage{array}
\usepackage{clrscode3e}
\usepackage{amsmath}
\usepackage{kbordermatrix}

\setlength{\parindent}{0em}
\setlength{\parskip}{1em}

\begin{document}

\title{Efficient Calculation of Interaction Features on Sparse Matrices}
\author{Andrew Nystrom}
\date{}

\maketitle

\begin{abstract}%   <- trailing '%' for backward compatibility of .sty file
FILL THIS IN
\end{abstract}

\section{Introduction}

Introduction
Interaction features are a way of capturing correlations between features in a machine
learning setting. A feature vector $\vec{x}$ of dimensionality $D$ has second degree interaction features
$\{x_i \cdot x_j : i, j \in \{0,1,..., D-1\} \land i < j\}$,
so a $D$ dimensional vector has $\binom{D}{2} = \frac{D^2-D}{2}$ second degree interaction features. A naive
approach to calculating these features is to simply iterate through the combinations of the column indices.
For a sparse vector, many of the resulting interaction features would be zero, and could therefore be ignored.
This work describes a method to efficiently calculate second degree interaction features for a sparse matrix
that has time and space complexities that decrease quadratically with the density of the input matrix with respect to the naive approach.

\section{Approach}
Let the list of nonzero columns for a given row $\vec{x}$ be denoted by $N_{zc}$. The nonzero second degree
interaction features are simply the products of all combinations of two elements whose
columns are in $N_{zc}$. However, to properly place an interaction feature into the correct column, a mapping from the column
index pairs of $N_{zc}$ into the columns of the interaction matrix is needed. The mapping is
from pairs $(a, b)$,  where $a$ and $b$ are in $1,2,..., D$, and $a < b$, to $1,2,..., \frac{D^2-D}{2}$,  Such a mapping essentially consists of mapping the indices of entries in the upper triangle of a matrix to indices in a flat
list. We now describe the construction of such a mapping.

\subsection{Mapping construction}
We seek a map from matrix indices $(i, j)$ (with $i < j$ and $0 \le i < D$) to numbers $f(i, j)$ with $0 \le f(i, j) < \frac{D(D-1)}{2}$, one that follows the pattern indicated by
\begin{align}
\begin{bmatrix}
x & 0 & 1 & 2 \\
x & x & 3 & 4 \\
x & x & x & 5 \\
x & x & x & x
\end{bmatrix}
\end{align}
It's considerably easier, however, to consider the same indices, but subtracted from $6$ (or more generally, from $\frac{D(D-1)}{2}$; that gives the pattern
\begin{align}
\label{eq:4x4mat}
\begin{bmatrix}
x & 6 & 5 & 4 \\
x & x & 3 & 2 \\
x & x & x & 1 \\
x & x & x & x
\end{bmatrix}
\end{align}
We'll call the function defined by this example $(i, j) \mapsto g(i, j)$, and then observe that
\begin{align}
f(i, j) = \frac{D(D-1)}{2} - g(i, j)
\end{align}

To simplify slightly, we introduce a notation for the $n$th triangular number,
\begin{align}
\begin{bmatrix}
T_2(n) = \frac{n(n+1)}{2}
\end{bmatrix}
\end{align}
\noindent
The subscript $2$ is there to indicate that these are triangles in two dimensions; we'll use $T_3(n)$ to indicate the $n$th tetrahedral number, and so on for higher dimensions.

The codomain of $g$ is now numbers from $1$ to $T_2(D-1)$, inclusive.

Observe that in Equation~\ref{eq:4x4mat}, each entry in row $i$ lies in the range
\begin{align}
T_2(D-i-1) &< e \le T_2(D-i).
\end{align}
\noindent
For instance, in row $2$ in our example, where $D = 4$, the entries range from $2$ to $3$, while $T_2(D-i-1) = T_2(1) = 1$ and $T_2(D-i) = T_2(2) = 3$. (Note that row indices start at zero.) Unfortunately, the numbers increase from right to left. The entry in column $j$ is just $T_2(D-i-1) + D-j$, which adds one for the rightmost column (because $D - (D-1) = 1$).
Thus, the formula for $g$ is simply
\begin{align}
g(i, j)
&= T_2(D-i-1) + (D-j) \\
&= \frac{(D-i-1)(D-i)}{2} + D-j \\
&= \frac{(D^2 - (2i)D - D - i^2 - i) + 2D - 2j}{2} \\
&= \frac{D^2 - (2i)D + D - i^2 - i - 2j}{2}
\end{align}
and hence
\begin{align}
f(i, j)
&= \frac{D(D-1)}{2} - g(i,j) \\
&= \frac{D^2-D}{2} - \frac{D^2 - (2i)D + D - i^2 - i - 2j}{2}\\
&= \frac{D^2-D - D^2 + (2i)D - D + i^2 + i + 2j}{2}\\
&= \frac{(2i)D - 2D + i^2 + i + 2j}{2}
\end{align}
[WRONG]
Correct result:
\begin{align}
R = \frac{ 2in - i^2  + 2j - 3i - 2}{2}.
\end{align}
\subsubsection{Other indices}
With one-based indexing, the formula above becomes
\begin{align}
f_1(i, j) &= ....
\end{align}

  column

INSERT JOHN'S PROOF HERE

With this mapping, an algorithm for generating second degree interaction features on a
matrix $A$ can be formulated as follows:

\begin{codebox}
\Procname{$\proc{Sparse Interaction}(A)$}
    \zi $\func{map}(a, b) = \frac{2Da-a^2+2b-3a-2}{2}$
    \zi $N \gets$ row count of $A$
    \zi $D \gets$ column count of $A$
    \zi $B$ $\gets$ Compressed Sparse Row Matrix of size $N \times \frac{D^2-D}{2}$
    \zi \For $\id{row}$ in $A$ \Do
    \zi     $N_{zc} \gets$ nonzero columns of $row$
    \zi     \For $i \gets 0 \To |N_{zc}|-1$ \Do
    \zi         \For $j \gets i+1 \To |N_{zc}|$ \Do
    \zi             $k \gets \func{map}(i, j)$
    \zi             $r \gets$ index of $\id{row}$
    \zi             $B[r, k] \gets \id{row}[i] \cdot \id{row}[j]$
                \End
            \End
       	\End
\end{codebox}

\section{Complexity Analysis}
Assume that A is a matrix with sparsity $0 < d < 1$, $N$ rows, and $D$ columns. Finding
interaction features with the proposed algorithm has time and space complexity
$\func{O}(d N D^2)$,
whereas a naive approach of using non-sparse matrices and multiplying all column
combinations has time and space complexity $\func{O}(N D^2)$. The algorithm is therefore an
improvement by a factor of the density factor of $A$.

This can represent a large gain in speed and time. For example, the 20 Newsgroups dataset
has density $d$ of 0.12 when its unigrams are represented in a vector space model. This
means the proposed approach would take less than $\frac{1}{8}$ time and memory.

The real benefit of this method is revealed when the average complexity is analysed. The
number of interactions calculated for a given row are $\binom{|N_{zc}|}{2}$. If the matrix has
density $d$, then on average, $N_{zc} = D d$, so the number of interaction features
calculated in total is

\begin{align*}
N \binom{d D}{2} &= N \frac{(Dd)!}{2!(Dd-2)!}\\
    \\
    &= N \frac{(D^2d^2-Dd)}{2}
\end{align*}

This means that the average complexity decreases quadratically with the density.

\section{Future Work}
The approach for generating second degree interaction features required a mapping from
combinations of two to the space $1,2,...,\frac{D^2-D}{2}$, which is isomorphic to a mapping from
the indices of an upper triangular matrix to the indices of a flat list of the same size.
To generate third degree interaction features, a mapping from combinations of three
$(a,b,c)$ to the space $1,2,...\frac{D^3-3D^2+2D}{6}$ (which is $\binom{D}{3}$), or the upper $3$-simplex of a tensor to a flat
list of the same size $\frac{D^3-3D^2+2D}{6}$ would be required. In general, for interaction
features of degree k, the upper $k$-simplex of a $k$-dimensional tensor must be mapped to the
space $1,2,...\frac{D!}{k!(D-k)!}$. A similar approach for finding these mappings could be taken
as the one used here for $k=2$.

Motivation for deriving mapping functions for higher orders
of interaction features is that the average complexity of generating degree $k$ interaction
features is $N \binom{Dd}{k}$, which decreases polynomially with respect to k compared to
generating the features naively.

\vskip 0.2in
\bibliography{sample}

\end{document}


\documentclass[letterpaper]{article}

\usepackage[utf8]{luainputenc}
\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{2 mars, 2015}
\maketitle

when does taylor series converge?

better question is when is $f(x)=\sum\limits{f^{(n)}(a)/n!\cdot (x-a)^n}$

\section*{taylor's thm}
$f\in C^{\infty}[a,b]$ and $f^{(n+1)}$ is defined on $[A,B]$ with $|f^{(n+1)}(x)|\le M$ for $x\in [A,B]$ then $R_n(x)=f(x)-\sum{\frac{f^{k}}{k!}(x-a)^k}$ satisfies $|R_n(x)|\le M|x-a|^{n+1}/(n+1)!$
\begin{enumerate}
\item
$f(x)=\lim\sum\frac{f^{(k)}}{k!}(x-a)^k$ if and only if $\lim R_n(x)=0$

the basic issue is that $M$ needs to not get too big to fast
\item
if $f\in C^\infty[A,B]$ then these hypotheses happen automatically (it's infinitely differentiable), although there is no guarantee that the taylor series converges to $f$.
\end{enumerate}

we want to use induction (what are we inducting on?)

we will show that $|R_n^{(n-k)}(x)|\le\frac{M|x-a|^{k+1}}{(k+1)!}$.

base case is $k=0$. $R_n(x)=f(x)-\sum{\frac{f^{(k)}}{k!}(x-a)^k}=f^{(n)}-\frac{n!}{n!}f^{(n)}x$ and so $|R_n(x)|=|f^{(n)}(x)-f^{(n)}(a)|$. by MVT we know $M|x-a|\ge f^{(n+1)}(c)|x-a|=|R_n(x)$. So base case is done

assume $|R_n^{(n-k)}(x)|\le M|x-a|^{k+1}/(k+1)!$. consider $|R_n^{(n-(k+1))}(x)|=|R_n^{(n-(k+1))}(a)-\int_a^x{R^{(n-k)}(t)\;\mathrm{d}t}|$

\subsection*{example}
$f(x)=\sin x$ and $a=\frac{\pi}{2}$.

$P_n(x)=\sum\limits_{k=0}^n{\frac{f^{(k)}(\frac{\pi}{2}(x-\frac{\pi}{2})^k}{k!}}$ and then we have $P_n=\sum\limits_{k=0}^n{(-1)^k(x-\frac{\pi}{2})^{2k}}/(2k)!$. Now $M_n=1$ (it is bounded by 1). Now then $R_n(x)\le \frac{1\cdot |x-a|^{n+1}}{(n+1)!}$ and because factorials are bigger than powers,  the limit is $R=0$.

and so the power series gives the same value as the function all the time.

$f(x)=\log x$ (natural log). taylor series at 1. note that zero is a problem. one is nice because it's symmetric and easy to compute.

$f'(x)=\frac{1}{x}, f''(x)=-\frac{1}{x^2}, f'''(x)=\frac{2}{x^3},\dots$

$P_n=\sum\limits_{k=1}^n{(-1)^{k+1}\frac{(k-1)!}{k!}(x-1)^k}$ now use ratio test

$\frac{\frac{(-1)^{k+2}}{k+1}}{\frac{(-1)^{k+1}}{k}}\to$ radius of convergent of 1, centered at zero, so interval is $(0,2)$
\end{document}


\section{Linkages and Frameworks}
--izmestiev def of framework
--freedom in machines def of linkage
--used to describe configurations and freedoms

\section{Geometric Configuration Space}

 --pictures of flexing.
\begin{figure}[ht]
  %\includegraphics[scale=0.1, angle=0]{tetris.eps}
\caption{Different geometric configurations of an octahedron intermediate.}
\label{fig:OctaGCS}
\end{figure}
--embedding

\begin{mydef}
The \textbf{constraint space} corresponding to a constraint function $c: \mathbbm{R}^n \to \mathbbm{R}^{n-m}$ is its zero-set,
\begin{align}
\Omega \doteq \{z \in \mathbbm{R}^n : c(z) = 0\}.
\end{align}
\end{mydef}
If $c$ is composed of polynomial functions, then the constraint space is an algebraic variety. If this variety has no singularities, then it is also a manifold.

Using the notion of a constraint space, we can define the space of different geometric configurations that a given intermediate can take. Since we are treating each intermediate as collection of rigid polygons attached to each other along hinged edges, there are two basic types of constraint we must enforce. First, we must ensure that the each individual face is rigid and has the correct geometric shape. Additionally, each edge to edge connection must remain fixed with only hinge-like motion allowed.

\begin{mydef}
The \textbf{geometric configuration space} of a building game intermediate is a constraint space that enforces the rigidity of individual faces and hinged motion along connected edges.
\end{mydef}

To mathematically describe a particular configuration, we must specify the locations of the vertices of each face. Thus, for an intermediate $[x]$, its building game configuration space can be represented as a subset of the ambient space $\mathbb{R}^{3\times N_x}$ where $N_x \doteq \sum_{f\in x} s_f$ where $s_f$ is the number of sides (and verticies) of face $f$. Using this representation, we must then identify the corresponding constaint equations that give rise to the combinatorial configuration space as a function of points in ambient space.

It is worth noting that while we represent each face with $3\times s_f$ coordinates, only 6 are required to speciafy a face's position and orientation if they are chosen carefully. With this in mind, we will typically use a function of $6$ of a face's vertex coordinates to constrain each of the face's remaining vertex coordinates.

Notationally, we refer to the $k$th vertex of the $j$th face of $x$ as $v^{jk} = \left(v^{jk}_x,v^{jk}_y,v^{jk}_z\right)$.
Since the context of a constraint space requires a function $c: \mathbbm{R}^n \to \mathbbm{R}^{n-m}$, we flatten the matrix of vertex coordinates into a vector of lenghth $n = 3N_x$.
\begin{align}
z = \begin{bmatrix} v^{1,1} \\ \vdots \\ v^{1,s_{f_1}} \\ \vdots \\ v^{|x|,1} \\ \vdots \\ v^{|x|,s_{f_{|x|}}} \end{bmatrix} \in \mathbbm{R}^n
\end{align}

There are four fundamental types of constraint equations: edge length constraints, angle constraints, and $2D$ face constraint to enforce the rigid structure of each face as well as vertex identification constraints to enforce the hinged connections.

Edge length constraints enforce that the lengths of the edges of each face in an intermediate cannot change. If the $k$th edge is definted to be that between the $(k-1)$st and $k$th vertices, we use the following function to constrain its lengths to a known value $\ell^{jk}$.
\begin{align}
c_{edge}^{j,k}\left(z\right)& = \left|v^{j,k} - v^{j,k-1}\right|^2 - (\ell_{j,k})^2 \\
& = \left(v_1^{j,k} - v_1^{j,k-1}\right)^2 +\left(v_2^{j,k} - v_2^{j,k-1}\right)^2 +\left(v_3^{j,k} - v_3^{j,k-1}\right)^2 - (\ell^{j,k})^2
\end{align}
This uses the notational convention that $v^{j,0} \doteq v^{j,s_j}$. For reasons that will be explained, we only explicitly enforce the lengths of two edges ($k=1,2$) per face, so there are a total of $2|x|$ edge length constraints.

Angle constraints ensure that each face's polygonal angles are conserved. Using the dot product formula for angles, we can write this constraint as a polynomial.
\begin{align}
c_{ang}^{j,k}\left(z\right) &= (v^{j,k-1} - v^{j,k})\cdot(v^{j,k+1} - v^{j,k})  - \ell^{j,k}\ell^{j,k+1}\cos(\theta^{j,k})
\end{align}
Here, $\theta^{j,k}$ is the angle at $v^{j,k}$ between $k$th and $(k+1)st$ edges which is a constant that is known a priori. In practice, we only enforce that the $k=1$st angle constraint for each face.

Now, between the two edge constraints and one angle constraints, we have described $3$ constraints as a function of $9$ vertex coordinates per face. Thus for any choice of the $6$ coordinates, $v^{j,1}_1, v^{j,1}_2, v^{j,1}_3, v^{j,2}_1, v^{j,2}_2, v^{j,0}_1$, the remaining $3$ coordinates, $v^{j,2}_3, v^{j,0}_2, v^{j,0}_3$, are specified by the $3$ constraint equations. Similarly, since each face's location and rotational orientation can be defined by this choice of $6$ coordinates, we have enough information to specify the coordinates of the remaining verticies.

Since the positions of the first three verticies dictate the locations of the remaining verticies, the $2D$ face constraints use a map from the known vertex coordinates to the yet unknown locations. Using a template for what the ideal polygonal stucture for each face should be, we use a rotation matrix to specify these remaining verticies. If this template has vertices $\hat{v}^{j,0}, \hat{v}^{j,1}, \hat{v}^{j,2}, \dots, \hat{v}^{j,k}, \dots$ and the locations for $v^{j,0}, v^{j,1}, \text{and } v^{j,2}$ are known, we can identify the location of $v^{j,k}$ for $k>2$. Using this template, we can define the following length and angle constants.
\begin{align}
\ell^{j,k_1, k_2} &\doteq |\hat{v}^{j,k_1} - \hat{v}^{j,k_2}| \\
\phi^{j,k_1,k_2,k_3} &\doteq \cos^{-1}\left(\frac{\left(\hat{v}^{j,k_1} - \hat{v}^{j,k_2}\right)\cdot\left(\hat{v}^{j,k_3} - \hat{v}^{j,k_2}\right)}{(\ell^{j,k_1, k_2})(\ell^{j,k_3, k_2})}\right)
\end{align}

\begin{figure}[ht]
  %\includegraphics[scale=0.1, angle=0]{tetris.eps}
\caption{Face template and 2d face constraints.}
\label{fig:2DFC}
\end{figure}

Our basic stratigy is to first place a point $\bar{v}^{j,k}$ in the span of $v^{j,0} - v^{j,1}$ at a distance of $|\bar{v}^{j,k} - v^{j,1}| = \ell^{j,k,1}$. The choice $\bar{v}^{j,k} = v^{j,1} + \frac{\ell^{j,k,1}}{\ell^{j,0,1}}(v^{j,0} - v^{j,1})$ will work, since
\begin{align}
  |\bar{v}^{j,k} - v^{j,1}| &= |\frac{\ell^{j,k,1}}{\ell^{j,0,1}}(v^{j,0} - v^{j,1})| \\
  &= \frac{\ell^{j,k,1}}{\ell^{j,0,1}}|v^{j,0} - v^{j,1}| \\
  &= \ell^{j,k,1}.
\end{align}
Then, a rotation matrix is used to rotate $\bar{v}^{j,k}$ by the correct angle into its position $v^{j,k}$.
The rotation matrix is centered at $v^{j,1}$ and its axis of rotation is defined by $u = \frac{1}{\ell^{j,0,1}\ell^{j,2,1}}(v^{j,0} - v^{j,1})\times(v^{j,2} - v^{j,1})$.  Similarly, the angle of rotation $\phi^{j,0,1,k}$ is the angle created by the two line segments in the template $(\hat{v}^{j,0},\hat{v}^{j,1})$ and  $(\hat{v}^{j,2},\hat{v}^{j,1})$. Thus, using $R = R(\phi^{j,0,1,k}, u)$ and our equation for $v^{j,k}$ is
\begin{align}
v^{j,k} &= v^{1,k} + R(\bar{v}^{j,k} - v^{j,1})\\
&= v^{1,k} + \ell^{j,k,1}R(v^{j,0} - v^{j,1})\\
\end{align}
Since $R$ is polynomial in $v^{j,0}, v^{j,1}, v^{j,2}$, we get the following polynomial $2$D face constraint for each $k > 2$.
\begin{align}
c_{2D}^{j,k}(z) \doteq v^{1,k} + \ell^{j,k,1}R(v^{j,0} - v^{j,1}) - v^{j,k}
\end{align}

The final constraint type, vertex identification, is used to enforce that the connection between edges of two faces has the mobility of a hinge. To do this, we simply need to ensure that that corresponding vertices on each edge share identical locations. This results in the relatively simple constraints:
\begin{align}
c_{ident}^{j_1,k_1,j_1,k_1,d}(z) \doteq v^{j_1,k_1}_d - v^{j_2,k_2}_d \\
\end{align}
where $v^{j_1,k_1}$ and $v^{j_2,k_2}$ are corresponding vertices from the faces $j_1$ and $j_2$ meeting at a hinged edge. If there are $|E_x|$ of these hinged connections in a building game state $x$, there must be $6|E_x|$ correcponding vertex identification constraints.

With all four constraint types explicitly defined, the aggregate constraint function for $x$ will have $n-m = 2|F_x|+ |F_x| + (N_x - 3|F_x|) + 6|E_x| = N_x + 6|E_x|$ total constraints.

%To mathematically represent each of the possible geometric configurations, we parameterize the embedding using its vertex locations. If a building game state is composed of the faces $\{f_1,\dots, f_i\}$, then any embedding of the intermediate can be described by a point in $\mathbbm{R}^n$, where $n$ is the three times sum $3\sum_{j=1}^i \#\text{vertices}(f_j)$, of the number of vertices in each face. Simply put, the point
% which is the concatenation of all of the vertex locations $v_1, v_2, \dots, v_{n/3} \in \mathbbm{R}^3$ for each verted on each face of an intermediate is used to represent a particular embedding.

\subsection{Special Case of Triangular Faces}

In the case where all of the faces of the polyhedron we consider are triangles (tetrahedron, octahedron, icosahedron, etc.) we notice that

\begin{mycon}
The geometric configuration space for each intermediate composed of equilateral triangular faces is a manifold.
\end{mycon}

%\begin{mycon}
%
%\end{mycon}

\section{Degrees of Freedom}
Roughly speeking, the degrees of freedom of a system are the different independent motions the system is able to exercise. Since the concept of degrees of freedom exists in many diverse scientific fields, such as mechanical engineering and statistical physics, many different formal definitions of degrees of freedom are used in the literature. McCarthy defines degrees of freedom of a mechanical system as follows. CITE
\begin{quote}
We derive formulas for the number of parameters needed to specify the configuration of a mechanism, in terms of the number of links and joints and the freedom of movement allowed at each joint. This number is the \textit{degrees of freedom} or \textit{mobility} of the mechanism. Changing the values of these parameters changes the configuration of the mechanism. Thus, if we view the set of all configuration available to a mechanism as a manifold, then the mobility of the mechanism is the dimension of this manifold.
\end{quote}

Since our geometric configuration space is an algebraic variety and not a manifold in general, this definition must be modified to make degrees of freedom a statistic of each individual configuration rather than a global statistic of the geometric configuration space.

\begin{mydef}
The number of \textbf{degrees of freedom} a building game intermediate at configuration $z$ is the dimension of the geometric configuration space algebraic variety at $z$. If $z$ is a singularity of the algebraic variety, then the degrees of freedom is undefined.
\end{mydef}

In most cases we consider, rigid rotation and rigid translation will preserve the value of the constraint function since they do not move the verticies relative to one another. In other words, if $c(z) = 0$ then we also have $c(Rz + T) = 0$ where $R \in \mathbbm{R}^{n}\times n$ rotates each vertex in the configuration by some $\hat{R} \in SO(3)$ and translates each vertex by $\hat{T} \in \mathbbm{R}^3$. With this definition, $R$ is the block diagonal matrix with each of the $n/3$ block being $\hat{R} \in SO(3)$ and $T \in \mathbbm{R}^n$ is composed of $n/3$ copies of $\hat{T} \in \mathbbm{R}^3$ stacked upon each other. These rigid body rotations account for $6$ of the configuration's degrees of freedom: $3$ rotational degrees of freedom and $3$ translational.
\begin{mydef}
The \textbf{trivial degrees of freedom} a building game intermediate at configuration $z$ are the $6$ degrees of freedom corresponding to rigid body rotations and translations.
\end{mydef}
Thus, the degrees of freedom we are most interested in are those that do represent movement of the verticies and faces relative to each other.
\begin{mydef}
The \textbf{internal degrees of freedom} a building game intermediate at configuration $z$ are the degrees of freedom that are not trival.
\end{mydef}

\begin{mydef}
\textbf{cannonical configuraion}
\end{mydef}

\subsection{Computing Degrees of Freedom}
--First for a general configuration space
--Jacobian equations
--rank of C
--extension to non triangular polyhedra
--implementing on computer

\[
	\frac{\partial\psi_{edge}^{j,k}}{\partial v} =
  	\begin{cases}
        	2\left(v^{j,k}_\bullet-v^{j,k-1}_\bullet\right) 	& \text{if } v = v^{j,k}_\bullet \\
   		-2\left(v^{j,k}_\bullet-v^{j,k-1}_\bullet\right) 	& \text{if } v = v^{j,k-1}_\bullet \\
   		0       & \text{else}
  	\end{cases}
\]

\subsection{Non-Manifold Example}
From this definition, it is worth noting that if the geometric configuration space is indeed a manifold, then the degrees of freedom is the dimension of that manifold which is a global property. Here we present an interesting example in which this is not the case and, depending on the specific configuration, a different number of degrees of freedom exist.

\begin{figure}[ht]
  %\includegraphics[scale=0.1, angle=0]{tetris.eps}
\caption{A linkage of 6 squares with configurations of different degrees of freedom.}
\label{fig:SixSq}
\end{figure}

\subsection{Results}

--Pictures!!

\subsection{Explicit Removal of Trivial Degrees of Freedom}

--fixe center of mass.

%We consider the linkage of rigid 2-dimensional polygons in $\mathbb{R}^3$ by means of ideal hinges located at the edges of the polygons. In this way, by treating intermediates of our various models as such a linkage, we can compute the non-trivial degrees of freedom (DoF) each intermediate has.

%Every linkage of this type has 6 trivial degrees of freedom: 3 corresponding to translational movement and 3 corresponding to rotational freedom. While these freedoms do change the orientation of the intermediate, they do not cause the faces to move relative to each other. For this reason, we label them as trivial degrees of freedom and focus on the calculation the remaining, non-trivial degrees of freedom.

%For our purposes, we define the \textbf{degrees of freedom} of a configuration to be the difference between the number of parameters needed to specify a configuration in an ambient parameter space $\Omega$ and the number of independent constraints imposed upon the configuration as a function of these parameters. Consequently, the non-trivial degrees of freedom is the number of degrees of freedom minus the six trivial degrees of freedom. If the ambiant parameter space is $\mathbb{R}^N$ and there are $M$ constraint equations given by $\varphi : \mathbb{R}^N \to \mathbb{R}^M$, we find the degrees of freedom to be $N - \operatorname{rank}(\varphi)$ and the non-trivial degrees of freedom to be $N - \operatorname{rank}(\varphi) - 6$. Here we need $\varphi$ to be a smooth map so that the rank is well defined. Furthermore, we define the \textbf{configuration space} to be the subset of parameter space $\{\omega \in \Omega : \varphi(\omega) = 0\}$ upon which the contraints are satisfied.

%In our case, the location and orientation of each polygon in the linkage can be represented by a total of 6 parameters: 3 for location and 3 for roational orientation. Thus, the entire configuration of an intermediate $x$ can be represented in the parameter space $\mathbb{R}^{F_x\times6}$. With constraints corresponding to hinged connections we construct a function $\varphi$ which is zero if and only if the constraints are satisfied. Thus the configuration space is $\{z \in \mathbb{R}^{F_x\times6} : \varphi(z) =0\}$. As we will show below, the constraint equation $\varphi$ can be constructed as a polynomial which means that the configuration space (as the zero set of this polynomial) is an algebraic variety.

% PARAGRAPH ON DIFFERENT DOF VALS AT DIFFERENT CONFIGS OF SAME INTERMEDIATE We refer to the embedded configuration of $x$ such that $x$ is a subset of the original polyhedron as the \textbf{canonical configuration}.

%The base constaints are put in place to fix one of the intermediate's faces in space to remove the 6 trivial degrees of freedom from the calculation. If $f_b$ is the face we wish to designate as the base we have the following $3\times s_b$ constraint equations
%\begin{align}
%\psi_{base}^{k,x}\left(\mathbf{v}\right)& = v^{b,k}_x - c^{b,k}_x \\
%\psi_{base}^{k,y}\left(\mathbf{v}\right)& = v^{b,k}_y - c^{b,k}_y \\
%\psi_{base}^{k,z}\left(\mathbf{v}\right)& = v^{b,k}_z - c^{b,k}_z
%\end{align}
%for $k = 1,\dots,s_b$ and with $c^{b,k}_\bullet$ a known constant. The corresponding Jacobian calculation for each constraint equations is:
%\[
%\frac{\partial\psi_{base}^{k,\bullet}}{\partial v} =
%  \begin{cases}
%   1 & \text{if } v = v^{b,k}_\bullet \\
%   0       & \text{else}
%  \end{cases}
%\]

%Edge length constraints enforce that the lengths of the edges of each face in an intermediate cannot change. Since there are $s_j$ edges for each face $f_j$ of intermediate $x$, there are $N_x$ corresponsing edge constraints.
%\begin{align}
%\psi_{edge}^{j,k}\left(\mathbf{v}\right)& = \left|v^{j,k} - v^{j,k-1}\right|^2 - \ell_{j,k}^2 \\
%& = \left(v_x^{j,k} - v_x^{j,k-1}\right)^2 +\left(v_y^{j,k} - v_y^{j,k-1}\right)^2 +\left(v_z^{j,k} - v_z^{j,k-1}\right)^2 - \ell_{j,k}^2
%\end{align}
%for all $j: f_j \subset x$ and with the convention that $v^{j,0} \doteq v^{j,s_j}$ and $\ell_{j,k}$ is a known constant. The resulting partial derivatives are
%\[
%	\frac{\partial\psi_{edge}^{j,k}}{\partial v} =
%  	\begin{cases}
%        	2\left(v^{j,k}_\bullet-v^{j,k-1}_\bullet\right) 	& \text{if } v = v^{j,k}_\bullet \\
%   		-2\left(v^{j,k}_\bullet-v^{j,k-1}_\bullet\right) 	& \text{if } v = v^{j,k-1}_\bullet \\
%   		0       & \text{else}
%  	\end{cases}
%\]
%--definiteion of 'base configuration'
%--representation of config space definition as system of constraint equations
%--solution of the constraint equations is a manifold except perhaps at some 'sp0ecial' configurations
%--define DoF as 6N - rank of the jacobian of the CEqs evaluated at the base config

%--go through computation of jacobian

\section{Cyclohexane Application}
??Include??
%\subsection{Sachse Model}
%
%Around the turn of the century, it was thought that cyclohexane's carbon atoms must lie in a plane. A young German assistant, Hermann Sachse, had the idea that allowing the carbons to lie outside the plan could alleviate the angle strain. Inspired by polyhedral geometry, he templates and outlined methods for creating 3D models of the chair and boat configurations his new theory conceptualized. Figure~\ref{fig:sachse} shows a construction of these two models. Despite his best efforts, Sachse's ideas were not accepted by that chemistry community until after his death.
%
%
%
%\begin{figure}
%        %\centering
%        %\begin{subfigure}[b]{0.24\textwidth}
%        %        \includegraphics[width=\textwidth]{chair_model_2.png}
%        %        \caption{Chair (top)}
%        %        \label{fig:CM1}
%        %\end{subfigure}%
%        %~ %add desired spacing between images, e. g. ~, \quad, \qquad etc.
%        %  %(or a blank line to force the subfigure onto a new line)
%        %\begin{subfigure}[b]{0.24\textwidth}
%        %        \includegraphics[width=\textwidth]{chair_model_1.png}
%        %        \caption{Chair (side)}
%        %        \label{fig:CM2}
%        %\end{subfigure}%
%        %~ %add desired spacing between images, e. g. ~, \quad, \qquad etc.
%        %  %(or a blank line to force the subfigure onto a new line)
%        %\begin{subfigure}[b]{0.24\textwidth}
%        %        \includegraphics[width=\textwidth]{boat_model_1.png}
%        %        \caption{Boat (top)}
%        %        \label{fig:BM1}
%        %\end{subfigure}%
%        %~ %add desired spacing between images, e. g. ~, \quad, \qquad etc.
%        %  %(or a blank line to force the subfigure onto a new line)
%        %\begin{subfigure}[b]{0.24\textwidth}
%        %        \includegraphics[width=\textwidth]{boat_model_2.png}
%        %        \caption{Boat (side)}
%        %        \label{fig:BM2}
%        %\end{subfigure}%
%        %\caption{Sachse Models}\label{fig:sachse}
%\end{figure}
%
%
%\subsection{Idealized Constraint Model}\label{building_game}
%
%Withe the eclipsing and angle strains in mind, we heuristically define an idealized model of cyclohexane by imposing geometric constraints. Each configuration is represented by the 3-dimensional locations of the center of its carbon atoms and we explicitly parameterize these locations as $v_1, v_2, \dots, v_6$ where $v_k \in \mathbb{R}^3$. For ease of exposition, $v_0 \doteq v_6$ and $v_{-1} \doteq v_5$ are notationally identified.
%
%First, we require that any two atoms sharing a bond have a known and fixed distance $\ell$ from each other. Since we can re-scale our coordinate system, we assume $\ell \doteq 1$. Additionally, we assume that the connectivity of the cyclohexane molecule is such that $v_k$ is bonded to $v_{k-1}$ for $k = 1,\dots,6$. This gives our first six constraint equations.
%$$0 = \varphi_{len}^k\left(v_{k-1},v_{k}\right) = \|v_k-v_{k-1}\| - 1 $$
%for $k = 1,\dots,6$.
%
%Additionally, we impose constraints representing the angle strain. Since the carbon atoms have the lowest energy when their bonds are at tetrahedral angles, we fix the angle of each set of three adjacent carbons to be at the tetrahedral angle. This is equivalent to the six angle constraint equations
%$$0 = \phi_{ang}^k\left(v_{k-2},v_{k-1},v_{k}\right) = \left(v_k-v_{k-1}\right)\cdot \left(v_{k-2}-v_{k-1}\right) + \frac{1}{3}$$
%for $k = 1,\dots,6$.
%
%\subsection{Degrees of Freedom in Ideal Model}
%
%The first step in answering these questions is determining whether there is any degrees of freedom to each configuration or whether they are rigid. This can be determined by using established theory on rigidity of linkages and degrees of freedom. By computing the Jacobian matrix $J(v)$ of the system of constraint equations, we have the following equation for the degrees of freedom for a constraint satisfying set of coordinates $v$.
%$$DoF(v) = 18 - rank(J(v))$$
%When the Jacobian is of full rank 18, none of the constraints are dependent on each other and thus there is no freedom in the linkage. When testing the chair, we found there to be $0$ degrees of freedom, meaning that it is impossible to make a transition from the chair to another configuration without breaking one or more of the constraint equations. For the boat, however, one degree of freedom was found. This result shows that it is possible to deform the boat continuously while satisfying the constraints, but it is not informative as to which configurations it can deform to. In particular, we are interested in finding a path to he twist boat or another boat configuration.
%

\section{Folding Configuration Space}
???include??
%
%The number of degrees of freedom measures the rigidity of an intermediate, but some intermediates with the same number of degrees of freedom may have varying degrees of mobility. We seek to quantify an intermediate's mobility by the relative amount of movement the intermediate has for a small movement in its configuration space. Intermediates with the most mobility will move less than the less mobile intermediates for the same amount of movement in configuration space.
%
%\subsection{Methods}
%% What is the configuration space?
%Each octahedron intermediate is composed of 8 equilateral triangles connected to each other along edges. We make the assumption that these triangles are rigid and cannot be deformed. Furthermore, when two triangles meet at an edge, they move relative to each other as if connected by an ideal hinge. Every configuration has 6 trivial degrees of freedom corresponding to the 3 translation degrees of freedom and the 3 rotational degrees of freedom. Since we are interested in the motion of an intermediate's faces relative to each other, we remove these trivial degrees of freedom by picking a single face to fix in space. Depending on the connectivity of the triangle's edges, an intermediate
%may still have degrees of freedom. The Octahedron intermediate (83), being a convex polyhedron, is rigid and has no non-trivial degrees of freedom as given by Cauchy's Theorem. However, this is not the case for most of the intermediates.
%
%We formalize the concept of the configuration space, by defining it to be the subset of an ambient parameter space that satisfies constraint equations that correspond to our assumptions. Since the are 8 faces in each intermediate and each face has 3 vertices each with 3 spacial coordinates (x,y,z), we use $\mathbb{R}^{8\times 3 \times 3} = \mathbb{R}^{72}$ as our ambient space. We then have 3 types of constraint equations: base face constraints, rigid face constraints, and hinge constraints. Every admissible configuration of an intermediate will be a point in $\mathbb{R}^{72}$ that satisfies the constraint equations and any admissible movement of a configuration (if possible) is a continuous movement in the subset of $\mathbb{R}^{72}$ where these constraints are satisfied.
%
%Since our constraint equations can all be expressed as polynomials, the configuration space which is the corresponding solution set is an Algebraic variety. The number of degrees of freedom of a configuration is the local dimension of this algebraic variety as a subset of $\mathbb{R}^{72}$. In the case of octahedron intermediates, the number of degrees of freedom is not an informative way of classifying which intermediates are dominant as intermediates $1-11$ have $7$ degrees of freedom, $12-33$ have $5$, $34-65$ have $3$, $66-82$ have $1$ and the Octahedron (83) and Boat (84) have $0$ degrees of freedom. Thus, a different measure the mobility of a configuration is required to differentiate intermediates. It is important to note that degrees of freedom of in intermediate can theoretically change based on the region of configuration space. However, in practice we have not observed an intermediate's degrees of freedom change in this way.
%
%We define the \textit{canonical configuration} of an intermediate, to be that in which each hinge forms a $180^\circ$ angle when the hinge does not have a vertex connection at either end and in the case of a closed vertex, the hinge's angle corresponds to the Octahedral angle. In cases where the intermediate cannot form an octahedron, we can use the Boat configuration's angles instead.
%
%% How do we move in the configuration space?
%Given a particular configuration $X$, if we wish to make a small move in configuration space, we first find the null space of the Jacobian matrix of the constraint equations. Since this null space corresponds to the directions in which the constraint equations are not changing, the constraints will remain satisfied. The null space can be represented by an orthonormal basis $N \in \mathbb{R}^{72\times d}$, where $d$ is the number of degrees of freedom of the configuration. Then, by taking a small step of size $\epsilon$ in each of the $d$ directions we get the configurations $X + \epsilon N_{\cdot,k}$.
%% How do we measure the magnitude of a movement in configuration space?
%
%We define the \textit{mobility} of a configuration to be the $L^2$ norm of the gradient of the configuration space. To approximate this, we measure the mean squared distance between each point of $X$ and $X \pm \epsilon N_{\cdot,k}$ in each of the $d$ directions of $N$, take the norm, and divide by $\epsilon$. This is given by Equation~\ref{eq:r1} where $\triangle_f$ refers to the $f$th face of the intermediate and $r\left(X\right)$ is the point we are integrating over and $r\left(X+\sigma \epsilon N_{\cdot,k}\right)$ is the corresponding point in the altered configuration.
%
%
%To explicitly define the rotation and translation of each face by the movement in configuration space, we use the algorithm given by Arun et al~\cite{Arunetal1987}. Given two sets of points $P,P' \in \mathbb{R}^{3\times n}$ in 3D space, the algorithm finds the rotation matrix $R$ and translation vector $b$ minimizing the least squares error of $P' \approx R P + b$. By using the three vertices of face $f$ as $P$ and their perturbed values as $P'$, we use this algorithm to find the rotation matrix $R_f$ and translation vector $b_f$ to describe the rigid movement of face $f$ in the configuration space. This leads to the formulation in Equation~\ref{eq:r3}.
%\begin{figure}[h]
%  %\centering
%  %\includegraphics[width=0.61\textwidth]{triangle_fig.png}
%  %\caption{Coordinates of rotated triangle for integration.}
%  \label{fig:tri}
%\end{figure}
%
%To make this integral easier to solve, we introduce a change of variables that enables us to integrate in the $x-y$ plane. If the the original triangle $f$ has vertices of $a', b',$ and $c'$, we consider the triangle with vertices at the coordinates $a = (0,0,0), b = (|a-b|,0,0),$ and $c = (\alpha,\beta,0)$ as seen in Figure~\ref{fig:tri}. Here, the choices $\alpha = \frac{|a-b|^2 + |a-c|^2 - |b-c|^2 }{2|a-b|}$ and $\beta = \sqrt{|a-c|^2-\alpha^2}$ yield a triangle congruent to the original. Using the Arun et al~\cite{Arunetal1987} algorithm again, we find the rotation matrix $S_f$ and translation vector $c_f$ that gives $[a',b',c'] = S_f[a,b,c] + c_f$. Using this we perform a change of variables and integrate over $s$ and $t$. This results in an simple double integral over a quadratic function of $s$ and $t$ with constants $u,v,w \in \mathbb{R}^3$ as seen in Equation~\ref{eq:r5}.
%{\tiny
%\begin{align}
%\mathcal{R}\left(x\right) &= \frac{1}{\epsilon}\left(\sum_{k=1}^d \sum_{f=1}^8 \int_{\triangle_f}\left|\frac{r\left(x+\epsilon N_{\cdot,k}\right) - r\left(x\right)}{2\sqrt{3}}\right|^2\right)^{\frac{1}{2}} \label{eq:r1}  \\
%&= \frac{1}{2\sqrt{3}\epsilon}\left( \sum_{k=1}^d \sum_{f=1}^8 \int_{\triangle_f}\left|R_fr+b_f - r\right|^2\right)^{\frac{1}{2}} \label{eq:r2} \\
%&= \frac{1}{2\sqrt{3}\epsilon}\left( \sum_{k=1}^d \sum_{f=1}^8 \int_{\triangle_f}\left|\left(R_f-I\right)r+b_f\right|^2\right)^{\frac{1}{2}} \label{eq:r3} \\
%&= \frac{1}{2\sqrt{3}\epsilon}\left( \sum_{k=1}^d \sum_{f=1}^8 \int_{0}^{\beta}\int_{\frac{\alpha}{\beta}t}^{|a-b| + \frac{\alpha- |a-b|}{\beta}t}\left|\left(R_f-I\right)\left(S_f\begin{bmatrix} s \\[0.3em] t \\[0.3em] 0 \end{bmatrix} + c_f\right)+b_f\right|^2ds dt\right)^{\frac{1}{2}} \label{eq:r4} \\
%&= \frac{1}{2\sqrt{3}\epsilon}\left( \sum_{k=1}^d \sum_{f=1}^8 \int_{0}^{\beta}\int_{\frac{\alpha}{\beta}t}^{|a-b| + \frac{\alpha- |a-b|}{\beta}t}\left|u + sv + tw\right|^2ds dt\right)^{\frac{1}{2}} \label{eq:r5}
%\end{align}
%}
%% Why does the choice of the base constraint matter and how do we deal with it?
%
%Unfortunately, the particular choice of base face affects the mobility. To rectify this bias, we compute the mobility with each of the eight faces as the base and take the average. This gives the final mobility in Equation~\ref{eq:r6}.
%{\tiny
%\begin{align}
%\mathcal{R} &= \frac{1}{8}\sum_{g=1}^8\mathcal{R}_g \\
%&= \frac{1}{16\sqrt{3}\epsilon}\sum_{g=1}^8\left( \sum_{k=1}^d \sum_{f=1}^8 \int_{0}^{\beta}\int_{\frac{\alpha}{\beta}t}^{|a-b| + \frac{\alpha- |a-b|}{\beta}t}\left|u + sv + tw\right|^2ds dt\right)^{\frac{1}{2}} \label{eq:r6}
%\end{align}
%}
%
%\section{Results}
%
%Mobility was computed for all octahedral intermediates in their canonical configuration. The perturbation parameter $\epsilon = 10^{-6}$ was selected, but the mobility showed to be robust to both smaller and larger values. Figure~\ref{fig:t1} outlines the results. From these calculations, it is clear that while mobility is highly correlated to degrees of freedom, mobility gives additional information about an intermediate's configuration space. Interestingly, all nets had the same mobility. Similarly, the symmetric pairs (14,16), (15,18), (19,20), (34,38), and (36,39) have matching mobility values. Of the intermediates with 5 degrees of freedom, 19 and 20 had the lowest mobility and 17 had the highest. As for intermediates with 3 degrees of freedom, 35 was the least mobile and 37 was the most mobile.
%
%\begin{figure}[h!]
%\label{fig:t1}
%%\centering
%\scalebox{.9}{
%\begin{tabular}{ l | c }
%  Intermediate & Mobility  \\
%  \hline\hline
%1 &  0.20518 \\
%2 &  0.20518 \\
%3 &  0.20518 \\
%4 &  0.20518 \\
%5 &  0.20518 \\
%6 &  0.20518 \\
%7 &  0.20518 \\
%8 &  0.20518 \\
%9 &  0.20518 \\
%10 & 0.20518 \\
%11 & 0.20518 \\\hline
%12 & 0.18376 \\
%13 & 0.18313 \\
%14 & 0.18249 \\
%15 & 0.18171 \\
%16 & 0.18249 \\
%17 & 0.18634 \\
%18 & 0.18171 \\
%19 & 0.18102 \\
%20 & 0.18102 \\
%21 & 0.18255 \\
%22 & 0.18485 \\\hline
%34 & 0.14487 \\
%35 & 0.14350 \\
%36 & 0.14530 \\
%37 & 0.14973 \\
%38 & 0.14487 \\
%39 & 0.14530 \\\hline
%66 & 0.07954 \\\hline
%83 & 0.00000 \\
%\end{tabular}
%}
%%\caption{A table with the mobility of octahedral intermediates.}
%\end{figure}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{geometry}
\usepackage[utf8]{inputenc}
\usepackage{enumitem}
\usepackage{physics}
\setlength{\parindent}{0pt}
\setlength{\parskip}{1em}
\relpenalty=10000
\binoppenalty=10000

\begin{document}

\section*{Graph theory}
\subsection*{Terminology}
\begin{enumerate}
	\item
	Graph $G(V,E)$ is set of \emph{vertices} $V$ and \emph{edges} $E$ (pairs of vertices). If there is an edge $(x,y) \in E$, then $x$ and $y$ are said to be \emph{connected}.
	\item
	A \emph{proper graph} is a graph with no more than one edge between each pair of vertices and in which no vertex is connected to itself.
	\item
	Proper graph in which each pair of vertices is connected is called a \emph{complete graph}.
	\item
	A \emph{complement} graph $\overline{G}$ has vertices equal to vertices of $G$ and its vertices $x$ and $y$ are connected if and only if $x$ and $y$ are not connected in $G$.

	\item
	An \emph{oriented graph} is one in which the pairs in the graph are ordered.
	\item
	The vertices of a \emph{k-partite graph} can be partitioned into $k$ non-empty disjoint sets in such way that there are no connections between vertices in each set.
	\item
	The \emph{degree} of a vertex $x$ is the number of times $x$ is the endpoint of an edge. Then
	$$\sum_{x \in V} d(x) = 2 \abs{E} $$
	\item
	A \emph{trajectory (or path)} is a sequence of vertices for which subsequent vertices are connected.
	\item
	A \emph{circuit} is a path that ends and starts in the same vertex.
	\item
	A \emph{cycle} is a circuit in which no vertex appears more than once (except the initial/final vertex).
	\item
	In a \emph{connected graph} there exists a path between any two points of the graph.
	\item
	\emph{Tree} is a connected graph with no circuits. A connected graph with $n$ vertices is a tree if and only if it has $n-1$ edges.
	\item
	\emph{Euler path} is a path in which every edge of the graph appears exactly once. Likewise \emph{Euler circuit} is a circuit in which every edge appears exactly once.
	\item
	If each vertex in a connected graph has even degree, then the graph contains an Euler circuit.
	\item
	If a connected graph has exactly two vertices with odd degree, it contains an Euler path.
	\item
	A \emph{Hamilton circuit} is a circuit in which each vertex appears exactly once.
	\item
	A \emph{planar graph} can be embedded in a plane with edges corresponding to non-intersecting lines (not necessarily straight). A planar graph with $n$ vertices has at most $3n-6$ edges.

\end{enumerate}

\newpage
\subsection*{Theorems}
	\begin{enumerate}
		\item \textbf{Ramsey's theorem: } For positive integers $r_1,\dots,r_k$, there exists a finite Ramsey number.
		\begin{enumerate}
		 	\item case $k=2$ \\
		 	Ramsey number $R(r_1,r_2)$ is a minimal number of people at the party such that there are either $r_1$ people who know each other or $r_2$ people who do not.
		 	\item General case for $k$ \\
		 	Ramsey's number of $R(r_1,r_2,\dots,r_k)$ is the minimal number of points in a complete graph whose edges are coloured with $k$ different colours, such that there exists a colour $i$ for which we can find $r_i$ vertices that are all connected with edges of colour $i$.
		\end{enumerate}

		\item \textbf{Kuratowski's theorem} % http://www.math.caltech.edu/~2014-15/2term/ma006b/10%20Planar3.pdf
		\begin{enumerate}
			\item $K_n$ is a complete graph with $n$ vertices and $K_{n,m}$ is a complete bipartite graph with $n$ and $m$ vertices in the subsets.
			\item Graphs $K_5$ and $K_{3,3}$ are non-planar.
			\item Every non-planar graph contains a subgraph which can be obtained from one of these graphs by a subdivision of its edges.

		\end{enumerate}
		\item \textbf{Hall's marriage theorem} % http://www.math.cornell.edu/~levine/18.312/alg-comb-lecture-16.pdf
		\begin{enumerate}
			\item In a set of $n$ women and $n$ men, a woman $a_i$ is happy to be married to any man in set $A_i$ and a man is happy to marry a woman who wants to marry him. Hall's marriage theorem states that each person can be happily married if and only if the sets $A_1,A_2,\dots,A_n$ meet the marriage condition.
			\item The marriage condition states that for any subset of women $I$, the number of men whom at least one woman would be happy to marry $\abs{\bigcup_{i \in I}A_i} \geq \abs{I}$
		\end{enumerate}
	\end{enumerate}

\subsection*{Problems}
\begin{enumerate}
	\item
	Prove that at least one of $G$ and $\overline{G}$ is connected.

	\item
	\emph{(Dirac's theorem).} Prove that a graph with $n$ vertices contains a Hamilton cycle if the degree of each vertex is at least $n/2$.

	\item
	\emph{(Euler's formula).}  A convex polyhedron has $E$ edges, $F$ faces and $V$ vertices. Prove that $E+2=F+V$.

	\item % http://www.math.olympiaadid.ut.ee/arhiiv/baltitee/bt1997/bw97est.pdf
	In a forest each of n animals ($n \geq 3$) lives in its own cave, and there is exactly one separate path
	between any two of these caves. Before the election for King of the Forest some of the animals
	make an election campaign. Each campaign-making animal visits each of the other caves exactly
	once, uses only the paths for moving from cave to cave, never turns from one path to another
	between the caves and returns to its own cave in the end of its campaign. It is also known that
	no path between two caves is used by more than one campaign-making animal.
	\begin{enumerate}
		\item Prove that for any prime $n$, the maximum possible number of campaign-making animals is $\tfrac{n-1}{2}$
		\item Find the maximum number of campaign-making animals for $n = 9$.
	\end{enumerate}

	\item % https://mks.mff.cuni.cz/kalva/usa/usoln/usol991.html
	Certain squares of an $n \times n$ board are coloured black and the rest white. Every white square shares a side with a black square. Every pair of black squares can be joined by chain of black squares, so that consecutive members of the chain share a side. Show that there are at least $\tfrac{n^2-2}{3}$ black squares.

	\item % https://cms.math.ca/Competitions/OMC/archive/sol2006.pdf
	Consider a round-robin tournament with $2n + 1$ teams, where each team plays each other team exactly once. We say
	that three teams $X$, $Y$ and $Z$, form a \emph{cycle triplet} if $X$ beats $Y$ , $Y$ beats $Z$, and $Z$ beats $X$. There are no ties.
	\begin{enumerate}
		\item Determine the minimum number of cycle triplets possible.
		\item Determine the maximum number of cycle triplets possible.
	\end{enumerate}

	\item % https://artofproblemsolving.com/wiki/index.php?title=2001_IMO_Shortlist_Problems/C3
	Define a $k$-clique to be a set of $k$ people such that every pair of them are acquainted with each other. At a certain party, every pair of 3-cliques has at least one person in common, and there are no 5-cliques. Prove that there are two or fewer people at the party whose departure leaves no 3-clique remaining.

	\item  % Latin squares
	For integers $r,n$ where $r<n$ a grid $r \times n$ is populated with numbers $1,2,\dots,n$, such that no number occurs twice in any row or column. Prove that it is possible to append $n-r$ rows, so that the condition is still met in the new $n \times n$ grid.

	\item % https://www.imo-official.org/problems/IMO2010SL.pdf p26
	On some planet, there are $2^N$ countries $(N \geq 4)$. Each country has a flag $N$ units wide and one unit high composed of $N$ fields of size $1 \times 1$, each field being either yellow or blue. No two countries have the same flag. We say that a set of $N$ flags is \emph{diverse} if these flags can be arranged into an $N \times N$ square so that all $N$ fields on its main diagonal will have the same colour. Determine the smallest positive integer $M$ such that among any $M$ distinct flags, there exist $N$ flags forming a diverse set.

	\item %http://www.math.olympiaadid.ut.ee/arhiiv/valik/vv2018/tvv2018.pdf ül5
	Let $k$ be a positive integer. Find all positive integers $n$ for which it is possible to choose $n$ points on the sides of a triangle (different from its vertices) and connect some of them with a line such that
	\begin{enumerate}
		\item There is at least $1$ point on each side
		\item For each pair of points $X$ and $Y$ which are on different sides of the triangle, there exists exactly $k$ points on the third side which are all connected to both $X$ and $Y$, and exactly $k$ points which are all connected to neither of $X$ or $Y$.
	\end{enumerate}

	\item %Valgevene (TVV2018-5-vana)
	A country has $n \geq 3$ airports. Two-way flights operate between between some pairs of the airports. It is known that for each pair of airports, there exists a third one which is not directly connected to either of them. Find the maximum number of two-way flights.

\end{enumerate}

\end{document}


\section{Additive Models}
Additive models are specific application of projection pursuit. They
are more useful in scientific applications.

In additive models we assume that the response is linear in the
predictors effects and that there is an additive error. This allows us to study the effect of each
predictor separately. The model is like (\ref{guasreg}) with
\[
f(X_1,\dots,X_p) = \sum_{j=1}^p f_j(X_j).
\]
Notice that this is projection pursuit with the projection
\[
\bg{\alpha}_j'\bX = X_j.
\]
The assumption made here is not as strong as in linear regression, but
its still quite strong. It's saying that the effect of each covariate
is additive. In practice this may not make sense.

Example: In the diabetes example consider an additive model that
models log(C-peptide)
in terms of age  $X_1$ and base deficit $X_2$. The additive
model assumes that for two
different ages $x_1$ and $x_1'$ the conditional expectation of $Y$
(seen as a random variable depending on base deficit):
\[
\E(Y | X_1=x_1, X_2) = f_1(x_1) + f_2(X_2)
\]
and
\[
\E(Y | X_1=x_1', X_2) = f_1(x_1') + f_2(X_2).
\]
This say that the way C-peptide depends on base deficit only varies by a
constant for different ages. It is not easy to disregard the
possibility that this dependence changes. For example, at
older ages the effect of high base deficit can be dramatically bigger. However,
in practice we have too make assumptions lik these in order to get
some kind of useful description of the data.

Comparing the non-additive smooth (seen above) and the additive model
smooth shows that it is not completely crazy to assume
additivity.

\centerline{\epsfig{figure=Plots/plot-07-04.ps,angle=270,width=\textwidth}}

Notice that in the first plots the curves defined for the different
ages are different. In the second plot they are all the same.

How did we create this last plot? How did we fit the additive
surface. We need to estimate $f_1$ and $f_2$. We will see this in the
next section.

Notice that one of the advantages of additive model is that no matter
the dimension of the covariates we know what the surface
$f(X_1,\dots,X_p)$ is like by drawing each $f_j(X_j)$ separately.

\centerline{\epsfig{figure=Plots/plot-07-05.ps,angle=270,width=.8\textwidth}}

\subsection{Fitting Additive Models: The Backfitting Algorithm}
Conditional expectations provide a simple intuitive motivation for the
backfitting algorithm.

If the additive model is correct then for any $k$
\[
\E\left(\left. Y - \alpha - \sum_{j\neq k} f_j(X_j) \, \right| \,X_k
\right) = f_k(X_k)
\]
This suggest an iterative algorithm for computing all the $f_j$.

Why? Let's say we have estimates $\hat{f}_1,\dots,\hat{f}_{p-1}$ and we
think they are ``good'' estimates in the sense that $\E\{{f}_j(X_j) -
f_j(X_j)\}$ is ``close to 0''. Then we have that
\[
\E\left(\left.Y - \hat{\alpha} - \sum_{j=1}^{p-1} \hat{f}_j(X_j) \,
  \right| \, X_p \right)
\approx f_p(X_p).
\]
This means that the partial residuals $\hat{\epsilon} = Y -
\hat{\alpha} - \sum_{j=1}^{p-1} \hat{f}_j(X_j) $
\[
\hat{\epsilon}_i \approx f_p(X_{ip}) + \delta_i
\]
with the $\delta_i$ approximately IID mean 0 independent of the
$X_p$'s. We have already discussed various ``smoothing'' techniques
for estimating $f_p$ in a model as the above.

Once we choose what type of smoothing technique we are using
for each covariate, say its defined by $S_j(\cdot)$, we obtain an
estimate for our
additive model following these steps
\begin{enumerate}
\item Define $\f_j = \{f_j(x_{1j}),\dots,f_j(x_{nj})\}'$ for all $j$.
\item Initialize: $\alpha^{(0)} = \mbox{ave}(y_i)$, $\f_j^{(0)} = $ linear
  estimate.
\item Cycle over $j=1,\dots,p$
\[
\f_j^{(1)} = S_j\left(\left. \by - \alpha^{(0)} - \sum_{k\neq j}
  \mathbf{f}^{(0)}_k
  \, \right| \bx_j \right)
\]
\item Continue previous step until functions ``don't change'', for
  example until
\[
\max_{j} \left|\left| \f_j^{(n)} - \f_j^{(n-1)} \right|\right| < \delta
\]
with $\delta$ is the smallest number recognized by your computer. In
my computer using S-Plus its: \begin{verbatim} .Machine$double.eps = 2.220446e-16 \end{verbatim} %$

\end{enumerate}

Things to think about:

Why is this algorithm valid? Is it the solution to some minimization
criterion?  Its not MLE or LS.

\subsection{Justifying the backfitting algorithm}
The backfitting algorithm seems to make sense. We can say that we have
given an intuitive justification.

However statisticians usually like to have more than this. In most
cases we can find a ``rigorous'' justification. In many cases
the assumptions made for the ``rigorous'' justifications too work are
carefully chosen so that we get the answer we want, in this case that the
back-fitting algorithm ``converges'' to the ``correct'' answer.

In the GAM book, H\&T find three ways to justify it: Finding
projections in $L^2$ function spaces, minimizing certain criterion
with solutions from reproducing-kernel Hilbert spaces, and as
the solution to penalized least squares. We will look at this last
one.

We extend the idea of penalized least squares by considering the
following criterion
\[
\sum_{i=1}^n \left\{ y_i - \sum_{j=1}^p f_j(x_{ij}) \right\}^2 +
  \sum_{j=1}^p \lambda_j \int \{f_j''(t)\}^2 \, dt
\]
over all p-tuples of functions $(f_1,\dots,f_p)$ that are twice
differentiable.

As before we can show that the solution to this problem is a p-tuple
of cubic splines with knots ``at the data'', thus we may rewrite the
criterion as
\[
\left(\by - \sum_{j=1}^p \f_j \right)'\left(\by - \sum_{j=1}^p \f_j
\right) + \sum_{j=1}^p \lambda_j \f_j \bK_j \f_j
\]
where the $\bK_j$s are penalty matrices for each predictor defined
analogously to the $\bK$ of section 3.3.

If we differentiate the above equation with respect to the function
$\f_j$ we obtain $-2(\by - \sum_k \f_k) + 2 \lambda_j \bK_j \f_j =
0$. The $\hat{\f}_j$'s that solve the above equation must satisfy:
\[
\hat{\f}_j = \left(\bI + \lambda_j \bK_j\right)^{-1}\left(\by -
\sum_{k\neq j} \hat{\f}_k\right), j=1,\dots,p
\]
If we define the smoother operator $\bS_j = \left(\bI + \lambda_j
\bK_j\right)^{-1}$ we can write out this equation in matrix notation as
\[
\left(
\begin{array}{cccc}
\bI&\bS_1&\dots&\bS_1\\
\bS_2&\bI&\dots&\bS_2\\
\vdots&\vdots&\ddots&\vdots\\
\bS_p&\bS_p&\dots&\bI
\end{array}
\right)
\left(
\begin{array}{c}
\f_1\\
\f_2\\
\vdots\\
\f_p
\end{array}
\right)
=
\left(
\begin{array}{c}
\bS_1 \by\\
\bS_2 \by\\
\vdots\\
\bS_p \by
\end{array}
\right)
\]
One way to solve this equation is to use the Gauss-Seidel algorithm
which in turn is  equivalent to
solving the back-fitting algorithm. See Buja, Hastie \& Tibshirani
(1989) Ann. Stat. 17, 435--555 for details.

Remember that
 that for any set of linear smoother
\[
\hat{\f}_j = \bS_j\by
\]
we can argue in reverse that it minimizes some penalized least squares
criteria of the form
\[
(\by - \sum_j\f_j)'(\by - \sum_j\f_j) + \sum_j \f_j'(\bS_j^{-} - I)\f_j
\]
and conclude that it is the solution to some penalized least squared
problem.

\subsection{Standard Error}
When using {\tt gam()} in S-Plus we get point-wise standard
errors. How are these obtained?

Notice that our estimates
$\hat{\f}_j$ are no longer of the form $\bS_j \by$ since we have used a
complicated backfitting algorithm. However, at convergence we can
express $\hat{\f}_j$ as $\bR_j \by$ for some $n \times n$ matrix
$\bR_j$. In practice this $\bR_j$ is obtained from the last calculation
of the $\hat{\f}_j$'s but finding a closed form is rarely
possible.

Ways of constructing confidence sets is not straight forward, and (to the best
of my knowledge) is an open area of research.


\section{Recurrences}

\begin{frame}{Recurrences}
  \[
	T(n) = aT(n/b) + f(n) \quad (a > 0, b > 1)
  \]

  \begin{equation*}
	\left.\begin{aligned}
	  f(n) \\
	  af(\frac{n}{b}) \\
	  a^2f(\frac{n}{b^2}) \\
	  \vdots \\
	  a^{\log_b^n}f(1) = n^{\log_b^a}
	\end{aligned}\right\}
	\pause
	\sum = \left\{\begin{aligned}
	  n^{\log_b^a}	\quad & q > 1 \\
	  n^{\log_b^a}\log n \quad & q = 1 \\
	  f(n)	\quad & q < 1 \quad \\
	\end{aligned}\right.
	\pause
	\quad \left.\begin{aligned}
	  f(n) &= O(n^{E-\epsilon}) \\
      f(n) &= \Theta(n^E) \\
      f(n) &= \Omega(n^{E+\epsilon})
	\end{aligned}\right.
  \end{equation*}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Solving recurrences (Problem 1.2.13, 1.2.16)}
  \begin{columns}
	\column{0.40\textwidth}
	  \begin{enumerate}
		\item $\Theta(n^{\log_3^2})$
		\item \textcolor{red}{$\Theta(\log^2 n)$}
		\item $\Theta(n)$
		\item $\Theta(n \log n)$
		\item \textcolor{red}{$\Theta(n \log^2 n)$}
		\item $\Theta(n^2)$
		\item $\Theta(n^{\frac{3}{2}}\log n)$
		\item $\Theta(n)$
		\item $\Theta(n^{c+1})$
		\item $\Theta(c^{n+1})$
		\item \textcolor{blue}{$\Theta(n)$}
	  \end{enumerate}
	\column{0.60\textwidth}
	  \[
		T(n) = T(n/2) + \log n
	  \]

	  \[
		T(n) = 2T(n/2) + n\log n
	  \]

	\pause
	\begin{alertblock}{Reference}
	  \[
		f(n) = \Theta(n^{\log_b^a} \lg^{k}n) \Rightarrow \Theta(n^{\log_b^a} \lg^{k+1}n)
	  \]
	\end{alertblock}
  \end{columns}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Solving recurrences (Problem 1.2.13, 1.2.16)}
  \[
	T(n) = T(n/2) + T(n/4) + T(n/8) + n
  \]

  \pause
  \vspace{0.30cm}
  \centerline{By recursion-tree.}

  \pause
  \vspace{0.50cm}

  \centerline{Exercise: Prove it by Mathematical Induction.}

  \pause
  \begin{alertblock}{Reference}
	``On the Solution of Linear Recurrence Equations'' by Akra \& Bazzi, 1996.

	\[
	  T(n) = \sum_{i=1}^{k} a_i T(n/b_i) + f(n)
	\]
  \end{alertblock}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Gaps (Problem 1.2.16)}
  \[
	T(n) = 2T(n/2) + \frac{n}{\log n} \pause = \Theta(n \log \log n)
  \]

  \pause

  The regularity condition in Case 3:
  \[
	bf(n/c) \le cf(n), \text{ for some } c < 1 \text{ and sufficiently large } n
  \]

  \[
	T(n) = T(n/2) + n(2 - \cos n)
  \]

  \[
	n^{E} = n^0 \quad f(n) = n(2 - \cos n) = \Omega(n^{0 + \epsilon})
  \]

  \pause
  \[
	n = 2\pi k (k \text{ odd}) \Rightarrow c \ge \frac{3}{2}
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{(Problem 1.2.15)}
  \begin{align*}
	\text{T}(n) &= \sqrt{n}\ \text{T}(\sqrt{n})+n \\
		&= n^{\frac{1}{2}}\ \text{T}\left(n^{\frac{1}{2}} \right )+n \\
		&= n^{\frac{1}{2}}\left( n^{\frac{1}{2^2}}\ \text{T}\left(n^{\frac{1}{2^2}} \right )+n^{\frac{1}{2}} \right )+n\\
		&= n^{\frac{1}{2}+\frac{1}{2^2}}\ \text{T}\left(n^{\frac{1}{2^2}}\right ) +2n\\
		&= n^{\frac{1}{2}+\frac{1}{2^2}}\left(n^{\frac{1}{2^3}}\ \text{T}\left(n^{\frac{1}{2^3}}\right ) +n^{\frac{1}{2^2}} \right )+2n\\
		&= n^{\frac{1}{2}+\frac{1}{2^2}+\frac{1}{2^3}}\ \text{T}\left(n^{\frac{1}{2^3}}\right ) +3n\\
		&= \cdots \\
		&= n^{\sum_{i=1}^{k}\frac{1}{2^i}}\ \text{T}\left(n^{\frac{1}{2^k}}\right ) +kn\\
  \end{align*}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{(Problem 1.2.15)}
  \[
	n^{\frac{1}{2^k}} = 2 \Rightarrow k = \log\log n
  \]

  \pause

  \begin{align*}
	\text{T}(n) &= n^{\sum_{i=1}^{k}\frac{1}{2^i}}\ \text{T}\left(n^{\frac{1}{2^k}}\right ) +kn\\
		&=n^{\sum_{i=1}^{\log\log n}\frac{1}{2^i}}\ \text{T}(2) + n \log\log n\\
  \end{align*}

  \pause
  \[
	\sum_{i=1}^{\log_2\log_2(n)}\frac{1}{2^i} < 1 \Rightarrow T(n) = \Theta(n \log \log n)
  \]

  \pause
  \vspace{0.30cm}
  \centerline{Exercise: Prove it by Mathematical Induction.}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{(Problem 1.2.15)}
  \[
	T(n) = \sqrt{n} T(\sqrt{n}) + n
  \]

  \[
	n = 2^k \quad \sqrt{n} = 2^{k/2} \quad k = \log n
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%