\section{Green's Theorem}\label{sec:GreensTheorem}

We now come to the first of three important theorems that extend the
Fundamental Theorem of Calculus to higher dimensions. (The Fundamental
Theorem of Line Integrals has already done this in one way, but in
that case we were still dealing with an essentially one-dimensional
integral.) They all share with the Fundamental Theorem the following
rather vague description: \emph{To compute a certain sort of integral over a
region, we may do a computation on the boundary of the region
that involves one fewer integrations.}

Note that this does indeed describe the Fundamental Theorem of
Calculus and the Fundamental Theorem of Line Integrals: to
compute a single integral over an interval, we do a computation on the
boundary (the endpoints) that involves one fewer integrations, namely,
no integrations at all.

\begin{theorem}{Green's Theorem}{greens theorem}
If the vector field $\vect{f}=\langle
f_1,f_2\rangle$ and the region $D$ are sufficiently nice, and if $C$ is
the boundary of $D$ ($C$ is a closed curve), then
$$\iint_D {\partial f_2\over\partial x}
-{\partial f_1\over\partial y} \,dA = \int_C f_1\,dx +f_@\,dy ,$$
provided the integration on the right is done counter-clockwise around
$C$.\index{Green's theorem}
\end{theorem}

The proof of Green's Theorem will follow a discussion and several examples.

To indicate that an integral $\ds\int_C$ is being done over a closed
curve in the counter-clockwise direction, we usually write
$\ds\oint_C$. We also use the notation $\partial D$ to mean the
boundary of $D$ \dfont{oriented} in the
counterclockwise direction. With this notation,
$\ds\oint_C=\int_{\partial D}$.

We already know one case, not particularly interesting, in which this
theorem is true: If $\vect{f}$ is conservative, we know that the integral
$\ds\oint_C \vect{f}\cdot d\vect{r}=0$, because any integral of a
conservative vector field around a closed curve is zero. We also know
in this case that $\partial f_1 /\partial y=\partial f_2 /\partial x$, so
the double integral  in the theorem is simply the integral of the zero
function, namely, 0. So in the case that $\vect{f}$ is conservative, the
theorem says simply that $0=0$.

\begin{example}{}{greenseg}
We illustrate the theorem by computing both sides of
$$\int_{\partial D} x^3\,dx + xy^2\,dy=\iint_D y-0\,dA,$$
where $D$ is the triangular region with corners $(0,0)$, $(1,0)$,
$(0,-1)$.
\end{example}
\begin{solution}
Starting with the double integral:
$$\iint_D y^2-0\,dA=\int_0^1\int_0^{-1+x} y^2\,dy\,dx=
\int_0^1
{(-1+x)^3\over3}\,dx=\left.{(-1+x)^4\over12}\right|_0^1=-{1\over12}.$$

There is no single formula to describe the boundary of $D$, so to
compute the left side directly we need to compute three separate
integrals corresponding to the three sides of the triangle, and each
of these integrals we break into two integrals, the ``$dx$'' part and
the ``$dy$'' part.
The three sides are described by $y=0$, $y=-1+x$, and $x=0$. The
integrals are then
\begin{align*}
\int_{\partial D}\> \> x^3\,dx + xy^2\,dy&=
\int_0^1 x^3\,dx+\int_0^0 0\,dy+\int_1^0 x^3\,dx+\int_0^{-1} (y+1)y^2\,dy+
\int_0^0 0\,dx+\int_0^{-1} 0\,dy	\\
&={1\over4}+0-{1\over4}+-{1\over12}+0+0=-{1\over12}.
\end{align*}

Alternately, we could describe the three sides in vector form as
$\langle t,0\rangle$ for $t$ from $0$ to $1$, $\langle t+1,t\rangle$ for $t$ from $0$ to $-1$, and $\langle 0,-1+t\rangle$ for $t$ from $0$ to $1$. Note that in each case, as $t$ ranges from lower to upper bound, we
follow the corresponding side in the correct direction. Now
\begin{align*}
\int_{\partial D} x^3\,dx + xy^2\,dy&=
\int_0^1 t^3 + t\cdot 0^2\,dt + \int_0^{-1} (t+1)^3 + (t+1)t^2\,dt
+\int_0^1 0 + 0(-1+t)\,dt	\\
&=\int_0^1 t^3\,dt + \int_0^{-1} (t+1)^3 + (t+1)t^2 \,dt
=-{1\over12}.
\end{align*}
% The three integrals on the right in the first line deserve a
% bit of explanation. The first arises from
% $$\int_C \vect{f}\cdot d\vect{r},$$
% where $C$ is given by $\vect{r}=\langle t,0\rangle$, that is, by
% $x=t$ and $y=0$. This implies that $dx=dt$ and $dy=0\cdot dt$.
% Now substituting into $x^4\,dx + xy\,dy$ gives
% $t^4\,dt+t\cdot0\cdot0\,dt$. In the second integral we have
% $\vect{r}=\langle 1-t,t\rangle$, so $x=1-t$, $y=t$,
% $dx=-dt$, and $dy=dt$. Substitution gives the integral shown. The
% third integral arises in the same way.
%
%Now computing the easy integrals gives
%$(1/5)+(-1/5+1/2-1/3)+(0)=1/6$ as before.
\end{solution}

In this case, none of the integrations are difficult, but the second
approach is somewhat tedious because of the necessity to set up three
different integrals. In different circumstances, either of the
integrals, the single or the double, might be easier to
compute. Sometimes it is worthwhile to turn a single integral into the
corresponding double integral, sometimes exactly the opposite approach
is best.

Here is a clever use of Green's Theorem: We know that areas can be
computed using double integrals, namely,
$$\iint_D 1\,dA$$
computes the area of region $D$. If we can find $f_1$ and $f_2$ so that
$\partial f_2 /\partial x-\partial f_1 /\partial y=1$, then the area is also
$$\int_{\partial D} f_1\,dx+f_2\,dy.$$
It is quite easy to do this: $f_1=0,f_2=x$ works, as do
$f_1=-y, f_2=0$ and $f_1=-y/2,f_2=x/2$.

\begin{example}{}{}
An ellipse centered at the origin, with its two principal axes
aligned with the $x$ and $y$ axes, is given by
$${x^2\over a^2}+{y^2\over b^2}=1.$$ Find the area of the
interior of the ellipse.
\end{example}
\begin{solution}
We find the area of the interior
of the ellipse
via Green's theorem. To do this we need a vector equation for the
boundary; one such equation is $\langle a\cos t,b\sin t\rangle$, as
$t$ ranges from 0 to $2\pi$. We
can easily verify this by substitution:
$${x^2\over a^2}+{y^2\over b^2}=
{a^2\cos^2 t\over a^2}+{b^2\sin^2t\over b^2}=
\cos^2t+\sin^2t=1.$$
Let's consider the three possibilities for $f_1$ and $f_2$ above:
Using 0 and $x$ gives
$$\oint_C 0\,dx+x\,dy=\int_0^{2\pi} a\cos(t)b\cos(t)\,dt=
\int_0^{2\pi} ab\cos^2(t)\,dt.$$
Using $-y$ and 0 gives
$$\oint_C -y\,dx+0\,dy=\int_0^{2\pi} -b\sin(t)(-a\sin(t))\,dt=
\int_0^{2\pi} ab\sin^2(t)\,dt.$$
Finally, using $-y/2$ and $x/2$ gives
\begin{align*}
\oint_C -{y\over2}\,dx+{x\over2}\,dy&=
\int_0^{2\pi} -{b\sin(t)\over2}(-a\sin(t))\,dt
+{a\cos(t)\over2}(b\cos(t))\,dt	\\
&=\int_0^{2\pi} {ab\sin^2t\over2}+{ab\cos^2t\over2}\,dt=
\int_0^{2\pi} {ab\over2}\,dt=\pi ab.
\end{align*}
The first two integrals are not particularly difficult, but the third
is very easy, though the choice of $f_1$ and $f_2$ seems more complicated.
\end{solution}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <1truecm,1truecm>
\setplotarea x from -3.1 to 3.1, y from -2.1 to 2.1
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 /
\put {$(0,b)$} [bl] <2pt,2pt> at 0 2
\put {$(a,0)$} [tl] <2pt,-2pt> at 3 0
\multiput {$\bullet$} at 0 2 3 0 /
\ellipticalarc axes ratio 3:2 360 degrees from 3 0 center at 0 0
\endpicture}}
\caption{A ``standard'' ellipse, ${x^2\over a^2}+{y^2\over b^2}=1$. \label{fig:standard ellipse}}
\end{figure}

Now we look at the proof of Green's Theorem.

\begin{proof}
We cannot here prove Green's Theorem in general, but we can do a
special case. We seek to prove that for a vector field $\vect{f} = \langle f_1, f_2 \rangle$
$$\oint_C f_1 \,dx + f_2\,dy = \iint_{D} {\partial f_2\over\partial x}
-{\partial f_1\over\partial y} \,dA.$$
It is sufficient to show that
$$\oint_C f_1 \,dx=\iint_{D}-{\partial f_1 \over\partial y} \,dA\qquad\hbox{and}
\qquad\oint_C f_2 \,dy=\iint_{D} {\partial f_2 \over\partial x}\,dA,$$
which we can do if we can compute the double integral in both possible
ways, that is, using $dA=dy\,dx$ and $dA=dx\,dy$.

For the first equation, we start with
$$\iint_{D}{\partial f_1\over\partial y}\,dA=
\int_a^b\int_{g_1(x)}^{g_2(x)} {\partial f_1\over \partial y}\,dy\,dx=
\int_a^b f_1(x,g_2(x))-f_1(x,g_1(x))\,dx.$$
Here we have simply used the ordinary Fundamental Theorem of Calculus,
since for the inner integral we are integrating a derivative with
respect to $y$: an antiderivative of $\partial f_1/\partial y$ with
respect to $y$ is simply $f_1(x,y)$, and then we substitute $g_1$ and
$g_2$ for $y$ and subtract.

Now we need to manipulate $\oint_C f_1\,dx$. The boundary of region $D$
consists of 4 parts, given by the equations $y=g_1(x)$, $x=b$,
$y=g_2(x)$, and $x=a$. On the portions $x=b$ and $x=a$, $dx=0\,dt$, so
the corresponding integrals are zero. For the other two portions, we
use the parametric forms $x=t$, $y=g_1(t)$, $a\le t\le b$, and
$x=t$, $y=g_2(t)$, letting $t$ range from $b$ to $a$, since we are
integrating counter-clockwise around the boundary.
The resulting integrals give us
\begin{align*}
\oint_C f_1\,dx = \int_a^b f_1(t,g_1(t))\,dt+\int_b^a f_1(t,g_2(t))\,dt
&=\int_a^b f_1(t,g_1(t))\,dt-\int_a^b f_1(t,g_2(t))\,dt	\\
&=\int_a^b f_1(t,g_1(t))-f_1(t,g_2(t))\,dt
\end{align*}
which is the result of the double integral times $-1$, as desired.

The equation involving $f_2$ is essentially the same, and left as an
exercise.
\end{proof}

We can now rewrite Green's Theorem using the concepts of divergence and curl; these rewritten
versions in turn are closer to some later theorems we will see.

Suppose we write a two dimensional vector field in the
form $\vect{f}=\langle f_1,f_2,0\rangle$, where $f_1$ and $f_2$ are functions
of $x$ and $y$. Then
$$\nabla\times \vect{f} =
\left|
\begin{matrix}
\vect{i}	&	\vect{j}	&	\vect{k}	\\
{\partial \over\partial x}	&	{\partial\over\partial y}	&	{\partial \over\partial z}	\\
f_1	&	f_2	&	0
\end{matrix}
\right|=
\langle 0,0,{\partial f_2\over\partial x}-{\partial f_1\over\partial y}\rangle,$$
and so $(\nabla\times \vect{f})\cdot\vect{k}=\langle 0,0,{\partial f_2\over\partial x}-{\partial f_1\over\partial y}\rangle\cdot
\langle 0,0,1\rangle = {\partial f_2\over\partial x}-{\partial f_1\over\partial y}$. So Green's Theorem says
\begin{equation}\label{eq:greens theorem second form}
\int_{\partial D} \vect{f}\cdot d\vect{r}=
\int_{\partial D} f_1\,dx +f_2dy = \iint_{D} {\partial f_2\over\partial x}   - {\partial f_1\over\partial y}   \,dA
=\iint_{D}(\nabla\times \vect{f})\cdot\vect{k}\,dA.
\end{equation}
Roughly speaking, the right-most integral adds up the curl (tendency
to swirl) at each point in the region; the left-most integral adds up
the tangential components of the vector field around the entire
boundary. Green's Theorem says these are equal, or roughly, that the
sum of the ``microscopic'' swirls over the region is the same as the
``macroscopic'' swirl around the boundary.

Next, suppose that the boundary $\partial D$ has a vector form
$\vect{r}(t)$, so that $\vect{r}'(t)$ is tangent to the boundary, and
$\vect{T}=\vect{r}'(t)/|\vect{r}'(t)|$ is the usual unit tangent vector.
Writing $\vect{r}=\langle x(t),y(t)\rangle$ we get
$$\vect{T}={\langle x',y'\rangle\over|\vect{r}'(t)|}$$
and then
$$\vect{N}={\langle y',-x'\rangle\over|\vect{r}'(t)|}$$
is a unit vector perpendicular to $\vect{T}$, that is, a unit normal to
the boundary.
Now
\begin{align*}
\int_{\partial D} \vect{f}\cdot\vect{N}\,ds&=
\int_{\partial D} \langle f_1, f_2\rangle\cdot{\langle
  y',-x'\rangle\over|\vect{r}'(t)|} |\vect{r}'(t)|dt=
\int_{\partial D} f_1 y'\,dt - f_2 x'\,dt	\\
&=\int_{\partial D} f_1 \,dy - f_2 \,dx
=\int_{\partial D} - f_2 \,dx+ f_1\,dy.
\end{align*}
So far, we've just rewritten the original integral using alternate
notation. The last integral looks just like the left side of Green's
Theorem (\ref{thm:greens theorem}) except that $f_1$ and $f_2$ have
traded places and $f_2$ has acquired a negative sign. Then applying
Green's Theorem we get
$$
\int_{\partial D} - f_2\,dx+f_1\,dy=\iint_{D} {\partial f_1\over\partial x}+{\partial f_2\over\partial y}\,dA=
\iint_{D} \nabla\cdot\vect{f}\,dA.$$
Summarizing the long string of equalities,
\begin{equation}\label{eq:greens theorem third form}
\int_{\partial D} \vect{f}\cdot\vect{N}\,ds=\iint_{D} \nabla\cdot\vect{f}\,dA.
\end{equation}
Roughly speaking, the first integral adds up the flow across the
boundary of the region, from inside to out, and the second sums the
divergence (tendency to spread) at each point in the interior. The
theorem roughly says that the sum of the ``microscopic'' spreads is
the same as the total spread across the boundary and out of the region.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:GreensTheorem}}

\begin{enumialphparenastyle}

\begin{ex}
Compute $\ds\int_{\partial D} 2y\,dx + 3x\,dy$,
where $D$ is described by $0\le x\le1$, $0\le y\le 1$.
\begin{sol}
	$1$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} xy\,dx + xy\,dy$,
where $D$ is described by $0\le x\le1$, $0\le y\le 1$.
\begin{sol}
	$0$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} e^{2x+3y}\,dx + e^{xy}\,dy$,
where $D$ is described by $-2\le x\le 2$, $-1\le y\le 1$.
\begin{sol}
	$1/(2e)-1/(2e^7)+e/2-e^7/2$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} y\cos x\,dx + y\sin x\,dy$,
where $D$ is described by $0\le x\le \pi/2$, $1\le y\le 2$.
\begin{sol}
	$1/2$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} x^2y\,dx + xy^2\,dy$,
where $D$ is described by $0\le x\le 1$, $0\le y\le x$.
\begin{sol}
	$-1/6$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} x\sqrt{y}\,dx + \sqrt{x+y}\,dy$,
where $D$ is described by $1\le x\le 2$, $2x\le y\le 4$.
\begin{sol}
	$(2\sqrt3-10\sqrt5+8\sqrt6)/3-2\sqrt2/5+1/5$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} (x/y)\,dx + (2+3x)\,dy$,
where $D$ is described by $1\le x\le 2$, $1\le y\le x^2$.
\begin{sol}
	$11/2-\ln(2)$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} \sin y\,dx + \sin x\,dy$,
where $D$ is described by $0\le x\le \pi/2$, $x\le y\le \pi/2$.
\begin{sol}
	$2-\pi/2$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} x\ln y\,dx$,
where $D$ is described by $1\le x\le 2$, $\ds e^x\le y\le e^{x^2}$.
\begin{sol}
	$-17/12$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} \sqrt{1+x^2}\,dy$,
where $D$ is described by $-1\le x\le 1$, $x^2\le y\le 1$.
\begin{sol}
	$0$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} x^2y\,dx - xy^2\,dy$,
where $D$ is described by $x^2+y^2\le 1$.
\begin{sol}
	$-\pi/2$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds\int_{\partial D} y^3\,dx + 2x^3\,dy$,
where $D$ is described by $x^2+y^2\le 4$.
\begin{sol}
	$12\pi$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\oint_C (y-\sin(x))\,dx + \cos(x) \, dy$,
where $C$ is the boundary of the triangle with vertices $(0,0)$,
$(1,0)$, and $(1,2)$ oriented counter-clockwise.
\begin{sol}
	$2\cos(1)-2\sin(1)-1$
\end{sol}
\end{ex}

\begin{ex}
Finish our proof of Green's Theorem by showing that
$\ds\oint_C f_2\,dy=\iint_{D} {\partial f_2\over\partial x}\,dA$.
\end{ex}

\end{enumialphparenastyle}


\documentclass[final]{siamart171218}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}

\setlength{\oddsidemargin}{0.65in}
\setlength{\evensidemargin}{0.65in}

\title{Differential count analysis with a topic model}

\author{Peter Carbonetto\thanks{Dept. of Human Genetics and the Research Computing Center, University of Chicago, Chicago, IL}}

\begin{document}

\maketitle

\section{Motivation, and overview of methods}

The aim of this document is to derive, from first principles, a method
for analysis of differential gene expression using a topic model (also
known as ``grade of membership model'' \cite{dey-2017}). This method
may have other uses --- say, to identify ``key words'' in a topic
modeling analysis of text documents --- but since our main motivation
is analysis of gene expression data, we describe the methods with that
application in mind.

Topic modeling brings a new twist to analysis of differential gene
expression: In conventional differential expression analysis, each
sample is assigned to a single condition; in topic modeling, the
assignments to each topic are {\em proportional.} This needs to be
accounted for in developing the new methods for differential
expression analysis.

For motivation, we begin with the ``log-fold change'' statistic
commonly used in microarray and RNA sequencing experiments to quantify
expression differences between two conditions (e.g.,
\cite{cui-churchill-2003, quackenbush-2002}). The log-fold change for
gene $j$ and condition $k$ is a ratio of two conditional expectations,
\begin{equation}
\beta_{jk} \equiv
\log_2 \frac{E[\,x_j \,|\, \mathrm{condition} = k\,]}
            {E[\,x_j \,|\, \mathrm{condition} \neq k\,]},
\label{eq:lfc}
\end{equation}
where $x_j$ is the measured expression level of gene
$j$.\footnote{Defining $x_{jk}$ as the total gene expression for gene
  $j$ among all samples (expression profiles) in condition $k$, $x_j$
  as the total gene expression for gene $j$ in all samples, $n_k$ as
  the number of samples in condition $k$, and $n$ as the total sample
  size, the log-fold change can be computed as $\beta_{jk} = \log_2
  \Big\{ \frac{x_{jk}}{x_j - x_{jk}} \times \frac{n - n_k}{n_k}
  \Big\}$.} The way in which the expression level $x_j$ is defined can
lead to different log-fold change statistics. For example, several
popular methods for analyzing differential expression compare changes
in {\em relative} expression (say, relative to total expression in a
cell) \cite{bullard-2010, voom, DESeq2, edgeR}. As we will see, a
topic modeling perspective provides a natural way to analyze
differential gene expression when comparing either relative or
absolute expression levels.

\section{A binomial model}

To develop the methods, we begin with a simple binomial model that
predicts expression of a single gene given the proportional
assignments to a topic:
\begin{equation}
x_i \sim \mathrm{Binom}(s_i, \pi_i).
\label{eq:binomial}
\end{equation}
Here, $x_i$ is the expression level of the target gene in sample $i$,
$s_i$ is the total expression in sample $i$, and $\mathrm{Binom}(n,
\theta)$ denotes the binomial distribution with $n$ trials and success
probability $\theta$. In this simple model, the binomial probabilities
are defined as
\begin{equation}
\pi_i = (1 - q_i) p_0 + q_i p_1,
\label{eq:binomial-prob}
\end{equation}
where $q_i \in [0,1]$ is the (known) proportion of sample $i$ that is
attributed to the topic, and $p_0, p_1 \in [0, 1]$ are two unknowns to
be estimated.\footnote{This is a special case of the {\em linear
    probability model} in econometrics called \cite{stock-watson}. The
  linear probability model would be typically written as $\pi_i = b_0
  + q_i b$, where $b_0 = p_0$ and $n = p_1 - p_0 \in [-1, +1]$. This
  is a regression model for the binomial probability $\pi_i$, in which
  $\pi_i$ increases linearly with topic proportion $q_i$.}

Statistical inference with this simple binomial model implements
analysis of differential gene expression. In particular, $\beta \equiv
\log_2(p_1/p_0)$ is the {\em log-fold change in relative expression.}
To show that this is so, consider the following statistical process
for generating the counts $x_1, \ldots, x_n$:
\vspace{1em}
\begin{itemize}

\item for $i = 1, \ldots, n$
\begin{enumerate}

  \item for $t = 1, \ldots, s_i$
  \begin{enumerate}

  \item Sample a topic, $z_{it} \sim \mathrm{Binom}(1,q_i)$.

  \item Sample a gene, $w_{it} \,|\, z_{it} \sim \left\{\begin{array}{ll}
  \mathrm{Binom}(1,p_1) & \mbox{if $z_{it} = 1$} \\
  \mathrm{Binom}(1,p_0) & \mbox{otherwise.}
  \end{array}\right.$

  \end{enumerate}

  \item Generate the final gene count,
  $x_i \leftarrow w_{i1} + \cdots + w_{is_i}$.

\end{enumerate}
\end{itemize}
\vspace{1em} In this statistical process, $q_i = p(z_{it} = 1)$ is the
topic probability, $p_1 = p(w_{it} = 1 \,|\, z_{it} = 1)$ is the
conditional probability that the gene is expressed given membership to
the topic, and $p_0 = p(w_{it} = 1 \,|\, z_{it} = 0)$ is the
probability that the gene is expressed when not belonging to the
topic. Therefore, we have
\begin{equation}
\beta \equiv \log_2 \frac{p_1}{p_0}
= \log_2 \frac{p(w_{it} = 1 \,|\, z_{it} = 1)}
            {p(w_{it} = 1 \,|\, z_{it} = 0)}.
\label{eq:lfc-binom}
\end{equation}
{\em This is the log-fold change statistic for relative expression
  given proportional topic assignments $q_1, \ldots, q_n$.} The
binomial model \eqref{eq:binomial} can in fact be derived from this
statistical process (proof not shown). Therefore, estimating $p_0,
p_1$ for the binomial model \eqref{eq:binomial} provides an estimate
of the log-fold change \eqref{eq:lfc-binom}.

Before continuing, we point out that the $s_i$'s, in practice, do not
need to be the total expression for each sample $i$ ({\em i.e.}, the
total counts); for example, some researchers have suggested setting
$s_i$ to some pre-defined quantile of the sample's nonzero count
distribution (e.g., \cite{bullard-2010}). This approach has been
motivated in settings where a small number of genes are much more
highly expressed than the others, and therefore these few genes have a
large effect relative expression levels. In short, the binomial model
is quite flexible, and can accommodate different relative differential
expression analyses defines the $s_i$'s. The only constraint on $s_i$
is that it cannot be smaller than $x_i$.

\section{A Poisson model}

The Poisson likelihood with rate $\lambda = n\theta$ closely
approximates the Binomial likelihood $\mathrm{Binom}(n, \theta)$ when
$\theta$ is small and $n$ is large. (The Poisson arises as the
limiting distribution of the binomial as $n \rightarrow \infty, \pi
\rightarrow 0, n\theta \rightarrow \lambda$.) This is usually the case in
gene expression studies; the total gene expression is large, whereas
the contribution of each gene is small, even for genes with the
highest levels of expression. This suggests a Poisson model of
expression $x_i$ given topic proportions $q_i$:
\begin{equation}
x_i \sim \mathrm{Poisson}(t_i \lambda_i),
\label{eq:poisson}
\end{equation}
in which the Poisson rates are defined as
\begin{equation}
\lambda_i = (1 - q_i) f_0 + q_i f_1,
\end{equation}
and the two unknowns to be estimated are $f_0, f_1 \geq 0$. Observe
that, unlike the binomial model, the unknowns for the Poisson model
are not constrained to be in $[0, 1]$, and so they do not necessarily
represent probabilities. However, by setting $t_i = s_i$, we have that
\begin{align*}
f_0 &\approx p_0 \\
f_1 &\approx p_1,
\end{align*}
when $p_0, p_1$ are close to zero and all $s_i$ are large. (See the
Appendix for an alternative motivation of the Poisson model.)

In the next section, we derive the mathematical expressions needed to
implement differential count analysis based on the Poisson model.

\section{Poisson model derivations}

To derive an algorithm for computing MLEs of $f_0, f_1$ in the Poisson
model \eqref{eq:poisson}, we begin with the log-likelihood,
\begin{equation}
\ell(f_0, f_1) \equiv \log p(x \,|\, f_0, f_1) =
\sum_{i=1}^n x_i \log (t_i \lambda_i) - t_i \lambda_i +
\mbox{const,}
\label{eq:loglik-poisson}
\end{equation}
in which the ``const'' captures all terms that do not depend on $f_0$
or $f_1$. A useful identity is that the partial derivative of the
log-likelihood with respect to $\lambda_i$ is
\begin{equation*}
\frac{\partial\ell}{\partial\lambda_i} = \frac{x_i}{\lambda_i} - t_i.
\end{equation*}
Making use of this result, the partial derivatives of the
log-likelihood with respect to the model parameters $f_0$ and $f_1$
are
\begin{align}
\frac{\partial\ell}{f_0} &=
\sum_{i=1}^n (x_i/\lambda_i - t_i) \times (1-q_i) \\
\frac{\partial\ell}{f_1} &=
\sum_{i=1}^n (x_i/\lambda_i - t_i) \times q_i.
\end{align}
We use a quasi-Newton method implemented in the R function {\tt optim}
to minimize the negative log-likelihood
\eqref{eq:loglik-poisson}. Note that in the special case in which all
the topic proportions are either 0 or 1, the MLEs have a simple
closed-form solution:
\begin{align}
f_0 &= \frac{\sum_{i=1}^n (1 - q_i) x_i}{\sum_{i=1}^n (1 - q_i) t_i} \\
f_1 &= \frac{\sum_{i=1}^n q_i x_i}{\sum_{i=1}^n q_i t_i}
\end{align}

\subsection{EM for Poisson model}

We also implement a simple EM algorithm for fitting the Poison model
\eqref{eq:poisson}. The key is to introduce latent variables $a_i \sim
\mathrm{Poisson}(t_i (1-q_i) f_0)$ and $b_i \sim \mathrm{Poisson}(t_i
q_i f_1)$, then work with the expected complete log-likelihood
$E[\,\log p(x, a, b \,|\, f_0, f_1)\,]$. The ``M-step'' updates work
out to
\begin{align}
f_0 &= \frac{\sum_{i=1}^n \phi_i}{\sum_{i=1}^n t_i(1-q_i)} \\
f_1 &= \frac{\sum_{i=1}^n \gamma_i}{\sum_{i=1}^n t_i q_i},
\end{align}
where we have introduced notation for the posterior expectations of
the latent variables, $\phi_i \equiv E[a_i]$ and $\gamma_i \equiv
E[b_i]$. It can be shown that the posterior distribution of $(a_i,
b_i)$ is multinomial with number of trials $x_i$ and event
probabilities proportional to $(1-q_i) f_0$ and $q_i f_1$. So the
posterior expectations computed in the ``E-step'' are
\begin{align}
\phi_i   &= x_i (1 - q_i) f_0 / \lambda_i \\
\gamma_i &= x_i q_i f_1 / \lambda_i.
\end{align}
This completes the description of the EM algorithm for the Poisson
model.

\subsection{glm identity parameterization}

Once we have obtained MLEs of $f_0$ and $f_1$, we would also like to
characterize uncertainty in these estimates---that is, compute the
standard error (s.e.). In particular, we are interested in the s.e. of
the log-fold change statistic, $\beta$. As an intermediate step, we
consider the ``glm identity'' parameterization $\lambda_i = b_0 + q_i
b$, where $b = f_1 - f_0$ and $b_0 = f_0$. This parameterization can
be implemented using {\tt glm} in R with {\tt family = poisson(link =
  "identity")}, and therefore can be used to verify our calculations.

Under the Laplace approximation to the likelihood at the MLE
$(\hat{b}_0, \hat{b})$, the covariance matrix is $-H^{-1}$, where $H$
is the $2 \times 2$ matrix of second-order partial derivatives,
\begin{equation*}
\frac{\partial^2\ell}{\partial b_0^2} =
-\sum_{i=1}^n \frac{x_i}{\lambda_i^2}, \qquad
\frac{\partial^2\ell}{\partial b^2} =
-\sum_{i=1}^n \frac{x_i q_i^2}{\lambda_i^2}, \qquad
\frac{\partial^2\ell}{\partial b_0 \partial b} =
-\sum_{i=1}^n \frac{x_i q_i}{\lambda_i^2}.
\end{equation*}
This result can be used to obtain the standard errors and $z$-scores
for $\hat{b}_0$ and $\hat{b}$.

\subsection{Log-fold change parameterization}

Here we derive the s.e. for the MLE of $\beta \equiv \log
(f_1/f_0)$. (For convenience, we use the natural logarithm here rather
than the base-2 logarithm; to obtain the base-2 log-fold change
statistic and its s.e., divide by $\log 2$.) With this new
parameterization, the Poisson rates are
\begin{equation}
\lambda_i = f_0 \times \{1 - q_i(1-e^{\beta})\}
\end{equation}
The second-order partial derivatives needed to compute the $2 \times
2$ Hessian are
\begin{align}
\frac{\partial\ell}{f_0} &= \frac{1}{f_0} \sum_{i=1}^n
x_i - t_i\lambda_i \\
\frac{\partial\ell}{\beta} &= f_1 \sum_{i=1}^n
(x_i/\lambda_i - t_i) \times q_i \\
\frac{\partial^2\ell}{f_0^2} &=
-\frac{1}{f_0^2} \sum_{i=1}^n x_i \\
\frac{\partial^2\ell}{\beta^2} &=
-f_1 \sum_{i=1}^n t_i q_i - x_i f_0 q_i (1-q_i)/\lambda_i^2 \\
\frac{\partial^2\ell}{\partial f_0 \partial\beta} &=
-\frac{f_1}{f_0} \sum_{i=1}^n t_i q_i.
\end{align}
The expression for $\frac{\partial^2\ell}{\partial f_0 \partial\beta}$
is the more complicated one, but fortuately it simplifies at the MLE,
$\beta = \hat{\beta}$ (at the MLE, the gradient of the log-likelihood
with respect to $\beta$ vanishes):
\begin{equation}
\frac{\partial^2\ell}{\beta^2} = -f_1^2 \sum_{i=1}^n x_i (q_i/\lambda_i)^2.
\end{equation}
Therefore, at the MLE $\beta = \hat{\beta}$, the standard error is
\begin{equation}
\mathrm{se}(\hat{\beta}) = \frac{1}{f_1} \times
\sqrt{\frac{\bar{a}}{\bar{a} \times \bar{c} - \bar{b}^2}},
\end{equation}
where I've defined
\begin{equation*}
\bar{a} = \sum_{i=1}^n x_i, \qquad
\bar{b} = \sum_{i=1}^n t_i q_i, \qquad
\bar{c} = \sum_{i=1}^n x_i (q_i/\lambda_i)^2.
\end{equation*}
From this, the $z$-score is recovered as $z =
\hat{\beta}/se(\hat{\beta})$.

\appendix

\section{More on Poisson model}

Consider the following process for generating the counts $x_1, \ldots,
x_n$:
\begin{itemize}

\item for $i = 1, \ldots, n$
\begin{enumerate}

\item $a_i \sim \mathrm{Poisson}(f_1)$
\hfill Sample the within-topic gene count. \hspace{2em}

\item $b_i \sim \mathrm{Poisson}(f_0)$
\hfill Sample the outside-topic gene count. \hspace{2em}

\item $a_i' \sim \mathrm{Binom}(a_i, q_i)$
\hfill Subsample the within-topic genes. \hspace{2em}

\item $b_i' \sim \mathrm{Binom}(b_i, 1-q_i)$
\hfill Subsample the outside-topic genes. \hspace{2em}

\item $x_i \leftarrow a_i' + b_i'$
\hfill Generate the final gene count. \hspace{2em}

\end{enumerate}
\end{itemize}
In this generative process, $f_1 = E[a_i]$ represents the
within-topic gene rate, and $f_0 = E[b_i]$ is the outside-topic gene
rate, and therefore
\begin{equation}
\beta^{\mathsf{abs}} \equiv
\log_2 \frac{f_1}{f_0} = \log_2 \frac{E[a_i]}{E[b_i]}
\label{eq:lfc-poisson}
\end{equation}
is the {\em log-fold change statistic in (absolute) expression given
  proportional topic assignments $q_1, \ldots, q_n$.} For intuition,
when all the topic proportions $q_i$ are 0 or 1, this statistical
process simplify to $x_i \sim \mathrm{Poisson}(f_1)$ if $q_i = 1$, and
$x_i \sim \mathrm{Poisson}(f_0)$ if $q_i = 0$, and so
\eqref{eq:lfc-poisson} would reduce to the ratio of the mean
expression levels inside and outside the topic. The Poisson model
\eqref{eq:poisson} can be derived from this statistical process, and
so estimating $f_0, f_1$ for the Poisson model \eqref{eq:poisson}
provides an estimate of the log-fold change \eqref{eq:lfc-poisson}.

\bibliographystyle{siamplain}
\bibliography{diffcount}

\end{document}


\section{Model of Muscle Contraction}\label{sec:model_muscle_contraction}

%Because of possibly large strains, a nonlinear hyperelastic formulation is used.
%For mathematical foundations in continuum mechanics, we refer to basic literature such as the books of Holzapfel \cite{holzapfel2000nonlinear} and Marsden and Hughes \cite{marsden1994mathematical}, as well as literature on the application of the finite element method in continuum mechanics \cite{zienkiewicz1977finite,SUSSMAN1987357,zienkiewicz2005finite}.

%The following section provides a more profound introduction of solid mechanics to complement the overview given in \cref{sec:model_muscle_contraction}. It also describes the finite element discretization of the solid mechanics model and the algorithms used to obtain a numeric solution.

Muscle contraction is described on the organ level by a solid mechanics model. The goal is to describe the deformation of the tissue caused by the internal forces that are generated by sarcomeres and as a response to outer constraints such as applied forces, the attachment to tendons and inertia effects.

Different modeling approaches exist to describe the mechanical muscle behavior. Dynamic \emph{finite elasticity} methods for large strains exist that use hyperelastic materials, both compressible and incompressible. Further, \emph{linear elasticity} descriptions with linearizations at various levels are used in appropriate applications where small strains can be assumed. The whole range from simplifying linearized models to accurate nonlinear approaches can be found in the literature, sometimes with varying conventions and symbols.
In this section, we introduce consistent notation and formulate the model equations for both approaches. The discretization and solution is discussed later in \cref{sec:discretization_mechanics}.

The derivation largely follows the book of Holzapfel \cite{holzapfel2000nonlinear} and the discretization follows the work of Zienkiewicz, Taylor et al. \cite{zienkiewicz1977finite,zienkiewicz2005finite}. Further details can be found also in the book of Marsden and Hughes \cite{marsden1994mathematical}.

%  The implementation of a solver for such generic descriptions exploiting parallel execution and integrating a multi-scale biomechanics model, being a contribution of this work, is an interdisciplinary endeavour. Therefore, we introduce consistent notation and summarize the required basics and the derivation up to the final algorithm  such that it may serve also readers that are not specialized in the field of continuum mechanics.

% -v-

\subsection{Geometric Description}\label{sec:geometric_description}

% introduce quantities
% F, C, E, E(u), variation δE
%  S, P, sigma

We begin with the geometric description of the material body and define the basic quantities that are subsequently used to describe the physics. We consider the 3D muscle domain $\Omega_0=\Omega_M \subset \R^3$ in reference configuration at time $t=0$ that deforms into a current configuration $\Omega_t$ at time $t$. The material points are given by $\bfX \in \Omega_0$. The corresponding points $\bfx \in \Omega_t$ in the current configuration are defined by the function $\bfx = \bfvarphi_t(\bfX)$.

In the following, capital letters refer to quantities in material or Lagrangian description, i.e., defined in the reference configuration and small letters refer to quantities in spatial or Eulerian description, i.e., defined in the current configuration.

The relation of point coordinates in the current configuration with respect to the reference configuration can also be described by the displacements field $\bfU$:
\begin{align*}
  \bfx(\bfX) = \bfX + \bfU(\bfX).
\end{align*}
The symbol $\bfu$ with $\bfu(\bfx(\bfX))=\bfU(\bfX)$ denotes the displacements formulated in current configuration.
The current velocity $\bfv$ is the time derivative of the displacements, $\bfv := \dot{\bfu}$.

The deformation gradient $\bfF$ is the second order tensor that is obtained by differentiating the function $\bfvarphi_t$. It is given using the unit vectors $\bfe_i$ and components $F_{aA}$:
\begin{align*}
  \bfF &= F_{aA}\,\bfe_a \otimes \bfe_A, \quad && F_{aA} = \p{x_a}{X_A}.
\end{align*}
Capital and small indices refer to reference and current configuration, respectively. The deformation gradient can also be expressed using the displacement field $\bfU$:
\begin{align}\label{eq:solid1}
  \bfF = \bfI + ∇\bfU.
\end{align}
%
Here and in the following, the gradient symbol $∇$ refers to differentiation with respect to material coordinates $\bfX$.
We assume Cartesian coordinates.

% tangent, normal, volume map
The determinant of the deformation gradient is $J:= \det \bfF>0$. It is positive for any physically valid transformation.
The deformation gradient is used to map geometric quantities from the reference to the current configuration:
\begin{subequations}\label{eq:geometry_maps}
  \begin{align}
    \bft &= \bfF\,\bfT, & \text{(tangent map)} \label{eq:tangent_map}\\[4mm]
    \bfa &= \cof(\bfF)\,\bfA, & \text{(normal map)} \label{eq:normal_map}\\[4mm]
    v &= J\,V. & \text{(volume map)}\label{eq:volume_map}
  \end{align}
\end{subequations}
%
As given in \cref{eq:tangent_map} and visualized in \cref{fig:geometric_quantities}, the tensor $\bfF$ maps material tangents $\bfT$ in $\Omega_0$ to the corresponding spatial line elements $\bft$ in $\Omega_t$.
Accordingly, the spatial stretch at a point $\bfx \in \Omega_t$ in a certain direction is given by $\lambda=\sqrt{\bflambda^\top\,\bflambda}$ with $\bflambda = \bfF\,\bfM$, where $\bfM$ is a material line element with unit length pointing in the respective Lagrangian direction.

In \cref{eq:normal_map}, the cofactor of $\bfF$ given by $\cof(\bfF) = J\,\bfF^{-\top}$ maps normals $\bfA$ and surface areas $|\bfA|$ from $\Omega_0$ to the corresponding values $\bfa$ and $|\bfa|$ in $\Omega_t$. Nanson's formula, $\d \bfa = \cof(\bfF)\,\d\bfA$, is used to transform surface integrals from Eulerian to Lagrangian description.
Note that tangents at a point $\bfX$ live in the tangent space $T_\bfX\Omega_0$ and normals live in the co-tangent space $T^\ast_\bfX\Omega_0$.

\Cref{eq:volume_map} describes the volume map from $\Omega_0$ to $\Omega_t$, which simply scales the reference volume $V$ by the determinant $J$ to obtain the volume $v$ in the current configuration.

% geometric quantities
\begin{figure}
  \centering%
  \def\svgwidth{0.7\textwidth}
  \input{images/theory/geometric_quantities.pdf_tex}%
  \caption{Vector spaces and variables used in the geometric description of the solid mechanics model. The left side shows the reference configuration with tangent and co-tangent space of point $\bfX$. The right side shows tangent and co-tangent space for the current domain and a point $\bfx$. The spatial stretch $\lambda$ is defined by mapping a material element $\bfM$ to the current configuration. The maps $\bfvarphi_t$, $\bfF$ and $\bfF^{-\top}$ map tangents $\bfT,\bft$ and normals $\bfA,\bfa$ between the configurations.}%
  \label{fig:geometric_quantities}%
\end{figure}

Furthermore, the deformation gradient $\bfF$ is used to define the right Cauchy Green tensor $\bfC = \bfF^\top\bfF$, which maps from tangent to co-tangent space in reference configuration, and subsequently the Green-Lagrange strain tensor:
%
\begin{align*}
  \bfE = \dfrac12(\bfC - \bfI).
\end{align*}
%
This strain measure can be interpreted as comparing the current Lagrangian metric $\bfC$, a measure for the symmetric part of the current deformation, with the reference metric which is the identity. Using \cref{eq:solid1}, the Green-Lagrange strain tensor can be formulated in terms of derivatives of the displacements:%
\begin{align}\label{eq:green_lagrange_u}
  \bfE &= \dfrac12\big((∇\bfU)^\top + ∇\bfU + ∇\bfU^\top ∇\bfU\big).
\end{align}

In case of small displacements, a simplification is to not distinguish between reference and current configuration.
The strain expression given in \cref{eq:green_lagrange_u} can be linearized by neglecting products of the derivatives and using the spatial displacements $\bfu$ instead of $\bfU$. As a result, the linearized strain tensor $\eps$ is given by:
\begin{align}\label{eq:linearized_helper3}
  \bfeps = \dfrac12\big((∇\bfu)^\top + ∇\bfu\big).
\end{align}
It can be used together with linear material models to derive a completely linear model.

\subsection{Stress Metrics}

Continuum mechanical models establish equations for the unknown displacement function $\bfu$ and its evolution in time via relations between stresses and strains. In the following, we introduce the required stress metrics.

% stress measures
The Cauchy stress tensor $\bfsigma$ results from Euler's cut principle: we consider the mechanical action on an arbitrary, virtual cut out of the body in current configuration. The contact forces on the cut surface at a point $\bfx$ are described by the traction force $\bft$.
The traction vector acts on the current configuration and is a function of the position $\bfx \in \Omega_t$ and the local orientation of the cut given by the normal vector $\bfn$. Cauchy's theorem states that this relation is linear and can be described by the second order Cauchy stress  tensor $\bfsigma$:
\begin{align}\label{eq:cauchy_theorem}
  \bft = \bfsigma \cdot \bfn.
\end{align}

Thus, the Cauchy stress describes the \say{true stress} of contact forces per deformed surface area. Both slots of the second order tensor are associated with the current configuration. More specifically, $\bfsigma$ is contravariant and maps from a normal $\bfn$ in co-tangent space $T^\ast_\bfx\Omega_t$ to the traction $\bft$ in tangent space $T_\bfx\Omega_t$.

While the physical description is natural in this Eulerian setting, the numerical treatment is more convenient in the Lagrangian setting, where we can integrate over a non-deforming domain.
Moreover, a two-point setting, where surface areas are measured in the undeformed configuration and traction forces are measured in the deformed configuration, is often useful in engineering. This is the natural setting, e.g., in tension tests. Therefore, other stress measures involving the reference configuration are defined.

% numerics -> physics
% pull-back, push-back
Using the mappings presented in \cref{eq:geometry_maps}, all quantities can be transformed between both configurations.
The physical derivation can be carried out equivalently in a Lagrangian or Eulerian setting and switching between them is possible at any point in the derivation. For this purpose, two operations are defined: the pull-back $\varphi^\ast(\bfa) = \bfF^\top\bfa\,\bfF$ and push-forward operations $\varphi_\ast(\bfA) = \bfF^{-\top}\bfA\,\bfF^{-1}$, which bring tensors from Eulerian to Lagrangian description and vice-versa.

% sigma, 1st PK, 2nd PK
The first Piola-Kirchhoff stress tensor $\bfP$ measures contact forces in the current configuration with regard to the area of the reference configuration and relates to the Cauchy stress as $\bfP = \bfsigma\,\cof(\bfF)$. The second Piola-Kirchhoff tensor $\bfS$ is a fully Lagrangian field given as the pull-back of the Cauchy stress scaled by $J$:%
\begin{align*}
  \bfS = \varphi^\ast(J\,\bfsigma) = J\,\bfF^{-1}\bfsigma\,\bfF^{-\top}.
\end{align*}
%It appears in the summary of the muscle contraction model in \cref{eq:contraction} in connection with the strain energy function $\Psi$.

% stress tensors
\begin{figure}
  \centering%
  \def\svgwidth{0.5\textwidth}
  \input{images/theory/stress_tensors.pdf_tex}%
  \caption{Stress tensors and geometric maps that can be used together in a solid mechanics formulation. The right Cauchy-Green tensor $\bfC$  and the second Piola-Kirchhoff stress $\bfS$ are dual Eulerian tensors and map between tangent space $T_\bfX\Omega_0$ and co-tangent space $T^\ast_\bfX\Omega_0$ in the reference domain. The deformation gradient $\bfF$ and the first Piola-Kirchhoff stress $\bfP$ are dual two-point tensors mapping from the reference to the current configuration. The Cauchy stress $\bfsigma$ is defined entirely in the Eulerian setting.
  %The Eulerian metric $\bfg$, which is the identity in cartesian coordinates and the Kirchhoff stress $J\,\bfsigma$ (where $\bfsigma$ is the Cauchy stress) are the dual objects in the Eulerian setting. All three pairs of dual tensors are linked together by transformations such as pull-back and push-forward.
  }%
  \label{fig:stress_tensors}%
\end{figure}

\Cref{fig:stress_tensors} summarizes the geometric maps by black arrows and the stress measures by red arrows.
%To relate strains and stresses in a material model, the corresponding tensors have to be dual objects. Here, three settings are possible: The Eulerian setting uses the metric $\bfg$ and the dual Kirchhoff stress $J\,\bfsigma$. The two-point setting uses the deformation gradient $\bfF$ and the dual first Piola-Kirchhoff stress $\bfP$. The Lagrangian setting uses the right Cauchy-Green tensor $\bfC$ and the dual second Piola-Kirchhoff stress $\bfS$.
%Different matching pairs of strain and stress tensors can be used to describe physical relations.
The Lagrangian setting defines the right Cauchy-Green tensor $\bfC$ and the second Piola-Kirchhoff stress tensor $\bfS$.
We use these quantities in the derivation of the discretized equations, because the Lagrangian formulation is natural for this task and allows integrating over the non-deforming domain $\Omega_0$.

The Cauchy stress $\bfsigma$ is completely defined in the Eulerian setting. It is used to formulate physical balance principles.

% ---
\subsection{Overview of the Physical Relations}

The previously introduced quantities are linked together by various relations, which are summarized in the diagram in \cref{fig:tonti_diagram}. The goal is to find the relationship between given forces (top left in \cref{fig:tonti_diagram}) and the resulting deformation of the body described by the displacements (top right in \cref{fig:tonti_diagram}).
Prescribed external traction forces $\bfT$ and external or inertial body forces $\bfB$ act on the body and result in stresses $\bfS$ satisfying the \emph{equilibrium} relation. A \emph{material law} connects stresses $\bfS$ and strains $\bfE$. The \emph{kinematics} of the body determine the relationship between displacements $\bfu$ and strains $\bfE$. Geometric Dirichlet boundary conditions prescribe displacements and Neumann boundary conditions such as traction forces contribute to the stress field.

Whereas the equilibrium relation is linear, the material and kinematic descriptions can both be chosen to be linear or nonlinear.
In cases of small strains, geometric and material linearity can be assumed. We derive two such formulations: a linear formulation where all relations are linear and a nonlinear formulation with nonlinear material and kinematic relations.

% Tonti diagram
\begin{figure}
  \centering%
  \def\svgwidth{\textwidth}
  \input{images/theory/tonti_diagram.pdf_tex}%
  \caption{The three relations between various quantities that compose the solid mechanics model: Equilibrium links traction and body forces $\bfT$ and $\bfB$ to the stresses $\bfS$. A material model connects them to strains $\bfE$. The kinematic relations yield the resulting displacement field $\bfu$. Note that all quantities in this diagram are given in Lagrangian formulation.}%
  \label{fig:tonti_diagram}%
\end{figure}
% linear, static and dynamic

\subsection{Assumptions and Model Equations}\label{sec:assumptions_and_model_equations}
% ----

The foundation of continuum mechanics usually builds on three balance principles: conservation of mass, of momentum and of angular momentum. In the following, these principles are presented in their Eulerian forms.

First, we assume \emph{conservation of mass} in terms of the densities $\rho_0(\bfX)$ and $\rho(\bfx)$ in reference and current configurations:
%
\begin{align*}
  \ds\int\limits_{V_0} \rho_0\,\d V = \int\limits_{V_t} \rho \,\d v.
\end{align*}
%
The equation holds for all corresponding subdomains $V_0\subset \Omega_0$ and $V_t \subset \Omega_t$. With the intermediate step of deducing $\d/\d t \int_{\Omega_t} \rho \,\d v=0$, we get the following differential equation:%
\begin{align}\label{eq:contraction_helper1}
  \dot{\rho}(\bfv,t) + \rho(\bfx,t)\,\div\big(\bfv(\bfx,t)\big) = 0.
\end{align}
%

As muscle tissue largely consists of water, it is typically assumed to be an incompressible domain. This is equivalent to a constant density, $\dot{\rho}=0$, and, thus, \cref{eq:contraction_helper1} reduces to
%
\begin{align}\label{eq:assumption_1_local}
  \div(\bfv(\bfx,t)) = 0.
\end{align}
%

The second assumption is the \emph{balance of momentum}, which is expressed as %
\begin{align}\label{eq:assumption_2_integral}
  \d{t} \ds\int\limits_{V_t} \rho\,\bfv\, \d v = \ds\int\limits_{V_t} \rho\,\bfb \,\d v + \ds\int\limits_{∂V_t} \bft \,\d a.
\end{align}
Here, $\bfb$ describes a body force and $\bft$ describes a traction force that acts on the surface of the domain $V_t$. Using the Cauchy theorem \cref{eq:cauchy_theorem}, it can be replaced by the Cauchy stress $\bfsigma$. The corresponding differential form is given by the following differential equation:%
\begin{align}\label{eq:assumption_2_local}
  \rho\,\dot{\bfv}(\bfx,t) = \rho\,\bfb(\bfx,t) + \div\bfsigma(\bfx,t).
\end{align}
It relates external forces to the internal stress field and describes the \emph{equilibrium} relation in \cref{fig:tonti_diagram} in Eulerian form.
For the discretization, a Lagrangian form is typically used.

For hyperelastic materials, which we consider in the muscle model, the equilibrium relation can also be formulated in terms of the \emph{Hellinger-Reissner energy functional} $\Pi_L(\bfu,p)$, which describes the potential energy of the system depending on the displacement and pressure functions $\bfu$ and $p$. Analog to the local form in \cref{eq:assumption_2_local}, it contains terms for the external loads and for the internal response of the body.
The functional is additively composed of external and internal potential energy:
\begin{align}\label{eq:hellinger_reissner}
  \Pi_L(\bfu,p) = \Pi_\text{ext}(\bfu) + \Pi_\text{int}(\bfu,p).
\end{align}
The external energy functional is formulated by
\begin{align}\label{eq:pi_ext}
  \Pi_\text{ext}(\bfu) = -\ds\int_{\Omega_0} \bfB\, \bfu\,\d V - \ds\int_{∂\Omega_0^t}\bar{\bfT}\,\bfu\,\d S,
\end{align}
with body force $\bfB$ in reference configuration and prescribed surface traction $\bar{\bfT}$ on the traction boundary $∂\Omega_0^t$. The body force term $\bfB$ also includes the inertial forces of mass density times acceleration, $\rho\,\dot{\bfv}$, in case of a dynamic model.
The internal energy functional $\Pi_\text{int}(\bfu,p)$ describes the strain energy of the system depending on the displacement field $\bfu$ and the hydrostatic pressure $p$. The term is defined in \cref{sec:section_with_pi_int}.

The \emph{principle of stationary potential energy} demands that the potential energy functional $\Pi_L$ is stationary.
Variational calculus and differentiation of \cref{eq:hellinger_reissner} lead to the local Eulerian description given in \cref{eq:assumption_2_integral,eq:assumption_2_local}.

The third assumption is the \emph{balance of angular momentum} and can be formulated using the 3D cross-product:%
\begin{align*}
  \d{t} \ds\int\limits_{V_t} \bfx \times (\rho\,\bfv)\, \d v = \ds\int\limits_{V_t} \bfx \times (\rho\,\bfb) \,\d v + \ds\int\limits_{∂V_t} \bfx \times \bft\,\d a.
\end{align*}
%
This can be shown to be equivalent to the symmetry of the Cauchy stress tensor, $\bfsigma = \bfsigma^\top$.

A further assumption in the multi-scale muscle framework is to only consider isothermal conditions.
An activated muscle performs work and energy is added to the system by metabolism. Further, the muscle is not thermodynamically isolated. The system is not closed regarding conversion and transfer of energy and, thus, the balance of energy cannot be modeled easily.

Regarding the required relations to obtain the deformation of the body from external loads given in \cref{fig:tonti_diagram}, the \emph{equilibrium} relation is given by \cref{eq:assumption_2_local} and the nonlinear \emph{kinematic} relation is given by \cref{eq:green_lagrange_u}.
The \emph{material} relation has yet to be defined. The mathematical description is closed by defining a constitutive relation between stresses and strains in the next sections.

\Cref{sec:material_linear_model} defines a linear material model that can be used together with linearized kinematics to formulate a fully linear model. \Cref{sec:material_nonlinear_model} presents the nonlinear material model to proceed with the fully nonlinear description.

\subsection{Linear Material Model}\label{sec:material_linear_model}

For a linear constitutive relation between strain and stress, the linearized strain tensor $\bfeps$, defined in \cref{eq:linearized_helper3} is used together with the Eulerian Cauchy stress $\bfsigma$. The generic linear material model is \emph{Hooke}'s law, given by
%
\begin{align}\label{eq:linearized_helper2}
  \bfsigma = \C:\bfeps
\end{align}
with the fourth order material tensor%
\begin{align*}
  \C_{abcd} = K\,δ_{ab}\,δ_{cd} + μ\,(δ_{ac}\,δ_{bd} + δ_{ad}\,δ_{bc} - \dfrac23 δ_{ab}\,δ_{cd}).
\end{align*}
The bulk modulus $K$ is a measure for the (in-)compressibility and the shear modulus $\mu$ specifies the elastic shear stiffness. $δ_{ab}$ is the Kronecker delta.
The material tensor $\C$ exhibits the following major and minor symmetries:%
\begin{subequations}\label{eq:symmetries}
\begin{align}
  \C_{abcd} &= \C_{cdab}, \quad &\text{(major symmetries)}\\[4mm]
  \C_{abcd} &= \C_{bacd} = \C_{abdc} = \C_{badc}, \quad & \text{(minor symmetries)}
\end{align}
\end{subequations}
effectively reducing the number of independent entries from 81 to 21 for 3D domains.

To incorporate force generation in the muscle, the stress can be additively composed of the passive stress $\bfsigma$ and an additional active stress term $\bfsigma_\text{active}$:%
\begin{align}\label{eq:active_stress_linear}
  \bfsigma^\text{total} = \bfsigma + \bfsigma^\text{active}.
\end{align}

\subsection{Nonlinear Material Modeling}\label{sec:material_modeling}
%
Next, we present the derivation of a nonlinear model that does not make any linearization assumptions of small strains as in the previous section.
We begin with the description of the material law, which links strains and stresses.

The scalar strain energy function $\Psi$ describes the elastic energy of the material depending on the deformation.
The definition of $\Psi$ suffices to describe the behavior of a hyperelastic material.
The strain energy function links the right Cauchy Green tensor $\bfC$ to the second Piola-Kirchhoff stress tensor $\bfS$ by the relation%
\begin{align}\label{eq:material_model_helper1}
  \bfS = 2\p{\Psi(\bfC)}{\bfC}.
\end{align}

The \emph{principle of material objectivity} requires that material properties are invariant under a change of observer. As a result, the \emph{representation theorem for isotropic materials} states that the stress tensor can be represented using three strain invariants $I_1, I_2$ and $I_3$. For a transversely isotropic material, two invariants $I_4$ and $I_5$ that depend on the anisotropy direction $\bfa_0$ (corresponding to a fiber direction) are added.
Consequently, we can formulate the strain energy function $\Psi=\Psi(I_1,I_2,I_3,I_4,I_5)$ in terms of these invariants. The principle strain invariants $I_1$ to $I_3$ of the right Cauchy-Green tensor $\bfC$ and the additional anisotropic invariants $I_4$ and $I_5$ are defined as:
\begin{align*}
  &I_1(\bfC) = \tr(\bfC),  &
  &I_2(\bfC) = \dfrac12\big(\tr(\bfC)^2 - \tr(\bfC^2)\big), &
  I_3(\bfC) = \det(\bfC) = J^2,\\[4mm]
  &I_4(\bfC,\bfa_0) = \bfa_0 \cdot \bfC \, \bfa_0, &
  &I_5(\bfC,\bfa_0) = \bfa_0 \cdot \bfC^2 \, \bfa_0. &
\end{align*}
The fiber stretch is related to the fourth invariant by $\lambda_f = \sqrt{I_4}$. Note that requiring incompressibility is equivalent to enforcing $J=1$, and, in this case, we get ${I_3(\bfC) = 1}$.

It is convenient to use a decoupled description, where the deformation gradient $\bfF$ and the right Cauchy-Green tensor $\bfC$ are multiplicatively decomposed into volume-changing (volumetric) and volume-preserving (isochoric) parts:%
\begin{align*}
  \bfF &= (J^{1/3}\bfI)\,\bar{\bfF},  & \bfC &= (J^{2/3}\bfI)\,\bar{\bfC}.
\end{align*}
%
Here, the volumetric parts are the identity tensors scaled by a power of the determinant $J$ of the deformation gradient. The isochoric or distortional parts $\bar{\bfF}$ and $\bar{\bfC}$ are given by%
\begin{align}\label{eq:reduced_fc}
  \bar{\bfF} &= J^{-1/3}\,\bfF,  & \bar\bfC &= J^{-2/3}\,\bfC.
\end{align}
The reduced invariants $\bar{I}_1$ to $\bar{I}_5$ of the reduced right Cauchy-Green tensor $\bar\bfC$ are defined accordingly.
Similarly, the strain energy function has a decoupled representation with volumetric part $\Psi_\text{vol}$ and isochoric part $\Psi_\text{iso}$:
\begin{align}\label{eq:psi_iso}
  \Psi = \Psi_\text{vol}(J) + \Psi_\text{iso}(\bar{\bfC}) = \Psi_\text{vol}(J) + \Psi_\text{iso}(\bar{I}_1,\bar{I}_2,\bar{I}_4,\bar{I}_5).
\end{align}

Using the decoupled form, any incompressible material can be modeled with the \emph{penalty method} as follows.
The material behavior is given by the isochoric strain energy $\Psi_\text{iso}(\bar{\bfC})$, e.g., by employing the Mooney-Rivlin model in \cref{eq:mooney_rivlin}. The volumetric part is defined as
\begin{align*}
  \Psi_\text{vol}(J) &= \kappa\,G(J) \qquad \text{with } G(J) = \dfrac12 (J-1)^2,
\end{align*}
with the incompressibility parameter $\kappa$ and the penalty function $G(J)$. This function is strictly convex and approaches zero as $J$ approaches 1. For large values of $\kappa$, the behavior is nearly incompressible. A disadvantage of this method is, that the resulting system becomes singular for $J \to 1$.

A better approach in this regard is to use a mixed formulation, where incompressibility is enforced exactly using a Lagrange multiplier. This approach is also implemented in OpenDiHu and is the preferred method for incompressible materials.

In OpenDiHu, the strain energy function of a new material can be given using the following four terms:
%
\begin{align}\label{eq:definition_psi}
  \Psi = \Psi_\text{vol}(J) + \Psi_\text{iso}(\bar{I}_1,\bar{I}_2,\bar{I}_4,\bar{I}_5) + \Psi_1(I_1,I_2,I_3) + \Psi_2(\bfC,\bfa_0).
\end{align}
The decoupled form is available with $\Psi_\text{vol}$ and $\Psi_\text{iso}$, the coupled form for isotropic materials can be used via $\Psi_1$. The term $\Psi_2$ gives the most flexibility, as the constitutive model can be directly formulated using the right Cauchy-Green tensor $\bfC$ and the fiber direction $\bfa_0$. The unused terms among $\Psi_\text{vol},\Psi_\text{iso},\Psi_1$ and $\Psi_2$ can be defined as constant zero. The incompressibility constraint using Lagrange multipliers can be switched on or off such that both incompressible and compressible materials can be computed.
%
% ---

\subsection{The Nonlinear Material Model for Muscle Contraction}\label{sec:material_nonlinear_model}

In the muscle contraction model of \cite{Heidlauf2013}, the strain energy function is additively composed of two passive terms, one isotropic, one anisotropic, and one additional active term:
\begin{align}\label{eq:transiso_mooney_rivlin}
  \Psi(\bfC) = \Psi_\text{isotropic}(I_1,I_2) + \Psi_\text{anisotropic}(\lambda_f) + \Psi_\text{active}(\gamma).
\end{align}
The isotropic term $\Psi_\text{isotropic}$ is formulated in terms of the strain invariants $I_1=\tr(\bfC)$ and $I_2=\big(\tr(\bfC)^2 - \tr(\bfC^2)\big)/2$. The anisotropic term $\Psi_\text{anisotropic}$ depends on the fiber stretch $\lambda_f$. The active term $\Psi_\text{active}$ yields the active stress that results from muscular activation, which is described by the activation parameter $\gamma$.
%Note the missing dependency on the third invariant $I_3$, which is constant because of the enforced incompressibility.

The passive behavior of muscle tissue is modeled by a transversely isotropic Mooney-Rivlin material.
The isotropic part is given by the Mooney-Rivlin formulation:%
\begin{align}\label{eq:mooney_rivlin}
  \Psi_\text{isotropic}(I_1,I_2) = c_1\,(I_1 - 3) + c_2\,(I_2-3).
\end{align}
The values of the two material parameters $c_1$ and $c_2$ can be determined by compression tests and are summarized in the work of \cite{Heidlauf2013}.

The anisotropic behavior depends only on the fiber stretch $\lambda_f$. The formulation in \cite{Heidlauf2013} uses two material parameters $b$ and $d$ and the following function:
\begin{align*}
  \Psi_\text{anisotropic}(\lambda_f) = \dfrac{b}{d}(\lambda_f^d - 1) - b\,\log(\lambda_f).
\end{align*}
%

The active contribution is directly formulated in terms of the second Piola-Kirchhoff stress $\bfS$. The relation between the active stress $\bfS_\text{active}$ and the active contribution $\Psi_\text{active}$ of the strain energy function as well as the definition of $\bfS_\text{active}$ is given as follows:
\begin{align}\label{eq:active_stress_term}
  \bfS_\text{active} = \dfrac{1}{\lambda_f}\p{\Psi_\text{active}}{\lambda_f} \bfA \otimes \bfA = \dfrac{1}{\lambda_f} \cdot S_\text{max,active}\cdot f_\ell(\lambda_f)\cdot\bar{\gamma}\, \bfA \otimes \bfA.
\end{align}
%
Here, the resulting active stress tensor $\bfS_\text{active}$ is the second order tensor oriented according to the material fiber direction $\bfA: \Omega_0 \to \R^3$ and given by the dyadic product $\bfA \otimes \bfA = A_{i}\,A_{j}\,\bfe_i \otimes \bfe_j$, scaled by the maximum active stress parameter $S_\text{max,active}$, a function $f_\ell$ that models the force-length relation, and the 3D homogenized value $\bar{\gamma}$ of the activation parameter $\gamma \in [0,1]$ following from the half-sarcomere model.

In the deforming body fat layer, the active stress contribution is disregarded. For simulating tendons, different material models can be used such as the model proposed by Carniel et al. \cite{Carniel2017}, which describes microstructural interactions between collagen fibers and their matrix in addition to the elastic response of the fibers themselves. To alter the material model, the definition of $\Psi$ can simply be changed while all other equations of the solid mechanics model remain intact.
%Similarly, other material models can be defined using the framework of the strain energy function.

\subsection{Summary of the Solid Mechanics Model Equations}
In summary, the model of solid mechanics for muscle contraction is solved for the unknown displacements $\bfu$ and additionally the velocities $\bfv$ if a dynamic formulation is considered.

The model equations follow from the following balance principles:
%
\begin{subequations}\label{eq:contraction}
  \begin{align}
    \div(\bfv) &= 0, \qquad &&\text{(incompressibility)} \label{eq:contraction_1}\\[4mm]
    \rho\,\dot{\bfv} &= \rho\,\bfb + \div\bfsigma, && \text{(balance of linear momentum)}\label{eq:contraction_2}\\[4mm]
    \bfsigma &= \bfsigma^\top, && \text{(balance of angular momentum)}\label{eq:contraction_3}
  \end{align}
\end{subequations}
with the constant density $\rho$, external body forces $\bfb$ and the Cauchy stress tensor $\bfsigma$.

Additionally, geometric relations between displacements $\bfu$ and strains $\bfE$ or $\bfeps$ are assumed, either fully nonlinear in \cref{eq:green_lagrange_u} or with corresponding linearization assumptions in \cref{eq:linearized_helper3}.
Furthermore, a material model is given that relates strains and stresses. A linear model is described in \cref{sec:material_linear_model}. The framework for nonlinear hyperelastic models uses a strain energy function $\Psi$ as described in \cref{sec:material_modeling}. A particular nonlinear material model for muscle contraction from the literature is described in \cref{sec:material_nonlinear_model}.

The description of the multi-scale model \cite{Roehrle2012,Heidlauf2013} assumes quasi-static conditions, which means that the velocities are set to zero, $\bfv=\bfzero$, and inertial terms are neglected. As a consequence, the incompressibility constraint in \cref{eq:contraction_1} has to be formulated differently and the balance of momentum in \cref{eq:contraction_2} reduces to $\rho\,\bfb + \div \bfsigma = 0$.
Our implementation extends the model to the fully dynamic formulation given in \cref{eq:contraction_1,eq:contraction_2,eq:contraction_3}.

Initial conditions for the displacements $\bfu$ and velocities $\bfv$ define the initial pose of the muscle tissue:
%
\begin{align*}
  \bfu(\bfx,0) &= \bfu_0(\bfx), & \bfv(\bfx,0) &= \bfv_0(\bfx) \quad &&\text{for } \bfx \in \Omega_M.
\end{align*}
%
Dirichlet boundary conditions for $\bfu$ and $\bfv$ can fix certain parts of the muscle, e.g., at the attachment points of the tendons:
\begin{align*}
  \bfu(\bfx,t) &= \bar{\bfu}(t), & \bfv(\bfx,t) &= \bar{\bfv}(t) \quad &&\text{for } \bfx \in ∂\Omega_\text{Dirichlet}.
\end{align*}
%
Additionally, Neumann boundary conditions can be used to prescribe traction forces on the surface.

The derivation of the finite element formulation and the resulting numerical scheme to obtain the solution functions $\bfu$ and $\bfv$ are discussed in \cref{sec:discretization_mechanics}.

%\subsection{Sensory Organs and Motor Neurons}
%text of paper:
%Muscle spindles and Golgi tendon organs are located in the muscle and sense fiber stretch, contraction velocity, contraction acceleration and muscle forces.
%They are connected via layers of inter neurons to the motor neurons, which reside in the spinal cord.
%In turn, the motor units innervate and activate the muscle.

%We use CellML models to compute the dynamics of muscle spindles \cite{Mileusnic2006} and motor neurons \cite{CisiKohn2008}.


\section*{Exercises}

\begin{ex}
  Find $\begin{mymatrix}{c}
    1 \\
    2 \\
    3
  \end{mymatrix}
  + \begin{mymatrix}{c}
    1 \\
    5 \\
    1
  \end{mymatrix}
  + \begin{mymatrix}{c}
    -1 \\
    2 \\
    -4
  \end{mymatrix}$.

  \begin{sol}
    $\begin{mymatrix}{r}
      1 \\
      9 \\
      0
    \end{mymatrix}$.
  \end{sol}
\end{ex}

\begin{ex}
  Use the properties of vector addition from
  Proposition~\ref{prop:properties-vector-addition} to show the following
  equalities. Justify every step.
  \begin{enumerate}
  \item $(\vect{u}+\vect{v})+\vect{w} = (\vect{v}+\vect{w})+\vect{u}$.
  \item $(\vect{u}+\vect{0})+(\vect{v}+(-\vect{u})) = \vect{v}$.
  \end{enumerate}

  \begin{sol}
    \begin{enumerate}
    \item
      $(\vect{u}+\vect{v})+\vect{w} = (\vect{v}+\vect{u})+\vect{w} =
      \vect{v}+(\vect{u}+\vect{w}) = \vect{v}+(\vect{w}+\vect{u}) =
      (\vect{v}+\vect{w})+\vect{u}$. Here we have used the commutative
      law, the associative law, the commutative law, and the
      associative law, in that order.
    \item
      $(\vect{u}+\vect{0})+(\vect{v}+(-\vect{u})) =
      \vect{u}+(\vect{v}+(-\vect{u})) = \vect{u}+((-\vect{u})+\vect{v})
      = (\vect{u}+(-\vect{u}))+\vect{v} = \vect{0}+\vect{v} =
      \vect{v}+\vect{0} = \vect{v}$. Here we have used the additive
      unit, the commutative law, the associative law, the additive
      inverse, the commutative law, and the additive unit, in that
      order.
    \end{enumerate}
  \end{sol}
\end{ex}


\chapter{Model Selection}
Suppose we observe a realization of a random variable $Y$, with
distribution defined by a parameter $\bb$
\begin{equation}
\label{gdef}
\prod_{\bx_i \in N_0} f(y_i; \bx_i,\bb) \equiv
f_\bY(\by ;\bX,\bb)
\end{equation}
where $\by$ is the observed response associated with the
covariates $\bX$ and
$\bb \in \mathbb{R}^P$ is a $P \times 1$ parameter vector.

We are interested in estimating $\bb$. Suppose that before
doing so, we need to choose from amongst $P$
competing models, generated by simply restricting the
general parameter space $R^P$ in which $\bb$ lies.

In terms of the
parameters, we represent {\sl the full model} with $P$ parameters as:
\[
\mbox{Model(P): } f_\bY(\by ;\bx,\bb_P),
  \bb_P = (\beta_1,\dots,\beta_p,\beta_{p+1},\dots,\beta_P)'.
\]
We denote the ``true value'' of the parameter vector $\bb$ with
$\bb^*$.

Akaike (1977)
formulates the problem of statistical model identification as one of
selecting a model $f_\bY(\by;\bx,\bb_p)$
based on the
observations from that distribution, where the particular restricted
model is defined by the constraint $\beta_{p+1} = \beta_{p+2} = \dots
= \beta_{P} = 0$, so that
\begin{equation}
\label{modelpdef}
\mbox{Model(p): } f_\bY(\by;\bx,\bb_p),
\bb_p =  (\beta_1,\dots,\beta_p,0,\dots,0)'
\end{equation}
We will refer to $p$ as the {\sl number of parameters} and to
$\Omega_p$ as the sub-space of $\mathbb{R}^P$ defined by restriction
(\ref{modelpdef}). For each $p=1,\dots,P$, we may assume model(p) to
estimate the non-zero
components of the vector $\bb^*$. We are interested in a
criterion that helps us chose amongst these $P$ competing estimates.

In this Chapter we consider 3 methods for model selection.

\input{section-09-01}
\input{section-09-02}
\input{section-09-03}
\input{references-09}


\documentclass[letterpaper]{article}

\usepackage{graphicx}
\usepackage{mathtools}
\usepackage{enumerate}
\usepackage[letterpaper,left=1.25in,right=1.25in,top=1in,bottom=1in]{geometry}
\usepackage{fancyhdr}
\usepackage[parfill]{parskip}
\usepackage{changepage}

\pagestyle{fancy}
\fancyfoot{}
% header/footer settings
\lhead{Kevin Nash (kjn33)}
\chead{EECS 391 -- P2}
\rhead{\today}
\cfoot{\thepage}

\begin{document}

\section{Linear Decision Boundaries}

\subsection{Plotting Iris Classes}
This project is written in Python, using Pandas dataframes to manipulate data
and Matplotlib to plot it.

The data is initialized like so,
\begin{verbatim}
# Read the file into a Pandas dataframe
iris_data = pd.read_csv(filename)
# Capitalize the column names and remove underscores
iris_data.columns = ["Sepal Length", "Sepal Width", "Petal Length",
                     "Petal Width", "Species"]
# Remove any species other than those specified by the assignment
iris_data = iris_data.loc[iris_data["Species"].isin(["versicolor", "virginica"])]
# Select the versicolor and virginica species individually
versi_data = iris_data.loc[iris_data["Species"] == "versicolor"]
virgi_data = iris_data.loc[iris_data["Species"] == "virginica"]
\end{verbatim}
The assignment asks us to consider only the versicolor and virginica species.
Let's take a look at the first few rows of each.
\begin{verbatim}
In [2]: versi_data.head(3)
    Sepal Length  Sepal Width  Petal Length  Petal Width     Species
50           7.0          3.2           4.7          1.4  versicolor
51           6.4          3.2           4.5          1.5  versicolor
52           6.9          3.1           4.9          1.5  versicolor

In [3]: virgi_data.head(3)
     Sepal Length  Sepal Width  Petal Length  Petal Width    Species
100           6.3          3.3           6.0          2.5  virginica
101           5.8          2.7           5.1          1.9  virginica
102           7.1          3.0           5.9          2.1  virginica
\end{verbatim}
We are interested in the petal dimensions. To plot them:
\begin{verbatim}
fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(x=dataset_1["Petal Length"], y=dataset_1["Petal Width"],
           color="b", marker="o", label="versicolor")
ax.scatter(x=dataset_2["Petal Length"], y=dataset_2["Petal Width"],
           color="r", marker="^", label="virginica")
plt.xlabel("Petal Length (cm)")
plt.ylabel("Petal Width (cm)")
plt.legend(loc="upper left")
plt.savefig("plot_1a.pdf", bbox_inches="tight")
\end{verbatim}
This gives us
\begin{center}
\includegraphics{plot_1a.pdf}
\end{center}
It is clear that at least two points of different classes overlap, which means
that our data is not going to be linearly separable.

\subsection{Choosing a Boundary by Hand}

I chose a boundary by physically drawing a line on the above plot using an
image editor. The line reaches from 2.4 on the y-axis to 6.5 on the x-axis.
Then we can get an equation in slope-intercept form from the coordinates
$(3, 2.4), (6.5, 1)$. $y=mx+b$ where $m=-0.4$ and $b=3.6$.

The code to generate this plot is identical to the above plot, except the line
\begin{verbatim}
plt.plot([3, 6.5], [2.4, 1], color="k", linestyle="-", linewidth=1)
\end{verbatim}
is now added before saving.
\begin{center}
\includegraphics{plot_1b.pdf}
\end{center}

\subsection{Defining a Threshold Classifier}

We can now define a threshold classifier using the above parameters to compare
against our data points. Simply put, if the point is above the line, it is
predicted to be virginica. Otherwise it is predicted to be versicolor.

The calculation is shown here, where $x$ and $y$ correspond to $petal\ length$
and $petal\ width$, respectively.
\begin{verbatim}
""" Returns the distance from point (x,y) to the line in slope-intercept form """
def dist_from_line(x, y, m, b):
    return y - (m * x + b)

""" Returns "virginica" if the point (length, width) is above the line """
def classify_linear(row, slope, intercept):
    if 0 < dist_from_line(row["Petal Length"], row["Petal Width"], slope, intercept):
        return "virginica"
    return "versicolor"
\end{verbatim}

To plot this data, we isolate the misclassified points into their own dataframes.
Misclassified points are defined as those for which $Species \neq Classification$.
\begin{verbatim}
# Classify data linearly for part 1c
iris_data["Classification"] = iris_data.apply(
    lambda row: classify_linear(row, -0.4, 3.6), axis=1
)
misclassified = pd.DataFrame(columns=iris_data.columns)

for i, row in iris_data.iterrows():
    if not row["Classification"] == row["Species"]:
        misclassified.loc[len(misclassified.index)] = row

print("Linear decision accuracy: {}%".format(100 - len(misclassified.index) /
      len(iris_data.index) * 100))

versi_class = misclassified.loc[misclassified["Species"] == "versicolor"]
virgi_class = misclassified.loc[misclassified["Species"] == "virginica"]
plot_linear_bound(versi_class, virgi_class, "plot_1c.pdf")
\end{verbatim}
This threshold classifier misclassifies six data points, as shown in the next
plot. Note that points \textit{above} the line are supposed to be virginica
(red triangles), and points \textit{below} the line are supposed to be
versicolor (blue circles), but the opposite is true of misclassified points.
\begin{center}
\includegraphics{plot_1c.pdf}
\end{center}

\subsection{Circle Decision Boundaries}

Instead of defining our decision boundary with a straight line, we can use
a circle instead. The technique is the same, although in this case the decision
is made considering whether a point is inside or outside of the circle, rather
than above or below a line. The functions are similar to those first shown in
section 1.3.
\begin{verbatim}
""" Returns the distance from point (x,y) to the edge of the circle """
def dist_from_circle(x, y, x_circ, y_circ, r):
    return ((x - x_circ)**2 + (y - y_circ)**2)**0.5 - r

""" Returns "virginica" if the point (length, width) is outside of the circle """
def classify_circular(row, x, y, radius):
    if 0 < dist_from_circle(row["Petal Length"], row["Petal Width"], x, y, radius):
        return "virginica"
    return "versicolor"
\end{verbatim}
Now we need to pick parameters for our circle ($x_{0}, y_{0}, r$). The
assignment instructs us to pick three sets. I started at the center of the plot
with an arbitrary radius of $\frac{1}{2}$. This, predictably, didn't give great
results, but with two adjustments the decision accuracy eventually met that of
the linear threshold ($94\%$). The parameters and results are shown below,
followed by the plot of each. Note that because the $x,y$ scales are not $1:1$,
the circle is \textit{mathematically} circular, \textit{not graphically} circular.
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline
$x$ & $y$ & $r$ & Accuracy\\
\hline\hline
$4.75$ & $1.7$ & $0.5$ & 58\%\\
\hline
$4.5$ & $1.2$ & $0.5$ & 79\%\\
\hline
$4.0$ & $1.2$ & $1$ & 94\%\\
\hline
\end{tabular}
\includegraphics{plot_1d_1.pdf}
\includegraphics{plot_1d_2.pdf}
\includegraphics{plot_1d_3.pdf}
\end{center}

The plotting code is easy to imagine. Instead of drawing a line, just draw
a circle according to
\begin{verbatim}
circle = plt.Circle((x, y), r, color="k", fill=False)
ax.add_artist(circle)
\end{verbatim}

\section{Objective Functions}

\subsection{Calculating Mean-Squared Error}

The mean squared error is given by
\begin{equation*}
E = \frac{1}{N}\sum_{n=1}^N\left(\boldsymbol{w}^T\boldsymbol{x}_n-c_n\right)^2
\end{equation*}
In our case the predicted and actual values are $\boldsymbol{w}^T\boldsymbol{x}_n$
and $c_n$, where class of $x$ in pattern vector $\boldsymbol{x}_n$ is given by
\[
    x \in
    \begin{cases}
    \text{class 1} & \text{if } y\geq 0\\
    \text{class 2} & \text{if } y < 0
    \end{cases}
\]
and we can number the classes $0$ or $1$ to give $c_n$ a value according to
\[
    c_n =
    \begin{cases}
    0 & \boldsymbol{x}_n \in \text{class 1}\\
    1 & \boldsymbol{x}_n \in \text{class 2}
    \end{cases}
\]
However by this definition, $(\boldsymbol{x}_n-c_n)\subset \{-1,0,1\}$ and
therefore $(\boldsymbol{x}_n-c_n)^2\subset \{0,1\}$. The mean of zeroes and
ones doesn't give us a very percise error value, and so we can redefine the
first term. Rather than assigning a zero or a one based on the predicted class,
we can plug distance into a modified version of the sigmoid function, shown
below.
\begin{equation*}
p = \frac{1}{1+10^{-d}}
\end{equation*}
where $d$ is the distance from the decision boundary and $p$ is the predictive
value. $p\in[0,1]$, which changes the prediction from a discrete set of one and
zero to a continuum from zero to one. By making this change, we qualify our
predictions with a degree of confidence and theoretically make our MSE value
more accurate.

From now on the MSE value will be superimposed over the graph. The current
boundary gives a value of $0.0645$.
\begin{center}
\includegraphics{plot_2a.pdf}
\end{center}
\begin{verbatim}
if show_mse:
    combined_data = pd.concat([dataset_1, dataset_2])
    mse = mean_squared_error(combined_data, slope, intercept)
    plt.text(0.85, 0.05, "MSE: %.4f" % mse, ha='center', va='center',
             fontsize=12, transform=ax.transAxes)
\end{verbatim}
The mean squared error calculation is calculated like so
\begin{verbatim}
def mean_squared_error(dataset, m, b, classes={"versicolor": 0, "virginica": 1}):
    sum_diffs = 0
    for i, row in dataset.iterrows():
        dist = dist_from_line(m=m, b=b, row=row)
        sum_diffs += (classes[row["Species"]] -
            classes[classify_prediction(dist)] * sigmoid(dist))**2
    return sum_diffs / len(dataset.index)

def sigmoid(x):
    return 1 / (1 + 10**(-x))
\end{verbatim}

\subsection{Examples of Large and Small Errors}

As was shown in the previous section, the MSE value for a line with $m=-0.4$, $b=3.6$ was
quite small ($0.0645$). We can generate a large MSE value ($0.6862$) by using a positive
slope. Here the boundary misclassifies the majority of the data points!
\begin{verbatim}
# Very high MSE
plot_linear_bound(versi_data, virgi_data, 1.5, -5.5, show_mse=True, filename="plot_2b_1.pdf")
\end{verbatim}
\begin{center}
\includegraphics{plot_2b_1.pdf}
\end{center}
To lower the MSE, we should move our parameters back to their ``approximately optimal''
values. Let's see what a zero slope about halfway through the data gives us.
\begin{verbatim}
# Lower MSE
plot_linear_bound(versi_data, virgi_data, 0, 1.65, show_mse=True, filename="plot_2b_2.pdf")
\end{verbatim}
\begin{center}
\includegraphics{plot_2b_2.pdf}
\end{center}
$0.0889$ is much closer to our original value of $0.0645$.

\subsection{Derivation of the Gradient}

Starting with the mean square equation from section 2.1,
\begin{equation*}
E = \frac{1}{N}\sum_{n=1}^N\left(\boldsymbol{w}^T\boldsymbol{x}_n-c_n\right)^2
\end{equation*}
Where $E$ is the mean-squared error, $N$ is the number of data points (in this
case 100), $\boldsymbol{w}^T$ is the vector of weights, $\boldsymbol{x}_n$ is
the vector of data point classifications, and $c_n$ is its actual class (in
this case species).

As was previously stated, I am improving the error function with a modified
sigmoid curve. The distance value supplied to that function is calculated with
\begin{equation*}
d = W_n - (w_1 \cdot L_n + w_2)
\end{equation*}
where $W_n$ is the petal width, $L_n$ is the petal length, $w_1$ is the first
weight corresponding to slope, $w_2$ is the second weight corresponding to
intercept.

The modified sigmoid function is then used to get a bias term ($w_0$).
\begin{equation*}
w_0 = S(d) = \frac{1}{1+10^{-d}}
\end{equation*}
The derivative of this sigmoid function is
\begin{equation*}
S'(d) = S(d)\cdot(1-S(d))\cdot\text{ln}|10|
\end{equation*}

If we combine these equations, the MSE equation is written
\begin{equation*}
E = \frac{1}{N}\sum_{n=1}^N\left(S(W_n - (w_1 \cdot L_n + w_2))\cdot\boldsymbol{x}_n-c_n\right)^2
\end{equation*}
The gradient for this MSE involves differentiating its equation, as is done below.
\begin{equation*}
\begin{split}
\frac{\partial E}{\partial\boldsymbol{w}} &= \frac{2}{N}(1-S(\boldsymbol{x}\boldsymbol{w}))(S(\boldsymbol{x}\boldsymbol{w})-\boldsymbol{c})\boldsymbol{x}\\
&= \frac{2}{N}(1-\boldsymbol{p})(\boldsymbol{p}-\boldsymbol{c})\boldsymbol{x}
\end{split}
\end{equation*}
At the end we replace the sigmoid function call with a vector of the predictive
values that it returns, notated $\boldsymbol{p}$.

This is the vector form. The scalar form will be shown in the next section.

\subsection{Gradients in Scalar and Vector Form}

We can expand the vector form of the gradient into a scalar form.
When expanded, it looks like:
\begin{equation*}
\text{Scalar:\quad} \frac{\partial E}{\partial w_i} = \frac{2}{N}\sum_{n=1}^N(x_{i,n}\cdot(1-S(x_n w)\cdot(S(x_n w)-c_n))
\end{equation*}
\begin{equation*}
\frac{2}{N}\sum_{n=1}^N(w_0x_{0,n}+\ldots+w_ix_{i,n}+\ldots+w_Mx_{M,n}-c_n)x_{i,n}
\end{equation*}
In this form the expression is summed over $N$ terms. The vector form is similar but
does not feature the summation, since it is implicit from the use of vectors
instead of scalar variables (e.g. $x_{i,n}$ and $\boldsymbol{x}$).
\begin{equation*}
\text{Vector:\quad} \frac{\partial E}{\partial\boldsymbol{w}} = \frac{2}{N}(1-S(\boldsymbol{x}\boldsymbol{w}))(S(\boldsymbol{x}\boldsymbol{w})-\boldsymbol{c})\boldsymbol{x}
\end{equation*}
Then to update the weight, we use an iteration function:
\begin{equation*}
w_i^{t+1}=w_1^t-\varepsilon\frac{\partial E}{\partial w_i}
\end{equation*}
Epsilon here is a small value that ensure we eventually converge on a minimum value.
At this point the difference between one MSE and the MSE of the next step is
negligible.

\subsection{Summing the Gradient for an Ensemble of Patterns}

We can now use a gradient to take steps that should ultimately minimize mean
squared error. To do this, we simply combine the tools outlined above.
Namely, we need the predictive function by which to compare the relative
confidence in each categorization, and the derivative, which tells us how to
improve this confidence (thereby lowering MSE).

\begin{verbatim}
def sum_gradient(dataset, m, b, classes={"versicolor": 0, "virginica": 1}):
    epsilon = 0.1 / len(dataset.index)
    gradient = 0
    for i, row in dataset.iterrows():
        pred = sigmoid(dist_from_line(m=m, b=b, row=row))
        error = pred - classes[row["Species"]]
        gradient += (2 / len(dataset.index)) * (1 - pred) * error
    if m < 0:
        new_m = m + epsilon * gradient
    else:
        new_m = m - epsilon * gradient
    new_b = b - epsilon * gradient
    return new_m, new_b
\end{verbatim}

The above function calculates the new slope and intercept for weights in the
next step in a gradient descent. Below are function calls to plotting routines
that produce the following plots. The plots show sequential steps (magenta)
from an initial decision boundary (black, dashed).

\begin{verbatim}
plot_gradient_descent(versi_data, virgi_data, -0.1, 5, snapshots=5, filename="plot_2e_1.pdf")
plot_gradient_descent(versi_data, virgi_data, -0.5, 3, snapshots=5, filename="plot_2e_2.pdf")
\end{verbatim}
\begin{center}
\includegraphics{plot_2e_1.pdf}
\includegraphics{plot_2e_2.pdf}
\end{center}

\section{Learning a Decision Boundary Through Optimization}

\subsection{Implementing Gradient Descent}

This section is basically the one above, since in the above plots I considered
it more helpful to show multiple steps at once. However since this section
covers the gradient descent implementation, I will point out that on the
previous graphs we notice initial and final MSE values, also reproduced
in the table below. The final value comes after 1000 steps, though not all
steps are always necessary.
\begin{center}
\begin{tabular}{|r|r|r|}
\hline
Initial MSE & Final MSE & Reduction\\
\hline\hline
0.4953 & 0.1175 & 76.3\%\\
\hline
0.2586 & 0.1093 & 57.7\%\\
\hline
\end{tabular}
\end{center}

We can also take a look at the descent function.
\begin{verbatim}
mse_0 = mean_squared_error(combined_data, slope, intercept)
m = slope
b = intercept
steps = 1000
for i in range(steps):
    m, b = sum_gradient(combined_data, m, b)
    if (i == 1) or (i % (steps / snapshots) == 0):
        # x_vals = np.array(ax.get_xlim())
        y_vals = b + m * x_vals
        plt.plot(x_vals, y_vals, color="m", linestyle='-', linewidth=1)
mse_f = mean_squared_error(combined_data, m, b)
\end{verbatim}
First the mean squared error is calculated for the inital weights. This is
$\text{MSE}_0$. Then we take $1000$ steps, and for every step a new gradient sum is
calculated, $m$ and $b$ are updated and plotted. After the $1000^{\text{th}}$
step, we calculated mean squared error once more. This is $\text{MSE}_f$.

The main point here is that there can be a significant reduction in MSE by
gradient descent.

\subsection{Showing the Learning Curve}

Now let us look at a learning curve for the first descent plot. This time it
is shown with a 100 steps visible, rather than just 5. You can see how the steps
``slow down'' as the boundary converges. Plus it might make a cool Moiré pattern
on your screen.
\begin{center}
\includegraphics{plot_3b_1.pdf}
\includegraphics{plot_3b_2.pdf}
\end{center}
The learning curve is shown above.
\begin{verbatim}
mse_vals = []
m = slope
b = intercept
steps = 10000
for i in range(steps):
    if i % 100 == 0:
        mse_vals.append(mean_squared_error(dataset, m, b))
    m, b = sum_gradient(dataset, m, b)

fig = plt.figure()
ax = fig.add_subplot(111)
# mse_vals = mse_vals[int(-len(mse_vals)/2):]
ax.scatter(x=list(range(1, len(mse_vals) + 1)), y=mse_vals, color="k", marker="|")
plt.xlabel("Steps")
plt.ylabel("MSE at Step")
plt.savefig(filename, bbox_inches="tight")
plt.close(fig)
\end{verbatim}

\subsection{Illustration with Random Weights}

Since this section uses a random element, the results will be different
every time the code is run. Therefore I can't comment on specifics here.
Enjoy the plots.
\subsubsection{Initial}
\begin{center}
\includegraphics{plot_3c_1.pdf}
\includegraphics{plot_3c_2.pdf}
\end{center}
\subsubsection{Middle}
\begin{center}
\includegraphics{plot_3c_3.pdf}
\includegraphics{plot_3c_4.pdf}
\end{center}
\subsubsection{Final}
\begin{center}
\includegraphics{plot_3c_5.pdf}
\includegraphics{plot_3c_6.pdf}
\end{center}

\subsection{Choosing the Gradient Step Size}

The gradient step size was selected similarly to an example discussed in
lecture. It is determined by the size of the data, namely
\begin{equation*}
\varepsilon = \frac{0.1}{N}
\end{equation*}
where $N$ is the size of the dataset, in this case 100.
This value was chosen because if the step size is too large, there is an
increased probability that the learning will diverge. On the other hand,
if the step size is too small, learning will take too long to converge.
By making the step size a function of the data size, we guarantee that the
step could adapt to a change in size of the data.

For this dataset and algorithm, I found that the step size was somewhat lenient.
I tested between 0.001 and 0.01 and both of these values worked properly.

\subsection{Choosing the Stopping Criterion}

I tested three different stopping criteria, keeping in mind that once the
learning converges, there is no point in continuing further.

First was to
detect a lack of change in the mean squared error (MSE), since it seems
reasonable to claim that once the MSE stops changing no more steps are needed.
However in practice this assumption is not always true. If the decision boundary
is ever completely outside of the clusters of points (\textit{all} points fall
on one side or another), then this method will falsely detect convergence, since
shifting the boundary will usually keep the MSE at around 0.50. Even if the
boundary is within the data, the data are granular enough that you can sometimes
shift the boundary without changing any classifications.

Second was to detect a lack of change in an ``upstream'' variable. This method
is more sensitive to stillness. I decided that if the boundary does not move
at all within some degree of freedom, then the learning has converged, and it
ended up working successfully for my code. The degree of freedom was adjusted
over testing. A very small number was initially used, then when the runtime was
long, it was sequentially reduced until it produced good results in an
acceptible time.
\begin{verbatim}
while 0.0001 < b_2 - b:
    (continue calculation)
\end{verbatim}
When this criterion is used, it stops slightly after 1300 steps.

Third we can set a limit on number of steps. This is arguably the best way
to do it if you have the empirical data to back it up. Since we are working
with a single dataset and not in a general application, this is easy to
calculate, however it is technically wasteful in some cases, as the
convergence will occur before the step limit is reached, and for this reason
I chose the second method instead.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Sets and Number Systems}

A \dfont{set} can be thought of as any collection of \ifont{distinct} objects, called {\bf{elements}} of the set.\\

In general, there are three ways to describe sets.

\begin{formulabox}[Ways to Describe Sets]

	\begin{enumerate}

		\item \textbf{Verbally:} Use a sentence to define a set.\index{set ! verbal description}

		\item \textbf{Roster Notation:}  Begin with a left brace `$\{$', list each element of the set \textit{only once} and then end with a right brace `$\}$'.\index{set ! roster method}

		\item \textbf{Set-Builder Notation:} A combination of verbal and roster notation using a ``dummy variable'' such as $x$.\index{set ! set-builder notation}\index{set-builder notation}
	\end{enumerate}
\end{formulabox}

\begin{example}{Roster Notation for Sets}{Sets}
	The collection $\{a,b,1,2\}$ is a set. It consists of the collection of
	four distinct objects, namely, $a$, $b$, $1$ and $2$. The order of the elements doesn't matter, so the same set could be described as $\{2,a,1,b\}$.
\end{example}

Typically, sets are represented using
\dfont{set-builder notation} and are surrounded by braces.
Recall that $(,)$ are called \dfont{parentheses} or \dfont{round brackets};
$[,]$ are called \dfont{square brackets}; and $\{,\}$ are called \dfont{braces} or \dfont{curly brackets}. \\

\begin{example}{Set-Builder Notation for Sets}{SetBuilder}
The expression $\displaystyle{ \{x \, \ssep  x \geq 0 \} }$ can be read as the set of all elements $x$ such that $x$ is greater than or equal to zero.  The vertical bar $\ssep$, which can also be written as a colon $:$, is a separator that can be read as "such that", "for which", or "with the property that".
\end{example}

Let $S$ be any set. We use the notation $\dfont{x\in S}$ to mean that $x$
is an element \ifont{inside} of the set $S$, and the notation $\dfont{x\not\in S}$ to mean that $x$ is \ifont{not} an element of the set $S$. \\

\begin{example}{Set Membership}{SetMembership}
If $S=\{a,b,c\}$, then $a\in S$ but $d\not\in S$.
\end{example}

%The \dfont{intersection} between two sets $S$ and $T$ is denoted
%by $S\cap T$ and is the collection of all elements that belong to
%\ifont{both} $S$ and $T$. The \dfont{union} between two sets $S$
%and $T$ is denoted by $S\cup T$ and is the collection of all elements
%that belong to \ifont{either} $S$ or $T$ (or both). \\
%
%\begin{example}{Union and Intersection}{UnionIntersection}
%Let $S=\{a,b,c\}$ and $T=\{b,d\}$.
%Then $S\cap T=\{b\}$ and $S\cup T=\{a,b,c,d\}$.
%Note that we do \ifont{not} write the element $b$ twice in $S\cup T$ even
%though $b$ is in both $S$ and $T$.
%\end{example}

Numbers can be classified into sets called \dfont{number systems}.

\begin{formulabox}[Sets of Numbers]

\begin{enumerate}

	\item The \textbf{Empty Set}:\index{set ! empty}\index{empty set} $\emptyset=\{ \}=\{x\,|\,\mbox{$x
		\neq x$}\}$.  This is the set with no elements.  Like the number `$0$,' it  plays a vital role in mathematics.

	\item The \textbf{Natural Numbers}:\index{natural number ! set of}\index{natural number ! definition of} $\mathbb N= \{ 1, 2, 3,  \ldots\}$ The periods of ellipsis here indicate that the natural numbers contain $1$, $2$, $3$, `and so forth'.

	\item The \textbf{Whole Numbers}:\index{whole number ! set of}\index{whole number ! definition of} $\mathbb W = \{ 0, 1, 2, \ldots \}$

	\item The \textbf{Integers}:\index{integer ! set of}\index{integer ! definition of} $\mathbb Z=\{ \ldots, -1, -2, -1, 0, 1, 2, 3, \ldots \}$

	\item The \textbf{Rational Numbers}:\index{rational number ! set of}\index{rational number ! definition of} $\mathbb Q=\left\{\frac{a}{b} \, | \, a \in \mathbb Z \, \mbox{and} \, b \in \mathbb Z \right\}$.  \underline{Ratio}nal numbers are the \underline{ratio}s of integers (provided the denominator is not zero!)  It turns out that another way to describe the rational numbers is: \[\mathbb Q=\{x\,|\,\mbox{$x$ possesses a repeating or terminating decimal representation.}\}\]

	\item The \textbf{Real Numbers}:\index{real number ! set of}\index{real number ! definition of} $\mathbb R = \{ x\,|\,\mbox{$x$ possesses a decimal representation.}\}$

	\item The \textbf{Irrational Numbers}:\index{irrational number ! set of}\index{irrational number ! definition of} $\mathbb P = \{x\,|\,\mbox{$x$ is a non-rational real number.}\}$  Said another way, an \underline{ir}rational number is a decimal which neither repeats nor terminates.

	\item The \textbf{Complex Numbers}:\index{complex number ! set of}\index{complex number ! definition of} $\mathbb C=\{a+bi\,|\,\mbox{$a$,$b \in \mathbb R$ and $i=\sqrt{-1}$}\}$  Despite their importance, the complex numbers play only a minor role in the text.

\end{enumerate}
\end{formulabox}

It is important to note that every natural number is a whole number, which, in turn, is an integer.   Each integer is a rational number (take $b =1$ in the above definition for $\mathbb Q$) and the rational numbers are all real numbers, since they possess decimal representations.   If we take $b=0$ in the above definition of $\mathbb C$, we see that every real number is a complex number.  In this sense, the sets $\mathbb N$, $\mathbb W$, $\mathbb Z$, $\mathbb Q$, $\mathbb R$, and $\mathbb C$ are nested.

%\begin{center}
%  \begin{tabular}{| c || c | c |}
%    \hline
%    $\mathbb{N}$ & the \dfont{natural} numbers
%    		& $\{1, 2, 3,\ldots\}$ \\ \hline
%    $\mathbb{Z}$ & the \dfont{integers}
%    		& $\{\dots,-3,-2, -1, 0, 1, 2, 3,\dots\}$ \\ \hline
%    $\mathbb{Q}$ & the \dfont{rational} numbers
%    		& Ratios of integers: $\left\{\frac{p}{q}\, \ssep\,p,q\in\mathbb{Z},q\not=0\right\}$ \\ \hline
%    $\mathbb{R}$ & the \dfont{real} numbers
%    		& Can be written using a finite or infinite \ifont{decimal expansion} \\ \hline
%    $\mathbb{C}$ & the \dfont{complex} numbers
%    		& These allow us to solve equations such as $x^2+1=0$ \\
%    \hline
%  \end{tabular}
%\end{center}

%In the table, the set of rational numbers is written using
%set-builder notation. The vertical bar $\ssep$, which can also be written as a colon $:$, is a separator that can be read as "such that", "for which", or "with the property that".
%The expression $\left\{\frac{p}{q}\, \ssep\,p,q\in\mathbb{Z},q\not=0\right\}$ can be read out loud as
%\ifont{the set of all fractions $p$ over $q$ such that $p$ and $q$ are both integers and $q$ is not equal to zero}.
%\\

\begin{example}{Rational Numbers}{RationalNumbers}
The numbers $-\frac{3}{4}$, $2.647$, $17$, $0.\bar{7}$ are all rational numbers.
You can think of rational numbers as \ifont{fractions} of one integer over another.
Note that 2.647 can be written as a fraction:
\[ 2.647=2.647\times\frac{1000}{1000}=\frac{2647}{1000} \]
Also note that in the expression $0.\bar{7}$, the bar over the $7$ indicates
that the $7$ is repeated forever:
\[ 0.77777777\ldots=\frac{7}{9}\]
\vspace{-0.5cm}
\end{example}

All rational numbers are real numbers with the property that
their decimal expansion either \ifont{terminates} after a finite number
of digits or begins to \ifont{repeat} the same finite sequence of digits over and over.
Real numbers that are not rational are called \dfont{irrational}. \\

\begin{example}{Irrational Numbers}{IrrationalNumbers}
Some of the most common irrational numbers include:
\begin{itemize}
	\item $\sqrt 2$.\quad
			Can you prove this is irrational? (The proof uses a technique called \ifont{contradiction}.)
	\item $\pi$.\quad
			Recall that $\pi$ (\dfont{pi}) is defined as the ratio of the circumference of a circle to its diameter and can be approximated by $3.14159265$.
	\item $e$.\quad
			Sometimes called Euler's number, $e$ can be approximated by $2.718281828459$.
			We will review the definition of $e$ in a later chapter.
\end{itemize}
\end{example}

Let $S$ and $T$ be two sets. If every element of $S$ is also an element of $T$, then
we say $S$ is a \dfont{subset} of $T$ and write $S\subseteq T$. Furthermore, if $S$ is
a subset of $T$ but not equal to $T$, we often write $S\subset T$.
The five sets of numbers in the table give an increasing sequence of sets:
\[ \mathbb{N} \subset \mathbb{Z} \subset \mathbb{Q} \subset \mathbb{R} \subset \mathbb{C}. \]

That is, all natural numbers are also integers, all integers are also rational numbers, all rational numbers are also real numbers, and all real numbers are also complex numbers.\\

For the most part, this textbook focuses on sets whose elements come from the real numbers $\mathbb R$.  Recall that we may visualize $\mathbb R$ as a line. Segments of this line are called \textbf{intervals}\index{interval ! definition of} of numbers. Below is a summary of the so-called \textbf{interval notation}\index{interval ! notation for} associated with given sets of numbers.  For intervals with finite endpoints, we list the left endpoint, then the right endpoint.  We use square brackets, `$[$' or `$]$', if the endpoint is included in the interval and use a filled-in or `closed' dot to indicate membership in the interval. Otherwise, we use parentheses, `$($' or `$)$' and an `open' circle to indicate that the endpoint is not part of the set.  If the interval does not have finite endpoints, we use the symbols $-\infty$ to indicate that the interval extends indefinitely to the left and $\infty$ to indicate that the interval extends indefinitely to the right.  Since infinity is a concept, and not a number, we always use parentheses when using these symbols in interval notation, and use an appropriate arrow to indicate that the interval extends indefinitely in one (or both) directions.\\