\chapter{Alternating Current}

We have discussed the voltage and current created by a battery.  A
battery pushes the electrons in one direction at a constant voltage;
this is known as \newterm{Direct Current} or DC. A battery typically
provides between 1.5 and 9 volts.

The electrical power that comes into your home on wires is
different. If you plotted the voltage over time, it would look like
this:

\begin{tikzpicture}[
tl/.style = {% tick labels fill=white, inner sep=1pt,
  font=\scriptsize, }, ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.425] (0,-1.85) grid
(6.6,1.85);

% y tick label
\foreach \y in {-170, -85, 85, 170}{\node[tl,left=1mm] at (0,{\y/100})
  {$\y$};}

% x tick label
\foreach \x in {0.004,0.008, 0.012, 0.016}{\node[tl,below=1mm] at
  ({392.67*\x},0) {$\x$};}

% axes
\draw[->,thick] (0,0) -- (6.5,0) node[right] {$seconds$};
\draw[->,thick] (0,-1.8) -- (0, 1.8) node[above] {$volts$};
% curve
\draw[<->,thick,draw=black,domain=0:6.5,samples=300,variable=\x] plot
(\x,{sin(deg(\x)) * 1.7});
\end{tikzpicture}

The $x$ axis here represents ground. When you insert a two-prong plug
into an outlet, one is ``hot'' and the other is ``ground''. Ground
represents 0 volts and should be the same voltage as the dirt under
the building.

The voltage is a sine wave at 60Hz. Your voltage fluctuates between
-170v and 170v. Think for a second what that means: The power company
pushes electrons at 170v and then pulls electrons at 170v.  It
alternates back and forth this way 60 times per second.

\section{Power of AC}

Let's say you turn on your toaster which has a resistance of 14.4
ohms. How much energy (in watts) does it change from electrical energy
to heat?  We know that $I = V/R$ and we know that watts of power is
$IV$. So given a voltage of $V$, the toaster is consuming $V^2/R$
watts.

However, $V$ is fluctuating. Let's plot the power the toaster is consuming:

\begin{tikzpicture}[
tl/.style = {% tick labels
    fill=white, inner sep=1pt, font=\scriptsize,
            },                        ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.5] (0,0) grid (6.6,2.1);

% y tick label
\foreach \y in {500, 1000, 1500, 2000}{\node[tl,left=1mm] at (0,{\y/1000}) {$\y$};}

% x tick label
\foreach \x in {0.004,0.008, 0.012, 0.016}{\node[tl,below=1mm] at ({392.67*\x},0) {$\x$};}

% axes
\draw[->,thick] (0,0) -- (6.5,0) node[right] {$seconds$};
\draw[->,thick] (0,0) -- (0, 2.1) node[above] {$watts$};
% curve
\draw[<->,thick,draw=black,domain=0:6.5,samples=300,variable=\x]
      plot (\x,{sin(deg(\x))^2 * 2.007});
\end{tikzpicture}

Another sine wave! Here is a lesser-known trig identity: $\left( \sin(x) \right)^2 = \frac{1}{2} - \frac{1}{2}\cos(2x)$

So this is actually a cosine wave flipped upside down, scaled down by
half the peak power and translated up so that it is never
negative. Note that it is also twice the frequency of the voltage sine
wave.

If we say the peak voltage is $V_p$ and the resistance of the toaster
is $R$, the power is given by

$$\frac{V_p^2}{2R} - \frac{V_p^2}{2R} \cos \left(\frac{2\pi t}{120} \right)$$

As a toaster user and as someone who pays a power bill, you are mostly
interested in the average power.  To get the average power, you take
the area under the power graph and divide it by the amount of time.

We can think of the area under the curve as two easy-to-integrate quantites summed:
\begin{itemize}
\item A constant function of $y = frac{V_p^2}{2R}$
\item A wave $y = - \frac{V_p^2}{2R} \cos \left(\frac{2\pi t}{120} \right)$
\end{itemize}

\begin{tikzpicture}[
tl/.style = {% tick labels
    fill=white, inner sep=1pt, font=\scriptsize,
            },                        ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.5] (0,-1.1) grid (6.6,1.1);

% y tick label
\foreach \y in {-1000, -500, 500, 1000}{\node[tl,left=1mm] at (0,{\y/1000}) {$\y$};}

% x tick label
\foreach \x in {0.004,0.008, 0.012, 0.016}{\node[tl,below=1mm] at ({392.67*\x},0) {$\x$};}

% axes
\draw[->,thick] (0,0) -- (6.5,0) node[right] {$seconds$};
\draw[<->,thick] (0,-1.1) -- (0, 1.1) node[above] {$watts$};
% curve
\draw[<->,thick,draw=black] (0,1.003) -- (6.5, 1.003) node[right] {Constant: $\frac{V_p^2}{2R}$};
\draw[<->,thick,draw=black,domain=0:6.5,samples=300,variable=\x]
      plot (\x,{-1 * cos(deg(\x) * 2) * 1.003}) node[right]{Wave:$- \frac{V_p^2}{2R} \cos \left(\frac{2\pi t}{120} \right)$};
\end{tikzpicture}

When we integrate that constant function we get $\frac{t V_p^2}{2R}$

When we integrate that wave for a complete cycle we get...zero! The
positive side of the wave is canceled out by the negative side.

So, the average power is $\frac{V_p^2}{2R}$ watts.

Someone at some point said ``I'm used to power being $V^2/R$. Can
we define a voltage measure for AC power such that this is always true?''

So we started using $V_{rms}$ which is just
$\frac{V_p}{\sqrt{2}}$. If you look on the back of anything that plugs
into a standard US power outlet, it will say something like ``For
120v''.  What they mean is ``For 120v RMS, so we expect the voltage to
fluctuate back and forth from 170v to -170v.''

Notice that this is the same Root-Mean-Squared that we defined
earlier, but now we know that if $y = \sin(x)$, the RMS of $y$ is
$1/\sqrt{2} \approx 0.707$.

For current, we do the same thing: If the current is AC, the power
consumed by a resistor is $I_{RMS}^2 R$, where $I_{RMS}$ is the peak
current divided by $sqrt{2}$.

\section{Power Line Losses}

A wire has some resistance. Thinner wires tend to have more resistance
than thicker ones. Aluminum wires tend to have more resistance than
copper wires.

Let's say that the power that comes to your house has to travel 20 km
from the generator in a cable that has about $1 \Omega$ of resistance
per km.  Let's say that your home is consuming 12 kilowatts of power.

If that power is 120v RMS from the generator to your home, what
percentage of the power is lost heating the power line? 10 amps RMS
flow through your home. When that current goes through the wire, $I^2
R = (100)(20) = 2000 watts$ is lost to heat.

So the power company would need to supply 14 kilowatts of power,
knowing that 2 kilowatts would be lost on the wires.

What if the power company moved the power at 120,000 volts RMS? Now
only 0.01 amps RMS flow through your home. When that current goes
through the wire $I^2R = (0.0001)(20) = .002$ watts of power are lost
on the power lines.

It is much, much more efficent. The only problem is that 120,000 volts
would be incredibly dangerous.  So the power company moves power long
distances at very high voltages, like 765 kV.  Before the power is
brought into your home, it is converted into a lower voltage using a
\newterm{transformer}.

\section{Transformers}

A transformer is a device that converts electrical power from one
voltage to another. A good tranformer is more than 95\% efficient. The
details of magnetic fields, flux, and inductance are beyond this scope
of this chapter, so I am going to give a relatively simple explanation
and admit that it is incomplete.

A transformer is a ring with two sets of coils wrapped around it.

\includegraphics[width=0.5\textwidth]{transformer.png}

(Diagram from Wikipedia)

When AC current is run through the primary winding, it create magnetic
flux in the ring.  The magnetic flux induces current in the secondary
winding.

If $V_P$ is the voltage across the primary winding and $V_S$ is the
voltage across the secondary winding, they are related by the
following equation:

$$\frac{V_P}{V_S} = \frac{N_P}{N_S}$$

where $N_P$ and $N_S$ are the number of turns in the primary and
secondary windings.

There are usually at least two transformers between you and the very
high voltage lines.  There are tranformers at the substation that make
the voltage low enough to travel on regular utility poles. On the
utility poles, you will see cans that contain smaller
transformers. Those step the voltage down to make the power safe to
enter your home.

\section{Phase and 3-phase power}

If two waves that are ``in sync'' we say they have the same \newterm{phase}.

\begin{tikzpicture}[
tl/.style = {% tick labels fill=white, inner sep=1pt,
  font=\scriptsize, }, ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.425] (0,-1.85) grid
(13,1.85);

% axes
\draw[->,thick] (0,0) -- (13,0);
% curve
\draw[thick,draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x)) * 1.7});
\draw[thick,dashed, draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x)) * 0.9});
\end{tikzpicture}

If they are the same frequency, but are not in-sync, we can talk about
the difference in their phase.

\begin{tikzpicture}[
tl/.style = {% tick labels fill=white, inner sep=1pt,
  font=\scriptsize, }, ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.425] (0,-1.85) grid
(13,1.85);

% axes
\draw[->,thick] (0,0) -- (13,0);
% curve
\draw[thick,draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x)) * 1.7});
\draw[thick,dashed, draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x) - 90) * 0.9});
\end{tikzpicture}

Here we see that the smaller wave is lagging by $\pi/2$ or $90^\circ$.

In most power grids, there are usually 3 wires carrying the power.
The voltage on each is $2\pi/3$ out of phase with the other two:

\begin{tikzpicture}[
tl/.style = {% tick labels fill=white, inner sep=1pt,
  font=\scriptsize, }, ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.425] (0,-1.85) grid
(13,1.85);

% axes
\draw[->,thick] (0,0) -- (13,0);
% curve
\draw[thick,draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x)) * 1.7});
\draw[thick,dashed, draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x) - 120) * 1.7});
\draw[thick,draw=sdkblue,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x) + 120) * 1.7});
\end{tikzpicture}

This is nice in two ways:
\begin{itemize}
\item While the power in each wire is fluctuating, the total power is not fluctuating at all.
\item While the power plant is pushing and pulling electrons on each
  wire, the total number number of electrons leaving the load is zero.
\end{itemize}
(Both these assume that there each wire is attached to a load with the same constant resistance.)

In big industrial factories, you will see all three wires enter the
building. Large amounts of smooth power delivery means a lot to an
industrial user.

In residential settings, each home gets its power from one of the three
wires. However two wires typically carry power into the home. Each
one carries 120V RMS, but they are out of phase by 180 degree. Lights
and small appliances are connected to one of the wires and ground, so
they get 120V RMS.  Large appliances, like air conditioners and
washing machines, are connected across the two wires so they get 240V
RMS.

\begin{tikzpicture}[
tl/.style = {% tick labels fill=white, inner sep=1pt,
  font=\scriptsize, }, ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.425] (0,-1.85) grid
(13,1.85);

% axes
\draw[->,thick] (0,0) -- (13,0);
% curve
\draw[thick,draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x)) * 1.7});
\draw[thick,dashed, draw=black,domain=0:13,samples=500,variable=\x] plot
(\x,{sin(deg(\x) + 180) * 1.7});
\draw[<->, thick, draw=black] (7.85398, 0) -- (7.85398, 1.7) node [midway, right] {120V};
\draw[<->, thick, draw=black] (4.712, -1.7) -- (4.712, 1.7);
\draw (4.712, 0.8) node[right] {240V};
\end{tikzpicture}

How do you get two circuits, 180 degrees out of phase, from one
circuit?  Using a center-tap transformer.

FIXME: Diagram here


\section{The bias-variance trade-off}

In smoothing in general there is a fundamental trade-off between the
bias and variance of the estimate, and this trade-off is governed by
the smoothing parameter.

Through out this section we will be using an artifical example defined
by
\begin{equation}
\label{sinexample}
y_i = 5 \sin (1/x) + \epsilon_i, i=1,\dots,n
\end{equation}
with the $\epsilon_i$ IID $N(0,1)$ or $t_3$.

\begin{figure}
\centerline{\epsfig{figure=Plots/plot-05-01.ps,angle=270,width=.8\textwidth}}
\caption{Outcomes of model with $f(x) = 5\sin(1/x)$ and IID normal
  errors with $\sigma^2=1$}
\end{figure}

The trade-off is most easily seen in the case of the running mean
smoother. The fitted running-mean smooth can be written as
\[
\hat{f}_k(x_0) = \frac{1}{2k+1} \sum_{i \in N^S_k(x_0)} y_i
\]
Under model (\ref{simplemodel}). The variance is easy to
compute. What is it?

The bias is
\[
\E[\hat{f}_k(x_0)] -f(x_0) =  \frac{1}{2k+1} \sum_{i \in N^S_k(x_0)}
\{ f(x_i) - f(x_0)   \}
\]

Notice that as $k$, in this case the smoothing parameter, grows the
variances decreases. However, the bigger the $k$ the more $f(x_i)$'s
get into the bias.

We have no idea of what $\sum_{i \in N^S_k(x_0)} f(x_i)$ is
because we don't know $f$! Let's see this in a more precise (not
much more) way.

Say we think that $f$ is smooth enough for us to assume that its second
derivative $f''(x_0)$ is bounded. Taylor's theorem says we can write
\[
f(x_i) = f(x_0) + f'(x_0) (x_i - x_0) + \frac{1}{2}f''(x_0)(x_i-x_0)^2 +
o(|x_i-x_0|^2).
\]
Because $ \frac{1}{2}f''(x_0)(x_i-x_0)^2$ is $O(|x_i-x_0|^2)$ we stop
being precise and write
\[
f(x_i) \approx f(x_0) + f'(x_0) (x_i - x_0) + \frac{1}{2}f''(x_0)(x_i-x_0)^2.
\]
Implicit here is the assumption that $|x_i-x_0|$ is small. This is the
way these asymptotics work. We assume that the kernel size
goes to 0 as $n$ gets big.

Why did we only go up to the second derivative?

To makes things simple, let's assume that the covariates $x$ are {\it equally spaced} and
let $\Delta = x_{j+1}-x_j$ we can write
\[
(2k+1)^{-1}\sum_{i \in N^S_k(x_0)} f(x_i) \approx f(x_0) + (2k+1)^{-1}\frac{k(k+1)}{6} f''(x_0) \Delta^2
\]
So now we see that the bias increases with $k^2$ and the second
derivative of the ``true'' function $f$. This agrees with our
intuition.

Now that we have
\[
\E\{\hat{f}_k(x_0) - f(x_0)\}^2 \approx \frac{\sigma^2}{2k+1} + \frac{k(k+1)}{6} f''(x_0) \Delta^2
\]
we can actually find an optimal $k$
\[
k_{opt} = \left\{ \frac{9\sigma^2}{2\Delta^4\{f''(x_i)\}^2}\right\}
\]
Usually this is not useful in practice because we have no idea of what
$f''(x)$ is like. So how do we chose smoothing parameters?

In Figure \ref{f5.1.2} we show the smooths obtained with a running
mean smoother with bandwidths of 0.01 and 0.1 on 25 replicates defined
by (\ref{sinexample}). The bias-variance trade-off can be clearly seen.

\begin{figure}
\centerline{\epsfig{figure=Plots/plot-05-02.ps,angle=270,width=.8\textwidth}}
\caption{\label{f5.1.2}Smooths using running-mean smoother with bandwidths of .01
  and 0.1. To the right are the smooths 25 replicates}
\end{figure}

\subsection{Bias-variance trade-off for linear smoothers}
Define $\bS_{\lambda}$ as the hat matrix for a particular smoother when
the smoothing parameter $\lambda$ is used. The ``smooth'' will be
written as $\hat{\f}_{\lambda} = \bS_{\lambda} \by$.

Define
\[
\bv_{\lambda} = \f - \E(\bS_{\lambda}\by)
\]
as the {\it bias} vector.

Define $\ave(\bx^2) = n^{-1} \sum_{i=1}^n x_i^2$ for any vector
$\bx$. We can derive the following formulas:
\begin{eqnarray*}
\mbox{MSE}(\lambda) &=& n^{-1} \sum_{i=1}^n
\var\{\hat{f}_{\lambda}(x_i)\} + \ave ( \bv_{\lambda}^2 ) \\
&=& n^{-1}\tr(\bS_{\lambda}\bS_{\lambda}') \sigma^2 + n^{-1}
\bv_{\lambda}'\bv_{\lambda}\\
\mbox{PSE}(\lambda) &=& \{1 + n^{-1} \tr(\bS_{\lambda}\bS_{\lambda}')
\} \sigma^2 +  n^{-1}
\bv_{\lambda}'\bv_{\lambda}.
\end{eqnarray*}

Notice for least-squares regression $\bS_{\lambda}$ is idempotent so
that $\tr(\bS_{\lambda}\bS_{\lambda}') = \tr(\bS_{\lambda}) =
\mbox{rank}(\bS_{\lambda})$ which is usually the
number of parameters in the model. This is why we will sometimes refer
to $\tr(\bS_{\lambda}\bS_{\lambda}')$ as the {\it equivalent number of
  parameters} or degrees of freedom of our smoother.


\chapter{Abstract}

Many contemporary proof assistants based on dependent type theories such as Coq and Agda
are founded on the types-as-propositions paradigm where type checking a program
corresponds to verifying a proof of some proposition in a higher-order predicate logic.
To ensure decidability of type checking and consistency of the logic,
these proof assistants forbid nonterminating recursive functions
using guard predicates that only allow structurally recursive functions
recurring only on syntactically smaller arguments.
However, these guard predicates are sometimes too restrictive
and reject obviously-terminating functions that aren't otherwise structurally recursive,
creating extra work for the programmer to convince the guard checker.

An alternative is to use type-based termination checking such as sized types,
where inductively-defined types are annotated with sizes.
Successful type checking guarantees that functions recur only on arguments whose types have smaller sizes,
rather than merely on syntactic subarguments.
There exist many sized dependent type theories,
but none simultaneously feature higher-rank size quantification,
which allows for passing around size-preserving functions,
and bounded size quantification,
which eliminates the need for complex monotonicity checks required by prior sized type systems.

In this thesis, I design a sized dependent type theory with higher-rank and bounded sizes (\lang),
and show that it's suitable for theorem proving by proving its logical consistency with a syntactic model:
by compiling \lang into the Extensional Calculus of Inductive Constructions (\CICE),
a variant of Coq's core type theory,
and showing that this translation is type preserving,
the consistency of \lang follows from the consistency of \CICE.
This approach, unfortunately, refutes the existence of an ``infinite'' size strictly greater than all sizes,
which is present in prior sized type systems to overcome the limitations of finitary size expressions,
meaning that some infinitary constructs aren't definable in \lang.

Even so, \lang provides a valid foundation for sized types in a proof assistant,
opening the way for future work on recovering expressivity lost from the lack of an infinite size
and on restricting sized types in Agda,
the only (inconsistent!) implementation of sized types in a major proof assistant,
to be consistent.


\documentclass[a4paper]{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{xcolor}
\usepackage{amsthm}
\usepackage[mathcal]{euscript}

\usepackage{url}

\newcommand{\Hcal}{\mathcal{H}}
\newcommand{\real}{\mathbb{R}}
\newcommand{\T}{\mathrm{T}}

\title{Some notes on Boyle, Dykstra (1986)}
\author{Nazarov Ivan}

\date{\today}

\begin{document}
\maketitle

Let $(\Hcal, \langle\cdot,\cdot\rangle)$ be a Hilbert space. We say that $(x_n)_{n\geq1}
\in \Hcal$ converges {\bf strongly} to $x\in\Hcal$ if $\|x_n - x\| \to 0$. A sequence
converges to $x$ weakly, if $\langle x_n, z \rangle \to \langle x, z\rangle$ for all
$z\in \Hcal$. Due to continuity of $\langle\cdot,\cdot\rangle$ w.r.t the product norm
topology and $\real$, strong convergence implies weak convergence.

A property of sequences, crucial for optimization problems in finite-dimensional spaces,
is strong sequential compactness: that every bounded sequence has a strongly convergent
subsequence (Bolzano-Weierstrass theorem). But in finite dimensional spaces it fails to
hold.

\subsection*{Projection onto one convex set} % (fold)
\label{sub:projection_onto_one_convex_set}

Let $C \subseteq \Hcal$ be a non-empty closed convex subset of $\Hcal$. Then for any
$g\in \Hcal$ there exists $g^*\in C$ such that $\|g - g^*\| = \inf_{h\in C} \|g - h\|$.
The element $g^*$ is a projection if and only if $\langle g - g^*, g^* - h\rangle
\geq 0$ for all $h \in C$. This follows from the minimization problem and the convexity
of $C$.

Indeed, let $f_t = g^* + t (h - g ^*)$ for any $t \in [0, 1]$. Then we have $f_t \in C$
and $\|f_t - g\| \geq \|g^* - g\|$ from the optimality of $g^*$. However
\begin{align*}
  \|f_t - g \pm g^*\|^2 - \|g^* - g\|^2
    &= \|f_t - g^*\|^2 + 2 \langle f_t - g^*, g^* - g\rangle
    \\
    &= t^2 \|h - g^*\|^2 + 2 t \langle h - g^*, g^* - g\rangle
    \,,
\end{align*}
whence
\begin{equation} \label{eq:proj_prop}
  \langle h - g^*, g^* - g\rangle
    = \lim_{t \downarrow 0}
      \tfrac1{2 t} \bigl( t^2 \|h - g^*\|^2 + 2 t \langle h - g^*, g^* - g\rangle \bigr)
      \geq 0
      \,.
\end{equation}
Conversely, such $g^*\in C$ is a minimizer of $\|h - g\|$ for $h \in C$:
\begin{equation*}
  \|h - g\|^2 - \|g - g^*\|^2
    = \|h - g^*\|^2 + 2 \langle g^* - h, g - g^*\rangle
    \geq \|h - g^*\|^2 \geq 0
    \,.
\end{equation*}

% subsection* projection_onto_one_convex_set (end)

\subsection*{Projection onto and intersection of convex sets} % (fold)
\label{sub:projection_onto_and_intersection_of_convex_sets}

Let $K$ non-empty closed convex sets $C_k \subseteq \Hcal$ with non-empty intersection.
Consider the following problem:
\begin{equation} \label{eq:k-projection}  % \tag{Proj}
  \begin{aligned}
    & \underset{h \in \Hcal}{\text{minimize}}
      & & \tfrac12 \|h - g\|^2
          \,, \\
    & \text{s.t.}
      & & h \in C = \bigcap_{k=1}^K C_k
          \,.
  \end{aligned}
\end{equation}
Dykstra's algorithm assumes that it is cheap an easy to project onto $C_k$ for any $k$,
and proposes the following iterations: {\bf starting} with $\delta^0_k = 0$ and $g^0_K = g$
{\bf do}
\begin{equation} \label{eq:dykstra_iter}  % \tag{}
  \begin{aligned}
    g^t_k
      &= \mathrm{proj}_{C_k}\bigl( g^t_{k-1} + \delta^{t-1}_k \bigr)
      \,,
      \, g^t_0 = g^{t-1}_K
      \,, \\
    \delta^t_k
      &= (g^t_{k-1} + \delta^{t-1}_k) - g^t_k
      \,,
      \, k = 1\ldots K
  \end{aligned}
\end{equation}
where $\mathrm{proj}_{C_k}(g) = \arg\min_{h \in C_k} \tfrac12 \|g - h\|^2$. Note
that, since $g^t_k$ is the projection of $g^t_{k-1} + \delta^{t-1}_k$ onto $C_k$,
the projection property \eqref{eq:proj_prop} implies that $\delta^t_k$ in
\eqref{eq:dykstra_iter} satisfy
\begin{equation} \label{eq:dykstra_prop}
  \forall{h\in C_k}
  \,\,
  \langle \delta^t_k, g^t_k - h \rangle
  = \bigl\langle (g^t_{k-1} + \delta^{t-1}_k) - g^t_k, g^t_k - h \bigr \rangle
  \geq 0
  \,.
\end{equation}
In particular, $h$ can be $g^s_k \in C_k$ for any $s\geq 1$ or any element of
$\bigcap_{j=1}^K C_j \subseteq C_k$.

Below we shall show that iterations \eqref{eq:dykstra_iter} converge to $g^*$, the
solution of \eqref{eq:k-projection}.

% subsection* projection_onto_and_intersection_of_convex_sets (end)

\subsection*{Working with the iterations} % (fold)
\label{sub:working_with_the_iterations}

The proof hinges on a key ``balance'' property of the updates in \eqref{eq:dykstra_iter}
and the property of the projection \eqref{eq:proj_prop}. The relation $g^t_{k-1} - g^t_k =
\delta^t_k - \delta^{t-1}_k$ implies
\begin{align}
  % g^t_k - g^t_{k-1} &= \delta^{t-1}_k - \delta^t_k
  %   \,, \notag \\
  g^t_k - g^{t-1}_K
    &= g^t_k - g^t_0
      = \sum_{j=1}^k (\delta^{t-1}_j - \delta^t_j)
    \,, \notag \\
  % g^t_K - g^{t-1}_K
  %   &= \sum_{j=1}^K \delta^{t-1}_j - \delta^t_j
  %   \,, \notag \\
  g^t_K - g^0_K
    &= \sum_{s=1}^t (g^s_K - g^{s-1}_K)
    % = \sum_{s=1}^t \sum_{j=1}^K \delta^{s-1}_j - \delta^s_j
    = \sum_{j=1}^K \sum_{s=1}^t (\delta^{s-1}_j - \delta^s_j)
    % = \sum_{j=1}^K \delta^0_j - \delta^t_j
    = \sum_{j=1}^K \delta^0_j - \sum_{j=1}^K \delta^t_j
    \,, \label{eq:tK_iterate} \\
  g^t_k - g
    &= g^t_k - g^0_K
    = - \sum_{j=1}^K \delta^{t-1}_j + \sum_{j=1}^k \delta^{t-1}_j - \delta^t_j
    = - \sum_{j=1}^k \delta^t_j - \sum_{j=k+1}^K \delta^{t-1}_j
    \,. \notag
\end{align}
This relation also implies that the difference $\|g^t_{k-1} - g^*\|^2 - \|g^t_k - g^*\|^2
= \ldots$ is
\begin{align*}
  \ldots
    % &= 2 \langle g^t_{k-1} - g^t_k, g^t_k - g^* \rangle
    %   + \| \delta^t_k - \delta^{t-1}_k \|^2
    &= 2 \langle \delta^t_k - \delta^{t-1}_k, g^t_k - g^* \rangle
      + \| \delta^t_k - \delta^{t-1}_k \|^2
    \\
    &= 2 \langle \delta^t_k, g^t_k - g^* \rangle
      - 2 \langle \delta^{t-1}_k, g^t_k - g^* \rangle
      + \| \delta^t_k - \delta^{t-1}_k \|^2
    \\
    &= \underbrace{2 \langle \delta^t_k, g^t_k - g^* \rangle}_{b^t_k}
      - \underbrace{2 \langle \delta^{t-1}_k, g^{t-1}_k - g^* \rangle}_{b^{t-1}_k}
      + \underbrace{2 \langle \delta^{t-1}_k, g^{t-1}_k - g^t_k\rangle}_{a^{t-1}_k}
      + \underbrace{\| \delta^t_k - \delta^{t-1}_k \|^2}_{c^t_k}
    \,,
\end{align*}
with $b^0_k = 2 \langle \delta^0_k, g^0_k - g^* \rangle = 0$ and $a^0_k = 2 \langle
\delta^0_k, g^0_k - g^1_k\rangle = 0$. Furthermore
\begin{align}
  % \|g^t_{k-1} - g^*\|^2 - \|g^t_k - g^*\|^2
  %   &= (b^t_k - b^{t-1}_k) + a^{t-1}_k + c^t_k
  %   \,, \notag \\
  \|g^t_0 - g^*\|^2 - \|g^t_k - g^*\|^2
    &= \sum_{j=1}^k (b^t_j - b^{t-1}_j)
      + \sum_{j=1}^k \bigl( a^{t-1}_j + c^t_j \bigr)
    \,, \label{eq:t0k_norm} \\
  \bigl \| g^{t_1}_K - g^* \bigr \|^2 - \bigl \| g^{t_2}_K - g^* \bigr \|^2
    % &= \sum_{s=t_1+1}^{t_2} \|g^{s-1}_K - g^*\|^2 - \|g^s_K - g^*\|^2
    %   \notag \\
    &= \sum_{t=t_1+1}^{t_2} \Bigl(
        \sum_{j=1}^K (b^t_j - b^{t-1}_j)
          + \sum_{j=1}^K \bigl( a^{t-1}_j + c^t_j \bigr)
        \Bigr)
      \notag \\
    % &= \sum_{j=1}^K \sum_{t=t_1+1}^{t_2} (b^t_j - b^{t-1}_j)
    %     + \sum_{j=1}^K \sum_{t=t_1+1}^{t_2} a^{t-1}_j
    %     + \sum_{t=t_1+1}^{t_2} \sum_{j=1}^K c^t_j
    %   \notag \\
    &= \sum_{j=1}^K (b^{t_2}_j - b^{t_1}_j)
        + \sum_{j=1}^K \sum_{t=t_1}^{t_2-1} a^t_j
        + \sum_{t=t_1+1}^{t_2} \sum_{j=1}^K c^t_j
      \notag \\
    &= \sum_{j=1}^K \Bigl( b^{t_2}_j + a^{t_1}_j - b^{t_1}_j
          + \sum_{t=t_1+1}^{t_2-1} a^t_j + \sum_{t=t_1+1}^{t_2} c^t_j
      \Bigr)
    \,, \label{eq:t1t2K_norm}
\end{align}
For $t_1 = 0$ and $t_2 = t$, we get
\begin{align}
  \|g^0_K - g^*\|^2 - \|g^t_K - g^*\|^2
    &= \sum_{k=1}^K b^t_k
        + \sum_{k=1}^K \sum_{s=1}^{t-1} a^s_k
        + \sum_{s=1}^t \sum_{k=1}^K \| \delta^s_k - \delta^{s-1}_k \|^2
      \notag \\
    &\geq
      \sum_{s=1}^t \sum_{k=1}^K \| \delta^s_k - \delta^{s-1}_k \|^2
    = \sum_{s=1}^t \sum_{k=1}^K \| g^s_{k-1} - g^s_k \|^2
    \,, \label{eq:gtk_bound}
\end{align}
since $b^t_k \geq 0$ and $a^t_k \geq 0$ for all $t\geq 0$ by \eqref{eq:dykstra_prop}.
The first and the last lines imply that $\|g^0_K - g^*\| \geq \|g^t_k - g^*\|$, whence
$(g^t_k)_{t\geq1}$ is bounded for all $k$.

% subsection* working_with_the_iterations (end)

\subsection*{Limiting properties} % (fold)
\label{sub:limiting_properties}

This section somewhat follows the proof in \cite{boyledykstra1986}.

The partial sum in the right hand side of \eqref{eq:gtk_bound} is bounded, therefore
the limit exists and the series $\sum_{t\geq 1} \sum_{k=1}^K \| g^t_{k-1} - g^t_k \|^2$
converge. This implies that for any $k$ we have $\|g^t_{k-1} - g^t_k\|^2 \to 0$ as
$t\to \infty$. Furthermore, if $(g^t_k)_{t\geq 1}\in \Hcal$ converges strongly to
$h$, then so does $(g^t_p)_{t\geq 1}$ for any $p=1,\,\ldots,\,K$: indeed
\begin{equation*}
  \limsup_{t\to \infty} \| g^t_p - h \|
    \leq \limsup_{t\to \infty} \|g^t_k - h\|
      + \sum_{j=p+1}^k \limsup_{t\to \infty} \| g^t_{j-1} - g^t_j \|
    \,.
\end{equation*}
Therefore proving the convergence of $(g^t_K)_{t\geq 1}$ is sufficient.

For any $f\in C = \bigcap_{k=1}^K C_k$ we have $\langle \delta^t_k,  g^t_k - f \rangle
\geq 0$ by \eqref{eq:dykstra_prop}, whence
\begin{equation*}
  \langle g^t_1 - f, g^t_K - g \rangle
    % = - \sum_{k=1}^K \langle g^t_1 \pm g^t_k - f, \delta^t_k \rangle
    = \sum_{k=1}^K \langle g^t_k - g^t_1, \delta^t_k \rangle
      + (-1) \sum_{k=1}^K \langle g^t_k - f, \delta^t_k \rangle
    \leq \sum_{k=2}^K \langle g^t_k - g^t_1, \delta^t_k \rangle
      \,.
\end{equation*}
If we put $d_t = \sum_{k=2}^K \| g^t_{k-1} - g^t_k \|$, then we get
\begin{align*}
  \sum_{k=2}^K \bigl\lvert \langle g^t_k - g^t_1, \delta^t_k \rangle \bigr\rvert
    &\leq \sum_{k=2}^K \|\delta^t_k\| \| g^t_k - g^t_1 \|
    \leq \sum_{k=2}^K \|\delta^t_k\| \sum_{j=2}^k \| g^t_{j-1} - g^t_j \|
    \\
    &\leq \sum_{k=2}^K \|\delta^t_k\| \, d_t
    = \sum_{k=2}^K d_t \Bigl\| \sum_{s=1}^t (\delta^s_k - \delta^{s-1}_k) \Bigr\|
    \\
    &\leq \sum_{k=2}^K \sum_{s=1}^t d_t \|\delta^s_k - \delta^{s-1}_k \|
    = \sum_{s=1}^t \sum_{k=2}^K d_t \|g^s_{k-1} - g^s_k \|
    = \sum_{s=1}^t d_t d_s
      \,.
\end{align*}
Observe that by Cauchy-Schwartz (H{\"o}lder for $p=q=2$) inequality we have
\begin{equation*}
  d_t
    \leq \sum_{k=1}^K 1 \cdot \| g^t_{k-1} - g^t_k \|
    \leq K^{\tfrac12} \biggl(
      \sum_{k=1}^K \| g^t_{k-1} - g^t_k \|^2
    \biggr)^{\tfrac12}
    \,,
\end{equation*}
which implies that
\begin{equation*}
  \sum_{t\geq 1} d^2_t
    \leq K \sum_{t\geq 1} \sum_{k=1}^K \| g^t_{k-1} - g^t_k \|^2
      < +\infty
    \,.
\end{equation*}
For such series there is $(t_n)_{n\geq1} \uparrow$ such that $\sum_{s=1}^{t_n} d_s
d_{t_n} \to 0$ ({\bf \color{red} WHY?}). Hence
\begin{equation*}
  \limsup_{n \to \infty} \langle g^{t_n}_1 - f, g^{t_n}_K - g \rangle
    \leq \limsup_{n \to \infty} \sum_{s=1}^{t_n} d_s d_{t_n}
    = 0
    \,.
\end{equation*}
Since $\|g^t_{k-1} - g^t_k \| \to 0$ as $t\to \infty$ and $(g^t_k)_{t\geq1}$ is
bounded for all $k$, we have
\begin{align*}
  \langle g^{t_n}_K - f, g^{t_n}_K - g \rangle
    &= \langle g^{t_n}_1 - f, g^{t_n}_K - g \rangle
        + \sum_{j=2}^K \langle g^{t_n}_j - g^{t_n}_{j-1}, g^{t_n}_K - g \rangle
      \\
    &\leq \langle g^{t_n}_1 - f, g^{t_n}_K - g \rangle
      + \bigl(
        \|g\| + \sup_{t,k\geq 1} \|g^t_k\|
      \bigr) \sum_{j=2}^K \|g^{t_n}_j - g^{t_n}_{j-1}\|
      \,.
\end{align*}
Therefore the right-hand side not greater than zero in the upper limit.

Next, since $\|g^{t_n}_k\|$ is bounded, it has a convergent subsequence in $\real$.
We can assume that $t_n$ itself is this subsequence, since refining convergent
sequences does not affect its limiting properties.

Finally, by weak compactness the bounded sequence $(g^{t_n}_K)_{n\geq 1}$ has another
subsequence that weakly converges to some $h \in \Hcal$. Again, we are free to let
$(t_n)_{n\geq 1}$ be that subsequence.

% So far we have shown that the iterates $g^t_k$ have the following properties:
% \begin{itemize}
%   \item $(g^t_k)_{t\geq 1}$ is bounded for all $k$;
%   \item if $(g^t_K)_{t\geq 1}$ converges strongly then so do $(g^t_k)_{\geq t}$ for $k=1..K$;
%   \item there is $(t_n)_{n\geq1} \uparrow$ with $\lim_{n\to 0} \langle g^{t_n}_K - f,
%   g^{t_n}_K - g \rangle \leq 0$ for all $f\in C$.
% \end{itemize}

% subsection* limiting_properties (end)

\subsection*{Finding a convergent sequence} % (fold)
\label{sub:finding_a_convergent_sequence}

We have shown that the iterates $g^t_k$ of \eqref{eq:dykstra_iter} admit
$(t_n)_{n \geq 1} \uparrow$ such that
\begin{itemize}
  \item there is $h\in \Hcal$ with $g^{t_n}_K \rightharpoonup h$;
  \item there exists $L \in \real$ with $\|g^{t_n}_K\| \to L$;
  \item $\limsup_{n\to \infty} \langle g^{t_n}_K - f, g^{t_n}_K - g \rangle \leq 0$
  for all $f\in C = \bigcap_{k=1}^K C_k$.
\end{itemize}
Observe also, that if $x_n \rightharpoonup x$ and $\|x_n\| \to L$, then from
\begin{equation*}
  \|x\|^2
    = \limsup_{n\to \infty} \langle x_n, x\rangle
    \leq \limsup_{n\to \infty} \|x_n\| \|x\|
    = L \|x\|
    \,,
\end{equation*}
we conclude that $\|x\| \leq L$. We apply this to $x_n = g^{t_n}_K$ and $x = h$ to
get $\|h\|\leq L$.

From the properties of $(t_n)_{n\geq1}$ we have
\begin{align} \label{eq:weak_lim_equ}
  \limsup_{n\to \infty}
    \langle g^{t_n}_K - f, g^{t_n}_K - g \rangle
    &= \limsup_{n\to \infty} \bigl(
        \|g^{t_n}_K\|^2 - \langle g^{t_n}_K, g \rangle - \langle f, g^{t_n}_K \rangle
      \bigr) + \langle f, g \rangle
    \notag \\
    &= L^2 - \langle h, g \rangle - \langle h, g \rangle
      + \langle f, g \rangle \pm \|h\|^2
    \notag \\
    &= L^2 - \|h\|^2 + \langle h - f, h - g \rangle
  \,,
\end{align}
and thus $\langle h - f, h - g \rangle \leq 0$ for any $f \in C$.

Next, for any $f\in \Hcal$ and $k=1..K$ we have the following inequality
\begin{equation*}
  \limsup_{n\to\infty}\,
    \bigl\lvert
      \langle g^{t_n}_K, f \rangle - \langle g^{t_n}_k, f \rangle
    \bigr\rvert
    \leq
      \|f\| \sum_{j=k+1}^K \limsup_{n\to\infty}
        \bigl \|g^{t_n}_{j-1} - g^{t_n}_j \bigr \|
      \,,
\end{equation*}
which implies that $g^{t_n}_k \rightharpoonup h$ for any $k$. However, for any $k$
from the weak convergence of $x_n = g^{t_n}_k$ to $h$ it is possible to deduce that
there is a subsequence $(n^k_i)_{i\geq1}$ with $\tfrac1m \sum_{i=1}^m x_{n^k_i} \to h$
strongly ({\bf \color{red} WHY?}). Therefore $h \in C_k$, since $x_{n^k_i} \in C_k$
for all $i$, and $C_k$ is convex and {\bf strongly closed}.

Therefore, the weak limit $h$ of $g^{t_n}_k$ is in $C$. From \eqref{eq:weak_lim_equ}
for $f = h$ we conclude that $0 \geq L^2 - \|h\|^2$, whence $\|h\| = L$. This implies
that $g^{t_n}_k \to h$, because
\begin{equation*}
  \limsup_{n\to\infty} \| g^{t_n}_k - h \|^2
    = \| h \|^2 + \limsup_{n\to\infty} \|g^{t_n}_k\|^2 - 2\langle g^{t_n}_k, h \rangle
    = \| h \|^2 + L^2 - 2 \| h \|^2
    \,.
\end{equation*}
To conclude, we have found a subsequence $(t_n)_{n\geq1} \uparrow$ such that
\begin{itemize}
  \item $g^{t_n}_k \to h$ strongly for any $k$ and $h=g^*$, since $h \in C$ and
  satisfies \eqref{eq:proj_prop}.
\end{itemize}

% subsection* finding_a_convergent_sequence (end)

\subsection*{Back to the iterations} % (fold)
\label{sub:back_to_the_iterations}

For any $t \geq t_n$ the result in \eqref{eq:t1t2K_norm} implies
% \begin{align*}
%   \bigl \| g^{t_n}_K - g^* \bigr \|^2
%     &= \bigl \| g^t_K - g^* \bigr \|^2
%       + \sum_{j=1}^K (b^t_j - b^{t_n}_j)
%       + \sum_{j=1}^K \sum_{s=t_n}^{t-1} a^s_j
%       + \sum_{j=1}^K \sum_{s=t_n+1}^t c^s_j
%       \\
%     &\geq \bigl \| g^t_K - g^* \bigr \|^2
%       - \sum_{j=1}^K b^{t_n}_j
%     \,,
% \end{align*}
\begin{equation*}
  \bigl \| g^t_K - g^* \bigr \|^2
    \leq \bigl \| g^{t_n}_K - g^* \bigr \|^2
    + \sum_{k=1}^K b^{t_n}_k
    \leq \bigl \| g^{t_n}_K - g^* \bigr \|^2
    + 2 \Bigl \lvert
        \sum_{k=1}^K \langle \delta^{t_n}_k, g^{t_n}_k - g^* \rangle
      \Bigr \rvert
    \,.
\end{equation*}
Since \eqref{eq:tK_iterate} implies $g - g^t_K = \sum_{k=1}^K \delta^t_k$, we
can get the following upper bound
\begin{align*}
  \Bigl \lvert \sum_{k=1}^K \langle \delta^t_k, g^* - g^t_k \rangle \Bigr \rvert
    &= \Bigl \lvert
      \Bigl\langle \sum_{k=1}^K \delta^t_k, g^* - g^t_1 \Bigr \rangle
      + \sum_{k=1}^K \langle \delta^t_k, g^t_1 - g^t_k \rangle
      \Bigr \rvert
      \\
    &\leq \bigl\lvert \langle g^t_K - g, g^t_1 - g^* \rangle \bigr\rvert
      + \sum_{k=2}^K \bigl\lvert \langle \delta^t_k, g^t_k - g^t_1 \rangle \bigr\rvert
      \\
    &\leq \bigl\lvert \langle g^t_K - g, g^t_1 - g^* \rangle \bigr\rvert
      + \sum_{s=1}^t d_t d_s
      \,.
\end{align*}
Applying this bound to the subsequence $(t_n)_{n\geq1}$ we get in the limit
\begin{align*}
  \limsup_{n\to \infty}
    \Bigl \lvert \sum_{k=1}^K \langle \delta^{t_n}_k, g^* - g^{t_n}_k \rangle \Bigr \rvert
    &\leq \limsup_{n\to \infty}
      \bigl\lvert \langle g^{t_n}_K - g, g^{t_n}_1 - g^* \rangle \bigr\rvert
      + \limsup_{n\to \infty} \sum_{s=1}^{t_n} d_s d_{t_n}
      \\
    &\leq \limsup_{n\to \infty} \|g^{t_n}_K - g\| \|g^{t_n}_1 - g^* \|
      = 0
    \,,
\end{align*}
because $g^{t_n}_k \to g^*$ and $g^t_k$ is bounded. Therefore for any $\varepsilon > 0$
there is $N_\varepsilon \geq 1$ such that $\|g^{t_n}_K - g^*\|^2 < \tfrac\varepsilon2$
and $\lvert \sum_{k=1}^K \langle \delta^{t_n}_k, g^* - g^{t_n}_k \rangle \rvert <
\tfrac\varepsilon4$ for every $n \geq N_\varepsilon$. Hence for all $t \geq T_\varepsilon$,
$T_\varepsilon = t_{N_\varepsilon + 1}$, we get
\begin{equation*}
  \bigl \| g^t_K - g^* \bigr \|^2
    \leq \bigl \| g^{t_n}_K - g^* \bigr \|^2
    + 2 \Bigl \lvert
        \sum_{k=1}^K \langle \delta^{t_n}_k, g^{t_n}_k - g^* \rangle
      \Bigr \rvert
    < \tfrac\varepsilon2 + 2 \tfrac\varepsilon4
    \,.
\end{equation*}
Therefore $g^t_k$ converges strongly to $g^*$ for any $k=1\,, \ldots\,,K$.

% subsection* back_to_the_iterations (end)

\subsection*{Application to the GLS estimate} % (fold)
\label{sub:application_to_the_gls_estimate}

The GLS estimate of $y\sim X$ with weights $\Omega$ is $\hat{\beta} = \bigl(X^\T \Omega^{-1} X
\bigr)^{-1} X^\T \Omega^{-1} y$ and solves the unconstrained problem
\begin{equation} \label{eq:gls_unc}
  \begin{aligned}
    & \underset{\beta\in \real^p}{\text{minimize}}
      & & \tfrac12 \bigl(y - X \beta\bigr)^\T \Omega^{-1} \bigl(y - X \beta\bigr)
          \,.
  \end{aligned}
\end{equation}

Noting that the objective can be rewritten as
\begin{align*}
  \bigl(y - X \beta\bigr)^\T \Omega^{-1} \bigl(y - X \beta\bigr)
    &= \bigl(y - X \hat{\beta}\bigr)^\T \Omega^{-1} \bigl(y - X \hat{\beta}\bigr)
    \\
    &+ 2 \bigl(y - X \hat{\beta}\bigr)^\T \Omega^{-1} X \bigl(\hat{\beta} - \beta\bigr)
    \\
    &+ \bigl(X \hat{\beta} - X \beta\bigr)^\T \Omega^{-1} \bigl(X \hat{\beta} - X \beta\bigr)
    \,.
\end{align*}
and that
\begin{align*}
  \bigl(y - X \hat{\beta}\bigr)^\T
    &= \bigl(y - X \bigl(X^\T \Omega^{-1} X \bigr)^{-1} X^\T \Omega^{-1} y\bigr)^\T \Omega^{-1} X
    \\
    &= y^\T \Omega^{-\T} \bigl(\Omega - X \bigl(X^\T \Omega^{-1} X \bigr)^{-1} X^\T \bigr)^\T \Omega^{-1} X
    \\
    &= y^\T \Omega^{-1} \bigl(X - X \bigl(X^\T \Omega^{-1} X \bigr)^{-1} X^\T \Omega^{-1} X \bigr)
    = y^\T \Omega^{-1} \bigl(X - X\bigr)
    \,,
\end{align*}
we get a reformulation of the constrained GLS problem
\begin{equation} \label{eq:gls_proj}
  \begin{aligned}
    & \underset{\beta\in C}{\text{minimize}}
      & & \tfrac12 \bigl(\beta - \hat{\beta} \bigr)^\T X^\T \Omega^{-1} X \bigl(\beta - \hat{\beta}\bigr)
          \,,
  \end{aligned}
\end{equation}
as a projection of $\hat{\beta}$ onto $C$ with respect a special metric.

% subsection* application_to_the_gls_estimate (end)

\end{document}


% !TEX root = main.tex

%-------------------------------------------------
\section{Central limit theorem}\label{sec:clt}

We will need the following definition of the exponential function $e^x$. This is one of several equivalent definitions.
\begin{definition}[The compound interest formula]\label{def:compound_interest_formula}
The exponential function can be defined by
\[
e^x = \lim_{n\to\infty}\left(1+\frac{x}{n}\right)^{n}.
\]
\end{definition}

%\begin{lemma}\label{def:compound_interest_formula}
%For any constant $c\in\R$,
%\[
%\left(1+\frac{c}{n}\right)^n\to e^c \quad\text{as}\quad n\to\infty.
%\]
%\end{lemma}
%\begin{proof}
%By the binomial theorem,
%\begin{align*}
%\left(1+\frac{c}{n}\right)^n
%	& = \sum_{k=1}^n \binom{n}{k}\left(\frac{c}{n}\right)^k \\
%	& = \sum_{k=1}^n \frac{n!}{(n-k)!k!}\left(\frac{c^k}{n^k}\right) \\
%	& = \sum_{k=1}^n \frac{c^k}{k!}\left(\frac{n(n-1)\ldots(n-k+1)}{n^k}\right) \\
%	& = \sum_{k=1}^n \frac{c^k}{k!}\left[1\left(1-\frac{1}{n}\right)\left(1-\frac{2}{n}\right)\cdots\left(1-\frac{k+1}{n}\right)\right] \\
%	& \to \sum_{k=1}^{\infty} \frac{c^k}{k!} = e^c \quad\text{as}\quad n\to\infty.
%\end{align*}
%\end{proof}

%-----------------------------
\subsection{The Poisson limit theorem}

The Poisson limit theorem asserts that the $\text{Binomial}(n,p)$ distribution can be approximated by the $\text{Poisson}(np)$ distribution when $n$ is large and $p$ is small.

% theorem
\begin{theorem}[Poisson limit theorem]
If $X_n\sim\text{Binomial}(n,\lambda/n)$ then the distribution of $X_n$ converges to the $\text{Poisson}(\lambda)$ distribution as $n\to\infty$.
\end{theorem}

% proof
\begin{proof}
By the inversion theorem for CFs, it is enough to show that the CF of $X_n$ converges to the CF of the $\text{Poisson}(\lambda)$ distribution as $n\to\infty$.

\bigskip
Recall that
\bit
\it the CF of the $\text{Binomial}(n,p)$ distribution is $\phi(t)=(1-p+pe^{it})^n$, and
\it the CF of the $\text{Poisson}(\lambda)$ distribution is $\phi(t)=e^{\lambda(e^{it}-1)}$.
\eit
The CF of $X_n\sim\text{Binomial}(n,\lambda/n)$ is therefore
\[
\phi_{X_n}(t)
	= \expe(e^{itX_n})
	= \left(1-\frac{\lambda}{n} + \frac{\lambda}{n}e^{it}\right)^n
	= \left[1 + \frac{\lambda(e^{it}-1)}{n}\right]^n
\]
By definition~\ref{def:compound_interest_formula},
\[
\phi_{X_n}(t) \to e^{\lambda(e^{it}-1)}\quad\text{as}\quad n\to\infty.
\]
This is the CF of the $\text{Poisson}(\lambda)$ distribution, and the result follows by the inversion theorem for CFs.
\end{proof}

%-----------------------------
%\subsubsection{Taylor approximation}
Let $X$ be a random variable and let $\phi_X$ be its characteristic function (CF). We can use the series expansion of the exponential function to show that
\begin{align*}
\phi_X(t)
	= \expe(e^{itX})
	& = \sum_{k=0}^{\infty} \frac{\expe(X^k)}{k!}(it)^k \\
	& = 1 + i\expe(X)t - \frac{\expe(X^2)}{2}t^2 -i\frac{\expe(X^3)}{6}t^3 + \ldots
\end{align*}

%Characteristic functions $\phi(t)$ converge for all $t$, and are therefore equal to their \emph{Taylor series}:
%\[
%\phi(t) = \sum_{k=0}^{\infty} \frac{\phi^{(k)}(0)}{k!} t^k
%\]
%where $\phi^{(k)}(0)$ is the $k$th derivative of $\phi(t)$ evaluated at $0$, which reduces to
%\[
%\phi(t)	= 1 + i\expe(X)t - \frac{\expe(X^2)}{2}t^2 -i\frac{\expe(X^3)}{6}t^3 + \ldots
%	\qquad\text{for all $|t|<R$.}
%\]
When $t$ is small, $t^2$ is smaller and $t^3$ is smaller still: as $k$ increases the corresponding terms thus contribute less and less to the value of the sum. We can therefore approximate $\phi(t)$ by taking only the first few terms of the sum, and this approximation becomes increasingly accurate as $t$ tends to zero.

\bigskip
\begin{tabular}{lll}\hline
Linear (first order)	& $\phi(t) =	1 + i\expe(X)t + o(t)$ 								& as $t\to 0$, \\
Quadratic (second order)& $\phi(t) =	1 + i\expe(X)t - \frac{1}{2}\expe(X^2)t^2 + o(t^2)$ 	& as $t\to 0$, \\
Cubic (third order) 	& $\phi(t) =	1 + \expe(X)t + \frac{1}{2}\expe(X^2)t^2 + \frac{1}{6}\expe(X^3)t^3 + o(t^3)$ & as $t\to 0$. \\ \hline
\end{tabular}
\bigskip

Here $o(t^k)$ represents a quantity that converges to zero faster than $t^k$ in the sense that $o(t^k)/t^k\to 0$ as $t\to 0$. Such quantities are represented in this way to indicate that they can be safely ignored when $t$ is sufficiently small. A second order approximation will be needed to prove the central limit theorem.

%-----------------------------
\subsection{The central limit theorem}

Let $X_1,X_2,\ldots$ be independent and identically distributed random variables and consider the sequence of partial sums
\[
S_n = X_1 + X_2 + \ldots + X_n.
\]
By independence, $\expe(S_n)=n\mu$ and $\var(S_n)=n\sigma^2$.

By the law of large numbers, $S_n$ is approximately equal to its mean $n\mu$ when $n$ is large. This however does not say much about the \emph{distribution} of $S_n$ when $n$ is large.
%\eit

\bigskip
It turns out that if the $X_i$ have finite mean and variance then \emph{irrespective of the distribution of the $X_i$} the distribution of the standardised sums
\[
Z_n = \frac{S_n-\expe(S_n)}{\sqrt{\var(S_n)}} %= \frac{S_n - n\mu}{\sigma\sqrt{n}}
\qquad\text{or equivalently}\qquad
Z_n=\frac{1}{\sqrt{n}}\sum_{i=1}^n\left(\frac{X_i-\mu}{\sigma}\right)
\]
converges to the standard normal distribution $N(0,1)$ as $n\to\infty$.

% theorem
\begin{theorem}[Central limit theorem]
Let $X_1,X_2,\ldots$ be a sequence of independent and identically distributed with common mean $\mu$ and variance $\sigma^2$. If $\mu$ and $\sigma^2$ are both finite, then the distribution of the standardised sums
%\[
%Z_n = \frac{S_n - n\mu}{\sigma\sqrt{n}}\qquad\text{where}\qquad S_n=X_1+\ldots+X_n,
%\]
\[
%Z_n=\frac{\sum_{i=1}^n X_i - n\mu}{\sigma\sqrt{n}}
%\qquad\text{or equivalently}\qquad
Z_n=\frac{1}{\sqrt{n}}\sum_{i=1}^n\left(\frac{X_i-\mu}{\sigma}\right)
\]
converges to the standard normal distribution $\text{N}(0,1)$ as $n\to\infty$.
\end{theorem}

\begin{proof}
Let $\displaystyle Y_i = \frac{X_i-\mu}{\sigma}$. Then $\expe(Y_i)=0$ and $\var(Y_i)=1$, and
\[
Z_n  = \frac{1}{\sqrt{n}}\sum_{i=1}^n Y_i
\]

\bit
\it Let $\phi_Y(t)$ denote the common CF of the $Y_i$.
\it Let $\phi_{Z_n}(t)$ denote the CF of $Z_n$.
\eit
By Taylor's theorem,
\[
\phi_Y(t) = \expe(e^{itY}) = \sum_{j=0}^k \frac{\expe(Y^j)}{j!}(it)^j + o((it)^k) \qquad\text{as}\quad t\to 0.
\]

Taking the first three terms in the Taylor expansion of $\phi_Y(t)$, since $\expe(Y)=0$ and $\expe(Y^2)=1$ we get
\[
\phi_Y(t) =  1 - \frac{1}{2}t^2 + o(t^2) \quad\text{as}\quad t\to 0
\]

By the properties of CFs,
\begin{align*}
\phi_{Z_n}(t)
	& = \phi_{\frac{1}{\sqrt{n}}(Y_1+Y_2+\ldots+Y_n)}(t) \\
	& = \phi_{Y_1+Y_2+\ldots+Y_n}\left(\frac{t}{\sqrt{n}}\right) \\
	& = \left[\phi_Y\left(\frac{t}{\sqrt{n}}\right)\right]^n \\
	& = \left[1 - \frac{t^2}{2n} + o\left(\frac{t^2}{n}\right)\right]^n \quad\text{as $t\to 0$}\\
	& \to e^{-\frac{1}{2}t^2} \quad\text{as $n\to\infty$},
\end{align*}
where the last step follows by definition~\ref{def:compound_interest_formula}. This is the CF of the $\text{N}(0,1)$ distribution, and the result follows by the inversion theorem for CFs.
\end{proof}

% example (Erlang)
\begin{example}
The \emph{Erlang distribution} with parameters $n\in\N$ and $\lambda>0$ is defined to be the sum of $n$ independent and identically distributed random variables $X_1,X_2,\ldots,X_n$, where each $X_i$ is exponentially distributed with (rate) parameter $\lambda$. Show that if $S_n\sim\text{Erlang}(n,\lambda)$, then the random variable
\[
Z_n=\displaystyle\frac{\lambda S_n-n}{\sqrt{n}}
\]
has approximately the standard normal distribution when $n$ is large.
\end{example}

\begin{solution}
Let $S_n\sim\text{Erlang}(n,\lambda)$. Then $S_n$ can be written as the sum of $n$ independent and identically distributed random variables $X_i$:
\[
S_n = X_1 + X_2 + \ldots + X_n\qquad\text{where}\quad X_i\sim\text{Exponential}(\lambda).
\]
Since $X_i\sim\text{Exponential}(\lambda)$ with $\lambda > 0$, we have
\[
\expe(X_i) = \frac{1}{\lambda} <\infty
\text{\quad and\quad}
\var(X_i) = \frac{1}{\lambda^2} <\infty.
\]

Furthermore, by independence we have
\[
\expe(S_n)= \sum_{i=1}^n \expe(X_i) = \frac{n}{\lambda}
\qquad\text{and}\qquad
\var(S_n) =  \sum_{i=1}^n \var(X_i) = \frac{n}{\lambda^2}.
\]
Hence,
\[
\frac{S_n - \expe(S_n)}{\sqrt{\var(S_n)}}
	= \frac{S_n - n/\lambda}{\sqrt{n/\lambda^2}}
	= \frac{\lambda S_n - n}{\sqrt{n}}
\]
By the central limit theorem,
\[
\frac{S_n - \expe(S_n)}{\sqrt{\var(S_n)}} \to Z \quad\text{in distribution as $n\to\infty$,}
\]
where $Z\sim\text{N}(0,1)$.
\end{solution}

\begin{exercise}
\begin{questions}

\question
The continuous uniform distribution on $(a,b)$ has the following PDF:
\[
f(x) = \begin{cases}
	\displaystyle\frac{1}{b-a} 	& a < x < b,  \\[2ex]
	0				& \text{otherwise.}
\end{cases}
\]
Use the central limit theorem to deduce the approximate distribution of the sample mean of $n$ independent observations from this distribution when $n$ is large.
\begin{answer}
The mean is
\begin{align*}
\mu
	& = \int_{a}^{b}\frac{x}{b-a}\,dx = \frac{a+b}{2},
\intertext{and the second moment is}
\mu_{2}
	& = \int_{a}^{b}\frac{x^{2}}{b-a}\,dx = \frac{a^{2}+ab+b^{2}}{3},
\intertext{so the variance is}
\sigma^{2}
	& = \expe(X^{2}) - \expe(X)^{2} = \frac{(b-a)^{2}}{12}
\end{align*}
By the central limit theorem, if $X$ is a random variable with mean $\mu$ and variance $\sigma^{2}$, the distribution of the sample mean $\bar{X}$ of a random sample of $n$ independent observations is approximately $N(\mu,\frac{\sigma^{2}}{n})$, the approximation being better for larger $n$. In this case, the approximate distribution of $\bar{X}$ is $N\left(\frac{a+b}{2},\frac{(b-a)^{2}}{12n}\right)$.
\end{answer}

\question
The exponential distribution with scale parameter $\theta>0$ has the following PDF:
\[
f(x) = \begin{cases}
	\displaystyle\frac{1}{\theta} e^{-x/\theta} 	& x > 0,  \\[2ex]
	0					& \text{otherwise.}
\end{cases}
\]
Use the central limit theorem to deduce the approximate distribution of the sample mean of $n$ independent observations from this distribution when $n$ is large.
\begin{answer} % <<<
\begin{align*}
\expe(X)
	& = \frac{1}{\theta}\int_{0}^{\infty} x e^{-x/\theta}\,dx = \theta, \\
\expe(X^2)
	& = \frac{1}{\theta}\int_{0}^{\infty} x^2 e^{-x/\theta}\,dx  = 2\theta^2 \\
\var(X)
	& = \expe(X^2) - \expe(X)^{2} = \theta^2.
\end{align*}

By the CLT, the distribution of $\bar{X}$ is approximately $\text{N}(\mu,\frac{\sigma^{2}}{n})$, the approximation being better for larger $n$. In this case, the approximate distribution of $\bar{X}$ is $\text{N}\left(\theta,\theta^2/n\right)$.
\end{answer}

\question
We perform a sequence of independent Bernoulli trials, each with probability of success $p$, until a fixed number $r$ of successes is obtained. The total number of failures $Y$ (up to the $r$th succes) has the \emph{negative binomial} distribution with parameters $r$ and $p$, so the PMF of $Y$ is
\[
\prob(Y=k) = \binom{k+r-1}{k} (1-p)^k p^r,\qquad k=0,1,2,\ldots
\]
Using the fact that $Y$ can be written as the sum of $r$ independent geometric random variables, show that this distribution can be approximated by a normal distribution when $r$ is large.

\begin{answer}
If $Y\sim\text{NB}(r,p)$, we can write
\[
Y = X_1 + X_2 + \ldots + X_r\qquad\text{where}\quad X_i\sim\text{Geometric}(p).
\]
Let $X\sim\text{Geometric}(p)$. Since $\var(X)<\infty$, it follows by the central limit theorem that
\[
\frac{Y - r\expe(X)}{\sqrt{r\var(X)}} \to \text{N}(0,1) \quad\text{in distribution as }r\to\infty.
\]
In fact, since $\expe(X)=(1-p)/p$ and $\var(X)=(1-p)/p^2$, we see that $Y$ can be approximated by the
$\displaystyle\text{N}\left(\frac{r(1-p)}{p}, \frac{r(1-p)}{p^2}\right)$ distribution as $r\to\infty$.
\end{answer}

\end{questions}
\end{exercise}