\chapter{Floating Point Numbers}
\label{chapter:floatingpoint}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{IEEE-754 Floating Point Number Representation}
\label{chapter::floatingpoint}

This section provides an overview of the IEEE-754 32-bit binary floating
point format.\cite{ieee:754}

\begin{itemize}
\item Recall that the place values for integer binary numbers are:
\begin{verbatim}
   ... 128 64 32 16 8 4 2 1
\end{verbatim}
\item We can extend this to the right in binary similar to the way we do for
decimal numbers:
\begin{verbatim}
   ... 128 64 32 16 8 4 2 1 . 1/2 1/4 1/8 1/16 1/32 1/64 1/128 ...
\end{verbatim}
The `.' in a binary number is a binary point, not a decimal point.

\item We use scientific notation as in $2.7 \times 10^{-47}$ to express either
small fractions or large numbers when we are not concerned every last digit
needed to represent the entire, exact, value of a number.

\item The format of a number in scientific notation is $mantissa \times base^{exponent}$

\item In binary we have $mantissa \times 2^{exponent}$

\item IEEE-754 format requires binary numbers to be {\em normalized} to
$1.significand \times 2^{exponent}$ where the {\em significand}
is the portion of the {\em mantissa} that is to the right of the binary-point.

\begin{itemize}
\item The unnormalized binary value of $-2.625$ is $-10.101$
\item The normalized value of $-2.625$ is $-1.0101 \times 2^1$
\end{itemize}

\item We need not store the `1.' part because {\em all} normalized floating
point numbers will start that way.  Thus we can save memory when storing
normalized values by inserting a `1.' to the left of significand.

{
\small
\setlength{\unitlength}{.15in}
\begin{picture}(32,4)(0,0)
	\put(0,1){\line(1,0){32}}		% bottom line
	\put(0,2){\line(1,0){32}}		% top line

	\put(0,1){\line(0,1){2}}		% left vertical
	\put(0,2){\makebox(1,1){\tiny 31}}	% left end bit number marker

	\put(32,1){\line(0,1){2}}		% vertical right end
	\put(31,2){\makebox(1,1){\tiny 0}}	% right end bit number marker

	\put(0,0){\makebox(1,1){\small sign}}
	\put(1,0){\makebox(8,1){\small exponent}}
	\put(9,0){\makebox(23,1){\small significand}}

    \put(0,1){\makebox(1,1){1}}		% sign

	\put(1,1){\line(0,1){2}}		% seperator
	\put(1,2){\makebox(1,1){\tiny 30}}	% bit marker

    \put(1,1){\makebox(1,1){1}}		% exponent
    \put(2,1){\makebox(1,1){0}}
    \put(3,1){\makebox(1,1){0}}
    \put(4,1){\makebox(1,1){0}}
    \put(5,1){\makebox(1,1){0}}
    \put(6,1){\makebox(1,1){0}}
    \put(7,1){\makebox(1,1){0}}
    \put(8,1){\makebox(1,1){0}}

	\put(8,2){\makebox(1,1){\tiny 23}}	% bit marker
	\put(9,1){\line(0,1){2}}		% seperator
	\put(9,2){\makebox(1,1){\tiny 22}}	% bit marker

    \put(9,1){\makebox(1,1){0}}
    \put(10,1){\makebox(1,1){1}}
    \put(11,1){\makebox(1,1){0}}
    \put(12,1){\makebox(1,1){1}}
    \put(13,1){\makebox(1,1){0}}
    \put(14,1){\makebox(1,1){0}}
    \put(15,1){\makebox(1,1){0}}
    \put(16,1){\makebox(1,1){0}}
    \put(17,1){\makebox(1,1){0}}
    \put(18,1){\makebox(1,1){0}}
    \put(19,1){\makebox(1,1){0}}
    \put(20,1){\makebox(1,1){0}}
    \put(21,1){\makebox(1,1){0}}
    \put(22,1){\makebox(1,1){0}}
    \put(23,1){\makebox(1,1){0}}
    \put(24,1){\makebox(1,1){0}}
    \put(25,1){\makebox(1,1){0}}
    \put(26,1){\makebox(1,1){0}}
    \put(27,1){\makebox(1,1){0}}
    \put(28,1){\makebox(1,1){0}}
    \put(29,1){\makebox(1,1){0}}
    \put(30,1){\makebox(1,1){0}}
    \put(31,1){\makebox(1,1){0}}
\end{picture}
}

%\item $-((1 + \frac{1}{4} + \frac{1}{16}) \times 2^{128-127}) = -(1 \frac{5}{16} \times 2^{1}) = -(1.3125 \times 2^{1}) = -2.625$
\item $-((1 + \frac{1}{4} + \frac{1}{16}) \times 2^{128-127}) = -((1 + \frac{1}{4} + \frac{1}{16}) \times 2^1) = -(2 + \frac{1}{2} + \frac{1}{8}) = -(2 + .5 + .125) = -2.625$

\item IEEE-754 formats:

\begin{tabular}{|l|l|l|}
\hline
				& IEEE-754 32-bit	& IEEE-754 64-bit	\\
\hline
sign			& 1 bit				& 1 bit			\\
exponent		& 8 bits (excess-127)			& 11 bits (excess-1023)		\\
mantissa		& 23 bits			& 52 bits		\\
max exponent	& 127				& 1023			\\
min exponent	& -126				& -1022			\\
\hline
\end{tabular}

\item When the exponent is all ones, the significand is all zeros, and
the sign is zero, the number represents positive infinity.

\item When the exponent is all ones, the significand is all zeros, and
the sign is one, the number represents negative infinity.

\item Note that the binary representation of an IEEE-754 number in memory
can be compared for magnitude with another one using the same logic as for
comparing two's complement signed integers because the magnitude of an
IEEE number grows upward and downward in the same fashion as signed integers.
This is why we use excess notation and locate the significand's sign bit on
the left of the exponent.

\item Note that zero is a special case number.  Recall that a normalized
number has an implied 1-bit to the left of the significand\ldots\ which
means that there is no way to represent zero!
Zero is represented by an exponent of all-zeros and a significand of
all-zeros.  This definition allows for a positive and a negative zero
if we observe that the sign can be either 1 or 0.

\item On the number-line, numbers between zero and the smallest fraction in
either direction are in the {\em \gls{underflow}} areas.
\enote{Need to add the standard lecture number-line diagram showing
where the over/under-flow areas are and why.}

\item On the number line, numbers greater than the mantissa of all-ones and the
largest exponent allowed are in the {\em \gls{overflow}} areas.

\item Note that numbers have a higher resolution on the number line when the
exponent is smaller.

\item The largest and smallest possible exponent values are reserved to represent
things requiring special cases. For example, the infinities, values representing
``not a number'' (such as the result of dividing by zero), and for a way to represent
values that are not normalized. For more information on special cases see \cite{ieee:754}.

\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Floating Point Number Accuracy}
Due to the finite number of bits used to store the value of a floating point
number, it is not possible to represent every one of the infinite values
on the real number line.  The following C programs illustrate this point.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Powers Of Two}
Just like the integer numbers, the powers of two that have bits to represent
them can be represented perfectly\ldots\ as can their sums (provided that the
significand requires no more than 23 bits.)

\listing{powersoftwo.c}{Precise Powers of Two}
\listing{powersoftwo.out}{Output from {\tt powersoftwo.c}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Clean Decimal Numbers}
When dealing with decimal values, you will find that they don't map simply
into binary floating point values.
% (the same holds true for binary integer numbers).

Note how the decimal numbers are not accurately represented as they get larger.
The decimal number on line 10 of \listingRef{cleandecimal.out}
can be perfectly represented in IEEE format.  However, a problem arises in
the 11Th loop iteration.  It is due to the fact that the
binary number can not be represented accurately in IEEE format.  Its least
significant bits were truncated in a best-effort attempt at rounding the value
off in order to fit the value into the bits provided.  This is an example of
{\em low order truncation}.  Once this happens, the value of \verb@x.f@ is
no longer as precise as it could be given more bits in which to save its value.

\listing{cleandecimal.c}{Print Clean Decimal Numbers}
\listing{cleandecimal.out}{Output from {\tt cleandecimal.c}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Accumulation of Error}
These  rounding errors can be exaggerated when the number we multiply
the \verb@x.f@ value by is, itself, something that can not be accurately
represented in IEEE
form.\footnote{Applications requiring accurate decimal values, such as
financial accounting systems, can use a packed-decimal numeric format
to avoid unexpected oddities caused by the use of binary numbers.}
\enote{In a lecture one would show that one tenth is a repeating
non-terminating binary number that gets truncated.  This discussion
should be reproduced here in text form.}

For example, if we multiply our \verb@x.f@ value by $\frac{1}{10}$ each time,
we can never be accurate and we start accumulating errors immediately.

\listing{erroraccumulation.c}{Accumulation of Error}
\listing{erroraccumulation.out}{Output from {\tt erroraccumulation.c}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Reducing Error Accumulation}
In order to use floating point numbers in a program without causing
excessive rounding problems an algorithm can be redesigned such that the
accumulation is eliminated.
This example is similar to the previous one, but this time we recalculate the
desired value from a known-accurate integer value.
Some rounding errors remain present, but they can not accumulate.

\listing{errorcompensation.c}{Accumulation of Error}
\listing{errorcompensation.out}{Output from {\tt erroraccumulation.c}}


\chapter{Incremental Gradient Methods}

\section{Kaczmarz Algorithm and MSC for Linear Problem}
%Now we have the problem with data $\{X^i, Y^i\}_{i=1}^N$ with $X^i \in \mathbb{R}^n$ and $Y^i \in \mathbb{R}$. One way to fit those data is to use linear regression, i.e
%\begin{problem}Find $W^{*} \in \mathbb{R}^{n}$ and $\theta^{*} \in \mathbb{R}$ s.t:
%\begin{equation}
%(W^*,\theta^*) = \mathop{\arg\min}_{W \in \mathbb{R}^{n},\theta \in \mathbb{R}} \sum_{i=1}^N\|X^i \cdot W + \theta - Y^i\|^2.
%\end{equation}
%\end{problem}

%Now, we can note $\tilde{W} = (W, \theta)$ and $\tilde{X}^i = (X^i, 1)$ so, we can reform the problem by:
%\begin{problem}Find $\tilde{W}^{*} \in \mathbb{R}^{n+1}$  s.t:
%\begin{equation}\label{equ:splitform-opt}
%\tilde{W}^* = \mathop{\arg\min}_{\tilde{W \in \mathbb{R}^{n+1}}} \sum_{i=1}^N\|\tilde{X}^i \cdot \tilde{W} - Y^i\|^2.
%\end{equation}
%\end{problem}

%And if we note a matrix $A \in \mathbb{R}^{N \times (n+1)}$ with i-th row of $A$, $A_i = \tilde{X}^i$ and $b = (Y^1, \cdots, Y^N)^{T} \in \mathbb{R}^N$, then the problem can be reformed as:
%\begin{problem}Find $\tilde{W}^{*} \in \mathbb{R}^{n+1}$  s.t:
%\begin{equation}\label{equ:matrixform-opt}
%\tilde{W}^* = \mathop{\arg\min}_{\tilde{W} \in \mathbb{R}^{n+1}} \|A \tilde{W} - b\|^2.
%\end{equation}
%\end{problem}
%And it is equivalent to solve the next normal equation:
%\begin{problem} $\tilde{W}^{*} \in \mathbb{R}^{n+1}$  is the solution of \ref{equ:matrixform-opt} iff:
%\begin{equation}\label{equ:matrixform-equ}
%A^TA \tilde{W}^* = A^Tb.
%\end{equation}
%\end{problem}

%So, it is easy to see that the gradient descent method for \ref{equ:splitform-opt} and \ref{equ:matrixform-opt} is exact the gradient descent method for the equation \ref{equ:matrixform-equ}.
\subsection{Least square problem}
The general least square problem is:
\begin{problem}\label{pro:standerLS}
	Find ${x}^{*} \in \mathbb{R}^{n}$  s.t:
	\begin{equation}\label{equ:leastsquare}
	{x}^* = \mathop{\arg\min}_{{x} \in \mathbb{R}^{n}} \frac{1}{2}\|A {x} - b\|^2 \Leftrightarrow {x}^* = \mathop{\arg\min}_{{x} \in \mathbb{R}^{n}} \sum_{i=1}^n \frac{1}{2}(A_i \cdot {x} - b_i)^2,
	\end{equation}
	with $A \in \mathbb{R}^{m\times n}$ and $\rm{rank}(A) = n$, and $A_i$
	is the i-th row of $A$, but we take it as an column vector.
\end{problem}
If we take $f_i = \frac{1}{2}(A_i \cdot {x} - b_i)^2$, then we can use the general incremental gradient descent method to solve the above problem. So the iterative step is:
\begin{equation}\label{equ:IGDforLS}
x_{t+1} = x_t - \eta_t \nabla f_{i_t}(x_t) = x_t - \eta_t (A_{i_t} \cdot x_t - b_{i_t})A_{i_t}.
\end{equation}
The only difference is the way to choose $i_t$, if we take:
\begin{align}
\mathbb{P}(i_t = s) = \frac{1}{m}, \quad s = 1:m,  \\
i_1, \cdots, i_T \quad \text{are independent},
\end{align}
we will get SGD. If we take
\begin{equation}
i_t \equiv t \mod(m), \quad i_t \in 1:m,
\end{equation}
we will get the cycle incremental gradient method.

\newpage

\subsection{Kaczmarz algorithm}
First, Kaczmarz algorithm is proposed for consistent problem i.e. $Ax
= b$ has solution for $A \in \mathbb{R}^{m\times n}$. So the solution
can be expressed as:

$$
A^T=  (A_1,\ldots, A_m)
$$

$$
A=
\begin{pmatrix}
A_1^T\\
\vdots\\
A_m^T
\end{pmatrix}
$$

\begin{equation}
\bigcap_{i=1}^m \{x \in \mathbb{R}^n ~:~ A_i \cdot x = b_i\}
\end{equation}
Use this structure, a simple idea is to set that $x_{t+1} - x_{t} \in  \{x \in \mathbb{R}^n ~:~ A_i \cdot x = b_i\}$ in \ref{equ:IGDforLS}. This lead to:
\begin{equation}\label{equ:KacamarzAlgo}
x_{t+1} = x_t -  \frac{1}{\|A_{i_t}\|^2}(A_{i_t} \cdot x_t - b_{i_t})A_{i_t}.
\end{equation}

\begin{eqnarray*}
(A_{i_t} \cdot x_t - b_{i_t})A_{i_t}
&=&A_{i_t} (A_{i_t}^T x_t - b_{i_t})
=A_{i_t} (Ax_t -b)\cdot e_{i_t}=A_{i_t} e_{i_t}^T(Ax_t -b)\\
& =& (e_{i_t}A_{i_t}^T)^T(Ax_t -b)
=(e_{i_t}e_{i_t}^TA)^T(Ax_t -b) \\
&=&A^Te_{i_t}e_{i_t}^T(Ax_t -b)
=[e_{i_t}^T(Ax_t -b)] A^Te_{i_t}
\end{eqnarray*}

or
$$
x_{t+1} - x_{t} \in  {\rm range}(A^Te_t)
$$
Recall
$$
y_{t+1} - y_{t} \in  {\rm range}(e_t)
$$
and
$$
(y_{t+1} , e_t)_{AA^T} = (b,e_t)
$$
$$
(x_{t+1}, A^Te_t) = (b,e_t)=(x,A^Te_t)
$$
If
$$
x_{t+1} - x_{t}=\eta=\alpha A^Te_t
$$
then
$$
(x_{t}+\alpha A^Te_t, A^Te_t) = (b,e_t)
$$
$$
(A(x_{t}+\eta), e_t) = (b,e_t)
$$
or, find $\eta\in V_t$ such that
$$
(A(x_{t}+\eta)-b, e_t) = 0.
$$

and
$$
\alpha=\frac{1}{(A^Te_t, A^Te_t)}(b-Ax_t,e_t)
$$
Let
$$
V_t={\rm range}(A^Te_t),
$$
and, consider the $\ell^2$ projection
$$
Q_t:  \mathbb R^m\mapsto V_t
$$
then
$$
Q_t(x_{t+1} - x)=0.
$$

Another way(in Prof. Xu's paper about random MSC) to connect Kaczmarz and MSC is to solve:
\begin{equation}\label{norm}
AA^T y = b,\quad x=A^Ty
\end{equation}
then we use G-S for the above equation, and then multiply $A^T$ for the iteration process and get the Kaczmarz process.

Apply Gauss-Seidel to \eqref{norm},
$$
y_{t+1} = y_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(AA^Ty_t - b) e_{i_t}.
$$
and
$$
A^Ty_{t+1} =A^T y_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(AA^Ty_t - b) A^Te_{i_t}.
$$
$$
x_{t+1} =x_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(Ax_t - b) A^Te_{i_t}.
$$

\subsection{Relationship between IGD and Kaczmarz}
Consider the least square problem
$$
\|Ax-b\|=\min.
$$
In some case, the solution satisfies
$$
A^TAx^*=A^Tb.
$$
If we write
$$
f(x)=\|Ax-b\|^2=\sum_{i=1}^mf_i(x)
$$
Then the IGD algorithm for the above form is:
\begin{equation}\label{equ:IGDforLS}
x_{t+1} = x_t - \eta_t \nabla f_{i_t}(x_t) = x_t - \eta_t (A_{i_t} \cdot x_t - b_{i_t})A_{i_t}.
\end{equation}
but the Kaczmarz algorithm for this problem is:
\begin{equation}\label{equ:KacamarzAlgo}
x_{t+1} = x_t -  \frac{1}{\|A_{i_t}\|^2}(A_{i_t} \cdot x_t - b_{i_t})A_{i_t}.
\end{equation}

The IGD for the above problem is equivalent to apply damped
Gauss-Seidel method to the normal equation
\begin{equation}
\label{normA}
AA^Ty=b
\end{equation}
\begin{enumerate}
	\item If the above problem is consistent, then the IGD method would
	converge to a solution of the above problem.  But the solution of
	equation \eqref{normA} may not be the least square solution???

	\item If the above problem \eqref{normA} is inconsistent, if the IGD
	converges to something, say $\tilde x$, then
	$$
	Ax-b\perp \mathbb R^m?
	$$
	\item But we know SGD (with diminishing learning rate) always converge
	to the original least square?
\end{enumerate}

{\bf{Juncai's simple answers}:}
\begin{enumerate}
	\item If the above problem is consistent, i.e. $Ax = b$ has solution( this solution will be unique if $\rm{rank}(A) = n$), then it is not sure to say what $y_t$ for \eqref{normA} will converges to, but we can make sure that $A^T y_t$ will converges to $x^*$ i.e. the solution for $Ax = b$.

	\item If \eqref{normA} is inconsistent, then $Ax = b$ is also inconsistent. So the SGD with diminishing stepsize will convergence to the solution for $A^TA x = A^Tb$, and the means that a special damped random G-S type MSC will convergence into a high dimension affine space like $y^* + \rm{ker}(A^T)$ such that $A^T y^* = x^*$.

	\item For inconsistent case( if $b \neq 0$, the inconsistent for $Ax=b$ and $AA^Ty = b$ is equivalent). The Kaczmarz algorithm and the IGD for $\min \frac{1}{2}\|Ax - b\|^2$ is different with the ``special" choice for the stepsize $\eta_t$, is should be a random variable! Because in Kaczmarz algorithm, it is same to choose
	\begin{equation}
	\eta_t = \frac{1}{\|A_{i_t}\|^2},
	\end{equation}
	in the IGD algorithms for  $\min \frac{1}{2}\|Ax - b\|^2$. If we didn't want such special choice for $\eta_t$, the Kaczmarz algorithm is equal to choose $\eta_t = 1$ for
	\begin{equation}
	\min \frac{1}{2}\|D^{-1}(Ax - b)\|^2 = \min \frac{1}{2} \|Ax-b\|^2_{\rm{diag}(AA^T)^{-1}},
	\end{equation}
	where $D = \rm{diag}(AA^T)^{\frac{1}{2}}$.

	\item For inconsistent system, use $\min \frac{1}{2}\|Ax - b\|^2$ as example, the stander SGD method fort this problem will not converges without diminishing stepsize because:
	\begin{equation}
	\mathbb{E}\|x_{t+1} - x^*\|^2 \ge (1 - 2 \lambda_{max}(A^TA) \eta_t) \mathbb{E}\|x_t - x^*\|^2 + \eta_t^2 (\min_i \|A_i\|^2)\|Ax^* - b\|^2.
	\end{equation}
	And it seems that the above inequality can be extended to general convex problem see

	\item If $AA^T y = b$ is inconsistent, then the damped G-S without diminishing(Kaczmarz for Ax = b)  cannot converges to anything. Because, we can can assume that will converges to some thing like $y^*$, then by taking limit in both side of the next damped G-S
	\begin{equation}
	\lim_{t} y_{t+1} = \lim_t (y_t -  \frac{1}{\|A_{i_t}\|^2}e_{i_t}^T(AA^Ty_t - b) e_{i_t}).
	\end{equation}
	this will lead to that
	$$
	AA^Ty^* - b = 0,
	$$
	this is contrary to the inconsistent assumption. Here, even we multiply $A^T$ in both side, we can know that $A^T y_t$ will converges to nothing. We can use the same idea to proof this, if it converges, then
	$$
	Ax^* - b = 0,
	$$
	and because of the fact that $A x = b$ is also inconsistent if $AA^T y = b$ is inconsistent.
\end{enumerate}

\subsection{A special case: A is SPD}
If $A \in \mathbb{R}^n$ is a SPD matrix, the next problem:
\begin{problem}\label{pro:standerLS}
	Find ${x}^{*} \in \mathbb{R}^{n}$  s.t:
	\begin{equation}\label{equ:spd-opt-equ}
	{x}^* = \mathop{\arg\min}_{{x} \in \mathbb{R}^{n}} \frac{1}{2}\|A {x} - b\|^2 \Leftrightarrow {x}^* = \mathop{\arg\min}_{{x} \in \mathbb{R}^{n}} \sum_{i=1}^n\frac{1}{2\|A_i\|^2}(A_i \cdot {x} - b_i)^2 \Leftrightarrow Ax^{*} = b
	\end{equation}
	here $A_i$ is the i-th row of $A$, but we take it as an column vector.
\end{problem}
So the incremental gradient method  with $f_i = \frac{1}{2\|A_i\|^2}(A_i \cdot {x} - b_i)^2$ for a single step can be taken as:
\begin{equation}
x_{t+1} = x_t - \eta_t \nabla f_{i_t}(x_t) = x_t - \eta_t \frac{(A_{i_t} \cdot x_t - b_{i_t})}{\|A_{i_t}\|^2}A_{i_t}.
\end{equation}

Then we can prove that, incremental gradient method for both cycle or random type is linear convergence. The crucial point is $Ax^* = b$, so we will have
\begin{equation}
\nabla f_i(x) = \frac{A_i \cdot (x - x^*)}{\|A_i\|^2}A_i.
\end{equation}

If we take $\eta_t = 1$, then this method is the Kaczmarz algorithm to solve least squares problem.

\subsubsection{MSC for linear problems}
Now we would like to discuss the relation between the Karczmarz
algorithm above and the method of subspace correction for solving $Ax
= b$.

If $A \in \mathbb{R}^{n \times n}$ is a SPD matrix,  we now consider the subspace correction method to solve
\begin{equation}\label{equ:SPD}
Ax = b,
\end{equation}
with the space decomposition
\begin{equation}
\mathbb{R}^n = \rm{span}\{A_i, i = 1, \cdots, n\},
\end{equation}
where $A_i$ is the same as above.

For the subspace correction method, the correction formula for subspace $V_i = \rm{span}\{A_i\}$ is:
\begin{equation}
x_{new} = x_{old} + I_i \bm{A}_i^{-1} Q_i r(x_{old}),
\end{equation}
where $P_i$ and $I_i$ are the projection(restriction) and interpolation operator w.r.t subspace $V_i$, and $\bm{A}_i$ is the inverse of the restricted problem in $V_i$, $r(x_{old}) = b - Ax_{old}$ is the residual.

So we can investigate a single step in MSC for this equation under this space decomposition as:
\begin{equation}
x_{k+1} = x_k - \frac{A_i^T(Ax_k - b)}{A_i^T A A_i}A_i,
\end{equation}
here we can rewrite it as:
\begin{equation}\label{equ:MSC}
x_{k+1} = x_k - \frac{(A_i, x_k - x^*)_A}{\|A_i\|^2_A}A_i.
\end{equation}

So the relationship between those two methods is that, we need to change the inner product. For the stander problem \ref{pro:standerLS},
\begin{equation}
f_i = \frac{(A_i \cdot x - b_i)^2}{2\|A_i\|^2} = \frac{(A_i, x - x^*)^2}{2\|A_i\|^2},
\end{equation}
and this is equivalent to solve $Ax = b$, but in fact this is also equivalent to define $f_i$ as:
\begin{equation}\label{equ:newinnerpro}
f_i = \frac{(A_i, x - x^*)_A^2}{2\|A_i\|_A^2},
\end{equation}
thus we have:
\begin{equation}
\nabla f_i = \frac{(A_i, x_k - x^*)_A}{\|A_i\|^2_A} A A_i.
\end{equation}

So we have, the incremental gradient method with $f_i$ defined by \ref{equ:newinnerpro} is have the next relation with the MSC \ref{equ:MSC}.
\begin{equation}
-A^{-1} \nabla f_i(x) = I_i \bm{A}_i^{-1}Q_i r(x),
\end{equation}

\begin{lemma}
	Steepest descent direction under the inner product of $(\cdot, \cdot)_A$ for $f_i$ is $- A^{-1} \nabla f_i$.
\end{lemma}
\begin{proof}
	If we assume the next descent direction is $d$, for the first order approximation for $f_i$ we have:
	\begin{equation}
	|(\nabla f_i, d)| = |(A^{-1} \nabla f_i, A d)| = |(A^{-1} \nabla f_i, d)_A| \le \|A^{-1} \nabla f_i\|_A \|d\|_A.
	\end{equation}
	So, under the inner product $(\cdot, \cdot)_A$ to get equality in above inequality, we just need $ d $ and $A^{-1} \nabla f_i$ is parallel.
\end{proof}

All in all, we have the next relation between MSC for $Ax = b$ with decomposition $\mathbb{R}^n = \rm{span}\{A_i, i = 1, \cdots, n\}$and gradient or incremental gradient method for $\min \sum_i f_i$ with
$f_i = \frac{(A_i, x - x^*)_A^2}{2\|A_i\|_A^2}$ and descent under the inner production of $(\cdot, \cdot)_A$.
\begin{itemize}
	\item The PSC for $Ax = b$ is equal to the gradient descent for $\min \sum_i f_i$.
	\item The SSC for $Ax = b$ is equal to the cycle incremental gradient descent for $\min \sum_i f_i$.
	\item The random SSC for $Ax = b$ is equal to the SGD for $\min \sum_i f_i$.
\end{itemize}

\subsubsection{convergence analysis}
Just use the stander MSC analysis and the newly proposed random MSC theory by Prof. Xu, we can analysis the random SSC as:
\begin{equation}
\mathbb{E} \|x_{t+1} - x^*\|_A^2 \le (1 - \frac{\delta_t}{n})\mathbb{E} \|x_{t} - x^*\|_A^2.
\end{equation}
For the SGD for $\sum_i f_i$ under metric $(\cdot,\cdot)_A$, to analysis its convergence performance, we should also use the inner product $(\cdot,\cdot)_A$, so this can be covered by the above result naturally.

\section{SGD with small step size and weak convergence for linear case}
Now we start our problem with solving
\begin{equation}
\min \frac{1}{2}\|Ax - b\|^2,
\end{equation}
with $A \in \mathbb{R}^{m \times n}$ and $\rm{rank}(A) = n$.
Here we note $f_i = \frac{1}{2}(A_i^T x - b_i)$ with $A_i$ is the i-th row of A. So we write the general SGD for the above problem as:
\begin{algorithm}\caption{General SGD}
	\label{alg:gSGD}
	\begin{equation}\label{equ:sgd-iteration}
	x_{t+1} = x_{t} - \eta_t \nabla f_{i_t}(x_t), \quad t = 0:T,
	\end{equation}
	\begin{equation}
	\mathbb{P}(i_t = s) = p_i, \quad s = 1:m,
	\end{equation}
	\begin{equation}
	i_1, \cdots, i_T \quad \text{are independent}.
	\end{equation}
\end{algorithm}

So now, we would like to discuss the relation for tanditional SGD and Kaczmarz algorithm under the general SGD framework and weak convergence property for inconsistent case.

\subsection{Kaczmarz algorithm}
If we take:
\begin{equation}
\eta_t = \frac{1}{\|A_{i_t}\|^2},
\end{equation}
as an {\bf random variable} without diminishing, then this is the general random Kaczmarz.

\subsection{Weak convergence}
Here the result is:
\begin{theorem}
	For those next two choice of $\eta_t$ and $p_i$,
	\begin{itemize}
		\item Traditional SGD for LS:
		\begin{equation}
		\eta_t = \eta \quad p_i = \frac{1}{m}.
		\end{equation}
		\item Random damped Kaczmarz for LS:
		\begin{equation}
		\eta_t = \frac{\eta}{\|A_{i_t}\|^2} \quad p_i = \frac{\|A_i\|^2}{\|A\|_F^2}.
		\end{equation}
	\end{itemize}
	We have the weak convergence without diminishing step size to $x^*$ as the solution of $\min \frac{1}{2}\|Ax - b\|^2$, i.e.
	\begin{equation}
	\lim_{t \to \infty}\|\mathbb{E}x_t - x^*\|^2  = 0.
	\end{equation}
\end{theorem}

\begin{proof}
	For the first case:
	\begin{align}
	\mathbb{E}( x_{t+1}| x_t) &= x_t - \mathbb{E}(\eta_t \nabla f_{i_t}(x_t) | x_t) \\
	&= x_t - \frac{\eta}{m}A^T(b - Ax_t)
	\end{align}
	Take $\mathbb{E}_{x_t}$ on both side of the above equation, we have:
	\begin{align}
	\mathbb{E}_{x_t}\mathbb{E}( x_{t+1}| x_t) &= \mathbb{E}x_{t+1}
	= \mathbb{E}x_t - \frac{\eta}{m}A^T(A\mathbb{E}x_t - b) \\
	&= \mathbb{E}x_t - \frac{\eta}{m}A^TA( \mathbb{E}x_t - x^*)
	\end{align}
	So we have:
	\begin{align}
	\|\mathbb{E}x_{t+1} - x^*\|
	= \|(I - \frac{\eta}{m}A^TA)\| \|(\mathbb{E}x_t - x^*)\|
	\end{align}
	Similarly for the second case, we have:
	\begin{align}
	\|\mathbb{E}x_{t+1} - x^*\|
	= \|(I - \frac{\eta}{\|A\|_F^2}A^TA)\| \|(\mathbb{E}x_t - x^*)\|
	\end{align}
\end{proof}

\begin{theorem}
	For the next choice of $\eta_t$ and $p_i$,
	\begin{equation}
	\eta_t = \eta \quad p_i = \frac{\|A_i\|^2}{\|A\|_F^2}.
	\end{equation}
	this weak convergence to
	\begin{equation}
	\frac{1}{2}\|Ax - b\|^2_{D}
	\end{equation}
	with
	\begin{align}
	D = \rm{diag}(\|A_1\|^2, \ldots, \|A_m\|^2) = \rm{diag}(AA^T).
	\end{align}

	And for the next choice of $\eta_t$ and $p_i$,
	\begin{equation}
	\eta_t = \frac{\eta}{\|A_{i_t}\|^2} \quad p_i = \frac{1}{m}.
	\end{equation}
	this weak convergence to
	\begin{equation}
	\frac{1}{2}\|Ax - b\|^2_{G}
	\end{equation}
	with
	\begin{align}
	G = \rm{diag}(\|A_1\|^{-1}, \ldots, \|A_m\|^{-1}) = D^{-1}.
	\end{align}
\end{theorem}
This means that, even the general SGD method for $\min \frac{1}{2}\|Ax-b\|^2$ with constant step size and non-uniform will weak converges to a solution for another problem. The similar situation happens to damped Kaczmarz. {\bf All this happened because of the face that
	\begin{equation}
	\mathbb{E} \nabla f_i \neq \nabla f.
	\end{equation}}

\section{Exact and nearly exact interpolation case}

For general deep learning problem, the object function is like:
\begin{problem}
	Find $W^* \in \mathbb{R}^n$ such that:
	\begin{equation}
	W^* \in \mathop{\arg\min}_{W} \frac{1}{m}\sum_{i=1}^mf_i(W) = \mathop{\arg\min}_{W} \frac{1}{m}\sum_{i=1}^m \|f(W;X^i) - Y^i\|^2.
	\end{equation}
\end{problem}

Interpolation means that:
\begin{equation}
\min_{W} \frac{1}{m}\sum_{i=1}^mf_i(W) = 0,
\end{equation}
with $f_i(W) =  \|f(W;X^i) - Y^i\|^2$.For simple expression, we use $f_i(x)$ not $f_i(W)$.

As we know, even for general convex problem, SGD method connot to be proved convergence with small constant step size. But here, we extend assumptions with similar to exact interpolation case like:
\begin{assumption}\label{assum:ideal}
	$f_i \ge 0$ satisfies the next conditions:
	\begin{itemize}
		\item $ f$ satisfies the $\lambda$-strong convex property.
		\item $\nabla f_i$ is Lipschitz continuous with constant $H$.
		\item We can interpolate every point exactly, i.e.
		\begin{equation}
		x^* = \mathop{\arg\min}_{x} \sum_{i=1}^mf_i(x)  \in \mathop{\arg\min}_{x} f_i(x), \quad \forall i  =  1:m.
		\end{equation}
	\end{itemize}
\end{assumption}

So, in some degree this means that:
\begin{equation}
x^* = \cap_{i=1}^m \mathop{\arg\min}_{x} f_i(x).
\end{equation}
and this can be seen as the generalization of the consistence in linear regression problem.
\begin{theorem}
	Under the assumption \ref{assum:ideal}, we can prove the SGD method \ref{alg:SGD} with small constant step size $\eta$ has the linear convergence, i.e
	\begin{equation}
	\mathbb{E}\|x_t - x^*\|^2 \le \delta^t \|x_0 - x^*\|^2,
	\end{equation}
	for some constant $ 0 < \delta < 1$.
\end{theorem}

\begin{proof}
	For SGD method, we have:
	\begin{align}
	\mathbb{E} \|x_{t+1} - x^*\|^2 &\le \mathbb{E} \|x_t - x^*\|^2  - 2 \eta_t \mathbb{E} (\nabla f_{i_t}(x_t) \cdot (x_t - x^*)) + \eta_t^2 \mathbb{E} \|\nabla f_{i_t}(x_t)\|^2\\
	&= \mathbb{E} \|x_t - x^*\|^2  - 2 \eta_t \mathbb{E} (\nabla f(x_t) \cdot (x_t - x^*))  + \eta_t^2 \mathbb{E} \|\nabla f_{i_t}(x_t)\|^2 \\
	&\le \mathbb{E} \|x_t - x^*\|^2 - 2\eta_t \lambda \mathbb{E}\|x_t - x^*\|^2 + \eta_t^2 \mathbb{E}\|\nabla f_{i_t}(x_t) - \nabla f_{i_t}(x^*)\|^2 \\
	&\le (1- 2\eta_t \lambda + \eta_t^2 H^2) \mathbb{E}\|x_t - x^*\|^2.
	\end{align}
	So if
	\begin{equation}
	\eta_t < \frac{2\lambda}{H^2},
	\end{equation}
	then we have $1- 2\eta_t \lambda + \eta_t^2 H^2 = \delta < 1$, which finishes this proof.
\end{proof}

This result can be taken as an extension for the convergence result for random Kacamarz algorithm for consistent system.

In many cases, the third assumption in assumption \ref{assum:ideal} is hard to satisfy. But in many applications, we found that, we don't need real diminishing stepsize to preserve the convergence, we just need some even not very small stepsize or decrease it just few steps to make $\sum_i f_i$ becomes convergence or just very small oscillation in the last of iterations.  Here we try to gives another reasonable assumption to explain why those happens:
\begin{assumption}\label{assum:nearideal}
	$f_i \ge 0$ satisfies the next conditions:
	\begin{itemize}
		\item $ f$ satisfies the $\lambda$-strong convex property.
		\item $\nabla f_i$ is Lipschitz continuous with constant $H$.
		\item For the minimizer $x^*$, we have the near consistent property:
		\begin{equation}
		\sum_{i}^m\frac{1}{m}\|\nabla f_i(x^*)\| \le \epsilon,
		\end{equation}
		here $\epsilon$ is a small positive constant.
	\end{itemize}
\end{assumption}

\begin{theorem}
	Under the assumption \ref{assum:nearideal}, we can prove the SGD method \ref{alg:SGD} with step size $\eta_t$ has the next properties, i.e
	\begin{equation}
	\mathbb{E}\|x_t - x^*\|^2 \le (1- 2\eta_t \lambda + \eta_t^2 H^2) \mathbb{E} \|x_{t-1} - x^*\|^2 + (\eta_t \epsilon)^2.
	\end{equation}
\end{theorem}
\begin{proof}
	For SGD method, we have:
	\begin{align}
	\mathbb{E} \|x_{t+1} - x^*\|^2 &\le \mathbb{E} \|x_t - x^*\|^2  - 2 \eta_t \mathbb{E} (\nabla f_{i_t}(x_t) \cdot (x_t - x^*)) + \eta_t^2 \mathbb{E} \|\nabla f_{i_t}(x_t)\|^2\\
	&= \mathbb{E} \|x_t - x^*\|^2  - 2 \eta_t \mathbb{E} (\nabla f(x_t) \cdot (x_t - x^*))  + \eta_t^2 \mathbb{E} \|\nabla f_{i_t}(x_t)\|^2 \\
	&\le \mathbb{E} \|x_t - x^*\|^2 - 2\eta_t \lambda \mathbb{E}\|x_t - x^*\|^2 + \eta_t^2 \mathbb{E}\|\nabla f_{i_t}(x_t) - \nabla f_{i_t}(x^*)\|^2 +  (\eta_t \epsilon)^2\\
	&\le (1- 2\eta_t \lambda + \eta_t^2 H^2) \mathbb{E}\|x_t - x^*\|^2 + (\eta_t \epsilon)^2.
	\end{align}
\end{proof}

By using supermartingale convergence theorem, fro diminishing stepsize one can prove that:
\begin{theorem}
	Take $\eta_t$ as diminishing stepsize with $\sum_t \eta_t = \infty$ and $\sum_t \eta^2_t < \infty$, then
	$\mathbb{E}\|x_t - x^*\|$ converges.
\end{theorem}
But it seems hard to find the convergence rate.  And for constant stepsize:
\begin{theorem}
	For constant stepsize, if $\eta_t = \eta$ is a constant and small enough, then
	\begin{equation}
	\mathbb{E} \|x_{t} - x^*\|^2 \le \delta^t \|x_0 - x^*\| + \frac{1 - \delta^t}{1- \delta}(\eta\epsilon)^2.
	\end{equation}
\end{theorem}

\subsection{Inconsistent case}
Like the inconsistent case in linear regression, we can have the next assumption:
\begin{assumption}\label{assum:inconsistent}
	$f_i \ge 0$ satisfies the next conditions:
	\begin{itemize}
		\item $ f$ satisfies the $\lambda$-strong convex property.
		\item $\nabla f_i$ is Lipschitz continuous with constant $H$.
		\item We have the next inconsistent property: there exist a $\delta > 0$ such that
		\begin{equation}
		\min_{x } \sum_i^m \|\nabla f_i(x)\|^2 \ge \delta.
		\end{equation}
	\end{itemize}
	In fact, the last assumption is exactly the inverse of the consistent case.
\end{assumption}
Then we can prove that, without the diminishing stepsize, the problem cannot converges.
\begin{theorem}
	We have the next estimate for the iteration process:
	\begin{equation}
	\mathbb{E}\|x_{t+1} - x^*\|^2 \ge (1 - 2\eta_t H)\mathbb{E} \|x_t - x^*\|^2 + \eta_t^2 \frac{\delta}{m}.
	\end{equation}
\end{theorem}

\section{Deep learning cases}
\subsection{``consistent" cases: separable!}
As we will see, the third assumption in \ref{assum:ideal} makes the key role in proving the convergence. Here we would like to show that, in some cases in deep learning, this will happen. Now we can consider a simple example as one dimension function interpolate with artificial neural network. So we have the data set $(x^i, y^i)$ with $i = 1, \cdots, m$. As we know, if we use the ReLU function
\begin{equation}
\rm{ReLU}(x) = \max\{0,x\},
\end{equation}
the one-dimension artificial neural network can cover the piecewise linear function. So for a general ANN model from $\mathbb{R}$ to $\mathbb{R}$, $f(W;x)$ can fixed all data $(x_i, y_i)$ exactly. Thus, we have
\begin{equation}
0 \le \min_W \sum_i^m f_i = 0 = \sum_i^m \min_W (f(W;x^i) - y^i)^2,
\end{equation}
so this satisfy the third assumption in \ref{assum:ideal}.

\subsection{near consistent case}
Here we can talk about the near consistent assumption in assumption \ref{assum:nearideal}. Because of the powerful expression power for ANN, even you cannot interpolate those data exactly, we can use the next analysis to show some reasonableness for our assumption.

To be added...

%When we consider about ANN for classification problem, first we would like to divide our data into $C$ classes:
%\begin{equation}
%S_k := \{i :Y_i = e_k\}, k = 1,\cdots,C.
%\end{equation}
%Let us define $\hat{x}^k$ by
%\begin{equation}
%\hat{x}^k \in \mathop{\arg\min}_{x} f_i
%\end{equation}

Here in real classification application like ImageNet problem, because even the Top 5 error is not zero, which means that $f_i(x^*)$ cannot be very close to $\min f_i$, especially for those $i$ that is classified into wrong classes.

\section{SGD as smoother}
%\subsection{GD and G-S}
%An interesting result is that, in fact the convergence result for both traditional subspace correction(Jacobi or G-S) and the gradient descent method, they both converges linearly like:
%\begin{equation}
%\|x_k - x^*\|_{*}^2 \le \delta^k \|x_0 - x^*\|_{*}^2.
%\end{equation}
%So the general behaviour for GD likes G-S very much, the most important properties is that: it convergence very fast at beginning and then slow. Here is a typical performance for GD in small deep learning problem:
%\begin{figure}[!htb]
%	\center{\includegraphics[width=8cm] {NN_GD.png}}
%	\caption{Gradient descent for a small deep learning model for MNISET}
%	\label{Kernels}
%\end{figure}

\subsection{SGD and GD}
The most important properties for SGD is that if you take expectation for SGD then you get SD. So in some degree, SGD may works like GD, but the convergence result for SGD is not good as GD, because now we only have:
\begin{equation}
\mathbb{E}\|x_k - x^*\|^2 \le \frac{M}{k}.
\end{equation}
However, in recent paper by using stochastic differential equation to analysis SGD, they show a special case with
\begin{equation}
\min_{x \in \mathbb{R}} f(x) = f_1(x) + f_2(x),
\end{equation}
with $f_1(x) = (x - a)^2$ and $f_2(x) = (x - b)^2$. We know that $\alpha = \frac{a+b}{2}$ is the solution for this problem. If we take the stepsize for SGD(learning rate) as a constant $\eta$, so we get an approximation SDE system:
\begin{align}
d X_t = -2(X_t - \alpha )dt + 2\sqrt{\eta}dB_t, \quad X_0 = x_0.
\end{align}
So we can solve the above system and get
\begin{align}
X_t \sim \mathcal{N}(\alpha + (x_0 - \alpha)e^{-2t}, \eta(1 - e^{-4t})).
\end{align}
So in the beginning, the variance is small, thus the behaviour of SGD resembles GD and has the same convergence rate as GD. {\bf However, as the SGD requires only one evaluation of the gradient at each iteration, it is exactly twice as fast as GD in the beginning.}

So we think that, GD often works like G-S very much, as converges fast in the beginning. And SGD with small constant learning rate in the first few steps has the same performance for GD and even faster w.r.t the computational cost. So, SGD can be a good smoother in some sense.


\section{$p$-adic integers}
Let $p$ be a prime number. To construct the $p$-adic integers, we have different ways of definition. We are going to present all of them.
\subsection{Topological approach}
Fix $p$ as above then $\mathfrak{p} := \left<p\right> = p\zz$ is a prime ideal in $\zz$ (moreover, it is maximal). We are going to use the notion of linear topological rings.
\begin{defi}
Let $(R,+,0,-,\cdot,1)$ be a ring with topology $\tau$. We call $R$ a topological ring if $+, \cdot : R^2 \longrightarrow R$ are continuous wrt. product topology. We call a topological ring $R$ linear topological if there is a open neighborhood basis of zero wrt. $\tau$:
$$\beta(0) := \{U \in \tau : U\ \trm{open},\ 0 \in U\}$$
\end{defi}
\bsp Clearly, $\zz$ is a ring which is a topological ring wrt. discrete topology $\tau_\zz = \mathcal{P}(\zz) = \{A \subset \zz\}$. More interestingly, let us consider the field of real numbers $\rz$ with standard topology $\tau_{\mathrm{stand}}$ with
$$\tau_{\mathrm{stand}} = \left<B(y,\eps) := \{x \in \rz : d(x,y) < \eps\} : y \in \rz, \eps \in \rz_{+}\right>.$$ To show continuity of addition we first prove that negation $- : \rz \longrightarrow \rz, x \longmapsto -x$ is continuous:
$$\bao{rclcl}
-^{-1} B(x,\eps) &=& \{y \in \rz : d(-y,x) < \eps\} &=& \{y \in \rz : -\eps < -(x + y) < \eps\}\\
&&&&\\
 &=& \{y \in \rz : \eps < y - (-x) < \eps\} &=&  \{y \in \rz : d(y,-x) < \eps\}\\
 &&&&\\ &=& B(-x,\eps).\\
 \ea$$
Thus the preimage is indeed a generator for the standard topology on $\rz$ for all generators $B(x,\eps)$. We claim the it suffice to show that subtraction $- : \rz^2 \longrightarrow \rz$ is continuous at zero. Clearly:
$$\bao{rcl}
-^{-1} (B(0,\eps)) &=& \{(x,y) \in \rz^2 : d(x - y, 0) < \eps\}\\&&\\ &=& \{(x,y) : -\eps < x - y < \eps\}\\
&&\\
& =& \bigcup_{x \in \rz,\delta \in (0,\eps)} B(x,\delta) \times B(-x,\eps - \delta)\\&&\\
&=& \bigcup_{y \in \rz,\ \delta \in (0,\eps)} B(-y,\eps - \delta) \times B(y,\delta)\\
\ea$$
is an open element in the product topology proving our previous claim. Since all elements $x \in \rz$ have a neighborhood system $B(x,\eps) = x + B(0,\eps)$ we may apply the last proof and see each preimage is of type
$$\bigcup_{y \in \rz,\ \delta \in (0,\eps)} B(y,\delta) \times B(x - y, \eps - \delta).$$
Analogously, we claim that $^{-1} : \rz^\times \longrightarrow \rz^\times$ is continuous for all $x \in \rz^\times$ and $B(x,\eps)$. For simplicity, we assume $0 \notin B(x,\eps)$ (otherwise we may decompose $B(x,\eps)$ to $B(x,\eps) \cap \rz_{<0}$ and $B(x,\eps) \cap \rz_{>0}$ and proceed with each component).
%We have $B(x^{-1},\eps/x)$ as preimage being clearly an open element in $\tau_{\mathrm{stand}}$.
$$\bao{rcl}
(1/)^{-1}(B(x,\eps)) &=& \{y \in \rz^\times : d(y^{-1},x) < \eps\}\\
&&\\
&=& \{y : \rz^\times : x - \eps < 1/y < x + \eps\}\\
&&\\
&=& \left\{y : \rz^\times : \frac{1}{x - \eps} > y > \frac{1}{x + \eps}\right\}\\
&&\\
&=& B(x/(2(x^2 - \eps^2)), \eps/(x^2 - \eps^2))\\
\ea$$
clearly being open in $\tau_{\mathrm{stand}}$. Now we will show that the division map $/ : \left(\rz^\times\right)^2 \longrightarrow \rz^\times, (x,y) \longmapsto x/y$ is continuous in the product topology at one.
$$\bao{rcl}
/^{-1}(B(1,\eps)) &=& \left\{(x,y) \in \left(\rz^\times\right)^2 : d(x/y, 1) < \eps\right\}\\
&&\\
&=& \{(x,y) : 1 - \eps < x/y < 1 + \eps\}\\
&&\\
&=& \left\{(x,y) \in \left(\rz_{>0}\right)^2 : \frac{x}{1 - \eps} > y > \frac{x}{1 + \eps}\right\}\\&&\\&& \cup \left\{(x,y) \in \left(\rz_{<0}\right)^2: \frac{x}{1 - \eps} < y < \frac{x}{1 + \eps}\right\}\\
&&\\
&=& B(x,\delta) \times B\left(\frac{x}{2(1 - \eps^2)}, \frac{\eps x}{1 - \eps^2}\right) \cup \{\mathrm{bollocks}\}\\
\ea$$
%The product topology is $\tau_{\rz \times \rz} = \left<U \times \rz, \rz \times V: U, V \in \tau\right>$. Then for $B(y,\eps)$, we find an $0 < \delta < \eps$ such that
%$$\bao{rcl}
%+^{-1}(B(y,\eps)) &=& \left\{(x,x') \in \rz^2 : d(x + x', y) < \eps\right\}\\
%&&\\
% &=& \bigcup_{\delta \in (0,\eps)} \left\{(z,z') \in \rz^2 : \forall x \in \rz,\ z \in B(x,\delta) \wedge z' \in B(y - x, \eps - \delta)\right\}\\
%&&\\
% &=:& \bigcup_{\substack{x \in \rz\\\delta \in (0,\eps)\\}} B(x,\delta) \times B(y - x,\eps - \delta)\\
% &&\\
% &=& \bigcup_{\delta \in (0,\eps)}\left\{(z,z') \in \rz^2 : \forall x' \in \rz,\ z' \in B(x',\delta) \wedge z \in B(y - x', \eps - \delta)\right\}\\
% &&\\
% &=& \bigcup_{\substack{x' \in \rz\\\delta \in (0,\eps)\\}} B(y - x', \eps - \delta) \times B(x',\delta)\\
% \ea$$
%Without loss of generality, each subset $B(x,\delta) \times B(y - x, \eps - \delta)$, indexed by $x \in \rz$ and $\delta \in (0,\eps)$,  is contained in a well chosen family of ascending neighborhoods of $x$ and $y - x$. Therefore, the preimages are open in the product topology. For the continuity of the multiplication, we first distinguish the unit group $(\rz^\times,\cdot,1)$ and then complete with zero. Thus, let $y \in \rz^\times$ and the other parameter as above.
%$$\bao{rcl}
%\cdot^{-1}(B(y,\eps)) &=& \{(x,x') \in \rz^2 : d(x x', y) < \eps\}\\
%&&\\
%&=& \bigcup_{\delta \in (0, \min(|x|, \eps))} \{(z,z') : \forall x \in \rz^\times,\ z \in B(x,\delta), z' \in \underbrace{B(y/(x - \delta), \eps/(x - \delta))}_{U_{-,\delta}}\}\\
%&&\\
%&& \cap \{(z,z') : \forall x \in \rz^\times,\ z \in B(x,\delta), z' \in \underbrace{B(y/(x + \delta, \eps/(x + \delta))}_{U_{+,\delta}}\}\\
%&&\\
%&=& \bigcup_{\substack{x \in \rz^\times\\0 < \delta < \min(|x|, \eps)}} B(x,\delta) \times (U_{-,\delta} \cap U_{+,\delta})
%\ea$$
%By symmetry of the metric, we may reverse both factors. This shows $(R^\times, \cdot, 1, ^{-1}, \tau)$ is a topological group.\\
\indent Next, we construct a neighborhood basis of zero for any commutative ring with one and an ideal $0 \subsetneq I \subsetneq R$. Pick $\beta(0) := \{I^i \subset R : i \geq 1\}$ then this defines a neighborhood system of zero as
$$I \supsetneq I^2 \supsetneq \ldots \supsetneq I^i \supsetneq I^{i+1} \supsetneq \ldots \supsetneq 0,$$
i.e. each ideal contains the zero module and, therefore, the zero element.


\subsection{Vector bundles}

A vector bundle consists of a base manifold (a base space), and a real vector space at each point in the base manifold.

\subsubsection{Example}

For example we can have a base manifold of a circle, and have a \(1\)-dimensional vector space at each point on the circle to create an infinitely extended cylinder.


In this section, we turn our attention to temperature-aware reliability
analysis.

Let $(\Omega, \F, \probability)$ be a probability space (see
\xref{probability-theory}), and let $\life: \Omega \to \real$ be a random
variable that represents the lifetime of the system. The lifetime is the time
until the system experiences a fault after which it no longer meets certain
requirements. Also, let $F(\cdot | \vg)$ be the distribution function of \life,
which gives the probability of failure before a certain moment in time, where
\vg is a vector of parameters. The expectation $\expectation{\life}$ is called
the \ac{MTTF}. Lastly, the complementary distribution function of \life is
\[
  R(t | \vg) = 1 - F(t | \vg),
\]
which, in the context of reliability analysis, gives the probability of survival
up to a certain moment in time and is called the reliability function of the
system.

The lifetime \life is a function of the lifetimes of the \np processing elements
that the system is composed of. Denote these individual lifetimes by
$\set{\life_i}{\Omega \to \real}_{i = 1}^\np$. Each $\life_i$ is characterized
by a physical model of wear \cite{jedec2016} that describes the stress that
processing element~$i$ is exposed to. Each $\life_i$ is also assigned an
individual $R_i(\cdot | \vg_i)$, which models the failures due to this stress.

The structure of $R(\cdot | \vg)$ with respect to $\set{R_i(\cdot | \vg_i)}_{i =
1}^\np$ is problem specific, and it can be especially diverse in the context of
fault-tolerant systems. Therefore, $R(\cdot | \vg)$ is to be devised by the
designer of the system under consideration. To give an example, suppose that the
failure of any of the \np processing elements makes the whole system fail, and
that $\set{\life_i}_{i = 1}^\np$ are conditionally independent given the
parameters gathered in \vg. In this scenario,
\begin{equation} \elab{reliability-model}
  \begin{split}
    & \life = \min_{i = 1}^\np \life_i \text{ and} \\
    & R(t | \vg) = \prod_{i = 1}^\np R_i(t | \vg_i).
  \end{split}
\end{equation}

\subsection{Periodic Thermal Stress}

In this subsection, we present a model that is frequently used for
characterizing the lifetime $\life_i$ of a single processing element when it is
exposed to repetitive temperature-induced stress \cite{huang2009b, xiang2010}.
The scenario under consideration is that the system is experiencing a periodic
workload with a certain period, which is denoted by \period. The resulting
temperature profile is a dynamic steady-state temperature profile, which will be
discussed in \sref{dynamic-steady-analysis}.

The lifetime $\life_i$ is assumed to have a Weibull distribution as follows:
\[
  \life_i | \vg_i \sim \mathrm{Weibull}(\scale_i, \shape_i)
\]
where $\scale_i$ and $\shape_i$ are called the scale and shape parameters,
respectively, and $\vg_i = (\scale_i, \shape_i)$. The distribution function of
$\life_i$ is
\begin{equation} \elab{weibull-distribution}
  F_i(t | \vg_i) = 1 - \exp\left(-\left(\frac{t}{\scale_i}\right)^{\shape_i}\right),
\end{equation}
the reliability function of the processing element is
\begin{equation} \elab{weibull-reliability}
  R_i(t | \vg_i) = 1 - F_i(t | \vg_i) = \exp\left(-\left(\frac{t}{\scale_i}\right)^{\shape_i}\right),
\end{equation}
and the corresponding \ac{MTTF} is
\begin{equation} \elab{weibull-expectation}
  \mean_i = \expectation{\life_i} = \scale_i \, \Gamma\left(1 + \frac{1}{\shape_i}\right)
\end{equation}
where $\Gamma$ stands for the gamma function.

\begin{remark} \rlab{weibull-homogeneity}
The Weibull model has a special case: assuming that $\shape_i = \shape$ for $i =
\range{1}{\np}$, the reliability function in \eref{reliability-model} belongs to
a Weibull distribution whose shape parameter is $\shape$ and scale parameter is
given by
\[
  \scale = \left(\sum\left(\frac{1}{\scale_i}\right)^\shape\right)^{-\frac{1}{\shape}}.
\]
\end{remark}

It is natural to expect that the usage conditions and hence the corresponding
stress change over the period \period. The distribution of $\life_i$ should then
reflect this aspect, which is not prominent in \eref{weibull-distribution}. In
order to take this into account, the period is split into \nk{i} time intervals
$\set{\dt_{ij}}_{j = 1}^{\nk{i}}$ so that the conditions that are relevant to
the model remain unchanged in each interval. Let
\[
  \life_{ij} | \vg_{ij} \sim \mathrm{Weibull}(\scale_{ij}, \shape_{ij})
\]
be the time to failure that the processing element would have if interval~$j$
was the only interval present and denote the corresponding \ac{MTTF} by
$\mean_{ij}$.

In the case of temperature-induced failures, we specify that $\shape_{ij} =
\shape_i$ for $j = \range{1}{\nk{i}}$; that is, the shape parameters are all
equal. The reason is that, unlike the scale parameters, the shape parameters are
independent of the operating temperature \cite{chang2006}. As shown in
\cite{xiang2010}, in this scenario, the reliability function of the processing
element can still be approximated by means of \eref{weibull-reliability} with
the following scale parameter:
\[
  \scale_i = \frac{\sum_{j = 1}^{\nk{i}} \dt_{ij}}{\sum_{j = 1}^{\nk{i}} \frac{\Delta t_{ij}}{\scale_{ij}}}.
\]
Applying \eref{weibull-expectation} at the level of individual intervals,
$\scale_i$ is rewritten as
\[
  \scale_i = \frac{\sum_{j = 1}^{\nk{i}} \dt_{ij}}{\Gamma\left(1 + \frac{1}{\shape_i}\right) \sum_{j = 1}^{\nk{i}} \frac{\Delta t_{ij}}{\mean_{ij}}}.
\]
Note now that the model in \eref{weibull-reliability} becomes fully specified as
soon as $\dt_{ij}$ and $\mean_{ij}$ are identified for $j = \range{1}{\nk{i}}$.
This part depends on the particular failure mechanism that is being considered,
which we discuss next.

\subsection{Thermal-Cycling Fatigue}
\slab{thermal-cycling-fatigue}

Let us tailor the above Weibull model to thermal-cycling fatigue
\cite{jedec2016}. This type of fatigue is of particular interest to us due to
its prominent dependence on temperature fluctuations: apart from the average and
maximum values, the frequencies and amplitudes of temperature oscillations
matter in this case.

Time intervals with constant relevant conditions correspond to thermal cycles.
In order to detect them in a given temperature curve, the curve is first
analyzed using a peak-detection algorithm in order to extract a set of extrema.
The rainflow counting method \cite{xiang2010} is then applied to these extrema.
The result is a set of \nk{i} thermal cycles. Each detected cycle is
characterized by a number of properties, including the desired duration
$\dt_{ij}$. Regarding the corresponding $\mean_{ij}$, it can be expressed as
follows:
\[
  \mean_{ij} = \nk{ij} \dt_{ij}
\]
where \nk{ij} stands for the mean number of such cycles to failure.

\begin{remark}
A cycle detected by the rainflow counting method does not have to be formed by
adjacent extrema; cycles can overlap. This feature makes the counting method
very efficient at mitigating overestimation. A cycle could be a half cycle,
meaning that only an upward or downward swing is present in the time series,
which is assumed to be adequately accounted for.
\end{remark}

The number \nk{ij} is estimated using a modified version of the Coffin--Manson
equation with the Arrhenius term as follows \cite{xiang2010, jedec2016}:
\begin{equation} \elab{thermal-cycling-mean-cycles}
  \nk{ij} = a_i (\Delta\q_{ij} - \Delta\q_{0, ij})^{-b_i} \exp\left(\frac{c_i}{k \q_{\maximum, ij}}\right)
\end{equation}
where $a_i$, $b_i$ (called the Coffin--Manson exponent), and $c_i$ (called the
activation energy) are empirically determined constants; $k$ is the Boltzmann
constant; $\Delta\q_{ij}$ is the excursion of the cycle in question;
$\Delta\q_{0, ij}$ is the portion of the temperature excursion that resides in
the elastic region, which does not cause damage; and $\q_{\maximum, ij}$ is the
maximum temperature during the cycle.

The reliability model of a single processing element is now fully specified. The
reliability function is the one in \eref{weibull-reliability} with
\begin{equation} \elab{thermal-cycling-scale}
  \scale_i = \frac{\sum_{j = 1}^{\nk{i}} \dt_{ij}}{\Gamma\left(1 + \frac{1}{\shape_i}\right) \sum_{j = 1}^{\nk{i}} \frac{1}{\nk{ij}}}
\end{equation}
where \nk{ij} is given in \eref{thermal-cycling-mean-cycles}. Using
\eref{weibull-expectation}, the \ac{MTTF} of the processing element is as
follows:
\begin{equation} \elab{thermal-cycling-mean-time}
  \mean_i = \frac{\sum_{j = 1}^{\nk{i}} \dt_{ij}}{\sum_{j = 1}^{\nk{i}} \frac{1}{\nk{ij}}}.
\end{equation}

In conclusion, it is worth emphasizing that the reliability model requires
detailed information about the thermal cycles to which the processing element is
exposed, which is the topic of \cref{certainty-development} and, in particular,
\sref{dynamic-steady-analysis} where dynamic steady-state temperature analysis
is discussed in detail.


\documentclass[12pt]{cdblatex}
\usepackage{fancyhdr}
\usepackage{footer}

\begin{document}

% =================================================================================================
% create checkpoint file

\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   import cdblib
   checkpoint_file = 'tests/semantic/output/detg2.json'
   cdblib.create (checkpoint_file)
   checkpoint = []
\end{cadabra}
\egroup

% =================================================================================================
\section*{The determinant of the metric}

Our game here is to compute (the leading terms) in $\det g$ of the metric in RNC form
\begin{dgroup*}
   \begin{dmath*} g_{a b}(x) = \cdb{gab.001}+\BigO{\eps^5} \end{dmath*}
\end{dgroup*}
For the sake of simplicity let's assume that we are working in 3-dimensions. The following
analysis is easily generalsied to other dimensions (and the final answers for $\det g$ and
friends are unchanged).

Define $\eps^{abc}_{ijk}$ by
\begin{align}
   \eps^{abc}_{ijk} =
        \delta^a_i \delta^b_j \delta^c_k - \delta^b_i \delta^a_j \delta^c_k
      + \delta^c_i \delta^a_j \delta^b_k - \delta^c_i \delta^b_j \delta^a_k
      + \delta^b_i \delta^c_j \delta^a_k - \delta^a_i \delta^c_j \delta^b_k
\end{align}
It is easy to see that $\eps^{abc}_{ijk}$ is anti-symmetric in both its upper and lower
indices. A trivial computation shows that for any $3{}\times{}3$ square matrix $M_{ab}$,
\begin{align}
   \eps^{abc}_{123} M_{1a} M_{2b} M_{3c}
   = \left(
          \delta^a_1 \delta^b_2 \delta^c_3 - \delta^b_1 \delta^a_2 \delta^c_3
        + \delta^c_1 \delta^a_2 \delta^b_3 - \delta^c_1 \delta^b_2 \delta^a_3
        + \delta^b_1 \delta^c_2 \delta^a_3 - \delta^a_1 \delta^c_2 \delta^b_3
     \right)M_{1a} M_{2b} M_{3c}
   = \det M
\end{align}
This can be easily generalised to
\begin{align}
   \eps^{abc}_{ijk} M_{pa} M_{qb} M_{rc}
   =
   \begin{cases}
      \pm \det M &\text{when $(ijk)$ and $(pqr)$ are permutations of $(123)$}\\
      0 & \text{otherwise}
   \end{cases}
\end{align}
The $\pm$ sign in the above depends on the particular permutations of $(ijk)$ and $(pqr)$. If
both permutations are even or both odd then the sign is $+1$ otherwise the sign is $-1$.
The same arguments can also be applied to a matrix inverse $N^{-1}$ leading to
\begin{align}
   \eps^{ijk}_{uvw} N^{pu} N^{qv} M^{rw}
   =
   \begin{cases}
      \pm \det {N^{-1}} &\text{when $(ijk)$ and $(pqr)$ are permutations of $(123)$}\\
      0 & \text{otherwise}
   \end{cases}
\end{align}
Note that the $\pm$ in this case will match exactly that for the case of $\det M$. Thus,
multiplying both expressions and summing over all choices for $(ijk)$ and $(pqr)$ leads
to
\begin{align}
   \sum_{\substack{(ijk)\\(pqr)}}\left(\det N^{-1}\right) \det M
   = \eps^{ijk}_{uvw} N^{pu} N^{qv} M^{rw} \eps^{abc}_{ijk} M_{pa} M_{qb} M_{rc}
\end{align}
where the sum on the left hand side includes just those $(ijk)$ and $(prq)$ that are
permutations of $(123)$. There are $3!$ choices for $(ijk)$ and $3!$ choices for
$(pqr)$ and thus the left hand side is easily reduced to $(3!)^2 \det M/\det N$ where
$\det N = 1/\det N^{-1}$. For the right hand side notice that
\begin{align}
   \eps^{ijk}_{uvw} \eps^{abc}_{ijk} = 3! \eps^{abc}_{uvw}
\end{align}
which leads to
\begin{align}
   \det M = \frac{1}{3!} \det N \eps^{abc}_{uvw} M_{pa} M_{qb} M_{rc} N^{pu} N^{qv} N^{rw}
\end{align}

For our RNC metric we will set $N^{ab} = g^{ab}$ and $M_{ij} = g_{ij}(x)$. Since $g^{ab}$ is
of the form ${\rm diag}(-1,1,1,1)$ we have $\det g = -1$ and thus
\begin{align}
   \det g(x) = - \frac{1}{3!} \eps^{abc}_{ijk}\, g_{pa}(x)\, g_{qb}(x)\, g_{rc}(x)\, g^{ip} g^{jq} g^{kr}
\end{align}

The $\eps^{abc}_{ijk}$ can be constructed in Cadabra by applying the \verb|asym| algorithm
to the upper indices of $\delta^a_i \delta^b_j \delta^c_k$. Note that \verb|asym| will
include the $1/3!$ coeffcient as part of its output.

The following code computes $-\det g$ rather than $\det g$.

{\bf Note} that Calzetta etal. use an opposite sign for $R_{abcd}$ so when comparing the
following results against Calzetta do take note of this flipped sign in $R_{abcd}$.

\clearpage

\begin{cadabra}
   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w#}::Indices(position=independent).

   {a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w#}::Integer(1..2).

   \nabla{#}::Derivative.

   d{#}::KroneckerDelta.

   g^{a b}::Symmetric.
   g_{a b}::Symmetric.

   R_{a b c d}::RiemannTensor.

   x^{a}::Weight(label=numx,value=1).

   def truncate (obj,n):

       ans = Ex(0)

       for i in range (0,n+1):
          foo := @(obj).
          bah = Ex("numx = " + str(i))
          keep_weight (foo, bah)
          ans = ans + foo

       return ans

   import cdblib

   g0ab = cdblib.get('g_ab_0','metric.json')
   g1ab = cdblib.get('g_ab_1','metric.json')  # zero in RNC
   g2ab = cdblib.get('g_ab_2','metric.json')
   g3ab = cdblib.get('g_ab_3','metric.json')
   g4ab = cdblib.get('g_ab_4','metric.json')
   g5ab = cdblib.get('g_ab_5','metric.json')

   gab := @(g0ab) + @(g1ab) + @(g2ab) + @(g3ab) + @(g4ab) + @(g5ab).  # cdb (gab.001,gab)
   gxab := gx_{a b} -> @(gab).

   eps := d^{a}_{i} d^{b}_{j}.   # cdb(eps.001,eps)
   asym (eps,$^{a},^{b}$)        # cdb(eps.002,eps) # includes a factor of 1/2!

   # compute negative Ndetg rather than det g
   Ndetg := @(eps) gx_{p a} gx_{q b} g^{i p} g^{j q}.  # note 1/2! included in eps

   substitute       (Ndetg,gxab)
   distribute       (Ndetg)
   Ndetg = truncate (Ndetg,5)                                          # cdb (Ndetg.001,Ndetg)
   substitute       (Ndetg,$g^{a b} g_{b c} -> d^{a}_{c}$,repeat=True) # cdb (Ndetg.002,Ndetg)
   eliminate_kronecker (Ndetg)                                         # cdb (Ndetg.003,Ndetg)
   sort_product     (Ndetg)                                            # cdb (Ndetg.004,Ndetg)
   rename_dummies   (Ndetg)                                            # cdb (Ndetg.005,Ndetg)
   canonicalise     (Ndetg)                                            # cdb (Ndetg.006,Ndetg)

   # introduce the Ricci tensor

   substitute     (Ndetg,$R_{a b c d} g^{a c} -> R_{b d}$,repeat=True)                                  # cdb (Ndetg.101,Ndetg)
   substitute     (Ndetg,$\nabla_{a}{R_{b c d e}} g^{b d}  -> \nabla_{a}{R_{c e}}$,repeat=True)         # cdb (Ndetg.102,Ndetg)
   substitute     (Ndetg,$\nabla_{a b}{R_{c d e f}} g^{c e}  -> \nabla_{a b}{R_{d f}}$,repeat=True)     # cdb (Ndetg.103,Ndetg)
   substitute     (Ndetg,$\nabla_{a b c}{R_{d e f g}} g^{d f}  -> \nabla_{a b c}{R_{e g}}$,repeat=True) # cdb (Ndetg.104,Ndetg)

   # the following are based on sqrt-Ndetg.tex

   sqrtNdetg := 1/2 + (1/2) @(Ndetg)
               - (1/8) (1/9) R_{a b} R_{c d} x^{a} x^{b} x^{c} x^{d}
               - (1/4) (1/18) R_{a b} \nabla_{c}{R_{d e}} x^{a} x^{b} x^{c} x^{d} x^{e}.
               # cdb (sqrtNdetg.001,sqrtNdetg)

   sort_product   (sqrtNdetg)                                          # cdb (sqrtNdetg.002,sqrtNdetg)
   rename_dummies (sqrtNdetg)                                          # cdb (sqrtNdetg.003,sqrtNdetg)
   canonicalise   (sqrtNdetg)                                          # cdb (sqrtNdetg.004,sqrtNdetg)

   logNdetg := -1 + @(Ndetg)
               - (1/2) (1/9) R_{a b} R_{c d} x^{a} x^{b} x^{c} x^{d}
               - (1/18) R_{a b} \nabla_{c}{R_{d e}} x^{a} x^{b} x^{c} x^{d} x^{e}.
               # cdb (logNdetg.001,logNdetg)

   sort_product   (logNdetg)                                           # cdb (logNdetg.002,logNdetg)
   rename_dummies (logNdetg)                                           # cdb (logNdetg.003,logNdetg)
   canonicalise   (logNdetg)                                           # cdb (logNdetg.004,logNdetg)

\end{cadabra}

\clearpage

\begin{dgroup*}
   \begin{dmath*} \cdb*{eps.001} \end{dmath*}
   \begin{dmath*} \cdb*{eps.002} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.001} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.002} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.003} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.004} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.005} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.006} \end{dmath*}
   \begin{dmath*} \cdb*{Ndetg.104} \end{dmath*}
   \begin{dmath*} \cdb*{sqrtNdetg.004} \end{dmath*}
   \begin{dmath*} \cdb*{logNdetg.004} \end{dmath*}
\end{dgroup*}

% =================================================================================================
% the remaining code is just for pretty printing

\clearpage

\begin{cadabra}
   # note: keeping numbering as is (out of order) to ensure R appears before \nabla R etc.
   def product_sort (obj):
       substitute (obj,$ x^{a}                            -> A000^{a}               $)
       substitute (obj,$ g^{a b}                          -> A001^{a b}             $)
       substitute (obj,$ \nabla_{c d e f}{R_{a b}}        -> A007_{a b c d e f}     $)
       substitute (obj,$ \nabla_{c d e}{R_{a b}}          -> A006_{a b c d e}       $)
       substitute (obj,$ \nabla_{c d}{R_{a b}}            -> A005_{a b c d}         $)
       substitute (obj,$ \nabla_{c}{R_{a b}}              -> A004_{a b c}           $)
       substitute (obj,$ \nabla_{e f g h}{R_{a b c d}}    -> A011_{a b c d e f g h} $)
       substitute (obj,$ \nabla_{e f g}{R_{a b c d}}      -> A010_{a b c d e f g}   $)
       substitute (obj,$ \nabla_{e f}{R_{a b c d}}        -> A009_{a b c d e f}     $)
       substitute (obj,$ \nabla_{e}{R_{a b c d}}          -> A008_{a b c d e}       $)
       substitute (obj,$ R_{a b}                          -> A002_{a b}             $)
       substitute (obj,$ R_{a b c d}                      -> A003_{a b c d}         $)
       sort_product   (obj)
       rename_dummies (obj)
       substitute (obj,$ A000^{a}                 -> x^{a}                          $)
       substitute (obj,$ A001^{a b}               -> g^{a b}                        $)
       substitute (obj,$ A002_{a b}               -> R_{a b}                        $)
       substitute (obj,$ A003_{a b c d}           -> R_{a b c d}                    $)
       substitute (obj,$ A004_{a b c}             -> \nabla_{c}{R_{a b}}            $)
       substitute (obj,$ A005_{a b c d}           -> \nabla_{c d}{R_{a b}}          $)
       substitute (obj,$ A006_{a b c d e}         -> \nabla_{c d e}{R_{a b}}        $)
       substitute (obj,$ A007_{a b c d e f}       -> \nabla_{c d e f}{R_{a b}}      $)
       substitute (obj,$ A008_{a b c d e}         -> \nabla_{e}{R_{a b c d}}        $)
       substitute (obj,$ A009_{a b c d e f}       -> \nabla_{e f}{R_{a b c d}}      $)
       substitute (obj,$ A010_{a b c d e f g}     -> \nabla_{e f g}{R_{a b c d}}    $)
       substitute (obj,$ A011_{a b c d e f g h}   -> \nabla_{e f g h}{R_{a b c d}}  $)

       return obj

   def get_term (obj,n):

       x^{a}::Weight(label=numx).

       foo := @(obj).
       bah  = Ex("numx = " + str(n))
       keep_weight (foo,bah)

       return foo

   def reformat (obj,scale):
       foo  = Ex(str(scale))
       bah := @(foo) @(obj).
       distribute     (bah)
       bah = product_sort (bah)
       rename_dummies (bah)
       canonicalise   (bah)
       sort_sum       (bah)
       factor_out     (bah,$x^{a?}$)
       ans := @(bah) / @(foo).
       return ans

   def rescale (obj,scale):
       foo  = Ex(str(scale))
       bah := @(foo) @(obj).
       distribute  (bah)
       factor_out  (bah,$x^{a?}$)
       return bah

   # ---------------------------------------------------------------
   # reformat Ndetg

   Rterm0 = get_term (Ndetg,0)       # cdb(Rterm0.701,Rterm0)
   Rterm1 = get_term (Ndetg,1)       # cdb(Rterm1.701,Rterm1)
   Rterm2 = get_term (Ndetg,2)       # cdb(Rterm2.701,Rterm2)
   Rterm3 = get_term (Ndetg,3)       # cdb(Rterm3.701,Rterm3)
   Rterm4 = get_term (Ndetg,4)       # cdb(Rterm4.701,Rterm4)
   Rterm5 = get_term (Ndetg,5)       # cdb(Rterm5.701,Rterm5)

   Rterm0 = reformat (Rterm0,  1)    # cdb(Rterm0.702,Rterm0)
   Rterm1 = reformat (Rterm1,  1)    # cdb(Rterm1.702,Rterm1)
   Rterm2 = reformat (Rterm2,  3)    # cdb(Rterm2.702,Rterm2)
   Rterm3 = reformat (Rterm3,  6)    # cdb(Rterm3.702,Rterm3)
   Rterm4 = reformat (Rterm4,180)    # cdb(Rterm4.702,Rterm4)
   Rterm5 = reformat (Rterm5, 90)    # cdb(Rterm5.702,Rterm5)

   Ndetg := @(Rterm0) + @(Rterm1) + @(Rterm2) + @(Rterm3) + @(Rterm4) + @(Rterm5).  # cdb (Ndetg.701,Ndetg)

   # ---------------------------------------------------------------
   # reformat sqrtNdetg

   Rterm0 = get_term (sqrtNdetg,0)   # cdb(Rterm0.801,Rterm0)
   Rterm1 = get_term (sqrtNdetg,1)   # cdb(Rterm1.801,Rterm1)
   Rterm2 = get_term (sqrtNdetg,2)   # cdb(Rterm2.801,Rterm2)
   Rterm3 = get_term (sqrtNdetg,3)   # cdb(Rterm3.801,Rterm3)
   Rterm4 = get_term (sqrtNdetg,4)   # cdb(Rterm4.801,Rterm4)
   Rterm5 = get_term (sqrtNdetg,5)   # cdb(Rterm5.801,Rterm5)

   Rterm0 = reformat (Rterm0,  1)    # cdb(Rterm0.802,Rterm0)
   Rterm1 = reformat (Rterm1,  1)    # cdb(Rterm1.802,Rterm1)
   Rterm2 = reformat (Rterm2,  6)    # cdb(Rterm2.802,Rterm2)
   Rterm3 = reformat (Rterm3, 12)    # cdb(Rterm3.802,Rterm3)
   Rterm4 = reformat (Rterm4,360)    # cdb(Rterm4.802,Rterm4)
   Rterm5 = reformat (Rterm5,360)    # cdb(Rterm5.802,Rterm5)

   sqrtNdetg := @(Rterm0) + @(Rterm1) + @(Rterm2) + @(Rterm3) + @(Rterm4) + @(Rterm5).  # cdb (sqrtNdetg.801,sqrtNdetg)

   # ---------------------------------------------------------------
   # reformat logNdetg

   Rterm0 = get_term (logNdetg,0)    # cdb(Rterm0.901,Rterm0)
   Rterm1 = get_term (logNdetg,1)    # cdb(Rterm1.901,Rterm1)
   Rterm2 = get_term (logNdetg,2)    # cdb(Rterm2.901,Rterm2)
   Rterm3 = get_term (logNdetg,3)    # cdb(Rterm3.901,Rterm3)
   Rterm4 = get_term (logNdetg,4)    # cdb(Rterm4.901,Rterm4)
   Rterm5 = get_term (logNdetg,5)    # cdb(Rterm5.901,Rterm5)

   Rterm0 = reformat (Rterm0,  1)    # cdb(Rterm0.902,Rterm0)
   Rterm1 = reformat (Rterm1,  1)    # cdb(Rterm1.902,Rterm1)
   Rterm2 = reformat (Rterm2,  3)    # cdb(Rterm2.902,Rterm2)
   Rterm3 = reformat (Rterm3,  6)    # cdb(Rterm3.902,Rterm3)
   Rterm4 = reformat (Rterm4,180)    # cdb(Rterm4.902,Rterm4)
   Rterm5 = reformat (Rterm5, 90)    # cdb(Rterm5.902,Rterm5)

   logNdetg := @(Rterm0) + @(Rterm1) + @(Rterm2) + @(Rterm3) + @(Rterm4) + @(Rterm5).  # cdb (logNdetg.901,logNdetg)

\end{cadabra}

\clearpage

% =================================================================================================
\section*{The metric determinant in Riemann normal coordinates}

\begin{dgroup*}
   \Dmath*{-\det g(x) = \cdb{Ndetg.701}+\BigO{\eps^6}}
\end{dgroup*}

% =================================================================================================
\section*{The volume element in Riemann normal coordinates}

If $-\det g(x)$ is non-negative then we also have
%
\begin{dgroup*}
   \Dmath*{\sqrt{-\det g(x)} = \cdb{sqrtNdetg.801}+\BigO{\eps^6}}
\end{dgroup*}

% =================================================================================================
\section*{The log of -detg in Riemann normal coordinates}

Apart from the signs, this matches exactly the expression given by Calzetta etal. (eq. A14)

\begin{dgroup*}
   \Dmath*{\log\left(-\det g(x)\right) = \cdb{logNdetg.901}+\BigO{\eps^6}}
\end{dgroup*}

\clearpage

% =================================================================================================
% export selected objects, these will later be imported into a library
% these are the objects that will appear in the paper

\begin{cadabra}
   cdblib.create ('detg2.export')

   cdblib.put ('Ndetg',    Ndetg,    'detg2.export')
   cdblib.put ('sqrtNdetg',sqrtNdetg,'detg2.export')
   cdblib.put ('logNdetg', logNdetg, 'detg2.export')

   checkpoint.append (Ndetg)
   checkpoint.append (sqrtNdetg)
   checkpoint.append (logNdetg)

\end{cadabra}

% =================================================================================================
% export checkpoints in json format

\bgroup
\CdbSetup{action=hide}
\begin{cadabra}
   for i in range( len(checkpoint) ):
      cdblib.put ('check{:03d}'.format(i),checkpoint[i],checkpoint_file)
\end{cadabra}
\egroup

\end{document}


\chapter{Statistical distance}

% \begin{figure}[H]
%     \centering
%     \includegraphics[width=.45\textwidth]{scatter_dist_question.png}
%     % \caption{A comic}
%     \label{fig:scatter_dist_questions}
% \end{figure}

\begin{figure}[H]
    \captionsetup[subfigure]{labelformat=empty}
    \centering
    \subfloat[$X \thicksim P$]{{\includegraphics[width=5cm]{red_scatter.png} }}%
    \qquad
    \subfloat[$Y \thicksim Q$]{{\includegraphics[width=5cm]{other_scatter.png} }}%
    \caption{Samples from two different one dimensional sources, $X$ and $Y$; the x
    axis represents the values that they take.
    How can we tell if they come from the same distribution?}
    \label{fig:scatter_dist_questions}%
\end{figure}

Suppose that we are given samples from two unknown distributions $P$ and $Q$, an
important question to ask is: are $P$ and $Q$ equal?

The Integral Probability Metric (IPM) and f-divergence are two very rich and well studied
families of measures of "distance" between probability measures.

We start by introducing the Reproducing Kernel Hilbert Spaces (RKHS), which will serve as a
building block for the maximum mean discrepancy (\cite{TwoSampleTestGrettonBernhard}), an important instance of IPM.

As we saw in the previous chapter, to understand causality, it is crucial to be able to
measure statistical dependence between random variables; therefore our ability of
performing causal inference will largely hinge on our ability to measure dependence
between random variables.

\newpage
\section{Reproducing Kernel Hilbert Space}

Kernels are an important component of the RKHS, and we will begin by defining them.

\subsection{Kernels}

\begin{definition}
    Let $\mathcal{X}$ be a non-empty set. A function
    $k: \mathcal{X} \times \mathcal{X} \rightarrow \R$ is a kernel if
    \begin{enumerate}
        \item $k$ is symmetric: $k(x, y) = k(y, x)$, $\forall x, y \in \mathcal{X}$
        \item $k$ is positive semi-definite, i.e. $\forall x_1, ..., x_n \in \mathcal{X}$,
        the "Gram Matrix" $K$, defined by $K_{ij} = k(x_i, x_j)$ is positive semi-definite
        \footnote{A matrix $M \in \R^{n\times n}$ is positive semi-definite if $\forall a \in \R^n$
        , $a^\intercal Ma \geq 0$}.
    \end{enumerate}
\end{definition}

It is easy construct new kernels since they are preserved under addition, multiplication and other operations.
(See for example \cite{GrettonNotes}).

One example of a kernel --- and one of the most popular ones --- is the Gaussian Kernel defined on $\R^d$:

$$
    k(x, y) = \exp (-\gamma^{-2}\norm{x - y}^2)
$$

\subsection{Constructing the Reproducing Kernel Hilbert Space}

Let $\mathcal{X}$ be an arbitrary set and $\mathcal{H}$ a Hilbert space of real valued functions
on $\mathcal{X}$. As per general convention, addition and multiplication are define pointwise:

\begin{equation}
    \begin{array}{lll}
    (\lambda \cdot f)(x) & :=\lambda \cdot f(x) & \forall \lambda \in \R, \forall f \in \mathcal{H} \text { and } \forall x \in \mathcal{X} \\
    (f+g)(x) & :=f(x)+g(x) & \forall f \in \mathcal{H}, \forall g \in \mathcal{H} \text { and } \forall x \in \mathcal{X}
    \end{array}
\end{equation}

% One powerful fact about Hilbert spaces is that every Hilbert space admits an orthonormal basis,
% and each vectorin the Hilbert space can be expanded as a series in terms of this orthonormal basis.

% For instance L2 blah blah

% $f \in L^1$ then we can decompose $f$ as follows:

% $$
%     f(x) = \int_{-\infty}^{\infty} \hat{f}(\omega) e^{2 \pi i x \omega} d \omega
%     = \inp*{\hat{f}}{\phi(x)}
% $$

We will now take a look at Hilbert spaces whose structure is highly linked with a kernel.
Note that if we pick some $x \in \mathcal{X}$, then $k(x, .)$ is a function from $\mathcal{X}$ to $\R$.

\begin{definition}
    Let $\mathcal{H}$ be a Hilbert space of functions $f: \mathcal{X} \rightarrow \R$.
    $\mathcal{H}$ is called a Reproducing Kernel Hilbert Space (RKHS) if there is a kernel k such that

    \begin{enumerate}
        \item $ k(x, \cdot) \in \mathcal{H} \quad \forall x \in \mathcal{X}$
        \item $ \inp*{f}{k(x, \cdot)} = f(x) \quad \forall f \in \mathcal{H}$
    \end{enumerate}

\end{definition}

Given the kernel $k$ it is convenient to define the feature map $\phi: \mathcal{X} \rightarrow \mathcal{H}$ as:

$$
    \phi(x) = k(x, \cdot)
$$

The intuition is that in this space, we can view functions as linear combinations\footnote{
    Note that if $f(x)$ is an element of $\mathcal{H}$, then we write $f$ as the coefficients
    for the feature representation.
} of features:

$$
    f(x) =  \inp*{f}{k(x, \cdot)} = \inp*{f}{\phi(x)}
$$

The power of this setup --- which is known as the kernel trick --- is that inner products between
features (which can live in infinite spaces) are simple function evaluations;
indeed by letting $f(x) = k(x, x^\prime)$ we get

$$
    \inp*{k(x^\prime, \cdot)}{k(x, \cdot)} = k(x, x^\prime)
$$

Observe that both conditions imply that $k$ spans $\mathcal{H}$, i.e.

\begin{equation}
    \mathcal{H}=\overline{\operatorname{span}\{k(\cdot, x): x \in \mathcal{X}\}}
\end{equation}

Indeed is is possible to go the other way around\footnote{See the excellent lecture notes on
RKHS \cite{BartlettNotes} for more details.} and first define the following vector space

\begin{equation}
    \operatorname{span}(\{\phi(x): x \in \mathcal{X}\})=\left\{f(\cdot)=
    \sum_{i=1}^{n} \alpha_{i} k\left(\cdot, x_{i}\right): n \in \N, x_{i}
    \in \mathcal{X}, \alpha_{i} \in \R\right\}
\end{equation}

We can then equip this space with an inner product and to show that it is complete in order to create
a Hilbert Space (at which point we will have created a RKHS).

\subsection{The kernel trick in action}

We will now show an application to illustrate both the power of the RHKS and to refine our intuition of it.
Suppose that we have
some data say $\{ x_i, y_i\}_{i \in [n]}$; we believe for example $y$ to be a smooth function of $x$ and we
expect some independent additive noise.

We can estimate $f$ as follows
\footnote{Note that it is not obvious how to implement the optimization as $\mathcal{H}$ may be infinite. However,
this setup with a Gaussian kernel is in fact equivalent to a Gaussian Processes, which can be easily
implemented in practice (see \cite{JordanNotes}).}, pick an RHKS $\mathcal{H}$ with a Gaussian kernel, and some
$\Omega > 0$:

\begin{equation}
    f^{*}=\arg \min _{f \in \mathcal{H}}\left(
        \sum_{i=1}^{n}
        \left(y_{i} - \inp*{f}{\phi\left(x_{i}\right)}_{\mathcal{H}} \right)^{2}
        + \Omega \norm{f}_{\mathcal{H}}^{2}\right)
\end{equation}

An amazing result is that an optimization of the above form will always admit a representation of the
form:

\[
    f^{*} = \sum_{i=1}^{n} \alpha_{i} \phi\left(x_{i}\right)
\]
where $\alpha_{i} \in \mathbb{R}$ for all $1 \leq i \leq n$.

This is known as the Representer Theorem (\cite{scholkopf2001generalized}); all it requires is
that we be in the usual RHKS setup, and that
the regularization be a strictly increasing\footnote{In our case regularization is linear,
we thus simply need to pick $\Omega \geq 0$.} real valued function.
If we wish to approximate a prediction for some new sample $x$, we can do so as follows:

$$
    f^{*}(x) = \inp*{f^{*}}{\phi\left(x\right)} =
    \sum_{i=1}^{n} \alpha_{i} \inp*{\phi\left(x_i\right)}{\phi\left(x\right)} =
    \sum_{i=1}^{n} \alpha_{i} k\left(x_i, x\right)
$$

It is precisely because the solution is of this form, that we may exploit the kernel trick. We can
also quickly see what the role of the kernel is. If for example, $k$ is the Gaussian Kernel, then
the solution will be a linear combination of scaled Gaussians centered at the data points
\footnote{In fact this will always be the case when we can write $k\left(x, y\right) = \tilde{k}\left(x - y\right)$}.

As a final remark we will explain the role of the penalty $\Omega \norm{f}_{\mathcal{H}}^{2}$; from
statistical models, we know that this kind of term is known as regularization and is supposed to help
choose a "simpler" model. As we will now show, this is also the case here.

To see this, we will use Mercer's Theorem (as stated in the lecture notes of \cite{BartlettNotes}) --- a Generalization of the spectral theorem for positive-semidefinite
matrices\footnote{Recall that our Kernel $k$ is a generalization of a positive-semidefinite Matrix}.

\begin{theorem}[Mercer's]

Suppose $k$ is a continuous positive semi-definite kernel on a compact set $\mathcal{X}$, then if
, for all $f \in L_{2}(\mathcal{X})$

\[
\int_{\mathcal{X}} k(u, v) f(u) f(v) d u d v \geq 0
\]

then $k$ has the following decomposition

\begin{equation}
    k(u, v)=\sum_{i=1}^{\infty} \lambda_{i} \psi_{i}(u) \psi_{i}(v)
\end{equation}

where $\{\psi_i\}$ forms an orthonormal basis of $L_2(\mathcal{X})$,
such that the corresponding sequence of eigenvalues $\{\lambda_i\}$ are non-negative.

The convergence is absolute and uniform, that is,
\[
\lim _{n \rightarrow \infty} \sup _{u, v}\left|k(u, v)-\sum_{i=1}^{n} \lambda_{i} \psi_{i}(u) \psi_{i}(v)\right|=0
\]

\end{theorem}

We can now use this decomposition of the Kernel to get further insight, using Mercer's theorem we can
thus write --- assuming the conditions are met:

$$
    k \left(x, x^{\prime}\right)=\sum_{i=1}^{\infty}
    \underbrace{\left[\sqrt{\lambda_{i}} \psi_{i}(x)\right]}_{\phi_{i}(x)}
    \underbrace{\left[\sqrt{\lambda_{i}} \psi_{i}\left(x^{\prime}\right)\right]}_{\phi_{i}\left(x^{\prime}\right)}
$$

We can thus rewrite the solution as follows

$$
    f^{*}(x) = \sum_{i=1}^{n} \alpha_{i} k\left(x_i, x\right) =
    \sum_{i=1}^{\infty} \phi_i\left(x\right)
    \sum_{j=1}^{n} \alpha_{j}  \phi_i\left(x_j\right) =
    \sum_{i=1}^{\infty} \sqrt{\lambda_{i}} \psi_{i}(x) f^{*}_i
$$

Note that due to the $\Omega \norm{f}_{\mathcal{H}}^{2}$ penalty, $f^{*}_i$ must decay for higher values of $i$. Note that
for example for the Fourier Transform
, in the basis $\{\psi_i\}$, higher values of $i$ correspond to higher
frequency functions; similarly, for the Gaussian Kernel, higher indices basis functions correspond to
higher frequencies\footnote{In the fourier space, we have the following
basis $\psi_\omega = \exp (2\pi i x \omega)$}. Thus, a higher $\Omega$ will force a faster decay on $f_i$ and thus result in smoother functions
-- in principle, this will reduce overfitting.

% \begin{figure}[htp]
\begin{figure}[H]

    \centering
    \includegraphics[width=.33\textwidth]{gp_goodfit.png}
    \includegraphics[width=.33\textwidth]{gp_underfit.png}
    \includegraphics[width=.33\textwidth]{gp_overfit.png}
    \caption{Small RKHS norm results in smooth functions.
    From left to right $\Omega = 2$, $\Omega = 14$, $\Omega = 0.2$,
    we fix the Gaussian kernel with $\gamma = 0.6$}
    \label{fig:kernel_smoothness}
\end{figure}

\section{Integral Probability Metric}

\subsection{Introduction}

We now turn to the question of statistical distance, i.e. given samples of $P$ and
$Q$, how can we determine if $P = Q$?

Observe that if two random variables $X$, $Y$ share the same distribution, then

$$
\E(g(X)) = \E(g(Y))
$$

for any continuous and bounded function $g : \R \rightarrow \R$.
It turns out that the reciprocal statement holds as well. (See \cite{TwoSampleTestGrettonBernhard})

This motivates the following construction

$$
D_\mathcal{F} (P, Q) =
\sup _{g \in \mathcal{F}} \mid \mathop{\E}_{X \sim P} g(X) - \mathop{\E}_{Y \sim Q} g(Y) \mid
$$

where $\mathcal{F}$ is a class of real-valued bounded measurable functions.

% See for example \cite{sriperumbudur2009integral} for a detailed analaysis

This defines a rich class of distance measures known as
integral probability metrics (IPMs) (see \cite{muller1997integral}). Depending
on how we choose $\mathcal{F}$ we may end up with different popular distance measures, such as
the Wasserstein distance or the Total variation distance to name a few.

The goal is to design an $\mathcal{F}$ that is "expressive" enough so that the IPM goes to zero iff $P = Q$,
and on the other hand, we need $\mathcal{F}$ to be "restrictive" enough so as to have fast and
reliable guarantees of the empirical estimate of the IPM (\cite{TwoSampleTestGrettonBernhard}.)

\subsection{MMD}

Consider $\mathcal{F}=\left\{f:\|f\|_{\mathcal{H}} \leq 1\right\}$, this is known
as the maximum mean discrepancy (MMD); where $\mathcal{H}$, is a reproducing kernel Hilbert space
(RHKS) with $k$ as its reproducing kernel.

We will next extend the notion of the feature map to the \textbf{embedding of probability distributions}.
Recall that if $\phi$ is the associated feature map to the kernel $k$ from
RKHS $\mathcal{H}$ then we have $g(x) = \inp*{g}{\phi(x)}$.

We define $\mu_P \in \mathcal{H}$, s.t. $\forall g \in \mathcal{H}$, we have that
$\mathop{\E}_{X} g(X) = \inp*{g}{\mu_P}$. We will now show under which conditions $\mu_P$ exits
(we follow the derivations as done by \cite{Peters2008diploma}).

\begin{lemma}\label{embedding_existance}
    If k is measurable and $\mathop{\E}_{X} \sqrt{k(X, X)} < \infty$ then
    $\mu_P \in \mathcal{H}$
\end{lemma}

\begin{proof}
\begin{align*}
    \left|\E_{X} g(X)\right| &\leq
    \E_{X}|g(X)| \\
    &=
    \E_{X}\left| \inp*{g}{\phi(X)}_{\mathcal{H}}\right| \\
    &\leq
    \E_{X} \norm{g}_{\mathcal{H}} \norm{\phi(X)}_{\mathcal{H}} \\
    &= \norm{g}_{\mathcal{H}} \E_{X} \sqrt{k(X, X)}
\end{align*}

The first inequality follows from the triangle inequality, and the following equality by using
the feature map representation; we conclude by applying Cauchy-Schwartz.

Thus $E_{X} g(X)$ is a bounded linear operator $\forall g \in \mathcal{F}$, and by the
Riesz representer theorem it follows that there exists a $\mu_P \in \mathcal{H}$
s.t. $\mathop{\E}_{X} g(X) = \inp*{g}{\mu_P}$.
\end{proof}

We can also see that the mean embedding of the distribution $P$ is the expectation under $P$
of the feature map $\phi$.

$$
    \mathop{\E}_{X \sim P} g(X) =
     \inp*{g}{ \mathop{\E}_{X \sim P} \phi(X)} = \inp*{g}{\mu_P}
$$

Assuming Lemma \ref{embedding_existance} --- and using Cauchy-Schwartz,
we can explicitly solve the MMD in terms of the mean embeddings:

\begin{align*}
    \text{MMD}_\mathcal{F} (P, Q)
    &=
    \sup _{g \in \mathcal{F}} \mid \mathop{\E}_{X \sim P} g(X) - \mathop{\E}_{Y \sim Q} g(Y) \mid \\
    &=
    \sup _{g \in \mathcal{F}} \mid \inp*{g}{\mu_P - \mu_Q} \mid \\
    &=
    \norm{\mu_P - \mu_Q}_\mathcal{H}
\end{align*}

We can therefore see the MMD as the feature mean difference of the distributions; we can further
expand this expression to get the result as a function of the kernel.

\begin{align*}
    \text{MMD}_\mathcal{F}^2 (P, Q)
    &=
    \norm{\mathop{\E}_{X \sim P} \phi(X)  - \mathop{\E}_{Y \sim Q} \phi(Y) }_\mathcal{H}^2 \\
    &= \mathop{\E}_{X \sim P} \mathop{\E}_{X^\prime \sim P} \inp*{\phi(X)}{\phi(X^\prime)} -
    2 \mathop{\E}_{X \sim P} \mathop{\E}_{Y \sim Q} \inp*{\phi(X)}{\phi(Y)} +
    \mathop{\E}_{Y \sim Q} \mathop{\E}_{Y^\prime \sim Q} \inp*{\phi(Y)}{\phi(Y^\prime)} \\
    &= \mathop{\E}_{X \sim P} \mathop{\E}_{X^\prime \sim P} k \left( X, X^\prime \right) -
    2 \mathop{\E}_{X \sim P} \mathop{\E}_{Y \sim Q} k \left( X, Y \right) +
    \mathop{\E}_{Y \sim Q} \mathop{\E}_{Y^\prime \sim Q} k \left( Y, Y^\prime \right)
\end{align*}

Note that we can straightforwardly estimate with samples the above expression; all the we require is
to specify a kernel: \textit{so how do we choose a kernel}?

We need to ensure that $\text{MMD}(P, Q) = 0$ iff $P = Q$, in other words, $\mu_P$ needs to be injective
as a function of $P$. Intuitively this means that $\mathcal{F}$ needs to be expressive enough to
reproduce enough continuous functions. One can show that to check if the resulting embedding
$\mu_P$ is injective, we may check either of these sufficient conditions (\cite{sriperumbudur2008injective})
on the Kernel $k$:

\begin{enumerate}
    \item $k$ is a universal kernel.
    \item $k$ is a convolution kernel on $\R^n$, for which the Radon-Nikodym derivative of
    its inverse Fourier transform is supported almost everywhere.
\end{enumerate}

The first condition is basically what we knew intuitively:
If we consider a compact metric space, say $(\mathcal{X}, d)$, then a Kernel $k$ on $\mathcal{X}$
is called universe if the corresponding RKHS is dense
in the space $C(\mathcal{X})$ of all continuous functions. The drawback is that the input space $\mathcal{X}$ needs to be compact --- which excludes $\R^n$;
this means that we cannot use universality to check our Gaussian kernel. Luckily the second condition
is enough.

Assuming $k$ is a bounded continuous positive definite function, then if we can write
$k(x, y) = \psi (x - y)$ we say that $k$ is a convolution kernel.

From inspection it is clear that the Gaussian kernel is convolutional

$$
    k(x, y) = \exp (-\gamma^{-2}\norm{x - y}^2)
$$

A cool fact about the Gaussian is that it is the fixed point of convolution
, which trivially implies
that the inverse Fourier transform of a Gaussian is supported everywhere.
 This means that the Gaussian kernel satisfies the second
condition, and it therefore generates an injective embedding $\mu_P$.

We note that HSIC is to MMD, what the Mutual Information is to the Kullback–Leibler divergence (in the
sense that they measure the distance between the joint and product of the marginal distributions to test for independence).

In practice the Gaussian kernel is very popular, it is used in the HSIC test when used as a
score function by \cite{Mooij2016jmlr}; but how do we find the parameter --- sometimes referred to
as the bandwidth --- of the kernel?

One approach is the median heuristic (\cite{scholkopf2002learning}):

$$
\hat{\gamma}(\mathbf{u}):=\operatorname{median}\left\{\left\|u_{i}-u_{j}\right\|:  i<j ,\left\|u_{i}-u_{j}\right\| \neq 0\right\}
$$

\subsection{The case for MMD}

In their study, \cite{sriperumbudur2009integral}) argue that the "IPM is much
simpler than estimating f-divergences, and that the estimators
are strongly consistent while exhibiting good rates of convergence. IPMs also account for the properties of
the underlying space $\mathcal{M}$ through the Kernel in case of MMD. This is especially
useful when considering disjoint supports between P and Q"

Another argument for the MMD, is that we only need to choose a kernel; in contrast,
when applying the f-divergence in practice we need to quantize in order to get an
empirical distribution. While both can be seen as a hyperparameter, the effect of
discretisation is not as obvious as that of choosing a kernel.

\section{f-divergence}

The f-divergence (\cite{csiszar2004information}) is another family of probability measures, and more simple than IPM.

\begin{definition}[f-divergence]
    ~
Let $P$ and $Q$ be two probability distributions over a space $\Omega$,
such that $P$ is absolutely continuous with respect to $Q$; and let
$f: \R_+ \rightarrow \R$ be a convex function satisfying $f(1) = 0$. The $f$ divergence of
$P$ from $Q$ is defined as

$$
    \infdiv{P}{Q} := \int_\Omega f \left( \frac{dP}{dQ} \right) dQ
$$

\end{definition}

We first show why divergence has some desirable properties for a probability measure:

\begin{align*}
    \infdiv{P}{Q} &= \E_Q \left[ f \left( \frac{dP}{dQ} \right) \right] \\
    &\geq f \left( \E_Q \left[  \frac{dP}{dQ} \right]  \right) \\
    &= f \left( \int_\Omega \frac{dP}{dQ} dQ  \right) \\
    &= f(1) \\
    &= 0
\end{align*}

The inequality follows from the convexity of $f$, this tells us that
$\infdiv{P}{Q} \geq 0$. From the definition it is clear that $\infdiv{P}{P} = 0$; further,
if $f$ is \textit{strictly} convex at $1$, then we have that $\infdiv{P}{Q} = 0$ iff $P = Q$.

Therefore roughly speaking, all f-divergences define a way to measure similarities between distributions.

However, in general it is not symmetric in $P$ and $Q$, so it is not a metric.

The following are some examples of f-divergences:

\begin{itemize}
    \item[--] \textbf{Kullback-Leibler (KL) divergence}: $f(x) = x \log(x)$
    \item[--] \textbf{Total Variation (TV)}: $f(x) = \frac{1}{2} \abs{x - 1}$, note that
    in this case we have
    $$
        \infdiv{P}{Q} = \frac{1}{2} \E_Q \left[ \abs{ \frac{dP}{dQ} - 1} \right]
        = \frac{1}{2} \int_\Omega \abs{ dP - dQ}
    $$
    Note that the TV is also a metric on the space of probability distributions.
\end{itemize}

% While seemingly less flexible than IPM, the f-divergences have a very rich theory behind them,

% There are many very useful theoretical results about f-divergences, among them is the
% data processing inequality which also plays a central role in many proofs in Information Theory.

% \begin{lemma}[Data Processing Inequality (DPI)]
%     Let $X$ and $Y$ be two random variables, and let $P_X$ and $Q_X$ be two measures on $X$. If we
%     pass these measures through a kernel, say $P_{Y|X}$ the resulting measures $P_X$ and $Q_X$ will
%     satisfy:

%     $$
%         \infdiv{P_X}{Q_X} \geq \infdiv{P_Y}{Q_Y}
%     $$
%     \label{lemma:dpi}
% \end{lemma}

% A proof of this lemma and more ineteresting facts about f-divergences can be found in chapter
% 2 of \cite{duchiEE377}.

\section{Independence tests}

It is rather straightforward to come up with independence tests once we are we are able to test
for the distance between distributions. Say we are given two random variables $X$ and $Y$, with
values over the product space $\mathcal{X} \times \mathcal{Y}$. If their joint distribution is
$P_{X, Y}$, and their marginal distributions are $P_X$ and $P_Y$. Then to check if $X \bigCI Y$
we need to verify if $P_{X, Y} = P_X \otimes P_Y$.

If we want to create an independence test from an f-divergence, say $f(x) = x\log(x)$, then we
can do as follows:

$$
    I (X; Y) :=  \infdiv{P_{X, Y}}{P_X \otimes P_Y}
$$

This is in fact the well known Mutual Information from Information Theory!

Recall that for the MMD we need to provide a kernel, since we are now in a product space, we need
to provide a product kernel on the space ($\mathcal{X}, \mathcal{Y}$): (\cite{Peters2008diploma})

\begin{align*}
    \mathcal{X} \times \mathcal{Y} \quad &\rightarrow \qquad \R \\
    ( (x, y) (\tilde{x}, \tilde{y})) &\mapsto k(x, \tilde{x}) \cdot l(y, \tilde{y})
\end{align*}

where $k$ and $l$ are kernels on $\mathcal{X}$ and $\mathcal{Y}$ respectively. We can then define the
MMD to test independence as follows:

$$
    \operatorname{MMD}\left( P_{X, Y}, P_X \otimes P_Y  \right)
$$

Observe that

$$
    \operatorname{MMD}\left( P_{X, Y}, P_X \otimes P_Y  \right)^2 = \operatorname{HSIC}(P_{X, Y})
$$

The HSIC is the MMD distance between the joint and product distribution.


\section{Practical Training Method
 in Large-Scale Machine Learning Problems}

\subsection{Basic Gradient Descent Type Algorithms}
Now we suppose we have a machine learning model
\begin{equation}
f(x; w),
\end{equation}
where $f$ can be any machine learning model like: linear regression, SVM or deep learning models.

We have data as
$$
z_i=\{x_i, y_i\}\quad i = 1:N,
$$
and the loss between the label and prediction is:
$$
f_i(w) = l(f(x_i, w), y_i),
$$
one example of $l$ is:
$$
l(f(x_i, w), y_i) = \|f(x_i, w)- y_i\|^2.
$$
What we want to solve is:
\begin{equation}
\mathop{\min}_{{w} \in \mathbb{R}^{n}} \frac{1}{N}\sum_{i=1}^N f_i(w).
\end{equation}

\subsubsection{Mini-Batch Method}
Now we introduce the Mini-Batch algorithm which is mostly used in basic algorithm for above problem:
\begin{algorithm}[H]
\caption{Mini-Batch}
\label{alg:mini-batch}
{\bf Input}: learning rate $\eta_t$, batch size $m$, parameter initialization $ w_0$, number of epochs $K$. \\
%For iteration $t = 1: K\frac{N}{m}$ \\
For Epoch $k = 1:K$\\
Shuffle data and get mini-batch $B_1, \cdots, B_{\frac{N}{m}}$, choose mini-batch as: $B_{i_t}$ with
$$
i_t \equiv t \mod(\frac{N}{m}),
$$
Compute the gradient on $B_{i_t}$:
$$
g_t = \nabla_{w} \frac{1}{m} \sum_{i \in B_{i_t}} f_i(w_{t})
$$
Update $w$:
\begin{equation}
w_{t+1} = w_t - \eta_t g_t.
\end{equation}
\end{algorithm}

This method is a little different from pure SGD type method as we still have
$$
\mathbb{E} g_t = \nabla f,
$$
except  the choice of $\nabla f_i$ and $\nabla f_{i+1}$ are dependent.
But this method works very well with some suitable batch-size. Recently, many researches show that small batch-size may lead to some flat minimizer with better generalization error. And some research suggests that, you can use a small batch size at the beginning and increase the batch size slowly with iteration.

\subsection{Deterministic Algorithm(a specially case):  Incremental Gradient Descent Method}
The cycle incremental gradient descent method:
\begin{algorithm}\caption{CIGD}
	\label{alg:CIGD}
	\begin{equation}\label{equ:GS-iteration}
	x_{t+1} = \Pi_{\mathcal{X}}(x_{t} - \eta_t \nabla f_{i_t}(x_t)), \quad t = 0:T,
	\end{equation}
	\begin{equation}
	i_t \equiv t \mod(m), \quad i_t \in 1:m,
	\end{equation}
\end{algorithm}

\subsection{Convergence for CIGD}
We start with the iteration scheme step by step, and analyze it first, and then sum a cycle iteration together.

A simple sufficient condition for the above assumption is that $f_i \in C^2(\mathcal{X})$ and is convex with $\mathcal{X}$ is a compact convex set.

Here we prove an important lemma:
\begin{lemma}\label{lem:CIGDexpand}Let $\{x_k\}$ be the sequence generated by algorithm \ref{alg:CIGD} with $\eta_t = \eta_k$ for $t = km:((k+1)m-1)$, then for any $y \in \mathcal{X}$, we have
	\begin{equation}\label{equ:CIGDexpand}
	\|x_{(k+1)m} - y\|^2 \le \|x_{km} - y\|^2 - 2m\eta_k(f(x_{km}) - f(y)) + \eta_k^2\beta m^2M^2,
	\end{equation}
	where $\beta = 2 - \frac{1}{m}$
\end{lemma}

\begin{proof}
	By the definition of iteration scheme:
	\begin{equation}\label{equ:basicinequICGD}
	\|x_{km+j} - y\|^2 \le \|x_{km+j-1} - y\|^2 -2\eta_k \nabla f_{j}(x_{km+j-1})^{T}(x_{km+j-1}-y) + \eta_k^2 M^2, \quad \forall j = 1:m.
	\end{equation}
	Because of the convexity of $f_i$, we have
	\begin{equation}
	\nabla f_{j}(x_{km+j-1})^{T}(x_{km+j-1}-y) \ge f_{j}(x_{km+j-1}) - f_{j}(y).
	\end{equation}
	Add all $j = 1:m$ together
	\begin{equation}
	\|x_{(k+1)m} - y\|^2 \le \|x_{km} - y\|^2 -2m\eta_k (f(x_{km}) - f(y))+ m\eta_k^2 M^2 + 2\eta_k\sum_{j=1}^m(f_{j}(x_{km}) - f_{j}(x_{km+j-1}) ).
	\end{equation}
	For the last term we have:
	\begin{equation}
	f_{j}(x_{km}) - f_{j}(x_{km+j-1}) \le M\| x_{km} - x_{km + 1} \| + \cdots + M\|x_{km+j-2} - x_{km+j-1}\| \le 2(j-1)\eta_k M^2,
	\end{equation}
	the last inequality can be proven by take $y = x_{km + j -1}$ in \ref{equ:basicinequICGD}.

	Finally we have
	\begin{equation}
	\|x_{(k+1)m} - y\|^2 \le \|x_{km} - y\|^2 -2m\eta_k (f(x_{km}) - f(y))+ m\eta_k^2 M^2 + 2\eta_k^2 M^2m(m-1),
	\end{equation}
	which is the form of \ref{equ:CIGDexpand} with $\beta = 2 - \frac{1}{m}$.
\end{proof}

Using this lemma, we can have the next results:
\begin{theorem}
	Let $\eta_k$ be fixed at some positive constant $\eta$.
	\begin{itemize}
		\item If $ f^* = \min_{x \in \mathcal{X}}f(x) = -\infty$, then
		\begin{equation}
		\mathop{\lim\inf}_{k \to \infty} f(x_k) = f^*.
		\end{equation}
		\item If $f^* > -\infty$, then
		\begin{equation}
		\mathop{\lim\inf}_{k \to \infty} f(x_k) \le f^* + \frac{\eta \beta m M^2}{2},
		\end{equation}
		where $\beta$ is the same in lemma \ref{lem:CIGDexpand}.
	\end{itemize}
\end{theorem}

\begin{theorem}
	If $X^* = \mathop{\arg\min}f(x)$ is nonempty. Then for $\epsilon > 0$, we have
	\begin{equation}
	\min_{ 0\le k \le N} f(x_k) \le f^* + \frac{\eta \beta m^2 M^2 + \epsilon}{2m},
	\end{equation}
	where $N$ is given by
	\begin{equation}
	N = m \lfloor \frac{\rm{dist}(x_0;X^*)^2}{\eta \epsilon}\rfloor.
	\end{equation}
\end{theorem}

\begin{theorem}
	If the stepsize $\eta_k$ satisfy
	\begin{equation}
	\lim_{k \to \infty} \eta_k = 0, \quad \sum_{k=0}^{\infty} \eta_k = \infty.
	\end{equation}
	Then,
	\begin{equation}
	\mathop{\lim\inf}_{k\to \infty} f(x_k) = f^*.
	\end{equation}
	Furthermore, if $X^*$ is nonempty and
	\begin{equation}
	\sum_{k=0}^\infty \eta_k^2 < \infty,
	\end{equation}
	then $\{x_k\}$ converges to some $x^* \in X^*$.
\end{theorem}

As far as we know, it is still an open problem to establish rate of
convergence for CIGD method.  In contrast, convergence rate in $L^2$
can be established for stochastic gradient (SGD) method.
\subsection{Some comments}
Minimizing $f_1$ and $f_2$ (or $\sum_i f_i$) alternatively by using
gradient descent, this is the so-called ``cycle incremental gradient
method".   This kind of incremental methods are obviously closely
related to subspace correction method for both
stochastic and cycle types.  Such connections need to be further
investigated.

With some assumption such as differentiability, convexity, Lipschitz
continuity, boundedness for gradient etc, qualitatively convergence
can be established for cycle type methods when diminishing stepsize
(or learning rate) is applied, but it is not known how convergence rate
can be established without randomization.

Some relevant discussions can be found in Bertsekas \cite{bertsekas2015convex, bertsekas2011incremental}


\documentclass{article}
\usepackage{amsmath}
\usepackage[margin=1.0in]{geometry}
\usepackage{xcolor}

\begin{document}

\noindent
Does $\displaystyle \sum_{n=1}^\infty \frac{\cos \pi n}{n}$
diverge, converge absolutely, or converge conditionally?

\subsection*{Solution}

The series $\displaystyle \sum_{n=1}^\infty \frac{\cos \pi n}{n}$ is an alternating series. Note $b_n = |a_n| = \frac1n$. Since the sequence $b_n$ is decreasing and $b_n \to 0$, by the Alternating Series Test, $\displaystyle \sum_{n=1}^\infty \frac{\cos \pi n}{n}$ converges.

To figure out whether $\displaystyle \sum_{n=1}^\infty \frac{\cos \pi n}{n}$ converges absolutely or conditionally, we consider the series
\[
\sum_{n=1}^\infty \left|\frac{\cos \pi n}{n}\right|
= \sum_{n=1}^\infty \frac{1}{n}
\]
which diverges by the $p$-test, so the series $\displaystyle \sum_{n=1}^\infty \frac{\cos \pi n}{n}$ converges conditionally.

\end{document}%%%%%%%%%%%%%%%%%

\begin{align*}
L&=\lim_{n \to \infty} \sqrt[n]{|a_n|}\\
&= \lim_{n \to \infty} \sqrt[n]{\left| \right|}\\
\end{align*}

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty AAAAAAAAAAAAAA$ converges absolutely.

Since $|r| < 1$, the series ...  converges by the Geometric Series Test.

Since $|r| \geq 1$, the series ...  diverges by the Geometric Series Test.

The function $f(x)=\frac{}{}$ is continuous, positive, and decreasing on $[1,\infty)$.

\subsection*{Solution}