%!TEX root = ceres-solver.tex
\chapter{Solving}
Effective use of Ceres requires some familiarity with the basic components of a nonlinear least squares solver, so before we describe how to configure the solver, we will begin by taking a brief look at how some of the core optimization algorithms in Ceres work and the various linear solvers and preconditioners that power it.

\section{Trust Region Methods}
Let $x \in \mathbb{R}^{n}$ be an $n$-dimensional vector of variables, and
$ F(x) = \left[f_1(x),   \hdots,  f_{m}(x) \right]^{\top}$ be a $m$-dimensional function of $x$.  We are interested in solving the following optimization problem~\footnote{At the level of the non-linear solver, the block and residual structure is not relevant, therefore our discussion here is in terms of an optimization problem defined over a state vector of size $n$.},
\begin{equation}
        \arg \min_x \frac{1}{2}\|F(x)\|^2\ .
        \label{eq:nonlinsq}
\end{equation}
Here, the Jacobian $J(x)$ of $F(x)$ is an $m\times n$ matrix, where $J_{ij}(x) = \partial_j f_i(x)$  and the gradient vector $g(x) = \nabla  \frac{1}{2}\|F(x)\|^2 = J(x)^\top F(x)$. Since the efficient global optimization of~\eqref{eq:nonlinsq} for general $F(x)$ is an intractable problem, we will have to settle for finding a local minimum.

The general strategy when solving non-linear optimization problems is to solve a sequence of approximations to the original problem~\cite{nocedal2000numerical}. At each iteration, the approximation is solved to determine a correction $\Delta x$ to the vector $x$. For non-linear least squares, an approximation can be constructed by using the linearization $F(x+\Delta x) \approx F(x) + J(x)\Delta x$, which leads to the following linear least squares  problem:
\begin{equation}
         \min_{\Delta x} \frac{1}{2}\|J(x)\Delta x + F(x)\|^2
        \label{eq:linearapprox}
\end{equation}
Unfortunately, na\"ively solving a sequence of these problems and updating $x \leftarrow x+ \Delta x$ leads to an algorithm that may not converge.  To get a convergent algorithm, we need to control the size of the step $\Delta x$. And this is where the idea of a trust-region comes in. The generic trust-region loop for non-linear least squares problems looks something like this

\begin{algorithmic}
\REQUIRE Initial point $x$ and a trust region radius $\mu$.
\LOOP
\STATE{Solve $\arg \min_{\Delta x} \frac{1}{2}\|J(x)\Delta x + F(x)\|^2$ s.t. $\|D(x)\Delta x\|^2 \le \mu$}
\STATE{$\rho = \frac{\displaystyle \|F(x + \Delta x)\|^2 - \|F(x)\|^2}{\displaystyle \|J(x)\Delta x + F(x)\|^2 - \|F(x)\|^2}$}
\IF {$\rho > \epsilon$}
\STATE{$x = x + \Delta x$}
\ENDIF
\IF {$\rho > \eta_1$}
\STATE{$\rho = 2 * \rho$}
\ELSE
\IF {$\rho < \eta_2$}
\STATE {$\rho = 0.5 * \rho$}
\ENDIF
\ENDIF
\ENDLOOP
\end{algorithmic}

Here, $\mu$ is the trust region radius, $D(x)$ is some matrix used to define a metric on the domain of $F(x)$ and $\rho$ measures the quality of the step $\Delta x$, i.e., how well did the linear model predict the decrease in the value of the non-linear objective. The idea is to increase or decrease the radius of the trust region depending on how well the linearization predicts the behavior of the non-linear objective, which in turn is reflected in the value of $\rho$.

The key computational step in a trust-region algorithm is the solution of the constrained optimization problem
\begin{align}
        \arg\min_{\Delta x}& \frac{1}{2}\|J(x)\Delta x + F(x)\|^2 \\
        \text{such that}&\quad  \|D(x)\Delta x\|^2 \le \mu
\label{eq:trp}
\end{align}

There are a number of different ways of solving this problem, each giving rise to a different concrete trust-region algorithm. Currently Ceres, implements two trust-region algorithms - Levenberg-Marquardt and Powell's Dogleg.

\subsection{Levenberg-Marquardt}
The Levenberg-Marquardt algorithm~\cite{levenberg1944method, marquardt1963algorithm} is the most popular algorithm for solving non-linear least squares problems.  It was also the first trust region algorithm to be developed~\cite{levenberg1944method,marquardt1963algorithm}. Ceres implements an exact step~\cite{madsen2004methods} and an inexact step variant of the Levenberg-Marquardt algorithm~\cite{wright1985inexact,nash1990assessing}.

It can be shown, that the solution to~\eqref{eq:trp} can be obtained by solving an unconstrained optimization of the form
\begin{align}
        \arg\min_{\Delta x}& \frac{1}{2}\|J(x)\Delta x + F(x)\|^2 +\lambda  \|D(x)\Delta x\|^2
\end{align}
Where, $\lambda$ is a Lagrange multiplier that is inverse related to $\mu$. In Ceres, we solve for
\begin{align}
        \arg\min_{\Delta x}& \frac{1}{2}\|J(x)\Delta x + F(x)\|^2 + \frac{1}{\mu} \|D(x)\Delta x\|^2
\label{eq:lsqr}
\end{align}
The matrix $D(x)$ is a non-negative diagonal matrix, typically the square root of the diagonal of the matrix $J(x)^\top J(x)$.

Before going further, let us make some notational simplifications. We will assume that the matrix $\sqrt{\mu} D$ has been concatenated at the bottom of the matrix $J$ and similarly a vector of zeros has been added to the bottom of the vector $f$ and the rest of our discussion will be in terms of $J$ and $f$, \ie the linear least squares problem.
\begin{align}
 \min_{\Delta x} \frac{1}{2} \|J(x)\Delta x + f(x)\|^2 .
 \label{eq:simple}
\end{align}
For all but the smallest problems the solution of~\eqref{eq:simple} in each iteration of the Levenberg-Marquardt algorithm is the dominant computational cost in Ceres. Ceres provides a number of different options for solving~\eqref{eq:simple}. There are two major classes of methods - factorization and iterative.

The factorization methods are based on computing an exact solution of~\eqref{eq:lsqr} using a Cholesky or a QR factorization and lead to an exact step Levenberg-Marquardt algorithm. But it is not clear if an exact solution of~\eqref{eq:lsqr} is necessary at each step of the LM algorithm to solve~\eqref{eq:nonlinsq}. In fact, we have already seen evidence that this may not be the case, as~\eqref{eq:lsqr} is itself a regularized version of~\eqref{eq:linearapprox}. Indeed, it is possible to construct non-linear optimization algorithms in which the linearized problem is solved approximately. These algorithms are known as inexact Newton or truncated Newton methods~\cite{nocedal2000numerical}.

An inexact Newton method requires two ingredients. First, a cheap method for approximately solving systems of linear equations. Typically an iterative linear solver like the Conjugate Gradients method is used for this purpose~\cite{nocedal2000numerical}. Second, a termination rule for the iterative solver. A typical termination rule is of the form
\begin{equation}
        \|H(x) \Delta x + g(x)\| \leq \eta_k \|g(x)\|. \label{eq:inexact}
\end{equation}
Here, $k$ indicates the Levenberg-Marquardt iteration number and $0 < \eta_k <1$ is known as the forcing sequence.  Wright \& Holt \cite{wright1985inexact} prove that a truncated Levenberg-Marquardt algorithm that uses an inexact Newton step based on~\eqref{eq:inexact} converges for any sequence $\eta_k \leq \eta_0 < 1$ and the rate of convergence depends on the choice of the forcing sequence $\eta_k$.

Ceres supports both exact and inexact step solution strategies. When the user chooses a factorization based linear solver, the exact step Levenberg-Marquardt algorithm is used. When the user chooses an iterative linear solver, the inexact step Levenberg-Marquardt algorithm is used.

\subsection{Powell's Dogleg}
Another strategy for solving the trust region problem~\eqref{eq:trp} was introduced by M. J. D. Powell. The key idea there is to compute two vectors
\begin{align}
        \Delta x^{\text{Gauss-Newton}} &= \arg \min_{\Delta x}\frac{1}{2} \|J(x)\Delta x + f(x)\|^2.\\
        \Delta x^{\text{Cauchy}} &= -\frac{\|g(x)\|^2}{\|J(x)g(x)\|^2}g(x).
\end{align}
Note that the vector $\Delta x^{\text{Gauss-Newton}}$ is the solution to~\eqref{eq:linearapprox} and $\Delta x^{\text{Cauchy}}$ is the vector that minimizes the linear approximation if we restrict ourselves to moving along the direction of the gradient.

Then Powell's Dogleg method finds a vector $\Delta x$ in the two dimensional subspace defined by $\Delta x^{\text{Gauss-Newton}}$ and $\Delta x^{\text{Cauchy}}$ that solves the trust region problem. For more details on the exact reasoning and computations, please see Madsen et al~\cite{madsen2004methods}.

The key advantage of the Dogleg over Levenberg Marquardt is that if the step computation for a particular choice of $\mu$ does not result in sufficient decrease in the value of the objective function, Levenberg-Marquardt solves the linear approximation from scratch with a small value of $\mu$. Dogleg on the other hand, only needs to compute the interpolation between the Gauss-Newton and the Cauchy vectors, as neither of them depend on the value of $\mu$.

The Dogleg method can only be used with the exact factorization based linear solvers.

\section{\texttt{LinearSolver}}
Recall that in both of the trust-region methods described above, the key computational cost is the solution of a linear least squares problem of the form
\begin{align}
 \min_{\Delta x} \frac{1}{2} \|J(x)\Delta x + f(x)\|^2 .
 \label{eq:simple2}
\end{align}

Let $H(x)= J(x)^\top J(x)$ and $g(x) = -J(x)^\top  f(x)$. For notational convenience let us also drop the dependence on $x$. Then it is easy to see that solving~\eqref{eq:simple2} is equivalent to solving the {\em normal equations}
\begin{align}
H \Delta x  &= g \label{eq:normal}
\end{align}

Ceres provides a number of different options for solving~\eqref{eq:normal}.

\subsection{\texttt{DENSE\_QR}}
For small problems (a couple of hundred parameters and a few thousand residuals) with relatively dense Jacobians, \texttt{DENSE\_QR} is the method of choice~\cite{bjorck1996numerical}. Let $J = QR$ be the QR-decomposition of $J$, where $Q$ is an orthonormal matrix and $R$ is an upper triangular matrix~\cite{trefethen1997numerical}. Then it can be shown that the solution to~\eqref{eq:normal} is given by
\begin{align}
    \Delta x^* = -R^{-1}Q^\top f
\end{align}
Ceres uses \texttt{Eigen}'s dense QR decomposition routines.

\subsection{\texttt{SPARSE\_NORMAL\_CHOLESKY}}
Large non-linear least square problems are usually sparse. In such cases, using a dense QR factorization is inefficient. Let $H = R^\top R$ be the Cholesky factorization of the normal equations, where $R$ is an upper triangular matrix, then the  solution to ~\eqref{eq:normal} is given by
\begin{equation}
    \Delta x^* = R^{-1} R^{-\top} g.
\end{equation}
The observant reader will note that the $R$ in the Cholesky factorization of $H$ is the same upper triangular matrix $R$ in the QR factorization of $J$. Since $Q$ is an orthonormal matrix, $J=QR$ implies that $J^\top J = R^\top Q^\top Q R = R^\top R$.

There are two variants of Cholesky factorization -- sparse and dense. \texttt{SPARSE\_NORMAL\_CHOLESKY}, as the name implies performs a sparse Cholesky factorization of the normal equations. This leads to substantial savings in time and memory for large sparse problems. We use the Professor Tim Davis' \texttt{CHOLMOD} library (part of the \texttt{SuiteSparse} package) to perform the sparse cholesky~\cite{chen2006acs}.

\subsection{\texttt{DENSE\_SCHUR} \& \texttt{SPARSE\_SCHUR}}
While it is possible to use \texttt{SPARSE\_NORMAL\_CHOLESKY} to solve bundle adjustment problems, bundle adjustment problem have a special structure, and a more efficient scheme for solving~\eqref{eq:normal} can be constructed.

Suppose that the SfM problem consists of $p$ cameras and $q$ points and the variable vector $x$ has the  block structure $x = [y_{1},\hdots,y_{p},z_{1},\hdots,z_{q}]$. Where, $y$ and $z$ correspond to camera and point parameters, respectively.  Further, let the camera blocks be of size $c$ and the point blocks be of size $s$ (for most problems $c$ =  $6$--$9$ and $s = 3$). Ceres does not impose any constancy requirement on these block sizes, but choosing them to be constant simplifies the exposition.

A key characteristic of the bundle adjustment problem is that there is no term $f_{i}$ that includes two or more point blocks.  This in turn implies that the matrix $H$ is of the form
\begin{equation}
        H =  \left[
                \begin{matrix} B & E\\ E^\top & C
                \end{matrix}
                \right]\ ,
\label{eq:hblock}
\end{equation}
where, $B \in \reals^{pc\times pc}$ is a block sparse matrix with $p$ blocks of size $c\times c$ and  $C \in \reals^{qs\times qs}$ is a block diagonal matrix with $q$ blocks of size $s\times s$. $E \in \reals^{pc\times qs}$ is a general block sparse matrix, with a block of size $c\times s$ for each observation. Let us now block partition $\Delta x = [\Delta y,\Delta z]$ and $g=[v,w]$ to restate~\eqref{eq:normal} as the block structured linear system
\begin{equation}
        \left[
                \begin{matrix} B & E\\ E^\top & C
                \end{matrix}
                \right]\left[
                        \begin{matrix} \Delta y \\ \Delta z
                        \end{matrix}
                        \right]
                        =
                        \left[
                                \begin{matrix} v\\ w
                                \end{matrix}
                                \right]\ ,
\label{eq:linear2}
\end{equation}
and apply Gaussian elimination to it. As we noted above, $C$ is a block diagonal matrix, with small diagonal blocks of size $s\times s$.
Thus, calculating the inverse of $C$ by inverting each of these blocks is  cheap. This allows us to  eliminate $\Delta z$ by observing that $\Delta z = C^{-1}(w - E^\top \Delta y)$, giving us
\begin{equation}
        \left[B - EC^{-1}E^\top\right] \Delta y = v - EC^{-1}w\ .  \label{eq:schur}
\end{equation}
The matrix
\begin{equation}
S = B - EC^{-1}E^\top\ ,
\end{equation}
is the Schur complement of $C$ in $H$. It is also known as the {\em reduced camera matrix}, because the only variables participating in~\eqref{eq:schur} are the ones corresponding to the cameras. $S \in \reals^{pc\times pc}$ is a block structured symmetric positive definite matrix, with blocks of size $c\times c$. The block $S_{ij}$ corresponding to the pair of images $i$ and $j$ is non-zero if and only if the two images observe at least one common point.

Now, \eqref{eq:linear2}~can  be solved by first forming $S$, solving for $\Delta y$, and then back-substituting $\Delta y$ to obtain the value of $\Delta z$.
Thus, the solution of what was an $n\times n$, $n=pc+qs$ linear system is reduced to the inversion of the block diagonal matrix $C$, a few matrix-matrix and matrix-vector multiplies, and the solution of block sparse $pc\times pc$ linear system~\eqref{eq:schur}.  For almost all  problems, the number of cameras is much smaller than the number of points, $p \ll q$, thus solving~\eqref{eq:schur} is significantly cheaper than solving~\eqref{eq:linear2}. This is the {\em Schur complement trick}~\cite{brown-58}.

This still leaves open the question of solving~\eqref{eq:schur}. The
method of choice for solving symmetric positive definite systems
exactly is via the Cholesky
factorization~\cite{trefethen1997numerical} and depending upon the
structure of the matrix, there are, in general, two options. The first
is direct factorization, where we store and factor $S$ as a dense
matrix~\cite{trefethen1997numerical}. This method has $O(p^2)$ space complexity and $O(p^3)$ time
complexity and is only practical for problems with up to a few hundred
cameras. Ceres implements this strategy as the \texttt{DENSE\_SCHUR} solver.

 But, $S$ is typically a fairly sparse matrix, as most images
only see a small fraction of the scene. This leads us to the second
option: sparse direct methods. These methods store $S$ as a sparse
matrix, use row and column re-ordering algorithms to maximize the
sparsity of the Cholesky decomposition, and focus their compute effort
on the non-zero part of the factorization~\cite{chen2006acs}.
Sparse direct methods, depending on the exact sparsity structure of the Schur complement,
allow bundle adjustment algorithms to significantly scale up over those based on dense
factorization. Ceres implements this strategy as the \texttt{SPARSE\_SCHUR} solver.

\subsection{\texttt{CGNR}}
For general sparse problems, if the problem is too large for \texttt{CHOLMOD} or a sparse linear algebra library is not linked into Ceres, another option is the \texttt{CGNR} solver. This solver uses the Conjugate Gradients solver on the {\em normal equations}, but without forming the normal equations explicitly. It exploits the relation
\begin{align}
    H x = J^\top J x = J^\top(J x)
\end{align}
When the user chooses \texttt{ITERATIVE\_SCHUR} as the linear solver, Ceres automatically switches from the exact step algorithm to an inexact step algorithm.

%Currently only the \texttt{JACOBI} preconditioner is available for use with this solver. It uses the block diagonal of $H$ as a preconditioner.

\subsection{\texttt{ITERATIVE\_SCHUR}}
Another option for bundle adjustment problems is to apply PCG to the reduced camera matrix $S$ instead of $H$. One reason to do this is that $S$ is a much smaller matrix than $H$, but more importantly, it can be shown that $\kappa(S)\leq \kappa(H)$.  Ceres implements PCG on $S$ as the \texttt{ITERATIVE\_SCHUR} solver. When the user chooses \texttt{ITERATIVE\_SCHUR} as the linear solver, Ceres automatically switches from the exact step algorithm to an inexact step algorithm.

The cost of forming and storing the Schur complement $S$ can be prohibitive for large problems. Indeed, for an inexact Newton solver that computes $S$ and runs PCG on it, almost all of its time is spent in constructing $S$; the time spent inside the PCG algorithm is negligible in comparison. Because  PCG only needs access to $S$ via its product with a vector, one way to evaluate $Sx$ is to observe that
\begin{align}
  x_1 &= E^\top x \notag \\
  x_2 &= C^{-1} x_1 \notag\\
  x_3 &= Ex_2 \notag\\
  x_4 &= Bx \notag\\
  Sx &= x_4 - x_3\ .\label{eq:schurtrick1}
\end{align}
Thus, we can run PCG on $S$ with the same computational effort per iteration as PCG on $H$, while reaping the benefits of a more powerful preconditioner. In fact, we do not even need to compute $H$, \eqref{eq:schurtrick1} can be implemented using just the columns of $J$.

Equation~\eqref{eq:schurtrick1} is closely related to {\em Domain Decomposition methods} for solving large linear systems that arise in structural engineering and partial differential equations. In the language of Domain Decomposition, each point in a bundle adjustment problem is a domain, and the cameras form the interface between these domains. The iterative solution of the Schur complement then falls within the sub-category of techniques known as Iterative Sub-structuring~\cite{saad2003iterative,mathew2008domain}.

\section{Preconditioner}
The convergence rate of Conjugate Gradients  for solving~\eqref{eq:normal} depends on the distribution of eigenvalues of $H$~\cite{saad2003iterative}. A useful upper bound is $\sqrt{\kappa(H)}$, where, $\kappa(H)$f is the condition number of the matrix $H$. For most bundle adjustment problems, $\kappa(H)$ is high and a direct application of Conjugate Gradients to~\eqref{eq:normal} results in extremely poor performance.

The solution to this problem is to replace~\eqref{eq:normal} with a {\em preconditioned} system.  Given a linear system, $Ax =b$ and a preconditioner $M$ the preconditioned system is given by $M^{-1}Ax = M^{-1}b$. The resulting algorithm is known as Preconditioned Conjugate Gradients algorithm (PCG) and its  worst case complexity now depends on the condition number of the {\em preconditioned} matrix $\kappa(M^{-1}A)$.

The computational cost of using a preconditioner $M$ is the cost of computing $M$ and evaluating the product $M^{-1}y$ for arbitrary vectors $y$. Thus, there are two competing factors to consider: How much of $H$'s structure is captured by $M$ so that the condition number $\kappa(HM^{-1})$ is low, and the computational cost of constructing and using $M$.  The ideal preconditioner would be one for which $\kappa(M^{-1}A) =1$. $M=A$ achieves this, but it is not a practical choice, as applying this preconditioner would require solving a linear system equivalent to the unpreconditioned problem.  It is usually the case that the more information $M$ has about $H$, the more expensive it is use. For example, Incomplete Cholesky factorization based preconditioners  have much better convergence behavior than the Jacobi preconditioner, but are also much more expensive.

The simplest of all preconditioners is the diagonal or Jacobi preconditioner, \ie,  $M=\operatorname{diag}(A)$, which for block structured matrices like $H$ can be generalized to the block Jacobi preconditioner.

For \texttt{ITERATIVE\_SCHUR} there are two obvious choices for block diagonal preconditioners for $S$. The block diagonal of the matrix $B$~\cite{mandel1990block} and the block diagonal $S$, \ie the block Jacobi preconditioner for $S$. Ceres's implements both of these preconditioners and refers to them as  \texttt{JACOBI} and \texttt{SCHUR\_JACOBI} respectively.

For bundle adjustment problems arising in reconstruction from community photo collections, more effective preconditioners can be constructed by analyzing and exploiting the camera-point visibility structure of the scene~\cite{kushal2012}. Ceres implements the two visibility based preconditioners described by Kushal \& Agarwal as \texttt{CLUSTER\_JACOBI} and \texttt{CLUSTER\_TRIDIAGONAL}. These are fairly new preconditioners and Ceres' implementation of them is in its early stages and is not as mature as the other preconditioners described above.

\section{Ordering}
All three of the Schur based solvers depend on the user indicating to the solver, which of the parameter blocks correspond to the points and which correspond to the cameras. Ceres refers to them as \texttt{e\_block}s and \texttt{f\_blocks}. The only constraint on \texttt{e\_block}s is that there should be no term in the objective function with two or more \texttt{e\_block}s.

As we saw in Section~\ref{chapter:tutorial:bundleadjustment}, there are two ways to indicate \texttt{e\_block}s to Ceres. The first is to explicitly create an ordering vector \texttt{Solver::Options::ordering} containing the parameter blocks such that all the \texttt{e\_block}s/points occur before the \texttt{f\_blocks}, and setting \texttt{Solver::Options::num\_eliminate\_blocks} to the number \texttt{e\_block}s.

For some problems this is an easy thing to do and we recommend its use. In some problems though, this is onerous and it would be better if Ceres could automatically determine \texttt{e\_block}s. Setting \texttt{Solver::Options::ordering\_type} to \texttt{SCHUR} achieves this.

The \texttt{SCHUR} ordering algorithm is based on the observation that
the constraint that no two \texttt{e\_block} co-occur in a residual
block means that if we were to treat the sparsity structure of the
block matrix $H$ as a graph, then the set of \texttt{e\_block}s is an
independent set in this graph. The larger the number of
\texttt{e\_block}, the smaller is the size of the Schur complement $S$. Indeed the reason Schur based solvers are so efficient at solving bundle adjustment problems is because the number of points in a bundle adjustment problem is usually an order of magnitude or two larger than the number of cameras.

Thus, the aim of the \texttt{SCHUR} ordering algorithm is to identify the largest independent set in the graph of $H$. Unfortunately this is an NP-Hard problem. But there is a  greedy approximation algorithm that performs well~\cite{li2007miqr} and we use it to identify \texttt{e\_block}s in Ceres.

\section{\texttt{Solver::Options}}

\texttt{Solver::Options} controls the overall behavior of the solver. We list the various settings and their default values below.

\begin{enumerate}

\item{\texttt{trust\_region\_strategy\_type }} (\texttt{LEVENBERG\_MARQUARDT}) The  trust region step computation algorithm used by Ceres. Currently \texttt{LEVENBERG\_MARQUARDT } and \texttt{DOGLEG} are the two valid choices.

\item{\texttt{max\_num\_iterations }}(\texttt{50}) Maximum number of iterations for Levenberg-Marquardt.

\item{\texttt{max\_solver\_time\_in\_seconds }} ($10^9$) Maximum amount of time for which the solver should run.

\item{\texttt{num\_threads }}(\texttt{1})
Number of threads used by Ceres to evaluate the Jacobian.

\item{\texttt{initial\_trust\_region\_radius } ($10^4$)} The size of the initial trust region. When the \texttt{LEVENBERG\_MARQUARDT} strategy is used, the reciprocal of this number is the initial regularization parameter.

\item{\texttt{max\_trust\_region\_radius } ($10^{16}$)} The trust region radius is not allowed to grow beyond this value.
\item{\texttt{max\_trust\_region\_radius } ($10^{-32}$)} The solver terminates, when the trust region becomes smaller than this value.

\item{\texttt{min\_relative\_decrease }}($10^{-3}$) Lower threshold for relative decrease before a Levenberg-Marquardt step is acceped.

\item{\texttt{lm\_min\_diagonal } ($10^6$)} The \texttt{LEVENBERG\_MARQUARDT} strategy, uses a diagonal matrix to regularize the the trust region step. This is the lower bound on the values of this diagonal matrix.

\item{\texttt{lm\_max\_diagonal } ($10^{32}$)}  The \texttt{LEVENBERG\_MARQUARDT} strategy, uses a diagonal matrix to regularize the the trust region step. This is the upper bound on the values of this diagonal matrix.

\item{\texttt{max\_num\_consecutive\_invalid\_steps } (5)} The step returned by a trust region strategy can sometimes be numerically invalid, usually because of conditioning issues. Instead of crashing or stopping the optimization, the optimizer can go ahead and try solving with a smaller trust region/better conditioned problem. This parameter sets the number of consecutive retries before the minimizer gives up.

\item{\texttt{function\_tolerance }}($10^{-6}$) Solver terminates if
\begin{align}
\frac{|\Delta \text{cost}|}{\text{cost}} < \texttt{function\_tolerance}
\end{align}
where, $\Delta \text{cost}$ is the change in objective function value (up or down) in the current iteration of Levenberg-Marquardt.

\item \texttt{Solver::Options::gradient\_tolerance } Solver terminates if
\begin{equation}
    \frac{\|g(x)\|_\infty}{\|g(x_0)\|_\infty} < \texttt{gradient\_tolerance}
\end{equation}
where $\|\cdot\|_\infty$ refers to the max norm, and $x_0$ is the vector of initial parameter values.

\item{\texttt{parameter\_tolerance }}($10^{-8}$) Solver terminates if
\begin{equation}
    \frac{\|\Delta x\|}{\|x\| + \texttt{parameter\_tolerance}} < \texttt{parameter\_tolerance}
\end{equation}
where $\Delta x$ is the step computed by the linear solver in the current iteration of Levenberg-Marquardt.

\item{\texttt{linear\_solver\_type }(\texttt{SPARSE\_NORMAL\_CHOLESKY})}

\item{\texttt{linear\_solver\_type }}(\texttt{SPARSE\_NORMAL\_CHOLESKY}/\texttt{DENSE\_QR}) Type of linear solver used to compute the solution to the linear least squares problem in each iteration of the Levenberg-Marquardt algorithm. If Ceres is build with \suitesparse linked in  then the default is \texttt{SPARSE\_NORMAL\_CHOLESKY}, it is \texttt{DENSE\_QR} otherwise.

\item{\texttt{preconditioner\_type }}(\texttt{JACOBI}) The preconditioner used by the iterative linear solver. The default is the block Jacobi preconditioner. Valid values are (in increasing order of complexity) \texttt{IDENTITY},\texttt{JACOBI}, \texttt{SCHUR\_JACOBI}, \texttt{CLUSTER\_JACOBI} and \texttt{CLUSTER\_TRIDIAGONAL}.

\item{\texttt{sparse\_linear\_algebra\_library } (\texttt{SUITE\_SPARSE})} Ceres supports the use of two sparse linear algebra libraries, \texttt{SuiteSparse}, which is enabled by setting this parameter to \texttt{SUITE\_SPARSE} and \texttt{CXSparse}, which can be selected by setting this parameter to $\texttt{CX\_SPARSE}$. \texttt{SuiteSparse} is a sophisticated and complex sparse linear algebra library and should be used in general. If your needs/platforms prevent you from using \texttt{SuiteSparse}, consider using \texttt{CXSparse}, which is a much smaller, easier to build library. As can be expected, its performance on large problems is not comparable to that of \texttt{SuiteSparse}.

\item{\texttt{num\_linear\_solver\_threads }}(\texttt{1}) Number of threads used by the linear solver.

\item{\texttt{num\_eliminate\_blocks }}(\texttt{0})
For Schur reduction based methods, the first 0 to num blocks are
    eliminated using the Schur reduction. For example, when solving
     traditional structure from motion problems where the parameters are in
     two classes (cameras and points) then \texttt{num\_eliminate\_blocks} would be the
     number of points.

\item{\texttt{ordering\_type }}(\texttt{NATURAL})
 Internally Ceres reorders the parameter blocks to help the
 various linear solvers. This parameter allows the user to
     influence the re-ordering strategy used. For structure from
     motion problems use \texttt{SCHUR}, for other problems \texttt{NATURAL} (default)
     is a good choice. In case you wish to specify your own ordering
     scheme, for example in conjunction with \texttt{num\_eliminate\_blocks},
     use \texttt{USER}.

\item{\texttt{ordering }} The ordering of the parameter blocks. The solver pays attention
    to it if the \texttt{ordering\_type} is set to \texttt{USER} and the ordering vector is
    non-empty.

\item{\texttt{use\_block\_amd } (\texttt{true})} By virtue of the modeling layer in Ceres being block oriented,
all the matrices used by Ceres are also block oriented.
When doing sparse direct factorization of these matrices, the
fill-reducing ordering algorithms can either be run on the
block or the scalar form of these matrices. Running it on the
block form exposes more of the super-nodal structure of the
matrix to the Cholesky factorization routines. This leads to
substantial gains in factorization performance. Setting this parameter to true, enables the use of a block oriented Approximate Minimum Degree ordering algorithm. Settings it to \texttt{false}, uses a scalar AMD algorithm. This option only makes sense when using \texttt{sparse\_linear\_algebra\_library = SUITE\_SPARSE} as it uses the \texttt{AMD} package that is part of \texttt{SuiteSparse}.

\item{\texttt{linear\_solver\_min\_num\_iterations }}(\texttt{1}) Minimum number of iterations used by the linear solver. This only makes sense when the linear solver is an iterative solver, e.g., \texttt{ITERATIVE\_SCHUR}.

\item{\texttt{linear\_solver\_max\_num\_iterations }}(\texttt{500}) Minimum number of iterations used by the linear solver. This only makes sense when the linear solver is an iterative solver, e.g., \texttt{ITERATIVE\_SCHUR}.

\item{\texttt{eta }} ($10^{-1}$)
 Forcing sequence parameter. The truncated Newton solver uses
    this number to control the relative accuracy with which the
     Newton step is computed. This constant is passed to ConjugateGradientsSolver which uses
     it to terminate the iterations when
\begin{equation}
      \frac{Q_i - Q_{i-1}}{Q_i} < \frac{\eta}{i}
\end{equation}

\item{\texttt{jacobi\_scaling }}(\texttt{true}) \texttt{true} means that the Jacobian is scaled by the norm of its columns before being passed to the linear solver. This improves the numerical conditioning of the normal equations.

\item{\texttt{logging\_type }}(\texttt{PER\_MINIMIZER\_ITERATION})

\item{\texttt{minimizer\_progress\_to\_stdout }}(\texttt{false})
By default the Minimizer progress is logged to \texttt{STDERR} depending on the \texttt{vlog} level. If this flag is
set to true, and \texttt{logging\_type } is not \texttt{SILENT}, the logging output
is sent to \texttt{STDOUT}.

\item{\texttt{return\_initial\_residuals }}(\texttt{false})
\item{\texttt{return\_final\_residuals }}(\texttt{false})
If true, the vectors \texttt{Solver::Summary::initial\_residuals } and \texttt{Solver::Summary::final\_residuals } are filled with the residuals before and after the optimization. The entries of these vectors are in the order in which ResidualBlocks were added to the Problem object.

\item{\texttt{return\_initial\_gradient }}(\texttt{false})
\item{\texttt{return\_final\_gradient }}(\texttt{false})
If true, the vectors \texttt{Solver::Summary::initial\_gradient } and \texttt{Solver::Summary::final\_gradient } are filled with the gradient before and after the optimization. The entries of these vectors are in the order in which ParameterBlocks were added to the Problem object.

Since \texttt{AddResidualBlock } adds ParameterBlocks to the \texttt{Problem } automatically if they do not already exist, if you wish to have explicit control over the ordering of the vectors, then use \texttt{Problem::AddParameterBlock } to explicitly add the ParameterBlocks in the order desired.

\item{\texttt{return\_initial\_jacobian }}(\texttt{false})
\item{\texttt{return\_initial\_jacobian }}(\texttt{false})
If true, the Jacobian matrices before and after the optimization are returned in \texttt{Solver::Summary::initial\_jacobian } and \texttt{Solver::Summary::final\_jacobian } respectively.

The rows of these matrices are in the same order in which the ResidualBlocks were added to the Problem object. The columns are in the same order in which the ParameterBlocks were added to the Problem object.

Since \texttt{AddResidualBlock } adds ParameterBlocks to the \texttt{Problem } automatically if they do not already exist, if you wish to have explicit control over the column ordering of the matrix, then use \texttt{Problem::AddParameterBlock } to explicitly add the ParameterBlocks in the order desired.

The Jacobian matrices are stored as compressed row sparse matrices. Please see \texttt{ceres/crs\_matrix.h } for more details of the format.

\item{\texttt{lsqp\_iterations\_to\_dump }}
 List of iterations at which the optimizer should dump the
     linear least squares problem to disk. Useful for testing and
     benchmarking. If empty (default), no problems are dumped.

\item{\texttt{lsqp\_dump\_directory }} (\texttt{/tmp})
 If \texttt{lsqp\_iterations\_to\_dump} is non-empty, then this setting determines the directory to which the files containing the linear least squares problems are written to.

\item{\texttt{lsqp\_dump\_format }}(\texttt{TEXTFILE}) The format in which linear least squares problems should be logged
when \texttt{lsqp\_iterations\_to\_dump} is non-empty.  There are three options
\begin{itemize}
\item{\texttt{CONSOLE }} prints the linear least squares problem in a human readable format
  to \texttt{stderr}. The Jacobian is printed as a dense matrix. The vectors
   $D$, $x$ and $f$ are printed as dense vectors. This should only be used
   for small problems.
\item{\texttt{PROTOBUF }}
   Write out the linear least squares problem to the directory
   pointed to by \texttt{lsqp\_dump\_directory} as a protocol
   buffer. \texttt{linear\_least\_squares\_problems.h/cc} contains routines for
   loading these problems. For details on the on disk format used,
   see \texttt{matrix.proto}. The files are named \texttt{lm\_iteration\_???.lsqp}. This requires that \texttt{protobuf} be linked into Ceres Solver.
\item{\texttt{TEXTFILE }}
   Write out the linear least squares problem to the directory
   pointed to by \texttt{lsqp\_dump\_directory} as text files
   which can be read into \texttt{MATLAB/Octave}. The Jacobian is dumped as a
   text file containing $(i,j,s)$ triplets, the vectors $D$, $x$ and $f$ are
   dumped as text files containing a list of their values.

   A \texttt{MATLAB/Octave} script called \texttt{lm\_iteration\_???.m} is also output,
   which can be used to parse and load the problem into memory.
\end{itemize}

\item{\texttt{check\_gradients }}(\texttt{false})
 Check all Jacobians computed by each residual block with finite
     differences. This is expensive since it involves computing the
     derivative by normal means (e.g. user specified, autodiff,
     etc), then also computing it using finite differences. The
     results are compared, and if they differ substantially, details
     are printed to the log.

\item{\texttt{gradient\_check\_relative\_precision }} ($10^{-8}$)
  Relative precision to check for in the gradient checker. If the
  relative difference between an element in a Jacobian exceeds
  this number, then the Jacobian for that cost term is dumped.

\item{\texttt{numeric\_derivative\_relative\_step\_size }} ($10^{-6}$)
 Relative shift used for taking numeric derivatives. For finite
     differencing, each dimension is evaluated at slightly shifted
     values, \eg for forward differences, the numerical derivative is

\begin{align}
       \delta &= \texttt{numeric\_derivative\_relative\_step\_size}\\
       \Delta f &= \frac{f((1 + \delta)  x) - f(x)}{\delta x}
\end{align}

     The finite differencing is done along each dimension. The
     reason to use a relative (rather than absolute) step size is
     that this way, numeric differentiation works for functions where
     the arguments are typically large (e.g. $10^9$) and when the
     values are small (e.g. $10^{-5}$). It is possible to construct
     "torture cases" which break this finite difference heuristic,
     but they do not come up often in practice.

\item{\texttt{callbacks }}
  Callbacks that are executed at the end of each iteration of the
     \texttt{Minimizer}. They are executed in the order that they are
     specified in this vector. By default, parameter blocks are
     updated only at the end of the optimization, i.e when the
     \texttt{Minimizer} terminates. This behavior is controlled by
     \texttt{update\_state\_every\_variable}. If the user wishes to have access
     to the update parameter blocks when his/her callbacks are
     executed, then set \texttt{update\_state\_every\_iteration} to true.

     The solver does NOT take ownership of these pointers.

\item{\texttt{update\_state\_every\_iteration }}(\texttt{false})
Normally the parameter blocks are only updated when the solver terminates. Setting this to true update them in every iteration. This setting is useful when building an interactive application using Ceres and using an \texttt{IterationCallback}.
\end{enumerate}

\section{\texttt{Solver::Summary}}
TBD


\section{A basic machine learning problem: image classification}
%The ultimate goal of supervised learning algorithms is to classify the data into correct categories.
%Classification is a very basic  yet important problem
%which is based on a training set of data containing observations (or instances) whose category memberships are known. An example is to classify a given email into ``spam" or ``non-spam" classes. Assigning a diagnosis to a given patient based on the observed characteristics of the patient (such as gender, blood pressure, presence or absence of certain symptoms, etc.) would be another example. To understand what it the image classification problem, let us propose the following question:
Classification problem is a very important part of machine learning. Its goal is to determine which class a new data belongs to based on a set of training data whose class is known. For example, in mail management, classifying an e-mail as ``spam" or ``non-spam" is a typical binary classification problem, and the classification of credit rating of credit card customers by banks belong to multiple classification problems. More specifically, in order to understand image classification problem, let us pose the following question:

\begin{itemize}
	\item Given a set of images of cat, dog and rabbit, how does a machine classify the three different classes?
\end{itemize}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=.4\textwidth, height=.2\textheight]{figures/cat-dog-1.png}
	\end{center}
	%	\caption{}
\end{figure}

\break

First of all, we introduce how an image is represented on computer. Mathematically, {gray-scale image can be considered as a matrix  in $ \mathbb{R}^{n_0\times n_0}$ shown below, where the left one is the image from human vision and the right one is the matrix represented in computer. Each entry in this $ \mathbb{R}^{n_0\times n_0}$ corresponds to the value of a pixel belong to $[0,255]$.
	\begin{center}
		\includegraphics[width=.4\textwidth, height=.2\textheight]{6DL/figures/gray-1.png}
	\end{center}
	%The next figure shows different results from: {human vision and computer representation}:{\color{red}Lian comments: should it be a 3D tensor?}
	%\begin{figure}[H]
	%	\begin{center}
	%		\includegraphics[width=.4\textwidth, height=.2\textheight]{figures/ImagePixels.png}
	%	\end{center}
	%\end{figure}
	A color image can be taken as 3D tensor (matrix with $3$ channels (RGB)) in $ \mathbb{R}^{n_0\times n_0 \times 3}$
	\begin{center}
		\includegraphics[width=.4\textwidth, height=.2\textheight]{6DL/figures/corlor-1.png}
	\end{center}

	%\begin{itemize}
	%	\item An image is just a big grid of numbers between [0, 255]
	%	\begin{itemize}
	%		\item e.g. 800$\times$600$\times$3 (3 channels RGB)
	%	\end{itemize}
	%\end{itemize}

	\break

	Then, let us think about the image classification problem of cat, dog and rabbit. Each image is a big vector of pixel values, for example
	$$ d=1280\times720\times 3  (\text{width} \times \text{height} \times \text{RGB channel}) \approx 3\text{M},$$
	which can be considered as a point $x \in \mathbb{R}^d$. The question becomes: given 3 different sets of points (cat, dog and rabbit) in $\mathbb{R}^d$, how does a machine classify them?
	\begin{figure}[H]
		\begin{center}
			\includegraphics[width=.3\textwidth, height=.3\textwidth]{figures/cat-dog-2.png}   \quad \quad  \quad \quad
			\includegraphics[width=.3\textwidth, height=.3\textwidth]{figures/cat-dog-3.png}
		\end{center}
	\end{figure}

	\noindent To answer this question, we consider a mathematical problem: Find $f(\cdot; \bm \theta): \mathbb{R}^d \to \mathbb{R}^3$ such that:
	$$
	f(\includegraphics[width=.07\textwidth]{figures/cat.png}; \Theta)
	\approx \begin{pmatrix}
	1\\ 0 \\ 0
	\end{pmatrix}
	,\quad
	f(\includegraphics[width=.07\textwidth]{figures/dog.png}; \Theta)
	\approx \begin{pmatrix}
	0\\ 1 \\ 0
	\end{pmatrix}
	,\quad
	f(\includegraphics[width=.07\textwidth]{figures/rabbit.png};
	\Theta) \approx
	\begin{pmatrix}
	0\\ 0 \\ 1
	\end{pmatrix}
	.
	$$
	$f(\cdot; \bm \theta)$ maps a given image to a 3-dimensional vector, which is a probability distribution  $\begin{pmatrix}
	p_1\\ p_2 \\ p_3
	\end{pmatrix}$
	, where $p_1, p_2, p_3$ are probabilities of the given image being cat, dog, rabbit, respectively. For example

	$$
	f(\includegraphics[width=.07\textwidth]{figures/cat.png}; \bm \theta)
	=
	\begin{pmatrix}
	0.7\\ 0.2 \\ 0.1
	\end{pmatrix}
	\quad \Longrightarrow \quad \includegraphics[width=.07\textwidth]{figures/cat.png}= {\rm cat}.
	$$
	$f$ is a classifier that can be used to classify what a given image is.

%%	As an example, we introduce the image classification of cat, dog, rabbit above. More general, classification problem is to find some curves or surfaces to separate some sets of data. As examples, the following two images show how a circle and a curve separate different sets of points
%%	\begin{figure}[H]
%%		\begin{center}
%%			\includegraphics[width=1.5in]{NLinearS1.png}  \quad
%%			\quad  \includegraphics[width=1.5in]{NLinearS2.png}
%%		\end{center}
%%	\end{figure}
%%	\noindent It begins with a data set (training data)
%%	$$
%%	D := \{(x_j, y_j)\}_{j=1}^N,
%%	$$
%%	and
%%	$$
%%	A = \{ x_1, x_2, \cdots, x_N\} \quad \text{with} \quad A = A_1\cup A_2\cup \cdots \cup A_k, ~A_i\cap A_j = \emptyset, \forall i \neq j,
%%	$$
%%	where $A_1,...,A_k$ are a collection of subsets of A,
%%	$y_j \in \mathbb{R}^{k}$ is the label of data $x_j$ with
%%	$y_j[i]$ as the probability of $x_j$ in class $i$ or $x_j \in A_i$. If $x_j \in A_{i_j}$, we often choose
%%	\begin{equation}\label{key}
%%	y_j = e_{i_j},
%%	\end{equation}
%%	or we say $x_j$ has real label $i_j$.
%%
%%	\noindent Then, an classification problem can be thought as a data fitting
%%	problem in a high dimensional space $\mathbb{R}^d$: Given data $\{x_j, y_j\}_{j=1}^N$, we need to find a mapping
%%	$f:  \mathbb R^{d}\mapsto \mathbb R^k,$
%%	such that, for a given data $(x_j,y_j)$,
%%	\begin{equation}\label{eq:idealouput}
%%	f(x_j)\approx y_j = e_{i_j} \in \mathbb R^k,
%%	\end{equation}
%%	for all $x_j \in A$.
%%	For the general setting above, we use a probatilistic model for understanding the
%%	output $f(x) \in \mathbb{R}^{k}$ as a discrete
%%	distribution on $\{1, \cdots,k\}$, with $[f(x)]_i$ as the probability
%%	for $x$ in the class $i$, namely
%%	\begin{equation}
%%	\label{distrib}
%%	0 \le [f(x)]_i \le 1,\quad
%%	\sum_{i=1}^k  [f(x)]_i=1.
%%	\end{equation}
%%	Then, we choose
%%	\begin{equation}\label{eq:maxchoose}
%%	\mathop{\arg\max}_{i}\{[f(x)]_i~:~ i = 1:k\},
%%	\end{equation}
%%	as the label for a data $x$, which is ideally close to
%%	\eqref{eq:idealouput}.  The remaining key issue is the construction of
%%	the classification mapping $f$.
%%
%%
%%	In order to find a good $f$, we consider to solve the following {optimization problem}
%%	$$\mathcal L(\bm \theta):= \mathbb E_{(x,y)\sim \mathcal D} [\ell(f(x;\theta),y)]$$
%%	which defines the ideal loss function which measures the distance over all possible data for the parameterized function model $f(x;\theta)$. However, in practice, we can only have the finite sampled data set $D = \{ (x_j,y_j)\}_{j=1}^N$ from the original data distribution $\mathcal D$.
%%	Based on the statistical mechanism, we approximate the expectation in the ideal loss function by sampling, which leads to the following approximation of $\mathcal L(\theta)$
%%	\begin{equation}\label{key}
%%	\mathcal L(\bm \theta)\approx L(\theta) := \frac{1}{N} \sum_{j=1}^N \ell(f(x_j,\theta), y_j).
%%	\end{equation}
%%	As examples, two commonly used distances are
%%	\begin{itemize}
%%		\item $\ell^2$ distance:
%%		$$
%%		\ell(y_j,f(x_j; \bm \theta)) = \|y_j - f(x_j; \bm \theta)\|^2.
%%		$$
%%		\item KL-divergence distance:
%%		$$
%%		\ell(y_j, f(x_j; \bm \theta)) = \sum_{i=1}^k [y_j]_i \log\frac{[y_j]_i }{[f(x_j;\bm \theta)]_i}.
%%		$$
%%	\end{itemize}
%%
%%
%%In order to verify the performance of trained model $f$, there will be a test set
%%\begin{equation}\label{key}
%%T = \{ (x_j ,y_j) \}_{j=1}^M,
%%\end{equation}
%%with the same dimension of training data $D$, but is not known before
%%we finish the training process.
%%
%%In the following, we briefly introduce linear models and logistic regression in order to better understand the classification problem.
%%\subsection{Linear models: decision boundaries given by hyper-planes}
%%This is a demo of classification for binary and multi-classed cases.
%%The original data sets are
%%\begin{figure}[H]
%%	\begin{center}
%%		\includegraphics[width=.35\textwidth]{lr2noboundary} \quad \quad  \includegraphics[width=.35\textwidth]{lr3noboundary}
%%	\end{center}
%%\end{figure}
%%Then the decision boundaries given by hyper-planes would be:
%%\begin{figure}[H]
%%	\begin{center}
%%		\includegraphics[width=.35\textwidth]{lr2boundary} \quad \quad  \includegraphics[width=.35\textwidth]{lr3boundary}
%%	\end{center}
%%\end{figure}
%%
%%\break
%%\subsection{How to find these hyper-planes:  logistic regressions}
%%For a collection of subsets $A_1,...,A_k\subset \mathbb{R}^d$, try to find
%%\begin{equation}
%%\label{Wb}
%%W=
%%\begin{pmatrix}
%%w_1\\
%%\vdots\\
%%w_k
%%\end{pmatrix}
%%\in \mathbb{R}^{k\times d},
%%b=
%%\begin{pmatrix}
%%b_1\\
%%\vdots\\
%%b_k
%%\end{pmatrix}
%%\in \mathbb{R}^{k\times d},
%%\end{equation}
%%such that,   for each $1\le i\le k$ and $ j \neq i$
%%\begin{equation}
%%\label{eq:3}
%%(Wx+b)_i > (Wx+b)_j,\ \forall x\in A_i,
%%\end{equation}
%%or
%%\begin{equation}
%%\label{eq:3}
%%w_ix+b_i > w_jx+b_j,\ \forall x\in A_i.
%%\end{equation}
%
%More details of logistic regression will be discussed later.

\break
\section{Some popular data sets in image classification}
In this subsection, we will introduce some popular and standard data sets
in image classification. The most popular four data sets, MNIST, CIFAR-10, CIFAR-100 and ImageNet are shown in Table \ref{popular_dataset}.

\begin{table}[H]
	\centering
	\begin{tabular}{|c|c|c|c|c|c|}
		\hline
		% after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
		dataset &  training (N) & test (M)   & classes (k) & channels (c)& input size (d)\\\hline
		MNIST	&	60K	&	10K	&	10	& Greyscale & 28*28  \\\hline
		CIFAR-10	&	50K 	&	10K	&	10 & RGB & 32*32   \\\hline
		CIFAR-100	&	50K 	&	10K	&	100 & RGB & 32*32   \\\hline
		ImageNet	&	1.2M 	&	50K	&	1000 &	RGB & 224*224  \\\hline
	\end{tabular}
	\caption{Basic descriptions about popular datasets }
	\label{popular_dataset}
\end{table}

\break
\subsection{MNIST (Modified National Institute of Standards and Technology Database)}
MNIST\cite{lecun1998mnist} is a database for handwritten digits. It is a simple database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. In order to use MNIST for the classification problem, the following setup is often used:
\begin{itemize}
	\item Training set : $N = 60,000$;
	\item Test set : $M = 10,000$;
	\item Image size : $d =28*28*1=784$;
	\item Classes: $ k = 10$;
\end{itemize}
\begin{figure}[H]
	\begin{center}
		\includegraphics[height=.2\textheight]{mnist_short.png}
		\caption{Some images in MNIST.}
	\end{center}
\end{figure}
The following example shows  an image of handwritten digits is represented mathematically
\break
$$
x=\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist2_1.jpg}}
=
\begin{pmatrix}
  x_1\\
x_2\\
\vdots\\
x_{784}
\end{pmatrix}
\in \mathbb R^{784}.
$$
MNIST has 10 classes denoted by $\{A_k\}_{k=1}^{10}$ as shown in the following, where $A_{k}$ is the set of handwritten digits $k$ for $k=1,2,3,...,9$ and $A_{10}$ is the set of handwritten digits $0$
%\begin{equation}
%\label{A1}
%A_1=
%\left\{
%\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist1_1.jpg}},
%\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist1_2.jpg}},
%\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist1_3.jpg}}, \cdots
%\right\}
%\subset \mathbb R^{784}
%\end{equation}
\begin{equation}
  \label{A2}
A_2=
\left\{
\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist2_1.jpg}},
\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist2_2.jpg}},\cdots
%\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist2_3.jpg}}, \cdots
\right\},~
%\subset \mathbb R^{784}
A_{9}=
\left\{
\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist9_1.jpg}},
\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist9_2.jpg}},\cdots
%\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist9_3.jpg}}, \cdots
\right\},~
A_{10}=
\left\{
\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist0_1.jpg}},
\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist0_2.jpg}}, \cdots
%\adjustbox{valign=c}{\includegraphics[height=.03\textheight]{mnist0_3.jpg}}, \cdots
\right\}
\subset \mathbb R^{784}.
\end{equation}

\break
\subsection{CIFAR}
\paragraph{CIFAR-10}
CIFAR-10\cite{krizhevsky2009learning} is a set of images that can be used to teach a computer how to recognize objects. It contains 60,000 32x32 color images in 10 different classes,  with 6000 images per class. The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. In order to use CIFAR-10 for the classification problem, people usually use the following setup:
\begin{itemize}
	\item Training set : $N = 50,000$
	\item Test set : $M = 10,000$
	\item Image size : $d  = 32*32*3$
	\item Classes: $k = 10$
\end{itemize}
\begin{figure}[H]
	\begin{center}
		\includegraphics[height=.26\textheight]{cifar10.jpg}
		\includegraphics[height=0.26\textheight]{cifar100.png}
		\label{Fig: CIFAR-10}
		\caption{Left: Some images in CIFAR-10. Right: Some images in CIFAR-100.}
	\end{center}
\end{figure}

%\begin{figure}[H]
%	\begin{center}
%		\includegraphics[height=.4\textheight]{cifar10.jpg}
%		\label{Fig: CIFAR-10}
%		\caption{Some images in CIFAR-10.}
%	\end{center}
%\end{figure}
%\begin{figure}[H]
%	\begin{center}
%		\includegraphics[height=0.35\textheight]{cifar100.png}
%		\caption{Some images in CIFAR-100.}
%	\end{center}
%\end{figure}

\break

\paragraph{CIFAR-100}
CIFAR-100\cite{krizhevsky2009learning} is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. The 100 classes in the CIFAR-100 are grouped into 20 superclasses and each superclass has 5 classes. Each image comes with a ``fine" label (the class to which it belongs) and a ``coarse" label (the superclass to which it belongs). In order to use CIFAR-100 for the classification problem, people usually use the following setup:

\begin{itemize}
	\item Training set : $N = 50,000$
	\item Test set : $M = 10,000$
	\item Image size : $d = 32*32*3$
	\item Classes: $k = 100$
	\end{itemize}
%\begin{figure}[H]
%	\begin{center}
%		\includegraphics[height=0.35\textheight]{cifar100.png}
%		\caption{Some images in CIFAR-100.}
%	\end{center}
%\end{figure}

\subsection{ImageNet}
The ImageNet\cite{deng2009imagenet} project is a large visual database designed for use in visual object recognition software research. More than 1 million images have been hand labeled to indicate what objects are pictured. It is organized according to the WordNet hierarchy (currently only the nouns), in which each node of the hierarchy is depicted by hundreds and thousands of images.
Since 2010, the ImageNet runs an annual software contest, the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), where software programs compete to correctly classify and detect objects and scenes. In order to use ImageNet for the classification problem, we list the setup of ILSVRC2012 in the following
\begin{itemize}
	\item Training set : $N = 1,200,000$
	\item Test set : $M = 50,000$
	\item Image size : $d = 224*224*3$
	\item Classes: $k = 1,000$
\end{itemize}
%\begin{figure}[H]
%	\begin{center}
%		\includegraphics[height=0.25\textheight]{imagenet.jpeg}
%
%	\end{center}
%\end{figure}

\begin{figure}[H]
	\begin{center}
\includegraphics[height=0.22\textheight]{imagenet-example.png}
		\caption{Some images in ImageNet.}
	\end{center}
\end{figure}

\break
%\section{Classification and decision boundaries}
%A classification problem is to find some curves or surfaces to separate some sets of data. As examples, the following two images show how a circle and a curve separate different sets of points
%\begin{figure}[H]
%	\begin{center}
%		\includegraphics[width=1.5in]{NLinearS1.png}  \quad
%                \quad  \includegraphics[width=1.5in]{NLinearS2.png}
%	\end{center}
%\end{figure}

%\begin{figure}[H]
%	\begin{center}
%		 \includegraphics[width=.35\textwidth,
%                 height=0.3\textheight]{nlr3boundary}
%	\end{center}
%\end{figure}
%\input{Jianqing}

497-6DL/1 Machine Learning and Image Classification/1.2-ClassificationProblem.tex


% Section: Gaussian Process Regression
\section{Gaussian Process Regression} \label{sec:intro}

\subsection{Regression}

\para{Regression}
Regression is probably one of the most fundamental problems in a wide range of fields including \emph{Statistics}, \emph{Signal Processing} and \emph{Machine Learning}, etc. A regression problem is usually formulated as follows:
Given a training set $D = \{ (\textbf{x}_{i}, y_{i}) | i = 1,2,...,n \}$, we assume that $\textbf{x}_{i}, y_{i}$ have the following relationship:
	\begin{equation}
		y_{i} = f(\textbf{x}_{i}) + e
	\end{equation}
where $e$ is the error noise. By finding such $f(\cdot)$, we can predict what a corresponding $y^{*}$ is in some test case $\textbf{x}^{*}$. Note that $\textbf{x}$ can either be a vector or a scalar.

\para{Generalized Linear Model}
A widely used regression model is called Generalized Linear Model(GLM)\cite{mccullagh1984generalized}, in which a regression function can be expressed as a linear combination:
	\begin{equation}
		f(\textbf{x}) = \sum_{i=1}^{M} w_{i}\phi_i (\textbf{x})
	\end{equation}
where $\phi_{i}(x)$ is called the basis function.
In a regular GLM analysis, we have to firstly determine what our basis functions we are going to use, and subsequently can we use the training dataset to derive the parameters in the basis functions and the coefficients in the regression function.

\para{Mean Square Error}
We use a measurement called the Mean Square Error(MSE) to evaluate the performance of the regression function. It is defined as follows:
	\begin{equation}
		MSE = \frac{1}{m}\sum^{m}_{i=1} (f(\textbf{x}^{*})-y_{i}^{*})^{2}
	\end{equation}
where $f(x)$ represents the regression function. A smaller MSE represents a better regression function on a test set.

\subsection{Gaussian Process Regression}

\para{Gaussian Process}
A Gaussian Process(GP) is any distribution over functions such that any finite set of function values $f(x_1), f(x_2), ..., f(x_N)$ have a joint Gaussian distribution\cite{rasmussen2006gaussian}. It can usually be represented as
\begin{equation}
N\{ E[ f(x) ], Cov[f(x), f(x^{'})] \}
\end{equation}
where $E[ f(x) ]$ refers to its \emph{Mean Function}, and $Cov[f(x), f(x^{'})]$ refers to its \emph{Covariance Function}.

\para{Gaussian Process Regression}
Gauss Process Regression(GPR)\cite{rasmussen2006gaussian} is a popular regression method these years.
The key of this method is to model the regression function $\{ f(\textbf{x}) | \textbf{x} \in S\}$ as a GP
\begin{equation}
f(x) \gets N \{ m(\textbf{x}), K (\textbf{x},\textbf{x}^{'}) \}
\end{equation}
where $m(\textbf{x})$ is the \emph{Mean function} and $K (\textbf{x},\textbf{x}^{'})$ is the \emph{Kernel Function}.

In a GPR, we don't have to derive the exact form of the regression function, we just need to determine the form of the above two functions.
As introduced in GP, the \emph{Kernel Function} $K (\textbf{x},\textbf{x}^{'})$ is actually the covariance between $f(\textbf{x})$ and $f(\textbf{x}^{'})$, and if it is a zero-mean GP, the covariance turns into correlation. So a \emph{Kernel Function} represents the relationship between $f(\textbf{x})$ and $f(\textbf{x}^{'})$.

By calculating the posterior probability of the desired $f(\textbf{x}^{*})$, \emph{i.e.} $p(f(\textbf{x}^{*}) | \textbf{x}^{*}, D)$, we can derive the mean value along with the standard deviation of this estimation.
On the other hand, we must noted that calculating the posterior probability will become an intractable work when we have a high-dimensional dataset. It is a need that we introduce some inference method to estimate this work. \\

To summarize, choosing a suitable \emph{Mean Functions}, \emph{Kernel Functions} as well as the \emph{Likelihood Functions} and the \emph{Inference Methods} is the key of a GPR model.
Rasmussen's \emph{Gaussian Processes for Machine Learning}\cite{rasmussen2006gaussian} has implemented some marvellous Matlab/Octave code of GPR, it is available on his website\footnote{Available at \color{blue}\href{http://www.gaussianprocess.org/gpml/code/matlab/doc/}{http://www.gaussianprocess.org/gpml/code/}} known as \textbf{\emph{GPML}}.


\chapter{Sample Spaces and Events}
In the previous chapter we discussed games that do not involve move by chance.
A theory that studies experiments, processes and interactions that are subject
to chance is called \emph{probability theory}; we are going to study this theory
in this part.

The most important assumption of probability theory is that nonetheless the
outcome of an experiment is not known in advance --- the set of all possible
outcomes is known. This set is called the \emph{sample space} or the
\emph{probability space}.

For example, if our experiment consists of tossing a coin, then the sample space
consists of two outcomes $\set{H, T}$, where $H$ stands for heads and $T$ stands
for tails.
\begin{exercise}
  Write the sample space for the experiment consisting of tossing two coins.
\end{exercise}
Each element of the sample space is called an \emph{outcome} or an
\emph{elementary event}. Typically, we are interested in observing several
outcomes; e.g., in the experiment consisting of tossing five coins, the set
$\set{HTTTT, THTTT, TTHTT, TTTHT, TTTTH}$ describe all possible outcomes when
exactly one coin shows heads. We say that a set of outcomes is an \emph{event}.

Another assumption of probability theory is that every outcome $\omega$ of a
sample space $\Omega$ is assigned some probability $\Distribution{D}(\omega)$;
intuitively, $\Distribution{D}(\omega)$ is the likelihood that the outcome
$\omega$ occurs in the experiment. It is convenient to normalize probabilities,
so we require that $0 \le \Distribution{D}(\omega) \le 1$ for all $\omega \in
\Omega$ and the sum of $\Distribution{D}(\omega)$ for $\omega \in \Omega$ is
equal to $1$.
(In the sequel, we use $\sum_{x \in X} f(x)$ to describe the sum of $f(x)$ for
$x \in X$; see \Cref{section:generalized-sum} for the formal definition.)
\nomenclature[C]{$\sum_{i \in S ~:~ P(i)} \alpha_i$}{denotes $\alpha_{i_1} +
\dots + \alpha_{i_k}$, where $\set[P(i)]{i \in S} = \set{i_1, \dots, i_k}$}
The function $\Distribution{D}$ is called a \emph{probability distribution} on
$\Omega$.

The pair of a sample space and a probability distribution on the space is called
a \emph{finite discrete probability space}.

We can extend the notion of probability from elementary events to all events as
follows. Let $E \subseteq \Omega$ be an event in the finite discrete probability
space $(\Omega, \Distribution{D})$. Then $\Pr_{\Distribution{D}}(E)$ denotes
$\sum_{\omega \in E} \Distribution{D}(\omega)$.

\begin{exercise}
  In many cases we consider \emph{uniform distribution} on a set $\Omega$, the
  distribution $\Uniform{\Omega}$ such that all the outcomes are equally
  likely.
  Let $\Omega = \set{HH, HT, TH, TT}$ and $\Uniform{\Omega}$ be the
  uniform distribution on $\Omega$. Find the probability of the event
  $\set{HT, TH, TT}$, and give an informal interpretation of the answer.
\end{exercise}

\section{Basic Principles}

Events are subsets of the sample space; hence, they can be combined using the
standard set operations. So it is natural to ask whether the probabilities
$\Pr_{\Distribution{D}}(A \cup B)$ and $\Pr_{\Distribution{D}}(A \cap B)$ can be
expressed in terms of $\Pr_{\Distribution{D}}(A)$ and $\Pr_{\Distribution{D}}(B)$.

\begin{theorem}[The Additive Principle]
  Let $(\Omega, \Distribution{D})$ be a finite discrete probability space, and let $A, B
  \subseteq \Omega$ be two disjoint events. Then
  $\Pr_{\Distribution{D}}(A \cup B) = \Pr_{\Distribution{D}}(A) +
  \Pr_{\Distribution{D}}(B)$.
\end{theorem}
\begin{proof}[Proof Sketch, see \Cref{theorem:additive-principle}]
  Let $A = \set{\omega_{1, 1}, \dots, \omega_{1, k}}$ and
  let $B = \set{\omega_{2, 1}, \dots, \omega_{2, \ell}}$.
  Then $A \cup B = \set{\omega_{1, 1}, \dots, \omega_{1, k},
    \omega_{2, 1}, \dots, \omega_{2, \ell}}$.
  Therefore $\Pr_{\Distribution{D}}(A \cup B) =
    \Pr_{\Distribution{D}}(\omega_{1, 1}) + \dots +
    \Pr_{\Distribution{D}}(\omega_{2, k}) +
    \Pr_{\Distribution{D}}(\omega_{2, 1}) + \dots +
    \Pr_{\Distribution{D}}(\omega_{2, \ell}) =
    \Pr_{\Distribution{D}}(A) +
    \Pr_{\Distribution{D}}(B)$.
\end{proof}

\begin{exercise}
  Let $(\Omega, \Distribution{D})$ be a finite discrete probability space, and let $A
  \subseteq \Omega$ be an event. Show that
  $\Pr_{\Distribution{D}}(\Omega \setminus A) = 1 - \Pr_{\Distribution{D}}(A)$.
\end{exercise}

This result can be easily extended to the cases when $A$ and $B$ are not
disjoint.
\begin{corollary}[The Inclusion-exclusion Principle]
\label{corollary:inclusion-exclusion-probability}
  Let $(\Omega, \Pr)$ be a finite discrete probability space, and let $A, B
  \subseteq \Omega$ be two events. Then
  $\Pr_{\Distribution{D}}(A \cup B) = \Pr_{\Distribution{D}}(A) +
   \Pr_{\Distribution{D}}(B) -
   \Pr_{\Distribution{D}}(A \cap B)$.
\end{corollary}

Unfortunately, $\Pr_{\Distribution{D}}(A \cap B)$ cannot be expressed via
$\Pr_{\Distribution{D}}(A)$ and $\Pr_{\Distribution{D}}(B)$.
However, in many cases $\Pr_{\Distribution{D}}(A \cap B) =
\Pr_{\Distribution{D}}(A) \Pr_{\Distribution{D}}(B)$; if this equality holds,
we say that $A$ and $B$ are \emph{independent}.

For example, let us consider an experiment where we toss two fair coins; i.e.,
let us consider $\Omega = \set{HH, HT, TH, TT}$ and let
$\Uniform{\Omega}$ be the uniform distribution on $\Omega$. It is easy to
see that $\Pr_{\Uniform{\Omega}}(\set{HH, HT}) = 1 / 2$,
$\Pr_{\Uniform{\Omega}}(\set{HH, TH}) = 1 / 2$, and
$\Pr_{\Uniform{\Omega}}(\set{HH, HT} \cap \set{HH, TH}) =
\Pr_{\Uniform{\Omega}}(\set{HH}) =  1 / 4$. Hence, these two events are
independent.

To analyze experiments consisting of tossing several coins, we need to be able
to study products of finite discrete probability spaces.
\begin{theorem}[The Multiplicative Principle]
\label{theorem:multiplicative-principle-probability}
  Let $\Omega = \Omega_1 \times \Omega_2$ and
  let $(\Omega_1, \Distribution{D}_1)$ and
  $(\Omega_2, \Distribution{D}_2)$ be finite discrete probability spaces. Then
  $\Distribution{D} : \Omega \to \R$ such that
  $\Distribution{D}(\omega_1, \omega_2) =
  \Pr_{\Distribution{D}_1}(\omega_1) \cdot \Pr_{\Distribution{D}_2}(\omega_2)$
  is a probability distribution on $\Omega$.
  Moreover, $\Pr_{\Distribution{D}}(E_1 \times E_2) =
  \Pr_{\Distribution{D}_1}(E_1) \cdot \Pr_{\Distribution{D}_2}(E_2)$ for all
  $E_1 \subseteq \Omega_1$ and $E_2 \subseteq \Omega_2$.
\end{theorem}

Using this principle we can show that in the experiment consisting of tossing
five coins, the event where the first flip is $H$ and the event where the second
flip is $H$ are independent. Indeed, let $\Omega = \set{H, T}^5$;
$\Pr$ be the uniform distribution on $\Omega$; and
$E_1 = \set[t_1 = H]{t \in \set{H, T}^5}$ and
$E_2 = \set[t_2 = H]{t \in \set{H, T}^5}$. By
\Cref{theorem:multiplicative-principle-probability},
$\Pr(E_1) = \Pr(E_2) = 1 / 2$. Moreover,
$E_1 \cap E_2 = \set[t_1 = H \text{ and } t_2 = H]{t \in \set{H, T}^5}$;
therefore, $\Pr_{\Distribution{D}}(E_1 \cap E_2) = \frac{1}{4}$.

\section{Random Variables}
Sometimes we are more interested in some function of the result of the
experiment rather than the result itself. For example, Sasha may play Dungeons
and Dragons and be interested in his chances to roll $7$ on two dice together.
Let us formalize the question. Let $\Omega = \range{6}^2$ and $\Pr$ be a the
uniform distribution on $\Omega$. Sasha is interested in the probability of the
event $\set[x + y = 7]{(x, y) \in \range{6}^2}$.

More generally, let $(\Omega, \Distribution{D})$ be a finite discrete
probability space. Then a function $\chi : \Omega \to \R$ is called a
\emph{random variable} and
$\Pr_{\Distribution{D}}(\chi = a)$ denotes
$\Pr_{\Distribution{D}}(\set[\chi(\omega) = a]{\omega \in \Omega})$.

In the example about Dungeons and Dragons, $\chi(x, y) = x + y$ and
we are interested in $\Pr_{\Distribution{D}}(\chi = 7) =
\Pr_{\Distribution{D}}(\set{(1, 6), (2, 5), \dots, (6, 1)} = 1 / 6$.
\begin{exercise}
  Let $\Omega = \range{6}^2$ and $\Uniform{\Omega}$ be the uniform distribution
  on $\Omega$. Let $\chi : \Omega \to \R$ be the random variable such that
  $\chi(x, y) = x + y$. Find $\Pr_{\Uniform{\Omega}}(\chi = 1)$, \dots,
  $\Pr_{\Uniform{\Omega}}(\chi = 12)$.
\end{exercise}

We are going to adopt some simple additional notation, if
$\chi_1, \chi_2 : \Omega \to \R$ are random variables then
$(\chi_1 + \chi_2), (\chi_1 \cdot \chi_2) : \Omega \to \R$ are the random
variables such that $(\chi_1 + \chi_2)(\omega) = \chi_1(\omega) +
\chi_2(\omega)$ and $(\chi_1 \cdot \chi_2)(\omega) = \chi_1(\omega) \cdot
\chi_2(\omega)$.

\begin{chapterendexercises}
  \exercise Prove \Cref{corollary:inclusion-exclusion-probability}.
  \exercise Let $\Omega = \set{HH, HT, TH, TT}$ and let $\Uniform{\Omega}$ be
    the uniform distribution on $\Omega$. Show that $\set{HH}$ and $\set{TT}$
    are not independent.
  \exercise Alice is rolling a dice $n$ times, compute the probability
    that Alice sees $6$, $6$, and $6$ in three consecutive rolls.
  \exercise Alice is rolling a dice $n$ times, compute the probability
    that Alice sees $4$, $5$, and $6$ in three consecutive rolls.
\end{chapterendexercises}


% A simple template for LaTeX documents
%
% To produce pdf run:
%   $ pdflatex paper.tex
%

\documentclass[10pt, twocolumn]{article}
%\documentclass[12pt]{article}

% Begin paragraphs with new line
\usepackage{parskip}

% Change margin size
\usepackage[margin=0.5in]{geometry}

% Graphics Example:  (PDF's make for good plots)
\usepackage{graphicx}
% \centerline{\includegraphics{figure.pdf}}

% Allows hyperlinks
\usepackage{hyperref}

% Blocks of code
\usepackage{listings}
\lstset{basicstyle=\ttfamily, title=\lstname}
% Insert code like this. replace `plot.R` with file name.
% \lstinputlisting{plot.R}

% Monospaced fonts
%\usepackage{inconsolata}
% GNU \texttt{make} is a nice tool.

% Supports proof environment
\usepackage{amsthm}

% Allows writing \implies and align*
\usepackage{amsmath}

% Allows mathbb{R}
\usepackage{amsfonts}

% Numbers in scientific notation
\usepackage{siunitx}

% Use tables generated by pandas
\usepackage{booktabs}

% norm and infinity norm
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\inorm}[1]{\left\lVert#1\right\rVert_\infty}

% Statistics essentials
\newcommand{\iid}{\text{ iid }}
\newcommand{\Expect}{\operatorname{E}}
\newcommand{\Var}{\operatorname{Var}}
\newcommand{\Cov}{\operatorname{Cov}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\textbf{Binomial}
$X \sim B(n, p)$
\[
    p(k) = \binom{n}{k} p^k (1-p)^{n-k}
    \qquad k = 0, 1, \dots, n
\]
$\Expect X = np, \quad \Var X = np(1-p)$

mgf: $M_X (t) = (pe^t + 1 - p)^n$

Beta is conjugate prior, Fisher info $I(p) = \frac{1}{p(1 - p)}$

\textbf{Poisson}
$X \sim P(\lambda)$
\[
    p(k) = \frac{e^{-\lambda} \lambda^k}{k!}
    \qquad k = 0, 1, \dots
\]
$\Expect X = \lambda, \quad \Var X = \lambda$

mgf: $M_X (t) = e^{\lambda (e^t -1)}$ Use recursive relation to compute
$\Expect(X_i)$.

Gamma is conjugate prior, Fisher info $I(\lambda) = \frac{1}{\lambda}$

\textbf{Normal}
$X \sim N(\mu, \Sigma)$, $\Sigma$ positive definite
\[
    f(x) = \frac{\exp\{ - \frac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu) \}}
        {(2\pi)^{\frac{k}{2}} \sqrt{\det(\Sigma)}}
        = \frac{1}{\sqrt{2 \pi} \sigma} e^{-\frac{(x - \mu)^2}{2
        \sigma^2}}
\]
mgf: $M_X (t) = \exp (\mu' t + \frac{1}{2} t' \Sigma t)$

Normal is conj. prior, Fisher info $I(\mu, \sigma^2) =
[\begin{smallmatrix}
        1 / \sigma^2 & 0 \\
        0 & 1 / 2\sigma^4 \\
\end{smallmatrix}]$

\textbf{Beta}
$ X \sim \text{Beta}(\alpha, \beta)$
\[
    f(x) = \frac{x^{\alpha-1}(1 - x)^{\beta-1}}{B(\alpha, \beta)}
    \qquad 0 \leq x \leq 1
\]
$\Expect X = \frac{\alpha}{\alpha + \beta},
\quad \Var X = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$

using the beta function:
\[
    B(\alpha, \beta) =
    \frac{\Gamma(\alpha) \Gamma(\beta)}{\Gamma(\alpha+ \beta)} =
    \int_0^1 t^{\alpha -1} (1-t)^{\beta - 1}dt
\]

\textbf{Gamma}
$X \sim \text{Gamma}(\alpha, \beta)$
\[
    f(x) = \frac{\beta^\alpha x^{\alpha-1} e^{-\beta x}}{\Gamma(\alpha)}
    \qquad x > 0
\]
$\Expect X = \frac{\alpha}{\beta},
\quad \Var X = \frac{\alpha}{\beta^2}$

mgf: $M_X (t) = (1 - \frac{t}{\beta})^{-\alpha}, t < \beta$

$X \sim \text{Gamma}(\alpha, \beta) \iff \beta X \sim \text{Gamma}(\alpha, 1)$

$X_i \iid \text{Gamma}(\alpha_i, \beta)$, then
\[
    \sum X_i \sim \text{Gamma}(\sum \alpha_i, \beta)
\]
Gamma function: $\Gamma(\alpha) = \int_0^\infty t^{\alpha-1} e^{-t} dt$.

$\Gamma(\frac{1}{2}) = \sqrt{\pi}$.

$\Gamma(\alpha + 1) = \alpha \Gamma(\alpha)$

$\Gamma(k) = (k-1)!$ for $k$ positive integer.

\textbf{Exponential}
Special case: $X \sim \text{Exp}(\lambda) \equiv \text{Gamma}(1, \lambda)$

$
f(x) = \lambda e^{-\lambda x},
\quad x > 0
\qquad \Expect X = \frac{1}{\lambda},
\quad \Var X = \frac{1}{\lambda ^2}
$

CDF $F(x) = 1 - e^{-\lambda x}$

\textbf{Chi square}
Special case: $X \sim \chi^2_n \equiv \text{Gamma}(\frac{n}{2}, \frac{1}{2})$

$
f(x) \propto x^{\frac{n}{2} - 1} e^{\frac{-x}{2}},
\quad x > 0
\qquad \Expect X = n,
\quad \Var X = 2n
$

Let $Z_i$ be iid $N(0, 1)$.

$\sum_{i=1}^n Z_i^2 \sim \chi^2_n$

Noncentral $\chi^2$. Let $Y \sim N(\mu, I)$ be an $n$ vector. Then
\[
    \norm{Y}^2 \sim \chi^2_n(\norm{\mu}^2)
\]

\textbf{F}
\[
    F(m, n) \equiv \frac{\frac{\chi^2_m}{m}}
        {\frac{\chi^2_n}{n}}
\]
Where numerator and denominator are independent $\chi^2$.

\textbf{T}
\[
    t(n) = \frac{N(0, 1)}
    {\sqrt{\frac{\chi^2_n}{n}}}
\]
Where numerator and denominator are independent.

\vspace{0.2in}
\hrule

\textbf{Transformations} If $g$ 1:1 with continuous derivatives and nonzero
Jacobian, and $Y = g(X)$, then the density
\[
    f_Y(y) = f_X(g^{-1}(y)) |J_{g^{-1}}(y)|
\]
For affine transformation $Y = AX + c$ then
\[
    f_Y(y) = f_X(A^{-1}(y - c)) |\det A|^{-1}
\]

Moment generating functions determine distribution
\[
    M_X(t) \equiv \Expect (e^{tX}),
    \quad M_X'(0) = \Expect(X)
\]
$X_i$ independently distributed $\iff$
\[
    M_{\sum X_i} (t) = \prod M_{X_i} (t)
\]
\textbf{Characteristic function}
\[
    \phi(t) = \Expect (e^{i t^T X)}
    = \Expect (\cos (t^T X)) + i \Expect(\sin(t^T X))
\]
Order statistics for sorted sample $X_{(1)}, \dots, X_{(n)}$ has pdf:
\[
    n! \prod_{i=1}^n f(X_{(i)}) \quad I(X_{(1)} < \dots < X_{(n)})
\]

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{TODO - add measure theory}

\textbf{Jensen's Inequality} if $S \subset R^k$ convex and closed, $g$ convex on $S$, $P[X
\in S] = 1$, and $\Expect X$ is finite, then $\Expect X \in S$, $\Expect
g(X)$ exists, and
\[
    \Expect g(X) \geq g(\Expect X)
\]
\textbf{Holder's Inequality} if $r, s > 1$ and $\frac{1}{r} + \frac{1}{s} =
1$ then
\[
    \Expect |XY| \leq (\Expect |X|^r)^{\frac{1}{r}}(\Expect |X|^s)^{\frac{1}{s}}
\]

$T(X)$ Sufficient means the distribution of $X | T(X)$ does not depend on
$\theta$.

Factorization theorem: $T(x)$ is sufficient $\iff$
\[
    f_{\theta}(x) = h(x) g(\theta, T(x))
\]
$L_x (\theta) = p_\theta (x) = p(x, \theta)$ likelihood is function of
$\theta$, density is function of $x$.

The likelihood ratio
\[
    \lambda_x (\theta) = \frac{L_x (\theta)}{L_x (\theta_0)}
\]
is minimal sufficient. To show $T(x)$ is minimal sufficient show that it is
sufficient and a function of the likelihood $\lambda_x (\theta)$.

\textbf{Fisher information}
\[
    I(\theta) = \Expect_\theta \left[ \frac{\partial}{\partial \theta}
    \log L_X (\theta) \right]^2
    = \Expect_\theta \left[ - \frac{\partial^2}{\partial \theta^2}
    \log L_X (\theta) \right]
\]
$\text{bias } \hat{v} \equiv \Expect (\hat{v}) - v$
\[
    MSE(\hat{v}) \equiv \Expect (\hat{v} - v)^2
    = \Var (\hat{v}) + (\text{bias } \hat{v})^2
\]
\textbf{Rao-Blackwell} Let $S(X)$ be an unbiased point estimator for
$g(\theta)$. Conditioning on a sufficient statistic $T(X)$ reduces
variance.
\[
    \Var_{\theta} (S(X)) \geq \Var_{\theta} (\Expect (S(X) | T(X)))
\]
Also holds for more general convex loss function $L$:
\[
    R(\theta, S) \equiv \Expect_{\theta} L(\theta, S(X)) \geq
    \Expect_{\theta} L(\theta, \Expect( S(X) | T(X)))
\]
\textbf{Completeness} $T(X)$ is complete if $\Expect g(T(X)) = 0$
implies $g = 0$ almost surely for all $\theta$.

\textbf{Cramer Rao Inequality} Let $g: \Theta \rightarrow R$. Suppose there
exists an unbiased estimator $U(X)$, $\Expect U(X) = g(\theta)$. Then
\[
    \Var_\theta U(X) \geq
    \left( \frac{\partial g(\theta)}{\partial \theta} \right)^T
    I(\theta)^{-1}
    \left( \frac{\partial g(\theta)}{\partial \theta} \right)
\]
Basu's Theorem - If $T(X)$ complete sufficient statistic and $A(X)$ is
ancillary then $A(X)$ and $T(X)$ are independent.

\textbf{Lehmann - Scheffe} Suppose $T(X)$ is complete sufficient. Then
there exists unique unbiased estimator $\Expect h(T(x))$ of $g(\theta) \in
R$ with smallest variance (MVUE).

\textbf{Exponential Families} $T(x)$ is natural sufficient statistic and is
complete sufficient if the $k$ parameter exponential family is full rank.
\[
    p(x, \theta) = h(x) \exp \{ \eta(\theta)^T T(x) - B(\theta) \}
\]
Canonical form model indexed by $\eta$.
\[
    q(x, \eta) = h(x) \exp \{ \eta^T T(x) - A(\eta) \}
\]
\[
    \dot{A} (\eta) = \Expect_\eta (T(X)) \quad
    \ddot{A} (\eta) = I(\eta) = \Var_\eta (T(X))
\]
Then moment generating function for $T(X)$ is
\[
    M_{T(X)}(t) = \exp \{A(t + \eta) - A(\eta) \}
\]
Equivalent statements useful for GLM's such as $Y \sim N(X \beta,
\sigma_0^2 I)$, where $Z$ is $n \times p$:

1. $I(\beta) = \frac{1}{\sigma_0^2} X^T X$ positive definite
2. rank$(X) = p$
3. model is identifiable. More generally another equivalent statement is
$\Var (T(X)) = \ddot{A} (\eta)$ is positive definite.

\subsection*{Decision Theory}

\textbf{Decision rule} $\delta: \mathcal{X} \rightarrow \mathcal{A}$, where
$\delta \in \mathcal{D}$, the space of possible decision rules and
$\mathcal{A}$ is the action space.

\textbf{Loss function} $l: \Theta \times \mathcal{A} \rightarrow \mathbb{R}^+$
Posterior mean minimizes square error loss; median minimizes absolute loss.

\textbf{Risk function} $R: \Theta \times \mathcal{D} \rightarrow
\mathbb{R}^+$ expected loss for a particular value of $\theta$
\[
    R(\theta, \delta) = \Expect_\theta l(\theta, \delta(X)) = \int
    l(\theta, \delta(x)) \cdot p_\theta (x) dx
\]
Bayes setup:
\[
    \pi(\theta | x) = \frac{p_\theta (x) \pi(\theta)}{m(x)}
\]
\textbf{Bayes decision rule} If there exists $\delta_\pi \in \mathcal{D}$
w.r.t prior $\pi$ such that
\[
    r(\pi, \delta_\pi) = \inf_{\delta \in \mathcal{D}} r(\pi, \delta)
\]
To find Bayes rule minimize the posterior risk:
\[
    \delta_\pi (x) = \min_{a \in \mathcal{A}} r_\pi (a | x)
\]

\textbf{Bayes risk} $r_\pi : \mathcal{D} \rightarrow
\mathbb{R}^+$ expected loss for fixed prior $\pi$
\[
    r_\pi (\delta) = \Expect_\pi R(\theta, \delta)
    = \int_\Theta R(\theta, \delta) \pi (d \theta)
    = \int_\mathcal{X} r_\pi(\delta(x) | x) m(x) dx
\]
To find Bayes risk: 1) find the Bayes rule 2) compute the risk function 3)
take the expectation of the risk wrt prior $\pi$.

\textbf{Minimax} decision rule $\delta^*$ minimizes the worst case
scenario, satisfies
\[
    \sup_{\theta} R(\theta, \delta^*) = \inf_{\delta} \sup_{\theta}
    R(\theta, \delta)
\]
To show $\delta^*$ is minimax, first check for constant risk $R(\theta,
\delta^* = c$ for all $\theta$, then find a
prior $\pi$ such that $\delta^*$ is the Bayes rule. This $\pi$ is least
favorable. More generally can find
a sequence of priors $(\pi_k)$ such that the Bayes risk $r_{\pi_k}
(\delta_{\pi_k}) \rightarrow c$.

\newpage
\subsection*{Asymptotics}

\textbf{Almost Sure convergence}
\[
    X_n \xrightarrow{a.s.} X \quad \text{means} \quad P(X_n \rightarrow X)
    = 1
\]
Theorem: $\iff P(\sup_{m \geq n} |X_m - X| > \epsilon) \rightarrow 0
\quad \forall \epsilon > 0$.

\textbf{Convergence in Probability}
\[
    X_n \xrightarrow{p} X
\]
means that $P( |X_n - X| > \epsilon ) \rightarrow 0$ for all $\epsilon > 0$.

\textbf{Generalized Chebychev Inequality}
Let $X$ be a r.v. and $g$ be a
nonnegative function increasing on the range of $X$. Then
\[
    P(X \geq a) \leq \frac{\Expect g(X)}{g(a)}
\]

\textbf{Borel Cantelli Lemma}

\textbf{Hoeffding Inequality}
Let $X_1, \dots, X_n$ be independent (not necessarily iid) with $a_i \leq
X_i \leq b_i$ and $\Expect X_i = 0$. Then
\[
    P \left( \sum_{i=1}^n X_i \geq \eta \right) \leq
    \exp \left\{ -\frac{2\eta^2}{\sum_{i=1}^n (b_i - a_i)^2} \right\}
\]

\newpage
\subsection*{Multivariate Normal}

log likelihood for $k$ vector $x \sim N(\mu, \Sigma)$
\[
    l_x = -\frac{k}{2} \log 2 \pi - \frac{1}{2}
    \{ \log \det \Sigma + (x - \mu)^T \Sigma^{-1} (x - \mu) \}
\]
Stein's formula: $X \sim N(\mu, \sigma)$
\[
    \Expect (g(X) (X - \mu)) = \sigma^2 \Expect(g'(X))
\]
assuming these expectations are finite.

$X \sim N(\mu, \Sigma)$, $A$ an $m \times n$ matrix,
then
\[
    AX \sim N(A \mu, A \Sigma A^t)
\]
For $\Sigma$ full rank it's possible to transform between $Z \sim
N(0, I)$ and $X$:
\[
    X = \Sigma^{1/2} Z + \mu \qquad Z = \Sigma^{-1/2} (X - \mu)
\]
In block matrix form:
\[
    X =
    \begin{bmatrix}
        X_1 \\
        X_2 \\
    \end{bmatrix}
    \sim N \left(
    \begin{bmatrix}
        \mu_1 \\
        \mu_2 \\
    \end{bmatrix}
    ,
    \begin{bmatrix}
        \Sigma_{11} & \Sigma_{12} \\
        \Sigma_{21} & \Sigma_{22} \\
    \end{bmatrix}
\right)
\]
Assuming $\Sigma_{11}$ is positive definite then the conditional
distribution
\[
    X_2 | X_1 \sim N(\mu_2 + \Sigma_{21} \Sigma_{11}^{-1} (X_1 - \mu_1),
    \Sigma_{22} - \Sigma_{21} \Sigma_{11}^{-1} \Sigma_{12})
\]

\subsection*{Conditional Distributions}

Conditional pdf:
\[
    f_{X|Y}(x | y) \equiv \frac{f_{X, Y}(x, y)}{f_Y(y)}
\]
Iterated expectation:
\[
    E(Y) = E(E(Y | X))
\]
Conditional variance formula:
\[
    \Var(Y) = \Var(E(Y | X)) + E(\Var(Y | X))
\]

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{General Techniques}

Singular Value Decompostion (SVD) Any matrix $X$ can be written
\[
    X = UDV^T
\]
with $U, V$ orthogonal, and $D$ diagonal.

Moore Penrose Psuedoinverse $A^+$ exists uniquely for every matrix $A$.

Projection matrix $P$ are symmetric and idempotent. They have eigenvalues
either 0 or 1.
\[
    P = P^T \qquad P^2 = P
\]

Covariance of linear transformations
\[
    Cov(Ay, Bx) = A Cov(y, x) B^T
\]
Invert $2 \times 2$ matrix:
$
    A =
    [\begin{smallmatrix}
        a & b \\
        c & d \\
    \end{smallmatrix}]
$
\[
    A^{-1} =
    \frac{1}{\det (A)}
    \begin{bmatrix}
        d & -b \\
        -c & a \\
    \end{bmatrix}
\]

Sum identities:
\[
    \sum_{k=0}^{\infty} p^k = \frac{p}{1 - p} \qquad
    \sum_{k=0}^{\infty} k p^k = \frac{p}{(1 - p)^2} \qquad |p| < 1
\]

Integration by parts:
\[
    \int uv' = uv - \int u'v
\]

Matrix / Vector differentiation

$\frac{\partial A^T \beta}{\partial \beta} = A$,
$\frac{\partial \beta^T A \beta}{\partial \beta} = (A + A^t) \beta =
2A\beta$ for $A$ symmetric.

$\frac{\partial}{\partial \theta_i} \log (|A|) =
tr( A^{-1} \frac{\partial A}{\partial \theta_i}$)

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{Linear Models}

Least Squares Principle
\[
    \text{arg min}_\beta \norm{Y - X\beta}^2
\]

Normal Model
\[
    Y = X\beta + \epsilon, \qquad \epsilon \sim N(0, \sigma^2 I)
\]

Normal Equations - Any $b$ satistfying this solves the least squares
\[
    X^T X b = X^T y
\]

Gauss Markov Theorem - $\hat{\beta}$ is Best Linear Unbiased
Estimator (BLUE) of $\beta$.
\[
    \hat{\beta} = (X^T X)^{-1} X^T y \sim N(\beta, \sigma^2 (X^T X)^{-1})
\]

Estimating the variance: $\frac{\norm{y - X \hat{\beta}}^2}{\sigma^2} \sim
\chi^2_{n-p}$.
\[
    \hat{\sigma}^2 = \frac{\norm{y - X \hat{\beta}}^2}{n - p}
\]

Use t test for hypothesis testing and confidence intervals for the value of
a particular $\beta_j$ coefficient.
Let $w_{ii}$ be the $i$th diagonal entry of $(X^T X)^{-1}$.
\[
    \frac{\beta_j - \beta_j^*}{\hat{\sigma} \sqrt{w_{ii}}} \sim t_{n-p}
\]
$1 - \alpha$ Confidence intervals for new observation $Y_h$ at $x_h$ and $E[Y_h]$:
\[
    E[y_h] \approx \hat{y_h} \pm t(n-p, 1 - \frac{\alpha}{2}) \hat{\sigma}
        \sqrt{x_h^T (X^T X)^{-1} x_h}
\]
\[
    y_h \approx \hat{y_h} \pm t(n-p, 1 - \frac{\alpha}{2}) \hat{\sigma}
        \sqrt{1 + x_h^T (X^T X)^{-1} x_h}
\]
Simultaneous (Working-Hotelling) confidence interval for $\Expect (y_h)$:
\[
    \hat{y}_h \pm \sqrt{p F_{p, n - p, 1 - \alpha}} se\{ \hat{y}_h \}
\]
\[
    \frac{(\hat{\beta} - \beta)^T X^T X (\hat{\beta} - \beta) /
    p}{\hat{\sigma}^2}
    \sim F_{p, n - p}
\]

General linear tests. Partition $\beta = (\beta_1, \beta_2)$ where $\beta_1$
is an $r$ vector and $\beta_2$ is $p - r$. Null hypothesis $H_0: \beta_2 =
\beta_2^*$ (often 0), and $H_a: \beta_2 \neq \beta_2^*$. Then
$SSE_r = \norm{y - X_2 \beta_2^* - X_1 \tilde{\beta_1}}^2$ is the sum of
squared error for the reduced model and
$SSE_f = \norm{y - X \hat{\beta}}^2$ is the squared sum of error for the
full model.
Under $H_0$:
\[
    \frac{\frac{SSE_r - SSE_f}{p - r}}
         {\frac{SSE_f}{n - p}}
         \sim F_{p-r, n-p}
\]

Alternate forms of linear test, and testing a linear combination if $R\beta
= r$, for $R$ full rank $s \times p$ matrix.
\[
    \frac{(R \hat{\beta} - r)^T (R(X^T X)^{-1} R^T)^{-1} (R \beta - r) / s}
    {\hat{\sigma}^2} \sim F_{s, n-p}
\]

\subsection*{Model selection and diagnostics}

$SSTO = \sum_{i=1}^n (y_i - \bar{y}) = \norm{y - \bar{y} 1_n }^2
        = \norm{(I - J)y}^2$

$SSR = \sum_{i=1}^n (\hat{y}_i - \bar{y}) = \norm{\hat{y} - \bar{y} 1_n}^2
    = \norm{(H - P)y}^2$

$SSE = \sum_{i=1}^n (y_i - \hat{y}_i) = \norm{y - \hat{y}}^2
        = \norm{(I - H)y}^2$

If the model contains the intercept in the column space of $X$  then $SSTO = SSR + SSE$.

$R^2 = 1 - \frac{SSE}{SSTO}$

Adjusted $R^2_a = 1 - \frac{SSE / (n-p)}{SSTO / (n-1)}$

$AIC = n \log SSE + 2p$

$BIC = n \log SSE + p \log n$

$Cp = \frac{SSE}{MSE} - (n - 2p)$

Residuals: $\hat{\epsilon}_i = y_i - \hat{y}_i$

Studentized residuals (\texttt{rstandard} in R):
\[
    \gamma_i =
    \frac{\hat{\epsilon}_i}{ s \{ \hat{\epsilon}_i \} } =
    \frac{\hat{\epsilon}_i}{\hat{\sigma} \sqrt{1 - h_{ii}}}
\]

Prediction sum of squares (PRESS) is the same as leave one out cross
validation (LOOCV). Prediction error on $i$th observation is called deleted
residuals:
\[
    y_i - \hat{y}_{i (-i)} = \frac{y_i - \hat{y}_i}{1 - H_{ii}}
\]
Works for ridge regression also, letting
$H = X(X^T X + \lambda I)^{-1} X^T$.

Studentized deleted residuals:
\[
    t_i = \frac{\hat{\epsilon}_i}{\sqrt{MSE_{(-i)} (1 - h_{ii})}} \sim t_{n - p -1}
\]
Where $MSE_{(-i)} = SSE_{(-i)} / (n - 1 - p)$ and
$SSE_{(-i)} = SSE - \frac{\hat{\epsilon}_i}{1 - h_{ii}}$ can be used to
calculate without refitting model.

\subsection*{ANOVA}

Three principles of experimental design: 1) Replication 2) Randomization 3)
Blocking

One way ANOVA with $n$ total observations, $K$ groups:

{
\centering
\begin{tabular}{lll}
    SS   &  & DF     \\
    SSTR & $\sum_{j=1}^K n_j (\bar{y_{j \cdot}} - \bar{y_{\cdot \cdot}})^2$  & K - 1 \\
    SSE  & $\sum_{i=1}^n (y_{ij} - \bar{y_{j \cdot}})^2$  & n - K \\
    SSTO & $\sum_{i=1}^n (y_{ij} - \bar{y_{\cdot \cdot}})^2$  & n - 1
\end{tabular}
}

Contrasts are sums of the form $\Phi = \sum_{i=1}^K c_i \mu_i$ with
$\sum_{i=1}^K c_i = 0$.
Tukey's works for all pairwise contrasts.
Scheffe's and extended Tukey works for all contrasts.
Bonferroni's is for a limited number of pre specified contrasts.

\textbf{Ridge Regression} for $\lambda > 0$ solves
\[
    \min_\beta ||Y - X\beta||^2 + \lambda ||\beta||^2
\]

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\textbf{Linear mixed models}
\[
    y = X \beta + Z \alpha + \epsilon
\]
Standard assumptions:

$\Expect ( \alpha) = 0$, $\Expect ( \epsilon) = 0$,
$\Cov(\epsilon, \alpha) = 0$

$\alpha \sim N(0, G), \epsilon \sim N(0, R)$ are jointly normal.

Marginal model $y \sim N(X \beta, V)$ where $V = R + ZGZ'$.

The idea of reduced maximum likelihood (REML) is to first estimate the random components of the model.
This is done through a transformation of the data to create a new model
containing only the random components, and no fixed components.

Let $\xi = b' \beta + a' \alpha$ be a mixed effect. These are what we're
interested in estimating. We call them predictions rather than estimations
because we're predicting a random component.

BLUE - Best linear unbiased estimator,
$\tilde{\beta} = (X' V^{-1} X)^{-1} X' V^{-1} y$. This is the MLE of
$\beta$.

BP - Best predictor, $\Expect (\xi | y) = b' \beta + a' G Z' V^{-1} (y - X
\beta)$ a theoretical ideal that's
usually difficult or impossible to derive.

BLUP - Best linear predictor, which plugs in $\tilde{\beta}$ into BP.

EBLUP - Empirical best linear predictor, plugs in $\hat{\theta}$. This is
typically the one we compute and use.

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{Applied Lectures - 232B}

4 Jan - Matrix form for linear mixed model, assumptions

6 Jan - Examples mixed models, time series, non gaussian, marginal, max
likelihood estimation, REML

11 Jan - Lambs example, parametric bootstrap

13 Jan - Asymptotic covariance matrix of fixed effects $\hat{\beta}$,
computing examples

18 Jan - Holiday

20 Jan - Mixed model prediction, BP, BLUE, BLUP, example algebra in
discussion

25 Jan - Empirical BLUP, Fay Herriot model, jackknife

27 Jan - EBLUP for random effects, intro to GLM's

1 Feb - GLMM with several examples, Monte carlo and the E-M algorithm,
importance sampling

3 Feb - Rejection sampling, Markov Chain Monte Carlo (MCMC), Markov chain
convergence theorem, Gibb's Sampler

8 Feb - Gibbs Sampler, transition kernel, MCMC, threshold model, data
cloning

10 Feb - Midterm

\subsection*{Math Lectures - 231B}

4 Jan - Maximum likelihood, determining MLE

6 Jan - Normal examples MLE, constraints, MLE of pdf $f$ decreasing on $[0,
    \infty)$.

11 Jan - MLE maximizing $\Expect l_X (\theta_0)$, KL divergence and
properties, M estimation

13 Jan - least squares and mean absolute deviation (MAD) as examples of m
estimators, existence and uniqueness theorems for optimization, review of
exponential families

18 Jan - Holiday

20 Jan - Existence and uniqueness of MLE for canonical and curved
exponential families, review convexity

25 Jan - Long proof of existence and uniqueness of MLE

27 Jan - Z estimation aka estimating equation estimation, method of
moments, gamma example, plug in principle

1 Feb - Asymptotics, modes of convergence and associated theorems,
Borel-Cantelli lemma, generalized Chebyshev

3 Feb - Hoeffding's inequality and proof, strong, weak, and uniform
consistency

8 Feb - Consistency theorem for MLE's in canonical exp families,
consistency of M estimator

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{Applied Lectures - 232A}

30 Nov - Outliers in x and y, leverage, DFFITS, cook's distance, influence
plot, add variable plot, robust regression

25 Nov - Model selection criteria, deleted residuals, forward and backward
selection in lab

23 Nov - BIC, AIC derivations, Mallow's cp, stepwise selection algorithms,
outliers and studentized residuals

18 Nov - F test with orthogonalized X, model selection criteria,
bootstrap t method in lab

16 Nov - Multicollinearity, Variance Inflation Factor,
ridge regression, bias variance tradeoff, AIC, BIC, cross validation, proof leave
one out cross validation formula for OLS

11 Nov - Holiday

9 Nov - Linear model with random $X$, transformations of $y$ and $X$, box
cox procedure, bootstrap with percentile-t and fixed $X$ sampling, weighted
least squares

4 Nov - Interaction plots for two way ANOVA with balanced design, Linear
models with random $X$

2 Nov - Midterm

28 Oct - Kronecker product formulae for two way ANOVA

26 Oct - Kronecker product 1 way ANOVA, decomposition of two way ANOVA,
noncentral $\chi^2$ distributions for ANOVA table SSA, SSB, SSAB

21 Oct - Tukey's method for pairwise contrasts, Bonferroni's method,
definition and properties of Kronecker product

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{Math Lectures - 231A}

30 Nov - Bayesian inference, risk, decision rules, conjugate families,
Binomial, normal examples

25 Nov - Exponential families, GLM's, full rank exp families, decision
theory, Bayes risk

23 Nov - Fisher information, Cramer Rao inequality, Exponential families
and properties

18 Nov - Rao-Blackwell theorem, Lehmann-Scheffe theorem, UMVUE examples for
normal, uniform, Poisson, Fisher information

16 Nov - Minimal sufficiency, likelihood ratio, ancillary statistics,
completeness, Basu's theorem, loss functions

11 Nov - Holiday

9 Nov - Distribution of order statistics, factorization theorem, sufficient
statistics for Exponential families and uniform dist

4 Nov - Midterm

2 Nov - Location-scale families, invariance, ancillary and sufficient
statistics, order statistics, multinomial distribution

28 Oct - Transformation of discrete and continuous random variables,
Jacobian, examples with beta distributions, Dirichlet distribution

26 Oct - Jensen's and Holder's inequality, convex functions and sets,
products of normal random variables

21 Oct - Convolution formula, examples with Uniform, Gamma, Poisson,
marginal and conditional distributions for multivariate normal
\newpage

\subsection*{Problem Solving Strategies}

\textbf{Read the whole question}

Read carefully and do the right problem! If there's a hint, it should
probably be used. Early parts of a question can help for later parts, and later
parts occasionaly provide insight for earlier parts.

\textbf{First principles}

When in doubt, work from definitions.

\textbf{Look for Distributions}

Can the question be solved by knowing the distribution of some
quantity?  Ex: $\sum (x_i - \bar{x})^2$ is $\chi^2_{n-1}$ for
$x_i \sim N(\mu, 1)$.

\textbf{Fast and correct algebra}

Better to write more than to make a simple algebra mistake.
Practice common manipulations so don't have to think about them
when testing.

\begin{table}[]
    \centering
    \caption{Problems in past 231 exams - Came from a brief glance at the
    question statements. TODO- make second updated table after solving
    questions that shows which techniques are used.}
    \label{231problems}
    \begin{tabular}{rl}

        Binomial                & ******* \\
        Poisson                 & ****** \\
        Uniform                 & ****** \\
        Normal                  & ****** \\
        Gamma                   & *** \\
        Exponential             & ** \\
        Negative binomial       & * \\
        Beta                    & * \\
        Geometric               & * \\
        MLE                     & ************* \\
        asymptotic distribution & *********** \\
        Bayes estimator / risk  & ********** \\
        UMVUE / Cramer-Rao      & ********* \\
        minimiax                & ****** \\
        UMP test                & ****** \\
        linear regression       & ***** \\
        likelihood ratio        & **** \\
        Wald's test             & *** \\
        sufficient statistic    & ** \\
        Hierarchical model      & * \\
        method of moments       & * \\
        hypothesis testing      & * \\
        order statistics        & * \\

    \end{tabular}
\end{table}

\end{document}


\problemname{Jumbled Communication}

\illustration{0.4}{SDC10461-scaled.jpg}{\copyright{} NWERC Jury}%
Your best friend Adam has recently bought a Raspberry Pi and some equipment, including a wireless temperature sensor and a 433MHz receiver to receive the signals the sensors sends.
Adam plans to use the Raspberry Pi as an in-door display for his weather sensor.
As he is very good with electronics, he quickly managed to get the receiver to receive the signals of the sensor.
However, when he looked at the bytes sent by the sensor he could not make heads or tails of them.
After some hours looking through a lot of websites, he found a document explaining that his weather sensor scrambles the data it sends, to prevent it from being used together with products from other manufacturers.

Luckily, the document also describes how the sensor scrambles its communication.
The document states that the sensor applies the expression \verb|x ^ (x << 1)|
to every byte sent.  The \verb|^| operator is bit-wise XOR\footnote{In bit-wise
XOR, the $i$th bit of the result is $1$ if and only if exactly one of the two
arguments has the $i$th bit set.}, e.g., $\verb|10110000 ^ 01100100| =
\verb|11010100|$.
The \verb|<<| operator is a (non-circular) left shift of a byte value\footnote{In
\texttt{x <{}< $j$}, the bits of \texttt{x} are moved $j$ steps to the left.
The $j$ most significant bits of $x$ are discarded, and $j$ zeroes are added
as the least significant bits of the result.}, e.g.,
$\verb|10111001 << 1| = \verb|01110010|$.

In order for Adam's Raspberry Pi to correctly interpret the bytes sent by the weather sensor, the transmission needs to be unscrambled.
However, Adam is not good at programming (actually he is a pretty bad programmer). So he asked you to help him and as a good friend, you are always happy to oblige.
Can you help Adam by implementing the unscrambling algorithm?

\section*{Input}

The input consists of:
\begin{itemize}
\item one line with an integer $n$ ($1 \le n \le 10^5$), the number of bytes in the message sent by the weather sensor;
\item one line with $n$ integers $b_1, \ldots, b_n$ ($0 \leq b_i \leq 255$ for all $i$), the byte values of the message.
\end{itemize}

\section*{Output}

Output $n$ byte values (in decimal encoding), the unscrambled message.