\section{Gaussian processes and the multivariate normal distribution}

Gaussian processes generalize the multivariate normal distribution from vectors to functions, like the multivariate normal distribution generalizes the univariate normal distribution from scalars to vectors. The progression is as follows:
\begin{equation}
    \begin{array}{ll}
        y\sim\textup N(\mu,V): & \textup{$y$, $\mu$, $V$ are scalars}\\\\
        \vec y\sim\textup N(\vec \mu,C): & \textup{$\vec y$ and $\vec \mu$ are vectors, $C$ is a matrix}\\\\
        f\sim\textup{GP}(M, C): & \textup{$f$ and $M$ are functions of one variable, $C$ is a function of two variables}
    \end{array}
\end{equation}

One of the nice things about all Gaussian distributions (parameterized by mean and covariance) is that they're easy to marginalize. For example, each element of a vector with a multivariate normal distribution has a univariate normal distribution:
\begin{eqnarray*}
    \vec y\sim\textup N(\vec \mu,C)\\
    \Rightarrow \vec y_i\sim\textup N(\vec \mu_i,C_{i,i}),
\end{eqnarray*}
and any subvector of a vector with a multivariate normal distribution has a multivariate normal distribution:
\begin{eqnarray*}
    \vec y\sim\textup N(\vec \mu,C)\\
    \Rightarrow \vec y_{i_1\ldots i_2}\sim\textup N(\vec \mu_{i_1\ldots i_2},C_{i_1\ldots i_2,i_1\ldots i_2}).
\end{eqnarray*}

This marginalizability applies to GP's as well. If $\vec x$ is a vector of values,
\begin{equation}
    \begin{array}{l}
        f\sim\textup{GP}(M, C)\\\\
        \Rightarrow f(\vec x) \sim\textup N(M(\vec x), C(\vec x,\vec x)).
    \end{array}
\end{equation}
In other words, any evaluation of a Gaussian process realization has a multivariate normal distribution. Its mean is the corresponding evaluation of the associated mean function, and its covariance is the corresponding evaluation of the associated covariance function. You can probably start to see why this fact is important for working with GPs on a computer.

\subsection{Observations}
As mentioned earlier, if $f$ has a GP prior and normally-distributed observations of $f$ are made at a finite set of values, $f$'s posterior is a Gaussian process also:
\begin{eqnarray*}
    \left.\begin{array}{l}
        d_i \stackrel{\tiny{\textup{ind}}}{\sim} \textup{N}(f(o_i), V_i)\\
        f \sim \textup{GP}(M,C)
    \end{array}\right\}\Rightarrow f|d \sim \texttt{GP}(M_o, C_o).
\end{eqnarray*}
Denoting by $o$ the array of observation values $[o_0\ldots o_{N-1}]$ and $V$ a matrix with observation variances $[V_0\ldots V_{N-1}]$ on its diagonal, $M_o(x)$ and $C_o(x,y)$ are as follows for arbitrary input vectors $x$ and $y$:
\begin{eqnarray*}
    M_o(x) = M(x) + C(x,o)[C(o,o) + V]^{-1}(f(o)-M(o))\\
    C_o(x,y) = C(x,y) - C(x,o)[C(o,o) + V]^{-1}C(o,y).
\end{eqnarray*}

\subsection{Low-rank observations}

If $C(o,o)+V$ is singular, some elements of $f(o)$ can be computed from others with no uncertainty. In other words, there exists a partition $[o_*, o_{**}]$ of $o$ and corresponding partition of the data such that $C(o_*,o_*)+V$ is full-rank and
\begin{eqnarray*}
    f(o_{**}) = M_{o_*}(o_{**}),
\end{eqnarray*}
where $M_{o_*}$ can be computed from the formula above.

In such cases, this package's strategy is to observe $f$ at a subvector $o_*$ of $o$, such that $C(o_*,o_*)+V$ is full-rank but if any elements were added to $o_*$ it would pick up a very small eigenvalue. The function \function{predictive_check} optionally checks the remaining data values $d_{**}$ against $M_{o_*}(o_{**})$, and raises an error if the two aren't equal up to a user-defined threshold.

This threshold is parameter \code{relative_precision} from \texttt{Covariance}'s init method, multiplied by the largest value on the diagonal of $C(o_*,o_*)+V$; intuitively, the maximal variance of $f(o_{**})$ given $f(o_*)$ is a multiple of the maximal variance of $f(o_*)$.

\section{Incomplete Cholesky factorizations}\label{sec:ichol} % (fold)

In addition to numpy's linear algebra support, this package uses some Fortran subroutines wrapped using \citetitle[http://cens.ioc.ee/projects/f2py2e/]{f2py}. They require \citetitle[http://www.netlib.org/blas/]{BLAS} and \citetitle[http://www.netlib.org/lapack/]{LAPACK} libraries (eg, \citetitle[http://math-atlas.sourceforge.net/]{ATLAS}) on your system, and the \file{setup.py} script will attempt to find these. If you don't have optimized BLAS and LAPACK installed, it's a good idea to install them; this package and numpy in general will be much faster. Some operating systems (such as Mac OS X) ship with optimized BLAS and LAPACK libraries included.

The following incomplete Cholesky factorization functions are found in the Fortran files \file{linalg_utils.f} and \file{incomplete_chol.f} :
\begin{description}

    % \item[\function{dtrsm_wrap(a,b,uplo='U',transa='N',alpha=1.)}:] A wrapper for the BLAS routine \citetitle[http://www.netlib.org/blas/dtrsmf.f]{DTRSM}, which solves triangular systems \texttt{$\alpha$a$^{op}$ x = b} in-place (that is, \texttt{b} is overwritten by $x$).
    % \begin{itemize}
    %     \item If \texttt{uplo='U'}, \texttt{a} is assumed to be an upper triangle; if \texttt{uplo='L'}, \texttt{a} is assumed to be a lower triangle.
    %     \item If \texttt{transa='T'}, \texttt{$\alpha$a.T*x = b} is solved. If \texttt{transa='N'}, \texttt{$\alpha$a*x = b} is solved.
    % \end{itemize}
    %
    % \item[\function{dtrmm_wrap(a,b,uplo='U',transa='N',alpha=1.)}:] A wrapper for the BLAS routine \citetitle[http://www.netlib.org/blas/dtrsmf.f]{DTRMM}, which does the triangular matrix multiplication \texttt{$\alpha$a$^{op}$x = b} in-place (that is, \texttt{b} is overwritten by $x$).
    % \begin{itemize}
    %     \item If \texttt{uplo='U'}, \texttt{a} is assumed to be an upper triangle; if \texttt{uplo='L'}, \texttt{a} is assumed to be a lower triangle.
    %     \item If \texttt{transa='T'}, \texttt{$\alpha$a.T*x} is computed. If \texttt{transa='N'}, \texttt{$\alpha$a*x} is computed.
    % \end{itemize}
    %
    % \item[\function{info = dpotrf_wrap(a)}:] A wrapper for the LAPACK routine \citetitle[http://www.netlib.org/lapack/double/dpotrf.f]{DPOTRF}, which tries to overwrite positive-definite matrix \texttt{a} with an upper-triangular Cholesky factor in-place. If the return value \texttt{info} is positive, \texttt{a} is not positive definite and the algorithm has failed.

    \item[\function{U, m, piv = ichol(diag, reltol, rowfun)}:] An implementation of the incomplete Cholesky decomposition, based on a port of one of the functions in the \citetitle[http://www.kyb.tuebingen.mpg.de/bs/people/seeger/]{chol_incomplete} package by Matthias Seeger.

The arguments are:
    \begin{description}
        \item[\texttt{diag}:] The diagonal of an \texttt{n} by \texttt{n} covariance matrix C.
        \item[\texttt{reltol}:] If the ratio of the \texttt{i}'th pivot to the maximum pivot is found to be less than \texttt{reltol}, the \texttt{i}'th pivot is assumed to be zero.
        \item[\texttt{rowfun}:] A Python function. The call \function{rowfun(i,p,rowvec)} should perform the update \texttt{rowvec[i,i+1:]=C[i,p[i+1:]]} in-place.
    \end{description}

The outputs are:
    \begin{description}
        \item[M:] The rank of C. Note that M$\le$\texttt{n}.
        \item[\texttt{piv}:] A length-\texttt{n} vector of pivots.
        \item[\texttt{U}:] An M-by-\texttt{n} upper-triangular matrix that satisfies \texttt{U[:,argsort(piv)].T * U[:,argsort(piv)] = C}.
    \end{description}

Because the full matrix $C$ does not need to be computed ahead of time, the algorithm is able to factor $C$ in $O(m^2 n)$ operations \cite{incompchol}. The algorithm is implemented in Fortran, but a Python version would be as follows:

\begin{verbatim}

def swap(vec,i,j):
    temp = vec[i]
    vec[i] = vec[j]
    vec[j] = temp

def ichol(diag, reltol, rowfun):

    piv = arange(n)
    U = zeros((n,n),dtype=float).view(matrix)
    rowvec = zeros(n,dtype=float)

    for i in range(n):
        l = diag[i:].argmax()
        maxdiag = diag[l]

        if maxdiag < reltol:
            m=i
            return U[:m,:], m, piv

        swap(diag,i,l)
        swap(p,i,l)

        temp = U[:i,i]
        U[:i,i] = U[:i,l]
        U[:i,l] = temp

        U[i,i] = sqrt(diag[i])
        rowvec[i:] = C[i,piv[i+1:]]

        if i > 0:
            rowvec -= U[:i,i].T * U[:i,i+1:]

        U[i,i+1:] = rowvec[i+1:] / U[i,i]
        diag[i+1:] -= U[i,i+1:].view(ndarray) ** 2

    m=n
    return U, m, piv
\end{verbatim}

Function \function{ichol} is wrapped by the \class{Covariance} method \method{cholesky}.

\item[\function{m, piv = ichol_continue(U, diag, reltol, rowfun, piv)}:]
This function computes the Cholesky factorization of an \texttt{n} by \texttt{n} covariance matrix C from the factor of its upper-left \texttt{n}$_*$ by \texttt{n}$_*$ submatrix C$_*$. Its input arguments are as follows:
\begin{description}
    \item[\texttt{U}:] Unlike \function{ichol}, this function overwrites a matrix in-place. Suppose the Cholesky factor of C$_*$, \texttt{U}$_*$, is of rank M$_*$. On input, \texttt{U} must be an \texttt{[m + (n-n$_*$)]}-by-\texttt{n} matrix arranged like this:
    \begin{eqnarray*}
        \left[
        \begin{array}{ccc}
            \texttt{U}_*\texttt{[:,:m}_*\texttt{]} & \texttt{U}_*\texttt{[:,:m}_*\texttt{].T.I C[:m}_*\texttt{,n}_*\texttt{:]} & \texttt{U}_*\texttt{[:,m}_*\texttt{:]}\\
            \texttt{0}&\texttt{0}&\texttt{0}
        \end{array}
        \right]
    \end{eqnarray*}
On exit, \texttt{U} will be an M-by-\texttt{n} upper-triangular matrix that satisfies \texttt{U[:,argsort(piv)].T * U[:,argsort(piv)]=C}.
    \item[\texttt{piv}:] Denote by \texttt{piv}$_*$ the pivot vector associated with \texttt{U}$_*$ On input, \texttt{piv} must be a length-\texttt{n} vector laid out like this:
    \begin{eqnarray*}
        \begin{array}{ccc}
            [\texttt{piv}_*\texttt{[:m}_*\texttt{]} & \texttt{arange(n-n$_*$)} & \texttt{piv}_*\texttt{[m}_*\texttt{:]}]
        \end{array}
    \end{eqnarray*}
    \item[\texttt{diag}:] The length \texttt{n-n}$_*$ diagonal of \texttt{C[n$_*$:,n$_*$]}.
    \item[\texttt{rowfun}:] The call \function{rowfun(i,p,rowvec)} should perform the update \texttt{rowvec[i,i+1:]} \texttt{=} \texttt{C[i,p[i+1:]]} in-place.
\end{description}

    The input parameter \texttt{reltol}, as well as the output parameters \texttt{piv} and M and the updated matrix \texttt{U}, should be interpreted just like their counterparts in \function{ichol}.

    The algorithm is just like the algorithm in ichol, but the index \texttt{i} iterates over \texttt{range(m$_*$,m$_*$+n-n$_*$)} (and the index of \texttt{diag} is downshifted appropriately).

    \function{ichol_continue} is wrapped by the \class{Covariance} method \method{continue_cholesky}, so it should rarely be necessary to call it directly.

    \item[\function{U, m, piv = ichol_full(c, reltol)}:] Just like \texttt{ichol}, but instead of a diagonal and a `get-row' function a full covariance matrix C is required as an input.

    \item[\function{U, m, piv = ichol_full(basis, nug, reltol)}:] Just like \texttt{ichol}, but the following arguments are required:
    \begin{description}
        \item[\texttt{basis}:] The evaluation of a basis function on the mesh, multiplied by the Cholesky factor of the coefficients' covariance matrix. This is itself a square root of the covariance matrix, but running it through \texttt{ichol_basis} allows for observations with nonzero variance and essentially pivots for better accuracy even in the zero-variance case.
        \item[\texttt{nug}:] A vector that will effectively be added to the diagonal of the covariance matrix.
    \end{description}
\end{description}

%     \item[\function{x = trisolve(A,b,uplo='U',transa='N',inplace=False)}:] Solves the triangular system \texttt{$\alpha$a$^{op}$ x = b} using \function{dtrsm_wrap} and returns $x$. If \texttt{A} is found to be singular, an error is raised.
%     \begin{itemize}
%         \item \texttt{A} is assumed upper-triangular if \code{uplo='U'} and lower-triangular if \code{uplo='L'}.
%         \item If \texttt{transa='T'}, \texttt{$\alpha$a.T*x = b} is solved. If \texttt{transa='N'}, \texttt{$\alpha$a*x = b} is solved.
%         \item If \texttt{inplace=True}, \texttt{b} is overwritten with $x$ in-place and returned. If \texttt{inplace=False}, \texttt{b} is not modified.
%     \end{itemize}
%
%     \item[\function{b = trimult(A,x,uplo='U',transa='N',inplace=False)}:] Does the triangular multiplication \texttt{$\alpha$a$^{op}$x = b} using \function{dtrmm_wrap} and returns \texttt{b}.
%     \begin{itemize}
%         \item \texttt{A} is assumed upper-triangular if \code{uplo='U'} and lower-triangular if \code{uplo='L'}.
%         \item If \texttt{transa='T'}, \texttt{$\alpha$a.T8x} is computed. If \texttt{transa='N'}, \texttt{$\alpha$a*x} is computed.
%         \item If \texttt{inplace=True}, $x$ is overwritten with \texttt{b} in-place and returned. If \texttt{inplace=False}, $x$ is not modified.
%     \end{itemize}
% \end{description}

    % \item[\function{observe(M, C, obs_mesh, obs_vals, obs_V, lintrans, cross_validate = True)}:] This function calls \texttt{C.observe}, calls \texttt{M.observe} with the output, and then if \texttt{cross_validate=True} calls \texttt{predictive_check}.

%     If the return of \texttt{predictive_check} value is \texttt{False}, the values of some observations can be predicted with negligible uncertainty from the values of others, but they don't match their predicted values; in other words, the data are extremely improbable. A \texttt{ZeroProbability} exception is raised with some helpful suggestions if this is the case.
%
%     \item[\function{OK = predictive_check(obs_vals, obs_mesh, M, posdef_indices, tolerance)} :] Checks the value of M evaluated at \texttt{obs_mesh} sliced at the complement of \texttt{posdef_indices} against the corresponding \texttt{obs_vals}. If any of the differences are greater than \texttt{tolerance}, \texttt{False} is returned. Otherwise \texttt{True} is returned.
% \end{description}
% \section{Object internals}\label{sec:internals}
%
% The methods of \class{Mean}, \class{Covariance} and \class{Realization} will be described here.
%
% \section{Covariance}\label{sec:covarianceInt}
% \begin{description}
%
%     \item[\method{cholesky(x, apply_pivot = True, observed=True)}:] Returns an incomplete Cholesky factor of \texttt{C(x,x)}. The other arguments' meanings are as follows:
%     \begin{description}
%         \item[\texttt{apply_pivot}:] If \texttt{True}, the return value is a matrix \texttt{U} such that \texttt{U.T*U = C(x,x)}. If \texttt{False}, the return value is a dictionary. Element \texttt{'pivots'} is a vector of pivots returned by function \function{ichol}, and element \texttt{'U'} is the matrix \texttt{sig} returned by \function{ichol}.
%         \item[\texttt{observed}:] If \texttt{True}, the matrix \texttt{C(x,x)} is obtained conditional on observations that have been made involving C. If \texttt{False}, the matrix is obtained without regard to those observations.
%     \end{description}
%
%         \item[\method{continue_cholesky(x, x_old, chol_dict_old[, apply_pivot, observed])}:] Returns an incomplete Cholesky factor of \texttt{C(concatenate(x_old,x), concatenate(x_old,x))}. Arguments \texttt{apply_pivot} and \texttt{observed} are the same as for \texttt{cholesky}. \texttt{chol_dict_old} should be the incomplete Cholesky factorization of \texttt{C(x_old, x_old)} in the form of a dictionary such as the one produced by \texttt{C.cholesky}.
%
%         This method returns either a matrix or a dictionary depending on the value of \texttt{apply_pivot}. See \texttt{cholesky}.
%
%
%         \item[\method{relevant_slice, obs_mesh_new, U_for_draw = observe(obs_mesh, obs_V=0.)}:] All subsequent calls will be made under the assumption that observations of the random field at inputs \texttt{obs_mesh} have been made with variance \texttt{obs_V}. This method updates the following attributes of C:
%         \begin{itemize}
%             \item \texttt{observed}: This flag is set to \texttt{True}.
%             \item \texttt{obs_U} and \texttt{obs_piv}: The output of \texttt{self.cholesky(obs_mesh, apply_pivot=False, observed=False)}.
%             \item \texttt{obs_mesh}: The mesh on which self has been observed, sliced by \texttt{obs_piv[:m]}, where M is the rank of \texttt{obs_U}.
%             \item \texttt{obs_V}: The variances associated with the observations, sliced as \texttt{obs_mesh} is.
%             \item \texttt{obs_len}: The length of \texttt{obs_V}.
%         \end{itemize}
%
%         The return values are:
%         \begin{itemize}
%             \item \texttt{relevant_slice}: The indices included in the incomplete Cholesky factorization. These correspond to the values of \texttt{obs_mesh} that determine the other values, but not one another.
%             \item \texttt{obs_mesh_new}: \texttt{obs_mesh} sliced according to \texttt{relevant_slice}.
%             \item \texttt{U_for_draw}: An upper-triangular Cholesky factor of \texttt{self}'s evaluation on \texttt{obs_mesh} conditional on all previous observations.
%         \end{itemize}
%
%
%
%     \item[\method{__call__(x[, y, observed])}:]
%
%     \begin{itemize}
%         \item If only one argument $x$ is provided, \texttt{C(x,x) = V(f(x))} will be returned conditional on any observations that have been made.
%         \begin{itemize}
%             \item If C is unobserved or \texttt{observed = False}, its underlying function C is evaluated at each element of $x$. The \texttt{i}'th element of the return value is \texttt{c(x_i,x_i)}.
%             \item If C is observed and \texttt{observed = True}, its underlying function C is evaluated at each element of $x$. The \texttt{i}'th element of the return value is
%             \begin{eqnarray*}
%                 \texttt{c(x_i,x_i) - (U.T.I c(o,x_i)).T (U.T.I c(o,x_i))},
%             \end{eqnarray*}
%             where \texttt{o} is \code{self.obs_mesh} and \texttt{U} is \texttt{self.obs_U}, both of which were provided by \function{observe}.
%             \end{itemize}
%
%         \item If two arguments \texttt{(x,y)} are provided, \texttt{C(x,y)} will be returned conditional on any observations that have been made.
%         \begin{itemize}
%             \item If C is unobserved, \texttt{c(x,y)} will be returned where C is the underlying function. If $x$ and $y$ are references to the same array, only half the matrix will actually be computed; the other half will be filled in.
%         \item If C is observed, the return value will be
%         \begin{eqnarray*}
%             \texttt{c(x,y) - (U.T.I c(o,x)).T (U.T.I c(o,y))}
%         \end{eqnarray*}
%         where again \texttt{o} is \code{self.obs_mesh} and \texttt{U} is \texttt{self.obs_U}.
%         \end{itemize}
%
%     \end{itemize}
%     \end{description}
%
% \section{BasisCovariance}\label{sec:basisCovariance}
% \class{BasisCovariance} has methods and attributes that act just like those of \texttt{Covariance}, but of course the internal computations are different and tend to be faster.
%
% \section{Mean}\label{sec:meanInt}
% \begin{description}
%     \item[\method{observe(C, obs_mesh_new, obs_vals_new)}:] All subsequent calls will be made under the assumption that observations of the random field at inputs \texttt{obs_mesh} have been made with variance \texttt{obs_V}. Assumes that \texttt{C(obs_mesh_new, obs_mesh_new)} is positive definite; the \emph{function} \function{observe} slices \texttt{obs_vals} according to \texttt{Covariance.observe}'s output value \texttt{relevant_slice} before passing it to \texttt{Mean.observe}. This method updates the following attributes of M:
%     \begin{itemize}
%         \item \texttt{self.C}: The covariance of the random field (which has self as mean) that was observed. Several of $C$'s attributes are used by this method.
%         \item \texttt{self.obs_U}: \texttt{C.obs_U}.
%         \item \texttt{self.obs_mesh}: \texttt{C.obs_mesh}.
%         \item \texttt{obs_V}: \texttt{C.obs_V}.
%         \item \texttt{obs_len}: \texttt{C.obs_len}.
%         \item \texttt{self.dev}: \texttt{obs_mesh - self.__call__(obs_mesh, observed=False)}.
%         \item \texttt{self.reg_vec}: \texttt{self.obs_U[:,:m].T*self.dev}, where M is the rank of \texttt{self.obs_U}.
%     \end{itemize}
%
%     Note that \texttt{Mean.observe} will not cross-validate; if inconsistent observation values are used, it will simply ignore them. You'll need to call \texttt{predictive_check} yourself if you want cross-validation.
%
%     \item[\method{__call__(x[, observed])}:]
%     \begin{itemize}
%         \item If M has not been observed or \texttt{observed = False}, the return value of \texttt{M(x)} will just be \texttt{m(x)} where M is the underlying mean function.
%         \item If M has been observed and \texttt{observed = True}, the return value will be
%         \begin{eqnarray*}
%             \texttt{c(x,o) U.I r},
%         \end{eqnarray*}
%         where \texttt{r} is \code{self.reg_vec}, \texttt{U} is \code{self.obs_U} and C is \code{self.cov_fun}, all of which were provided by \function{observe}.
%
%     \end{itemize}
%
% \end{description}

% \section{Realization}\label{sec:realInt}
% A realization f maintains the following attributes:
% \begin{itemize}
%     \item \texttt{x_sofar}: The values at which self has already been evaluated.
%     \item \texttt{f_sofar}: Self's evaluation at \texttt{x_sofar}.
%     \item \texttt{M_internal}: An observed copy of self's mean.
%     \item \texttt{C_internal}: An observed copy of self's covariance.
% \end{itemize}
%
% When the evaluation \texttt{f(x)} is requested, the following happen:
% \begin{enumerate}
%     \item \texttt{x_sofar} is searched for elements of $x$. If any are found, the corresponding values of f aren't recomputed. Call the remaining elements \texttt{x_new}.
%     \item \texttt{C_internal} is observed at \texttt{x_new} with \texttt{obs_V=0}.
%     \item The output value \texttt{U_for_draw} is used to generate a normal random variable \texttt{f_new} with mean \texttt{M_internal(x_new)} and covariance \texttt{C_internal(x_new, x_new)}.
%     \item \texttt{M_internal} is observed with \texttt{obs_mesh = x_new}, \texttt{obs_vals = f_new}, \texttt{obs_taus = Inf}.
%     \item \texttt{f_new} is appropriately combined with f's evaluation at previously computed values of $x$ and the result is returned.
% \end{enumerate}
%
% If a realization's covariance is an instance of \texttt{BasisCovariance}, the behavior is different. At instantiation, the realization will draw coefficients for the basis terms; calls to the realization will simply be handled by calling the basis.

% chapter numerics (end)

% \chapter{Wishlist}\label{cha:wishlist}
% \section{Features}
% \begin{itemize}
%     \item Linear transformations.
%     \begin{itemize}
%         \item Create new GP's from old GP's via linear transformations. Maybe try to make a \class{LinearOperator} class.
%         \item Observe linear transformations of evaluations of GP.
%     \end{itemize}
%     \item Observations can be dependent (\texttt{obs_V} can be square rather than a diagonal).
%     \item Specific support for anisotropy.
%     \item Specific support for spline covariances.
%     \item Specific support for vector-valued GPs (cokriging).
% \end{itemize}
%
% \section{Optimizations}
% \begin{itemize}
%     % \item Markov random field covariances, which take advantage of the sparseness of the covariance for speed.
%     \item Speed up realization calls with just one value. All the array manipulation really bogs this case down, and this makes dynamical/autoregressive applications slow.
%     \begin{itemize}
%         \item There isn't a clear bottleneck, unfortunately. It might be necessary to write all the call methods in f2py or Pyrex.
%     \end{itemize}
% \item \function{ichol} and friends:
% \begin{itemize}
%     \item Don't allocate full-rank memory up-front.
%     \item Use a symbolic Cholesky decomposition to save some work.
%     \item Increase granularity of BLAS calls to take better advantage of threaded libraries.
%     \item Distributed memory.
% \end{itemize}
% \end{itemize}
%
% \section{Other}
% \begin{itemize}
%     \item Clean up the return values from \class{Covariance.observe}.
%     \item Try to figure out relative performance of incomplete Cholesky w/low-rank covariances to basis covariances, and relative performance of sparse Cholesky to Markov Random Field covariances.
% \end{itemize}

% section new_features (end)

% \chapter{Quick reference}\label{cha:reference}


\subsection{Dot Products}
\noindent
A dot product is a way of multiplying two vectors so that the result is a scalar.
$\vec{a}\cdot\vec{b} = \norm{\vec{a}}\norm{\vec{b}}\cos{\theta}$ where $\theta$ is the angle between $\vec{a}$ and $\vec{b}$.
One way to think of the dot product is as a measure of how much two vectors point in the same direction.
We can also show using the law of cosines that $\vec{a}\cdot\vec{b} = a_1b_1+a_2b_2+...+a_nb_n$.
Knowing the lengths of two vectors and their dot product we can calculate the angle between them as
\begin{equation*}
	\theta = \arccos{\left(\frac{\vec{a} \cdot \vec{b}}{\norm{\vec{a}} \norm{\vec{b}}}\right)}.
\end{equation*}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.25\textwidth]{../common/vectorsMatrices/DotProduct.png}
	\caption{Two vectors and the angle between them}
\end{figure}

\noindent
Although similar to scalar multiplication, dot products have some properties that set them apart.
\begin{enumerate}[label=]
	\item \textbf{Commutative}
		\begin{equation*}
			\vec{a}\cdot\vec{b} = \vec{b}\cdot\vec{a}
		\end{equation*}
		the same as scalar multiplication.
	\item \textbf{Distributive}
		\begin{equation*}
			\vec{a}\cdot\left(\vec{b}+\vec{c}\right) = \vec{a}\cdot\vec{b}+\vec{a}\cdot\vec{c}
		\end{equation*}
		the same as scalar multiplication.
	\item \textbf{\underline{NOT} Associative}
		$\left(\vec{a}\cdot\vec{b}\right)\cdot\vec{c}$ is a nonsense expression.
		However, like scalar multiplication, dot products are scalar associative.
		\begin{equation*}
			\left(c\cdot\vec{a}\right)\cdot\vec{b} = \vec{a}\cdot\left(c\cdot\vec{b}\right)
		\end{equation*}
\end{enumerate}


% !TEX root = main.tex

%-------------------------------------------------
\section{Appendix}\label{sec:}

\subsection{Proof of the Cauchy-Schwarz inequality for random variables}

We need to show that for any two random variables $X$ and $Y$,
\[
\expe(XY)^2 \leq \expe(X^2)\expe(Y^2)
\]
with equality if and only if $\prob(Y=aX)=1$ for some $a\in\R$.

First we need the following technical result (which we shall not prove).
% lemma
\begin{lemma}\label{lem:pos_rv_expe_zero}
If $X\geq 0$ and $\expe(X)=0$ then $\prob(X=0)=1$.
\end{lemma}

\begin{proof}[Cauchy-Schwarz]
$X^2$ and $Y^2$ are non-negative random variables, so by Lemma~\ref{lem:pos_rv_expe_zero} we can assume that $\expe(X^2)>0$ and $\expe(Y^2)>0$ (otherwise both sides of the inequality are identically zero).

Let $a\in\R$ consider the random variable $Z=aX-Y$. By the properties of expectation,
\begin{align*}
Z^2\geq 0
	& \Rightarrow \expe(Z^2)\geq 0  \qquad\text{(positivity)}\\
	& \Rightarrow \expe(a^2X^2 - 2aXY + Y^2) \geq 0  \\
	& \Rightarrow a^2\expe(X^2) - 2a\expe(XY) + \expe(Y^2) \geq 0 \qquad\text{(linearity).}
\end{align*}

\bit
\it Let $q(a)=a^2\expe(X^2) - 2a\expe(XY) + \expe(Y^2)$. This is a quadratic expression in $a$.
\eit
Since $q(a)\geq 0$ for all $a\in\R$, the roots of the quadratic equation $q(a)=0$, given by
\[
a = \frac{\expe(XY)\pm\sqrt{\expe(XY)^2-\expe(X^2)\expe(Y^2)}}{\expe(X^2)}
\]
are either both complex (discriminant is negative) or co-incide (discriminant is zero).
\bit
\it Hence $\expe(XY)^2-\expe(X^2)\expe(Y^2) \leq 0$, or equivalently $\expe(XY)^2\leq \expe(X^2)\expe(Y^2)$.
\eit

The discriminant is zero if and only if the quadratic has one real root, which occurs if and only if $\expe(Z^2)=0$ for some $a\in\R$, i.e.
\[
\expe\left((aX-Y)^2\right) = 0\qquad\text{for some $a\in\R$.}
\]
Because $(aX-Y)^2$ is a non-negative random variable, we have by Lemma~\ref{lem:pos_rv_expe_zero} that
\[
\prob\big[(aX-Y)^2 = 0\big] = 1
\]
or equivalently, $\prob\big(Y = aX) = 1$, as required.
\end{proof}


\documentclass[12pt]{article}
\input{physics1}
\begin{document}

\section*{NYU Physics I---small variations}

\paragraph{\theproblem}\refstepcounter{problem}%
You have a cubic block of ice, $1.000\,\m$ on a side.  By what
fractional amount does its mass decrease if you shave a millimeter off
of it in each dimension (ie, so that it becomes a cubic block of ice,
$0.999\,\m$ on a side)?

\paragraph{\theproblem}\refstepcounter{problem}%
Explain why your answer to the above was so close to $3\times
10^{-3}$.  You have two explanations to give, \textsl{(a)}~one from
the point of view of the three operations you did (shave one face,
then the next, then the next), and \textsl{(b)}~one from the point of
view of small variations (write the formula for the volume $V$ in
terms of the side length $\ell$ and differentiate with respect to
$\ell$).

\paragraph{\theproblem}\refstepcounter{problem}%
Give a general argument---using calculus---that
\begin{equation}
\lim_{\epsilon\rightarrow 0}\,(1+\epsilon)^n=1+n\,\epsilon \quad .
\end{equation}

\paragraph{\theproblem}\refstepcounter{problem}%
With a calculator, compute the sine of the angle $\theta=0.1\,\rad$.
Now compute the error you make if you use $\sin\theta=\theta$.  Now
compare that error to the quantity $\theta^2/2$ and $\theta^3/6$.  Is
it close to either?

\paragraph{\theproblem}\refstepcounter{problem}%
Now compute the error you make if you use
\begin{equation}
\lim_{\theta\rightarrow 0}\,\sin\theta=\theta-\frac{1}{6}\,\theta^3 \quad .
\label{eq:third}
\end{equation}
Do you think the \emph{next} term in the Taylor series for sine will
be negative or positive on this basis?

\paragraph{\theproblem}\refstepcounter{problem}%
Inside of what angle is the approximation $\sin\theta=\theta$ good to
one percent?  What is this angle in degrees?

\paragraph{\theproblem}\refstepcounter{problem}%
Now square the expression in Equation~(\ref{eq:third}) and compare it
to the second-order expression for cosine:
\begin{equation}
\lim_{\theta\rightarrow 0}\,\cos\theta=1-\frac{1}{2}\,\theta^2 \quad .
\end{equation}
Show that this is consistent with the trigonometric identity
\begin{equation}
\sin^2\theta = \frac{1 - \cos(2\,\theta)}{2} \quad .
\end{equation}

\end{document}


\section*{Exercises}

\begin{ex}
  Use row operations to calculate the following determinants:
  \begin{equation*}
    (a)~
    \begin{absmatrix}{rrr}
      1 & 2 & 1 \\
      2 & 3 & 2 \\
      -4 & 1 & 2
    \end{absmatrix},
    \quad
    (b)~
    \begin{absmatrix}{rrr}
      2 & 1 & 3 \\
      2 & 4 & 2 \\
      1 & 4 & -5
    \end{absmatrix},
    \quad
    (c)~
    \begin{absmatrix}{rrrr}
      1 & 2 & 1 & 2 \\
      3 & 1 & -2 & 3 \\
      -1 & 0 & 3 & 1 \\
      2 & 3 & 2 & -2
    \end{absmatrix},
    \quad
    (d)~
    \begin{absmatrix}{rrrr}
      1 & 4 & 1 & 2 \\
      3 & 2 & -2 & 3 \\
      -1 & 0 & 3 & 3 \\
      2 & 1 & 2 & -2
    \end{absmatrix}.
  \end{equation*}
  \begin{sol}
    \begin{enumerate}
    \item
      \begin{equation*}
        \begin{absmatrix}{rrr}
          1 & 2 & 1 \\
          2 & 3 & 2 \\
          -4 & 1 & 2
        \end{absmatrix} = -6.
      \end{equation*}
    \item
      \begin{equation*}
        \begin{absmatrix}{rrr}
          2 & 1 & 3 \\
          2 & 4 & 2 \\
          1 & 4 & -5
        \end{absmatrix} = -32.
      \end{equation*}
    \item One can row reduce this, using only row operations of the
      third kind, to
      \begin{equation*}
        \def\arraystretch{1.2}
        \begin{mymatrix}{rrrr}
          1 & 2 & 1 & 2 \\
          0 & -5 & -5 & -3 \\
          0 & 0 & 2 & \frac{9}{5} \\
          0 & 0 & 0 & -\frac{63}{10}
        \end{mymatrix}.
      \end{equation*}
      Therefore, the determinant is $-63$.
    \item One can row reduce this, using only row operations of the
      third kind, to
      \begin{equation*}
        \def\arraystretch{1.2}
        \begin{mymatrix}{rrrr}
          1 & 4 & 1 & 2 \\
          0 & -10 & -5 & -3 \\
          0 & 0 & 2 & \frac{19}{5} \\
          0 & 0 & 0 & -\frac{211}{20}
        \end{mymatrix}.
      \end{equation*}
      Thus the determinant is $211$.
    \end{enumerate}
  \end{sol}
\end{ex}


\lab{Algorithms}{Elementary Matrices}{Elementary Matrices}
\label{lab:LUdecomp}
\objective{In this section we will use elementary matrices to find the RREF and to find the LU decomposition.}

In Linear algebra there are 3 elementary row operations: switching two rows, multiplying a row by a constant, and adding a multiple of one row to another row.  We carry out each of these operations with a corresponding elementary matrix.  These matrices are easy to construct. Suppose $A$ is an $m \times n$ matrix and you want to perform one of the three elementary operations on $A$. You can do this be constructing the $m \times m$ identity matrix, $I$, performing the elementary row operation on $I$ to obtain $E$ and then multiplying $EA$.  For example, consider the matrix
\[
A = \begin{pmatrix}
a_{11}&a_{12}&a_{13}&a_{14}\\
a_{21}&a_{22}&a_{23}&a_{24}\\
a_{31}&a_{32}&a_{33}&a_{34}
\end{pmatrix}
\]
If we want to swap the first two rows, we can left multiply the
matrix $A$ by:
\[
E = \begin{pmatrix}
0&1&0\\
1&0&0\\
0&0&1
\end{pmatrix},
\]
then
\[
E A =
\begin{pmatrix}
a_{21}&a_{22}&a_{23}&a_{24}\\
a_{11}&a_{12}&a_{13}&a_{14}\\
a_{31}&a_{32}&a_{33}&a_{34}
\end{pmatrix}.
\]
E in this case is called a type I matrix.

Now let's examine the next row operation.  If we want to multiply,
say, the second row of $A$ by the constant $b$, we can left multiply
the matrix $A$ by the following matrix:
\[
\tilde{E} = \begin{pmatrix}
1&0&0\\
0&b&0\\
0&0&1
\end{pmatrix}.
\]
Then
\[
\tilde{E} A =
\begin{pmatrix}
a_{11}&a_{12}&a_{13}&a_{14}\\
b a_{21}&b a_{22}&b a_{23}&b a_{24}\\
a_{31}&a_{32}&a_{33}&a_{34}
\end{pmatrix}.
\]
$\tilde{E}$ is called a type II matrix.

Now let's examine the last row operation.  If we want to multiply,
say, the first row of $A$ by a constant $c$ and add it to the second
row, we can left multiply the matrix $A$ by the following matrix:
\[
\widehat{E} = \begin{pmatrix}
1&0&0\\
c&1&0\\
0&0&1
\end{pmatrix}.
\]
Then
\[
\widehat{E} A =
\begin{pmatrix}
a_{11}&a_{12}&a_{13}&a_{14}\\
c a_{11} + a_{21}&c a_{12} + a_{22}&c a_{13} + a_{23}&c a_{14} + a_{24}\\
a_{31}&a_{32}&a_{33}&a_{34}
\end{pmatrix}.
\]
$\widehat{E}$ is called a type III matrix.

Below, the elementary matrices corresponding to each row operation is implemented in Python.
\lstinputlisting[style=python, style=fromfile, name=row_opers.py]{row_opers.py}

\section*{Programming Row Reduction}

A fundamental problem in linear algebra is using matrix representations to solve systems of linear equations.  In this section, we do this by using elementary matrices to reduce a matrix into ``row echelon form'' (REF), as opposed to ``reduced row echelon form'' (RREF).  We remark that to solve a linear system, it is actually faster computationally to use REF and then finish with back-substitution, than it is to use RREF.  Consider the following matrix:

\[
\begin{pmatrix}
4&5&6&3 \\
2&4&6&4 \\
7&8&0&5
\end{pmatrix}
\]

By iteratively left multiplying by elementary matrices, we can reduce as follows:

Remember that our functions returns the elementary array corresponding to the desired row operation.  Also note that setting the type of our initial array is crucial.
\begin{lstlisting}[style=python]
: import scipy as sp
: import row_opers as op
: A = sp.array([[4, 5, 6, 3],[2, 4, 6, 4],[7, 8, 0, 5]], dtype='float32')
array([[ 4.,  5.,  6.,  3.],
       [ 2.,  4.,  6.,  4.],
       [ 7.,  8.,  0.,  5.]], dtype=float32)
: A1 = sp.dot(op.cmultadd(3,1,0,-A[1,0]/A[0,0]), A); A1
array([[ 4. ,  5. ,  6. ,  3. ],
       [ 0. ,  1.5,  3. ,  2.5],
       [ 7. ,  8. ,  0. ,  5. ]])
: A2 = sp.dot(op.cmultadd(3,2,0,-A1[2,0]/A1[0,0]), A1); A2
array([[  4.  ,   5.  ,   6.  ,   3.  ],
       [  0.  ,   1.5 ,   3.  ,   2.5 ],
       [  0.  ,  -0.75, -10.5 ,  -0.25]])
: A3 = sp.dot(op.cmultadd(3,2,1,-A2[2,1]/A2[1,1]), A2); A3
array([[ 4. ,  5. ,  6. ,  3. ],
       [ 0. ,  1.5,  3. ,  2.5],
       [ 0. ,  0. , -9. ,  1. ]])
\end{lstlisting}

To complete REF we would need to divide each row by its leading
coefficient.  We can do that using Type II matrices.  We leave it to
you to carry this out.

\begin{problem}
\label{prob:REF}
Write a Python function, which takes as input an
$n\times (n+1)$ matrix (in other words and \emph{augmented} matrix) and performs the above naive row reduction to REF using elementary matrices.
\end{problem}

\section*{LU Decomposition}

Again, consider the matrix $A$. By iteratively left multiplying by
elementary matrices, we reduce as follows:

\begin{lstlisting}[style=python]
: E1 = op.cmultadd(3,1,0,-A[1,0]/A[0,0]); E1
array([[ 1. ,  0. ,  0. ],
       [-0.5,  1. ,  0. ],
       [ 0. ,  0. ,  1. ]])
: B1 = sp.dot(E1, A)
array([[ 4. ,  5. ,  6. ,  3. ],
       [ 0. ,  1.5,  3. ,  2.5],
       [ 7. ,  8. ,  0. ,  5. ]])
: E2 = op.cmultadd(3,2,0,-B1[2,0]/B1[0,0])
: B2 = sp.dot(E2, B1)
: E3 = op.cmultadd(3,2,1,-B2[2,1]/B2[1,1])
: U = sp.dot(E3, B2); U
array([[ 4. ,  5. ,  6. ,  3. ],
       [ 0. ,  1.5,  3. ,  2.5],
       [ 0. ,  0. , -9. ,  1. ]])
\end{lstlisting}
Note that we have reduced the above matrix into upper-triangular
form, denoted as $U$.  Hence, we have
\[
U = E_3 E_2 E_1 A.
\]
Since the elementary matrices are invertible, we also have
\[
(E_3 E_2 E_1)^{-1} U =  A.
\]
This can be re-written as
\[
E_1^{-1} E_2^{-1} E_3^{-1} U =  A.
\]
Then we define $L$ to be
\[
L = E_1^{-1} E_2^{-1} E_3^{-1},
\]
which yields $L U = A$.
\begin{lstlisting}[style=python]
: from scipy import linalg as la
: I = lambda x: la.inv(x)
: L = sp.dot(sp.dot(I(E1), I(E2)), I(E3)); L
array([[ 1.  ,  0.  ,  0.  ],
       [ 0.5 ,  1.  ,  0.  ],
       [ 1.75, -0.5 ,  1.  ]])
: sp.dot(L, U)
array([[ 4.,  5.,  6.,  3.],
       [ 2.,  4.,  6.,  4.],
       [ 7.,  8.,  0.,  5.]])
\end{lstlisting}
What makes $LU$ decomposition so easy is that the inverses of
elementary matrices are elementary matrices.  For example, the
inverse of a Type 3 elementary matrix is the same matrix with the
opposite sign in the $(j,k)$ entry.  In the above problem, we have:
Note that the minus signs are gone.  Doing it this way, we don't
have to actually invert anything to compute $L$.  This makes the
computation much faster.

\section*{Why Should I Care?}

The LU Decomposition isn't very useful when doing matrix computation
by hand.  It is, however, very important in scientific computation
for the following reasons:
\begin{itemize}
\item If you want to solve the matrix equation $A x = b$, for several different $b's$, you can replace $A$ with $L$ and $U$, giving $L U x = b$.  Then solve the equations $L y = b$ and $U x = y$, using forward and backward substitution, respectively.  This is actually faster than solving them with row reduction.
\item The $LU$ decomposition allows quick computation of both inverses and determinants.
\item For very large matrices the $LU$ decomposition is crucial.  Indeed one can perform the $LU$ decomposition on a given matrix $A$ without needing additional space, that is, the program actually over-writes $A$ with $L$ and $U$.  Note that since the diagonal of $L$ are all ones, they don't need to be stored, and so the upper diagonal (including the diagonal) is $U$ and the lower diagonal (not including the diagonal) is $L$.
\end{itemize}

\begin{problem}
\label{prob:LU}
Write a Python function which takes as input
a random $n\times n$ matrix, performs the LU decomposition and
returns $L$ and $U$.  To verify that it works, multiply $L$ and $U$ together
and compare to $A$. Note: you should not use the \li{inv} function when you do this. You should only use the elementary matrices that we just created.
Additionally, have your function count the number of operations needed to perform the LU decomposition.
\end{problem}

\begin{problem}
\label{prob:det}
Write a Python function which uses the solution to Problem \ref{prob:LU} to find the determinant of $A$.
\end{problem}


\chapter{Some Limit Theorems}

\section{Important/Useful Theorems}

\subsection{Weak Law of Large Numbers}
Let $\xi_1, \xi_2, ..., \xi_n$ be $n$ independent indentically distributed random variables with mean $a$ and variance $\sigma^2$.  Then, given any $\delta > 0$ and $\epsilon >0$, however small, there is an integer, $n$ such that:
\begin{equation}
	a- \epsilon \leq \frac{1}{n}(\xi_1 +\xi_2 + \cdots + \xi_n) \leq a + \epsilon
\end{equation}
with probability greater than $1- \delta$.

\subsection{Generating Functions}

For the \textbf{discrete} random variable $\xi$ with probability distribution $P_{\xi}(k)$, the generating function is defined as

\begin{equation}
	F_{\xi}(z) = \sum_{k=0}^{\infty}P_{\xi}(k)z^k
\end{equation}
which yields some cool relations:
\begin{eqnarray}
	P_{\xi}(k) = \frac{1}{k!}F_{\xi}^{(k)}(0) \\
	\textbf{E}\xi = F'(1) \\
	\sigma^2 =F''(1) + F'(1) - [F'(1)]^2
\end{eqnarray}

Note, also, that while you sum together random variables, you multiply generating functions.

\subsection{Thm. 6.2}

The sequence of probability distributions, $P_n(k)$ with generating functions $F_n(z)$ where $n=1, 2, 3, ....$ converges weakly to the distribution $P(k)$ with distribution function $F(z)$ iff

\begin{equation}
	\lim_{n\rightarrow \infty} F_n(z) = F(z)
\end{equation}

\subsection{Characteristic Functions}

For a real random variable $\xi$, its generating function is defined as

\begin{equation}
	f_{\xi}(t) = \textbf{E}e^{i \xi t} = F_{\xi}(e^{i t}) = \sum_{k=0}^{\infty} P_{\xi}(k)e^{i k t}
\end{equation}
Or, if the variable is continuous:
\begin{equation}
	f_{\xi}(t) = \int_{-\infty}^{\infty} P_{\xi}(x)e^{i x t} dx
\end{equation}
Where it is clear that is represents the discrete or continuous fourier transform of the probability distribution and as such, the inverse is true:
\begin{equation}
	P_{\xi}(x) = \frac{1}{2 \pi}\int_{-\infty}^{\infty} f_{\xi}(t)e^{-i x t} dt
\end{equation}
Like Generating functions, there are some cool properties that can be exploited:
\begin{eqnarray}
	\textbf{E}\xi = -if_{\xi}'(0) \\
	\textbf{D}\xi = -f_{\xi}''(0) + [f_{\xi}'(0)]^2
\end{eqnarray}
And just like generating functions, they multiply together for random variables that add together.

\subsection{The Central Limit Theorem}

A sequence of random variables $\xi_1, \xi_2, \xi_3, ...$ is said to satisfy the central limit theorem if
\begin{equation}
	\lim_{n\rightarrow \infty} \textbf{P}\{ x' \leq \frac{S_n - \textbf{E}S_n}{\sqrt{\textbf{D}S_n}} \leq x''\} = \int_{x'}^{x''}e^{\frac{-x^2}{2}}dx
\end{equation}
Where
\begin{equation}
	S_n = \sum_{k=1}^n \xi_k
\end{equation}

\subsection{Thm. 6.3}

If the same series of random variables as above each has mean $a_k$ and variance $\sigma^2_k$ and satisfies the Lyapunov condition:
\begin{equation}
	\lim_{n\rightarrow \infty} \frac{1}{B_n^3} \sum_{k=1}^n \textbf{E}|\xi_k - a_k|^3 = 0
\end{equation}
Where
\begin{equation}
	B_n^2 = \textbf{D}S_n = \sum_{k=1}^n \sigma^2_k
\end{equation}
then the sequence will satisfy the central limit theorem.

\section{Answers to Problems}
\subsection{}
%problem6.1
\begin{eqnarray}
	a- \epsilon \leq \frac{1}{n}(\xi_1 +\xi_2 + \cdots + \xi_n) \leq a + \epsilon \\
	a- \epsilon \leq \frac{1}{n}\sum_{k=1}^n \xi_i \leq a + \epsilon \\
	- \epsilon \leq \frac{1}{n}\sum_{k=1}^n \xi_i -a \leq \epsilon \\
	\left| \frac{1}{n}\sum_{k=1}^n \xi_i -a  \right| < \epsilon
\end{eqnarray}
And we know this happens with probability
\begin{equation}
	P_n = 1- \delta
\end{equation}
\begin{equation}
	\textbf{P} \left[ \left| \frac{1}{n}\sum_{k=1}^n \xi_i -a  \right| < \epsilon  \right] = 1 - \delta
\end{equation}
\begin{equation}
	\lim_{n\rightarrow \infty} \textbf{P} \left[ \left| \frac{1}{n}\sum_{k=1}^n \xi_i -a  \right| < \epsilon  \right] = 1
\end{equation}
\textbf{Answer not verified}

\subsection{}
%problem6.2

Since we know the mean, we most certainly can use the equation to estimate the variance: look what it's doing.  It's taking the sum of the square of the distances from the known mean and dividing by the number of trials: the classic standard deviation calculation.
\textbf{Answer not verified}

\subsection{}
%problem6.3
We need to start with some calculations on the distribution:
\begin{eqnarray}
	E\xi = \int_0^{\infty} x \frac{x^m}{m!}e^{-x} dx = m+1 \\
	E\xi^2 = \int_0^{\infty} x^2 \frac{x^m}{m!}e^{-x} dx = (m+2)(m+1)
\end{eqnarray}

\begin{equation}
	P \{  | \xi - a | > \epsilon  \} \leq \frac{1}{\epsilon^2}\textbf{E}(\xi - a)^2
\end{equation}
clearly we can manipulate it in this manner:
\begin{equation}
	-P \{  | \xi - a | > \epsilon  \} \geq  - \frac{1}{\epsilon^2}\textbf{E}(\xi - a)^2
\end{equation}
\begin{equation}
	1-P \{  | \xi - a | > \epsilon  \} \geq 1 - \frac{1}{\epsilon^2}\textbf{E}(\xi - a)^2
\end{equation}
\begin{equation}
	P \{  | \xi - a | < \epsilon  \} \geq 1 - \frac{1}{\epsilon^2}\textbf{E}(\xi - a)^2
\end{equation}
let $\epsilon = a = m+1$
\begin{equation}
	P \{ 0 \leq \xi  \leq 2(m+1)  \} \leq 1 - \frac{1}{(m+1)^2}[(m+2)(m+1) - (m+1)^2]
\end{equation}
\begin{equation}
	P \{ 0 \leq \xi  \leq 2(m+1)  \} \leq \frac{m}{m+1}
\end{equation}

\textbf{Answer not verified}

\subsection{}
%problem6.4
Under these circumstances, we expect 500 A and the standard deviation is $\sqrt{250}= 5 \sqrt{10} \approx 15$ therefore the $\pm 100$ from the mean is more than $\pm 6 \sigma$ meaning there is indeed far more that .97 probability of seeing the mean between these two values.

\textbf{Answer not verified}

\subsection{}
%problem6.5
\begin{eqnarray}
	F_{\xi}(z) = \frac{z +z^2 + z^3 + z^4 + z^5 +z^6}{6}
\end{eqnarray}

\textbf{Answer not verified}

\subsection{}
%problem6.6
\begin{eqnarray}
	F'_{\xi}(z) = \frac{1+ 2z + 3z^2 + 4z^3 + 5z^4 + 6z^5}{6}
\end{eqnarray}
\begin{eqnarray}
	F'_{\xi}(1) = \frac{21}{6} = \frac{7}{2} = \textbf{E}\xi
\end{eqnarray}
\begin{eqnarray}
	F''_{\xi}(z) = \frac{2+ 6z + 12z^2 + 20z^3 + 30z^4 }{6}
\end{eqnarray}
\begin{eqnarray}
	F''_{\xi}(1) = \frac{70}{6} = \frac{35}{3}
\end{eqnarray}
\begin{eqnarray}
	\sigma^2 = F''_{\xi}(1) + F'_{\xi}(1) + [F'_{\xi}(1)]^2  = \frac{35}{3} + \frac{7}{2} - \frac{49}{4}  = \frac{35}{12}
\end{eqnarray}
\textbf{Answer verified}

\subsection{}
%problem6.7
In order to do this problem with 6.6, we need the generating function
\begin{eqnarray}
	F_{\xi}(z) = \sum_{k=0}^{\infty} \frac{ a^k }{k!}e^{-a} z^k = e^{a(z-1)}
\end{eqnarray}
\begin{eqnarray}
	F'_{\xi}(z) = a e^{a (z-1)} \\
	F'_{\xi}(1) = a e^{0}  = a = \textbf{E}\xi\\
\end{eqnarray}
\begin{eqnarray}
	F''_{\xi}(z) = a^2 e^{a (z-1)} \\
	F''_{\xi}(1) = a^2
\end{eqnarray}
\begin{eqnarray}
	\sigma^2 = F''_{\xi}(1) + F'_{\xi}(1) - [F'_{\xi}(1)]^2  = a^2 + a - a^2 = a
\end{eqnarray}

\subsection{}
%problem6.8
In order to do this problem with 6.6, we need the generating function
\begin{eqnarray}
	F_{\xi}(z) = \sum_{k=0}^{\infty} \frac{ a^k }{(1+a)^{k+1}} z^k = \frac{-a-1}{(a+1) (a z-a-1)}
\end{eqnarray}
\begin{eqnarray}
	F'_{\xi}(z) = \frac{(a+1) a}{(a+1) (a z-a-1)^2} = \frac{a}{(a z-a-1)^2} \\
	F'_{\xi}(1) = a =  \textbf{E}\xi\\
\end{eqnarray}
\begin{eqnarray}
	F''_{\xi}(z) = \frac{-2 a^2}{(a z-a-1)^3} \\
	F''_{\xi}(1) = 2 a^2
\end{eqnarray}
\begin{eqnarray}
	\sigma^2 = F''_{\xi}(1) + F'_{\xi}(1) - [F'_{\xi}(1)]^2  = 2 a^2 + a - a^2 = a^2 - a = \textbf{D}\xi
\end{eqnarray}

\textbf{Answer not verified}

\subsection{}
%problem6.9
These two distributions have generating functions:
\begin{eqnarray}
	F_{\xi_1} = e^{a(z-1)} \\
	F_{\xi_2} = e^{a'(z-1)}
\end{eqnarray}
which means that $\eta$ has this generating function:
\begin{eqnarray}
	F_{\eta} =F_{\xi_1}F_{\xi_2} = e^{a(z-1)}e^{a'(z-1)} = e^{(a'+ a)(z-1)}  \\
\end{eqnarray}
Which, according to theorem 6.2 means there that $\eta$ is a Poisson distribution with mean $a+a'$.
\textbf{Answer not verified}

\subsection{}
%problem6.10
For any given experiment we can define an individual generating function: $F_i(z) = q_i + z p_i$ meaning that for the entire function
\begin{eqnarray}
	F(z) = \prod_{i=1}^n F_i(z) = \prod_{i=1}^n (q_i + z p_i)
\end{eqnarray}
The trick that Feller uses at this point is to take the log
\begin{eqnarray}
	\log F(z) = \sum_{i=1}^n \log F_i(z) = \sum_{i=1}^n \log (q_i + z p_i) \\
	\log F(z) = \sum_{i=1}^n \log (1 - p_i + z p_i) = \sum_{i=1}^n \log (1 - p_i( z -1))
\end{eqnarray}
We were, however, told that the largest probability goes to zero so we take the taylor series of each term, $\log (1-x) \approx - x$
\begin{eqnarray}
	\lim_{n \rightarrow \infty} \log F(z) =  \sum_{i=1}^n \log (1 - p_i( z -1)) = \sum_{i=1}^n  p_i( 1 - z ) = - \lambda (z - 1) \\
	\lim_{n \rightarrow \infty} F(z) = e^{- \lambda (z - 1)}
\end{eqnarray}
And, according to theorem 6.2, the probability distribution is Poissonian.
\textbf{Answer verified}

\subsection{}
%problem6.11

\begin{eqnarray}
	p_{\xi}(x) = \frac{1}{2} e^{-|x|} \\
	f_{\xi}(t) = \int_{-\infty}^{\infty} \frac{1}{2} e^{ i x t } e^{-|x|} dx \\
	f_{\xi}(t) = \frac{1}{t^2+1}
\end{eqnarray}

\textbf{Answer verified}

\subsection{}
%problem6.12
\begin{eqnarray}
	f_{\xi}(t) = \frac{1}{t^2+1} \\
	f'_{\xi}(t) =  -\frac{2 t}{\left(t^2+1\right)^2} = i \textbf{E}\xi \\
	f'_{\xi}(0) =  i \textbf{E}\xi = 0 \\
	f''_{\xi}(t) = \frac{8 t^2}{\left(t^2+1\right)^3}-\frac{2}{\left(t^2+1\right)^2} \\
	f''_{\xi}(0) = - \sigma^2 = -2
\end{eqnarray}

\textbf{Answer verified}

\subsection{}
%problem6.13
For a uniform distribution
\begin{eqnarray}
	p(x) = \frac{1}{b-a}
\end{eqnarray}
Therefore the characteristic is:
For a uniform distribution
\begin{eqnarray}
	f(t) = \frac{1}{b-a} \int_a^b e^{i x t} dx = \frac{i \left(e^{i a t}-e^{i b t}\right)}{t(b-a)}
\end{eqnarray}
\textbf{Answer verified}

\subsection{}
%problem6.14
\begin{eqnarray}
	f_{\xi} (t) = e^{-a|t|} \\
	p(x) = \frac{1}{2 \pi} \inf_{\infty}^{\infty} e^{-a|t|} e^{- i x t} \\
	p(x) = \frac{a}{\pi(a^2+x^2)}
\end{eqnarray}
\textbf{Answer verified}

\subsection{}
%problem6.15
Looking at the characteristic, there is clearly going to be a discontinuity in the deirvative at $t=0$, meaning that the derivative does not exist at that point.  This jibes well with what we've done before because we proved in problem 43 on page 24 that the probability distribution this characteristic produces does not have a mean or a standard deviation.

\textbf{Answer not verified}

\subsection{}
%problem6.16
For the single die, $E\xi = 3.5$ and $D\xi = \frac{35}{12}$.  The distribution for $n$ dice rolls is binomial but in the limit of large $n$, it approximates a normal distribution with $E\xi = 3.5n$ and $D\xi = n \frac{35}{12}$
\begin{eqnarray}
	P\{ 3450 \leq x \leq 3550  \} = \int_{3450}^{3550} \sqrt{\frac{1}{\frac{70 \pi \cdot 10^3}{12}}}e^{-\frac{(x-3500)^2}{\frac{70 \cdot 10^3}{12}}} dx \\
	= 0.645461  \\
\end{eqnarray}
\textbf{Answer not verified}

\subsection{}
%problem6.17

We want to prove here that the distribution satisfies the Lyapunov condition:
\begin{equation}
	\lim_{n\rightarrow \infty} \frac{1}{B_n^3} \sum_{k=1}^n \textbf{E}|\xi_k - a_k|^3 = 0
\end{equation}
Where
\begin{equation}
	B_n^2 = \textbf{D}S_n = \sum_{k=1}^n \sigma^2_k
\end{equation}

The machinations of which are left to the reader... sorry!

\textbf{Answer not verified}

%%answer template
%\subsection{}
%%problem n.n
%
%
%\begin{equation}
%
%\label{answern.n}
%\end{equation}
%\textbf{Answer [not] verified}


\section{Bonus lecture: Primitive Roots}
We work in the multiplicative group $\mathbb Z_p^\times$ or $\mathbb Z_p^\star$ consisting of $(\mathbb Z_p\setminus\{0\},\times,1)$.
\begin{definition}
    The order of $x\in\mathbb Z_p^\star$ is the least $n\in\mathbb N$ with $x^n\equiv 1\pmod{p}$.
\end{definition}
It is trivial that an order indeed exists by F$\ell$T.
\begin{example}
    The order of $2$ in $\mathbb Z_7^\star$ is $3$, and the order of $3$ there is $6$.
\end{example}
\begin{definition}
    A $x\in \mathbb Z_p^\star$ is said to be a primitive root, or generator, if $x$ has order $p-1$.
\end{definition}
Indeed, if such an $x$ exists, then every element in $\mathbb Z_p^\star$ can be written in the form $x^n$ for some $n\in\mathbb N$.
\begin{theorem}\label{primitive}
    For any $p$, there exists a generator.
\end{theorem}
That is, $\mathbb Z_p^\star\cong C_{p-1}$.
\begin{proposition}
    If $x$ has order $n$, then if $x^d\equiv 1\pmod{p}$, then $n|d$.
\end{proposition}
\begin{proof}
    Trivial by minimality of order.
\end{proof}
\begin{proposition}
    Let $x$ have order $a$ and $y$ have order $b$ where $a,b$ are coprime, then $xy$ has order $ab$.
\end{proposition}
\begin{proof}
    Obviously $(xy)^{ab}\equiv 1\pmod{p}$.
    Conversely, if $(xy)^d\equiv 1\pmod{p}$, then $x^dy^d\equiv 1\pmod{p}$, so $x^{bd}\equiv 1\pmod{p}$ hence $a|bd$, but $a,b$ are coprime, so $a|d$.
    Similarly $b|d$, thus $ab|d$ since again $a,b$ are coprime.
\end{proof}
Also the condition of $a,b$ being coprime is necessary.
Indeed, by the same idea, we obtain
\begin{proposition}
    Let $x$ have order $a$ and $y$ have order $b$, then there is some element have order $l$, the LCM of $a,b$.
\end{proposition}
\begin{proof}
    Let
    $$a=p_1^{a_1}\cdots p_k^{a_k}q_1^{c_1}\cdots q_j^{c_j},b=p_1^{b_1}\cdots p_k^{b_k}q_1^{d_1}\cdots q_j^{d_j}$$
    where $p_i,q_i$ are distinct primes where $a_i\ge b_i,c_i\le d_i$.
    Then $x^{q_1^{c_1}\cdots q_j^{c_j}}$ has order $p_1^{a_1}\cdots p_k^{a_k}$, and $y^{p_1^{b_1}\cdots p_k^{b_k}}$ has order $q_1^{d_1}\cdots q_j^{d_j}$.
    So $x^{q_1^{c_1}\cdots q_j^{c_j}}y^{p_1^{b_1}\cdots p_k^{b_k}}$ has order $p_1^{a_1}\cdots p_k^{a_k}q_1^{d_1}\cdots q_j^{d_j}=l$ by above.
\end{proof}
\begin{corollary}
    If $d$ is the biggest order in $\mathbb Z_p^\star$, then any order divides $d$.
\end{corollary}
Note that all the above works in any finite abelian group.
We now bring in number theory.
\begin{proposition}
    Let $f$ be a polynomial in $\mathbb Z_p$ or degree $k>0$, the $f$ has at most $k$ roots in $\mathbb Z_p$.
\end{proposition}
\begin{proof}
    Division and induction.
\end{proof}
\begin{remark}
    This is NOT true in $\mathbb Z_n$ in general since we used division (which requires it to be a field).
    For example, $x^2\equiv 1$ has $4$ roots modulo $8$.
\end{remark}
\begin{proof}[Proof of Theorem \ref{primitive}]
    Take $d$ to be the biggest order in $\mathbb Z_p^\star$.
    We want $d=p-1$.
    Note that for any $x\in\mathbb Z_p^\star$, $x^d\equiv 1\pmod{p}$ by the preceding corollary.
    But the polynomial $x^d-1$ can have at most $d$ roots.
    Then we must have $d\ge p-1$, but $d|p-1\implies d\le p-1$, hence $d=p-1$.
\end{proof}
There are some consequences by this theorem.
We let $g$ be the generator hereafter.
For example, we can prove Fermat by takeing $x\equiv g^a\pmod{p}$, then $x^{p-1}\equiv g^{a(p-1)}\equiv 1\pmod{p}$.\\
From now on we set $p$ odd.
So we have $\mathbb Z_p^\star$ contains exactly $(p-1)/2$ roots by again looking at the exponent.
Also, a non-square times a non-square gives a square.
Now we look at $-1$.
Then we must have $g^{(p-1)/2}\equiv -1\pmod{p}$, so $-1$ is a square if and only if $(p-1)/2$ is even, which happens if and only if $p\equiv 1\pmod{4}$.\\
For Wilson's Theorem, we can simply write $(p-1)!\equiv g^{1+2+3+\cdots +(p-1)}\equiv g^{p(p-1)/2}\equiv (-1)^p\equiv -1\pmod{p}$.\\
For $p=3k+1$, the cubes are exactly $g^3,g^6,\ldots,g^{3k}$, so there are exactly $k=(p-1)/3$ cubes.
Otherwise, everything is a cube (as shown in example sheets).
\begin{remark}
    The proof does not tell us how we can find the generator as it only shows existence.
    Even today, it is not well-understood which element would be a generator.
    For example, when is $2$ a generator?
    It is true for $\mathbb Z_5$ but not $\mathbb Z_7$.
    Nobody knows for which primes $2$ is a primitive roots, it is not even known if $2$ is a primitive root modulo infinitely many primes $p$ (Artin's Conjecture).
\end{remark}


%!TEX root = ../thesis.tex
\chapter{Supplementary Material}
\section{Matrix definitions}\label{app:GellMann}
The standard form of the $2\times 2$ Pauli matrices is
%
\begin{equation}
\sigma_1 = \begin{pmatrix}
0 & 1\\
1 & 0
\end{pmatrix},
\hspace{0.5cm}
\sigma_2 = \begin{pmatrix}
0 & -i\\
i & 0
\end{pmatrix},
\hspace{0.5cm}
\sigma_3 = \begin{pmatrix}
1 & 0\\
0 & -1
\end{pmatrix}\\, .
\end{equation}
%
The $4\times 4$ gamma matrices in the Dirac representation in Euclidean space are therefore given by
%
\begin{equation}
\gamma_i = \begin{pmatrix}
0 & \sigma_i\\
-\sigma_i & 0
\end{pmatrix},
\hspace{0.5cm}
\gamma_4 = \begin{pmatrix}
I & 0\\
0 & -I
\end{pmatrix}\, .
\end{equation}
%
The gamma matrices in this representation are evidently Hermitian, and satisfy the anti-commutation relationship,
%
\begin{equation}
\lbrace \gamma_\mu\, ,\, \gamma_\mu \rbrace = 2\,\delta_{\mu\nu}\, .
\end{equation}
%
The Gell-Mann matrices $\lambda_a$ are related to the generators of $SU(3)$, $t_a$, by~\cite{peskin2018introduction}
%
\begin{equation}
t_a = \frac{\lambda_a}{2}\, .
\end{equation}
%
The matrices are given by
%
\begin{align}
\lambda _ { 1 } &= \left( \begin{array} { c c c } { 0 } & { 1 } & { 0 } \\ { 1 } & { 0 } & { 0 } \\ { 0 } & { 0 } & { 0 } \end{array} \right) \quad \lambda _ { 2 } = \left( \begin{array} { c c c } { 0 } & { - i } & { 0 } \\ { i } & { 0 } & { 0 } \\ { 0 } & { 0 } & { 0 } \end{array} \right) \quad \lambda _ { 3 } = \left( \begin{array} { c c c } { 1 } & { 0 } & { 0 } \\ { 0 } & { - 1 } & { 0 } \\ { 0 } & { 0 } & { 0 } \end{array} \right)\nonumber\\
\lambda _ { 4 } &= \left( \begin{array} { c c c } { 0 } & { 0 } & { 1 } \\ { 0 } & { 0 } & { 0 } \\ { 1 } & { 0 } & { 0 } \end{array} \right) \quad \lambda _ { 5 } = \left( \begin{array} { c c c } { 0 } & { 0 } & { - i } \\ { 0 } & { 0 } & { 0 } \\ { i } & { 0 } & { 0 } \end{array} \right)\nonumber\\
\lambda _ { 6 } &= \left( \begin{array} { c c c } { 0 } & { 0 } & { 0 } \\ { 0 } & { 0 } & { 1 } \\ { 0 } & { 1 } & { 0 } \end{array} \right) \quad \lambda _ { 7 } = \left( \begin{array} { c c c } { 0 } & { 0 } & { 0 } \\ { 0 } & { 0 } & { - i } \\ { 0 } & { i } & { 0 } \end{array} \right) \quad \lambda _ { 8 } = \frac { 1 } { \sqrt { 3 } } \left( \begin{array} { c c c } { 1 } & { 0 } & { 0 } \\ { 0 } & { 1 } & { 0 } \\ { 0 } & { 0 } & { - 2 } \end{array} \right)\, .
\end{align}

\section{Gauge transformation of $F_{\mu\nu}$}\label{app:GTF}
Recall from Eq.~\eqref{eq:FieldStrengthTensor} that the field strength tensor is defined to be
%
\begin{equation}
F_{\mu\nu}=\partial_\mu A_\nu - \partial_\nu A_\mu + ig[A_\mu,\,A_\nu]\, .
\label{eq:FieldStrength(app)}
\end{equation}
%
To calculate how $F_{\mu\nu}$ transforms under a gauge transformation, we will also make use of the gauge transformation property for $A_\mu$,
%
\begin{equation}
A_\mu \rightarrow \Omega\,A_\mu\,\Omega^\dag + \frac{i}{g}\,(\partial_\mu\,\Omega)\,\Omega^\dag\, .
\label{eq:PotentialGT(app)}
\end{equation}
%
We will also make repeated use of the unitarity of $\Omega$, specifically the fact that
%
\begin{align}
\partial_\mu \left(\Omega\, \Omega^\dagger\right) &= \left(\partial_\mu\,\Omega\right)\Omega^\dagger + \Omega\left(\partial_\mu\, \Omega^\dagger\right)\nonumber\\
&=\partial_\mu \, I\nonumber\\
&= 0\nonumber\\
\implies & \left(\partial_\mu\,\Omega\right)\Omega^\dagger = -\Omega\left(\partial_\mu\, \Omega^\dagger\right)\, .
\end{align}
%
Substituting Eq.~\eqref{eq:PotentialGT(app)} into Eq.~\eqref{eq:FieldStrength(app)} we obtain
%
\begin{align}
F_{\mu\nu} \rightarrow &\left(\partial_\mu \, \Omega\right)A_\nu \, \Omega^\dagger + \Omega\left(\partial_\mu\,A_\nu\right)\Omega^\dagger + \Omega \, A_\nu\left(\partial_\mu \, \Omega^\dagger\right)\nonumber\\
& + \frac{i}{g}\left(\partial_\nu\,\partial_\mu\, \Omega\right)\Omega^\dagger + \frac{i}{g}\left(\partial_\nu\,\Omega\right)\left(\partial_\mu\,\Omega^\dagger\right)\nonumber\\
& - \left(\partial_\nu \, \Omega\right)A_\mu \, \Omega^\dagger - \Omega\left(\partial_\nu\,A_\mu\right)\Omega^\dagger - \Omega \, A_\mu\left(\partial_\nu \, \Omega^\dagger\right)\nonumber\\
& - \frac{i}{g}\left(\partial_\nu\,\partial_\mu\, \Omega\right)\Omega^\dagger - \frac{i}{g}\left(\partial_\mu\,\Omega\right)\left(\partial_\nu\,\Omega^\dagger\right)\nonumber\\
& +ig \, \Omega \, A_\mu \, A_\nu \, \Omega^\dagger - \left(\partial_\mu \, \Omega\right)A_\nu \, \Omega^\dagger + \Omega\,A_\mu\left(\partial_\nu \, \Omega^\dagger\right) + \frac{i}{g}\left(\partial_\mu\,\Omega\right)\left(\partial_\nu\,\Omega^\dagger\right)\nonumber\\
& -ig \, \Omega \, A_\nu \, A_\mu \, \Omega^\dagger + \left(\partial_\nu \, \Omega\right)A_\mu \, \Omega^\dagger - \Omega\,A_\nu\left(\partial_\mu \, \Omega^\dagger\right) - \frac{i}{g}\left(\partial_\nu\,\Omega\right)\left(\partial_\mu\,\Omega^\dagger\right)\, .
\end{align}
Cancelling off terms reduces the above expression to the desired result,
%
\begin{align}
F_{\mu\nu} \rightarrow &\, \Omega\left(\partial_\mu\,A_\nu\right)\Omega^\dagger - \Omega\left(\partial_\nu\,A_\mu\right)\Omega^\dagger +ig\,\Omega\left[A_\mu,\,A_\nu\right]\Omega^\dagger\nonumber\\
&= \Omega\,F_{\mu\nu}\,\Omega^\dagger\, .
\end{align}

\section{Wilson line gauge transformation}\label{app:WilsonLineGT}

We wish to show that the Wilson line obeys the gauge transformation property
%
\begin{equation}
U_\mu(x) \rightarrow \Omega(x)\,U_\mu(x)\,\Omega^\dagger(x)
\end{equation}
%
We can apply a gauge transformation to $A_\mu$ to obtain
%
\begin{equation}
U_\mu(x)\rightarrow\mathcal{P}\exp\left(-iag\int_0^1\,dt\, \Omega(x(t))\,A_\mu(x(t))\,\Omega^\dagger(x(t)) + \frac{i}{g}\,(\partial_\mu\,\Omega(x(t)))\,\Omega^\dagger(x(t))\right)\, ,
\label{eq:WilsonLineGT(app)}
\end{equation}
%
where $x(t)=x+at\hat{\mu}$. To simplify this expression, we need to make use of an equivalent expression for the path-ordered exponential. For a generic path-ordered exponential of a function $a(t)$ we can write
%
\begin{align}
&\mathcal{P}\left(\int_0^t dt^\prime\, a(t^\prime)\right) = \lim_{N\rightarrow\infty} \left( e^{a(t_N)\Delta t}\,e^{a(t_{N-1})\Delta t}\,\cdots\,e^{a(t_0)\Delta t}\right)\nonumber\\
\implies &\mathcal{P}\left(-\int_0^t dt^\prime\, a(t^\prime)\right) = \lim_{N\rightarrow\infty} \left( e^{a(t_0)\Delta t}\,e^{a(t_{1})\Delta t}\,\cdots\,e^{a(t_N)\Delta t}\right)\, , \label{eq:OrderedExponentialLimit}
\end{align}
%
where $\lbrace t_0 = 0, \, \cdots\, , \, t_N = t\rbrace$ is a partition of the integration range into equal slices of length $\Delta t = \frac{t}{N}$. We also wish to utilise the fact that
%
\begin{equation}
e^{ig\,\Omega(x)\,A_\mu(x)\,\Omega^\dagger(x)} = \Omega(x)\,e^{ig\,A_\mu(x)}\,\Omega^\dagger(x)\, ,
\end{equation}
as can be easily derived from writing the exponential as an infinite sum.\\

Writing Eq.~\eqref{eq:WilsonLineGT(app)} in the form of Eq.~\eqref{eq:OrderedExponentialLimit} and employing the fact that path-ordering permits us to only retain the first order terms in the Baker-Campbell-Hausdorff identity (see Eq.~\eqref{eq:BCH}), we find that
%
\begin{align}
U_\mu(x)\rightarrow \lim_{N\rightarrow\infty} &\left( \Omega(x_0)\,e^{ig\,A_\mu(x_0)\,\Delta x}\,\Omega^\dagger(x_0) \exp\left((\partial_\mu\,\Omega(x_0))\,\Omega^\dagger(x_0)\right)\right.\nonumber\\
&\times \Omega(x_1)\,e^{ig\,A_\mu(x_1)\,\Delta x}\,\Omega^\dagger(x_1) \exp\left((\partial_\mu\,\Omega(x_1))\,\Omega^\dagger(x_1)\Delta x\right) \times \, \cdots \nonumber\\
&\left. \times \, \Omega(x_{N-1})\,e^{ig\,A_\mu(x_{N-1})\,\Delta x}\,\Omega^\dagger(x_{N-1}) \exp\left((\partial_\mu\,\Omega(x_{N-1}))\,\Omega^\dagger(x_{N-1})\Delta x\right) \right)\, , \label{eq:WilsonLineExpansion(app)}
\end{align}
%
where $\Delta x = \frac{a}{N}$, $x_0=x$ and $x_N = x+a\hat{\mu}$. However, in the limit as $N\rightarrow\infty$, $\exp\left((\partial_\mu\,\Omega(x_i))\,\Omega^\dagger(x_i)\Delta x \right)$ is precisely the parallel transport operator for $\Omega$ over the distance $\Delta x$, and hence satisfies
%
\begin{equation}
\Omega^\dagger(x_i)\exp\left((\partial_\mu\,\Omega(x_i))\,\Omega^\dagger(x_i)\Delta x \right) = \Omega^\dagger(x_{i+1})\, .
\label{eq:RotationTransport(app)}
\end{equation}
%
Substituting Eq.~\eqref{eq:RotationTransport(app)} into Eq.~\eqref{eq:WilsonLineExpansion(app)} eliminates all the gauge transformation terms except for the first and last transformations. This then reduces to our desired result,
%
\begin{equation}
U_\mu(x)\rightarrow \Omega(x)\,U_\mu(x)\,\Omega^\dagger(x+a\hat{\mu})\, .
\end{equation}

\section{Taylor expansion of $P_{\mu\nu}$}\label{app:TEPlaquette}
Here we sketch out how to construct the field strength tensor in terms of the plaquette. We will neglect a discussion of the $\mathcal{O}(a^3)$ terms found in Eq.~\eqref{eq:PlaquetteExponantial(app)} as a more careful treatment outside the scope of this work is required to show that these higher order terms do not render the expansion shown in Eq.~\eqref{eq:PlaquetteExpansion} incorrect. However, this derivation highlights the key steps in arriving at the desired result. First, we recall the definitions of the gauge link in the continuum
%
\begin{equation}
U_\mu(x) = \mathcal{P}\exp\left(-iag\int_0^1 \,dt \,A_\mu(x + at\hat{\mu})\right)  \, ,
\label{eq:GaugeLink(app)}
\end{equation}
%
and the plaquette formed from the product of the gauge links around a $1\times 1$ loop
\begin{equation}
P_{\mu\nu} = U_\mu(x)\,U_\nu(x+a\hat{\mu})\,U^\dagger_\mu(x+a\hat{\nu})\,U^\dagger_\nu(x)\, .
\label{eq:Plaquette(app)}
\end{equation}
%
We can approximate the integral in Eq.~\eqref{eq:GaugeLink(app)} by Taylor expanding $A_\mu$ around $a=0$ and explicitly evaluating the integral. Note that once we have Taylor expanded $A_\mu$, the term within the integral commutes with itself for all values of $t$, allowing us to omit the path ordering from Eq.~\eqref{eq:GaugeLink(app)}. Performing the expansion, we find that
%
\begin{align}
U_\mu(x)&=\exp\left(-iag\int_0^1 dt \left(A_\mu\left(x\right) + at\partial_\mu A_\mu(x) + \mathcal{O}(a^3)\right)\right)\nonumber\\
&=\exp\left(-iag A_\mu\left(x\right) - \frac{1}{2}ia^2 g\, \partial_\mu A_\mu\left(x\right)+ \mathcal{O}(a^3)\right)\, . \label{eq:UTaylor}
\end{align}
%
Similarly, we evaluate
%
\begin{align}
U_\nu(x+a\hat{\mu}) &= \exp\left(-iag\int_0^1 dt \left(A_\nu(x) + a\partial_\mu A_\nu + at\partial_\nu A_\nu + \mathcal{O}(a^3)\right)\right)\nonumber\\
&= \exp\left(-iag A_\nu(x) - ia^2g\,\partial_\mu A_\nu(x) - \frac{1}{2}ia^2 g\,\partial_\nu A_\nu(x)+ \mathcal{O}(a^3)\right)\, . \label{eq:UTaylor2}
\end{align}
%
We will also require the Baker-Campbell-Hausdorff identity for non-Abelian matrix exponentials
%
\begin{equation}
\exp(A)\,\exp(B) = \exp\left(A + B +\frac{1}{2}[A,\,B]\right)\, .
\label{eq:BCH}
\end{equation}
%
Substituting Eq.~\eqref{eq:UTaylor} and Eq.~\eqref{eq:UTaylor2} into Eq.~\eqref{eq:Plaquette(app)} and retaining only terms up to $\mathcal{O}(a^2)$ we find that
%
\begin{alignat}{2}
P_{\mu\nu} &\simeq &&\exp\left(-ig\left(a\,A_\mu(x)+\frac{1}{2}a^2\,\partial_\mu A_\mu(x) \right)\right)\nonumber\\
& &&\times\exp\left(-ig\left(aA_\nu(x) + \frac{1}{2}a^2\,\partial_\nu A_\nu(x) + a^2\,\partial_\mu A_\nu(x)\right)\right)\nonumber\\
& &&\times\exp\left(ig\left(aA_\mu(x) + \frac{1}{2}a^2\,\partial_\mu A_\mu(x) + a^2\,\partial_\nu A_\mu(x)\right)\right)\nonumber\\
& &&\times\exp\left(ig\left(a\,A_\nu(x)+\frac{1}{2}a^2\,\partial_\nu A_\nu(x)\right)\right)\nonumber\\
&\simeq &&\exp\left(-ia^2g\,\partial_\mu A_\nu(x) + ia^2g\,\partial_\nu A_\mu(x) +\frac{a^2g^2}{2}[A_\mu(x),\,A_\nu(x)] - \frac{a^2g^2}{2}[A_\nu(x),\,A_\mu(x)]\right)\nonumber\\
&= &&\exp\left(-ia^2g F_{\mu\nu}(x) + \mathcal{O}(a^3)\right)\, .
\label{eq:PlaquetteExponantial(app)}
\end{alignat}
\section{Properties of the adjoint representation}
\label{app:RepMapProof}
Consider the mapping $H:SU(3)^\text{fundamental}\rightarrow SU(3)^\text{adjoint}$ defined by
%
\begin{equation}
\left[H(U)\right]_{ij} = \frac{1}{2}\Tr\left(\lambda_i \, U \, \lambda_j\, U^\dagger\right)\, .
\label{eq:FundMap(app)}
\end{equation}
%
We want to show that for $U,V\in SU(3)^F$
%
\begin{equation}
\left[H(U)\right]_{ij}\left[H(V)\right]^{jk} = \left[H(U V)\right]_i^{~k}\, .
\label{eq:MapPreserve(app)}
\end{equation}
%
To do this, we will need to make use of the following Fierz completeness relations for the $SU(3)$  generators
%
\begin{equation}
\lambda _ { b } ^ { a } \lambda _ { c } ^ { d } = 2\, \delta _ { c } ^ { a } \delta _ { b } ^ { d } - \frac { 2 } { 3 } \delta _ { b } ^ { a } \delta _ { c } ^ { d }\, .
\end{equation}
%
Substituting Eq.~\eqref{eq:FundMap(app)} into Eq.~\eqref{eq:MapPreserve(app)} and noting that repeated indicies are summed over, we have
\begin{align}
\left[H(U)\right]_{\alpha\beta}\left[H(V)\right]^{\beta\gamma} &= \frac{1}{2} \lambda^\alpha_{ab}\, U_{bc}\,\lambda^\beta_{cd}\, U^\dagger_{da} \times \frac{1}{2} \lambda^\beta_{ef}\, V_{fg}\,\lambda^\gamma_{gh}\, V^\dagger_{he}\nonumber\\
&=\frac{1}{2} U_{bc}\,U^\dagger_{da}\,V_{fg}\,V^\dagger_{he}\,\lambda^\alpha_{ab}\, \lambda^\gamma_{gh} \left(\delta_{cf}\,\delta_{de} - \frac{1}{3}\delta_{cd}\delta_{ef}\right)\nonumber\\
&=\frac{1}{2}U_{bc} \, V_{cg} \, \lambda^\gamma_{gh} \, V^\dagger_{hd} \, U^\dagger_{da} \, \lambda^\alpha_{ab} - \frac{1}{6} U_{bc} \, U^\dagger_{ca} \, V^\dagger_{he} \, V_{eg} \lambda^\alpha_{ab}\, \lambda^\gamma_{gh}\nonumber\\
&= \frac{1}{2}U_{bc} \, V_{cg} \, \lambda^\gamma_{gh} \, V^\dagger_{hd} \, U^\dagger_{da} \, \lambda^\alpha_{ab} - \frac{1}{6}\delta_{ba}\delta_{hg} \lambda^\alpha_{ab}\, \lambda^\gamma_{gh}\nonumber\\
&=\frac{1}{2}\Tr\left(U \, V \, \lambda^\gamma \, (U \, V)^\dagger \lambda^\alpha\right) - \frac{1}{6}\Tr\left(\lambda^\alpha\right) \, \Tr\left(\lambda^\gamma\right)\, .
\end{align}
Making use of the cyclic property of the trace and the fact that the Gell-Mann matrices are traceless, we find the desired result,
%
\begin{align}
\left[H(U)\right]_{\alpha\beta}\left[H(V)\right]^{\beta\gamma} &= \frac{1}{2}\Tr\left(\lambda^\alpha U \, V \, \lambda^\gamma \, (U \, V)^\dagger \right)\nonumber\\
&= \left[H(UV)\right]_{\alpha\gamma} \, .
\end{align}\\

We also wish to show that for $U^A\in SU(3)^A$ and $U\in SU(3)^F$ that
%
\begin{equation}
\Tr\left(U^A\right) = \left|\Tr(U)\right|^2 -1\, .
\end{equation}
%
Making use of Eq.~\eqref{eq:FundMap(app)}, we have
\begin{align}
\Tr\left(U^A\right) &= \sum_{\alpha=1}^8 \frac{1}{2}\Tr\left(\lambda^\alpha \, U_\mu(x) \, \lambda^\alpha \, U_\mu^\dagger(x)\right)\nonumber\\
&= \frac{1}{2}U_{bc} \, U^\dagger_{da} \, \lambda_{ab}^\alpha \, \lambda_{cd}^\alpha\nonumber\\
&= U_{bc} \, U^\dagger_{da} \left(\delta_{ad}\,\delta_{bc} - \frac{1}{3}\delta_{ab}\,\delta_{cd}\right)\nonumber\\
&= \Tr\left(U\right) \, \Tr\left( U^\dagger \right) - \frac{1}{3}\Tr\left(U\, U^\dagger\right)\nonumber\\
&= \left|\Tr(U)\right|^2 -1
\end{align}

\section{Cooling algorithm derivation}\label{app:Cooling}
We wish to find $a_i$ such that
%
\begin{equation}
U_\mu^\prime = a_3 \, a_2 \, a_1 \, U_\mu\, ,
\label{eq:UPrime(app)}
\end{equation}
%
minimises the local Wilson action associated with the link $U_\mu$. This minimisation is equivalent to maximising
%
\begin{equation}
R = \operatorname{Re} \Tr (U_\mu^\prime \, \bar{U})\, .
\label{eq:CoolFunctional(app)}
\end{equation}
%
To determine the optimal choice for the $a_i$, we define the following three functions, $F_i(V)$, such that $F_i : SU(3)\rightarrow SU(2)$:
%
\begin{align}
F_1(V) &=\frac{1}{k_1} \begin{pmatrix}
\frac{1}{2}\left(V_{11} + V_{22}^*\right) & \frac{1}{2}\left(V_{12} - V_{21}^*\right) & 0\\
\frac{1}{2}\left(V_{21} - V_{12}^*\right) & \frac{1}{2}\left(V_{11}^* + V_{22}\right) & 0\\
0 & 0 & k_1
\end{pmatrix}\, ,\\
F_2(V) &=\frac{1}{k_2} \begin{pmatrix}
\frac{1}{2}\left(V_{11} + V_{33}^*\right) & 0 &\frac{1}{2}\left(V_{13} - V_{31}^*\right)\\
0 & k_2 & 0\\
\frac{1}{2}\left(V_{31} - V_{13}^*\right) & 0 & \frac{1}{2}\left(V_{11}^* + V_{33}\right)
\end{pmatrix}\, ,\\
F_3(V) &=\frac{1}{k_3} \begin{pmatrix}
k_3 & 0 & 0\\
0 & \frac{1}{2}\left(V_{22} + V_{33}^*\right) & \frac{1}{2}\left(V_{23} - V_{32}^*\right)\\
0 & \frac{1}{2}\left(V_{32} - V_{23}^*\right) & \frac{1}{2}\left(V_{22}^* + V_{33}\right)
\end{pmatrix}\, ,
\end{align}
%
where $k_i^2$ is the determinant of the $2\times 2$ $SU(2)$ sub-block. The $\frac{1}{k_i}$ factor therefore fixes the determinant such that $\det(F_i(V))=1$. We now wish to find a suitable $V$ to define our $a_i$'s. Consider the first case, $U^{(1)}_\mu = a_1\,U_\mu$, and let $a_1 = F_1(U_\mu\,\bar{U})^\dagger$. It is worth stating explicitly that it is this step where the fact that a sum of $SU(2)$ matrices is proportional to an $SU(2)$ matrix is utilised. Despite the fact that $U_\mu\,\bar{U} \notin SU(3)$, $U_\mu \, \tilde{U}_\alpha\in SU(3)~\forall\, \alpha$, which implies that $F_i(U_\mu\, \tilde{U}_\alpha)\in SU(2)$. Then we have, for example,
%
\begin{align}
F_1(U_\mu \, \bar{U}) &= \frac{1}{k_1}\sum_\alpha
\begin{pmatrix}
\frac{1}{2}\left((U_\mu \, \tilde{U}_\alpha)_{11} + (U_\mu \, \tilde{U}_\alpha)^*_{22}\right) & \frac{1}{2}\left((U_\mu \, \tilde{U}_\alpha)_{12} - (U_\mu \, \tilde{U}_\alpha)^*_{21}\right) & 0\\
\frac{1}{2}\left((U_\mu \, \tilde{U}_\alpha)_{21} - (U_\mu \, \tilde{U}_\alpha)^*_{12}\right) & \frac{1}{2}\left((U_\mu \, \tilde{U}_\alpha)^*_{11} + (U_\mu \, \tilde{U}_\alpha)_{22}\right) & 0\\
0 & 0 & k_1
\end{pmatrix}\nonumber\\
&=\frac{1}{k_1}\sum_\alpha
\begin{pmatrix}
k_{1,\,\alpha}\,\left(F_1(U_\mu\,\tilde{U}_\alpha)\right)_{11} & k_{1,\,\alpha}\,\left(F_1(U_\mu\,\tilde{U}_\alpha)\right)_{12} & 0\\
k_{1,\,\alpha}\,\left(F_1(U_\mu\,\tilde{U}_\alpha)\right)_{21} & k_{1,\,\alpha}\,\left(F_1(U_\mu\,\tilde{U}_\alpha)\right)_{22} & 0\\
0 & 0 & k_1
\end{pmatrix}\nonumber\\
&=\frac{1}{k_1}\sum_\alpha
\left(\begin{array}{@{}c|c@{}}
k_{1,\,\alpha} \,F_1(U_\mu\,\tilde{U}_\alpha)_{2\times 2} & 0\\
  \hline
0 & k_1
\end{array}\right)
\label{eq:FExpansion}
\end{align}
%
The $2\times 2$ block in Eq.~\eqref{eq:FExpansion} is a sum over terms proportional to $SU(2)$ matrices, and hence the result is proportional to an $SU(2)$ matrix. Thus with the appropriate normalisation from the $k_1$ factor, we see that $F_1(U_\mu \, \bar{U})\in SU(2)$. The same result holds true for $F_2$ and $F_3$.\\

With the definition $a_1 = F_1(U_\mu\,\bar{U})^\dagger$, the functional in Eq.~\eqref{eq:CoolFunctional(app)} we are seeking to maximise can now be directly evaluated. Setting $U = U_\mu\,\bar{U}$, we can write Eq.~\eqref{eq:CoolFunctional(app)} as
%
\begin{align}
\Re\Tr(F_1(U)^\dagger\, U) &=\Re \left(\frac{1}{k_1}\left[ \frac{1}{2}\,U_{11} \left(U_{22} + U_{11}^*\right) - \frac{1}{2}\,U_{21} \left(U_{12} - U_{21}^*\right)\right.\right.\nonumber\\
&~~~~+ \left. \left.\frac{1}{2}\,U_{22} \left(U_{11} + U_{22}^*\right)
- \frac{1}{2}\,U_{12} \left(U_{21} - U_{12}^*\right) +k_1\,U_{33}\right]\right)\nonumber\\
&= \frac{1}{k_1}\Re\left(\frac{|U_{11}|^2}{2} + \frac{|U_{22}|^2}{2} + \frac{|U_{12}|^2}{2} + \frac{|U_{21}|^2}{2} + U_{11}\,U_{22} - U_{12}\,U_{21} + k_1\,U_{33}\right)\, .\label{eq:FunctionalExpanded}
\end{align}
%
Here we have used the fact that by the construction of $F_1(U)$, $k_1$ is real and can therefore be brought to the front of Eq.~\eqref{eq:FunctionalExpanded}. We now wish to make use of the known determinant of $F_1(U)$ to simplify this expression.
%
\begin{align}
\det(F_1(U)) &= \frac{1}{4\,k_1^2} \left(\left(U_{11}+ U_{22}^*\right)\left(U_{11}^* + U_{22}\right)-\left(U_{12} - U_{21}^*\right)\left( U_{21} - U_{12}^*\right)\right)\nonumber\\
&=\frac{1}{4\,k_1^2}\left( |U_{11}|^2 + |U_{22}|^2 + |U_{12}|^2 + |U_{21}|^2\right.\\
&~~~~+ U_{11}\,U_{22} + U_{11}^*\,U_{22}^* + U_{12}\,U_{21} + U_{12}^*\,U_{21}^*\big)\nonumber\\
&= \frac{1}{2\,k_1^2}\Re\left(\frac{|U_{11}|^2}{2} + \frac{|U_{22}|^2}{2} + \frac{|U_{12}|^2}{2} + \frac{|U_{21}|^2}{2} + U_{11}\,U_{22} - U_{12}\,U_{21}\right)\nonumber\\
&=1\, .
\end{align}
%
Substituting this determinant back into Eq.~\eqref{eq:FunctionalExpanded}, we find that
%
\begin{align}
\Re\Tr(F_1(U)^\dagger\, U) &= \Re\left(\frac{1}{k_1}(2k_1^2 + k_1\,U_{33}))\right)\nonumber\\
&=2\,k_1 + \Re\left(U_{33}\right)\, .
\end{align}
%
Finally, substituting back $U = U_\mu\,\bar{U}$ brings us to the desired result,
%
\begin{equation}
\Re \Tr (F_1(U_\mu\,\bar{U})^\dagger \, U_\mu \, \bar{U}) = 2k_1 + \Re\left( (U_\mu\, \bar{U})_{33}\right)\, .
\label{eq:MaximisedCooling}
\end{equation}
%
By the matrix structure of $F_1(V)$ it is apparent that $\Re(U_\mu\, \bar{U})_{33}$ is invariant under pre-multiplication by $a_1$, so it is clear that Eq.~\eqref{eq:MaximisedCooling} represents the maximum attainable value for this form of $U^{(1)}_\mu$. Similarly, we let $a_2=F_2(U_\mu\,\bar{U})^\dagger$ and $a_3=F_3(U_\mu\,\bar{U})^\dagger$ to obtain the final value of $U^\prime_\mu$ according to Eq.~\eqref{eq:UPrime(app)}. This construction results in a choice of $U_\mu^\prime \in SU(3)$ that minimises the local Wilson action.


\section{Modeling \& Optimization}
Now that we know some ways to find extreme values, we can apply them to answer optimization problems.
The general steps needed to solve an optimization problem are:
\begin{enumerate}
	\item Write an equation that represents what you're trying to maximize/minimize. This is called your primary equation.
	\item Use additional information to eliminate excess variables.
	\item Find extreme values.
	\item Select the extreme values that fit the problem's constraints. Make sure your answer is what the problem is asking for.
\end{enumerate}

\begin{example}
	A farmer has 1000 linear feet of fence and wants to create a rectangular pasture.
	The pasture borders a river, which doesn't need a fence.
	What is the maximum area he can enclose?
\end{example}
\begin{answer}
	\begin{enumerate}
		\item Since the pasture is rectangular, we know the two lengths of fence perpendicular to the river will be the same length, which we'll call $x$.
			We'll say the remaining side parallel to the river has length $y$.
			So, the area enclosed is
			\begin{equation*}
				A(x,y) = xy.
			\end{equation*}
		\item There is no reason for the farmer not to use all 1000 linear feet of fence, so we'd expect the sum of the lengths of the 3 fenced sides to be equal to 1000 feet: $2x + y = 1000$, or $y = 1000 - 2x$.
			We can then substitute back into our primary equation to get it in terms of just $x$.
			\begin{equation*}
				A(y) = x(1000-2x) = 1000x - 2x^2.
			\end{equation*}
		\item We'll do a first derivative test to find extreme values.
		\begin{align*}
			A^\prime(y) &= 1000 - 4x \\
			A^\prime(y) &= 0 \text{ at } x = 250 \\
			A^\prime(200) &= 200 > 0 \\
			A^\prime(300) &= -200 < 0.
		\end{align*}
		\item So, $x=250 \implies y = 500$ is a local maximum.
			This corresponds to an area of $A = 250\cdot 500 = 125000\text{ft}^2$.
	\end{enumerate}
\end{answer}

\begin{example}
	What is the maximum area of a rectangle that has two vertices on the $x-axis$ and two vertices on the portion of the graph $y=8-x^2$ where $y > 0$?
\end{example}
\begin{answer}
	\begin{enumerate}
		\item We can define any such trapeziod (which include all such rectangles) by the $x$ values of the vertices on the $x$ axis.
			We'll call these two values $x_1$ and $x_2$.
			We'll say arbitrarily that $x_1 < x_2$.
			So, the area enclosed is
			\begin{equation*}
				A(x_1, x_2) = (x_2 - x_1)\frac{f(x_1) + f(x_2)}{2}.
			\end{equation*}
		\item However, the problem specifically restricts us to a rectangle, not a trapezoid.
			So, the heights of each side must be equal.
			\begin{align*}
				f(x_1) &= f(x_2) \\
				8 - x_1^2 &= 8 - x_2^2 \\
				x_1^2 &= x_2^2 \\
				\pm x_1 &\ \pm x_2 \\
				-x_1 &= x_2 \text{ because $x_1 \neq x_2$}.
			\end{align*}
			Substituting back into our primary equation,
			\begin{equation*}
				A(x_2) = (x_2 - (-x_2))\frac{f(x_2) + f(-x_2)}{2} = 2x_2f(x_2) = 16x_2 - 2x_2^3.
			\end{equation*}
		\item We'll do a second derivative test for fin extreme values.
			\begin{align*}
				A^\prime(x_2) &= 16 - 6x_2^2 \\
				A^\prime(x_2) &= 0 \text{ at } x = \pm\frac{4}{\sqrt{6}} \\
				A^{\prime\prime}(x_2) &= -12x_2 \\
				A^{\prime\prime}\left(-\frac{4}{\sqrt{6}}\right) &= 8\sqrt{6} > 0 \\
				A^{\prime\prime}\left(\frac{4}{\sqrt{6}}\right) &= -8\sqrt{6} < 0.
			\end{align*}
		\item So, $x_2 = 4/\sqrt{6}$ is a local maximum, and $x_2 = -4/\sqrt{6}$ is a local minimum.
			The problem asks for a maximum, so we select $x_2 = 4/\sqrt{6}$.
			The problem asks for a maximum area, so
			\begin{equation*}
				A_{max} = 16\left(\frac{4}{\sqrt{6}}\right) - 2\left(\frac{4}{\sqrt{6}}\right)^3 = \frac{128}{3\sqrt{6}} \approx 17.419.
			\end{equation*}
	\end{enumerate}
\end{answer}


\section{Exercises in Continuous Random Variables}\label{S:xsContinuousRVs}
\begin{ExerciseList}
\Exercise
Consider the probability density function
$$f(x)\;=\;\begin{cases} k &-4 \leq x\leq4\\0&\textrm{otherwise}\end{cases}\,.$$

\be
\item Find  the value of $k$.
\item Find the distribution function, $F$.
\item   Graph $f$ and  $F$.
\ee
\Answer
\be
\item Since $f(x)$ is a (continuous) probability density function which integrates to one,
$$\int_{-4}^4kdx\;=\;1\,.$$
That is,
\ba{k \,x\bigg]^4_{-4}&=\;1\\[3pt]k(4-(-4))&=\;1\\[3pt]8k&=\;1\\[3pt]k&=\;\frac{1}{8}}

\item First note that if $x <  -4$, then  $$F(x)\;=\;\int^x_{-\infty}0\,dv\;=\;0\,.$$
If $-4 \leq x \leq 4$, then
\begin{align*}F(x)&=\;\int^{-4}_{-\infty}0\,dv\;+\;\int^x_{-4} \frac{1}{8}\,dv\\[3pt]
&=\;0\;+\;\left[ \frac{1}{8}\, v \right]^x_{-4}\\[3pt]
&=\;  \frac{1}{8} (x +4)\end{align*}
If $x\geq 4$, then
\begin{align*}F(x)&=\int^{-4}_{-\infty}0\,dv\;+\;\int^{4}_{-4}
  \frac{1}{8} \,dv\;+\;\int^x_{4} 0 \,dv\\[3pt]
&=0\;+\;\left[\frac{1}{8} v \right]^4_{-4}\;+\;0\\[3pt]
&=\; 1\end{align*}
Hence  $$F(x)\;=\;\begin{cases}0&x <  -4\\ \frac{1}{8} (x+4) &-4 \leq
  x\leq 4\\1&x\geq 4\end{cases}$$

\item  The graphs of $f(x)$ and $F(x)$ for  random variable $X$ are as follows:
%TODO\centering   \makebox{\includegraphics[width=6.5in]{figures/fandFforConstantDensity.png}}

\ee

\Exercise
Assume that a new light bulb will burn out at time $t$ hours according to the probability density function given by
$$
f(t)\;=\;
\begin{cases}
\lambda e^{-\lambda t} & \text{ if } t >0 \enspace,\\
0 & \text{ otherwise} \enspace .
\end{cases}
$$
In this context, $\lambda$ is often called the failure rate of the bulb.

\be
\item[(a)]Assume that $\lambda=0.01$, and find the probability that the
  bulb will not burn out before $\tau$ hours. This $\tau$-specific probability is often
  called the reliability of the bulb.

Hint: Use the distribution function for an $\exponential(\lambda)$ random variable (recall, $F(\tau;\lambda)=\int_{-\infty}^{\tau} f(t) dt$)!

\item[(b)]For what  value of $\tau$ is the reliability of the bulb exactly $\frac{1}{2}$?
\ee
\Answer
\be
\item Since the distribution function is $F(t; \lambda) \,=\,
  1 - \exp( - \lambda t)$,
 \[\P(t>\tau)\;=\;1-\P(t<\tau)\;=\; 1 - F(\tau;\lambda=0.01) \;=\;1-(1-e^{-0.01\tau})\;=\;e^{-0.01\tau}\,.\]

\item Set  \[\P(t>\tau)\;=\;e^{-0.01\tau}\;=\;\frac{1}{2}\]
and solve for $\tau$ to get  then $\tau\,=\,-100\times \log(0.5)\,=\,69.3\quad (\text{3 sig. fig.})$\,.
\ee

\Exercise
Let the random variable $X$ be the time  after which certain ball bearings wear out, with density
$$f(x)\;=\;\begin{cases} ke^{-x}&0\leq x\leq 2\\0&\textrm{otherwise}\end{cases}\enspace.$$
Note: $X$ is measured in  years.
\be
\item Find $k$.
\item Find the probability that a bearing will last at least 1 year.
\ee
\Answer
\be
\item

\ba{\int^2_0 k\,e^{-x}\,dx &=\;1\\[3pt]
\left[ -k\,e^{-x}\right]^2_0&=\;1\\[3pt]
k\,(-e^{-2}+1)&=\;1\\[3pt]
k&=\;\frac{1}{1-e^{-2}} \quad (\approx 1.1565)
}

\item
\ba{\P(X\geq1)&=1\,-\,\P(X<1)\\[3pt]
&=\;1\;-\;\int^1_0 k e^{-x}\,dx\\[3pt]
&=\;1\;+\;k\left(e^{-x}\right]^1_0\\[3pt]
&=\;1\;+\;\frac{e^{-1}-1}{1-e^{-2}}\\[3pt]
&\approx\;0.2689}

\ee

\end{ExerciseList}


\chapter{Arithmetical Hierarchy}

\Cref{theorem:enumerable-via-computable-funcitons} proves that any decidable set
is the image of some total computable function from $\N$; identifying sets and
predicates we give the following reformulation.
\begin{theorem}
  A predicate$A(x)$ of natural numbers is enumerable iff there is a decidable
  predicate $B(x, y)$ such that the following formula is always true:
  \[
    A(x) \iff \exists y \  B(x, y).
  \]
  (Here and in the sequel, we are going to write $\exists x \  P(x)$ denotes the
  statement ``there is an $x$ from the domain of $P$ such that $P(x)$ is true.)
\end{theorem}

A natural questions can be asked: ``What can be said about other combinations of
quantifiers?''.

It is easy to see that if $A(x)$ is a predicate such that
\[
  A(x) \iff \exists y \  \exists z \  C(x, y, z),
\]
where $C(x, y, z)$ is decidable, then $A(x)$ is enumerable.
Indeed, let $B(x, \pair{y}{z}) = C(x, y, z)$; then $B(x, w)$ is decidable and
\[
  A(x) \iff \exists y \  \exists z \  B(x, \pair{y}{z}) \iff
    \exists w \  B(x, w).
\]
If $A(x)$ is a predicate such that
\[
  A(x) \iff \forall y \  B(x, y),
\]
where $B(x, y)$ is decidable, then the negation of $A(x)$ is enumerable (we call
such sets and predicates \emph{coenumerable}). Indeed,
\[
  \lnot A(x) \iff \exists y \  \lnot B(x, y),
\]
and $\lnot B(x, y)$ is decidable since the compliment of a decidable set is
decidable.

This question gives rise to the following definition.
\begin{definition}
  We say that a predicate $A$ on $\N$ belongs to the class $\Sigma_n$ iff there
  is a deciable predicate $B$ on $\N^{n + 1}$ such that
  \[
    A(x) \iff
    \exists y_1 \  \forall y_2 \  \exists y_3 \  \dots B(x, y_1, \dots, y_n).
  \]
  Similarly, a predicate $A$ on $\N$ belongs to the class $\Sigma_n$ iff there
  is a deciable predicate $B$ on $\N^{n + 1}$ such that
  \[
    A(x) \iff
    \forall y_1 \  \exists y_2 \  \forall y_3 \  \dots B(x, y_1, \dots, y_n).
  \]
\end{definition}

Previous observations can be generalized as follows.
\begin{theorem}
\label{theorem:arythmetic-hierarchy-basics}
  \begin{enumerate}
    \item The set $\Sigma_1$ consists of enumerable sets.
    \item If a predicate belongs to $\Sigma_n$, then its negation belongs to
      $\Pi_n$.
    \item The set $\Sigma_n$ does not change if we allow groups of quantifiers
      of the same type instead of single quantifiers.
  \end{enumerate}
\end{theorem}

Like enumerable and decidable sets, $\Sigma_n$ and $\Pi_n$ sets have several
good properties.
\begin{theorem}
  Union and intersection of two $\Sigma_n$ ($\Pi_n$) sets are also $\Sigma_n$
  ($\Pi_n$) sets.
\end{theorem}
\begin{proof}
  We prove the statement for the union of two $\Sigma_n$ sets, all other cases
  have similar proofs. Let $A_1, A_2 \in \Sigma_n$. By the definition of
  $\Sigma_n$, there are decidable $B_1$ and $B_2$ such that
  \begin{gather*}
    A_1(x) \iff \exists y_1 \  \forall y_2 \  \dots B_1(x, y_1, \dots, y_n) \\
    \text{and} \\
    A_2(x) \iff \exists z_1 \  \forall z_2 \  \dots B_2(x, z_1, \dots, z_n).
  \end{gather*}
  Therefore
  \begin{multline*}
      A_1(x) \land A_2(x) \iff \\
      \exists y_1, z_1 \  \forall y_2, z_2 \  \dots
        B_1(x, y_1, \dots, y_n) \land B_2(x, z_1, \dots, z_n);
  \end{multline*}
  which implies that $A_1 \land A_2$ is a $\Sigma_n$ predicate
  (\Cref{theorem:arythmetic-hierarchy-basics}).
\end{proof}

\begin{exercise}
  Prove that $\Sigma_n \cup \Pi_n \subseteq \Sigma_{n + 1} \cup \Pi_{n + 1}$ for
  any $n \in \N_0$.
\end{exercise}

We can also prove an extension of
\Cref{theorem:m-reduction-complexity-preservation}.
\begin{theorem}
  Let $n \in \N$, and let $A, B \subseteq \N$ such that $A \le_m B$.
  If $B \in \Sigma_n$, then $A \in \Sigma_n$.
\end{theorem}
\begin{proof}
  Let $f$ be the reduction from $A$ to $B$. Let $C$ be a decidable predicate
  such that
  \[
    B(x) \iff \exists y_1 \  \forall y_2 \  \dots C(x, y_1, \dots, y_n).
  \]
  Note that, by the definition of the reduction,
  \[
    A(x) \iff \exists y_1 \  \forall y_2 \  \dots C(f(x), y_1, \dots, y_n);
  \]
  hence, $A(x) \in \Sigma_n$ since $C(f(x), y_1, \dots, y_n)$ is decidable.
\end{proof}

\begin{exercise}
  Prove that if a set $A \in \Sigma_n$, then $A \times A \in \Sigma_n$.
\end{exercise}

We defined infinitely many classes of sets; however, we have not shown that all
these sets are different (except $\Sigma_0$ and $\Pi_0$ which are equal to the
class of all decidable sets). Like in the case of enumebrable sets, to prove
this we need to use universal sets.
\begin{theorem}
  For any $n \in \N$, there is a set $U \in \Sigma_n$ universal for $\Sigma_n$
  subsets of $\N$.
\end{theorem}
\begin{proof}
  We prove the statement using induction by $n$. The base case for $n = 1$ is
  true by
  \Cref{theorem:arythmetic-hierarchy-basics,theorem:universal-set-enumerable}.

  Let us prove the induction step. Assume $V \in \Sigma_n$ is a universal set
  for $\Sigma_n$ subsets of $\N$. First we show that $(\N \setminus V) \in \Pi_n$
  is a universal set for $\Pi_n$ subsets of $\N$. Let $A \in \Pi_n$. Then there
  is $k \in \N$ such that $\N \setminus A = V_k$ since $(\N \setminus A) \in
  \Sigma_n$. Therefore, $A = \N \setminus V_k$ which implies that
  $\N \setminus V$ is universal for $\Pi_n$ subsets of $\N$.

  Note that there is $W' \in \Pi_n$ that is universal for $\Pi_n$ subsets
  of $\N^2$. Indeeed, let $W' = \set[(n, \pair{x}{y}) \in W]{(n, x, y) \in \N^3}
  \in \Sigma_n$.
  Let $A' \in \Pi_n$ and $B'$ be decidable set such that
  \[
    A'(x, y) \iff \exists z_1 \  \forall z_2 \  \dots B'(x, y, z_1, \dots, z_n).
  \]
  Consider $A \subseteq \N$ such that
  \[
    A(\pair{x}{y}) \iff \exists z_1 \  \forall z_2 \  \dots
      B'(x, y, z_1, \dots, z_n);
  \]
  it is clear that $A \in \Pi_n$. Hence, there is $k \in \N$ such that
  $W_k = A$, which implies that $W'_n = A'$. Therefore $W'$ is universal for
  $\Pi_n$ subsets of $\N^2$.

  Let us consider the set
  $U = \set[\exists y \  (k, x, y) \in W']{(k, x) \in \N^2}$. We claim that it is
  universal for $\Sigma_{n + 1}$ subsets of $\N$.
  Let $A \in \Sigma_{n + 1}$. Then there is $B \in \Pi_n$ such that
  \[
    A(x) \iff \exists y \  B(x, y).
  \]
  In addittion, there is $k \in n$ such that $W'_k = B$. Therefore,
  \[
    A(x) \iff \exists y \  W'(k, x, y) \iff U(k, x).
  \]
  Hence, $U$ is universal for $\Sigma_{n + 1}$ subsets of $\N$.
\end{proof}

We proved that the compliment of an enumerable universal set for the class of
enumerable sets is not enumerable. Similar result can be shown for $\Sigma_n$.
\begin{theorem}
  Let $n \in \N$ and let $W \subseteq \N^2$ be universal for $\Sigma_n$ subsets
  of $\N$. Then $W \not\in \Pi_n$.
\end{theorem}
\begin{proof}
  Assume that $W \in \Pi_n$, this implies that that the set
  $D = \set[(n, n) \not\in W]{n \in \N}$ belongs to $\Sigma_n$.
  Note that there is $k \in \N$ such that $W_k = D$ since $W$ is universal for
  $\Sigma_n$ subsets if $\N$. However, $W_k(k)$ is true only if $W_k(k)$ is
  false, which is a contradiction. Therefore, $W \not\in \Pi_n$.
\end{proof}

\begin{corollary}
  For any $n \in \N$, $\Pi_n \not\subseteq \Sigma_{n + 1}$ and
  $\Sigma_n \not\subseteq \Pi_{n + 1}$.
\end{corollary}

\begin{chapterendexercises}
  \exercise Prove that if $A, B \in \Sigma_n$, then
    $A \setminus B \in \Sigma_{n + 1} \cap \Pi_{n + 1}$.
\end{chapterendexercises}


\section{08/31}
\subsection{Verifying Matrix Multiplication}
\begin{problem}{Verifying Matrix Multiplication}{}
    Let $A$, $B$, and $C$ be $n \times n$ matrices. Verify whether
    \[A \times B = C\]
\end{problem}
The best known deterministic algorithm for this requires $\bigO{n^{2.37286}}$
operations, improving over the previous best of $\bigO{n^{2.37287}}$
\cite{alman2020refined}. However, we can do substantially better using
randomization.

A principle in randomized algorithms is to take advantage of an ``abundance of
witness,'' that is, when some property that is helpful towards your algorithm
occurs often enough, to try and make use of that property.

In this case, the witness is that, if $AB = C$, then $AB\vec{r} = C\vec{r}$ for
every vector $\vec{r}$. Thus, a single vector will prove if $AB \neq C$, and
each vector for which $AB\vec{r} = C\vec{r}$ increases our confidence that $AB =
C$. This admits the following pseudocode
\begin{algorithm}
    \caption{Verify Matrix Multiplication}
    \begin{algorithmic}[1]
        \Function{Rand-Matrix-Mult-Verify}{A, B, C}
            \State $\vec{r} \gets \Call{Sample}{\set{0,1}^n}$
            \If{$A(B\vec{r}) \neq C\vec{r}$}
                \State \Return \False
            \Else
                \State \Return \True
            \EndIf
        \EndFunction
    \end{algorithmic}
\end{algorithm}
Notice that this algorithm only requires $\bigO{n^2}$ operations. As stated
previously, it does not admit false negatives --- if $AB = C$, it will never
output \False, because we must have $AB\vec{r} = C\vec{r}$ for all $\vec{r}$.

Let us determine the probability that this algorithm outputs \True when $AB \neq
C$. Write $D = AB - C \neq 0_{n,n}$, and observe that $D$ must have at least one
non-zero entry. Without loss of generality, say it is $d_{1,1}$. Now, if
$AB\vec{r} = C\vec{r}$, then $D\vec{r} = \vec{0}$. In particular, writing
$\vec{r} = (r_1, r_2, \dots, r_n)$, we must have
\begin{align*}
    d_{1,1}r_1 + d_{1,2}r_2 + \dots + d_{1,n}r_n = 0\\
    r_1 = \frac{-1}{d_{1,1}}\sum_{i=2}^n d_{1,i}r_i
\end{align*}

To determine the probability that $r_1$ is equal to this value, we apply the
\emph{Principle of Deferred Decisions}
\begin{definition}{Principle of Deferred Decisions}{}
    The \emph{Principle of Deferred Decisions} states that, in the analysis of a
    randomized algorithm, we can \emph{defer} random choices until they are
    revealed to the algorithm.
\end{definition}

In this specific case, we can assume $r_2$, $r_3$, \dots, $r_n$ have already
been assigned their random values. Clearly, then, the probability that $r_1$ is
equal to $\frac{-1}{d_{1,1}}\sum_{i=2}^n d_{1,i}r_i$ is \emph{at most}
$\sfrac{1}{2}$.

This gives us an upper bound on the probability of a false positive. In
particular, after repeating the algorithm $k$ times, the probability of a false
positive is at most $\sfrac{1}{2^k}$. Setting $k = \log{n}$ yields the desired
upper bound of $\sfrac{1}{n}$. In other words, the following pseudocode verifies
matrix multiplication with high probability

\begin{algorithm}
    \caption{Verify Matrix Multiplication with High probability}
    \begin{algorithmic}[1]
        \Function{Rand-Matrix-Mult-Verify-WHP}{A, B, C}
            \ForRange{$i$}{1}{$\log{n}$}
                \State $\texttt{result} \gets \Call{Rand-Matrix-Mult-Verify}{A, B, C}$
                \If{$\texttt{result}$ is \False}
                    \State \Return \False
                \EndIf
            \EndForRange
            \State \Return \True
        \EndFunction
    \end{algorithmic}
\end{algorithm}

\subsection{Verifying Equality of Binary Strings}
\begin{problem}{Verifying Equality of Binary Strings}{}
    Given strings
    \begin{align*}
        \texttt{X} &= \texttt{x}_0 \texttt{x}_1 \dots \texttt{x}_{n-1}\\
        \texttt{Y} &= \texttt{y}_0 \texttt{y}_1 \dots \texttt{y}_{n-1}
    \end{align*}
    Determine if $X = Y$, i.e., if there exists an $i$ such that $\texttt{x}_i
    \neq \texttt{y}_i$.
\end{problem}

Here, we are interested in \emph{communication complexity}. Specifically suppose
Alice has string $\texttt{X}$ and Bob has string $\texttt{Y}$, and they wish to
determine if their strings are equal by transmitting a message of $k$ bits.

Clearly, there is a naive deterministic algorithm that requires $\bigO{n}$ time
by simplying comparing all $n$ bits. In fact, \emph{every} deterministic
algorithm requires $\bigOm{n}$ bits. However, we can construct a randomized
algorithm, via a technique called \emph{fingerprinting}, that requires
$\bigO{\log{n}}$ bits.

For a prime $p \in \set{1, 2, \dots, k}$, let us define $F_p(x) = x \bmod p$.
Additionally, we will identify the values $\texttt{X}$ and $\texttt{Y}$ by the
numerical value of their binary strings, that is,
\begin{align*}
    \texttt{X} &= \sum_{i=0}^{n-1}\texttt{x}_i 2^i\\
    \texttt{Y} &= \sum_{i=0}^{n-1}\texttt{y}_i 2^i
\end{align*}
If $F_p(\texttt{X}) \neq F_p(\texttt{Y})$, then we output \False, otherwise we
output \True. Again, this admits no false negatives, since if $\texttt{X} =
\texttt{Y}$, $F_p(\texttt{X}) = F_p(\texttt{Y})$ for any function $F_p$.

Observe that, if $F_p(\texttt{X}) = F_p(\texttt{Y})$, then
\[\sum_{i=0}^{n-1}\left(\texttt{x}_i - \texttt{y}_i\right) 2^i \bmod p = 0\]
Additionally, notice that
\[\sum_{i=0}^{n-1}\left(\texttt{x}_i - \texttt{y}_i\right) 2^i \leq 2^n\]
In particular, there are at most $n$ prime factors of
$\sum_{i=0}^{n-1}\left(\texttt{x}_i - \texttt{y}_i\right) 2^i$. Letting $\pi(k)$
denote the number of primes less than or equal to $k$, we have
\begin{align*}\prob{F_p(\texttt{X}) \neq F_p(\texttt{Y}) \given \texttt{X} \neq \texttt{Y}}
    &\leq \frac{n}{\pi(k)}\\
    &\approx \frac{n\ln{k}}{k}
\end{align*}
Taking $k = n^2\ln{n}$ yields
\[\prob{F_p(\texttt{X}) \neq F_p(\texttt{Y}) \given \texttt{X} \neq \texttt{Y}} \leq \frac{1}{n}\]

\subsection{Binary Consensus}
\begin{problem}{Binary Consensus}{}
    Given $n$ parties, of which at most $f$ are faulty (and can crash at any
    time), and who each hold a 0 or 1, Consensus is reached when:
    \begin{enumerate}
        \item Agreement --- all non-faulty parties agree to the same bit
        \item Validity --- if all parties start with a value, all parties must
              agree on that value
        \item Termination --- all parties terminate in finite time
    \end{enumerate}
\end{problem}

Let us assume that communication is asynchronous and that all nodes can
communicate with all other nodes (i.e., the network is a complete graph).
%TODO drawing

With \emph{no faults}, it is sufficient for each node to simply broadcast their
value and take the majority, breaking ties with 1. However, with even a single
fault, this problem \emph{cannot} be solved deterministically. Additionally, no
randomized algorithm can solve this problem when there are at least
$\sfrac{n}{2}$ faults.

%TODO prove the above for randomized.