\subsubsection{Na\"{i}ve Methods, Moving Averages, and Exponential Smoothing}
\label{ets}

Simple forecasting methods are often employed as a benchmark for more
    sophisticated ones.
The so-called na\"{i}ve and seasonal na\"{i}ve methods forecast the next time
    step in a time series, $y_{T+1}$, with the last observation, $y_T$,
    and, if a seasonal pattern is present, with the observation $k$ steps
    before, $y_{T+1-k}$.
As variants, both methods can be generalized to include drift terms in the
    presence of a trend or changing seasonal amplitude.

If a time series exhibits no trend, a simple moving average (SMA) is a
    generalization of the na\"{i}ve method that is more robust to outliers.
It is defined as follows: $\hat{y}_{T+1} = \frac{1}{h} \sum_{i=T-h}^{T} y_i$
    where $h$ is the horizon over which the average is calculated.
If a time series exhibits a seasonal pattern, setting $h$ to a multiple of the
    periodicity $k$ suffices that the forecast is unbiased.

Starting in the 1950s, another popular family of forecasting methods,
    so-called exponential smoothing methods, was introduced by
    \cite{brown1959}, \cite{holt1957}, and \cite{winters1960}.
The idea is that forecasts $\hat{y}_{T+1}$ are a weighted average of past
    observations where the weights decay over time; in the case of the simple
    exponential smoothing (SES) method we obtain:
$
\hat{y}_{T+1} = \alpha y_T + \alpha (1 - \alpha) y_{T-1}
                + \alpha (1 - \alpha)^2 y_{T-2}
                + \dots + \alpha (1 - \alpha)^{T-1} y_{1}
$
where $\alpha$ (with $0 \le \alpha \le 1$) is a smoothing parameter.

Exponential smoothing methods are often expressed in an alternative component
    form that consists of a forecast equation and one or more smoothing
    equations for unobservable components.
Below, we present a generalization of SES, the so-called Holt-Winters'
    seasonal method, in an additive formulation.
$\ell_t$, $b_t$, and $s_t$ represent the unobservable level, trend, and
    seasonal components inherent in $y_t$, and $\beta$ and $\gamma$ complement
    $\alpha$ as smoothing parameters:
\begin{align*}
\hat{y}_{t+1} & = \ell_t + b_t + s_{t+1-k} \\
\ell_t        & = \alpha(y_t - s_{t-k}) + (1 - \alpha)(\ell_{t-1} + b_{t-1}) \\
b_t           & = \beta (\ell_{t} - \ell_{t-1}) + (1 - \beta) b_{t-1} \\
s_t           & = \gamma (y_t - \ell_{t-1} - b_{t-1}) + (1-\gamma)s_{t-k}
\end{align*}
With $b_t$, $s_t$, $\beta$, and $\gamma$ removed, this formulation reduces to
    SES.
Distinct variations exist: Besides the three components, \cite{gardner1985}
    add dampening for the trend, \cite{pegels1969} provides multiplicative
    formulations, and \cite{taylor2003} adds dampening to the latter.
The accuracy measure commonly employed is the sum of squared errors between
    the observations and their forecasts.

Originally introduced by \cite{assimakopoulos2000}, \cite{hyndman2003} show
    how the Theta method can be regarded as an equivalent to SES with a drift
    term.
We mention this method here only because \cite{bell2018} emphasize that it
    performs well at Uber.
However, in our empirical study, we find that this is not true in general.

\cite{hyndman2002} introduce statistical processes, so-called innovations
    state-space models, to generalize the methods in this sub-section.
They call this family of models ETS as they capture error, trend, and seasonal
    terms.
Linear and additive ETS models have a structure like so:
\begin{align*}
y_t       & = \vec{w} \cdot \vec{x}_{t-1} + \epsilon_t \\
\vec{x_t} & = \mat{F} \vec{x}_{t-1} + \vec{g} \epsilon_t
\end{align*}
$y_t$ denote the observations as before while $\vec{x}_t$ is a state vector of
    unobserved components.
$\epsilon_t$ is a white noise series and the matrix $\mat{F}$ and the vectors
    $\vec{g}$ and $\vec{w}$ contain a model's coefficients.
Just as the models in the next sub-section, ETS models are commonly fitted
    with maximum likelihood and evaluated using information theoretical
    criteria against historical data.
We refer to \cite{hyndman2008b} for a thorough summary.


% Chapter 2
\chapter{Traditional \gls{RGBD} Cameras Calibration} % Main chapter title
\label{chapterTraditionalCalibration} % For referencing the chapter elsewhere, use \ref{sens_introduction}
%\indent
A pinhole-camera model can be used to describe an image sensor's field of view. When applying the pinhole camera model in world space, it explains the mapping relationship from world space to camera space, and then to image space. A $3\times4$ pinhole-camera matrix expresses the mappings mathematically. It consists of an intrinsic matrix mapping from the \gls{3D} camera space to 2D image space, and an extrinsic matrix map from \gls{3D} world space to \gls{3D} camera space. Traditionally, lens distortions correction is after, and separated from the pinhole-camera model calibration. In this chapter, we will introduce the camera calibration methods based on the pinhole-camera model in detail, and then discuss how to remove the lens distortions in traditional methods.
%
\begin{figure}[!b]
\centering
\includegraphics[width=0.45\textwidth]{PinholeCameraFigure}
\caption{The Pinhole Camera Inspection}
\label{PinholeCameraFigure}
\end{figure}%
%%
\section{Pinhole Camera}
\label{sectionPinholeCamera}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%
%%%%%%%%%%      1. Intrinsic, introduce camera model from \gls{3D} to 2D:         %%%%%%%%%%%
%%%%%%%%%%                 (X^c, Y^c, Z^c) --> \(r, \, c\)                                          %%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\indent
A pinhole camera is a simple optical imaging device in the shape of a closed box or chamber. A pinhole camera is completely dark on all the other sides of the box including the side where the pin-hole is created. Figure~\ref{PinholeCameraFigure} shows an inspection of a pinhole camera. In its front is a pin-hole that help create an image of the outside space on the back side of the box. When the shutter is opened, the light shines through the pin-hole and imprint an image onto a sensor (or photographic paper, or film) placed at the back side of the box. In order to analyze parameters like focal distance, field of view, etc., pinhole camera has its own three dimensional space (noted as \(\gls{cameraX}\), \(\gls{cameraY}\), and \(\gls{cameraZ}\)). Note that, according to Cartesian Coordinates \enquote{right hand} principle, the camera is looking down the negative of \(\gls{cameraZ}\)-axis, given \(\gls{cameraX}\)\(\gls{cameraY}\) directions as shown in the figure. Its focal length of the pinhole camera is the distance on the \(Z^c\)-axis, between the pinhole at the front of the camera and the paper or film at the back of the camera.
\\\indent
\begin{figure}[!t]
\centering
\includegraphics[width=0.5\textwidth]{PinHoleVirtualFocalPlane}
\caption{Virtual Focal Plane of a Pinhole Camera}
\label{PinHoleVirtualFocalPlane}
\end{figure}%
%
%
Pinhole cameras are characterized by the fact that they do not have a lens. It rely on the fact that light travels in straight lines, which is a principle called the rectilinear theory of light. This makes the image appear upside down in the camera, as shown in Fig.~\ref{PinHoleVirtualFocalPlane}. Tracing the corners of the camera sensor through the pin hole, those dark green lines show the limits of the field of view in \gls{3D} coordinate space. The back side plane of the pinhole camera, which is behind the origin at a positive \(\gls{cameraZ}\)-axis and also where our sensor sits, is also called the focal plane. It is not intuitive, nor convenient for mathematical analysis that the images on the focal plane are always upside down. So a virtual focal plane is defined in front of the pinhole on the negative \(\gls{cameraZ}\)-axis, which is equal distant from the focal point (pin hole) as the actual focal plane is behind. Notice that the limits of the field of view intersect with the virtual focal plane at the four corners of the up-right image just as they disseminate from the four corners of the sensor at the real focal plane.

\begin{figure}[t]
\centering
\includegraphics[width=0.65\textwidth]{CommonPinholeCameraModel}
\caption{Common Pinhole Camera Model for a $240\times320$ Pixels Camera Sensor}
\label{CommonPinholeCameraModel}
\end{figure}%
%

With the virtual focal plane, the camera body with the real focal plane could be removed. And the rest parts in front of the the camera body, the focal point and the virtual focal plane together, form the most common pinhole camera model. In order to employ this model to analyze arbitrary \gls{3D} object points inside the camera's field of view in 2D image space, the prior step is to define the relationship between points in \gls{3D} camera space and the 2D image space (\(\gls{imageRow}\) and \(\gls{imageColumn}\)). As shown in Fig.~\ref{CommonPinholeCameraModel}, The focal point is right at the origin of the camera \gls{3D} space coordinates, from where to the sensor is the vertical distance of \(f\), the focal distance. The 2D image coordinates are in dark green, and its origin is sitting at the up-left corner of the sensor. Only the a virtual sensor (in color red) is visible on the virtual focal plane, whose size in 2D image space is noted as 240 by 320 (using the size of PrimeSense camera). As long as both of the camera \gls{3D} space coordinates and image 2D space coordinates are defined, the next job is to build a mapping between them.
%
\begin{figure}[t]
\centering
\includegraphics[width=0.82\textwidth]{RelationshipCameraToImage}
\caption{Mapping from Camera Space to Image Space}
\label{RelationshipCameraToImage}
\end{figure}%
%

Select a random object point \(P^C\) in the camera space located at camera \gls{3D} coordinates (\(\gls{cameraPointX0}\), \(\gls{cameraPointY0}\), \(\gls{cameraPointZ0}\)). A line passing both of the point \(P^C\) and the \emph{Focal Point} intersects with the virtual focal plane at \(P^I\), with its image 2D coordinates (\(r, \, c\)). To determine the mapping function, we can start a the proportional relationship. As shown in Fig.~\ref{RelationshipCameraToImage}, the center point in the image coordinates, which is usually called \emph{\gls{principlePoint}}, could be determined by column of half-width (\(c_h\)) and row of half-height (\(r_h\)). Concretely, the \gls{principlePoint} (\(r_h\), \(c_h\)) is either (119.5, 159.5) if range is ([0:239], [0:319]), or (120, 320) if range is ([1:240], [1:320]). So, we could get the relative row and column distance of  \(r_r\) and \(c_r\) by:
%
\begin{equation}
\begin{aligned}
r_r &= r - r_h%
\\%
c_r &= c - c_h \ \ \ .%
\end{aligned}
\label{relativeCRforProportional}
\end{equation}%
%
Based on by triangulation, it is straight forward to tell the proportional relationship between \(f\)/\(\gls{cameraPointZ0}\) and \(c_r\)/\(\gls{cameraPointX0}\), \(r_r\)/\(\gls{cameraPointY0}\). Thus we get
%
\begin{equation}
\left[ \begin{array}{c} c_r \\ r_r \end{array} \right] %
= f %
\left[ \begin{array}{c} \gls{cameraPointX0}/\gls{cameraPointZ0} \\ \gls{cameraPointY0}/\gls{cameraPointZ0} \end{array} \right]  .%
\label{twoDRelativeFromCamToIm}
\end{equation}
\noindent
And by changing the relative distance \(r_r / c_r\) back to the 2D image coordinates \(r, \, c\), then eqn.~(\ref{twoDRelativeFromCamToIm}) will be written as
%
\begin{equation}
\left[ \begin{array}{c} c \\ r \end{array} \right] %
= f %
\left[ \begin{array}{c} \gls{cameraPointX0}/\gls{cameraPointZ0} \\ \gls{cameraPointY0}/\gls{cameraPointZ0} \end{array} \right]%
+
\left[ \begin{array}{c}  c_h \\  r_h \end{array} \right] .%
\label{linearRelationFromCamToIm}
\end{equation}
\noindent
If written in homogeneous coordinates, we will get eqn.~(\ref{HomoProportionalRelationFromCamToIm}):
\begin{equation}
%
\gls{cameraPointZ0} \left[ \begin{array}{c} c \\ r \\ 1 \end{array} \right] %
= %
\left[ \begin{array}{c} fx^c \\ fy^c \\ \gls{cameraPointZ0} \end{array} \right]%
+
\left[ \begin{array}{c}  \gls{cameraPointZ0}c_h \\  \gls{cameraPointZ0}r_h \\ 0\end{array} \right] %
=  \begin{bmatrix} f & 0 &  c_h  \\ 0 & f & r_h \\ 0 & 0 & 1 \end{bmatrix}%
\left[ \begin{array}{c} \gls{cameraPointX0} \\ \gls{cameraPointY0} \\ \gls{cameraPointZ0} \end{array} \right] .%
\label{HomoProportionalRelationFromCamToIm}
\end{equation}%
%
\\\indent
Till Now, we haven't consider the units translation between the camera \gls{3D} space the image 2D space. The random object point \(P^C\)'s mapping point \(P^I\) (\(r, \, c\)) on the image space is expressed in millimeters (or inches). Since it is necessary to express the image space coordinates (\(r, \, c\)) in pixels, we need to find out the resolution of the sensor in pixels/millimeter. Considering that, the pixels are not necessarily be square-shaped, we assume they are rectangle-shaped with resolution  $\alpha_c$ and \(\alpha_r\) pixels/millimeter in the \(\gls{imageColumn}\) and \(\gls{imageRow}\) direction respectively. Therefore, to express \(P^I\) in pixels, its \(c\) and \(r\) coordinates should be multiplied by \(\alpha_c\) and \(\alpha_r\) respectively, to get:
%
\begin{equation}
\left[ \begin{array}{c} \gls{cameraPointZ0} c \\ \gls{cameraPointZ0} r \\ \gls{cameraPointZ0}  \end{array} \right] %
= %
\left[ \begin{array}{c} f\alpha_c x^c \\ f \alpha_r y^c \\ \gls{cameraPointZ0} \end{array} \right]%
+
\left[ \begin{array}{c}  \gls{cameraPointZ0} \alpha_c c_h \\ \gls{cameraPointZ0} \alpha_r r_h \\ 0 \end{array} \right] %
=  \begin{bmatrix} \alpha_c f & 0 &  \alpha_c c_h  \\ 0 & \alpha_r f & \alpha_r r_h \\ 0 & 0 & 1 \end{bmatrix}%
\left[ \begin{array}{c} \gls{cameraPointX0} \\ \gls{cameraPointY0} \\ \gls{cameraPointZ0} \end{array} \right]%
= \gls{intrinsicMatrixK} \textbf{\textit{P}}^C  .
\label{HomoProportionalFromCamToImInPixels}
\end{equation}%

\noindent
Note that \(\gls{intrinsicMatrixK}\) only depends on the intrinsic camera parameters like its focal length, resolution in pixels, and sensor's width and height. Thus, the mapping matrix \(\gls{intrinsicMatrixK}\) is also called a camera's intrinsic matrix. Considering that the pixels might be parallelogram-shaped instead of rigid rectangle-shaped (when the image coordinate axis \(\gls{imageRow}\) and \(\gls{imageColumn}\) are not orthogonal to each other), usually \(\gls{intrinsicMatrixK}\) has a skew parameter \(s\), given by

\begin{equation}
\gls{intrinsicMatrixK}%
=  \begin{bmatrix}
f_c & s & t_c \\
 0 & f_r & t_r \\
 0 & 0 & 1 \end{bmatrix} ,%
\label{intrinsicKmatrix}
\end{equation}%
\noindent
where \(f_c = \alpha_c f\) and \(f_r = \alpha_r f\) are the focal length in pixels on the \(\gls{imageColumn}\) and \(\gls{imageRow}\) directions respectively,  \(t_c = \alpha_c r_h\) and \(t_r = \alpha_r r_h\) are the translation parameters that help move the origin of image coordinate to the \gls{principlePoint}.
\\\indent
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%
%%%%%%%%%%      2. Extrinsic, (X^w, Y^w, Z^w) --> (X^c, Y^c, Z^c)          %%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Now we have \(\gls{intrinsicMatrixK}\), which helps map between camera \gls{3D} space and image 2D space. But we are still not able to employ it yet. The camera \gls{3D} space is with respect to the camera sensor only. Neither can we directly tell the camera \gls{3D} coordinates of an object point, nor can we assign it. All we can do is to use the camera space as an intermediate space between the image coordinates and world coordinates, which we could assign by ourselves. %
%
\begin{figure}[!t]
\centering
\includegraphics[width=0.65\textwidth]{FromWorldToCameraSpace}
\caption{Pinhole Camera in World Space}
\label{FromWorldToCameraSpace}
\end{figure}%
%
Figure~\ref{FromWorldToCameraSpace} shows a pinhole camera observing an arbitrary object point P in the world  space. We assign the world coordinates so that the object point has world space coordinates \(P^W(x^w, \, y^w, \, z^w)\). Although the world space and camera space are two different spaces, we could easily transform between each other through rotation and translation, as long as both of the spaces are using rigid Cartesian Coordinates. With a standard rotation matrix \(\gls{exRotationR}\) and a translation matrix \(\gls{exTranslationT}\)
%
\begin{equation}
\gls{exRotationR}%
=  \begin{bmatrix}
r_{11} & r_{12} & r_{13} \\
r_{21} & r_{22} & r_{23} \\
r_{31} & r_{32} & r_{33}
 \end{bmatrix}%
, \, \,
\gls{exTranslationT}%
=  \begin{bmatrix}
t_{1} \\
t_{2} \\
t_{3}
 \end{bmatrix} ,%
\label{rotationTranslationMatrixRT}
\end{equation}%
%
we can get the transformation matrix \([\gls{exRotationR} ,\,\, \gls{exTranslationT}]\) from the world space to camera space:
%
\begin{equation}
\begin{bmatrix}
X^{C} \\
Y^{C} \\
Z^{C}
 \end{bmatrix}%
=  \gls{exRotationR} \begin{bmatrix}
X^{W} \\
Y^{W} \\
Z^{W}
 \end{bmatrix}%
 + \gls{exTranslationT}
=
\begin{bmatrix}
\gls{exRotationR} & \gls{exTranslationT} \\
\end{bmatrix}%
 \begin{bmatrix}
X^{W} \\
Y^{W} \\
Z^{W} \\
1
 \end{bmatrix}  .%
\label{mappingFromWorldToCameraSpace}
\end{equation}%
%
%%%% 2.5 Pinhole Camera Matrix (X^w, Y^w, Z^w) --> (X^c, Y^c, Z^c) --> \(r, \, c\)
%
\noindent
The parameters that help map from world space to camera space depend on how we assign the world coordinates. Since none of them are from the camera even though they are belongs to an important part of camera calibration, usually the matrix \([\gls{exRotationR} \,\, \gls{exTranslationT}]\) is called extrinsic camera matrix. With both of the extrinsic camera matrix (help map from world space to camera space) and the intrinsic camera matrix (help map from camera space to image space), we are now able to build the connection between the world space coordinates, which could be assigned by ourselves, and the image space \(\gls{imageRow}\) and \(\gls{imageColumn}\), which are the streams we retrieved from the camera.
\\\indent
To combine the intrinsic camera matrix and extrinsic camera matrix (combine eqn.~(\ref{HomoProportionalFromCamToImInPixels}) and eqn.(~\ref{mappingFromWorldToCameraSpace})), we get

\begin{equation}
\gls{cameraZ}\left[ \begin{array}{c} C \\ R \\ 1 \end{array} \right] %
=\gls{intrinsicMatrixK} \left[ \begin{array}{c} \gls{cameraX} \\ \gls{cameraY} \\ \gls{cameraZ}\end{array} \right]%
=\gls{intrinsicMatrixK} \begin{bmatrix} \gls{exRotationR} & \gls{exTranslationT} \end{bmatrix} \left[ \begin{array}{c} X^W \\ Y^W \\ Z^W \\ 1 \end{array} \right]%
=\gls{pinHoleCameraM} \left[ \begin{array}{c} X^W \\ Y^W \\ Z^W \\ 1 \end{array} \right]%
 , %
\label{pinholeCameraMatrixCalculation}
\end{equation}%
\noindent
where: %
\begin{equation}
\gls{pinHoleCameraM} = \gls{intrinsicMatrixK} \begin{bmatrix} \gls{exRotationR} & \gls{exTranslationT} \end{bmatrix}%
= \begin{bmatrix}
m_{11} & m_{12} & m_{13} & m_{14} \\
m_{21} & m_{22} & m_{23} & m_{24} \\
m_{31} & m_{32} & m_{33} & m_{34} \\
\end{bmatrix} . %
\label{pinholeMatrix3x4M}
\end{equation}%
%
\noindent
Note that, although \(\gls{cameraZ}\) values can be retrieved from the depth sensor streams, they will be employed during the calculation of \(\gls{pinHoleCameraM}\), because they will be expressed by the third row parameters in matrix \(\gls{pinHoleCameraM}\). \(\gls{cameraZ}\) will only be used in the step of \gls{3D} reconstruction after the pinhole camera matrix M is determined, as will be discussed in details in Section~\ref{section3DcameraCalibration}. Thus, \(\gls{cameraZ}\) in eqn.~(\ref{pinholeCameraMatrixCalculation}) is commonly substituted as an intermediate parameter \(k\). We did not change \(\gls{cameraZ}\) for the consistency of derivations.
To inspect the pinhole camera matrix \(\gls{pinHoleCameraM}\), it is composed of rotation/translation matrix for \gls{3D} space transforming and intrinsic perspective matrix for handling both of perspective view mapping and shape-skewing, all of which belong to linear processing. In other words, this $3\times4$ transformation matrix is specially for handling perspective view, or perspective distortion. The pinhole camera model is based on the homogeneous coordinates, which means its matrix \(\gls{pinHoleCameraM}\) is also limited by linear processing.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%
%%%%%%%%%%      3.        \gls{3D} Reconstruction from Depth                               %%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{\gls{3D} Camera Calibration}
\label{section3DcameraCalibration}
%    3. Pinhole camera matrix solving for matrix M
\indent
The calibration of a \gls{3D} camera aims to be able to generate the world coordinates (\(\gls{worldX}, \gls{worldY}, \gls{worldZ}\)) and corresponding \(RGB\) values for every single pixel, given the depth steams and RGB streams retrieved from the \gls{3D} camera. From Section~\ref{sectionPinholeCamera}, we know that the pinhole camera matrix \(\gls{pinHoleCameraM}\) (eqn.~(\ref{pinholeMatrix3x4M})) could help map from the world space to image space; however not able to directly transform image space data to world space coordinates. In order to determine \(\gls{worldX}/\gls{worldY}/\gls{worldZ}\) (based on eqn.(~\ref{HomoProportionalFromCamToImInPixels}) and eqn.(~\ref{mappingFromWorldToCameraSpace})), both of the intrinsic camera matrix and extrinsic camera matrix are needed, both of which are intermediate parameters and practically can only be determined through matrix \(\gls{pinHoleCameraM}\). Thus, the first job for \gls{3D} camera calibration is to solve the pinhole camera matrix \(\gls{pinHoleCameraM}\).
\\\indent
To solve the pinhole camera matrix, we can use least squares fit with known \gls{3D} points (\(\gls{worldX}\), \(\gls{worldY}\), \(\gls{worldZ}\)) and their corresponding image points (\(R, C\)). With one point, based on eqn.~(\ref{pinholeMatrix3x4M}) and (\ref{pinholeCameraMatrixCalculation}), we can get two equations.
\begin{equation}
\begin{aligned}
m_{11}\gls{worldX} + m_{12}\gls{worldY} + m_{13}\gls{worldZ} + m_{14} - m_{31}X^WC - m_{32}Y^WC - m_{33}Z^WC - m_{34}C = 0%
\\%
m_{21}\gls{worldX} + m_{22}\gls{worldY} + m_{23}\gls{worldZ} + m_{24} - m_{31}X^WR - m_{32}Y^WR - m_{33}Z^WR - m_{34}R = 0
\end{aligned}
\label{onePointEquationCR}
\end{equation}%
\noindent
There are totally 12 unknowns to solve, thus we need at least six points to solve the $3\times4$ pinhole camera matrix \(\gls{pinHoleCameraM}\). Using $n$-points least squares to solve the best fit, we can build a $2n$ equations matrix, given by eqn.~(\ref{nPoints2nEquationCR}).

\begin{equation}
\hspace*{-1cm}
\begin{bmatrix}
\gls{worldX}_1 & \gls{worldY}_1 & \gls{worldZ}_1 & 1 & 0 & 0 & 0 & 0 & -\gls{worldX}_1C_1 & -\gls{worldY}_1C_1 & -\gls{worldZ}_1C_1 & -C_1\\
0 & 0 & 0 & 0 & \gls{worldX}_1 & \gls{worldY}_1 & \gls{worldZ}_1 & 1 &  -\gls{worldX}_1R_1 & -\gls{worldY}_1R_1 & -\gls{worldZ}_1R_1 & -R_1\\
\gls{worldX}_2 & \gls{worldY}_2 & \gls{worldZ}_2 & 1 & 0 & 0 & 0 & 0 & -\gls{worldX}_2C_2 & -\gls{worldY}_2C_2 & -\gls{worldZ}_2C_2 & -C_2\\
0 & 0 & 0 & 0 & \gls{worldX}_2 & \gls{worldY}_2 & \gls{worldZ}_2 & 1 &  -\gls{worldX}_2R_2 & -\gls{worldY}_2R_2 & -\gls{worldZ}_2R_2 & -R_2\\
 & & & & & & & \vdots & & & & \\
\gls{worldX}_n & \gls{worldY}_n & Z^n_2 & 1 & 0 & 0 & 0 & 0 & -\gls{worldX}_nC_n & -\gls{worldY}_nC_n & -\gls{worldZ}_nC_n & -C_n\\
0 & 0 & 0 & 0 & \gls{worldX}_n & \gls{worldY}_n & \gls{worldZ}_n & 1 & -\gls{worldX}_nR_n & -\gls{worldY}_nR_n & -\gls{worldZ}_nR_n & -R_n
\end{bmatrix}
\begin{bmatrix}
m_{11} \\ m_{12} \\ m_{13} \\ m_{14} \\
\vdots
\\ m_{33} \\ m_{34}
\end{bmatrix}
=
\begin{bmatrix}
0 \\ 0 \\ 0 \\ 0 \\
\vdots \\ 0 \\ 0
\end{bmatrix}
\label{nPoints2nEquationCR}
\end{equation}%

\noindent
Considering that this matrix is build on homogeneous system, there is no unique solution. There can always be a total-zeros solution. %
\\\indent
To make the solution unique, we select \(m_{34} = 1\), so that the homogeneous eqn.~(\ref{nPoints2nEquationCR}) could be changed into an inhomogeneous format like \(\textbf{\textit{A}}X = B\), where the known matrix \(\textbf{\textit{A}}\) is a $2n\times11$ matrix and known matrix \(B\) is a $2n$ vector:

\begin{equation}
\hspace*{-0.1cm}
\begin{bmatrix}
\gls{worldX}_1 & \gls{worldY}_1 & \gls{worldZ}_1 & 1 & 0 & 0 & 0 & 0 & -\gls{worldX}_1C_1 & -\gls{worldY}_1C_1 & -\gls{worldZ}_1C_1\\
0 & 0 & 0 & 0 & \gls{worldX}_1 & \gls{worldY}_1 & \gls{worldZ}_1 & 1 &  -\gls{worldX}_1R_1 & -\gls{worldY}_1R_1 & -\gls{worldZ}_1R_1\\
\gls{worldX}_2 & \gls{worldY}_2 & \gls{worldZ}_2 & 1 & 0 & 0 & 0 & 0 & -\gls{worldX}_2C_2 & -\gls{worldY}_2C_2 & -\gls{worldZ}_2C_2\\
0 & 0 & 0 & 0 & \gls{worldX}_2 & \gls{worldY}_2 & \gls{worldZ}_2 & 1 &  -\gls{worldX}_2R_2 & -\gls{worldY}_2R_2 & -\gls{worldZ}_2R_2\\
 & & & & & & & \vdots & & &\\
\gls{worldX}_n & \gls{worldY}_n & Z^n_2 & 1 & 0 & 0 & 0 & 0 & -\gls{worldX}_nC_n & -\gls{worldY}_nC_n & -\gls{worldZ}_nC_n\\
0 & 0 & 0 & 0 & \gls{worldX}_n & \gls{worldY}_n & \gls{worldZ}_n & 1 & -\gls{worldX}_nR_n & -\gls{worldY}_nR_n & -\gls{worldZ}_nR_n
\end{bmatrix}
\begin{bmatrix}
m_{11} \\ m_{12} \\ m_{13} \\ m_{14} \\
\vdots
 \\ m_{32} \\ m_{33}
\end{bmatrix}
=
\begin{bmatrix}
C_1 \\ R_1 \\ C_2 \\ R_2 \\
\vdots \\ C_n \\ R_n
\end{bmatrix} .
\label{inHomogenousNPoints2nEquationCR}
\end{equation}%
\noindent
Using pseudo inverse, eqn.~(\ref{inHomogenousNPoints2nEquationCR}) can be solved by \(X = (\textbf{\textit{A}}^T\textbf{\textit{A}})^{-1}\textbf{\textit{A}}^TB\), where \(X\) is an 11-elements vector and \(X(1)\)  \texttildelow \, \(X(11)\) correspond to \(m_{11}\) \texttildelow \, \(m_{33}\). And the $3\times4$ pinhole camera matrix (eqn.~(\ref{pinholeMatrix3x4M})) will be solved as:

\begin{equation}
\gls{pinHoleCameraM} =
\begin{bmatrix}
X(1) & X(2) & X(3) & X(4) \\
X(5) & X(6) & X(7) & X(8) \\
X(9) & X(10) & X(11) & 1
\end{bmatrix}
\label{determinationOfPinhole3x4}
\end{equation}%
\\
\noindent
After we get the perspective projection matrix \(\gls{pinHoleCameraM}\), the next step is to recover the intrinsic and extrinsic camera matrix \(\gls{intrinsicMatrixK}\) and [\(\gls{exRotationR}, \, \gls{exTranslationT}\)], with which we could generate the world coordinates \(\gls{worldX}/\gls{worldY}/\gls{worldZ}\).
\\\indent
Starting from the decomposition of eqn.~(\ref{pinholeMatrix3x4M}) step by step:

\begin{equation}
\gls{pinHoleCameraM} =
\begin{bmatrix}
m_{11} & m_{12} & m_{13} &  \\
m_{21} & m_{22} & m_{23} & \textbf{\textit{O}}_{3*1} \\
m_{31} & m_{32} & m_{33} &
\end{bmatrix}%
+
\begin{bmatrix}
 &  &  & m_{14} \\
 & \textbf{\textit{O}}_{3*3} &  & m_{24} \\
 &  &  & m_{34} \\
\end{bmatrix} , %
\label{decomposePerspectiveProjectionOne}
\end{equation}%

\begin{equation}
\gls{intrinsicMatrixK} [\gls{exRotationR} \, \, \gls{exTranslationT}] =
\begin{bmatrix}
\gls{intrinsicMatrixK} \gls{exRotationR} & \textbf{\textit{O}}_{3*1}
 \end{bmatrix}%
+
\begin{bmatrix}
\textbf{\textit{O}}_{3*3} & \gls{intrinsicMatrixK} \gls{exTranslationT}
\end{bmatrix} , %
\label{decomposePerspectiveProjectionTwo}
\end{equation}%
%
and

\begin{equation}
\textbf{\textit{M}}_{3*3} =
\begin{bmatrix}
m_{11} & m_{12} & m_{13} \\
m_{21} & m_{22} & m_{23} \\
m_{31} & m_{32} & m_{33}
\end{bmatrix}%
=
\gls{intrinsicMatrixK} \gls{exRotationR}  ,
\label{QRdecompositionEquation}
\end{equation}%
\noindent
where \(\textbf{\textit{O}}\) denotes zero matrices with their sizes noted by subscripts. From eqn.~(\ref{rotationTranslationMatrixRT}), we know that \(\gls{exRotationR}\) is a standard rotation matrix, which has its property of orthogonal. Also from eqn.~(\ref{intrinsicKmatrix}), we know that \(\gls{intrinsicMatrixK}\) is an upper triangular matrix. Thus, all of the above fit in the prerequisites of RQ decomposition, which is a technique that could help us decompose the \(\textbf{\textit{M}}_{3*3}\) into the upper triangular intrinsic matrix \(\gls{intrinsicMatrixK}\) and rotation matrix \(\gls{exRotationR}\). After we got \(\gls{exRotationR}\), the translation matrix \(\gls{exTranslationT}\) could be determined with eqn.~(\ref{decomposePerspectiveProjectionTwo}).
\\\indent%
% 4. cite/introduce the static 90 degree angle calibration system, collect data for solving equation 1.11
%
Now we find the way to determine both of the intrinsic camera matrix and the extrinsic camera matrix. With depth streams measuring \(\gls{cameraZ}\), we are able to transform the 2D image data retrieved from the camera into \gls{3D} camera space point cloud by eqn.~(\ref{HomoProportionalFromCamToImInPixels}), and then generate the world space point cloud by eqn.~(\ref{mappingFromWorldToCameraSpace}). The basic pinhole camera model calculation is widely used in various camera calibration techniques. Based on different calibration systems, Zhengyou \cite{Zhengyou04} classified those calibration techniques into four categories: unknown scene points in the environment (self-calibration), 1D objects (wand with dots), 2D objects (planar patterns undergoing unknown motions) and \gls{3D} apparatus (two or three planes orthogonal to each other).
\\\indent
Self-calibration technique do not use any calibration object, and can be considered as zero-dimension approach because only image point correspondences are required. Just by moving a camera in a static scene, the rigidity of the scene provides in general two constraints \cite{selfCalibration3_1992} on the cameras' internal parameters from one camera displacement by using image information alone. Therefore, if images are taken by the same camera with fixed internal parameters, correspondences between three images are sufficient to recover both the internal and external parameters which allow us to reconstruct 3D structure up to a similarity \cite{selfCalibration2_1997, selfCalibration1_1994}.
\\\indent
One-Dimension, points-line calibration employs one dimension objects composed of a set of collinear points. With much lower cost than two dimensional or even three dimensional calibration system, using one dimension objects in camera calibration is not only a theoretical aspect, but is also very important in practice especially when multi-cameras are involved in the environment. To calibrate the relative geometry between multiple cameras, it is necessary for all involving cameras to simultaneously observe a number of points. It is hardly possible to achieve this with \gls{3D} or 2D calibration apparatus if one camera is mounted in the front of a room while another in the back. This is not a problem for 1D objects. Xiangjian \cite{oneDcalibration1_2006} shows how to estimate the internal and external parameters using one dimensional pattern in the camera calibration. And Zijian \cite{oneDcalibration2_2008} employed one dimensional objects as virtual environments in practical multiple cameras calibration.
\\\indent
Two and three dimensional object calibration systems usually give better calibrations. Sturm \textit{et al}. \cite{twoDcalibration1_1999} presented a general algorithm for plane-based calibration
that can deal with arbitrary numbers of views that observe a planar pattern shown at different orientations, so that almost anyone can make such a calibration pattern by him/her-self, and the setup is very easy. Both of Matlab and OpenCV have applied this two dimension plane calibration method in their applications. Zhengdong \cite{twoDcalibration2_2011} compared this two dimension plane camera calibration method and self-calibration method. Hamid \cite{twoDcalibration3_2015} applied this method into practical calibration and employed the calibrated camera into camera pose estimation and distance estimation application.
\\\indent
In three-dimensional object calibration technique, camera calibration is performed by observing a calibration object whose geometry in \gls{3D} space is known for very good precision. Calibration can be done very efficiently \cite{treeDcalibration1_1993}. The calibration objects usually consist of two or three planes orthogonal to each other. Paul \cite{treeDcalibration2_1996} applied the three dimension object calibration in his PHD project. Mattia \cite{threeDExample_2014} wrote a detailed tutorial from building the \gls{3D} object (Fig.~\ref{buildingThreeDCalibrationObject}) for calibration, to scanning using the calibrated camera. Figure~\ref{threeDSixPointsCalibrating} shows how six points are selected for calibration and Fig.~\ref{3DreconstructAfterCalibration} shows the \gls{3D} reconstruction after calibration.
%
\begin{figure}[!t]
\centering
\includegraphics[width=0.9\textwidth]{buildingThreeDCalibrationObject}
\caption{Building \gls{3D} Calibration Object \cite{threeDExample_2014}}
\label{buildingThreeDCalibrationObject}
\end{figure}%
%
\begin{figure}[!t]
\centering
\subfloat[Six Points to Calibrate]{
	\includegraphics[height=0.6\textwidth, width=0.45\textwidth]{threeDSixPointsCalibrating}
	\label{threeDSixPointsCalibrating}
}
\subfloat[Reconstruction After Calibration]{
	\includegraphics[height=0.6\textwidth, width=0.45\textwidth]{3DreconstructAfterCalibration}
	\label{3DreconstructAfterCalibration}
}
\caption{Three Dimension Object Camera Calibration \cite{threeDExample_2014}}
\label{twoPlanesCalibration}
\end{figure}%
%

%\begin{figure}[p]
%\centering
%\includegraphics[width=0.8\textwidth]{buildingThreeDCalibrationObject}
%\caption{Building Three Dimension Object}
%\label{buildingThreeDCalibrationObject}
%\end{figure}%
%
%\begin{figure}[b]
%\centering
%\includegraphics[width=0.8\textwidth]{buildingThreeDCalibrationObject}
%\caption{Building Three Dimension Object}
%\label{buildingThreeDCalibrationObject}
%\end{figure}%

%\\\\%

Zhengyou Zhang \cite{zhangCalibration1_2004, zhangCalibration2_2000, Zhengyou04} has deep studies on camera calibration from one-dimension calibration to tree-dimension calibration. The accuracy of calibration from 1D to \gls{3D} is getting better, but the calibration system set-up needs more and more work and cost as well. One dimension object is suitable for calibrating multiple cameras at once. Two dimension planer pattern approaches seems to be a good compromise, with good accuracy and simple setup. Also using the three dimension method for calibration, Kai \cite{Kai10} derived the per-pixel  beam equation, the linear relationship that could map to \(\gls{worldX}/\gls{worldY}\) from \(\gls{worldZ}\) as eqn.~(\ref{kaiBeamEquation}) shows, directly from pinhole camera matrix \(\gls{pinHoleCameraM}\). That is to say, we could easily look up \(\gls{worldX}/\gls{worldY}\) after calibration once found the way to get \(\gls{worldZ}\).

\begin{equation}
\begin{aligned}
\gls{worldX} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = a [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  \gls{worldZ} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]+d [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]
\\%
\gls{worldY} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] = c [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]  \gls{worldZ} [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]+d [\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}]
\end{aligned}
\label{kaiBeamEquation}
\end{equation}%
\noindent
where \(a/b/c/d\) are per-pixel coefficients for the linear beam equations, and the subscripts \([\gls{imDiscreteRow}, \, \gls{imDiscreteColumn}] \) are corresponding pixel address in image space.

%\gls{D}[r, c] ~= \gls{worldZ}[r, c] -- > (X^c, Y^c)
%\gls{worldZ}[r, c] = Z^c[r, c] = \gls{D}[r,c]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%
%%%%%%%%%%      4.        Lens Distortion Removal                                                %%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Lens Distortion}
%* distortion equation
\indent
All above in Chapter \ref{chapterTraditionalCalibration} are talking about the ideal pinhole camera, without lenses. Whereas in practice, as a result of several types of imperfections in the design and assembly of lenses composing the camera optical system, there are always lens distortions for a camera, and the expressions in eqn.~(\ref{twoDRelativeFromCamToIm}) are not valid any more. Lens distortion could be classified into two groups \cite{distortion1_1992} : radial distortion and tangential distortion. Imperfect lens shape causes light rays bending more near the edges of a lens than they do at its optical center. Barrel distortions happen commonly on wide angle lenses, where the field of view of the lens is much wider than the size of the image sensor \cite{whatisDistortion_2013}. Improper lens assembly will lead to tangential distortion, which occurs when the lens and the image plane are not parallel. Figure~\ref{RadialAndTangentialDistortion} shows how radial distortion \(d_\text{r}\) and tangential distortion \(d_\text{t}\) affect the object point position in the image. Note that both of radial distortion and tangential distortion are with respect to image space row and column, and what we will take later is negative distortion instead of positive. Distortions are present because the field of view (\gls{FoV}) in camera space has been affected by the lens.
%
\begin{figure}[b]
\centering
\includegraphics[width=0.65\textwidth]{RadialAndTangentialDistortion}
\caption{Radial and Tangential Distortion Affection In Image Space}
\label{RadialAndTangentialDistortion}
\end{figure}%
%
For most consumer \gls{RGBD} cameras with cheap lens, their distortions are usually barrel distortions (negative distortion) resulted by the enlarged field  of view in the camera space, because the larger view was squeezed into the sensor. Figure~\ref{DistortionComprehension} intuitively shows how the lens enlarged the field of view of in the camera space and then generates the barrel distortions.
%
\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{DistortionComprehension}
\caption{From Camera Space to Image Space with Lens Distortions}
\label{DistortionComprehension}
\end{figure}%

There are (a)(b)(c) three parts shown in Fig.~\ref{DistortionComprehension}. Each part has the pinhole camera only on the top, in contrast to the camera-with-lens situation at the bottom. To understand how the barrel distortion happens, we should go through from part (c) to part (a). In part (c), the gray background uniform grid is the \enquote{object} our that the camera is going to observe, and the blue frames shows the \gls{FoV} of the camera in the camera space. Due to the fact that, there will be worse and worse distortions as one pixel goes from the center to the edge, the enlarged \gls{FoV} of a camera with lens in the camera space is in pincushion (or star) shape. With the enlarged \gls{FoV} is mapped to the \enquote{Virtual Focal Plane}, as defined in Fig.~\ref{PinHoleVirtualFocalPlane}, the pincushion shape doesn't change because rays from the camera space have not gone through the lens yet. Note that we quoted the \enquote{Virtual Focal Plane} because the image on this virtual plane, when considering lens distortion, does not equal to the real focal plane (where the sensor is) any more. We can tell from part (b) that, even though the image space coordinates still are composed of \(\gls{imageColumn}\) and \(\gls{imageRow}\), their ranges have changed from positive integers only to the whole real integers that include negative ones. But the sensor never changes, and so the image space in part (a) still has its range of positive integers. With rays going through the lens, the pincushion-shape \gls{FoV} (the frame in blue) will be squeezed into a small rectangle, and thus we get the image in the real focal plane with its background grid showing a barrel distorted shape. %
%\\\indent%
With lens distortions counted, eqn.~(\ref{twoDRelativeFromCamToIm}) now needs to be changed into
%
\begin{equation}
\left[ \begin{array}{c} c'_r \\ r'_r \end{array} \right] %
= f %
\left[ \begin{array}{c} \gls{cameraPointX0}/\gls{cameraPointZ0} \\ \gls{cameraPointY0}/\gls{cameraPointZ0} \end{array} \right]%
\label{undistortedRelativeFromCamToIm}
\end{equation}
where \(c'_r\) and \(r'_r\) denote the relative pixel distance on the undistorted \enquote{Virtual Focal Plane}, whose \gls{FoV} is pincushion-shape and image coordinates' ranges include negative integers.%
\\\indent%
Duane \cite{distortion2_1966} gave the lens distortion equation, and the undistorted \(\gls{imageColumn}\) and \(\gls{imageRow}\) (\(C'/R'\) in our notation) can be expressed as power series in radial distance \(r = \sqrt{C^2 + R^2}\):
%
\begin{equation}
\begin{aligned}
\gls{UndistortedImColumn} =  C (1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + [p_1 (r^2 + 2 C^2) + 2 p_2 CR] %
\\
\gls{UndistortedImRow} =  R (1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + [p_2 (r^2 + 2 R^2) + 2 p_1 CR]
\end{aligned}
\label{lensDistortion}
\end{equation}%
%
\noindent
where higher order parameters are omitted for being negligible; \((\gls{UndistortedImColumn} , \, \gls{UndistortedImRow})\) denote the undistorted pixels in the \enquote{Virtual Focal Plane}, \((\gls{imageColumn}, \, \gls{imageRow} )\) denote the distorted pixel in real sensor image, \(k_i\)s are coefficients of radial distortion, and \(p_j\)s are coefficients of tangential distortion. The five parameters \(k_1/k_2/k_3/p_1/p_2\) are usually called distortion parameters. With the distortion parameters calculated, the distorted \((\gls{imageColumn}, \, \gls{imageRow} )\) could be undistorted into \((\gls{UndistortedImColumn} , \, \gls{UndistortedImRow})\), and then \((\gls{UndistortedImColumn} , \, \gls{UndistortedImRow})\) could be used to generate the world space \(\gls{worldX}/\gls{worldY}/\gls{worldZ}\) with intrinsic and extrinsic parameters.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%
%%%%%%%%%%      4.       Summation                                                                  %%%%%%%%%%%
%%%%%%%%%%                                                                                %%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Summation}
%%
\indent
\begin{figure}[t]
\centering
\includegraphics[width=0.6\textwidth]{flowChart}
\caption{Traditional Camera Calibration Flow Chart}
\label{flowChart}
\end{figure}%
\\\indent
Considering the lens distortion correction, Fig.~\ref{flowChart} shows the flow chart of the whole traditional camera calibration method based on the pinhole camera model. Considering the lens distortions, both of the pinhole camera model (matrix \(\gls{pinHoleCameraM}\)) and the lens distortions model (five parameters for undistortion) need to be determined. The pinhole camera model can help map from the world space (\(\gls{worldX}, \, \gls{worldY}, \, \gls{worldZ}\)) to the undistorted image space  \((\gls{UndistortedImColumn} , \, \gls{UndistortedImRow})\), which are on the \enquote{Virtual Focal Plane} as noted in Fig.~\ref{DistortionComprehension}. And the lens distortion model help remove the lens distortions by mapping from  \((\gls{UndistortedImColumn} , \, \gls{UndistortedImRow})\) to \((\gls{imageColumn}, \, \gls{imageRow} )\). The pinhole camera model can be determined by eqn.~(\ref{determinationOfPinhole3x4}), and the lens distortion model could be determined by eqn.~(\ref{lensDistortion}).

%
%\begin{equation*}%
%c_{[row, col]} %
%= \frac%
%{(m_{22}m_{33} - m_{23}m_{32})col + (m_{13}m_{32} - m_{12}m_{33})row + (m_{12}m_{23} - m_{13}m_{22})}%
%{(m_{21}m_{32} - m_{22}m_{31})col + (m_{12}m_{31} - m_{11}m_{32})row + (m_{11}m_{22} - m_{12}m_{21})} \, ,
%\end{equation*}
%%
%\begin{equation*}%
%d_{[row, col]} %
%= \frac%
%{(m_{22}m_{34} - m_{24}m_{32})col + (m_{14}m_{32} - m_{12}m_{34})row + (m_{12}m_{24} - m_{14}m_{22})}%
%{(m_{21}m_{32} - m_{22}m_{31})col + (m_{12}m_{31} - m_{11}m_{32})row + (m_{11}m_{22} - m_{12}m_{21})} \, ,
%\end{equation*}
%%
%\begin{equation*}%
%e_{[row, col]} %
%= \frac%
%{(m_{23}m_{31} - m_{21}m_{33})col + (m_{11}m_{33} - m_{13}m_{31})row + (m_{13}m_{21} - m_{11}m_{23})}%
%{(m_{21}m_{32} - m_{22}m_{31})col + (m_{12}m_{31} - m_{11}m_{32})row + (m_{11}m_{22} - m_{12}m_{21})} \, ,
%\end{equation*}
%%
%\begin{equation*}%
%f_{[row, col]} %
%= \frac%
%{(m_{23}m_{32} - m_{22}m_{33})col + (m_{12}m_{33} - m_{13}m_{32})row + (m_{13}m_{22} - m_{12}m_{23})}%
%{(m_{21}m_{32} - m_{22}m_{31})col + (m_{12}m_{31} - m_{11}m_{32})row + (m_{11}m_{22} - m_{12}m_{21})}
%\end{equation*}
%
%


\begin{appendices}

\section{Python code used for producing the over-underfitting example in figure \ref{fig:regr_example}} \label{app:overfitting}
\begin{lstlisting}
import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

def true_fun(X):
    return np.cos(1.5 * np.pi * X)

np.random.seed(0)

n_samples = 30
degrees = [1, 4, 15]

X = np.sort(np.random.rand(n_samples))
y = true_fun(X) + np.random.randn(n_samples) * 0.1

plt.figure(figsize=(14, 5))
for i in range(len(degrees)):
    ax = plt.subplot(1, len(degrees), i + 1)
    plt.setp(ax, xticks=(), yticks=())

    polynomial_features = PolynomialFeatures(degree=degrees[i],
                                             include_bias=False)
    linear_regression = LinearRegression()
    pipeline = Pipeline([("polynomial_features", polynomial_features),
                         ("linear_regression", linear_regression)])
    pipeline.fit(X[:, np.newaxis], y)

    # Evaluate the models using crossvalidation
    scores = cross_val_score(pipeline, X[:, np.newaxis], y,
                             scoring="neg_mean_squared_error", cv=10)

    X_test = np.linspace(0, 1, 100)
    plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
    plt.plot(X_test, true_fun(X_test),
             label="True function", linestyle='dashed')
    plt.scatter(X, y, edgecolor='b', s=20, label="Samples")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.xlim((0, 1))
    plt.ylim((-2, 2))
    plt.legend(loc="best")
    plt.title("Degree {}\nMSE_cv = {:.2e}(+/- {:.2e})".format(
        degrees[i], -scores.mean(), scores.std()))
plt.show()

\end{lstlisting}

\section{Python code for producing the activation functions in figure \ref{fig:act_funcs}} \label{app:act_funcs}

\begin{lstlisting}
import tensorflow as tf
import matplotlib.pyplot as plt

x = tf.linspace(-10., 10., 200)
y_sigmoid = tf.keras.activations.sigmoid(x)
y_tanh = tf.keras.activations.tanh(x)
y_relu = tf.keras.activations.relu(x)
y_elu = tf.keras.activations.elu(x)
y_step = x > 0

plt.plot(x, y_sigmoid, label='Sigmoid')
plt.plot(x, y_tanh, label=r'$tanh$')
plt.plot(x, y_relu, label='ReLU')
plt.plot(x, y_elu, label='ELU')
plt.plot(x, y_step, label='Step')
axes = plt.gca()
axes.set_ylim([-2, 2])
plt.xlabel('x')
plt.ylabel('g(x)')
plt.legend()
plt.savefig('act_func_fig.pdf')
plt.show()
\end{lstlisting}

\section{Python code for implementing the simple Bayesian neural network illustrated in figure \ref{fig:simple_BNN}} \label{app:simple_BNN}
\begin{lstlisting}
#import random as rn
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions

# -------------------------------- Creating sin-data -------------------------------

def true_fun(x):
    return np.sin(3 * x)  # np.sin(1.5 * np.pi * x)

np.random.seed(42)
n_x = 6
x_train = np.sort(np.random.rand(n_x))
y_train = true_fun(x_train)

# --------------------- Build and compile neural net -------------------------------

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(16, input_shape=([1, ]), activation='tanh'),
    tf.keras.layers.Dense(1, activation='tanh')
])
model.summary()

# --------------------- Sample weights through neural networks with acceptance-prop equal to likelihood ----------------------

n_NN = 10**5
weight_list = []
likelihood_list = []
sigma_k = 0.1  # sd of assumed gauss P(y \mid x), Neal's eq 1.8, Neal uses 0.1
tf.random.set_seed(42)
# Sample neural networks and save their likelihood and weights
for i in range(n_NN):
    print(i)
    # sample weights and set them to hidden layer "dense" and output layer "dense_1"
    for layer in model.layers:
        if layer.name == "dense":
            layer.set_weights([np.random.normal(0, 8, size=w.shape)
                               for w in layer.get_weights()])
        if layer.name == "dense_1":
            layer.set_weights([np.random.normal(
                0, 1/np.sqrt(16), size=w.shape) for w in layer.get_weights()])
    # save weights in list
    weight_list.append(model.get_weights())
    # Calculate gauss likelihood of y_train given weights and x_train, Neal's eq 1.11
    mean = model.predict(x_train)
    gauss = tfd.Normal(loc=mean, scale=sigma_k)
    likelihood = 1
    for j in range(n_x):
        likelihood *= gauss[j].prob(y_train[j])
    likelihood_list.append(likelihood)

# Normalize likelihood to max prob is 1
if max(likelihood_list) != 0:
    likelihood_list = likelihood_list/max(likelihood_list)

# Accept model with prob equal to normalized likelihood
accepted_weights = []
accepted_likelihood = []
for i in range(len(likelihood_list)):
    uniform_dist = tfd.Uniform(0, 1)
    if likelihood_list[i] >= uniform_dist.sample():
        accepted_weights.append(weight_list[i])
        accepted_likelihood.append(likelihood_list[i])

# --------------------- Use sampled weights for predicting y's ----------------------
x_pred = tf.linspace(0.0, 1, 200)
y_pred = []
for i in range(len(accepted_weights)):
    model.set_weights(accepted_weights[i])
    y_pred.append(model.predict(x_pred))

# mean y_pred
mean_y_pred = np.array(y_pred).mean(axis=0)

y_pred_std = np.array(y_pred).std(axis = 0)
# Lower std-line
lower_std = mean_y_pred - y_pred_std
# Upper std-line
upper_std = mean_y_pred + y_pred_std

# --------------------- Plot of BNN results ----------------------
plt.scatter(x_train, y_train,
            edgecolor='b', s=40, label="Datapoint")
for i in range(len(y_pred)):
    plt.plot(x_pred, y_pred[i], color='k', linestyle='dashed')
plt.plot(x_pred, mean_y_pred, color='coral', label="Average prediction")
plt.fill_between(x_pred, lower_std.flatten() , upper_std.flatten() , color='b', alpha=.1)
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.savefig('figure_simple_BNN.pdf')
plt.show()

\end{lstlisting}

\section{Python code for Metropolis implementation used for producing figure \ref{fig:MH_sampling} }\label{app:MH_code}
\begin{lstlisting}
# # ----------------------------- IMPORTS ---------------------------

import numpy as np
import scipy.stats as ss
import matplotlib.pyplot as plt

# # ----------------------------- Defining functions ---------------------------
# Defining target probability
def p(x):
    sigma = np.array([[1, 0.6], [0.6, 1]])  # Covariance matrix
    return ss.multivariate_normal.pdf(x, cov=sigma)

# # ----------------------------- Sampling  ---------------------------
samples = np.zeros((1000, 2))
np.random.seed(42)

x = np.array([7, 0])
for i in range(1000):
    samples[i] = x
    # Gaussian proposal for symmetry
    x_prime = np.random.multivariate_normal(mean=x, cov=np.eye(2), size=1).flatten()
    acceptance_prob = min(1, (p(x_prime) )/ (p(x)))
    u = np.random.uniform(0, 1)
    if u <= acceptance_prob:
        x = x_prime
    else:
        x = x

# # ----------------------------- Vizualising  ---------------------------

# For vizualising normal contours
X, Y = np.mgrid[-3:3:0.05, -3:3:0.05]
X, Y = np.mgrid[-3:3:0.05, -3:3:0.05]
XY = np.empty(X.shape + (2,))
XY[:,:,0] = X; XY[:,:,1] = Y
target_distribution = ss.multivariate_normal(mean=[0,0], cov=[[1, 0.6],[0.6, 1]])

plt.subplot(2, 2, 1) # row 1, col 2 index 1
plt.contour(X, Y, target_distribution.pdf(XY),cmap=plt.cm.Blues)
plt.ylim(-3,5)
plt.xlim(-3,8)
plt.subplot(2, 2, 2) # index 2
plt.plot(samples[0:100,0], samples[0:100,1], 'ro-', color="navy", linewidth=.2, markersize=.7,label="First 100 samples")
plt.contour(X, Y, target_distribution.pdf(XY),cmap=plt.cm.Blues)
plt.legend(loc="upper right",fontsize=9)
plt.ylim(-3,5)
plt.xlim(-3,8)
plt.subplot(2, 2, 3) # index 3
plt.plot(samples[0:200,0], samples[0:200,1], 'ro-', color="navy", linewidth=.2, markersize=.7,label="First 200 samples")
plt.contour(X, Y, target_distribution.pdf(XY),cmap=plt.cm.Blues)
plt.legend(loc="upper right",fontsize=9)
plt.ylim(-3,5)
plt.xlim(-3,8)
plt.subplot(2, 2, 4) # index 4
plt.plot(samples[0:300,0], samples[0:300,1], 'ro-', color="navy", linewidth=.2, markersize=.7, label="First 300 samples")
plt.contour(X, Y, target_distribution.pdf(XY),cmap=plt.cm.Blues)
plt.legend(loc="upper right",fontsize=9)
plt.ylim(-3,5)
plt.xlim(-3,8)
plt.savefig("metro_example.pdf")
plt.show()

\end{lstlisting}

\section{Python packages and specification for computer doing the evaluations in chapter \ref{chap:eval_NN}} \label{app:specs}
The code ran on a virtual machine for the Linux distribution Ubuntu 20.04.2 LTS. The virtual machine used VMWare Workstation 16.1.1 on a native Windows 10 64-bit version 2004. The hardware used is
\begin{itemize}
    \item GPU: NVidia GeForce GTX 970
    \item RAM allocated to virtual machine: 9.5 GB
    \item Processor: Intel® Core™ i5-6600K CPU @ 3.50GHz × 4
    \item A harddrive allocated only to this virtual machine with free space of 427 GB
\end{itemize}
The packages required for reproducing the evalutions are
\begin{itemize}
    \item \texttt{PyMC3}==3.11.2
    \item \texttt{Theano}==1.1.2
    \item \texttt{Arviz}==0.11.2
    \item \texttt{Numpy}==1.19.5
    \item \texttt{tensorflow}==2.4.1
    \item \texttt{tensorflow\_probability}==0.12.2
    \item \texttt{sklearn}==0.24.2
    \item \texttt{numpy}==1.19.5
    \item \texttt{seaborn}==0.11.1
    \item \texttt{matplotlib.pyplot}==3.4.1
\end{itemize}

\section{Python code for the neural networks in table \ref{tab:Boston_NN_performance}} \label{app:Boston_NN}
The network with with early stopping is performed using the code
\begin{lstlisting}
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import boston_housing
import time

tf.random.set_seed(40)

# ----------------------------- Prepare data ---------------------
(X_train, y_train), (X_test, y_test) = boston_housing.load_data(seed=3030)

# ----------------------------- Neural Network ---------------------
n_hidden = 10

model = tf.keras.Sequential([
    tf.keras.Input((13, ), name='feature'),
    tf.keras.layers.Dense(n_hidden, activation=tf.nn.relu),
    tf.keras.layers.Dense(1)
])
model.summary()

# Early stopping
es = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss', mode='min', patience=10, min_delta=0.1)

start_time = time.time()
# Compile, train, and evaluate.
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['mse'])
history = model.fit(X_train, y_train, epochs=300,
                    validation_split=0.3, callbacks=[es])

print("The algorithm ran", len(history.history['loss']), "epochs")

# ----------------------------- Overfitting? ---------------------
train_acc = model.evaluate(X_train, y_train, verbose=0)[-1]
test_acc = model.evaluate(X_test, y_test, verbose=0)[-1]
print("--- %s seconds ---" % (time.time() - start_time))
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))

plt.plot(history.history['loss'], label='train')
plt.plot(history.history['val_loss'], label='validation')
plt.legend()
plt.grid()
plt.show()

\end{lstlisting}
The networks not using early stopping and their visualization of train and validation loss in figure \ref{fig:Boston_NN_nohidden_wd_loss}, figure \ref{fig:Boston_NN_1hidden_wd_loss} and figure \ref{fig:Boston_NN_1hidden_noreg_loss} are produced using the code
\begin{lstlisting}
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import boston_housing
import time
from keras.regularizers import l2

tf.random.set_seed(40)
# ----------------------------- Prepare data ----------------------
(X_train, y_train), (X_test, y_test) = boston_housing.load_data(seed=3030)

# ----------------------------- Neural Network --------------------
reg_const = 0.3
n_hidden = 10

model = tf.keras.Sequential([
    tf.keras.Input((13, ), name='feature'),
    tf.keras.layers.Dense(n_hidden, activation=tf.nn.relu, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const)),
    tf.keras.layers.Dense(1, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const))
])
model.summary()

start_time = time.time()
# Compile, train, and evaluate.
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['mse'])
history = model.fit(X_train, y_train, epochs=300, validation_split=0.3)
model.evaluate(X_test, y_test)

# ----------------------------- Overfitting? --------------------
train_acc = model.evaluate(X_train, y_train, verbose=0)[-1]
test_acc = model.evaluate(X_test, y_test, verbose=0)[-1]
print("--- %s seconds ---" % (time.time() - start_time))
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))

plt.plot(history.history['loss'], label='Train')
plt.plot(history.history['val_loss'], label='Validation')
plt.legend()
plt.grid()
plt.ylabel('Loss')
plt.xlabel('Epochs')
plt.ylim(0, 200)
plt.savefig('figure_Boston_NN_1hidden_wd_loss.pdf')
plt.show()

\end{lstlisting}
where the network with no hidden layers are produced by removing the line \begin{lstlisting}
tf.keras.layers.Dense(n_hidden, activation=tf.nn.relu, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const))
\end{lstlisting}
and the network with 1 hidden layer and no regularization is produced by removing the regularization arguments \texttt{kernel\_regularizer} and \texttt{bias\_regularizer} in
\begin{lstlisting}
tf.keras.layers.Dense(n_hidden, activation=tf.nn.relu, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const)),
    tf.keras.layers.Dense(1, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const))
\end{lstlisting}

\section{Python code for the Bayesian neural networks in table \ref{tab:Boston_BNN_performance}} \label{app:Boston_BNN}
The Bayesian neural network with hierarchical model is implemented by the following code
\begin{lstlisting}
# # ----------------------------- IMPORTS -----------------------
import sys
import time
from keras.datasets import boston_housing
from sklearn import metrics
import numpy as np
import pymc3 as pm
import theano
import arviz as az
from arviz.utils import Numba
import theano.tensor as tt
Numba.disable_numba()
Numba.numba_flag
floatX = theano.config.floatX
import seaborn as sns
sns.set_style("white")
import tensorflow as tf

# Ignore warnings - NUTS provide many runtimeWarning
import warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)

tf.random.set_seed(42)
# # ----------------------------- Loading Boston data --------------
(X_train, y_train), (X_test, y_test) = boston_housing.load_data(seed=3030)

#pad Xs with 1's to add bias
ones_train=np.ones(X_train.shape[0])
ones_test=np.ones(X_test.shape[0])
X_train=np.insert(X_train,0,ones_train,axis=1)
X_test=np.insert(X_test,0,ones_test,axis=1)

# # ----------------------------- Implementing a BNN function -----

def construct_bnn(ann_input, ann_output, n_hidden):
    # Initialize random weights between each layer
    init_1 = np.random.randn(X_train.shape[1], n_hidden).astype(floatX)*.1
    init_out = np.random.randn(n_hidden,1).astype(floatX)*.1
    with pm.Model() as bayesian_neural_network:
        ann_input = pm.Data("ann_input", X_train)
        ann_output = pm.Data("ann_output", y_train)

    # prior on hyper parameters for weight 1
        #mu1 = pm.Normal('mu1',shape=(X_train.shape[1], n_hidden), mu=0, sigma=1)
        mu1 = pm.Cauchy('mu1',shape=(X_train.shape[1], n_hidden), alpha=0, beta=1)
        sigma1 = pm.HalfNormal('sigma1',shape=(X_train.shape[1], n_hidden), sigma=1)

    # Input -> Layer 1
        weights_1 = pm.Normal('w_1', mu=mu1, sd=sigma1,
                          shape=(X_train.shape[1], n_hidden),
                          testval=init_1)
        acts_1 = pm.Deterministic('activations_1', tt.nnet.relu(tt.dot(ann_input, weights_1)))

    # prior on hyper parameters for weight_out
        mu_out = pm.Cauchy('mu_out',shape=(n_hidden, 1), alpha=0, beta=1)
        sigma_out = pm.HalfNormal('sigma_out',shape=(n_hidden, 1), sigma=1)

    # Layer 1 -> Output Layer
        weights_out = pm.Normal('w_out', mu=mu_out, sd=sigma_out,
                            shape=(n_hidden, 1),
                            testval=init_out)
        acts_out = pm.Deterministic('activations_out',tt.dot(acts_1, weights_out))

    #Define likelihood
        out = pm.Normal('out', mu=acts_out[:,0], sd=1, observed=ann_output)

    return bayesian_neural_network

# # ---------------- Sampling from posterior ---------
# Start time
tic = time.perf_counter() # for timing
bayesian_neural_network_NUTS = construct_bnn(X_train, y_train, n_hidden=10)

# Sample from the posterior using the NUTS samplper
with bayesian_neural_network_NUTS:
    trace = pm.sample(draws=3000, tune=1000, chains=3,target_accept=.90)

# # ------------------ Making predictions on training data ---------
ppc1=pm.sample_posterior_predictive(trace, model=bayesian_neural_network_NUTS)

# Taking the mean over all samples to generate a prediction
y_train_pred = ppc1['out'].mean(axis=0)

# Replace shared variables with testing set
pm.set_data(new_data={"ann_input": X_test, "ann_output": y_test}, model=bayesian_neural_network_NUTS)

# # ---------------- Making predictions on test data ------
ppc2 = pm.sample_posterior_predictive(trace, model=bayesian_neural_network_NUTS)

# Taking the mean over all samples to generate a prediction
y_test_pred = ppc2['out'].mean(axis=0)

# End time
toc = time.perf_counter()
print(f"Run time {toc - tic:0.4f} seconds")

# Printing the performance measures
print('MSE (NUTS) on training data:', metrics.mean_squared_error(y_train, y_train_pred))
print('MSE (NUTS) on test data:', metrics.mean_squared_error(y_test, y_test_pred))

\end{lstlisting}
The Bayesian neural network with one hidden layer is implemented by the following code
\begin{lstlisting}
# # ----------------------------- IMPORTS -------------------
import sys
import time
from keras.datasets import boston_housing
from sklearn import metrics
import numpy as np
import pymc3 as pm
import theano
import arviz as az
from arviz.utils import Numba
import theano.tensor as tt
Numba.disable_numba()
Numba.numba_flag
floatX = theano.config.floatX
# seaborn for vizualzing
import seaborn as sns
sns.set_style("white")
import matplotlib.pyplot as plt
import tensorflow as tf
# # ----------------------------- Print versions -------------

print("Running on Python version %s" % sys.version)
print(f"Running on PyMC3 version{pm.__version__}")
print("Running on Theano version %s" % theano.__version__)
print("Running on Arviz version %s" % az.__version__)
print("Running on Numpy version %s" % np.__version__)

# Ignore warnings - NUTS provide many runtimeWarning
import warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)

tf.random.set_seed(42)
# # ----------------------------- Loading Boston data ------------
(X_train, y_train), (X_test, y_test) = boston_housing.load_data(seed=3030)

#pad Xs with 1's to add bias
ones_train=np.ones(X_train.shape[0])
ones_test=np.ones(X_test.shape[0])
X_train=np.insert(X_train,0,ones_train,axis=1)
X_test=np.insert(X_test,0,ones_test,axis=1)

# # ----------------------------- Implementing a BNN function ------

def construct_bnn(ann_input, ann_output, n_hidden, prior_std):
    # Initialize random weights between each layer
    init_1 = np.random.randn(X_train.shape[1], n_hidden).astype(floatX)*prior_std
    init_out = np.random.randn(n_hidden,1).astype(floatX)*prior_std

    with pm.Model() as bayesian_neural_network:
        ann_input = pm.Data("ann_input", X_train)
        ann_output = pm.Data("ann_output", y_train)

    # Input -> Layer 1
        weights_1 = pm.Normal('w_1', mu=0, sd=prior_std,
                          shape=(X_train.shape[1], n_hidden),
                          testval=init_1)
        acts_1 = tt.nnet.relu(tt.dot(ann_input, weights_1))

    # Layer 1 -> Output Layer
        weights_out = pm.Normal('w_out', mu=0, sd=prior_std,
                            shape=(n_hidden, 1),
                            testval=init_out)
        acts_out = tt.dot(acts_1, weights_out)

    #Define likelihood
        out = pm.Normal('out', mu=acts_out[:,0], sd=1, observed=ann_output)

    return bayesian_neural_network

# # ----------------------------- Sampling from posterior -------
# Start time
tic = time.perf_counter() # for timing
bayesian_neural_network_NUTS = construct_bnn(X_train, y_train, n_hidden=10, prior_std=.1)

# Sample from the posterior using the NUTS samplper
with bayesian_neural_network_NUTS:
    trace = pm.sample(draws=3000, tune=1000, chains=3,target_accept=.9, random_seed=42)

# # ------------------ Making predictions on training data --------
ppc1=pm.sample_posterior_predictive(trace, model=bayesian_neural_network_NUTS, random_seed=42)

# Taking the mean over all samples to generate a prediction
y_train_pred = ppc1['out'].mean(axis=0)

# Replace shared variables with testing set
pm.set_data(new_data={"ann_input": X_test, "ann_output": y_test}, model=bayesian_neural_network_NUTS)

# # ------------------- Making predictions on test data ------
ppc2 = pm.sample_posterior_predictive(trace, model=bayesian_neural_network_NUTS, random_seed=42)

# Taking the mean over all samples to generate a prediction
y_test_pred = ppc2['out'].mean(axis=0)

# End time
toc = time.perf_counter()
print(f"Run time {toc - tic:0.4f} seconds")

# Printing the performance measures
print('MSE (NUTS) on training data:', metrics.mean_squared_error(y_train, y_train_pred))
print('MSE (NUTS) on test data:', metrics.mean_squared_error(y_test, y_test_pred))

# -------------------------------- Plots --------------------
# Vizualize uncertainty
# Define examples for which you want to examine the posterior predictive:
example_vec=np.array([1,2,4,9,10,11,15,16,22,24,27,28,30,44,55,62,68,72,84,93])
for example in example_vec:
    plt_hist_array=np.array(ppc2['out'])
    plt.hist(plt_hist_array[:,example], density=1, color="lightsteelblue", bins=30)
    plt.xlabel(f"Predicted value for example {example}",fontsize=13)
    plt.ylabel("Density",fontsize=13)
    plt.savefig(f'Python_code/Boston_BNN_1hidden_postpred_{example}.pdf')
    plt.show()
\end{lstlisting}
where the network with no hidden layers is implemented by replacing the lines
\begin{lstlisting}
# Input -> Layer 1
    weights_1 = pm.Normal('w_1', mu=0, sd=prior_std,
                    shape=(X_train.shape[1], n_hidden),
                    testval=init_1)
    acts_1 = tt.nnet.relu(tt.dot(ann_input, weights_1))

# Layer 1 -> Output Layer
    weights_out = pm.Normal('w_out', mu=0, sd=prior_std,
                    shape=(n_hidden, 1),
                    testval=init_out)
    acts_out = tt.dot(acts_1, weights_out)
\end{lstlisting}
with
\begin{lstlisting}
 # Input layer -> Output layer
        weights_out = pm.Normal('w_out', mu=0, sd=prior_std,
        shape=(X_train.shape[1], 1), testval=init_out)
    acts_out = pm.Deterministic(
        'activations_out', tt.dot(ann_input, weights_out))
\end{lstlisting}

\section{Python code for the neural networks in table \ref{tab:credit_NN_performance}} \label{app:Credit_NN}
The neural network with early stopping is performed using the code
\begin{lstlisting}
 import time
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
import seaborn as sns

tf.random.set_seed(40)
# ----------------------------- Prepare data -------------------
credit_data = pd.read_csv(
    "Python_code/data/UCI_Credit_Card.csv", encoding="utf-8", index_col=0)
credit_data.head()

# Data to numpy
data = np.array(credit_data)

# Extract labels
data_X = data[:, 0:23]
data_y = data[:, 23]

# # -------------------------- Subsamling credit data -------
X_train, X_test, y_train, y_test = train_test_split(
    data_X, data_y, test_size=0.30, random_state=3030)

N = 300
N_test = 100
X_train = X_train[0:N, :]
y_train = y_train[0:N]
X_test = X_test[0:N_test, :]
y_test = y_test[0:N_test]

# ------------------------- Neural Network ---------------

model = tf.keras.Sequential([
    tf.keras.Input((23, ), name='feature'),
    tf.keras.layers.Dense(10, activation=tf.nn.tanh),
    tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)
])
model.summary()

# Early stopping
es = tf.keras.callbacks.EarlyStopping(
    monitor='val_loss', mode='min', patience=0, min_delta=0)

start_time = time.time()

# Compile, train, and evaluate.
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['binary_crossentropy'])
history = model.fit(X_train, y_train,  validation_split=0.3,
                    epochs=1000, callbacks=[es])
print("The algorithm ran", len(history.history['loss']), "epochs")

print("--- %s seconds ---" % (time.time() - start_time))

# ------------------------- Overfitting? ----------------
train_acc = model.evaluate(X_train, y_train, verbose=0)[-1]
test_acc = model.evaluate(X_test, y_test, verbose=0)[-1]
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))

# taking mean of summed cross-entropy loss
train_loss = np.array(history.history['loss'])
val_loss = np.array(history.history['val_loss'])

plt.plot(train_loss, label='train')
plt.plot(val_loss, label='validation')
plt.legend()
plt.grid()
plt.show()

\end{lstlisting}
The networks not using early stopping and their visualization of train and validation loss in figure \ref{fig:Credit_NN_nohidden_wd_loss}, \ref{fig:Credit_NN_1hidden_wd_loss} and figure \ref{fig:Credit_NN_1hidden_noreg_loss} are produced using the code
\begin{lstlisting}
from keras.regularizers import l2
import time
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
import seaborn as sns
start_time = time.time()
tf.random.set_seed(40)

# ----------------------------- Prepare data ----------------
credit_data = pd.read_csv(
    "Python_code/data/UCI_Credit_Card.csv", encoding="utf-8", index_col=0)
credit_data.head()

# Data to numpy
data = np.array(credit_data)

# Extract labels
data_X = data[:, 0:23]
data_y = data[:, 23]

# # ----------------------------- Subsamling credit data ------
X_train, X_test, y_train, y_test = train_test_split(
    data_X, data_y, test_size=0.30, random_state=3030)

N = 300
N_test = 100
X_train = X_train[0:N, :]
y_train = y_train[0:N]
X_test = X_test[0:N_test, :]
y_test = y_test[0:N_test]

# ----------------------------- Neural Network ----------------
reg_const = 0.1
n_hidden = 10

model = tf.keras.Sequential([
    tf.keras.Input((23, ), name='feature'),
    tf.keras.layers.Dense(n_hidden, activation=tf.nn.tanh, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const)),
    tf.keras.layers.Dense(1, activation=tf.nn.sigmoid, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const))
])
model.summary()

# Compile, train, and evaluate.
val_ratio = 0.3
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['binary_crossentropy'])
history = model.fit(X_train, y_train, epochs=1000,
                    validation_split=val_ratio)

model.evaluate(X_test, y_test)

print("--- %s seconds ---" % (time.time() - start_time))

# ----------------------------- Overfitting? ----------------

train_acc = model.evaluate(X_train, y_train, verbose=0)[-1]
test_acc = model.evaluate(X_test, y_test, verbose=0)[-1]
print('Train: %.3f, Test: %.3f' % (train_acc, test_acc))

# taking mean of summed cross-entropy loss
train_loss = np.array(history.history['loss'])
val_loss = np.array(history.history['val_loss'])

plt.plot(train_loss, label='train')
plt.plot(val_loss, label='validation')
plt.legend()
plt.grid()
plt.ylim(0.4, 1)
plt.savefig('Python_code/figure_Credit_NN_1hidden_wd_loss.pdf')
plt.show()

\end{lstlisting}
where the network with no hidden layers are produced by removing the line
\begin{lstlisting}
 tf.keras.layers.Dense(n_hidden, activation=tf.nn.tanh, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const)),
\end{lstlisting}
and the network with 1 hidden layer and no regularization is produced by removing the regularization arguments \texttt{kernel\_regularizer} and \texttt{bias\_regularizer} in
\begin{lstlisting}
tf.keras.layers.Dense(n_hidden, activation=tf.nn.tanh, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const)),
tf.keras.layers.Dense(1, activation=tf.nn.sigmoid, kernel_regularizer=l2(
        reg_const), bias_regularizer=l2(reg_const))
\end{lstlisting}

\section{Python code for the Bayesian neural networks in table \ref{tab:credit_BNN_performance}} \label{app:Credit_BNN}
The Bayesian neural network with hierarchical model is implemented by the following code
\begin{lstlisting}
# # ----------------------------- IMPORTS -----------------
import warnings
from sklearn.metrics import accuracy_score, log_loss
import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
import sys
import time
import numpy as np
from sklearn.model_selection import train_test_split
import pandas as pd
import pymc3 as pm
import theano
import arviz as az
from arviz.utils import Numba
import theano.tensor as tt
from scipy.stats import mode
Numba.disable_numba()
Numba.numba_flag
floatX = theano.config.floatX
sns.set_style("white")

# # ----------------------------- Print versions -----------------
print("Running on Python version %s" % sys.version)
print(f"Running on PyMC3 version{pm.__version__}")
print("Running on Theano version %s" % theano.__version__)
print("Running on Arviz version %s" % az.__version__)
print("Running on Numpy version %s" % np.__version__)

# Ignore warnings - NUTS provide many runtimeWarning
warnings.filterwarnings("ignore", category=RuntimeWarning)

tf.random.set_seed(42)

# # ----------------------------- Loading credit data ---
credit_data = pd.read_csv("Python_code/data/UCI_Credit_Card.csv",
                          encoding="utf-8", index_col=0, delimiter=",")
credit_data.head()
# Data to numpy
data = np.array(credit_data)
# seperating labels from features
data_X = data[:, 0:23]
data_y = data[:, 23]

# # ----------------------------- Subsamling credit data --------
X_train, X_test, y_train, y_test = train_test_split(
    data_X, data_y, test_size=0.30, random_state=3030)

N = 300
N_test = 100
X_train = X_train[0:N, :]
y_train = y_train[0:N]
X_test = X_test[0:N_test, :]
y_test = y_test[0:N_test]

# pad Xs with 1's to add bias
ones_train = np.ones(X_train.shape[0])
ones_test = np.ones(X_test.shape[0])
X_train = np.insert(X_train, 0, ones_train, axis=1)
X_test = np.insert(X_test, 0, ones_test, axis=1)

# # ----------------------------- Implementing a BNN function -----

def construct_bnn(ann_input, ann_output, n_hidden):

    with pm.Model() as bayesian_neural_network:
        ann_input = pm.Data("ann_input", X_train)
        ann_output = pm.Data("ann_output", y_train)

        # prior on hyper parameters for weight 1
        mu1 = pm.Cauchy('mu1', shape=(
            X_train.shape[1], n_hidden), alpha=0, beta=1)
        sigma1 = pm.HalfNormal('sigma1', shape=(
            X_train.shape[1], n_hidden), sigma=1)

        # Weights from input to hidden layer
        weights_in_1 = pm.Normal(
            "w_in_1", mu1, sigma1, shape=(X_train.shape[1], n_hidden))

        # prior on hyper parameters for weight_out
        mu_out = pm.Cauchy('mu_out', shape=(n_hidden, 1), alpha=0, beta=1)
        sigma_out = pm.HalfNormal('sigma_out', shape=(n_hidden, 1), sigma=1)
        # Weights from hidden layer to output
        weights_1_out = pm.Normal(
            "weights_out", mu_out, sigma=sigma_out, shape=(n_hidden, 1))

        # Build neural-network using tanh activation function
        act_1 = pm.math.tanh(pm.math.dot(ann_input, weights_in_1))

        output = pm.Deterministic(
            "output", pm.math.sigmoid(tt.dot(act_1, weights_1_out)))

        # Binary classification -> Bernoulli likelihood
        out = pm.Bernoulli(
            "out",
            output,
            observed=ann_output,
            total_size=y_train.shape[0],
        )

    return bayesian_neural_network

# # ----------------------------- Sampling from posterior --------
tic = time.time()  # for timing
bayesian_neural_network_NUTS = construct_bnn(X_train, y_train, n_hidden=10)

# Sample from the posterior using the NUTS samplper
draws = 1500
tune = 10**3
chains = 3
target_accept = .9
with bayesian_neural_network_NUTS:
    trace = pm.sample(draws=draws, tune=tune, chains=chains,
                      target_accept=target_accept)

y_train_pred = (trace["output"]).mean(axis=0)

# Replace shared variables with testing set
pm.set_data(new_data={"ann_input": X_test, "ann_output": y_test},
            model=bayesian_neural_network_NUTS)

ppc2 = pm.sample_posterior_predictive(
    trace, var_names=["output"], model=bayesian_neural_network_NUTS)
y_test_pred = (ppc2["output"]).mean(axis=0)

# y_test_pred = np.append(y_test_pred,1-y_test_pred,axis=1)

# end time
toc = time.time()
print(f"Running MCMC completed in {toc - tic:} seconds")

# Printing the performance measures
print('Cross-entropy loss on train data = {}'.format(log_loss(y_train, y_train_pred)))
print('Cross-entropy loss on test data = {}'.format(log_loss(y_test, y_test_pred)))

\end{lstlisting}
The Bayesian neural network with one hidden layer is implemented by the following code
\begin{lstlisting}
# # ----------------------------- IMPORTS ------------------
import warnings
from sklearn.metrics import accuracy_score, log_loss
import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
import sys
import time
import numpy as np
from sklearn.model_selection import train_test_split
import pandas as pd
import pymc3 as pm
import theano
import arviz as az
from arviz.utils import Numba
import theano.tensor as tt
from scipy.stats import mode
Numba.disable_numba()
Numba.numba_flag
floatX = theano.config.floatX
sns.set_style("white")

# # ----------------------------- Print versions -----------------
print("Running on Python version %s" % sys.version)
print(f"Running on PyMC3 version{pm.__version__}")
print("Running on Theano version %s" % theano.__version__)
print("Running on Arviz version %s" % az.__version__)
print("Running on Numpy version %s" % np.__version__)

# Ignore warnings - NUTS provide many runtimeWarning
warnings.filterwarnings("ignore", category=RuntimeWarning)

tf.random.set_seed(42)

# # ----------------------------- Loading credit data ------
credit_data = pd.read_csv("Python_code/data/UCI_Credit_Card.csv",
                          encoding="utf-8", index_col=0, delimiter=",")
credit_data.head()
# Data to numpy
data = np.array(credit_data)
# seperating labels from features
data_X = data[:, 0:23]
data_y = data[:, 23]

# # ----------------------------- Subsamling credit data -----
X_train, X_test, y_train, y_test = train_test_split(
    data_X, data_y, test_size=0.30, random_state=3030)

N = 300
N_test = 100
X_train = X_train[0:N, :]
y_train = y_train[0:N]
X_test = X_test[0:N_test, :]
y_test = y_test[0:N_test]

# pad Xs with 1's to add bias
ones_train = np.ones(X_train.shape[0])
ones_test = np.ones(X_test.shape[0])
X_train = np.insert(X_train, 0, ones_train, axis=1)
X_test = np.insert(X_test, 0, ones_test, axis=1)

# # ----------------------------- Implementing a BNN function -----
def construct_bnn(ann_input, ann_output, n_hidden, prior_std):

    with pm.Model() as bayesian_neural_network:
        ann_input = pm.Data("ann_input", X_train)
        ann_output = pm.Data("ann_output", y_train)

        # Weights from input to hidden layer
        weights_in_1 = pm.Normal(
            "w_in_1", 0, sigma=prior_std, shape=(X_train.shape[1], n_hidden))

        # Weights from hidden layer to output
        weights_1_out = pm.Normal(
            "weights_out", 0, sigma=prior_std, shape=(n_hidden, 1))

        # Build neural-network using tanh activation function
        act_1 = pm.math.tanh(pm.math.dot(ann_input, weights_in_1))
        output = pm.Deterministic(
            "output", pm.math.sigmoid(tt.dot(act_1, weights_1_out)))

        # Binary classification -> Bernoulli likelihood
        out = pm.Bernoulli(
            "out",
            output,
            observed=ann_output,
            total_size=y_train.shape[0],  # IMPORTANT for minibatches
        )

    return bayesian_neural_network

# # ------------------ Sampling from posterior -----
tic = time.time()  # for timing
bayesian_neural_network_NUTS = construct_bnn(
    X_train, y_train, n_hidden=10, prior_std=1)

# Sample from the posterior using the NUTS samplper
draws = 1500
tune = 10**3
chains = 3
target_accept = .9
with bayesian_neural_network_NUTS:
    trace = pm.sample(draws=draws, tune=tune, chains=chains,
                      target_accept=target_accept)

y_train_pred = (trace["output"]).mean(axis=0)

# Replace shared variables with testing set
pm.set_data(new_data={"ann_input": X_test, "ann_output": y_test},
            model=bayesian_neural_network_NUTS)

ppc2 = pm.sample_posterior_predictive(
    trace, var_names=["output"], model=bayesian_neural_network_NUTS)
y_test_pred = (ppc2["output"]).mean(axis=0)

# end time
toc = time.time()
print(f"Running MCMC completed in {toc - tic:} seconds")

# Printing the performance measures
print('Cross-entropy loss on train data = {}'.format(log_loss(y_train, y_train_pred)))
print('Cross-entropy loss on test data = {}'.format(log_loss(y_test, y_test_pred)))

# Vizualize uncertainty
# Define examples for which you want to examine the posterior predictive:
example_vec = np.array([5, 11, 25, 88])
for example in example_vec:
    plt_hist_array = np.array(ppc2['output'])
    plt.hist(plt_hist_array[:, example], density=1,
             color="lightsteelblue", bins=30)
    plt.xlabel(f"Predicted probability for example {example}", fontsize=13)
    plt.ylabel("Density", fontsize=13)
    plt.savefig(f'Python_code/Credit_BNN_1hidden_postpred_{example}.pdf')
    plt.show()

\end{lstlisting}
where the network with no hidden layers is implemented by replacing the lines
\begin{lstlisting}
 # Weights from input to hidden layer
        weights_in_1 = pm.Normal(
            "w_in_1", 0, sigma=prior_std, shape=(X_train.shape[1], n_hidden))

# Weights from hidden layer to output
    weights_1_out = pm.Normal(
            "weights_out", 0, sigma=prior_std, shape=(n_hidden, 1))

# Build neural-network using tanh activation function
    act_1 = pm.math.tanh(pm.math.dot(ann_input, weights_in_1))
    output = pm.Deterministic(
            "output", pm.math.sigmoid(tt.dot(act_1, weights_1_out)))
\end{lstlisting}
with
\begin{lstlisting}
 # Weights from hidden layer to output
        weights_in_out = pm.Normal("weights_out", 0, sigma=prior_std, shape=(X_train.shape[1],1))

        # Build neural-network using tanh activation function
        output = pm.Deterministic("output", pm.math.sigmoid(tt.dot(ann_input, weights_in_out)))
\end{lstlisting}

\end{appendices}


\subsection{Generating functions}

\subsubsection{Definition}

A series can be described as:

\(\sum_{i=0}^{\infty }s_i x^i\)

If we know the function equal to this series, we can identify the \(i\)th number.


%!TEX root = ../thesis.tex
% ******************************* Thesis Appendix A ****************************
\chapter{Spherical Harmonics on Hyperspheres}
\label{appendix:spherical-harmonics}

Spherical harmonics are a special set of functions defined on the hypersphere and play a central role in harmonic analysis and approximation theory \citep{wendland2005}. They originate from solving Laplace's equation, and form a complete set of orthogonal functions. Any sufficiently regular function defined on the sphere can be written as a sum of these spherical harmonics, similar to the Fourier series with sines and cosines. Spherical harmonics are defined in arbitrary dimensions \citep{frye2014,dai2013}, but lack explicit formulations and practical implementations in dimensions larger than three. %This is mainly due to the complexity of solving Laplace's equation in higher dimensions.

In this section, we propose a novel algorithm to construct spherical harmonics in $d$ dimensions. The algorithm is based on the existence of a fundamental system of points on the hypersphere, which we select in a greedy fashion through optimisation. This result in spherical harmonics that are a linear combination of zonal functions and form an ortho-normal basis on $\dsphere$. The algorithm lends itself well for implementation in Python and TensorFlow, which we provide at: \url{https://github.com/vdutor/SphericalHarmonics}. The code is accompanied by a series of tests that show that the properties of spherical harmonics, as detailed below, hold. Before outlining the algorithm, we briefly define and cover the important properties of spherical harmonics in $\Reals^d$. We refer the interested reader to \citet{dai2013,frye2014} for a comprehensive overview.
% exist bases of spherical harmonics consisting of entirely zonal harmonics

We adopt the usual $L_2$ inner product for functions $f: \dsphere \rightarrow \Reals$ and $g: \dsphere \rightarrow \Reals$ restricted to the sphere
\begin{equation}
     \langle f, g\rangle_{L_{2}(\dsphere)} = \frac{1}{\darea} \int_{\dsphere} f(x)\,g(x) \, \calcd{\omega},
\end{equation}
where $\calcd{\omega(x)}$ is the surface area measure such that $\darea$ denotes the surface area of $\dsphere$
\begin{equation}
\label{eq:surface}
    \darea = \int_{\dsphere} \calcd{\omega(x)} = \frac{2 \pi ^ {d/2}}{\Gamma(d/2)}.
\end{equation}

Throughout this section we use the following notation and definitions. For $x = (x_1, \ldots, x_d) \in \Reals^d$ and $\alpha = (\alpha_1, \ldots, \alpha_d) \in \Naturals^d$, a monomial $x^\alpha$ is a product $x^\alpha = x_1^{\alpha_1} \ldots x_d^{\alpha_d}$, which has degree $|\alpha| = \alpha_1 + \ldots \alpha_d$. A real homogeneous polynomial $P(x)$ of degree $n$ is a linear combination of monomials of degree $n$ with real coefficients, that is $P(x) = \sum_{|\alpha| = n} c_{\alpha} x^{\alpha}$, with $c_\alpha \in \Reals$. We denote $\mathcal{P}_n^d$ as the space of real homogeneous polynomials of degree $n$, and can show that, counting the cardinality of the set $\{\alpha \in \Naturals^d: |\alpha| = n\}$, that $\dim(\mathcal{P}_n^d) = \binom{n + d -1}{n}$. A function $f:\Reals^d \rightarrow \Reals$ is said to be \emph{harmonic} if $\Delta f = 0$, where $\Delta = \partial_{x_1}^2 + \ldots + \partial_{x_d}^2$ and $\partial_{x_i}$ the partial derivate w.r.t. the $i$-th variable.

\begin{definition}
    The spherical harmonics of degree $n$ of $d$ variables, denoted by $\mathcal{H}_{n}^d$, is the linear space of harmonic and homogeneous in degree $n$ polynomials on $\dsphere$, that is
    \begin{equation}
        \mathcal{H}_{n}^d = \{p \in \mathcal{P}_n^d: \Delta p = 0\ \text{and}\ p: \dsphere \rightarrow \Reals \}.
    \end{equation}
The dimensionality of $ \mathcal{H}_{n}^d$ is given by
\begin{equation}
\label{eq:numharmonics}
\dim(\mathcal{H}_{n}^d) = \frac{2 n + d - 2}{n} \binom{n + d - 3}{d - 1} := \dnumharmonicsforlevel.
\end{equation}
\end{definition}

The space $\mathcal{H}_{n}^d$ has an orthonormal basis consisting of $\dnumharmonicsforlevel$ functions, denoted by $\{\phi_{n,j}\}_{j=1}^{\dnumharmonicsforlevel}$. The basis satisfy the following properties
\begin{equation}
    \mathcal{H}_{n}^d  = \textrm{span}\left(\phi_{n,1}, \ldots, \phi_{n, \dnumharmonicsforlevel}\right),\quad\text{and}\quad\left\langle \phi_{n, j}, \phi_{n', j'}\right\rangle_{L_2(\dsphere)} = \delta_{n n'} \delta_{j j'}.
\end{equation}

From the completeness and orthonormality of the spherical harmonic basis $\{\phi_{n,j}\}_{n=0,j=1}^{\infty,\dnumharmonicsforlevel}$, it can be shown that they also form a basis of square-integrable functions \citep{frye2014}. This means that we can decompose a function $f: \dsphere \rightarrow \Reals$ as
\begin{equation}
    f = \sum_{n=0}^{\infty} \sum_{j=1}^{\dnumharmonicsforlevel} \widehat{f}_{n, j} \phi_{n, j},\quad\text{with}\quad\widehat{f}_{n, j} = \langle f, \phi_{n, j} \rangle_{L_2(\dsphere)},
\end{equation}
which can be seen as the spherical analogue of the Fourier decomposition of periodic functions onto a basis of sines and cosines.

Subsequently, we will coin the set $\{\phi_{n,j}\}$ as the spherical harmonics. They are indexed by $n$ and $j$, where $n=0,1,2,\ldots$ denotes the degree (or level) and $j=1,\cdots,\dnumharmonicsforlevel$ denotes the orientation of the spherical harmonic. We are interested in finding $\{\phi_{n,j}\}$ in arbitrary dimension. For $d=2$, is solving Laplace's equation ($\Delta p = 0$) directly relatively straightforward. Doing so reveals that $N^{2}_0 = 1$ with $\phi_{0, 1} = 1$ and $N^{2}_n = 2$ for all $n > 0$ with $\phi_{n, 1}(\theta) = \sqrt{2} \cos(n \theta)$ and $\phi_{n, 2}(\theta) = \sqrt{2} \sin(n \theta)$. This shows that on the unit circle $\sphere^1$, the spherical harmonics correspond to the Fourier basis. For $d=3$, we can also directly solve Laplace's diffential equation to find $\dnumharmonicsforlevel = 2n + 1$ and a closed form solution for $\{\phi_{n,j}\}$. However, for $d > 3$, explicit formulations for the spherical harmonics become very rare. To the best of our knowledge, the only explicit formulation we could find is in \citet[Theorem~5.1]{dai2013}, which consists of a product over polynomials. This makes the implementation cumbersome and numerically unstable, and only practically useful up to 10 dimensions \citep{Dutordoir2020spherical}. However, making use of the following two theorems, we can derive another formulation for the basis of spherical harmonics as a sum of polynomials, rather than a product. The connection between spherical harmonics and orthogonal polynomials becomes clear in the next theorem.
\begin{theorem}[Addition]
    \label{theorem:addition}
    Let $\{\phi_{n,j}\}_{j=1}^{\dnumharmonicsforlevel}$ be an orthonormal basis for the spherical harmonics of degree $n$ and $x,x' \in \dsphere$. Then the Gegenbauer polynomial $C_n^{(\alpha)}: [-1, 1] \rightarrow \Reals$ of degree $n$ can be written as
\begin{equation}
    \sum_{j=1}^{\dnumharmonicsforlevel} \phi_{n, j}(x) \phi _{n, j}(x') = \frac{n + \alpha}{\alpha}\,
    C_n^{(\alpha)}(x\transpose x')\quad\text{with}\quad \alpha = \frac{d-2}{2}.
\end{equation}
\end{theorem}
As a result of the relation between the Gegenbauer polynomial and the spherical harmonics, are the Gegenbauer polynomial sometimes referred to as ultraspherical polynomials. For $d=2$, Theorem~1 recovers the addition formula of the cosine function, as indeed $\cos(\theta) \cos(\theta') + \sin(\theta) \sin(\theta') = \cos(\theta - \theta')$ and $C_n^{(0)}(t) = \cos(n \arccos(t))$. The Gegenbauer polynomials with $\alpha=0$ are better known as the Chebyshev polynomials. Another connection between spherical harmonics and Gegenbauer polynomials is given by the Funk-Hecke theorem and applies to zonal functions. A zonal function on $\dsphere$ is a function that is rotationally invariant w.r.t. to a point on the sphere, $\eta \in \dsphere$. This means that the function only depends on the inner product $\eta\transpose x$, or equivalently, on the geodestic distance between $\eta$ and $x$.

\begin{theorem}[Funk-Hecke]
    \label{appendix:theorem:funk}
    Let $f$ be an integrable function such that $\int_{-1}^1 \| f(t)\| (1 - t^2)^{(d-3)/2} \calcd{t}$ is finite and $d \ge 2$. Then for every $\phi_{n,j}$  and $\eta \in \dsphere$
    \begin{equation}
        \frac{1}{\darea} \int_{\dsphere} f(\eta\transpose x)\,\phi_{n, j}(x)\, \calcd{\omega(x)} = {\lambda}_{n}\,\phi_{n,j}(\eta),
    \end{equation}
    where ${\lambda}_{n}$ is a constant defined by
    \begin{equation}
        \lambda_{n}  =
        % \frac{\Omega_{d-2}}{\Omega_{d-1}}
        \frac{\omega_{d}}{C_n^{(\alpha)}(1)} \int_{-1}^1 f(t)\,C_n^{(\alpha)}(t)\,(1 - t^2)^{\frac{d-3}{2}} \calcd{t},
    \end{equation}
    with $\alpha = \frac{d-2}{2}$ and $\omega_d = \frac{\Omega_{d-2}}{\Omega_{d-1}}$.
\end{theorem}

\section{Algorithm: Zonal Spherical Harmonics}
\label{sec:zonal-spherical-harmonics}

From the Funk-Hecke and the Addition theorem, it is clear that there is a strong connection between spherical harmonics and Gegenbauer polynomials. The next theorem develops this connection further as it states that a basis for spherical harmonics can be written as zonal Gegenbauer polynomials. % , that is $\phi_{n,j}(x) = \sum_i \beta_{j,i} C_n^{(\alpha)}(\eta_i\transpose x)$. In what comes next we show how to select the weights $\beta_{j,i}$ and zonal directions $\eta_i \in \dsphere$.

\begin{theorem}
    If $\{\eta_1, \ldots, \eta_{\dnumharmonicsforlevel} \} \in \dsphere$ is a fundamental system of points on the sphere, then $\{C_n^{(\alpha)}(\eta_i \cdot)\}_{i=1}^{\dnumharmonicsforlevel}$ is a basis for $\mathcal{H}_n^d$. A collection of points $\{\eta_1, \ldots, \eta_M \} \in \dsphere$ is called a fundamental system of degree $n$ consisting of $M$ points on the sphere if
    \begin{equation}
        \label{eq:fundamental-system}
        \textrm{det}\
        \begin{bmatrix}
            C_n^{(\alpha)}(1) & \ldots & C_n^{(\alpha)}(\eta_1\transpose\eta_M) \\
            \vdots & & \vdots \\
            C_n^{(\alpha)}(\eta_M\transpose\eta_1) & \ldots & C_n^{(\alpha)}(1)
        \end{bmatrix} > 0.
    \end{equation}
\end{theorem}
Finding a basis for $\mathcal{H}_n^d$ is thus equivalent to finding a set of $\dnumharmonicsforlevel$ points that satisfy \cref{eq:fundamental-system}. Crucially, \citet[Lemma~3]{dai2013} show that there always exists a fundamental system of degree $n$ and $\dnumharmonicsforlevel$ points.

Following the theorem, if we wish to construct $\{\phi_{n,j}\}_{n,j}$, an \emph{ortho-normal} basis for the spherical harmonics, we firstly need a fundamental system of points. Secondly, while $\{C_n^{(\alpha)}(\eta_i \cdot)\}_{i=1}^{\dnumharmonicsforlevel}$ forms a basis for $\mathcal{H}_n^d$, the basis is not orthonormal. We will thus have to apply a Gram-Schmidt process for ortho-normalising the basis. We detail both steps in the next paragraphs.

\paragraph{Construction of a fundamental system of points}
We propose to build a fundamental system of points in a greedy fashion by repeatedly adding a point on the sphere that maximises the determinant as given in \cref{eq:fundamental-system}. Therefore, let $\veta = \{\eta_1, \ldots, \eta_M\}$ contain the M points that are already in the fundamental system and define the following block-matrix of size $(M+1) \times (M+1)$ as
\begin{equation}
    \renewcommand\arraystretch{1.3}
    \MM(\veta, \eta_{*}) =
    \left[
        \begin{array}{c|c}
          C_n^{\alpha}(\veta\veta\transpose) \in \Reals^{M \times M} & C_n^{(\alpha)}(\veta\eta_{*}\transpose) \in \Reals^{M \times 1} \\
          \hline
          C_n^{(\alpha)}(\veta\eta_{*}\transpose)\transpose \in \Reals^{1 \times M} & C_n^{(\alpha)}(1) \in \Reals
        \end{array}
    \right],
\end{equation}
where $C_n^{(\alpha)}(\veta\veta\transpose)$ corresponds to element wise evaluating the Gegenbauer polynomial $C_n^{(\alpha)}: [-1, 1] \rightarrow \Reals$ for each element of $\veta \veta\transpose \in \Reals^{M \times M}$. A new point $\eta$ is added to the fundamental system if it maximises the determinant
\begin{equation}
    \eta = \argmax_{\eta_* \in \dsphere}\ \textrm{det}\left(\MM(\veta, \eta_{*})\right),
\end{equation}
in order to satisfy the condition in \cref{eq:fundamental-system}.
% \begin{equation}
%     \textrm{det}(\MC_n^{(\alpha)}(\veta, \eta_{*})) = \textrm{det}(C_n^{(\alpha)}(\veta\veta\transpose))\left(C_n^{(\alpha)}(1) - C_n^{(\alpha)}(\eta_{*}\veta\transpose) \left[C_n^{(\alpha)}(\veta\veta\transpose)\right]^{-1} C_n^{(\alpha)}(\veta\eta_{*}\transpose) \right).
% \end{equation}
Computing the determinant can be done efficiently using Schur' complement. Furthermore, as $\veta$ and $C_n^{(\alpha)}(1.0)$ are constants the optimisation problem boils down to
\begin{equation}
    \label{eq:optimisation-determinant}
    \eta = \argmin_{\eta_* \in \Reals^d}\ C_n^{(\alpha)}(\frac{\eta_{*}}{\norm{\eta_*}}\veta\transpose) \left[C_n^{(\alpha)}(\veta\veta\transpose)\right]^{-1} C_n^{(\alpha)}(\veta\frac{\eta_{*}\transpose}{\norm{\eta_*}}).
\end{equation}
The complete algorithm is given in \cref{alg:fundamental-system}.

\begin{algorithm}[H]
    \SetAlgoLined
    \DontPrintSemicolon
    \KwInput{Degree $n$ and dimension $d$}
    \KwResult{Fundamental system $\veta = \{\eta_0, \ldots \eta_{\dnumharmonicsforlevel}\}$}
    $\eta_1 = (0,0,\ldots,1)$ \tcp*{d-dimensional vector pointing north}
    $\veta = \{\eta_1\}$,
    $\alpha = \frac{d-2}{2}$,
    $i = 2$\;
     \While{$i \le \dnumharmonicsforlevel$}{
        $\eta = \argmax_{\eta_* \in \Reals^d}\ \textrm{det}(\MM(\veta, \frac{\eta_{*}}{\norm{\eta_*}}))$
        \tcp*{Using a local optimisation method (e.g., BFGS) and \cref{eq:optimisation-determinant}}
        Add $\eta$ to $\veta$\;
        i = i + 1\;
      }
     \caption{Construction of fundamental system\label{alg:fundamental-system}}
\end{algorithm}

\paragraph{Ortho-normalisation}

\begin{theorem}
   Let $\veta = \{\eta_1, \ldots, \eta_{\dnumharmonicsforlevel}\}$ be a fundamental system of degree $n$ consisting of $\dnumharmonicsforlevel$ points, and $\ML$ the inverse Cholesky factor of $C_n^{(\alpha)}(\veta\veta\transpose)$. Then for $j=1,\ldots,\dnumharmonicsforlevel$ and
    \begin{equation}
    \phi_{n,j}(x) = \sum_{i=1}^{\dnumharmonicsforlevel} \ML_{j,i}\,C_n^{(\alpha)}(\eta_i\transpose x)
    \end{equation}
    is $\{\phi_{n,j}\}$ an ortho-normal basis for the spherical harmonics $\mathcal{H}_n^d$.
\end{theorem}

The proof follows from $\langle C_n^{(\alpha)}(\eta_i\transpose \cdot), C_n^{(\alpha)}(\eta_j\transpose \cdot) \rangle_{L_2(\dsphere)} = C_n^{(\alpha)}(\eta_i\transpose\eta_j)$ as a result of the Funk-Hecke theorem.

% Let $\MM = C_n^{(\alpha)}(\veta\veta\transpose)$ and $\ML \ML\transpose = \MM$, which exists because the set $\veta$ satisfies \cref{eq:fundamental-system}.
% \begin{equation}
% \phi_{n,j}(x) = \sum_i [\ML\inv]_{j,i} C_n^{(\alpha)}(\eta_i\transpose x)
% \end{equation}

\begin{figure}
    \centering
    \includegraphics[width=.6\linewidth]{Appendix1/harmonics}
    \caption{Spherical Harmonics}
    \label{fig:appendix:harmonics}
\end{figure}


\subsubsection{Circular Cylinders}
\noindent
A circular cylinder is what we usually think of as a cylinder.
It is a circle extruded into 3D space.
One of its forms is
\begin{equation*}
	x^2 + y^2 = R^2,
\end{equation*}
where $R$ is the cylinder's radius.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.33\textwidth]{./differentialMultivariableCalculus/cylinder.png}
	\caption{A circular cylinder}
\end{figure}