\subsection{Problem Statement}

The best way to understand how neural networks work is to build one yourself from scratch. The understanding becomes even more comprehensive if there is a particular problem that can be solved using NNs. Therefore let's start our work by taking a look at Figure \ref{fig:digit-classification}.

\begin{figure}[H]
\centering
    \includegraphics[scale=0.35]{pics/problem.png}
    \caption{\label{fig:digit-classification} Digit Classification Problem}
\end{figure}

There are handwritten numbers that you want computer to correctly classify. This would be an easy task for a person but at least for a long period of time was an extremely complicated one for a machine.

Even though the computer is faster than the human brain in numeric computations, the brain outperforms the computer in some other tasks. Many of those tasks are related to the human ability for sentience (which is a concept different from intelligence). The trick is to find a way, so that the computer could apply its numeric computation skills to solve these later tasks (at least to some degree).

The first step would be to limit the scope of the task. In our particular case the broad task of image recognition will be addressed as a classification problem - a task of giving an object a label from a given set of labels.

As we will see during the process of building our own NN, its output is based almost exclusively on application of linear algebra methods. Despite the name (which is sometimes related to the fear of artificial intelligence), neural networks in fact are much more related to statistical methods (like regression analysis or curve fitting) than to the way human brain works (\cite{reid2014}).

NNs are inspired by human brain only to certain extent. For instance the main element that makes them similar is a multilayer net structure of simple elements that are connected in some way, receiving and transmitting information. But the structure of the human brain is much more complicated, besides it is self-organizing and adaptive in contrast to the fixed manually designed architecture of a NN. Hence, there is a good reason to stop being afraid of neural networks and instead to create one ourselves.

% \begin{figure}[H]
%     \includegraphics[width=\linewidth]{pics/neurons_net3.png}
%     \caption{\label{fig:real-neurons} Brain Neurons. Source: pixabay.com}
% \end{figure}

\subsection{Schematic Representation}
A complex multilayer structure that all neural networks have in common in a simplified way can be depicted using the Figure \ref{fig:nn-schema}.

\begin{figure}[H]
    \includegraphics[width=\linewidth]{pics/neural_network1.jpg}
    \caption{\label{fig:nn-schema} Neural Network Schema}
\end{figure}

All we need in order to implement such a structure is base Python and numpy, a library for numerical computation, which we will be using to do linear algebra.

First let's determine the elements of a neural network depicted on the Figure \ref{fig:nn-schema}: nodes, layers, weights across nodes and activation functions.

\begin{itemize}
    \item \textbf{Nodes}. A node is basically a point where data points are received, processed and then transferred to the node. A node could be either an endpoint or a redistribution point or even both when iterations are done through the learning algorithm. The number of nodes to use is optional.

    \item \textbf{Layers}. A layer consists of one or several nodes. The initial layer in the network is called the input layer and it is the entry point through which the data is fed into the neural net. The middle layers are called hidden layer because the computation results of them are not directly visible to someone interacting with the neural net. In the hidden layers, which can range from one to thousands, the features are transformed and most of the structure (both linear and nonlinear) is captured. Finally, there is the final layer, from which results are output. The nodes in each layer are fully interconnected to the ones in the next and the previous layers.

In our case we have a structure with 3 layers: input, output and one hidden layer. The number of nodes in the input  (i\_n), hidden (h\_n) and output (o\_n) layers are 3, 5 and 2 respectively. In Python, such a structure can be represented in the following way:

\begin{lstlisting}[language=Python]
    # Load the package to work with numbers
    import numpy as np

    # Determine the structure of the NN
    i_n = 3
    h_n = 5
    o_n = 2
\end{lstlisting}

    \item \textbf{Weights.} In order to transfer an input data point to the next layer, a predetermined number (called weight) is stored in each connection from the sender node to the receiver node. Each weight accounts for the impact between the interconnected nodes. Initially, we assign weights between nodes in neighboring layers randomly. This is needed only for the sake of initializing the structure. Later these weights will be changed in order to solve our classification problem. The weight updating will be better described in the following sections. Neural nets will have n-1 matrices of weights, where n is the number of layers in the NN. You can imagine these weight matrices sitting between two layers representing the strength of the connection between every single node of neighboring layers. Thus, each of these matrices will be of size f x p, where p is the number of nodes in the preceding layer and f is the number of nodes in the following layer.
    This becomes more clear once you check the code below that creates 2 matrices of weights: the 5x5 matrix of weights between input and hidden layers (w\_i\_h) and the 2x5 matrix of weights between hidden and output layers (w\_h\_o). Such a dimensions of matrices are necessary in order to accomplish matrix and vector multiplications that are done in the following stages.

\begin{lstlisting}[language=Python]
    # Randomly define the weights between the layers.
    w_i_h = np.random.rand(h_n, i_n) # create an array of the given shape and populate it with random values.
    w_h_o = np.random.rand(o_n, h_n)

    # Show matrices of randomly assigned weights.
    w_i_h
    # w_h_o # uncomment this line in order to see the values for w_h_o.
    # Use Cmd + / in MacOS and CTRL + / in MS Windows as a shortcut to comment/uncomment lines.
\end{lstlisting}

\begin{lstlisting}
    array([[0.63964736, 0.97236245, 0.83944375],
    [0.31439566, 0.54254369, 0.0456713 ],
    [0.93759599, 0.71292359, 0.11961199],
    [0.90587079, 0.0855728 , 0.55046849],
    [0.89559465, 0.47349711, 0.42168825]])
\end{lstlisting}

\item \textbf{Activation Function.} The remaining element of the NN's structure is an activation function - a function which transforms an input data point that it receives from the previous nodes to an output value which will be the input for the nodes in the next layer. The activation function plays an important role in the efficiency of the neural network as it accounts for non-linearity of data.
It is to certain extent inspired by the concept of firing, which means that neurons fire or transmit information further only if the input surpasses certain threshold. The simplest activation function can be represented by a step function as on the Figure \ref{fig:act-fct}.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.35]{pics/step_function.png}
    \caption{\label{fig:act-fct} Activation Functions. Source: Research Gate}
\end{figure}

In our NN, we will use a slightly more elaborate activation function, the sigmoid function (logistic), which allows for more efficient use of the input data. A description of various activation functions, their benefits and disadvantages is given in sections below.

\begin{lstlisting}[language=Python]
    # Determine activation function.
    def sigmoid(x):
        # np.exp() calculates the exponential
        # of all elements in the input array.
        return 1 / (1 + np.exp(-x))

    # Draw activation function.
    import matplotlib.pyplot as plt

    # return 100 evenly spaced numbers over an interval from -10 to 10.
    x = np.linspace(-10, 10, 100)
    # plot sigmoid function for sampled values.
    plt.plot(x, sigmoid(x))
    plt.show()
\end{lstlisting}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.5]{pics/sigmoid.png}
    \caption{\label{fig:sigmoid-fct} Sigmoid Function}
\end{figure}

\end{itemize}

\subsection{Data Inspection}

By now we have collected all the elements of the NN. Can we use this structure in order to solve the classification problem stated in the beginning? In order to answer this question we need first to get a better understanding of the data at our disposal.

We are trying to check whether NN is able to solve the classification problem using a collection of 70 000 handwritten numbers. Each of this handwritten number is represented as 28x28 image.

The original source of the data is THE MNIST DATABASE. In the links below, you can find:

\begin{itemize}
    \item http://yann.lecun.com/exdb/mnist/: A detailed description of the dataset and a summary of the performance results achieved by various classification algorithms.
    \item https://pjreddie.com/projects/mnist-in-csv/: the data we will use. Here the original images are saved in CSV, which allows to work with them directly.
\end{itemize}

For the purposes of demonstration below we use a smaller dataset (100 images), which will be expanded at a later stage.

\begin{lstlisting}[language=Python]
    # Load the data.
    # "r" stands for "read only" mode.
    raw_data = open("data/mnist_train_100.csv", 'r')
    # read all the lines of a file in a list.
    data = raw_data.readlines()
    # remove temporal file from the environment in order to save memory.
    raw_data.close()

    # Inspect the data - check the number of observations.
    # length of the object.
    len(data)
\end{lstlisting}

\begin{lstlisting}
    100
\end{lstlisting}

\begin{lstlisting}[language=Python]
    # Inspect a particular observation of the data.
    # show observation number 0 from the list
    # (remember that in Python numbering starts from 0).
    data[0]
\end{lstlisting}

\begin{lstlisting}[language=Python]
'3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,254,254,254,254,255,209,
126,67,32,0,0,0,0,0,0,0,0,0,0,0,
...
254,233,139,57,232,254,243,166,28,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,'
\end{lstlisting}

From the result above, we see that 1) a particular observation looks like a string of 785 elements (label of the image + 784 elements for each pixels of a 28x28 image), 2) each element representing a pixel is a number from 0 to 255 (from white to black color) and 3) the first element in the line is the label of the image and therefore is a number from 0 to 9.

Using `matplotlib`, we can also reconstruct the original image based on the data about each pixel in the string.

\begin{lstlisting}[language=Python]
    # Load the package to plot the data
    import matplotlib.pyplot as mpp

    # Plot the data
    observation = data[0].split(',') # break down observation number 0 (comma is used to identify each element).
    image = np.asfarray(observation[1:]).reshape((28,28)) # take all the elements starting from the element 1
    # (exclude element number 0, that corresponds to the label) and reshape them as an array with dimension 28 by 28.
    mpp.imshow(image, cmap='Blues', interpolation='None') # show the plot of this array using grey pallete.

    # Save an observation of the data as an input to work with.
    input = np.array(np.asfarray(observation[1:]), ndmin=2).T # save necessary elements in a vertical vector shape.
\end{lstlisting}

\begin{figure}[H]
    \centering
   \includegraphics[scale=0.5]{pics/3.png}
   \caption{\label{fig:number3} Plot of the handwritten number}
\end{figure}

    \subsection{Fitting the Structure of the NN to the Data}

After inspecting the data, we can conclude that the structure in Figure \ref{fig:nn-schema} with 3-5-2 nodes is probably not optimal and therefore should be updated in order to fit the data we have and peculiarities of the classification problem:

\begin{itemize}
    \item For each observation we have 784 elements as an input (label element is excluded). Accordingly, instead of 3 input nodes we should better have 784.
    \item Similarly, as we have 10 different options for the outcome (handwritten numbers are labeled from from 0 to 9) the number of output nodes should be 10 instead of 2.
    \item We also change the number of hidden nodes from 5 to 90. Such a number has been assigned based on some proportionality assumptions which will be checked later: 90 is 9 times higher than 10 and approximately 9 times smaller than 784.
\end{itemize}

As we have new structure of the NN we should reassign the weights - now the size of each weight matrix will increase as we have more nodes in each layer.

\begin{lstlisting}[language=Python]
    # Determine the new structure of the NN.
    i_n = 784
    h_n = 90
    o_n = 10

    # Determine the weights.
    w_i_h = np.random.rand(h_n, i_n)
    w_h_o = np.random.rand(o_n, h_n)
\end{lstlisting}

So far we have not used the first element of our observation - the label. It will be necessary to compare the predictions of the NN to the real state of the world and to train the NN to make correct predictions. The target should therefore have the same shape as the output layer of the NN, so that they could be comparable. We can represent the label as a vector of n binary (0 or 1) elements (n corresponds to the number of nodes in the output layer). There should be only one element equal to 1 and the position of this element should correspond to the index number of the label we want to predict.

\begin{lstlisting}[language=Python]
    # Create target array.
    target = np.array(np.zeros(o_n), ndmin=2).T
    # int() method returns an integer object
    # from any number or string.
    target[int(observation[0])] = 1

    # Inspect how the target looks like (remember that the label of observations is 5).
    target

    # Show the sizes of matrices of weights, input and target vectors.
    w_i_h.shape, input.shape, w_h_o.shape, target.shape
\end{lstlisting}

\begin{lstlisting}
((90, 784), (784, 1), (10, 90), (10, 1))
\end{lstlisting}

\subsection{Feedforwarding}

Once we have the structure of the NN updated for the specific task of classifying the numbers depicted on the images, we can run our network in order to get the first predictions that will be represented by a vector of 10 elements. This vector in its turn can be compared to the target.

To run the NN, i.e. to feed forward our input data in order to get some predictions, we should follow certain steps:

\begin{enumerate}
    \item Multiply an input vector by a matrix of weights that connects it with the next layer;
    \item Transform the result using activation function;
    \item Use the output obtained in the 2nd step as an input vector for the next layer.
\end{enumerate}

A sequence of this steps should be repeated n-1 times (where n corresponds to the number of layers). The output of the previous layer will always be the input vector for the next layer. In our case the procedure will happen twice.

In the Figures \ref{fig:multiplication} and \ref{fig:activation}, you can see the procedure necessary to obtain the output of the hidden layer. The result of matrix multiplication here is called Hidden\_Input. Result of the transformation of Hidden\_Input through activation function is called Hidden\_Output.

This output will be used as the input vector that should be multiplied by the next weight matrix and transformed through activation function in order to calculate the final output of the NN. If our NN would have more than one hidden layer, the procedure would be repeated more times.

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.4]{pics/multiplication.png}
    \caption{\label{fig:multiplication} How Weights Work}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[scale=0.3]{pics/activation.jpg}
    \caption{\label{fig:activation} How Activation Works}
\end{figure}

Below you can see the code implementation of all the steps for all layers of the NN.

\begin{lstlisting}[language=Python]
    # Calculate the output of hidden and output layers of our NN.
    # dot() performs matrix multiplication; "h_input" stands for "Hidden_Input".
    h_input = np.dot(w_i_h, input)
    # "Hidden_Output" - result after activation function.
    h_output = sigmoid(h_input)
    # "Output_Input" - input used for the next layer.
    o_input = np.dot(w_h_o, h_output)
    # "Output_Output" - final output of the NN.
    o_output = sigmoid(o_input)

    # Print output to the screen
    o_output
\end{lstlisting}

\begin{lstlisting}
    array([[ 1.],
    [ 1.],
    [ 1.],
    [ 1.],
    [ 1.],
    [ 1.],
    [ 1.],
    [ 1.],
    [ 1.],
    [ 1.]])
\end{lstlisting}

\subsection{Good Practices on Data Treatment}

Once we check the output of the NN and the results of each performed step, we can observe that already at the stage of the h\_output all the data converts to a vector in which all the values are equal to 1. Such a vector does not provide us with any helpful insight. Apparently, something is wrong with what we have done so far. There could be several reasons for the problem we face. Take a look again at our sigmoid function in Figure \ref{fig:sigmoid-fct}.

% \begin{figure}[H]
%     \includegraphics[width=\linewidth]{pics/sigmoid.png}
%     \caption{\label{fig:sigmoid-fct} Sigmoid Function}
% \end{figure}

As we can see the output of the sigmoid function will be almost identical once we feed a number bigger than 2. Similarly there is no significant difference between the outputs if numbers used are smaller than -2. Hence the application of sigmoid function to the original data leads to a loss of valuable information. The NN has problems learning something from the inputs, which are almost undifferentiable.

One solution is to transform the input we have. Ideally we should have our data in a range between 0 and 1. It is also desirable to avoid zeros as inputs, because the output of a zero input will always be zero, no matter how large the weights are, in which case the NN will not be able to use this input to learn.

We can perform a transformation of the original data as the one coded below.

\begin{lstlisting}[language=Python]
    # Good practice transformation of the input values:
    input = np.array((np.asfarray(observation[1:])/255.0*0.99) + 0.01, ndmin=2).T
    # Our values in our input vector are in the range from 0 to 255. Therefore we should divide input vector by 255,
    # multiply it by 0,99 and add 0,01 in order to get values in the range from 0,01 to 1.

    # Good practice transformation of the target value:
    target = np.array(np.zeros(o_n) + 0.01, ndmin=2).T
    target[int(observation[0])] = 0.99
\end{lstlisting}

Secondly, we can check our way to randomly assigned initial weights. Let's take a look once at the function we used to randomly assign them.

\begin{lstlisting}[language=Python]
    np.random.rand(3, 5)
\end{lstlisting}

\begin{lstlisting}
    array([[ 0.22583725,  0.12109248,  0.02590624,  0.12690234,  0.62018663],
    [ 0.81179976,  0.46311266,  0.71892529,  0.44099911,  0.43754171],
    [ 0.49648974,  0.30002541,  0.04783774,  0.69393036,  0.58613769]])
\end{lstlisting}

As we can see, all the weights are positive, while the actual relationship between the features in the data and the values of the output vector can be negative. Hence, the way we employ to assign random weights should allow for negative weights too.

Below there are too alternatives how this can be implemented in Python.

\begin{lstlisting}[language=Python]
    # Good practice for initial weights assignment:

    alternative1 = np.random.rand(3, 5) - 0.5
    # or
    alternative2 = np.random.normal(0.0, pow(3, -0.5), (3, 5))
    # arguments: Mean of the distribution, Standard deviation of the distribution, Output shape.
    # Second approach is better as it takes in account the standard deviation
    # that is related to the number of incoming links into a node, 1/√(number of incoming links).

    # alternative1
    alternative2

    # Define the weights in accordance with the best practice:
    w_i_h = np.random.normal(0.0, pow(h_n, -0.5), (h_n, i_n))
    w_h_o = np.random.normal(0.0, pow(o_n, -0.5), (o_n, h_n))
\end{lstlisting}

Now that we have all the elements assigned in accordance with good practices, we can feedforward the data once again and evaluate the results by comparing the predictions to labels.

\begin{lstlisting}[language=Python]
    # Run NN to get new classification of the particular observation:
    h_input = np.dot(w_i_h, input)
    h_output = sigmoid(h_input)
    o_input = np.dot(w_h_o, h_output)
    o_output = sigmoid(o_input)

    # Calculate the errors of the classification:
    o_errors = target - o_output
    o_errors
\end{lstlisting}

\begin{lstlisting}
    array([[-0.06373055],
    [-0.13887152],
    [-0.95694188],
    [ 0.28799234],
    [-0.23295729],
    [-0.21478192],
    [-0.74462671],
    [-0.65361842],
    [-0.36623992],
    [-0.46408493]])
\end{lstlisting}

The result we would like to achieve should look like a vector of values where almost all values are negligibly small except for the one value that has the position in the vector corresponding to the index of the true label.

It is not the case now. Nevertheless, one should remember that so far all the weights have been assigned randomly and no training has been performed. In any case, it is not a vector of ones anymore, which an improvement.

Thus, we can proceed to the next stage, which is to find out where the errors come from and how they can be minimized.


\chapter{Numeric and Mathematical Modules}
\label{numeric}

The modules described in this chapter provide
numeric and math-related functions and data types.
The \module{math} and \module{cmath} contain
various mathematical functions for floating-point and complex numbers.
For users more interested in decimal accuracy than in speed, the
\module{decimal} module supports exact representations of  decimal numbers.

The following modules are documented in this chapter:

\localmoduletable


\documentclass[a4paper,12pt]{report}

\usepackage{amsmath,amsfonts,mathtools}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{hyperref}

\begin{document}
\title{CHE260 Part 2 Abridged}
\author{Aman Bhargava}
\date{September 2019}
\maketitle

\tableofcontents

\chapter{Basics of Heat Transfer}
\section{Introduction}
Heat is energy transferred because of temperature differences. The modes of heat transfer include:
\begin{enumerate}
\item Conduction
\item Convection
\item Radiation
\end{enumerate}

\section{Conduction}
\paragraph{Fourier's Law of Heat Conduction: }
$$\dot{Q} = -kA\frac{\Delta T}{\Delta x}$$
\begin{enumerate}
\item $k$: THERMAL CONDUCTIVITY. It's really a function of temperature but we assume it's constant in this course.
\item $A$: Area of surface
\end{enumerate}

\paragraph{DIFFUSIVITY: } $\alpha = \frac{k}{\rho c_p}$ is the rate of heat propagation through a medium by volume. Small $\to$ mostly absorbed, not transmitted.

\section{Convection}
Heat transfer from solid to fluid in motion $\dot{Q}_{conv} = \dot{Q}_{cond} + \dot{Q}_{fluid motion}$

\paragraph{NEWTON'S LAW OF COOLING: }
$$\dot{Q}_{conv} = hA_s(T_s-T_{\infty}$$
Where $h$ is experimentally and subjectively determined.

\section{Radiation}
$$\dot{Q}_{emit} = \varepsilon \sigma A_s T_s^4$$
\begin{enumerate}
\item $\varepsilon$: Emissivity
\item $\sigma$: Stefan-Boltzmann constant
\end{enumerate}
$$\dot{Q}_{rad, net} = \varepsilon \sigma A_s(T_s^4 - T_{surr}^4)$$

$$\dot{Q}_{conv+rad} = h_{combined}A_s(T_s-T_{\infty})$$

\chapter{Steady Heat Conduction}
\section{Thermal Resistance Networks}
$$\dot{Q}=\frac{T_{\infty,1} - T_{\infty,2}}{R_{total}}$$
$V = IR \to \Delta T = \dot{Q}R$

Resistances add in the same ways as electrical resistances. For a wall, you must account for convective resistances on both sides and the
internal conductive resistance.

\paragraph{Thermal contact resistance: } Results from microscopic gaps between two objects in physical contact. $R_c$ is the resistance
per unit area of contact. Use the ratio of $R_{layer} : R_c$ to determine if the contact resistance is worth taking into consideration.

\section{Heat Conduction with Cylinders and Spheres}
\subsection{Cylinders}
By integrating $\dot{Q}_{cond, cyl} = -kA\frac{dT}{dr}$ we get $$R_{cyl} = \frac{1}{2\pi Lk}\ln(r_2/r_1)$$
Where $r_2$ is outer, $r_1$ is inner, $L$ is length, etc.

\subsection{Spheres}
By the same logic: $$R_{sph} = \frac{r_2-r_1}{4\pi r_1 r_2 k}$$

\paragraph{For both spheres and cylinders: } you must figure in $R_{conv, in} + R_{cyl} + R_{conv, out}$

\section{Critical Radius of Insulation}
For flat walls, more insulation $\to$ more thermal resistance.

$$r_{cr,cyl} = \frac{k}{h}$$
$$r_{cr,sph} = \frac{2k}{h}$$

\section{Chapter 10 Class Notes}
\begin{itemize}
\item Biot number: Ratio of convection : conduction = $Bi = \frac{hw}{k}$, $w$ is the width of the surface.
\item Diffusion time: $t_D = \frac{L^2}{\alpha}$
\item Rate Limiting Step: If $Bi < 1$: CONVECTION. Else, conduction.
\item Governing Conduction Equation: $$\rho C\frac{\partial T}{\partial t} = \nabla \cdot(k\nabla T) + \dot{S}$$
\begin{itemize}
\item $\rho$ = density
\item $C$ = heat capacity
\item $k$ = thermal CONDUCTIVITY
\item $\dot{S}$ = heat supplied per second
\end{itemize}
\end{itemize}

\section{Fins}
Fins allow you to increase the area of convection, leading to greater $\dot{Q}_{conv}$. We analyze fins at steady state with constant $k, h$.

\paragraph{FIN EQUATION: } For an infinitesimal slice of a fin, we have:
$$\dot{Q}_{cond, x} = \dot{Q}_{cond, x + \Delta x} + \dot{Q}_{conv}$$
$$\frac{d^2\theta}{dx^2} - m^2\theta = 0$$
Where $m^2 = \frac{h\rho}{kA_c}$, $\theta = T-T_\infty$
$$\theta(x) = c_1e^{mx} + c_2e^{-mx}$$
Boundary conditions: $\theta(0) = T_{base} - T_\infty$. Boundary condition 2 depends on situation...

Note that $p$ is the perimeter of the fin at the tip.

\begin{itemize}
\item \textbf{Infinite Fin Length: } $T_{tip} = T_\infty$.
\item \textbf{Insulated Tip: } $\dot{Q}_{tip} = 0 \to \frac{d\theta}{dx}|_{x=L} = 0$, so the general heat out is:
$$\dot{Q} = \sqrt{hp kA_c}(T_b-T_\infty) \tanh(mL)$$
\item Convective and radiative heat loss. Use insulated tip length with corrected length:$$L_c = L + A_c/p$$
$L_{c,rect} = L + t/2$, t = thickness. $L_{c, cyl} = L + D/4$.

\end{itemize}

\paragraph{FIN EFFICIENCY: } $\eta_{fin} = \frac{\dot{Q}_{fin}}{\dot{Q}_{fin, max}}$.
$$\dot{Q}_{max} = hA_{fin}(T_b-T_\infty)$$

\paragraph{FIN EFFECTIVENESS: } $$\varepsilon_{fin} = \frac{\dot{Q}_{fin}}{\dot{Q}_{nofin}} = \frac{\dot{Q}_{fin}}{hA_b(T_b-T_\infty)}$$
$$\varepsilon_{fin} = \frac{A_{fin}}{A_b}\eta_fin$$

\paragraph{Proper Fin Length: } Too long is a waste. $\frac{\dot{Q}_{fin}}{\dot{Q}_{long fin}} = tan{mL}$. Generally $mL = 1$ is a good compromise
for 76.2\% efficiency.

$$\dot{Q}_{fin} = \frac{T_b-T_\infty}{R}$$
$$R = \frac{1}{hA_{fin}\eta_{fin}}$$

\chapter{Transient Temperature Changes}
\section{Lump Analysis}
Treating the body like it just has one internal temperature that is the same throughout. Used when $Bi = \frac{hL_c}{k} = \leq 0.1$, $L_c = V/A_s$.
$$hA_s(T_\infty - T) dt = mc_p dT$$
$$\frac{T(t) - T_\infty}{T_i - T_\infty} = \exp(-bt)$$
Where $b = \frac{hA_s}{\rho V c_p}$

$$Q_{net} = mc_p[T(t) - T_i]$$

\section{Transient Heat Conduction}
We study 1-D transient heat conduction for walls, spheres, and cylinders.
\begin{itemize}
\item $\theta = \frac{T - T_\infty}{T_i-T_\infty}$ = dimensionless temperature
\item $X = x/L$ = dimensionless time
\item $Bi = hL/k$ = dimensionless heat transfer coefficient
\end{itemize}

The fourier series for each of these is on the formula sheet. When $\tau > 0.2$ we can use the single-term fourier approximation.
$\tau = \frac{\alpha t}{L^2}$.
\subsection{Illustrative Example: }
\paragraph{How long until the centre of an egg is 70 Celsius? }
\begin{enumerate}
\item $Bi = \frac{hr_0}{k}$. $Bi > 0.1 \to $ Transient 1-D analysis. Less means lump analysis.
\item Find $\lambda_1$ and $A_1$ from the table.
\item Find $\tau$, the fourier number. $$\frac{T_0-T_\infty}{T_i - T_\infty} = A_1 e^{-\lambda_1^2 \tau}$$ to determine the fourier number $\tau$.
$T_0$ is the final temperature.
\item $t = \frac{\tau r_0^2}{\alpha}$ = time until middle reaches final temperature $T_0$.
\end{enumerate}

\chapter{External Forced Convection}
\section{Physical Mechanisms of Convection}
\paragraph{No Slip Condition: } Fluid velocity is zero and `sticks' to the surface of objects it flows by. Therefore, we have pure conduction
at the surface of objects in fluids.

IMPORTANT NUMBERS:
\begin{enumerate}
\item \textbf{Nusslet} $Nu = \frac{h L_c}{k} = \frac{\dot{q}_{conv}}{\dot{q}_{cond}}$
\item
\item
\end{enumerate}

\section{Classifying Fluid Flows}
\begin{itemize}
\item Viscous: internal stickiness is significant. Inviscous: internal stickiness is not relevant.
\item \textbf{External} means flow is unbounded. \textbf{Internal} means flow is bounded on all sides (e.g. pipe).
\item \textbf{Compressible} corresponds to $Ma > 0.3$ or $v > 100m/s$.
\item \textbf{Natural} means that it's only stuff like buoyancy and convection.
\item \textbf{Steady} means stable through time. \textbf{Uniform} means stable with location.
\item \textbf{Transient} means developing steady flows. \textbf{Periodic} means oscillatory.
\end{itemize}

\section{Velocity Boundary Layer}
Consider fluid flowing over a surface. Due to the non-slip condition, the fluid at the surface doesn't move.
$\delta$ is the height above the surface at which the x-velocity reaches $0.99v$ where $v$ is the upstream velocity. It's the
\textbf{boundary layer thickness}. It divides the `boundary layer' from the `irrotational flow' area.

$F/A = \tau$ which is shear stress. $$\tau_s = \mu \frac{du}{dy}|_{y=0}$$

$\nu = \mu/\rho$ is kinematic viscosity. Measured in stokes (cm2/s)

More practically: $$\tau_s = C_f\frac{\rho V^2}{2}$$

\chapter{In-Class Things and Guides}

\section{Thermal Boundary Layer}
This is the same as the velocity boundary layer but for temperature. It's where $T_s - T = 0.99(T_\infty - T_s)$. Its size
increases in the direction of flow. $\dot{Q}_x$ depends strongly on temperature gradient at the surface, so it's important!

\textbf{Prantl Number} is the relative thickness of the velocity : thermal layer. $$Pr = \frac{\nu}{\alpha} = \frac{\mu c_p}{k}$$

\section{Laminar and Turbulent Flows}

Reynold numbers dictate whether you are in laminar, turbulent, or transitional flow. If $Re > Re_{cr}$, we have turbulent flow.

\section{Drag and Heat Transfer in External Flow}

Lift is just the sum of forces that are perpendicular to flow.

$$C_D = \frac{F_D}{\frac{1}{2}\rho V^2 A}$$

There is a pressure component and a frictional component to drag force. Pressure dominates with large Reynold's numbers while friction
dominates with small Reynold's numbers.

The area in that formula for drag coefficient is the frontal projection of the object for most objects, but is the actual area of a
flat plate that is in line with the flow.

\subsection{Heat Transfer}
Heat transfer relies on the same variables as drag. $$Nu = C_{onst} Re_i^m Pr^n$$ where $m, n$ are constant integer exponents and $C$ is
a constant that results from the geometry.

Since $C_D, h$ vary along the surface, we want the AVERAGE values of them.

$x_cr$ is the x coordinate where flow becomes turbulent over a plate. $Re_{cr} = 5 \times 10^5$ is the generally
accepted way to get the critical reynold's number.

\paragraph{Friction Coefficients at $x$:}
\begin{itemize}
\item Laminar: $$\delta_{v, x} = \frac{4.91 x}{Re_x^0.5}$$
$$C_{fx} = \frac{0.664}{Re_{x}^0.5}$$
$$Nu_x = \frac{h_xx}{k} = 0.332 Re_x^{0.5}Pr^{1/3}$$
\item Turbulent: $$\delta_{v, x} = \frac{0.38 x}{Re_x^{1/5}}$$
$$C_{fx} = \frac{0.059}{Re_x^{1/5}}$$
$$Nu_x = \frac{h_xx}{k} = 0.0296Re_x^{0.8}Pr^{1/3}$$
\end{itemize}

\paragraph{Average Friction Coefficients: }
\begin{itemize}
\item Laminar: $$C_f = \frac{1.33}{Re_L^{0.5}}$$
\item Turbulent: $$C_f = \frac{0.074}{Re_L^{1/5}}$$
\item Laminar AND Turbulent: $$C_f = \frac{0.74}{Re_L^{1/5}}$$
\end{itemize}

\textbf{Rough, Turbulent Surfaces: } $$C_f = (1.89 - 1.62 \log \frac{\varepsilon}{L})^{-2.5}$$

\paragraph{Average Heat Transfer Coefficients}
\begin{itemize}
\item Laminar: $Nu_{avg} = \frac{hL}{k} = 0.664 Re_L^0.5 Pr^{1/3}$
\item Turbulent: $Nu_{avg} = \frac{hL}{k} = 0.037 Re_L^{0.8}Pr^{1/3}$
\item Entire Plate: $Nu_{avg} = \frac{hL}{k} = (0.037 Re_l^{0.8} - 871)Pr^{1/3}$
\end{itemize}
There's some substantial deviation from these if the fluid has very niche characteristics (small Pr for liquid metals, etc.). Here is a
formula that applies to all fluids: $$Nu_x = \frac{h_xx}{k} = \frac{0.3387 Pr^{1/3} Re_x^{0.5}}{[1+(0.0468/Pr)^{2/3}]^{1/4}}$$

\subsection{Unheated Starting Length}
It's common for the first bit of the plate that the fluid goes over to not be heated. Let $\Xi$ be the distance from the physical and heated starting points.
\begin{itemize}
\item Laminar: $$Nu_x = \frac{Nu_{x, (\Xi = 0)}}{[1-(\Xi/x)^{3/4}]^{1/3}}$$
$$h_{avg} = \frac{2[1-(\Xi/x)^{3/4}]}{1-\Xi/L} h_{x=L}$$

\item Turbulent: $$Nu_x = \frac{Nu_{x, (\Xi = 0)}}{[1-(\Xi/x)^{9/10}]^{1/9}}$$
$$h_{avg} = \frac{5[1-(\Xi/x)^{9/10}]}{4(1-\Xi/L)}h_{x=L}$$
\end{itemize}

\subsection{Uniform Heat Flux}
When the plate has uniform heat flux instead of uniform heat. Relations for $\Xi \neq 0$ apply here too.
\begin{itemize}
\item Laminar: $$Nu_x = 0.453 Re_x^{0.5}Pr^{1/3}$$
\item Turbulent: $$Nu_x = 0.0308 Re_x^{0.8}Pr^{1/3}$$
\end{itemize}

$$\dot{Q} = \dot{q}_s A$$ $$\dot{q_s} = h_x[T_s(x) T_\infty] \to T_s(x) = T_\infty + \frac{\dot{q}_s}{h_x}$$

\end{document}


\chapter{Background and Literature Review}\label{c:background}

This chapter first provides a review of relevant source detection literature, which discusses the most well-used extended source detection techniques as well as recent efforts to apply Bayesian methods to the problem. The remainder of the chapter then explains necessary background concepts from statistics and machine learning.

\section{Literature Review}
Algorithms for the extraction of sources from astronomical images have been in development for at least three decades \cite{jarvis1981focas}. The most prominent of these is SExtractor\cite{bertin1996sextractor} (Source Extractor), which is proficient at finding high-intensity sources. It begins by estimating a Gaussian model of the background, and then finds pixels above some threshold (often $5\sigma$) of the background mean. These pixels are likely sources, and photometry is then performed to fit an ellipse to those pixels. SExtractor scales linearly with the number of pixels and is the de-facto standard for source detection. Other popular source detection algorithms, such as Duchamp \cite{whiting2012duchamp} (which finds sources in three-dimensional spectral line cubes) take a similar approach. These methods share a common flaw that, if no pixels in a source reaches the given threshold, then that source will not be detected even if it is large and clearly visible when taken as a whole. Nevertheless, SExtractor is very commonly used for extended source detection \cite{dalcanton1997number} \cite{longozautomated}.

The 2 Micron All Sky Survey \cite{jarrett20002mass} searched for low surface brightness galaxies (a type of extended source) using a similar thresholding system. Images are first pre-processed to remove all bright sources, and then downsampled and smoothed to improve the signal to noise ratio. A $3\sigma$ threshold is then applied to find peaks which are classed as sources. While fast, this technique is still ultimately limited to evaluating regions individually and achieves approximately a $70-80\%$ success rate on best-case data. It is also limited in that images require significant preprocessing before the algorithm can be run.

A large number of other algorithms for detecting low surface brightness galaxies have been used \cite{kniazev2004low} \cite{walsh2009invisibles} \cite{strauss2002spectroscopic} in the analysis of various sky surveys. The most common pattern with these algorithms is to use SExtractor (or one of its precursors) to find and remove point sources, and then perform extra filtering and post-processing to extract extended sources. While very effective, these algorithms often incorporate a large amount of knowledge specific to their survey, and are designed to detect only specific types of sources. Consequently, they are not useful as general source detection techniques.

\comment{DAOfind \cite{stetson1987daophot} is another popular (though somewhat historic) source detection algorithm, with slightly more involved statistics. It fits a Gaussian profile to each small region of pixels, and calls a region a source if the profile's parameters are close to the expected parameters of a star. This is a very different method of finding sources to SExtractor, as it requires a model of the distribution of a \emph{source}, rather than the distribution of the background. It is also a very restricted method, as the algorithm will only find the sources it knows to look for. DAOfind is particularly capable at performing photometry, but there are indications that SExtractor is still a more effective source detection tool \cite{annunziatella2013inside}. Nevertheless, DAOfind represents a step towards more complex statistical models of the sky.}

Wavelet transforms are another approach to source detection \cite{damiani1997method} \cite{fruscione2006ciao}, and are well-suited to finding extended sources. These transforms, used in a similar fashion to Fourier analysis, are able to extract information about both the shape and position of a source. This sensitivity to shape enables more effective extended source detection than thresholding methods. However, this approach has a preference for sources of a particular shape, is vulnerable to false positives \cite{valtchanov2001comparison}, and is not in widespread use.

Several modern source detection algorithms, most notably BlobCat \cite{hales2012blobcat} and Aegean \cite{aegean}, are based on the idea of `flood-filling' an image. Pixels of high intensity (often $5\sigma$ \cite{aegean}) are used to seed `islands' of source pixels, which are then expanded to nearby pixels. Aegean first applies Laplacian filtering to these islands, before creating curvature maps that are used to characterise the islands as sources. While this is a similar idea to the thresholding performed by SExtractor, the collection of algorithms built on this technique have achieved very good results. In particular, Aegean has been shown to be a highly reliable and effective algorithm, and is quickly gaining traction as a replacement for SExtractor on many radioastronomy tasks.

Source detection approaches based on Bayesian probability theory have been studied increasingly in the last decade. These attempts continue the trend of backing algorithms with rigorous statistical models to extract as much information from data as possible. Bayesian ideas are incorporated to varying degrees, from simple per-pixel model comparisons to sampling from the posterior of a full generative model. Most of these techniques employ Bayesian \emph{statistics}, rather than Bayesian \emph{machine learning}. In fact, there is a notable lack of machine learning tools being used to solve astronomical source detection problems.

Hobson et al. \cite{hobson2003bayesian} model sources as Gaussians and use Markov Chain Monte-Carlo (MCMC) to search for good parameters (position, mean, variance). The number of sources itself is also sampled, and a Poisson prior is used to make the sampler favour fewer sources in general. This forms a very large search space, requiring the algorithm to be run for many iterations. On a $200 \times 200$ pixel image, the authors obtained good results when the posterior is evaluated about 200\,000 times total.

Savage et al. \cite{savage2007bayesian} use Bayesian Information Criterion to find pixels that are more likely to come from a point source distribution than a purely-background distribution. MCMC is used to optimise the parameters of a point source model. While some modifications for extended sources are presented, this method remains primarily applicable to point source detection. This approach also relies on both the sources and the background being Gaussian, and the parameters of the background being known.

A more advanced Bayesian algorithm was proposed by Guglielmetti et. al \cite{guglielmetti2009background} for extended source detection. The algorithm uses a mixture model of two Poisson distributions, one for the background and one for the source, and finds pixels where the source distribution dominates the mixture. To find large sources, the algorithm performs `multiresolution analysis', in which pixels are aggregated into cells and the same procedure is applied. This technique crucially relies on a strong model of both background and source. Unfortunately the method as-presented is applicable only to X-ray telescope data, which is fundamentally different to radio or optical telescope data.

Frean et al. \cite{frean2014source} developed a Bayesian extended source detection with similar underpinnings to ours, based on Anna Friedlander's master's thesis. This used the Dirichlet Compound Multinomial (DCM) distribution as a generative model of sources and the background, which is combined with a binning (discretisation) scheme that captures uncertainty in where bin boundaries should be placed. Sources are modelled as a Gaussian envelope generated by a DCM distribution, and the envelopes are evaluated using a Bayes factor comparison to some model of the background. The algorithm itself does a greedy gradient-based search to find envelopes that are anomalous according to some model of the background. While successful at detecting extended sources, the lack of a full likelihood model means the method has no natural way of finding \emph{all} sources in an image at once. Instead, the algorithm can be run several times, each one with the previously found sources removed.

Source detection algorithms based on Bayesian methods have a common structure: they are composed of an evaluation mechanism and a search mechanism. The evaluation mechanism defines how a region of pixels is decided to be source or background, and is most commonly a direct comparison (such as a Bayes factor or Bayesian Information Criterion) between source and background models. Evaluation mechanisms are well-developed and have been studied extensively, and are the main differentiator between the discussed Bayesian algorithms. The search mechanism defines how regions of to-be-evaluated pixels are selected, and are generally more simple. Most algorithms, particularly point source detection algorithms, search by applying the evaluation function once to every pixel in the image.

Extended source detection algorithms must use a more complex search mechanism, as these sources are rarely visible at the single-pixel level. This alludes to one of the fundamental difficulties of finding extended sources: any algorithm must search for sources at several scales in the image, that is it must perform \emph{aggregation} of some kind. The techniques discussed above take significantly different approaches to this; Wavelet transforms encode scale in the transform itself, Hobson et al. (effectively) perform brute-force search, both Frean et al. and Savage et al. greedily optimise the shape of the Gaussian region, and Guglielmetti et al. combine regions into pixel cells.

\comment{Our algorithm takes a more principled approach to extended source detection than previous work. Oddity is guided entirely by a generative model of the image, so a candidate source is evaluated by comparing the likelihood of a model with that source to the likelihood of a model without. This is desirable, as it allows the algorithm to asymptotically find the \emph{optimal} configuration of sources under its model. Oddity shares a common ancestor with the previous work of Frean et al.\ in that the DCM distribution is used to model source pixels, however that work did not use the model's likelihood as its evaluation mechanism.}

Because of the large search space, robust aggregation is difficult to perform efficiently. If the most intuitive generative model of the sky is used, that is, regions with an arbitrary size and shape are selected to be sources, then aggregation becomes very expensive. Hence, most approaches so far have either been ad-hoc or intractable for large images.

\section{Generative Models} \label{s:generative_models}

A key concept in Bayesian machine learning is a \emph{generative model}. This is a probabilistic description of how observed data was created, and is described by the \emph{likelihood} $P(\text{data} \mid \text{model})$. A model often has some latent (unobserved) variables, such as class labels, for which we wish to infer good values. This is often performed by sampling from the \emph{posterior} $P(\text{model} \mid \text{data})$. The process of inferring the latent variables is known as \emph{inverting} the generative model.

\section{Dirichlet Compound Multinomial}

The Dirichlet Compound Multinomial (DCM) is a discrete, compound probability distribution that will be one of the main building blocks of our algorithm's underlying generative model. The DCM distribution works in two steps. First, a probability vector $\vec p$ is drawn from a \emph{Dirichlet} distribution. Then, some number of observations are drawn from a \emph{multinomial} distribution with probability vector $\vec p$. The DCM is `compound' because it is composed of these two distributions.

The Dirichlet distribution takes a vector of parameters, $\vec \alpha$, which determines the profile of the distributions drawn from it. First, if $\vec\alpha$ has $K$-long vector, then drawn probability vectors will have $K$ elements. The parameter vector can be broken into two components, the \emph{shape}, $\vec s$, and the \emph{concentration}, $C$, where $\vec s$ is a probability vector and $\vec \alpha = C \vec s$. The shape parameter determines the mean distribution drawn from the Dirichlet, and the concentration parameter determines the variance of each component of the vectors. With a very large concentration, draws will be very similar to $\vec s$, and if $\vec \alpha = (1, \ldots, 1)$, then draws are uniformly random.

The Dirichlet distribution can be thought of as drawing from points on the $K$-simplex, which is the collection of all probability vectors length $K$. For example, the 3-simplex can be viewed as the surface of an equilateral triangle in 3-dimensional space, and each point on the triangle is a distribution. The shape parameter determines the mean draw on this plane, and the concentration determines the variance in each dimension. In the context of the DCM distribution, the vector $\vec \alpha$ is called the \emph{hyperparameters}.

The DCM is an example of a generative model, and it has two useful features. First, it allows observations to be modelled as drawn from an \emph{unknown} multinomial distribution. Second, it is also unusual among compound distributions in that it is tractable to calculate the likelihood, $P_\text{DCM}(\vec x \mid \vec \alpha)$. This tractability comes from the fact that the intermediate multinomial does not need to be explicitly calculated. Specifically, the result is \begin{equation}
 \PDCM (\vec x \mid \vec \alpha) = \frac{\Gamma(A)}{\Gamma(N + A)} \prod_{k=1}^K \frac{\Gamma(n_k + \alpha_k)}{\Gamma(\alpha_k)} . \label{e:DCM}
\end{equation}
Where $n_k$ is the number of occurrences of event $k$ in the observations vector $\vec x$, $\Gamma(a)$ is the gamma function, $A = \sum_{k=1}^K \alpha_k$, and $N = \sum_{k=1}^K n_k$. Note that, for integer $a$, $\Gamma(a) = (a-1)!$. We will generally take logs of this equation, yielding \begin{equation}
 \lPDCM (\vec x \mid \vec \alpha) = \log \Gamma(A) - \log \Gamma(N + A) + \sum_{k=1}^K \left[ \log\Gamma(n_k + \alpha_k) - \log \Gamma(\alpha_k) \right] . \label{e:logDCM}
\end{equation}
While the observations $\vec x$ do not appear in the equation directly, they are used to calculate the histogram $n_k$. For convenience this report will consider $\lPDCM(\vec n \mid \vec \alpha)$ as synonymous with $\lPDCM(\vec x \mid \vec \alpha)$ (despite being non-standard notation).

\section{Markov Chains and Gibbs Sampling} \label{s:markov_background}

A Markov chain is a random process that transitions between different states over time. It is initialised to some (often arbitrary) state $\theta^0$, and at each step selects a new state to transition to from a \emph{transition distribution}. The initial state of the Markov chain affects its behaviour in future time steps, but the magnitude of this effect lessens as the chain is run for more steps. Every Markov chain has a \emph{stationary distribution}, which describes its behaviour in the absence of any effect from its initial state. When the behaviour of a Markov chain very closely matches its stationary distribution, that chain is said to have \emph{mixed}. Depending on the complexity of the transition matrix and the quality of the initial state, mixing can take a very long time.

The stationary distribution of a Markov chain can be sampled from by first running the chain until it is mixed, and then drawing samples with a large enough number of iterations between them so as to be i.i.d. This procedure does not require the stationary distribution to ever be explicitly defined, as it arises naturally from the transition distribution and the structure of the chain. This is the key usefulness of Markov chains; very commonly, one needs to sample from a distribution $P$ that is intractable or impossible to sample from exactly. If a Markov chain can be created in such a way that $P$ is its stationary distribution, approximate samples can be drawn from $P$ by running the chain. This procedure is known as Markov Chain Monte Carlo (MCMC).

Gibbs sampling \cite{mackay2003information} is a simple MCMC procedure. Suppose we are sampling from a distribution $P(\theta_1, \ldots, \theta_N)$. Before the first iteration, arbitrarily initialise each variable $\theta_i^0$ (where superscripts represent time). To perform one iteration of Gibbs at time $t+1$, draw new samples for each variable as follows: \begin{equation}
  \theta_i^{t+1}\ \ \sim\ \ P(\theta_i \mid \theta_1^{t+1}, \ldots, \theta_{i-1}^{t+1}, \theta_{i+1}^t, \ldots, \theta_N^t) \label{e:gibbs_bg_update}
\end{equation}
In other words, hold all variables fixed at their current value except the one to be updated, and then draw a new value for the changing variable's state. The distribution in equation \ref{e:gibbs_bg_update} is known as the \emph{full conditional} distribution of $\theta_i^{t+1}$, which is generally known and easy to calculate. Sampling in this fashion creates a Markov chain whose stationary distribution is the posterior. Hence, once the chain has mixed, the variables become a true sample from the posterior. Variables do not need to be updated in any particular or consistent order, as long as every variable is updated every iteration. Gibbs sampling is useful for learning the latent variables of a generative model, so long as the full conditional distributions are known.

\comment{\section{Metropolis Sampling} \label{s:metropolis_background}

The Metropolis-Hastings algorithm is a second sampling algorithm for approximating samples from a complex posterior. Metropolis also works iteratively, on a system of proposal transitions between states that are either accepted or rejected. Suppose $Q(\theta^*\mid\theta^t)$ is a \emph{transition distribution}, which can be sampled to offer a proposed transition, $\theta^*$, for the next iteration, which might be dependent on the previous state $\theta^t$. To run the algorithm, first initialise $\theta^0$ arbitrarily to some state with non-zero probability in the posterior. Then, at time $t+1$, sample a candidate proposal $\theta^* \sim Q$. Next, calculate the \emph{acceptance ratio} as follows \begin{equation}
 r = \frac{P(\theta^*) Q(\theta^t | \theta^*)}{P(\theta^t) Q(\theta^* | \theta^t)}. \label{e:acceptance_ratio}
\end{equation}
Finally, accept the proposal with probability $min(1,r)$. If the proposal is accepted, $\theta^{t+1} = \theta^*$, otherwise $\theta^{t+1} = \theta^t$. Much like with Gibbs sampling, as time tends to infinity the state tends to a true sample from the posterior $P$.

Gibbs sampling is a special case of Metropolis-Hastings, in which only a single parameter of the model is varied on any iteration. Gibbs is conceptually simpler, and is generally used when the parameters take on a small number of discrete values. Depending on the distribution being sampled from, changing parameters individually often avoids the need to evaluate the full posterior of $P$ which makes it a cheaper algorithm to run.}


\documentclass[]{BasiliskReportMemo}
\usepackage{AVS}

\newcommand{\submiterInstitute}{Autonomous Vehicle Simulation (AVS) Laboratory,\\ University of Colorado}

\newcommand{\ModuleName}{test\textunderscore sunlineSuKF}
\newcommand{\subject}{Sunline Switch-uKF Module and Test}
\newcommand{\status}{Initial document}
\newcommand{\preparer}{T. Teil}
\newcommand{\summary}{This module implements and tests a Switch Unscented Kalman Filter in order to estimate the sunline direction.}

\begin{document}

\makeCover

%
%	enter the revision documentation here
%	to add more lines, copy the table entry and the \hline, and paste after the current entry.
%
\pagestyle{empty}
{\renewcommand{\arraystretch}{2}
\noindent
\begin{longtable}{|p{0.5in}|p{4.5in}|p{1.14in}|}
\hline
{\bfseries Rev}: & {\bfseries Change Description} & {\bfseries By} \\
\hline
Draft & Initial Revision & T. Teil \\
\hline

\end{longtable}
}

\newpage
\setcounter{page}{1}
\pagestyle{fancy}

\tableofcontents
~\\ \hrule ~\\

%\begin{figure}[htb]
%	\centerline{
%	\includegraphics[]{Figures/Fig1}
%	}
%	\caption{Sample Figure Inclusion.}
%	\label{fig:Fig1}
%\end{figure}

\section{Introduction}
The Switch Unscented Kalman filter (SuKF) in the AVS Basilisk simulation is a sequential
filter implemented to give the best estimate of the desired states.
In this method we estimate the sun heading as well as the spacecraft rotation rate along the observable axes.
The SuKF reads in the message written by the coarse sun sensor, and writes a message
containing the sun estimate.

This document summarizes the content of the module, how to use it, and the test that
was implemented for it. More information on the filter derivation can be found in Reference [\citenum{Teil:2018fe}], and more information on the square root unscented filter can be found in Reference [\citenum{Wan2001}] (attached alongside this document).

\section{Filter kinematics}

\subsection{Filter Derivation} %%%

The Switch-uKF attempts to avoid subtracting any terms from the state, while still removing the unobservable component of the rate. In order to do this, an appropriate frame must be defined.
In order to not track the rate component alongside the sunline direction, that vector needs to be one of the basis vectors of the frame. It is decided to be the first vector for the frame, and therefore in that frame, $\omega_1$ the component of the rotation rate can be removed from the states.
This frame is called $\mathcal{S}_1 = \{\hat{\bm s}_1 = \frac{\bm d}{|\bm d|}, \hat{\bm s}_2, \hat{\bm s}_3 \}$. This is seen in Figure \ref{fig:Switches}, where the dotted line represents the $30 \dg$ threshold cone before switching frames.

\begin{figure}[t]
	\centering
	\includegraphics[]{./Figures/Switches}
	\caption{Frame built off the body frame for Switch filters}
	\label{fig:Switches}
\end{figure}

The second vector of the frame must be created using only $\bm d$, and the body frame vectors. The first intuitive decision, is to use $\hat{\bm b}_1$ of the body frame and define $\bm s_2$ in Equation \eqref{eq:s2}. The third vector $\hat{\bm s}_3$ of the $\mathcal{S}_1$ frame, is naturally created from the first two.
\begin{equation}\label{eq:s2}
\hat{\bm s}_2 = \frac{\hat{\bm s}_1 \times \hat{\bm b}_1}{|\hat{\bm s}_1 \times \hat{\bm b}_1|} \hspace{2cm} \hat{\bm s}_3 = \frac{\hat{\bm s}_1 \times \hat{\bm s}_2}{|\hat{\bm s}_1 \times \hat{\bm s}_2|}
\end{equation}
The problem that arises is the singularity that occurs when $\hat{\bm b}_1$ and $\bm d$ become aligned: this frame becomes undefined. In order to counteract this, using a similar process as the shadow set used for Modified Rodrigues Parameters [\citenum{schaub}], a second frame is created. This frame $\mathcal{S}_2 =  \{\hat{\bar{\bm s}}_1 = \hat{\bm s}_1, \hat{\bar{\bm s}}_2 , \hat{\bar{\bm s}}_3 \}$ is created with the same first vector, but constructs $\hat{ \bar{\bm s}}_2$ using $\hat{\bm b}_2$ of the body frame as in Equation \eqref{eq:s2bar}. The last vector, once again, finishes the orthonormal frame.
\begin{equation}\label{eq:s2bar}
\hat{\bar{\bm s}}_2 = \frac{\hat{\bar{\bm s}}_1 \times \hat{\bm b}_2}{|\hat{\bar{\bm s}}_1 \times \hat{\bm b}_2|}
\end{equation}
With both these frames, $\mathcal{S}_1$ and $\mathcal{S}_2$, the singularities can always be avoided. Indeed, $\mathcal{S}_1$ becomes singular when $\bm d$ approches $\hat{\bm b}_1$, while $\mathcal{S}_2$ becomes singular when the sunheading approaches $\hat{\bm b}_2$. By changing frames, whenever the sunline gets within a safe cone of $30 \dg$ (a modifiable value) of $\hat{\bm b}_1$, the frame is rotated into $\mathcal{S}_2$, which is not singular. Similarly, when $\bm d$ approches $\hat{\bm b}_2$ the frame is switched back to $\mathcal{S}_1$.

Because the two frames share the sunline vector $\bm d$, this vector is the same in both frames. This is a clear advantage as this is the vector we desire to estimate, and not having to rotate it avoids numerical issues.
The rotation of the rates is done by computing the following DCMs, of which all the vectors are known.
\begin{equation}\label{eq:DCMs}
 [\mathcal{B}\mathcal{S}_1] = \begin{bmatrix} \leftexp{B}{\hat{\bm s}_1} & \leftexp{B}{\hat{\bm s}_2} & \leftexp{B}{\hat{\bm s}_3}\end{bmatrix} \hspace{1cm}  [\mathcal{B}\mathcal{S}_2] = \begin{bmatrix} \leftexp{B}{\hat{\bar{\bm s}}_1} & \leftexp{B}{\hat{\bar{\bm s}}_2} & \leftexp{B}{\hat{\bar{\bm s}}_3}\end{bmatrix} \hspace{1cm}  [\mathcal{S}_2\mathcal{S}_1] = [\mathcal{B}\mathcal{S}_2] ^T [\mathcal{B}\mathcal{S}_1]
\end{equation}

\subsection{Filter Dynamics} %%%

The filter is therefore derived with the states being $\bm X =\begin{bmatrix}\leftexp{B}{\bm d} & \omega_2 & \omega_3 \end{bmatrix}^{T}$, given that $\bm \omega_{\mathcal{S}/\mathcal{B}} = \leftexp{S}{\begin{bmatrix} \omega_1 & \omega_2 &\omega_3\end{bmatrix}}^T$. The rates of $\mathcal{S}$ relative to the body and inertial frame are related as such: $\bm \omega_{\mathcal{S}/\mathcal{N}} - \bm \omega_{\mathcal{S}/\mathcal{B}}  = \bm \omega_{\mathcal{B}/\mathcal{N}}$. Since $\omega_1$ is unknown, it is set to zero. Furthermore, since the sun heading is considered to be constant in the inertial frame over the period of time required for attitude determination and control, the equation becomes  $- \bar{\bm \omega}_{\mathcal{S}/\mathcal{B}}  = \bar{\bm \omega}_{\mathcal{B}/\mathcal{N}}$.

${\bm \omega}_{\mathcal{S}/\mathcal{B}}$ is estimated directly by the filter, and its skew matrix can be computed by setting $\omega_1$ to zero (in the absence of information). This defines $\tilde{\bm \omega}_{\mathcal{B}/\mathcal{N}}$ as a function of known parameters. The dynamics are therefore given by Equations \eqref{eq:dynSwitch} and \eqref{eq:dynmatSwitch}, where $ \tilde{[\bm d]}(2,3)$ corresponds to the $2^{\text{nd}}$ and $3^{\text{rd}}$ columns of the $ \tilde{[\bm d]}$ matrix.
\begin{align}
\label{eq:dynSwitch}
\bm X' = \bm F(\bm X) &=  \begin{bmatrix}  \leftexp{B}{ \bm d'} \\  \omega_2' \\ \omega_3' \end{bmatrix} =   \begin{bmatrix}  -\leftexp{B}{\bar{\bm \omega}_{\mathcal{B}/\mathcal{N}}} \times \leftexp{B}{\bm d}\\ 0 \\ 0\end{bmatrix}  =   \begin{bmatrix} [\mathcal{B}\mathcal{S}]\leftexp{S}{\begin{bmatrix} 0 \\ \omega_2 \\ \omega_3\end{bmatrix}} \times \leftexp{B}{\bm d}\\ 0 \\ 0\end{bmatrix}
\\\label{eq:dynmatSwitch}
[A]&= \begin{bmatrix} \frac{\partial \bm F (\bm d, t_i)}{\partial \bm X}  \end{bmatrix}  =   \begin{bmatrix} [ \leftexp{B}{\tilde{\bar{\bm \omega}}_{\mathcal{S}/\mathcal{B}}}] & - \tilde{[\bm d]}[\mathcal{B}\mathcal{S}](2,3) \\  [0]_{2\times 3} &  [0]_{2\times 2} \end{bmatrix}
\end{align}
This formulation leads to simple dynamics, much simpler than those of the filter which subtracts the unobservable states, yet can actually estimate the observable of the rate, instead of using past estimates of $\bm d$.

In regard to the SR-uKF version of this filter, the same coefficients are used: $\alpha = 0.02$, and $\beta = 2$.

\subsection{Switching Frames} %%%

When switching occurs, the switch matrix $[W]$ can be computed in Equation \eqref{eq:switchMat} using the previously computed DCMs. This equation assumes the switch is going from frame 1 to frame 2 (the reciprocal is equivalent), and $[\mathcal{S}_2 \mathcal{S}_1](2,3)$ corresponds to the $2^{\text{nd}}$ and $3^{\text{rd}}$ columns of the $[\mathcal{S}_2 \mathcal{S}_1]$ matrix.
\begin{equation}\label{eq:switchMat}
[W] = \begin{bmatrix} [I]_{3\times 3} & [0]_{3 \times 2} \\ [0]_{2 \times 3} & [\mathcal{S}_2 \mathcal{S}_1](2,3)\end{bmatrix}
\end{equation}
The new states $\bm X$ and covariance [P] after the switch are therefore given in Equation \eqref{eq:switchEq}
\begin{equation}\label{eq:switchEq}
\bar{\bm X} = [W] \bm X \hspace{2cm} [\bar{P}] = [W] [P] [W]^T
\end{equation}
When writing out the values of the state and covariance, it is necessary to bring it back into the body frame, using the $[\mathcal{B}\mathcal{S}]$ DCM ($\mathcal{S}$ representing the current frame in use).

\subsection{Measurements}
The measurement model is given in equation \ref{eq:meas}, and the $H$ matrix defined as $H = \left[\frac{\partial \bm G (\bm X, t_i)}{\partial \bm X}\right]^{*}$ is given in equation $\ref{eq:Hmat}$.

In this filter, the only measurements used are from the coarse sun sensor. For the $i^\mathrm{th}$ sensor, the measurement is simply given by the dot product of the sunline heading and the normal to the sensor. This yields easy partial derivatives for the H matrix, which is a matrix formed of the rows of transposed normal vectors (only for those which received a measurement). Hence the $H$ matrix has a changing size depending on the amount of measurements.

\begin{equation}\label{eq:meas}
\bm G_i(\bm X) = \bm n_i \cdot \bm d
\end{equation}

\begin{equation}\label{eq:Hmat}
\bm H(\bm X) = \begin{bmatrix} \bm n_1^T \\ \vdots \\ \bm n_i^T \end{bmatrix}
\end{equation}

\section{Filter Set-up, initialization, and I/O}

\subsection{User initialization}

In order for the filter to run, the user must set a few parameters:

\begin{itemize}
\item The unscented filter has 3 parameters that need to be set, and are best as: \\
      \texttt{ filterObject.alpha = 0.02} \\
      \texttt{ filterObject.beta = 2.0} \\
      \texttt{ filterObject.kappa = 0.0}
\item The angle threshold under which the coarse sun sensors do not read the measurement: \\
\texttt{FilterContainer.sensorUseThresh = 0.}
\item The process noise matrix: \\
   \texttt{qNoiseIn = numpy.identity(5)} \\
   \texttt{ qNoiseIn[0:3, 0:3] = qNoiseIn[0:3, 0:3]*0.01*0.01} \\
   \texttt{ qNoiseIn[3:5, 3:5] = qNoiseIn[3:5, 3:5]*0.001*0.001} \\
    \texttt{filterObject.qNoise = qNoiseIn.reshape(25).tolist()}
\item The measurement noise value, for instance: \\
 \texttt{FilterContainer.qObsVal = 0.001}
\item The initial covariance: \\
 \texttt{Filter.covar =} \\
  \texttt{ [1., 0.0, 0.0, 0.0, 0.0, \\
                          0.0, 1., 0.0, 0.0, 0.0,\\
                          0.0, 0.0, 1., 0.0, 0.0,\\
                          0.0, 0.0, 0.0, 0.02, 0.0,\\
                          0.0, 0.0, 0.0, 0.0, 0.02]}
 \item The initial state :\\
 \texttt{Filter.state =[0.0, 0.0, 1.0, 0.0, 0.0]}
\end{itemize}

The messages must also be set as such:

\begin{itemize}
\item    \texttt{ filterObject.navStateOutMsgName = "sunline$\_$state$\_$estimate"}
\item    \texttt{ filterObject.filtDataOutMsgName = "sunline$\_$filter$\_$data"}
\item   \texttt{ filterObject.cssDataInMsgName = "css$\_$sensors$\_$data"}
\item   \texttt{ filterObject.cssConfInMsgName = "css$\_$config$\_$data"}
\end{itemize}

\subsection{Inputs and Outputs}

The SuKF reads in the measurements from the coarse sun sensors. These are under the form of a list of cosine values. Knowing the normals to each of the sensors, we can therefore use them to estimate sun heading.

\section{Test Design}
The unit test for the sunlineSuKF module is located in:\\

\noindent
{\tt fswAlgorithms/attDetermination/sunlineSuKF/$\_$UnitTest/test$\_$SunlineSuKF.py} \\

As well as another python file containing plotting functions:

\noindent
{\tt fswAlgorithms/attDetermination/sunlineSuKF/$\_$UnitTest/SunlineSuKF$\_$test$\_$utilities.py} \\

The test is split up into 3 subtests. The first test creaks up all of the individual filter methods and tests them individually. These notably go over the square-root unscented filter specific functions. The second test verifies that in the case where the state is zeroed out from the start of the simulation, it remains at zero. The third test verifies the behavior of the time update with a measurement modification in the middle of the run.

\subsection{Individual tests}

In each of these individual tests, random inputs are fed to the methods and their values are computed in parallel in python. These two values are then compared to assure that the correct computations are taking place.
\begin{itemize}
\item \underline{QR Decomposition}: This tests the QR decomposition function which returns just the R matrix. Tolerance to absolute error $\epsilon = 10^{-15}$.

\textcolor{ForestGreen}{Passed}
\item \underline{LU Decomposition}: This tests the LU Decomposition accuracy. Tolerance to absolute error $\epsilon = 10^{-14}$.

\textcolor{ForestGreen}{Passed}
\item \underline{LU backsolve}: This tests the LU Back-Solve accuracy. Tolerance to absolute error $\epsilon = 10^{-14}$.

\textcolor{ForestGreen}{Passed}
\item \underline{LU matrix inverse}: This tests the LU Matrix Inverse accuracy. Tolerance to absolute error $\epsilon = 10^{-14}$.

\textcolor{ForestGreen}{Passed}
\item \underline{Cholesky decomposition}: This tests the Cholesky Matrix Decomposition accuracy. Tolerance to absolute error $\epsilon = 10^{-14}$.

\textcolor{ForestGreen}{Passed}
\item \underline{L matrix inverse}: This tests the L Matrix Inverse accuracy. Tolerance to absolute error $\epsilon = 10^{-14}$.

\textcolor{ForestGreen}{Passed}

\item \underline{U matrix inverse}: This tests the U Matrix Inverse accuracy. Tolerance to absolute error $\epsilon = 10^{-12}$.

\textcolor{ForestGreen}{Passed}
\end{itemize}

\subsection{Static Propagation}

\input{AutoTeX/StatesPlotprop.tex}

This test also takes no measurements in, and propagates with the expectation of no change. It then tests that the states and covariance are as expected throughout the time of simulation. Plotted results are seen in Figure \ref{fig:StatesPlotprop}. We indeed see that the state and covariance that evolve nominally and without bias .

Tolerance to absolute error: $\epsilon = 10^{-10}$

\subsection{Full Filter test}

This test the filter working from start to finish. No measurements are taken in for the first 20 time steps. Then a heading is given through the CSS message. Halfway through the simulation, measurements stop, and 20 time steps later a different heading is read. The filter must be robust and detect this change. This test is parametrized for different test lengths, different initial conditions, different measured headings, and with or without measurement noise. All these are successful.

\vspace{0.2cm}
Tolerance to absolute error without measurement noise: $\epsilon = 10^{-10}$

\textcolor{ForestGreen}{Passed}

Plotted results are seen in Figures \ref{fig:StatesPlotupdate}, and \ref{fig:PostFitupdate}. Figure \ref{fig:StatesPlotupdate} shows the state error and covariance over the run. We see the covariance initially grow, then come down quickly as measurements are used. It grows once again as the measurements stop before bringing the state error back to zero with a change in sun heading.

Figure \ref{fig:PostFitupdate} shows the post fit residuals for the filter, with no measurement noise. We see that the observations are read in well an that the residuals are brought back down to zero.

\input{AutoTeX/StatesPlotupdate.tex}
\input{AutoTeX/PostFitupdate.tex}

\bibliographystyle{AAS_publication}   % Number the references.
\bibliography{references}   % Use references.bib to resolve the labels.

\end{document}


\section{Biconnectivity}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Biconnectivity algorithm in one word}
  % \fignocaption{width = 0.50\textwidth}{figs/back-logo.jpg}
  \centerline{\Huge Back!}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Biconnectivity algorithm in two questions}
  \begin{center}
	(1) When and how to update $\text{back}[v]$? \\[15pt] \pause
	(2) When and how to identify a bicomponent?
  \end{center}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Biconnectivity algorithm}
  \begin{exampleblock}{Initialization of $\text{back}[v]$ (Problem 5.6)}
	\[
	  \text{back}[v] = d[v] \text{\emph{ vs. }} \text{back}[v] = \infty, 2(n+1)
	\]
  \end{exampleblock}

  % $\text{back}[v]$: the earliest reachable ancestor of $v$ by tree + back edges
  % \begin{itemize}
  %   \item tree edge ($\to v$; discovered): $\text{back}[v] = d[v]$
  %   \item back edge ($v \to w$): $\text{back}[v] = \min \set{\text{back}[v], d[w]}$
  %   \item backtracking from $w$: $\text{back}[v] = \min \set{\text{back}[v], \text{back}[w] = \text{wBack}}$
  % \end{itemize}

  \pause
  \vspace{0.50cm}
  \begin{itemize}
	\item if updated
	  \pause
	\item if never updated:
	  \[
		\text{wBack} = \infty > d[v] \text{\emph{ vs. }} \pause \text{wBack} = d[w] > d[v]
	  \]
  \end{itemize}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Root cutnode}
  \begin{exampleblock}{Root cutnode (Problem 5.5)}
	\[
	  v \text{ is a cutnode} \iff \text{OutDegree}[v] \ge 2
	\]
  \end{exampleblock}

  \pause
  \fignocaption{width = 0.40\textwidth}{figs/cutnode-types.png}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{$K$-core of a graph}
  \begin{exampleblock}{Planning a party (Problem 5.27)}
	\begin{itemize}
	  \item undirected graph $G$
	  \item subgraph $G' = (V', E')$:
		\[
		  \forall v' \in V: K(v') \ge 5 \land D(v') \ge 5
		\]
	\end{itemize}
  \end{exampleblock}

  \pause
  \vspace{0.50cm}
  $K$-core of a graph:
  \[
	\forall v' \in V': \text{deg}[v'] \ge k
  \]

  \pause
  \vspace{0.50cm}
  \centerline{Iteratively delete nodes $v$ of $K(v) < 5 \lor D(v) < 5$}
\end{frame}
%%%%%%%%%%%%%%%%%%%%