\epigraph{``Since Newton, mankind has come to realise that the law of physics are always expressed in the language of differential equations"}{Steven Strogatz}

\section{Finite Difference Method}\label{fdm_section}
\subsection{Introduction}
The use of numerical models for the simulation of dynamics within the atmosphere typically involves the solution of a set of partial differential equations. These equations generally describe three processes: advection, adjustment and diffusion.

\begin{definition}
Advection is the transport of a substance by bulk motion.
\end{definition}

\begin{definition}
Adjustment is how the mass and wind fields adjust to one another.
\end{definition}

\begin{definition}
Diffusion is the movement of a liquid, or gas from an area of high concentration to an area of low concentration.
\end{definition}

Most meteorological problems involving partial differential equations generally fall into three distinct categories: initial value problems, boundary value problems, and eigenvalue problems. The meteorological problems associated with this project are solely initial value problems\cite{numerical_methods}. An initial value problem is a situation where you want to predict the future state of a given system, given the necessary initial conditions. Unfortunately, the equations describing the evolution of the atmosphere do not have exact analytical solutions.

\begin{definition}
An analytical function is the most precise way of representing a  physical field as it gives us the value of this field at any point in space, and at any instant in time.
\end{definition}

When an analytical solution does not exist, an approximate numerical solution is found using a specified computational technique\cite{numerical_methods}. For the purposes of this project, the finite difference method will be utilised. It must be noted, however, that there are several other methods to acquire an approximate numerical solution, with the finite element method and the spectral method being just a few.

The idea of the finite difference method is to approximate the derivatives in the partial differential equations with differences between adjacent points in space and in time. The advantages of this being that the problem becomes an algebraic one, and that a continuous problem becomes a discrete one.

\subsection{Derivation of the Finite Difference Method}

In order to derive the finite difference method, it is necessary to look at the Taylor series expansion of the function.

\begin{definition}
The Taylor series of a function is the limit of that function's Taylor polynomials as the degree increases.
\end{definition}

The Taylor series expansion of $f(x)$ can be represented as the following:

\begin{equation}
    f(x + \Delta x) = f(x) + \frac{d f}{d x}(x) \frac{\Delta x}{1!} + \frac{d^2 f}{d x^2}(x) \frac{\Delta x^2}{2!} + ... + \frac{d^n f}{d x^n}(x) \frac{\Delta x^n}{n!} + ...
\end{equation}
\begin{equation}
    f(x - \Delta x) = f(x) + \frac{d f}{d x}(x) \frac{-\Delta x}{1!} + \frac{d^2 f}{d x^2}(x) \frac{(-\Delta x)^2}{2!} + ... + \frac{d^n f}{d x^n}(x) \frac{(-\Delta x)^n}{n!} + ...
\end{equation}

The higher order terms, which will be represented as $\mathcal{O}(\Delta x)$ from this point on, become less important as $\Delta x$ approaches zero.

\begin{equation}
    \Rightarrow f(x + \Delta x) = f(x) + \frac{d f}{d x}(x) \frac{\Delta x}{1!} + \frac{d^2 f}{d x^2}(x) \frac{\Delta x^2}{2!} + [\mathcal{O}(\Delta x^3)]
    \label{fds}
\end{equation}
\begin{equation}
    \Rightarrow f(x - \Delta x) = f(x) - \frac{d f}{d x}(x) \frac{\Delta x}{1!} + \frac{d^2 f}{d x^2}(x) \frac{\Delta x^2}{2!} + [\mathcal{O}(\Delta x^3)]
    \label{bds}
\end{equation}

These higher order terms are neglected, and the following approximation for the derivative of $f(x)$ is found:

\begin{equation}
    \Rightarrow \frac{d f}{d x}(x) = \frac{f(x + \Delta x) - f(x)}{\Delta x} + \mathcal{O}(\Delta x)
\end{equation}
\begin{equation}
    \Rightarrow \frac{d f}{d x}(x) = \frac{f(x - \Delta x) - f(x)}{\Delta x} + \mathcal{O}(\Delta x)
\end{equation}

These are called the forward and backward difference schemes respectively, and by keeping only the leading order terms, an error of order $\mathcal{O}(\Delta x)$ is occurred. It is possible to obtain a better approximation by subtracting \ref{bds} from \ref{fds}, which yields equation \ref{eq_fds_bds_subtract}.

\begin{equation}
    \frac{d f}{d x}(x) = \frac{f(x + \Delta x) - f(x - \Delta x)}{\Delta x} + \mathcal{O}(\Delta x^2)
    \label{eq_fds_bds_subtract}
\end{equation}

This particular approximation is called the central difference scheme, and has an error of order $\mathcal{O}(\Delta x^2)$. Therefore, this scheme is more accurate than the previously mentioned forward and backward difference schemes. It is possible to take more and more terms from the Taylor series expansion, however, there is an inherent trade off between accuracy and computational efficiency.

In relation to the atmosphere, what this method does is divide the atmosphere into several discrete horizontal layers, and each layer is divided up into grid cells. Following which, each equation is evaluated at the centre of the cell. Similarly, the time interval under consideration is sliced into a number of discrete time steps. The size of the grid step $\Delta x$ and time step $\Delta t$ determines the accuracy of the scheme, with accuracy increasing as $\Delta x$ and $\Delta t$ approach zero. On a synoptic scale, $\Delta x$ is generally equal to 500 km. For higher resolutions, the grid-size is smaller, which corresponds to a greater computational burden. As such, there is a trade off between accuracy and computational performance. For Eulerian schemes, the typical time step is 2 minutes. As such, since the software will use an Eulerian scheme, the time step will be 2 minutes\cite{leapfrog_slides_one}.

\subsection{FTCS Scheme}\label{ftcs_section}

Given the information mentioned in the previous section, the most obvious scheme to approximate a differential equation, which will be used to predict the future state of the atmosphere, would be to combine the central difference scheme for space and the forward difference scheme for time (FTCS). This scheme would allow us access to the increased accuracy of the central difference scheme, while maintaining two time variable unknowns. If only it was that simple! Let's take the example of the 1-D linear advection equation for temperature. This equation is represented as the following:

\begin{equation}
    \frac{\partial T}{\partial t} + u \frac{\partial T}{\partial x} = 0
    \label{1d_temp_eq}
\end{equation}

Using the FTCS scheme mentioned above, this equation can be approximated as:

\begin{equation}
    \frac{T^{n+1}_{i} - T^{n}_{i}}{\Delta t} + u \frac{T^{n}_{i+1} - T^{n}_{i-1}}{2 \Delta x} = 0
\end{equation}

It can be shown, by using Fourier Series, that:

\begin{equation}
    |\lambda_j|^2 = 1 + \alpha^2(\sin{j \Delta x}^2)
\end{equation}

Therefore, $|\lambda_j|^2 \geq 1$, and so the scheme is said to be absolutely unstable. What it means for a scheme to be unstable is that if there is a slight change in the initial value, the result of the computation will change dramatically. The stability of a scheme is important in meteorological problems because if slight deviations from the mathematical model caused by unavoidable errors in measurement do not have a correspondingly slight effect on the approximate numerical solution, the mathematical equations describing the problem will not accurately predict the future outcome\cite{ftcs_leapfrog}. For a more detailed technical explanation of the stability of this scheme and the leapfrog scheme (discussed in section \ref{leapfrog}), please see the following article: \url{https://www.ecmwf.int/sites/default/files/elibrary/2002/16948-numerical-methods.pdf}.

\subsection{Leapfrog Scheme}\label{leapfrog}

This scheme is probably the most common scheme used for meteorological problems. The "leapfrog" refers to the centred
time difference which is used in conjunction with centred space differences.

Taking the 1-D linear advection equation for temperature seen in equation \ref{1d_temp_eq}, applying this scheme results in:

\begin{equation}
    \frac{T^{n+1}_{i} - T^{n-1}_{i}}{2 \Delta t} + u \frac{T^{n}_{i+1} - T^{n}_{i-1}}{2 \Delta x} = 0
\end{equation}

It can be shown that this scheme is stable using a similar technique mentioned in section \ref{ftcs_section}. This equation can then be rearranged for the forecast value $T^{n+1}_{i}$\cite{ftcs_leapfrog}:

\begin{equation}
    T^{n+1}_{i} = T^{n-1}_{i} - u \frac{2 \Delta t}{2 \Delta x}(T^{n}_{i+1} - T^{n}_{i-1})
\end{equation}

For the physical equation, a single initial condition $T^{0}$ is sufficient to determine the solution. One problem with the leapfrog scheme is that two values of $T$ are required to start the computation. In addition to the physical initial condition $T^{0}$, a computational initial condition $T^{1}$ is required. This cannot be obtained using the leapfrog scheme, so a non-centred step is used to provide the value at $t = \Delta t$. From which point on, the leapfrog scheme is used, however, the errors of the first step will persist. This method, however, still retains an error of order $\mathcal{O}(\Delta t^2)$. If you also use half of the time step for the forward time step, followed by leapfrog time steps; this will reduce the error introduced in the first step\cite{leapfrog_slides_two}. This will be the method utilised within the software.

\subsection{Nonlinear Instability}
A major problem which occurs while dealing with nonlinear partial differential equations is nonlinear instability. This is a problem where there is a nonlinear interaction between atmospheric waves\cite{nonlinear_instability}.

\begin{definition}
An atmospheric wave is a periodic disturbance in the fields of atmospheric variables (like geopotential height, temperature, or wind velocity) which may either propagate (travelling wave) or not (standing wave).
\end{definition}

If one of the waves involved in this nonlinear interaction have a wavelength less than $4 \Delta x$ something called aliasing causes a channelling of energy towards the small wavelengths. The continuous feedback of energy leads to a catastrophic rise in the kinetic energy of wavelengths between $2 \Delta x$ and $4 \Delta x$. Within the software, a smoothing operator, which reduces the amplitude of the short waves while having little effect on the meteorologically important waves, is utilised\cite{nonlinear_instability}.

Another problem to mention before moving on is that for nonlinear equations, the leapfrog scheme has a tendency to increase the amplitude of the computational mode with time This can separate the space dependence between the even and odd time steps. This problem can be rectified by applying a Robert-Asselin Time Filter. After $T^{n+1}$ is obtained a slight time smoothing is applied to $T^{n}$, where $\gamma$ is on the order of 0.1\cite{leapfrog_slides_two}:

\begin{equation}
    T^{n} = T^{n} + \gamma(T^{n+1} - 2 T^{n} + T^{n-1})
\end{equation}

\section{Ensemble Prediction System}
\subsection{Introduction}
Ensemble Prediction Systems (EPS) are numerical weather prediction systems that allow for the estimation of uncertainty in a weather forecast, as well as, providing a better prediction for the future state of the atmosphere. Instead of running a atmospheric dynamical simulation once (this would be regarded as deterministic), the simulation is run many different time with slightly different initial conditions. Due to the high computational resources required to run these simulations, they are often run at half the resolution of an equivalent deterministic simulation. The ensemble prediction system has a control simulation that doesn't have any perturbations to the initial conditions. Each simulation that makes up the system is called an ensemble member\cite{intro_efs}.

\subsection{Advantages of EPS}
\begin{figure}[H]
    \centering
    \includegraphics[width=.8\linewidth]{Images/efs.jpg}
    \caption{Visualisation of the Advantages of an Ensemble Prediction System}
    \label{efs}
\end{figure}

As a consequence of Chaos Theory, a tiny difference in the initial conditions in a large system, such as the atmosphere, can result in drastically different forecasted events, so that even with a tiny error, it can become a large error in the forecasted future state of the atmosphere. Even with the most accurate observations, error cannot be avoided. Therefore, it is simply not possible to make a better forecast or simulation. This is why an ensemble prediction system is utilised. In an ensemble simulation, small perturbations are made to the initial conditions, after which, the simulation is re-run. If there is a large degree of overlap between the ensemble members, there will be a higher degree of confidence in the ensemble forecast, with the opposite also holding true\cite{intro_efs}.

\subsection{Global EPS}
There are three distinct types of ensemble prediction systems: global, regional and convective-scale. Each system address different timescales, and different grid-sizes. A global ensemble prediction system is designed and used for medium-range forecasting between 3 and 15 days into the future. They use synoptic simulation models and are run at relatively low resolutions. Although they are primarily designed for use in the medium range, their global coverage means that they can also be used to provide short-range EPS forecasts in regions of the globe where no other EPS are currently available, and may be the only available option for certain countries. Considering the software is focused on synoptic scale simulations, this will be the system of interest\cite{intro_efs}, The default number of ensemble members in the software is fifteen, which is typical for a global ensemble prediction system. The grid-size, used by the software, can be specified by the end-user in order to get a more detailed forecast (this will increase the amount of computational resources required to run the simulation), however, the default grid-size is of the scale of 1000 km. There is also a hard limit of a $5^{\circ} \times 5^{\circ}$ cell, as a smaller cell size would result in inaccurate simulations due to the fact that the dynamical equations utilised by the software do not work on this scale. This will be discussed at greater depth in chapter \ref{4}.

\section{Recurrent Neural Network}
\subsection{Introduction}
Weather forecasting has traditionally been done by physical models of the atmosphere, which are unstable to perturbations, and thus are inaccurate for large periods of time\cite{why_rnn}. Since machine learning techniques are more robust to perturbations, it would be logical to combine a neural network with a physical model. Weather forecasting is a sequential data problem, therefore, a recurrent neural network is the most suitable option for this task.

\begin{definition}
A recurrent neural network is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence.
\end{definition}

Before, we delve into the specific example of using a recurrent neural network to predict the future state of the atmosphere, it is necessary to review what a recurrent neural network is. Recurrent Neural Networks (RNNs) are neural networks that are used in situations where data is presented in a sequence. For example, let's say you want to predict the future position of a fast-moving ball. Without information on the previous position of the ball, it is only possible to make an inaccurate guess. If you had, however, a large number of snapshots of the previous position, you are then able to predict the future position of the ball with some certainty. RNNs excel at modelling sequential data such as these. This is due to sequential memory.

In order to intuitively understand sequential memory, the prime example would be the alphabet. While it is easy to say the alphabet from A-Z, it is much harder to go from Z-A. There is a logical reason why this is difficult. As a child, you learn the alphabet in a sequence. Sequential memory is a mechanism that makes it easier for your brain to recognise sequence patterns.

In a traditional neural network, there is a input layer, hidden layer, and a output layer. In a recurrent neural network, a loop is added that can be added to pass information forward as seen in the diagram below (provided by Towards Data Science)\cite{intro_rnn}:

\begin{figure}[H]
    \centering
    \includegraphics[width=.2\linewidth]{Images/rnn.png}
    \caption{Visualisation of a Recurrent Neural Network}
\end{figure}

The information that is forwarded is the hidden layer, which is a representation of previous inputs. How this works in practise is that you initialise your network layers and the hidden the initial hidden state. The shape and dimension of the hidden state will be dependent on the shape and dimension of your recurrent neural network. Then you loop through your inputs, pass the relevant parameter and hidden state into the RNN. The RNN returns the output and a modified hidden state. Last you pass the output to the output layer, and it returns a prediction.

There is, however, a major problem known as short-term memory. Short-term memory is caused by something known as the vanishing gradient problem, which is also prevalent in other neural network architectures. As the RNN processes more steps, it has troubles retaining information from previous steps. Short-Term memory and the vanishing gradient is due to the nature of back-propagation. This can be comprehended through understanding how a neural network is trained\cite{intro_rnn}.

\begin{definition}
Back-propagation is an algorithm used to train and optimise neural networks.
\end{definition}

To train a recurrent neural network, you use an application of back-propagation called back-propagation through time. Training a neural network has three major steps. First, the relevant data vector is normalised between 0 and 1, the vector is feed into the RNN, and it goes through an activation function. The activation function utilised in the software is the rectified linear activation function\cite{lstm_rnn}.

\begin{definition}
The rectified linear activation function is a piece-wise linear function that will output the input directly if is positive, otherwise, it will output zero.
\end{definition}

The function is linear for values greater than zero, meaning it has a lot of the desirable properties of a linear activation function when training a neural network using back-propagation. Yet, it is a nonlinear function as negative values are always output as zero. As a result, the rectified function is linear for half of the input domain and nonlinear for the other half, it is referred to as a piece-wise linear function\cite{relu}. This nonlinear element is extremely important if the system has a nonlinear component, for example in predicting the evolution of the future state of the atmosphere.

\begin{figure}[H]
    \centering
    \includegraphics[width=.65\linewidth]{Images/relu.png}
    \caption{Sketch of the Rectified Linear Activation Function}
\end{figure}

Second, it outputs the results. Third, it compares the prediction to the ground truth using a loss function.

\begin{definition}
A loss function outputs an error value which is an estimate of how poorly the network is performing.
\end{definition}

The lost function that will be utilised in the software will be the function for mean squared error. The reason for choosing this particular function is that it heavily penalises large errors, as it squares the difference between the predicted and actual value. A large error in a weather forecast is highly undesirable, hence, the use of this function. The function is represented below:

\begin{equation}
    MSE = \frac{1}{n}\sum_{i=1}^n(Y_i-\hat{Y_i})^2
\end{equation}

If  a vector of $n$ predictions is generated from a sample of $n$ data points on all variables, and $Y$ is the vector of observed values of the variable being predicted, with $\hat{Y_i}$ being the predicted values.

\begin{definition}
Mean squared error is the average squared difference between the estimated values and the actual value.
\end{definition}

Returning to the training of the RNN, it uses that error value from the loss function. to do back propagation which calculates the gradients for each time step in the network. The gradient is the value used to adjust the networks internal weights, allowing the network to learn. The bigger the gradient, the bigger the adjustments and vice versa. Here is where the problem lies. When doing back propagation, the gradient of the current time step is calculated with respect to the effects of the gradients, in the time step before it. So if the adjustments to the time step before it is small, then adjustments to the current time step will be even smaller.  The gradient values will exponentially shrink as it propagates through each time step. That causes gradients to exponentially shrink as it back propagates down. The earlier layers fail to do any learning as the internal weights are barely being adjusted due to extremely small gradients.

Because of vanishing gradients, the RNN doesn’t learn the long-range dependencies across time steps. So not being able to learn on earlier time steps causes the network to have a short-term memory. In order to combat this, a long short-term memory is used\cite{intro_rnn}.

\subsection{LSTM}
LSTM's were created as a solution to the short-term memory problem. They have internal mechanisms called gates that can regulate the flow of information. These gates can learn which data in a sequence is important to keep or throw away. By doing that, it can pass relevant information down the long chain of sequences to make predictions. For example, if you were interested in buying a particular, you might read a review in order to determine if the purchase of the product is a good decision. When you read a review, your brain subconsciously only remembers important keywords. You pick up words like ``amazing", ``superb", or ``awful", you don't remember words such as "the", "as", or "because". This is what an LSTM does, it learns to keep only the relevant information to make predictions.

An LSTM has a similar control flow as a recurrent neural network. It processes data passing on information as it propagates forward. The differences are the operations within the LSTM’s cells. The core concept of LSTM’s are the cell state, and it’s various gates. The cell state is the method by which information is transferred down the sequence chain. The cell state, in theory, can carry relevant information throughout the processing of the sequence. So even information from the earlier time steps can make it’s way to later time steps, reducing the effects of short-term memory. As the cell state goes on its journey, information get’s added or removed to the cell state via gates\cite{lstm_rnn}.

\begin{definition}
A gate is an electric circuit with an output which depends on the combination of several inputs.
\end{definition}

Gates contain the sigmoid activation function. The sigmoid activation function squishes values between 0 and 1. That is helpful to update or forget data because any number getting multiplied by 0 is 0, causing values to disappears or be ``forgotten". Any number multiplied by 1 is the same value therefore that value stay’s the same or is ``kept".

\begin{figure}[H]
    \centering
    \includegraphics[width=.65\linewidth]{Images/sigmoid.png}
    \caption{Sketch of the Sigmoid Activation Function}
\end{figure}

There are three types of gates utilised within a neural network: a forget gate, an input gate, and an output gate. A forget gate decides what information should be thrown away or kept. Information from the previous hidden state and information from the current input is passed through the sigmoid function. An input gate is where the previous hidden state and current input into a sigmoid function. The output gate decides what the next hidden state should be. The hidden state is also used for predictions. First, we pass the previous hidden state and the current input into a sigmoid function. Then we pass the newly modified cell state to the rectified linear activation function. We multiply the rectified linear activation function output with the sigmoid output to decide what information the hidden state should carry. The output is the hidden state. The new cell state and the new hidden is then carried over to the next time step\cite{lstm_rnn}.

\subsection{Implementation}\label{implement_rnn}
The data set for the initial conditions (which is discussed in section \ref{noaa_initial_conditions}) consists of three features: geopotential height, air temperature, and relative humidity. For the purposes of this specific project, the RNN will solely be trained on air temperature and relative humidity. Unfortunately, due to the COVID-19, there was a time constraint on the developed of the RNN, which resulted in the inability to also train the RNN on geopotential height. This is due to the lack of computational resources at my disposable. The data set in question is updated every six hours by the National Oceanic and Atmospheric Administration. This means for a single day, there will be four observations. The goal for this project will be to, first predict the relevant atmospheric parameter in seven days time given the last thirty days of data and combine this RNN prediction with the physical model prediction in an attempt to make a more accurate prediction overall. In order to make such predictions, it is necessary to create a window of the last 120 ($30 \times 4$) observations to train the model\cite{time_series}.

At the start, a seed is set in order to ensure reproducibility. As mentioned previously, it is important to scale features before training a neural network. Normalisation is a common way of doing this scaling by subtracting the mean and dividing by the standard deviation of each feature. In order for the most optimal performance, the method ``MinMaxScaler" from the library, scikit-learn, is utilised within the software\cite{scikit-learn}. An LSTM requires a 1-dimensional sequence, however, the atmosphere is a 3-dimensional system. Hence, it is necessary to flatten the 3-dimensional vector that represents the state of the atmosphere. This is done in order to avoid the need of repeatably running the RNN. Batches are then created to split the data into manageable sequences. The diagram on the following page shows how the data is represented after flattening the data and batching it (provided by Tensorflow)\cite{time_series}.

\begin{figure}[H]
    \centering
    \includegraphics[width=.5\linewidth]{Images/data_rnn.png}
    \caption{Visualisation of how the data is represented after flattening and batching.}
\end{figure}

Following this process, the data is feed into the RNN. The LSTM model is built using Keras in TensorFlow, which is an free and open-source software library for machine learning. It was developed by the Google Brain Team\cite{tensorflow}. It is apparent that a multi-step model is needed as the model needs to learn to predict a range of future values. The source code for the LSTM model developed for the software is shown below:

\begin{minted}[mathescape,linenos,frame=lines]{python}
# Prepossessed historical data, which has been flatten and batched.
x_data, y_data = prepossessing_function(input_data)
# Prepossessed initial conditions, which has been flatten and batched.
initial_conditions = prepossessing_function(input_initialconditions)

# The network is shown data from the last 15 days.
past_history = 15 * 4

# The network predicts the next 7 days worth of steps.
future_target = 7 * 4

# Create, and train models.
# Optimiser.
opt = Adam(lr=1e-6, decay=1e-10, clipvalue=0.6)
# Create model.
model = Sequential()
model.add(
    LSTM(
        400, activation='relu', input_shape=(past_history, features)
    )
)
model.add(RepeatVector(future_target))
model.add(LSTM(400, activation='relu', return_sequences=True))
model.add(LSTM(400, activation='relu', return_sequences=True))
model.add(LSTM(400, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(features)))
model.compile(
    optimizer=opt, loss='mse', metrics=['mean_absolute_error']
)

# Train.
model.fit(
    x_data, y_data, epochs=epochs, batch_size=10
)

# Predict.
future_state = model.predict(initial_conditions)
# Invert normalisation, and flattening.
future_state = inverse_prepossessing(future_state)
\end{minted}

The model consists of four LSTM layers, which in combination are able to produce a more accurate and reliable prediction than a single LSTM layer. As is evident, the activation function for each LSTM is the rectified linear activation function, which is built into Keras. The number of epochs can be specified by the end user depending on the computational resources they have and what they need. More epochs will evidently lead to a more accurate neural network.

\begin{definition}
An epoch is one forward pass and one backward pass of all the training examples.
\end{definition}

\section{Initial Conditions}\label{noaa_initial_conditions}
\subsection{Global Data Assimilation System}
The initial conditions utilised by the software are from the Global Data Assimilation System (GDAS), which is provided by the National Oceanic and Atmospheric Adminstration in the United States. The primary reason for utilising this data is that it is freely available to the general public. In an ideal world, data from the European Centre for Medium-Range Weather Forecasts would be utilised, however unfortunately, there data is not freely available to the general public. This fundamentally violates the software's open source principles (these principles are discussed in chapter \ref{5}).

The GDAS is a model to place observations into a gridded model space for the purpose of initialising weather forecasts with observed data. This system is utilised by the National Center for Environmental Prediction for such a purpose. GDAS adds the following types of observations to a gridded, 3-D, model space: surface observations, balloon data, wind profiler data, aircraft reports, buoy observations, radar observations, and satellite observations\cite{gdas}. The initial conditions provided by the GDAS to the software have a vertical pressure co-ordinate, or the vertical co-ordiante is pressure. This co-ordinate system is known as isobaric co-ordinates.

\subsection{Isobaric Co-Ordinates}
In coordinate systems applied to the earth, the vertical coordinate describes position in the vertical direction (that is, parallel to the force of effective gravity). In meteorology, pressure can be a more convenient vertical coordinate than altitude. One reason is that until recently, radiosondes, which are the primary means of gathering observations of weather variables above the earth’s surface, measure and reported pressure, temperature, and humidity, but not altitude, as they rise through the atmosphere\cite{isobar_i}.

\begin{definition}
A radiosonde is an instrument carried by balloon to various levels of the atmosphere and transmitting measurements by radio.
\end{definition}

Another reason is that on scales large enough for the hydrostatic approximation to be valid (this is discussed in section \ref{hydro_balance}), the pressure-gradient force in the equations of motion becomes simpler and density no longer becomes an explicit variable in the tendency equations\cite{isobar_i}. Thus,a given geopotential gradient implies the same geostrophic wind at any height, whereas a given horizontal pressure gradient implies different values of the geostrophic wind depending on the density\cite{isobar_ii}.

From the Global Data Assimilation System, three prognostic variables are chosen: geopotential height, air temperature, and relative humidity.

\begin{definition}
Geopotential Height is the height above sea level of a pressure level. For example, if a station reports that the 500 hPa height at its location is 5600 m, it means that the level of the atmosphere over that station at which the atmospheric pressure is 500 hPa is 5600 meters above sea level.
\end{definition}

Geophysical sciences such as meteorology often prefer to express the horizontal pressure gradient force as the gradient of geopotential along a constant-pressure surface, because then it has the properties of a conservative force. For example, the primitive equations which weather forecast models solve use hydrostatic pressure as a vertical coordinate, and express the slopes of those pressure surfaces in terms of geopotential height. As such, this will be a parameter of great interest. From the aforementioned three selected parameters, any other parameter that is needed in the software can be calculated, including the wind. This will be discussed in greater depth in the next chapter.


\section{Computational Problems and Complexity}
\label{cc-sect}

We base this section on the definitions in Papadimitriou \cite{papad-cc},
to which we refer for further details.

A {\em deterministic Turing machine} $\mathcal{M}$ (we will imply the
``deterministic'' from now on) is a representation of an
algorithm that takes an {\em input}, runs a algorithm
manipulating the input on a {\em tape}, and returns an
{\em output} when it comes to a halting state (if it ever does).

The tape and its manipulation can be seen as follows: initially the tape
contains the input, given by a {\em first symbol} $\rhd$ followed by a
finite string of symbols $x\in (\Sigma \setminus \{ \sqcup \})^\ast$,
where $\Sigma$ is the {\em alphabet} of symbols of $\mathcal{M}$
and $\sqcup$ is a special
symbol representing the {\em blank} string.
The program is then carried on: a {\em cursor} starts at the first symbol
and follows the {\em algorithm} given by a
{\em transition function} $\delta$. The transition function depends on the
current {\em state} $p\in K$ and the current symbol $\sigma\in \Sigma$: the
former is the ``instruction'' to follow, the latter the symbol in the
position of the cursor. The transition function returns the next
state $q\in K$, the symbol $\tau\in \Sigma$ to be written at the position
of the cursor, and a direction in which the cursor will move on the tape.
The possible directions are ``left'', ``right'', or ``stay'', represented
respectively
by $\leftarrow,\rightarrow,-$. The first symbol is never
overwritten, and from there the cursor can only go left.

In some cases, the machine reaches the state
{\sc Yes} (the machine {\em accepts} the input), {\sc No} (the machine
{\em rejects} the input) or the {\em halting state} $h$
after a finite amount of time. The output
is then given by $\mathcal{M}(x)=\text{\sc Yes}$ or
$\mathcal{M}(x)=\text{\sc No}$ if the machine accepts or rejects the input.
If the halting state $h$ is reached, the output is $\mathcal{M}(x)=y$,
where $y$ is the finite (possibly empty) string of symbols that is left on
the tape after the symbol $\rhd$ and before any string of $\sqcup$'s at the
end. It is also possible that the machine doesn't halt on input $x$; we
denote this case as $\mathcal{M}(x)=\nearrow$.

A problem $P$ that requires an output that is either ``{\sc Yes}'' or
``{\sc No}'' is a {\em decision problem}; its {\em complement} is the
problem $\overline{P}$ that outputs ``{\sc No}'' for each instance
of $P$ that outputs ``{\sc Yes}'', and vice versa.
A problem where the output can be a string $\mathcal{M}(x)=y$ on the
output tape as above is a {\em function problem}; note that no guarantee
on the halting state is required. If the output is a
string $\mathcal{M}(x)=y$ satisfying a relation $R(x,y)$
or the input is rejected if it is not
possible to find any such string, the problem is a {\em search problem}.
A search problem where the input is never rejected is a {\em total
function problem}. By Theorem \ref{nash-thm}, the problem $n$-Nash of
Table \ref{n-nash} is a total function problem.

\begin{problem}
{$n$-Nash}
{A $n$-player game.}
{A Nash equilibrium of the game.}
\label{n-nash}
\end{problem}

Formally, a Turing machine is $\mathcal{M}=(K,\Sigma,\delta,s)$,
where $K$ and $\Sigma$ are finite, $\Sigma\cap K=\varnothing$ and $\Sigma$
always contains the symbols $\sqcup$ and $\rhd$.
The {\em transition function} is $\delta$, defined as follows,
where $\{ \leftarrow,\rightarrow,- \}\nsubseteq(K\cup\Sigma)$:
\[
\delta:K\times \Sigma\longrightarrow (K\cup\{ h,\ \text{{\sc Yes}},\ \text{{\sc No}} \})%
\times \Sigma \times \{ \leftarrow,\rightarrow,- \}.
\]

A slightly different, although equivalent, model allows for
{\em multiple strings}: $k\in \naturals$ cursors move
on $k$ strings $\sigma_i\in \Sigma^\ast$; the states are still
represented as $p\in K$. The input is given in the tape of the first string;
assuming the machine halts, the output is given in the tape of the $k$-th
string.

A {\em language} is a set of {\em strings of symbols}
$L\subseteq(\Sigma\setminus\{ \sqcup \})^\ast$; a {\em class} is a
set of languages. Let $\mathcal{M}$ be a Turing machine, and let
$x\in (\Sigma\setminus\{ \sqcup \})^\ast$.
We say that $\mathcal{M}$ {\em decides} $L$ if
either $\mathcal{M}(x)=\text{{\sc Yes}}$ if $x\in L$ or
$\mathcal{M}(x)=\text{{\sc No}}$
if $x\notin L$. A Turing
machine $\mathcal{M}$ {\em accepts} $L$ if
$x\in L$ is a necessary and sufficient condition for
$\mathcal{M}(x)=\text{{\sc Yes}}$, and if $x\notin L$ then
$\mathcal{M}(x)=\text{{\sc No}}$ or $\mathcal M$
does not halt.
Finally, $\mathcal{M}$ {\em computes} a function
$f:(\Sigma\setminus\{ \sqcup \})^\ast\to\Sigma^\ast$ if
$\mathcal{M}(x)=f(x)$ for every $x\in (\Sigma\setminus\{ \sqcup \})^\ast$.

Let $P_1$ be a problem and let $x$ be an instance of $P_1$ that is encoded
in $|x|$ bits. $P_1$ {\em reduces to the problem $P_2$ in polynomial time}
if there exists a function $f: \{0,1\}^\ast \to \{0,1\}^\ast$, a
Turing machine $\mathcal{M}$, and a polynomial $p$
such that for all $x\in\{0,1\}^\ast$
\begin{enumerate}
\item $x\in P_1\quad\iff\quad f(x)\in P_2$;
\item $\mathcal{M}$ computes $f(x)$;
\item $\mathcal{M}$ stops after $p(|x|)$ steps.
\end{enumerate}

For any class $\mathrm{C}$ of decision problems, the class of all complements
of the problems in $\mathrm{C}$ is the {\em complement class}
$\mathrm{co-C}$. A problem $P$ is {\em hard} for a class $\mathrm{C}$
if every problem in $\mathrm{C}$ is polynomial-time reducible to $P$; that is,
if $P$ is at least as hard to solve as every problem in $\mathrm{C}$. A
{\em complete} problem for the class $\mathrm{C}$ is a $\mathrm{C}-hard$ problem
that is also in $\mathrm{C}$.
Intuitively, if $P_1$ is polynomial-time reducible to $P_2$, it takes
polynomial time to ``translate'' $P_1$ to $P_2$, and then to
``translate back'' a solution of $P_2$ as a solution of $P_1$.
This is particularly useful if $P_2$ is hard: then
the problem $P_1$ is at least as ``difficult''. Furthermore, if the problem
$P_2$ is also complete in a class $\mathrm{C}$, it can be used as a test
of belonging to the class $C$.

The complexity class $\mathrm{\mathbf{P}}$ ({\em polynomial-time})
contains all the
{\em polynomially decidable problems}, that is, all problems $P$ such that
there exists a Turing machine $\mathcal{M}$ that outputs either {\sc Yes} or
{\sc No} for all inputs $x\in\{0,1\}^\ast$ of $P$ after $p(|x|)$ steps,
where $p$ is a polynomial.
A problem $P$ belongs to the class $\mathrm{\mathbf{NP}}$
({\em non-deterministic polynomial-time}) if there exists a
Turing machine $\mathcal{M}$ and polynomials $p_1,p_2$ such that
\begin{enumerate}
\item for all $x\in P$ there exists a {\em certificate} $y\in \{0,1\}^\ast$
such that \mbox{$|y|\leq p_1(|x|)$};
\item $\mathcal{M}$ accepts the combined input $xy$, stopping after at most
$p_2(|x| + |y|)$ steps;
\item for all $x\notin P$ there does not exist $y\in \{0,1\}^\ast$ such
that $\mathcal{M}$ accepts the combined input $xy$.
\end{enumerate}
An equivalent definition gives $\mathrm{\mathbf{NP}}$ as the class of all
languages $L$ for which the binary relation $R(x,y)$ such
that $L=\{ x\ |\ R(x,y) \text{ holds for some } y \}$ satisfies
the following conditions:
\begin{enumerate}
\item {\em (polynomially balanced)} if $(x,y)\in R$ then $|y|\leq |x|^k$ for
some $k\in \naturals$;
\item {\em (polynomially decidable)} if there is a Turing machine that
decides $L$ in polynomial time.
\end{enumerate}
Informally, a decision problem is in
$\mathrm{\mathbf{P}}$ if the answer to its question can be found in a
number of steps that is polynomial in the input of the problem, and
a decision problem is in $\mathrm{\mathbf{NP}}$ if it takes
polynomial time to verify whether the ``certificate'' $y$ is,
indeed, a correct answer to the question posed by the problem.
Notice that there may be many different certificates for each instance
of a problem. Consider for instance the $\mathrm{\mathbf{NP}}$ problem
{\sc Hamilton Path},
defined as ``given an input graph $x$, is it possible to find a Hamiltonian
path $y$ of $x$?'' There are graphs with more than one possible
Hamiltonian path, and each one of these can be used as certificate.

It is quite simple to see that $\mathbf{P}\subseteq \mathbf{NP}$, but it is
an important open
problem whether the inclusion is strict. If this were
not the case, it could be
argued that there isn't any substantial difference between finding a
solution and verifying its validity. This seems to contradict most of
the human intellectual experience, where the value
of an ``original'' discovery is perceived as fundamental; the
``conventional'' view is therefore that $\mathbf{P}\neq\mathbf{NP}$.
Another open problem is whether $\mathbf{NP} = \mathbf{co-NP}$; again,
this is widely believed to be false. Notice that if
$\mathbf{NP} \neq \mathbf{co-NP}$ then
also $\mathbf{P} \subsetneq \mathbf{NP}$, but not vice versa.

The classes $\mathbf{FP}$ and $\mathbf{FNP}$ are analogous to
$\mathbf{P}$ and $\mathbf{NP}$, respectively, but for function problems
instead of decision problems.
Formally, {\bf FNP} ({\em function non-deterministic polynomial-time})
is defined
in Megiddo and Papadimitriou \cite{megiddo-papad} as the class
of problems of the form ``given an $x\in\Sigma^\ast$,
find $y\in \Sigma^\ast$ such that $(x,y)\in R$, where R is a
polynomially balanced binary relation, or reject the input.''
$\mathbf{FP}$ ({\em function polynomial-time}) is the
class of all $\mathbf{FNP}$ problems that can be solved in polynomial time.
As in the case of decision problems, it is not known whether
$\mathbf{FP} = \mathbf{FNP}$; the question is actually equivalent
to whether $\mathbf{P} = \mathbf{NP}$.
Megiddo and Papadimitriou \cite{megiddo-papad} also define the class
$\mathbf{TFNP}$ ({\em total function non-deterministic polynomial-time})
as the class of all $\mathbf{FNP}$ problems where for every $x\in\Sigma^\ast$
a solution $y\in\Sigma^\ast$ is guaranteed to exist.
From another point of view, $\mathbf{TFNP} = \mathbf{F(NP\cap co-NP)}$,
and the existence of a $\mathbf{TFNP}$-complete problem would imply
that $\mathbf{NP} = \mathbf{co-NP}$.
The lack of $\mathbf{TFNP}$-complete problems has led to define
a number of new classes; we will see more of this in
section \ref{ppad-sect}.

The class $\mathrm{\mathbf{\# P}}$ is the class of all function
problems that output the
number of possible certificates
for a decision problem in $\mathrm{\mathbf{NP}}$.
Formally: the
{\em counting problem} associated with a binary
relation $Q(x,y)$ is ``given $x$, how many $y$ are there such
that $(x,y)\in Q$?'' Then $\mathrm{\mathbf{\# P}}$ is the class of
all
counting problems associated with binary relations that are both
polynomially
balanced and polynomially decidable.
It is interesting to notice that there are
$\mathrm{\mathbf{\# P}}$-complete
problems for which the corresponding search problem can
be solved
in polynomial time. An example close to the topic
of this thesis is given in Brightwell \cite{brightwell}: although finding
an Eulerian circuit in an
undirected graph takes polynomial time, counting the number of possible
circuits in the same graph is complete in $\mathrm{\mathbf{\# P}}$.

Finally, the class {\bf PSPACE} is the set of decision problems that can be
solved by a Turing machine using an amount of tape space that is polynomial
in the size of the input; although it can be proven that
$\mathbf{NP}\subseteq \mathbf{PSPACE}$, the possibility of an identity
is yet another open problem.


%
% STAT 100: Chance and Data Analysis - A Course Overview
% Section: Probability
%
% Author: Jeffrey Leung
%

\section{Probability}
	\label{sec:probability}
\subsection{Introduction}
	\label{subsec:probability:introduction}
\begin{easylist}

	& \emph{Experiment:} Process which creates a bag of outcomes
	& \emph{Sample space:} Set of all possible outcomes
		&& Denoted by $S$
	& \emph{Event:} Possible outcome which satisfies a given condition(s)
		&& Denoted by $E$

	& \emph{Probability:} Value between 0 and 1 (inclusive) representing the proportion of times an outcome/event is expected
		&& Denoted by $P(event)$
		&& Calculation:
		\begin{math}
			P(A) = \frac{|A|}{|S|}
		\end{math}
		where |X| is the number of outcomes for X
		&& 0 means the event is impossible; 1 means the event is certain
		&& Represented either by a fraction or a decimal value

		&& E.g. The probability of getting an odd number when rolling a 6-sided die:

		\medskip
		\Deactivate
		\begin{center}
			\begin{tabular}{ r @{ = } l }
				$S$ & Results of rolling a 6-sided die = 1, 2, 3, 4, 5, 6 \\
				$|S|$ & 6 \\
				$E$ & Getting an odd number = 1, 3, 5 \\
				$|E|$ & 3 \\
				$P(E)$ & $\frac{|E|}{|S|}$ = $\frac{3}{6}$ = 0.5
			\end{tabular}
			\Activate
		\end{center}

	& %TODO

\end{easylist}
\clearpage


\section{Reduced Order Modeling}
\label{sec:ROMs}
To provide a very simple idea of a ROM, assume that the final
response space of a physical system is governed by the transfer
function $H \left (  \overline{x}\right)$ (see Section
~\ref{sec:mathBackground}), which, from a practical point of
view, represents the outcome of the system based on the initial
conditions  $\overline{x}$. Now, sample the domain of variability of the
initial conditions $\overline{x}$ to create a
set of $N$ realizations of the input and response space $ \left ( \left (
\overline{x}_{i}, H \left (  \overline{x}_{i}\right) \right), i=1,N \right)$,
named ``training'' set. Based on the data set generated, it is possible
to construct a mathematical representation $G\left ( \overline{x}:
\overline{x}_{i}\right)$ of the
real system $H \left (  \overline{x}\right)$, which will approximate its
response (see Figure~\ref{fig:ROMexampleOfPhysicalSystem}):
\begin{equation}
\label{eq:regressor}
G\left ( \overline{x} \right ):\overline{x}_{i} \rightarrow G\left ( \overline{x}_{i} \right ) \cong H\left ( \overline{x}_{i} \right )
\end{equation}
The ROMs reported above are generally named ``regressors'', among
which all the most common data fitting algorithms are found (e.g.,
least square for construction of linear models).
An important class of ROMs for the work presented here after is the
one containing the so called ``classifiers''. A classifier is a ROM that is
capable of representing the system behavior from a binary point of
view (e.g., event happened/not happened or failure/success). It is a
model (set of equations) that identifies to which category an object
belongs in the feature (input) space. Referring to the example that
brought to Equation ~\ref{eq:regressor}, a classifier can be formally represented as follows (see
Figure~\ref{fig:ROMClassifierExampleOfPhysicalSystem}):
\begin{figure}[h!]
  \centering
  \includegraphics[width=1.0\textwidth]  {pics/ROMexampleOfPhysicalSystem.png}
  \caption{Example of reduced order model representation of physical system (regression).}
  \label{fig:ROMexampleOfPhysicalSystem}
\end{figure}
\begin{figure}[h!]
  \centering
  \includegraphics[width=1.0\textwidth]  {pics/ROMClassifierExampleOfPhysicalSystem.png}
  \caption{Example of reduced order model representation of physical system (classifier).}
  \label{fig:ROMClassifierExampleOfPhysicalSystem}
\end{figure}

\begin{equation}
\label{eq:classifier}
G\left ( \overline{x} \right ):\overline{x}_{i} \rightarrow G\left ( \overline{x}_{i} \right ) \cong
C \left ( H\left ( \overline{x}_{i} \right ) \right )
\end{equation}

The function $C\left (  H\left ( \overline{x}_{i}  \right ) = \overline{\theta}
\right ) $ is the so called ``goal'' function that is able to recast the
response of the system $H\left ( \overline{x}_{i}  \right )$ into a binary
form (e.g., failure/success). As an example, referring to
Figure~\ref{fig:ROMClassifierExampleOfPhysicalSystem}, the
``goal'' function would be:
\begin{equation}
\label{eq:goalFunctionClassifier}
C\left (   \overline{\theta}  \right ) = \left\{\begin{matrix}
1 & if \: \overline{\theta}>1.0 \\
0 &  if \: \overline{\theta} \leq 1.0
\end{matrix}\right.
\end{equation}
Hence, the ROM of type classifier $G\left (  \overline{x} \right )$  will operate in the space transformed through the ``goal''  function $C\left (   \overline{\theta}  \right )$.
\\The classifiers and regressors can be categorized into two main classes:
\begin{itemize}
  \item Model-based algorithms
  \item Data-based algorithms
\end{itemize}
In the first class, the created ROM aims to approximate the response
of the system as a function of the input parameters. These algorithms
construct a functional representation of the system. Examples of such ROM type are Support Vector Machines (SVMs), Kriging-based regressors, discriminant-based models, and polynomial chaos.

On the other side, data-based algorithms do not build a response-
function-based ROM but classify or predict the response of the
system from the neighborhood graph constructed from the training
data, without any dependencies on a particular prediction model.
These algorithms directly build a neighborhood structure as the
ROM (e.g., a relaxed Gabriel graph) on the initial training data. Examples of such ROM type are nearest neighbors and decision trees.

\textcolor{red}{\\It is important to NOTE that RAVEN uses a Z-score normalization of the training data before
  constructing most of the ROMs:
\begin{equation}
  \mathit{\mathbf{X}} = \frac{(\mathit{\mathbf{X}}-\mu )}{\sigma }
\end{equation}
 }
In order to identify which ROMs get trained with data normalized by the previous reported normalization approach, please refer to the RAVEN user manual \cite{RAVENuserManual}.

RAVEN has support of several different ROMs,
such as:
\begin{enumerate}
  \item \textit{Nearest Neighbors approaches}
  \item \textit{Support Vector Machines}
  \item \textit{Inverse Weight regressors}
  \item \textit{Spline regressors }, etc.
\end{enumerate}
In this section only few of them are going to be explained.

\subsection{Gaussian Process Models}
\label{sec:GPM}
Gaussian Processes (GPs)~\cite{Rasmussen_GPM} are algorithms that extend multivariate Gaussian distributions to infinite dimensionality. A Gaussian process generates a data set located throughout some domain such that any finite subset of the range follows a multivariate Gaussian distribution. Now, the n observations in an arbitrary data set, $y={y_1,\ldots,y_n}$, can always be imagined as a single point sampled from some multivariate ($n$-variate) Gaussian distribution.
What relates one observation to another in such cases is just the covariance function, $k(x,x')$. A popular choice is the squared exponential:
\begin{equation}
k(x,x')=\sigma_{f}^{2}  exp \left [ \frac{-(x-x')^2}{2 l^2} \right ]
\end{equation}
where the maximum allowable covariance is defined as $\sigma_{f}^{2}$; this should be high for functions that cover a broad range on the y axis. If $x \simeq x'$, then $k(x,x')$ approach this maximum meaning $f(x)$ is very correlated to $f(x')$. On the other hand, if $x$ is very distant from $x'$, then $k(x,x' ) \simeq 0$ (i.e., the two points cannot see each other.
So, for example, during interpolation at new $x$ values, distant observations will have negligible effect). How much effect this separation has will depend on the length parameter $l$.
Each observation $y$ can be thought of as related to an underlying function $f(x)$ through a Gaussian noise model:
\begin{equation}
y=f(x)+N(0,\sigma_{n}^{2})
\end{equation}
The new kernel function can be written as:
\begin{equation}
k(x,x')=\sigma_{f}^{2}  exp \left [ \frac{-(x-x')^2}{2 l^2} \right ] + \sigma_{n}^{2} \delta(x,x')
\end{equation}
So given $n$ observations $y$, the objective is to predict the value $y_*$ at the new point $x_*$. This process is performed by following this sequence of steps:
\begin{enumerate}
\item Calculate three matrices:
\begin{equation}
K=\begin{bmatrix}
k(x_1,x_1) &  \ldots & k(x_1,x_n)\\
\vdots  & \ddots &\vdots  \\
k(x_n,x_1) &  \ldots & k(x_n,x_n)
\end{bmatrix}
\end{equation}
\begin{equation}
K_*= \begin{bmatrix}
k(x_*,x_1) & \ldots & k(x_*,x_n)
\end{bmatrix}
\end{equation}
\begin{equation}
K_{**}=k(x_*,x_*)
\end{equation}
\item The basic assumption of GPM is that:
\begin{equation}
\begin{bmatrix}
y\\
y_*
\end{bmatrix}
=\mathcal{N}(0,\begin{bmatrix}
K & K_{*}^{T}\\
K_* & K_{**}
\end{bmatrix})
\end{equation}
\item The estimate $\bar{y_*} $ for $y_*$ is the mean of this distribution
\begin{equation}
\bar{y_*}=K_* K^{-1}y
\end{equation}
\item The uncertainty associated to the estimate $\bar{y_*} $ can be expressed in terms of variance of  $y_*$:
\begin{equation}
var(y_*)=K_{**}-k_* K^{-1} K_{*}^{T}
\end{equation}
\end{enumerate}

\subsection{Support Vector Machines}
\label{sec:SVM}
The Support Vector Machine (SVM)~\cite{SVM_Burges} classifier is a methodology that aims to determine the optimal separation hyperplane between data sets having different labels.
The training data consist of $N$ data points $(x_i,y_i)$ $i=1,\ldots,N$ where $x_i \in \mathbb{R}^M$ and $y_i \in {-1,1}$.
Assuming a linear property of the hyperplane  then its definition is:
\begin{equation}
\left \{ x: f(x)=x^T\beta+\beta_0=0 \right \}
\end{equation}
where $\beta$ is a unit vector.

The SVM parameters $\beta$ and $\beta_0$  are determined by solving this optimization problem:
\begin{equation}
\left\{\begin{matrix}
\underset{\beta,\beta_0}{min} \left \| \beta \right \|\\
\text{subject to } y_i(x_{i}^{T}\beta+\beta_0)\geq 1 , \quad i=1,\dots,N
\end{matrix}\right.
\end{equation}

Once the SVM parameters $\beta$ and $\beta_0$ are determined then the classification of a new point $\bar{x}$ is given by:
\begin{equation}
G(\bar{x})=sign(\bar{x}^T\beta+\beta_0)
\end{equation}

\subsection{KNN Classifier and KNR Regressor}
\label{sec:KNN_KNR}
The K Nearest Neighbor algorithm~\cite{altman_KNN} (KNN) is a non-parametric method used for both regression and classification. The only input parameter is the variable $K$ which indicates the number of neighbors to be considered in the classification/regression process. The special case where the class is predicted to be the class of the closest training sample (i.e. when $K = 1$) is called the nearest neighbor algorithm. In binary (two class) classification problems, it is helpful to choose k to be an odd number as this avoids tied votes. The output depends on whether KNN is used for classification or regression:
\begin{itemize}
\item In KNN classification, the output is a class membership. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its $K$ nearest neighbors ($K$ is a positive integer, typically small). If $K = 1$, then the object is simply assigned to the class of that single nearest neighbor.
\item In KNN regression, the output is the property value for the object. This value is the average of the values of its $K$ nearest neighbors.
\end{itemize}
Both for classification and regression, it can be useful to assign weight to the contributions of the neighbors, so that the nearer neighbors contribute more to the average than the more distant ones. For example, a common weighting scheme consists in giving each neighbor a weight of $1/d$, where $d$ is the distance to the neighbor.

\subsection{Multi-Dimensional Interpolation}
\label{sec:ND_interp}
This section covers the methods that have been implemented in the CROW statistical library:
\begin{itemize}
\item Shepard's Method (see Section~\ref{sec:shepard})
\item Multi-Dimensional Spline method (see Section~\ref{sec:ND_spline}).
\end{itemize}

These two methods are interpolation methods that can be used in any dimension.
In RAVEN they are employed in two major applications:
\begin{enumerate}
\item ROMs
\item Multi-dimensional distributions.
\end{enumerate}
For both applications, given a set of $N$ data points $ (x_i,u_i )$  $i=1,\ldots,N$ where $x_i$ are the coordinate in the input space $D \subset \mathbb{R}^M$ and $u_i \in \mathbb{R}$ is the outcome, the methods predicts the outcome $\tilde{u}$ for a new coordinate $\tilde{x}\in \mathbb{R}^n$.

\subsubsection{Shepard's Method}
\label{sec:shepard}
The Shepard interpolator~\cite{Shepard} is also know as Inverse Distance Weighting (IDW) interpolator.
The starting point is a set of $N$ data points $ (x_i,u_i )$ for $i=1,\ldots,N$.
The Inverse-Weight interpolator can be represented as a function $f_{IDW}(x)$ that, given a new coordinate in the input space $x$, generates a prediction on $u$ such that
\begin{equation}
u:x \in \mathbb{R}^M \rightarrow f_{IDW}(x) \in \mathbb{R}
\end{equation}
based on the distance $d(x,x_i)$ in the euclidean space between $x$ and $x_i$.

Such prediction $u=f_{IDW}(x)$ is performed by summing all data points $x_i$ $i=1,\ldots,N$ weighted by a weighting parameter $w_i (x)$ as follows:
\begin{equation}
f_{IDW}(x) =
\left\{
\begin{matrix}
\sum_{i=1}^{N} w(x_i) u_i &  \text{if } d(x,x_i) \neq 0 \\
 u_i &  \text{if } d(x,x_i) = 0
\end{matrix}\right.
\end{equation}
where
\begin{equation}
w(x_i) =\frac{w_i}{\sum_{i=1}^{N} w_i}
\end{equation}
and
\begin{equation}
w_i = \left ( \frac{1}{d(x,x_i)} \right )^p
\end{equation}
Large values of $p$ assign greater weight $w_i$ to data points $x_i$ closest to $x$, with the result turning into a mosaic of tiles (i.e., Voronoi diagram) with nearly constant interpolated value.

\subsubsection{Multi-Dimensional Spline}
\label{sec:ND_spline}
The Multi-Dimensional Spline (MDS)~\cite{MD_spline} is a method that requires the sampled points $x_i$ to be lying in multi-dimensional cartesian grid.
A generic grid $\Delta_m$ for each dimension $m$ will be indicated as follows:
\begin{equation}
\Delta_m = \{x_{0_m},x_{1_m},\ldots,x_{p_m}\} \text{ for } m=1,\ldots,M
\end{equation}
This methods construct a $M$-dimensional cubic spline so that, given a coordinate in the input space $x=(x_1,x_2,\ldots,x_M)$, generates a prediction on $u$ such that
\begin{equation}
u:x \in \mathbb{R}^M \rightarrow f_{MDS}(x) \in \mathbb{R}
\end{equation}
where
\begin{equation}
f_{MDS}(x)=\sum_{i_1=1}^{p_1+3} \sum_{i_2=1}^{p_2+3} \ldots \sum_{i_M=1}^{p_M+3} c_{i_1,i_2,\ldots,i_p} \prod_{m=1}^{M} u_{i_j} (x_m)
\end{equation}
where
\begin{equation}
u_{i_j} (x_m) = \Phi\left ( \frac{x_m-x_{0_m}}{h_j}+2-i_j  \right )
\end{equation}

The cubic kernel $\Phi(t)$ is defined as:
\begin{equation}
\Phi(t) = \left\{\begin{matrix}
(2-\left | t \right |)^3 & 1\leq \left | t \right |\leq 2 \\
4-6\left | t \right |^2+3\left | t \right |^3 & \left | t \right |\leq 1\\
0 & \text{elsewhere}
\end{matrix}\right.
\end{equation}

The set of $\prod_{m=1}^{M}(p_m+3)$ coefficients $c_{i_1,i_2,\ldots,i_p}$  is determined when the interpolator is initialized.


\chapter{Linear Models: Analysis of variance}
\label{ch:ANOVA}

Aims of this chapter \footnote{Here you work with the script file {\tt anova.R}}:
\begin{compactitem}
	\item Plotting boxplots and barplots using factors
	\item Fitting factors in linear models using analysis of variance
	\item Diagnostic plots for explanatory factors
	\item Exploring differences between levels of a factor
\end{compactitem}

\section{What is ANOVA?}

\begin{figure}
	\centering
	\includegraphics[width=.4\textwidth]{ANOVA_is_LM.pdf}

	\caption{A dataset where an ANOVA would be appropriate. performing an
	ANOVA test on this dataset is the same as fitting the linear model
	$y  = \beta_1  + \beta_2 x_s + \beta_3 x_a$, where $x_s$ and $x_a$
	are two levels. There are three ``treatments'' here with the first
	treatment, the control, captured by the baseline value $\beta_1$
	(the sample with the lowest value, on the far left)}

	\label{fig:anova1}

\end{figure}

A {\it One-way analysis of variance} (one-way ANOVA) is a technique
used to compare means of two or more samples representing numerical,
continuous data.

ANOVA tests the null hypothesis that samples from two or more groups
are drawn from populations with the {\it same mean value}. To do this,
ANOVA uses the {\it F}-statistic --- the ratio of the variance
calculated across the samples (groups) (the null hypothesis) to the
variance within the samples (groups). If the null hypothesis that the
group means are drawn from populations with the same mean is indeed
true, the between-group variance (numerator in the F-ratio) should be
lower than the within-group variance (denominator). A higher ratio (and
{\it F} value) therefore implies that the samples were drawn from
populations with different mean values.

This is same as asking whether a linear model with a predictor (or
explanatory variable) with at least two categorical levels (or
factors), better accounts for the variance (Explained Sum of Squares,
ESS) than a null model of the form $y  = \beta_1$ (Figure
\ref{fig:anova1}). Thus, ANOVA is just a type of linear model.

By the end of this chapter, it will make more sense to you how/why
linear regression models that we covered in Chapter \ref{ch:regress},
of the form $y = \beta_1  + \beta_2 x$ (where $x$ is a continuous
predictor variable),  require ANOVA to determine if the the model
better fits than a null model of the form $y  = \beta_1$.

Typically, one-way ANOVA is used to test for differences among at least
three groups, since the two-group (or levels or factors) case can be
covered by a $t$-test (see Chapter~\ref{ch:t_F_tests}). When there
are only two means to compare, the $t$-test and the F-test are
equivalent; the relation between ANOVA and t is given by $F = t^2$.

An extension of one-way ANOVA is two-way analysis of variance that
examines the influence of two different categorical independent
variables on one dependent variable --- we will look at multiple
predictor variables in Chapter \ref{ch:MulExpl} onwards.

\section{Calculating the ANOVA test statistic}

ANOVA ``partitions'' variability in your data as follows:

\begin{description}

	\item[Total sum of squares (TSS)] This is sum of the squared
	difference between the observed dependent variable ($y$) and the mean
	of the response variable $y$ (denoted by $\bar{y}$), i.e.,
	$$\text{TSS} = \sum_{i=1}^{n}(y_i - \bar{y})^2$$ TSS tells us how
	much variation there is in the dependent variable without having any
	other information (your null model). You might notice that TSS is the
	numerator of the sample variance you learned about in Chapter
	\ref{ch:ExpDesign}.

	\item [Explained sum of squares (ESS)] Sum of the squared differences
	between the predicted $y$'s (denoted $\hat{y}$'s) and $\bar{y}$, or,
	$$\text{ESS} = \sum_{i=1}^{n} (\hat{y}_i - \bar{y})^2$$ ESS tells us
	how much of the variation in the dependent variable our alternative
	(linear) model was able to explain. That is, it's the reduction in
	uncertainty that occurs when the linear model is used to predict the
	responses.

	\item [Residual sum of squares (RSS)] Sum of the squared differences
	between the observed $y$'s (denoted by $y_i$) and the predicted
	$\hat{y}$, or, $$\text{RSS} = \sum_{i=1}^{n} (\hat{y}_i - y_i)^2$$
	RSS tells us how much of the variation in the dependent variable our
	model could not explain. That is, it's the uncertainty that remains
	even after the linear model is used. The linear model is considered
	to be statistically significant if it can account for a large amount
	of variability in the response.

\end{description}

And of course, TSS = ESS + RSS; the OLS method ``decomposes'' the total variation in the dependent variable into an explained component (ESS; explained by the predictor) and an unexplained or residual component (the RSS).

These sums of squares can then be used to calculate the statistical
significance of the linear model (Regression, ANOVA, etc) through the
F-Value (or F-Ratio), as follows:

\begin{center}
\def\arraystretch{1.5}
\begin{tabular}{|>{\centering}m{2cm}|>{\centering}m{2.1cm}|>{\centering}m{2cm}|>{\centering}m{2cm}|p{1.5cm}|}
\hline
Type of Sum of Squares (SS)  & Calculation & Degrees of Freedom (DF) &
Mean Sum of Squares (MSS) & F-Value \\ \hline

TSS & $\sum_{i=1}^{n}(y_i - \bar{y})^2$   & $n-1$ &$\frac{TSS}{n-1}$&
\multirow{3}{*}{
$\frac{\left(\frac{ESS}{n_c-1}\right)}{\left(\frac{RSS}{n-n_c}\right)}$} \\ \cline{1-4}

ESS & $\sum_{i=1}^{n} (\hat{y}_i - \bar{y})^2$ & $n_c-1$ & $\frac{ESS}{n_c-1}$ &  \\ \cline{1-4}

RSS & $\sum_{i=1}^{n} (\hat{y}_i - y_i)^2$ &  $n-n_c$ &  $\frac{RSS}{n-n_c}$ & \\ \hline
\end{tabular}
\end{center}

\subsection{Degrees of freedom}
Thus each sum of squares has a corresponding degrees of freedom (DF)
associated with it that gives the Mean Sum of Squares (MSS)  --- the Sums of Squares divided by the corresponding degrees of freedom.

The TSS DF is one less than the number of observations $n-1$. This is because calculating TSS, needs $\bar y$ , which imposes loss of one degree of freedom. Note that MSS is thus nothing but the sample variance.

The ESS DF is one less than the number of coefficients ($n_c$)
(estimated parameters) in the model: $n_c-1$. Note that in the case
where the linear model is an ANOVA, it the number of coefficients
equals the number of ``treatments'' (the categories or levels in the
predictor). So for example, in Fig. \ref{fig:anova1}, there are three
treatments (predictors) and therefore three coefficients ($\beta_1$,
$\beta_2$, $\beta_3$), which means that the ESS degrees of freedom
there is $n_c-1 = 2$.

The RSS DF is the sample size $n$ minus the number of coefficients $n_c$, that is, $n - n_c$, because each estimated coefficient is an unknown parameter.

\subsection{The F-Value (or Ratio)}

Finally, The F-Value or F-Ratio, the test statistic used to decide
whether the linear model fit is statistically significant, is the ratio
of the Mean ESS to the Mean RSS. The null hypothesis is rejected if the
F-ratio is large --- the model explains a significant amount of
variance. The p-value is then calculated from the F-distribution as you
learned before, in Chapter \ref{ch:t_F_tests} (see Fig. \ref{fig:fdist}).

Also note that the Root Mean Square Error (RMSE), also known as the
standard error of the estimate, is the square root of the Mean RSS. It
is the standard deviation of the data about the Linear model, rather
than about the sample mean.

\subsection{The $R^{2}$}

Finally, $R^{2}$, also called the Coefficient of Determination, is the
proportion of total error (TSS) explained by the model (ESS), so the
ratio ESS/TSS. That is it is the proportion of the variability in the
response that is explained by  by the fitted model. Since TSS = ESS +
RSS, $R^{2}$ can be rewritten as (TSS-RSS)/TSS = 1 - RSS/TSS. If a
model has perfectly fits the data, $R^{2}=1$, and if it has no
predictive capability $R^{2}=0$. In
reality, $R^{2}$ will never be exactly 0 because even a null model will
explain some variance just by chance due to sampling error. Note that
$R$, the square root of $R^2$, is the multiple correlation coefficient:
the correlation between the observed values ($y$), and the predicted
values ($\hat{y}$).

As additional predictors (end therefore linear model coefficients) are
added to a linear model, $R^2$ increases even when the new predictors
add no real predictive capability. The adjusted-$R^2$ tries to addresses this
problem of over-specification or over-fitting by including the degrees
of freedom: Adjusted $R^2$ = 1 - (RSS/$n-n_c-2$)/(TSS/$n-1$)
\footnote{That is, it is 1 minus the ratio of the square of the
standard error of the estimate to the sample variance of the response}.
Thus additional predictors with little explanatory capability will increase
the ESS (and reduce the RSS), but they will also have lower RSS degrees of
freedom (because of the additional number of fitted coefficients,
$n_c$'s)\footnote{i.e., Standard error of the estimate won't
decrease}. Thus if the additional predictors have poor predictive
capability, these two reductions will cancel each other out. In other
words, the Adjusted $R^2$ penalizes the addition of new predictors to
the linear model, so you should always have a look at the Adjusted
$R^2$ as a corrected measure of $R^2$.

\section{A new dataset}

In this Chapter, we will use a new dataset of genome size and life
history in mammals to try out a one-way ANOVA. The dataset is a
composite of data taken from an online database of genome sizes and a
published database of mammalian life history:

\begin{compactdesc}
	\item[Genome size] Average genome sizes for available mammal species
	are taken from the online database
	\href{www.genomesize.com}{www.genomesize.com}.
	\item[Life history] Trait data for these species are taken from:
	\href{http://esapubs.org/archive/ecol/e090/184/metadata.htm}{
	Jones, K. E. {\it et al.} (2009) PanTHERIA: a species-level database
	of life history, ecology, and geography of extant and recently
	extinct mammals. Ecology 90, 2648--2648}.
\end{compactdesc}

\begin{compactitem}[$\quad\star$]
	\item Download the file {\tt MammalData.csv} from bitbucket and save
	to your {\tt Data} directory.
	\item Create a new blank script called {\tt ANOVA\_Prac.R} and add
	some introductory comments.
	\item Use {\tt read.csv} to load the data in the data frame
	{\tt mammals} and then {\tt str} and {\tt summary} to examine
	the data.
\end{compactitem}

There are nine variables. The first two are the latin binomial and
taxonomic order of each species, followed by the species mean genome
size (`C value', picograms), adult body mass (g), diet breadth, habitat
breadth, litter size and then two factors showing whether the species
are ground dwelling and their trophic level. For more information, see
the link above.

You will see from the output of {\tt summary} that there is lots of
missing data for the life history traits.

\section{Exploring the data with boxplots}

We are interested in finding out whether the mean C value for species
varies predictably for different levels of life history traits (a
typical one-way ANOVA question). For example:

\begin{compactitem}
\item Do carnivores or herbivores have larger genome sizes?
\item Do ground dwelling mammals have larger or smaller genome sizes?
\end{compactitem}

Before we fit any models, we want to plot the data to see if the means
within these groupings look different. We also want to check whether
the variance looks similar for each group: {\it constant normal
variance}! A simple way is to look at box and whisker plots, showing
the median and range of the data:

\begin{compactitem}[$\quad\star$]
	\item Use {\tt plot(meanCvalue \textasciitilde{} TrophicLevel,
	data= mammals)} to generate a boxplot of the differences in genome
	sizes between trophic levels.
	\item Looking at the plots, it is clear that there is more spread in
	the data above the median than below. Create a new variable {\tt
	logCvalue} in the {\tt mammals} data frame containing logged C
	values.
	\item Create a boxplot of log C values within trophic groups.
	\item Repeat the two plot commands to look at differences between
	ground dwelling and other species.
\end{compactitem}

\section{Differences in means with barplots}

Box and whisker plots show the median and spread in the data very
clearly, but we want to test whether the means are different. This is
$t$ test territory --- how different are the means given the standard
error --- so it is common to use barplots and standard error bars to
show these differences.

We're going to use some R code to construct a barplot by hand. We need
to calculate the means and standard errors within trophic groups, but
before we can do that, we need a new functions to calculate the
standard error of a mean:

\begin{lstlisting}

# get standard error of the mean from a set of values (x)

seMean <- function(x){
	x <- na.omit(x) # get rid of missing values

	se <- sqrt(var(x)/length(x)) # calculate the standard error

	return(se) 	# tell the function to return the standard error
}

\end{lstlisting}

Now we can use the function {\tt tapply}: it splits a variable up into
groups from a factor and calculates statistics on each group using a
function.
\begin{lstlisting}
trophMeans <- tapply(mammals$logCvalue, mammals$TrophicLevel, FUN =
mean, na.rm = TRUE)

print(trophMeans)

 Carnivore Herbivore  Omnivore
     1.085     1.197     1.236
\end{lstlisting}

\begin{lstlisting}
trophSE <- tapply(mammals$logCvalue, mammals$TrophicLevel, FUN = seMean)

print(trophSE)

 Carnivore Herbivore  Omnivore
   0.03983   0.02206   0.01844
\end{lstlisting}

Now we have to put these values together on the plot:

\begin{lstlisting}
# get the upper and lower limits of the error bars
upperSE <- trophMeans + trophSE
lowerSE <- trophMeans - trophSE

# get a barplot
# - this function can report where the middle of the bars are on the x-axis
# - set the y axis limits to contain the error bars

barMids <- barplot(trophMeans, ylim=c(0, max(upperSE)), ylab = 'log C value (pg)')

# Now use the function to add error bars
# - draws arrows between the points (x0,y0) and (x1,y1)
# - arrow heads at each end (code=3) and at 90 degree angles

arrows(barMids, upperSE, barMids, lowerSE, ang=90, code=3)

\end{lstlisting}

\begin{center}
	\includegraphics[width=0.7\textwidth]{TLbarplot.pdf}
\end{center}

Now we need to draw all these pieces together into a script and get
used to using them.
\begin{compactitem}[$\quad\star$]
	\item Copy all the lines of code from this section into your script.
	\item Run it and check you get the graph above.
	\item Use the second two chunks as a model to plot a similar graph
	for {\tt GroundDwelling}. You should get something like the plot
	below.
\end{compactitem}

\begin{center}
	\includegraphics[width=0.6\textwidth]{GDbarplot.pdf}
\end{center}

\section{An alternative to barplots}

That is a lot of work to go through for a plot. Doing it the hard way
uses some useful tricks, but one strength of R is that there is a huge
list of add-on packages that you can use to get new functions that
other people have written.

We will use the {\tt gplots} package to create  plots of group means
and confidence intervals. Rather than plotting the means $\pm$ 1
standard error, the option {\tt p=0.95} uses the standard error and the
number of data points to get 95\% confidence intervals. The default
{\tt connect=TRUE} option adds a line connecting the means, which isn't
useful here.

\begin{compactitem}[$\quad\star$]
	\item Replicate the code below into your script and run it to get the
	plots below.
\end{compactitem}

\begin{lstlisting}
#Load the gplots package
> library(gplots)

# Get plots of group means and standard errors
> par(mfrow=c(1,2))
> plotmeans(logCvalue ~ TrophicLevel, data=mammals, p=0.95, connect=FALSE)
> plotmeans(logCvalue ~ GroundDwelling, data=mammals, p=0.95, connect=FALSE)
\end{lstlisting}

\begin{center}
	\includegraphics[width = \textwidth]{test.pdf}
\end{center}

\section{Analysis of variance}

Hopefully, those plots should convince you that there are differences
in genome size between different trophic groups and between ground
dwelling and other mammals. We'll now use a linear model to test
whether those differences are significant.

\begin{compactitem}[$\quad\star$]
	\item Using your code from Chapter~\ref{ch:regress} as a guide,
	create a linear model called {\tt trophicLM} which models log C value
	as a function of trophic group.
	\item Use {\tt anova} and  {\tt summary} to look at the analysis of
	variance table and then the coefficients of the model.
\end{compactitem}

The ANOVA table for the model should look like the one below: trophic
level explains highly significant variation in genome size ($F= 7.22,
\textrm{df}=2 \textrm{ and } 300, p =0.0009$). {\it Note the style of
reporting the result} - the statistic ($F$), degrees of freedom and $p$
value are all provided in support. It is common to contract this style
to this: $F_{2,300}=7.22, p=0.0009$.
\begin{lstlisting}
> anova(trophicLM)

 Analysis of Variance Table

 Response: logCvalue
               Df Sum Sq Mean Sq F value  Pr(>F)
 TrophicLevel   2   0.83   0.413    7.22 0.00087 ***
 Residuals    300  17.18   0.057
 ---
 Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

\end{lstlisting}

However, look at the sum of squares column. Of a total of $17.18+0.83 =
18.01$ units of sums of squares, only 0.83 are explained by trophic
level: $0.83/18.01 \approx 0.046$ or 4.6\%. This ratio is called
$r^2$, a measure of explanatory power, and shows that, although the
model is very significant, it isn't very explanatory. We  care about
explanatory power or effect size, {\it not} $p$ values.

The coefficients table for the model looks like this:

\begin{lstlisting}
> summary(trophicLM)

 Call:
 lm(formula = logCvalue ~ TrophicLevel, data = mammals)

 Residuals:
     Min      1Q  Median      3Q     Max
 -0.5038 -0.1635 -0.0038  0.1511  0.9313

 Coefficients:
                       Estimate Std. Error t value Pr(>|t|)
 (Intercept)             1.0851     0.0335   32.38  < 2e-16 ***
 TrophicLevelHerbivore   0.1119     0.0396    2.83  0.00503 **
 TrophicLevelOmnivore    0.1513     0.0399    3.80  0.00018 ***
 ---
 Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

 Residual standard error: 0.239 on 300 degrees of freedom
   (76 observations deleted due to missingness)
 Multiple R-squared: 0.0459,	Adjusted R-squared: 0.0396
 F-statistic: 7.22 on 2 and 300 DF,  p-value: 0.000866

\end{lstlisting}

It shows the following:

\begin{compactitem}

	\item The reference level (or intercept) is for carnivores. Their
	mean genome size is significantly different from zero - this is not
	an exciting finding!

	\item The mean genome size for both herbivores and omnivores are both
	significantly different from carnivores. Both larger in fact:
	herbivore mean genome size = $1.085 + 0.112 = 1.197$ and omnivore
	mean genome size = $1.085 + 0.151 = 1.236$. These are the same group
	means we found above.

	\item The $r^2$ is shown and is the 4.6\% we calculated above. The
	{\it adjusted} $r^2$ reduces the raw $r^2$ to account for the number
	of variables included in the model. That 4.6\% would be even less
	impressive if we needed 6 explanatory variables to get it\ldots

	\item The $F$ statistic, as in the ANOVA table above.

\end{compactitem}

\begin{compactitem}[$\quad\star$]
	\item Repeat the analysis of variance above to look at the effects of
	ground dwelling on genome size.
\end{compactitem}

\section{Model criticism}

The next question must be  ---  and actually, we should do this before
we go anywhere near the model summaries --- is the model appropriate to
the data.

\begin{compactitem}[$\quad\star$]
	\item Using Chapter \ref{ch:regress} to guide you, get the four model
	diagnostic plots for the trophic level model on a single figure.
\end{compactitem}

The four plots are:
\begin{center}
	\includegraphics[width=0.8\textwidth]{modelDiag.pdf}
\end{center}

Note that in regression, the predicted (or fitted) values from the
model take a range along the relationship $y=a + bx$ (as we saw in the
Figures \ref{fig:DiagModDragon} \& \ref{fig:DiagModDamsel}). For ANOVA,
there are only a few predicted values --- one for each group mean. This
means that the plots above look different but we are looking for the
same things: is there constant variance at each fitted value and are
the residuals normally distributed? The answer for this model looks to
be yes.

\begin{compactitem}[$\quad\star$]
	\item Check the ground dwelling model in the same way.
\end{compactitem}

\section{Testing pairwise differences between levels}

The one thing that the trophic level model does not tell us is whether
there is a difference in genome size between omnivores and herbivores
--- both are compared to carnivores, but not to each other. This is
because of the multiple pairwise testing problem mentioned in Chapter
\ref{ch:t_F_tests} --- if you do
lots of tests then you may find small $p$ values by chance and say
something important is going on when it is just random chance. This is
called a false positive or Type I error.

With a 95\% confidence interval, there is a 5\% chance of a false
positive {\it per test} but there are ways of getting a 5\% chance
across a set (or family) of tests. For  linear models, we can use
Tukey's Honest Significant Difference test. We have to convert the {\tt
lm} object into an {\tt aov} object first.

\begin{lstlisting}

> TukeyTroph <- TukeyHSD(aov(trophicLM))
> print(TukeyTroph)

   Tukey multiple comparisons of means
     95% family-wise confidence level

 Fit: aov(formula = trophicLM)

 $TrophicLevel
                        diff      lwr    upr  p adj
 Herbivore-Carnivore 0.11186  0.01863 0.2051 0.0139
 Omnivore-Carnivore  0.15128  0.05741 0.2452 0.0005
 Omnivore-Herbivore  0.03942 -0.03161 0.1104 0.3923

\end{lstlisting}

The table shows the following:
\begin{compactitem}

	\item The differences between the three possible pairs and then the
	lower and upper bounds of the 95\% confidence interval for the
	difference and a $p$ value.

	\item In each case, we want to know if the difference could be zero:
	does the 95\% confidence interval for each pair include zero.

	\item For the first two pairs,  carnivores versus omnivores and
	herbivores, the confidence intervals do not include zero, so they are
	significantly different. For the comparison between herbivores and
	omnivores, the interval does include zero (difference = 0.039, 95\%
	CI's limits are -0.032 \& 0.110), so these groups are not
	significantly different.

	\item The $p$ values for the top two pairs are both larger (less
	significant) than in the summary table. The test has made it harder
	to find significant results.

\end{compactitem}

You can visualise these confidence intervals by plotting the Tukey
test. You have to tweak the graphics parameters to get a clean plot
though.

\begin{lstlisting}
> par(las=1, mar=c(4,10,3,1))
# las= 1 turns labels horizontal
# mar makes the left margin wider (bottom, left, top, right)
> plot(TukeyTroph)
\end{lstlisting}

The result should be:

{\centering \includegraphics[width=0.8\textwidth]{TukeyPLot.pdf} }

\begin{compactitem}[$\quad\star$]
	\item Run the Tukey test in your script for both the trophic level
	and ground dwelling models.
\end{compactitem}

\section{Are the factors independent?}

We've looked at two models, using trophic level and ground dwelling. It
is worth asking whether these are independent factors. What if, for
example, our herbivores are all big, ground dwellers? This is important
to know because otherwise, a two-way ANOVA would not be appropriate. We
will look at interactions in Chapter \ref{ch:MulExplInter}.

OK, so we want to know whether the two factors are independent. This is
a job for the $\chi^2$ test!

\subsection{The Chi-square test and count data}

The Chi-square test, also known as $\chi^{2}$ test or chi-square test,
is designed for scenarios where you want to statistically test how
likely it is that an observed distribution of values is due to chance.
It is also called a ``goodness of fit'' statistic, because it measures
how well the observed distribution of data fits with the distribution
that is expected if the variables of which measurements are made are
independent. In our mammals example below, the two variables are
trophic level and ground dwelling.

Note that a $\chi^{2}$ test is designed to analyze categorical data.
That is the data have been counted (count data) and divided into
categories. It is not meant for continuous data (such as body weight,
genome size, or height). For example, if you want to test whether
attending class influences how students perform on an exam, using test
scores (from 0-100) as data would not be appropriate for a Chi-square
test. However, arranging students into the categories ``Pass'' and
``Fail'' and counting up how many fall in each categories would be
appropriate. Additionally, the data in a Chi-square table (see below)
should not be in the form of percentages -- only count data are
allowed!

\subsubsection{The Chi-square test with the mammals data}

We can easily build a table for a Chi-square test on the mammals
data as follows:

\begin{lstlisting}
> factorTable <- table(mammals$GroundDwelling, mammals$TrophicLevel)
> print(factorTable)

       Carnivore Herbivore Omnivore
   No         26        45       64
   Yes        22        62       40
\end{lstlisting}

Now let's run the test:

\begin{lstlisting}
> chisq.test(factorTable)

 	Pearson's Chi-squared test

 data:  factorTable
 X-squared = 8.12, df = 2, p-value = 0.01725
\end{lstlisting}

The ``{\tt X-squared}'' value is the $\chi^{2}$ {\it test statistic}, akin to the
t-value of the t-test or W value in the Wilcox test.

The $\chi^{2}$ statistic is calculated as the sum of the quantity
$$ \frac{(\mathrm{Observed} - \mathrm{Expected})^2}{\mathrm{Expected}} $$
across all the cells/categories in the table (so the sum would be over
6 categories in our current mammals example).

``Observed'' is the observed proportion of data that fall in a
certain category. For example, there are 26 species observed in the
{\tt Carnivore}, {\tt No} category, and 22 in the {\tt Carnivore}, {\tt
Yes} category.

``Expected'' is what count would be expected if the values in each
category we truly independent. Each cell has its own expected value,
which is simply calculated as the count one would expect in each
category if the value were generated in proportion to the total number
seen in that category. So in our example, the expected value for the
{\tt Carnivore}, {\tt No} category would be
$$26+22 \mathrm{~(Total~number~of~carnivore~species)}
\times \frac{26+45+64 \mathrm{~(Total~number~in~the~''No''~category)}}{
26+22+45+62+64+40 \mathrm{~(Total~number~of~species)}}$$
$$= 48 \times \frac{135}{259} = 25.02$$

The sum of all six (one for each cell in the table above) such
calculations would be the $\chi^{2}$ value that R gave you through the
{\tt chisq.test()} above --- try it!

Now back to the R output from the {\tt chisq.test()} above. Why df = 2?
This is calculated as $DF = (r - 1) * (c - 1)$ where $r$ and $c$ are
the number of rows and columns in the $\chi^{2}$ table, respectively.
The same principle you learned before applies here; you lose one degree
of freedom for each new level of information you need to estimate:
there is uncertainity about the information (number of categories) in
both rows and columns, so you need to lose one degree of freedom for
each.

Finally, note that the p-value is significant --- we can conclude that the
factors aren't independent. From the table, carnivores can be either
ground dwelling or not, but herbivores tend to be ground dwelling and
omnivores tend not to be. Ah well... it's OK. We will look at a better
way to analyze these data using ``interactions'' in Chapter
\ref{ch:MulExplInter}.

\begin{compactitem}[$\quad\star$]
	\item Include and run the $\chi^2$ test in your script.
\end{compactitem}

\section{Saving data}

The last thing to do is to save a copy of the mammal data, including
our new column of log data, for use in later chapters.

\begin{compactitem}[$\quad\star$]
	\item Use this code in your script to create the saved data in you
	{\tt Data} directory :
\end{compactitem}

\begin{lstlisting}
save(mammals, file='../Data/mammals.Rdata')
\end{lstlisting}


\section{Graphs}
\begin{comment}
	Using adjacency list:
	\code{graphs/graph_template_adjlist.cpp}
	Using adjacency matrix:
	\code{graphs/graph_template_adjmat.cpp}
	Using edge list:
	\code{graphs/graph_template_edgelist.cpp}
\end{comment}
	\subsection{Single-Source Shortest Paths}
		\subsubsection{Dijkstra}
			\code{graphs/shortest_paths/dijkstra.cpp}
		\subsubsection{Bellman-Ford}
			\code{graphs/shortest_paths/bellman_ford.cpp}
    \subsubsection{Shortest Path Faster Algorithm}
      \code{graphs/shortest_paths/spfa.cpp}
	\subsection{All-Pairs Shortest Paths}
		\subsubsection{Floyd-Washall}
			\code{graphs/shortest_paths/floyd_warshall.cpp}
	\subsection{Strongly Connected Components}
		\subsubsection{Kosaraju}
      \code{graphs/scc/kosaraju.cpp}
    \subsubsectionRed{Tarjan's Offline Algorithm}
      \code{graphs/scc/tarjan.cpp}
  \subsectionRed{Minimum Mean Weight Cycle}
    Run this for each strongly connected component
    \code{graphs/min_mean_cycle.cpp}
	\subsection{Biconnected Components}
	  \subsubsection{Bridges and Articulation Points}
      \code{graphs/bcc/bridges_artics.cpp}
    \subsubsection{Block Cut Tree}
      \code{graphs/bcc/block_cut_tree.cpp}
		\subsubsection{Bridge Tree}
      \code{graphs/bcc/bridge_tree.cpp}
	\subsection{Minimum Spanning Tree}
		\subsubsection{Kruskal}
      \code{graphs/mst/kruskal.cpp}
		\subsubsection{Prim}
      \code{graphs/mst/prim.cpp}
	\subsectionRed{Euler Path/Cycle}
    \subsubsectionRed{Euler Path/Cycle in a Directed Graph}
      \code{graphs/euler_path.cpp}
    \subsubsectionRed{Euler Path/Cycle in an Undirected Graph}
      \code{graphs/euler_path_undirected.cpp}
	\subsectionRed{Bipartite Matching}
		\subsubsectionRed{Alternating Paths Algorithm}
      \code{graphs/bipartite_matching/bipartite_matching.cpp}
		\subsubsectionRed{Hopcroft-Karp Algorithm}
      \code{graphs/bipartite_matching/hopcroft_karp.cpp}
    \subsubsectionRed{Minimum Vertex Cover in Bipartite Graphs}
      \code{graphs/bipartite_matching/bipartite_mvc.cpp}
	\subsection{Maximum Flow}
		\subsubsectionBlack{Edmonds-Karp} $O(VE^2)$
\begin{comment}
			\code{graphs/max_flow/edmonds_karp.cpp}
\end{comment}
		\subsubsection{Dinic}
			$O(V^2E)$
			\code{graphs/max_flow/dinic.cpp}
		\subsubsection{Push-relabel}
			$\omega(VE+V^2\sqrt{E})$, $O(V^3)$
			\code{graphs/max_flow/push_relabel.cpp}
    \subsubsection{Gomory-Hu (All-pairs Maximum Flow)}
      $O(V^3E)$, possibly amortized $O(V^2E)$ with a big constant factor.
      \code{graphs/max_flow/gomory_hu_tree.cpp}
  \subsection{Minimum Cost Maximum Flow}
    \code{graphs/max_flow/mcst.cpp}
    \subsubsection{Hungarian Algorithm}
      \code{graphs/max_flow/hungarian.cpp}
  \subsectionRed{Minimum Arborescence}
    Given a weighted directed graph, finds a subset of edges of minimum
    total weight so that there is a unique path from the root $r$ to each
    vertex. Returns a vector of size $n$, where the $i$th element is the
    edge for the $i$th vertex. The answer for the root is undefined!
    \code{graphs/arborescence.cpp}
  \subsectionRed{Blossom algorithm}
    Finds a maximum matching in an arbitrary graph in $O(|V|^4)$ time. Be
    vary of loop edges.
    \code{graphs/blossom.cpp}
  \subsection{Maximum Density Subgraph}
    Given (weighted) undirected graph $G$. Binary search density. If $g$ is
    current density, construct flow network: $(S, u, m)$, $(u, T,
    m+2g-d_u)$, $(u,v,1)$, where $m$ is a large constant (larger than sum
    of edge weights). Run floating-point max-flow. If minimum cut has empty
    $S$-component, then maximum density is smaller than $g$, otherwise it's
    larger. Distance between valid densities is at least $1/(n(n-1))$. Edge
    case when density is $0$. This also works for weighted graphs by
    replacing $d_u$ by the weighted degree, and doing more iterations (if
    weights are not integers).
  \subsection{Maximum-Weight Closure}
    Given a vertex-weighted directed graph $G$. Turn the graph into a flow
    network, adding weight $\infty$ to each edge. Add vertices $S,T$. For
    each vertex $v$ of weight $w$, add edge $(S,v,w)$ if $w\geq 0$, or edge
    $(v,T,-w)$ if $w<0$. Sum of positive weights minus minimum $S-T$ cut is
    the answer. Vertices reachable from $S$ are in the closure. The
    maximum-weight closure is the same as the complement of the
    minimum-weight closure on the graph with edges reversed.
  \subsection{Maximum Weighted Ind. Set in a Bipartite Graph}
    This is the same as the minimum weighted vertex cover. Solve this by
    constructing a flow network with edges $(S,u,w(u))$ for $u\in L$,
    $(v,T,w(v))$ for $v\in R$ and $(u,v,\infty)$ for $(u,v)\in E$. The
    minimum $S,T$-cut is the answer. Vertices adjacent to a cut edge are
    in the vertex cover.
  \subsection{Synchronizing word problem}
    A DFA has a synchronizing word (an input sequence that moves all states
    to the same state) iff.\ each pair of states has a synchronizing word.
    That can be checked using reverse DFS over pairs of states. Finding the
    shortest synchronizing word is NP-complete.
  \subsection{Max flow with lower bounds on edges}
    % TODO: Test this!
    Change edge $(u,v,l\leq f\leq c)$ to $(u,v,f\leq c-l)$. Add edge
    $(t,s,\infty)$. Create super-nodes $S$, $T$. Let $M(u) = \sum_{v}
    l(v,u) - \sum_{v} l(u,v)$. If $M(u)<0$, add edge $(u,T,-M(u))$, else
    add edge $(S,u,M(u))$. Max flow from $S$ to $T$. If all edges from $S$
    are saturated, then we have a feasible flow. Continue running max flow
    from $s$ to $t$ in original graph.
    % TODO: Was there something similar for vertex capacities that we should add?
  \subsection{Tutte matrix for general matching}
    Create an $n\times n$ matrix $A$. For each edge $(i,j)$, $i<j$, let
    $A_{ij} = x_{ij}$ and $A_{ji} = -x_{ij}$. All other entries are $0$.
    The determinant of $A$ is zero iff.\ the graph has a perfect matching.
    A randomized algorithm uses the Schwartz--Zippel lemma to check if it is
    zero.
  \subsection{Heavy Light Decomposition}
    \code{graphs/heavy_light_decomposition.cpp}
	\subsectionRed{Centroid Decomposition}
    \code{graphs/centroid_decomposition.cpp}
	\subsection{Least Common Ancestor}
		\subsubsection{Binary Lifting}
      \code{graphs/lca/binary_lifting.cpp}
    \subsubsection{Euler Tour Sparse Table}
      \code{graphs/lca/lca_sparse_table.cpp}
    \subsubsection{Tarjan Off-line LCA}
      \code{graphs/lca/tarjan.cpp}
  \subsection{Counting Spanning Trees}
    Kirchoff's Theorem: The number of spanning trees of any graph is the
    determinant of any cofactor of the Laplacian matrix in $O(n^3)$.
    \begin{enumerate}
        \item Let $A$ be the adjacency matrix.
        \item Let $D$ be the degree matrix (matrix with vertex degrees on the diagonal).
        \item Get $D-A$ and delete exactly one row and column. Any row and
        column will do. This will be the cofactor matrix.
        \item Get the determinant of this cofactor matrix using Gauss-Jordan.
        \item $\text{Spanning Trees} = \left|\mathrm{cofactor}(D-A) \right|$
    \end{enumerate}
  \subsection{Erd\H{o}s-Gallai Theorem}
    A sequence of non-negative integers $d_1 \ge \cdots \ge d_n$ can be represented as the
    degree sequence of finite simple graph on $n$ vertices if and only if $d_1 + \cdots + d_n$ is
    even and the following holds for $1 \le k \le n$:
    \[
    \sum_{i=1}^n d_i \le k(k-1) + \sum_{i=k+1}^n \min\left(d_i, k\right)
    \]
  \subsection{Tree Isomorphism}
    \code{graphs/tree_isomorphism.cpp}


%!TEX root =  ../main.tex

\subsection{Difference Quotient}

\objective{Explain the various forms of the difference quotient and their meaning.}

To all appearances, limits seem to be about incredibly precise --- more precise
than anything in our universe --- or unmeasurably large inputs of functions.
This would seem to offer us nothing useful about the world we actually find ourselves
in.  Such is not the case, however.

If we want to know about the rate of change of a function, we must typically ask
``over what interval?''  We might suspect that there is a moment on the graph below
($x\approx.25$) where the average rate of change is 0.  How might we prove that?
We would have to travel some further amount in the $x$ direction, which would
produce some change in the $y$ value, because the rate of change is $\frac{\Delta y}
{\Delta x}$.

\begin{figure}
\begin{center}
\includegraphics[scale=0.6]{\chapdir/pics/dq.png}
% http://www.texample.net/tikz/examples/difference-quotient/
\caption{The difference quotient, as $\Delta x$ approaches 0\cite{tikzdq}.}
\end{center}
\end{figure}

In single-variable calculus, the difference quotient is usually the name for the expression
\begin{equation}
\lim_{h\rightarrow0}\frac {f(x+h)-f(x)}{(x+h)-x}
\end{equation}

What does this represent?  We are looking for the rate of change of the function.  We begin
by inputting $x$ (and, of course, receiving an output of $f(x)$).  We then move over a tiny
amount and measure the change in output, over the change in input.  That tiny amount is called
$h$, and we take the limit as $h$ goes to 0.

Because we are evaluating the entire function this way, if we are able to answer the limit,
we will receive a new function as out answer.  This function is called the \textbf{derivative} of
$f(x)$.

\begin{derivation}{Derivative}
The derivative of a function $f(x)$ may be denoted $f'(x)$.  This style of writing is
called \textbf{Lagrange's notation}, after Joseph-Louis Lagrange
\end{derivation}

\begin{figure}
\begin{center}
\includegraphics[scale=0.9]{\chapdir/pics/deriv.png}
\caption{The difference quotient produces a secant line \cite{derivativedefinition}.}
\end{center}
\end{figure}

Sometimes it is too difficult or time consuming to find the entire derivative equation.
We might want to find the derivative at one location only, typically called $c$

\begin{equation}
\lim_{x\rightarrow c}\frac{f(x)-f(c)}{x-c}
\end{equation}

These tend to be much easier to solve, as they are typically removable discontinuities
of the kind we solved in §2.1

\subsection{Theorems of Limits}
Let's us generalize the properties of limits we have seen in action this chapter:
\begin{enumerate}
\item The limit of a product is the same as a product of limits
\item The limit of a sum is the same as a sum of limits
\item The limit of a quotient is the same as a quotient of limits (but $\ne 0$)
\item The limit of a constant times a function is the same as a constant times a limit
\item the limit of a constant is a constant
\end{enumerate}