\documentclass[]{article}

\usepackage[english]{babel}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{float}
\usepackage{hyperref}

\title{Wind Prediction with LSTM}
\author{Juan Lao Tebar}

\begin{document}

\maketitle

\begin{abstract}

LSTM neural networks are powerful learning models, specially on long sequences with relations in long intervals of time. In this practice we analyze their behavior against the WIND dataset, implementing several Deep Learning techniques to improve the learning process and to prevent overfitting.

\end{abstract}

\section{Introduction}

In this practice we extend the experiments performed in the previous ``Guided Laboratory of the RNN Topic''\footnote{https://upc-mai-dl.github.io/rnn-lab-guided/}, not only predicting more steps and trying more complex topologies, but also testing different ways of preparing the data to exploit the true power of LSTM networks, learning the inherent data relations over long intervals of time.

As explained in the guided laboratory, the data for this example is a subset of the NREL Wind Integration National Dataset (WIND) Toolkit\footnote{https://www.nrel.gov/grid/wind-toolkit.html}. The original set consists of meteorological information for more than 126,000 sites across the USA from 2007 to 2013, sampled every 5 minutes. However, for our purposes, we only take into account the data from one site sampled every 15 minutes.

In the guided laboratory we took into account only one variable---wind speed at 100m---and we predicted the value it takes in the following step. In contrast, in this practice we predict the following 6 steps including the rest of the variables: air density, temperature and air pressure.

The workflow of this practice consists in defining a baseline model---similar to the one proposed in the guided laboratory---and improving it in the following experiments, step by step.

More in detail, the plan consists in the following:

\begin{enumerate}
	\item Study the effect of incrementing the number of hidden layers.
	\item Observe the response of the model for different activation functions.
	\item Find a better window size.
	\item If the model overfits the training data, apply dropout---on inter-layer connections and/or on LSTM gates.
	\item In case of experimenting explosive gradients, avoid them with gradient clipping.
	\item Present a new sequence-to-sequence model capable of learning from \emph{any-length} input samples.
\end{enumerate}

The code related to this work is public and available at github\footnote{https://github.com/juanlao7/WIND-LSTM}.

\section{Experiments}

The original data set consists of 245,472 measurements. In our process we use 122,736 of them for training---$\sim$3.5 years---, 61,368 for validation and 61,368 for testing---$\sim$1.75 years.

The loss of the model is measured with the mean squared error (MSE).

\subsection{Baseline}

Our first model is quite similar to the one proposed in the previous guided laboratory, as it consists of an LSTM layer of 512 units connected to a dense output layer of 24 units---6 predictions of 4 variables---, as it can be seen in figure \ref{f:model}. We use ReLU as the activation function of the LSTM units, the logistic function for their recurrent step and, of course, a linear function for the output neurons.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{model}
	\caption{Structure of the baseline model.}
	\label{f:model}
\end{figure}

We train the model in batches of 1000 samples using the adaptive optimizer RMSprop \cite{tieleman2012lecture}---recommended for training recurrent neural networks---with a learning rate of 0.0001, $ \rho = 0.9 $, $ \varepsilon = 1 \cdot 10^{-8} $ and no learning rate decay. The samples are obtained by \emph{lagging} the original samples of the data set with their previous 6 steps and foreseeing the following 6.

In this practice we change the stopping criteria of the training process, as we remove the limit of epochs and we implement a stopper based on the validation loss: when we detect a minimum value, we stop the training after 30 epochs if it does not decrease.

With this configuration we obtain the results shown in table \ref{t:baseline}.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}ccc@{}}
		\toprule
		Training loss & Validation loss & Test loss \\ \midrule
		0.02878       & 0.03151         & 0.03290   \\ \bottomrule
	\end{tabular}
	\caption{Results of the baseline model.}
	\label{t:baseline}
\end{table}

\subsection{Number of Layers}

Feedforward neural networks with two hidden layers can represent any decision boundary with arbitrary accuracy using the proper number of units, as well as approximating any smooth variable relation to any accuracy too.

For this reason, in this experiment we compare our previous results with the ones obtained with models of two and three hidden LSTM layers of 512 units each.

Table \ref{t:layers} shows the results obtained, while figure \ref{f:layers} shows the evolution of the training and validation loss in each case.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}cccc@{}}
		\toprule
		& Training loss & Validation loss & Test loss \\ \midrule
		1 hidden layer  & 0.02878       & 0.03151         & 0.03290   \\
		2 hidden layers & 0.02771       & 0.03088         & 0.03257   \\
		3 hidden layers & 0.02830       & 0.03207         & 0.03327   \\ \bottomrule
	\end{tabular}
	\caption{Results after testing different number of hidden layers.}
	\label{t:layers}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{layers}
	\caption{Validation and training loss with each number of hidden layers.}
	\label{f:layers}
\end{figure}

As expected, two hidden layers approximate better our data relations, while three hidden layers does not seem to improve the mark. In the following experiments we use an architecture of two hidden LSTM layers.

\subsection{Activation Function}

Clevert, Unterthiner, \& Hochreiter, 2016 \cite{clevert2015fast} found that using ELU instead of ReLU not only speeds up the learning process in deep neural networks, but also leads to higher classification accuracies: ``In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect''.

In this experiment we aim to reproduce similar results, testing different activation functions---softplus, logistic, hyperbolic tangent, ELU and ReLU---for the hidden LSTM layers of our model, but still using the logistic function for their recurrent step.

Table \ref{t:activation} shows the results obtained, while figure \ref{f:activation} shows the evolution of the training and validation loss in each case.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}ccccc@{}}
		\toprule
		& Training loss & Validation loss & Test loss & Epochs \\ \midrule
		Softplus           & 0.02972       & 0.03352         & 0.03485   & 288    \\
		Logistic           & 0.02986       & 0.03217         & 0.03348   & 622    \\
		Hyperbolic tangent & 0.02864       & 0.03133         & 0.03289   & 172    \\
		ELU                & 0.02856       & 0.03143         & 0.03251   & 186    \\
		ReLU               & 0.02771       & 0.03088         & 0.03257   & 206    \\ \bottomrule
	\end{tabular}
	\caption{Results after testing different activation functions.}
	\label{t:activation}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=\textwidth]{activation}
	\caption{Validation and training loss with each activation function.}
	\label{f:activation}
\end{figure}

As we can see, in our case ELUs and ReLUs are quite similar. We can see how ELUs take a slightly lesser number of epochs to converge, with almost the same test loss as with ReLUs but with higher training and validation losses.

In the following experiments we keep ReLU as the activation function for all the units of the hidden layers, since it is computationally cheap and we have not seen any considerable improvement on using ELU.

\subsection{Window Size}

Our data set is generated by going through the original samples with a window size of 12---6 measurements for the input + 6 for the output---. We must note that these measurements have been taken in intervals of 15 minutes, hence each input vector of the generated data set comprises an interval of 75 minutes.

In this experiment we test different window sizes, in order to determine if larger time intervals provide useful information for our model to obtain a lower loss.

Table \ref{t:window} shows the results obtained, while figure \ref{f:window} shows the evolution of the training and validation loss in each case.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}cccc@{}}
		\toprule
		& Training loss & Validation loss & Test loss \\ \midrule
		Input size of 1  & 0.03595       & 0.03743         & 0.03810   \\
		Input size of 6  & 0.02771       & 0.03088         & 0.03257   \\
		Input size of 12 & 0.02655       & 0.03190         & 0.03345   \\
		Input size of 24 & 0.02242       & 0.03655         & 0.03743   \\
		Input size of 48 & 0.01975       & 0.04247         & 0.04327   \\ \bottomrule
	\end{tabular}
	\caption{Results after testing different window sizes.}
	\label{t:window}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=\textwidth]{window}
	\caption{Validation and training loss with each window size.}
	\label{f:window}
\end{figure}

In this experiment we test input sizes up to 48. When we test higher values, at some point our gradients explode, loss becomes $ +\infty $ and the training process fails. There can be several reasons behind this behavior; we suspect that it is due to a combination of using ReLUs---which range is $ [0, +\infty) $---and a high learning rate. This problem can be easily solved just applying gradient clipping, but at this point we prefer just not to try bigger window sizes. Later, in this document, we will apply gradient clipping in another scenario.

Regarding to the results, as we can see, bigger windows lead to lower training losses. In fact, our model overfits the data when we train it with sequences of 24 or 48 inputs, as we can see how the validation loss increases while training loss decreases without any sign of convergence.

In the following experiments we continue working with the model trained with sequences of 6 inputs---the best test loss---and the model trained with sequences of 48 inputs---the best training loss.

\subsection{Dropout I}

Dropout is considered a simple and powerful regularization method. \cite{srivastava2014dropout} ``The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much. During training, dropout samples from an exponential number of different ``thinned'' networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights. This significantly reduces overfitting and gives major improvements over other regularization methods''.

In this experiment we apply different dropout factors---25\%, 50\% and 75\%---in each LSTM layer, not only at the inter-layer connections but also at the recurrent step.

Table \ref{t:drop1} shows the results obtained, while figures \ref{f:drop11} and \ref{f:drop12} show the evolution of the training and validation loss in each case.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}ccccc@{}}
		\toprule
		&      & Training loss             & Validation loss & Test loss       \\ \midrule
		Input size of 6  & 0\%  & 0.02771                   & 0.03088         & 0.03257         \\
		& 25\% & 0.14523                   & 0.09438         & 0.09005         \\
		& 50\% & 0.32983                   & 0.27392         & 0.25589         \\
		& 75\% & 0.63196                   & 0.55140         & 0.50822         \\
		\midrule
		Input size of 48 & 0\%  & 0.02231                   & 0.03864         & 0.04073         \\
		& 25\% & 40,982,524.64120          & 2,030,493.06455 & 2,327,123.48740 \\
		& 50\% & 0.29737                   & 0.26379         & 0.24811         \\
		& 75\% & 395,288,170,387,000.00000 & 191,750.24100   & 210,455.33092   \\ \bottomrule
	\end{tabular}
	\caption{Results after testing different dropout factors.}
	\label{t:drop1}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{drop11}
	\caption{Validation and training loss with each dropout factor, for input sizes of 6 samples.}
	\label{f:drop11}
\end{figure}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{drop12}
	\caption{Validation and training loss with each dropout factor, for input sizes of 48 samples.}
	\label{f:drop12}
\end{figure}

As we can see, on input sizes of 48 samples, applying a dropout factor makes gradients explode and loss takes very large values.

Gradient clipping prevents gradients from exploding. As we can see in table \ref{t:drop2} and figure \ref{f:drop2}, when all parameter gradients are clipped to a maximum norm of 1, we can successfully train our models while applying dropout.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}ccccc@{}}
		\toprule
		&      & Training loss & Validation loss & Test loss \\ \midrule
		Input size of 48 & 0\%  & 0.02248       & 0.03962         & 0.03925   \\
		& 25\% & 0.11694       & 0.09555         & 0.09036   \\
		& 50\% & 0.28648       & 0.27258         & 0.25334   \\
		& 75\% & 0.59369       & 0.51817         & 0.48055   \\ \bottomrule
	\end{tabular}
	\caption{Results after testing different dropout factors and applying gradient clipping.}
	\label{t:drop2}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{drop2}
	\caption{Validation and training loss with each dropout factor, for input sizes of 48 samples, applying gradient clipping.}
	\label{f:drop2}
\end{figure}

Gradient clipping has demonstrated to be a good method for avoiding exploding gradients, hence we will conserve it in the following experiments. On the other hand, we found that applying dropout at the inter-layer connections and at the recurrent step does not improve the performance of our model at all; on the contrary, it gets much worse.

\subsection{Dropout II}

Taking into account that LSTMs solve the vanishing gradient problem using an internal state controlled by gates, it does not sound right to apply dropout over those gates. Probably the bad results obtained in the previous experiment can be prevented just applying dropout at the inter-layer connections but not at the recurrent step.

In this experiment we apply different dropout factors---25\%, 50\% and 75\%---after each LSTM layer.

Table \ref{t:drop3} shows the results obtained, while figures \ref{f:drop31} and \ref{f:drop32} show the evolution of the training and validation loss in each case.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}ccccc@{}}
		\toprule
		&      & Training loss & Validation loss & Test loss \\ \midrule
		Input size of 6  & 0\%  & 0.02707       & 0.03102         & 0.03236   \\
		& 25\% & 0.03272       & 0.03107         & 0.03227   \\
		& 50\% & 0.04334       & 0.03215         & 0.03352   \\
		& 75\% & 0.06819       & 0.03617         & 0.03711   \\
		Input size of 48 & 0\%  & 0.02231       & 0.03864         & 0.04073   \\
		& 25\% & 0.03074       & 0.03451         & 0.03576   \\
		& 50\% & 0.03949       & 0.03425         & 0.03584   \\
		& 75\% & 0.06712       & 0.03815         & 0.03869   \\ \bottomrule
	\end{tabular}
	\caption{Results after testing different dropout factors, only between layers.}
	\label{t:drop3}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=\textwidth]{drop31}
	\caption{Validation and training loss with each dropout factor, for input sizes of 6 samples, only between layers.}
	\label{f:drop31}
\end{figure}

\begin{figure}[H]
	\centering
	\includegraphics[width=\textwidth]{drop32}
	\caption{Validation and training loss with each dropout factor, for input sizes of 48 samples, only between layers.}
	\label{f:drop32}
\end{figure}

As we can see, dropout does make any considerable improvement when the input size is 6---probably because that model was not overfitting the training data---. However, when the input size is 48, we can see how a dropout of 25\% successfully reduces the gap between training and validation losses, obtaining an overall lower test error.

In any case, this model still has a worse performance than the model with an input size of 6.

\subsection{Sequential Architecture}

LSTM networks can learn the inherent data relations over intervals of time of \emph{any} length; however, in our current architecture, we train and test the model with short subsequences generated with a small sliding window.

Why are we doing that? Is it possible to train the LSTM model with all the data set expressed as a single huge sequence? By doing that, the LSTM network will learn whatever it finds relevant, and we will not arbitrarily decide its scope; the model will find the optimum one. It may happen that it finds data relations over very large intervals of time---maybe weeks---, improving the overall performance of the system.

In our current architecture, if we try to process our data set with a huge window size, we find a physical problem: there is not enough RAM.

When we process the data set, we go through $ N $ samples of $ V $ variables with a window size of $ W $, generating $ N - W $ vectors of $ V \cdot (W + 1) $ variables each. Since we train in batches of 1000, we are working with $ 1000 \cdot V \cdot (W + 1) $ variables simultaneously.

The memory used by this architecture is directly related to the size of sequences we use for training, making it not scalable for training with very large sequences of samples. We would need large amounts of RAM to exploit the true power of LSTM networks.

To overcome this obstacle, in this experiment we present a new architecture consisting in a sequence-to-sequence model that gives one prediction for each input we feed to it (i.e. when we feed a sequence of $ N $ inputs, the model gives $ N $ outputs). This new architecture completely discards the sliding window process and directly feeds the original measurements to the model, targeting the next 6 measurements of the data set for each given input.

This new model uses an option defined by Keras framework in their LSTM layer implementation\footnote{https://keras.io/layers/recurrent/\#lstm}, the argument ``stateful'': ``Boolean (default False). If True, the last state for each sample at index $ i $ in a batch will be used as initial state for the sample of index $ i $ in the following batch''.

In this new architecture each element of the training sequences pertain to different batches, hence the RAM used by the training process does not depend on the sequences size anymore. Now the required RAM only depends on the batch size we choose.

Our current data set, since it is sequentially sorted by time, is not compatible with this new training process and we must make a conversion first. This conversion consists in reordering the measurements in such a way that, for a data set of $ N $ samples and a batch size of $ B $, samples from $ s_1 $ to $ s_{\frac{N}{B}} $ end up being the 1st element of every batch, in sequential order; then samples from $ s_{\frac{N}{B}+1} $ to $ s_{\frac{N}{B}+\frac{N}{B}} $ end up being the 2nd element of every batch and so on.

As we can see, with this conversion we are actually generating $ B $ different training sequences. We can decide their size: with a batch size of 1 we train our model with a single sequence of 122,736 samples; with a batch size of 4 we train our model 4 sequences of size 30,684.

In this experiment we leave the batch size as 1000---so we train with input sequences of 122 samples---, in order to compare the results with the ones obtained in the previous experiments. We prefer to leave the same batch size because it directly affects to the model’s generalization, and results may vary.

In this experiment we also clip all parameter gradients to a maximum norm of 0.000000001---very, very small---and even with that we still suffer from exploding gradients, as it can be seen in figure \ref{f:architecture1}. In addition, this strict clipping slows the learning process: note that the gradients explode after after 2964 epochs---1 hour and 39 minutes in our hardware---and the training loss is not even below 0.8. We could test lower clipping norms but then the training process would take an unfeasible amount of time to finish.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{architecture1}
	\caption{Exploding gradients when trying to train the sequential model.}
	\label{f:architecture1}
\end{figure}

Our gradients are exploding due to the usage of ReLU as activation function, since its range is $ [0, +\infty) $. If we replace it with the hyperbolic tangent---range $ (-1, 1) $---then we do not suffer this problem.

Table \ref{t:architecture2} shows the results obtained, while figure \ref{f:architecture2} shows the evolution of the training and validation loss in each case.

\begin{table}[H]
	\centering
	\begin{tabular}{@{}cccc@{}}
		\toprule
		& Training loss & Validation loss & Test loss \\ \midrule
		Sequential      & 0.03556       & 0.04878         & 0.05231   \\
		Input size of 6 & 0.02707       & 0.03102         & 0.03236   \\ \bottomrule
	\end{tabular}
	\caption{Results of the sliding window and the sequential model.}
	\label{t:architecture2}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{architecture2}
	\caption{Validation and training loss of the sliding window and the sequential model.}
	\label{f:architecture2}
\end{figure}

This new architecture may be useful for problems where the model needs to recall information far from the past (e.g., stock price prediction or paragraph understanding) but it does not offer any advantage over our previous architecture based on a sliding window. From the previous experiments, we already had the suspicion that all the relevant information for making a good prediction relies on the very short past.

\section{Discussion of the Results}

In this practice we confirmed several expectations but also found cases where established methodologies have no effect in our context.

For example, regarding to the network topology, since two hidden layers can approximate any smooth variable relation to any accuracy, we found that our model experiments an improvement when we go from one hidden layer to two, and we do not experiment any improvement when using three. We could apply some deep learning techniques, such as layer-wise pre-training deep autoencoders with several layers, but it was out of the scope of this RNN practice.

Regarding to the activation function, as found by Clevert et al, 2016 and confirmed by us in the prior DL-MAI practice, we expected to speed up the training process and obtain a lesser error with ELUs. However, in this case, the change had no effect.

We have the suspicion that the cause behind this behavior relies on the data we are working with: easy to approximate, no matter the activation function, but with a constant underlaying noise, impossible to predict with the variables we have. With more variables, this noise can be reduced.

The rest of the activation functions we tested (logistic, hyperbolic tangent, softplus) obtain similar results than ReLU, confirming our suspicion.

Now, regarding to specific RNN properties, we found that larger window sizes do not improve the performance of the system, even when regularizing overfited models using dropout. Input sequences of 6 steps are enough to predict with the maximum accuracy we found the following 6. In fact, we also found that our alternative architecture consisting in a sequence-to-sequence model trained with very long sequences is not able to beat that mark neither.

Finally, regarding to regularization, we tested dropout and we found that in our case it is better not to apply it on the recursive step of the network, since the network performance is dramatically reduced. We suspect that dropout may interfere with the underlying state of the LSTM units, causing them to forget important information. If the model overfits, in our case, it is better to apply dropout between layers.

\section{Conclusion and Future Work}

In this practice we extended the experiments performed in the previous ``Guided Laboratory of the RNN Topic'', predicting more steps, trying more complex topologies and testing a sequence-to-sequence model.

We presented and followed a plan of experiments that lead us to confirm and discard several hypothesis, as explained in the prior section.

Due to the nature of the data set and the results of the performed experiments, in this practice we did not test different optimizers or regularization methods. Due to lack of time, we have not tried our model against other sites either, and we did not check the variance of our error. It is recommendable to perform n-fold cross-validation instead of a single test.

Finally, it would be also interesting to find a data set where our sequence-to-sequence architecture may obtain better results than the sliding window process. As previously said, this model can be useful on cases where long-term relations have special importance, such as on stock pricing prediction or text understanding.

\bibliography{bibfile}{}
\bibliographystyle{plain}

\end{document}


\documentclass[11pt]{article}

\newcommand{\m}[1]{{\bf{#1}}}       % for matrices and vectors
\newcommand{\tr}{^{\sf T}}          % transpose

\topmargin 0in
\textheight 9in
\oddsidemargin 0pt
\evensidemargin 0pt
\textwidth 6.5in

%------------------------------------------------------------------------------
\begin{document}
%------------------------------------------------------------------------------

\title{User Guide for KLU and BTF}
\author{
Timothy A. Davis\thanks{
DrTimothyAldenDavis@gmail.com,
http://www.suitesparse.com.
This work was supported by Sandia National Labs, and the National
Science Foundation.
Portions of the work were done while on sabbatical at Stanford University
and Lawrence Berkeley National Laboratory (with funding from Stanford
University and the SciDAC program).
}
\and Eka Palamadai Natarajan}

\date{VERSION 1.3.8, May 4, 2016}
\maketitle

%------------------------------------------------------------------------------
\begin{abstract}
KLU is a set of routines for solving sparse linear systems of equations.
It is particularly well-suited to matrices arising in SPICE-like circuit
simulation applications.
It relies on a permutation to block triangular form (BTF), several methods
for finding a fill-reducing ordering (variants of approximate minimum degree,
and nested dissection), and a sparse left-looking LU factorization method
to factorize each block.  A MATLAB interface is included.
KLU appears as Collected Algorithm 907 of the ACM \cite{DavisNatarajan10}.
\end{abstract}
%------------------------------------------------------------------------------

\newpage
\tableofcontents
\newpage

%------------------------------------------------------------------------------
\section{License and Copyright}
%------------------------------------------------------------------------------

KLU, Copyright\copyright 2004-2011 University of Florida.
All Rights Reserved.
KLU is available under alternate licenses; contact T. Davis for details.

{\bf KLU License:} see KLU/Doc/License.txt for the license.

{\bf Availability:} {\tt http://www.suitesparse.com}

{\bf Acknowledgments:}

    This work was supported by Sandia National Laboratories (Mike Heroux)
    and the National Science Foundation under grants 0203270 and 0620286.

%------------------------------------------------------------------------------
\newpage
\section{Overview}
%------------------------------------------------------------------------------

KLU is a set of routines for solving sparse linear systems of equations.  It
first permutes the matrix into upper block triangular form, via the BTF
package.  This is done by first finding a permutation for a zero-free diagonal
(a maximum transversal) \cite{Duff81}. If there is no such permutation, then
the matrix is structurally rank-deficient, and is numerically singular.  Next,
Tarjan's method \cite{Duff78a,Tarjan72} is used to find the strongly-connected
components of the graph.  The block triangular form is essentially unique; any
method will lead to the same number and sizes of blocks, although the ordering
of the blocks may vary (consider a diagonal matrix, for example).  Assuming the
matrix has full structural rank, the permuted matrix has the following form:
\[
PAQ =
\left[
\begin{array}{ccc}
A_{11} & \cdots & A_{1k} \\
       & \ddots & \vdots \\
       &        & A_{kk} \\
\end{array}
\right],
\]
where each diagonal block is square with a zero-free diagonal.

Next, each diagonal block is factorized with a sparse left-looking method
\cite{GilbertPeierls88}.  The kernel of this factorization method is an
efficient method for solving $Lx=b$ when $L$, $x$, and $b$ are all sparse.
This kernel is used to compute each column of $L$ and $U$, one column at a time.
The total work performed by this method is always proportional to the number of
floating-point operations, something that is not true of any other sparse LU
factorization method.

Prior to factorizing each diagonal block, the blocks are ordered to reduce
fill-in.  By default, the symmetric approximate minimum degree (AMD) ordering
is used on $A_{ii}+A_{ii}^T$ \cite{AmestoyDavisDuff96,AmestoyDavisDuff03}.
Another ordering option is to find a column ordering via COLAMD
\cite{DavisGilbertLarimoreNg00_algo,DavisGilbertLarimoreNg00}.
Alternatively, a permutation can be provided by the user, or a pointer to
a user-provided ordering function can be passed, which is then used to order
each block.

Only the diagonal blocks need to be factorized.  Consider a linear system where
the matrix is permuted into three blocks, for example:
\[
\left[
\begin{array}{ccc}
A_{11} & A_{12} & A_{13} \\
       & A_{22} & A_{23} \\
       &        & A_{33} \\
\end{array}
\right]
\left[
\begin{array}{c}
x_{1} \\
x_{2} \\
x_{3} \\
\end{array}
\right]
=
\left[
\begin{array}{c}
b_{1} \\
b_{2} \\
b_{3} \\
\end{array}
\right].
\]

The non-singular system $A_{33} x_{3} = b_{3}$ can first be solved for $x_{3}$.
After a block back substitution, the resulting system becomes
\[
\left[
\begin{array}{cc}
A_{11} & A_{12} \\
       & A_{22} \\
\end{array}
\right]
\left[
\begin{array}{c}
x_{1} \\
x_{2} \\
\end{array}
\right]
=
\left[
\begin{array}{c}
b_{1} - A_{13} x_{3}\\
b_{2} - A_{23} x_{3}\\
\end{array}
\right]
=
\left[
\begin{array}{c}
b'_{1} \\
b'_{2} \\
\end{array}
\right]
\]
and the $A_{22} x_{2} = b'_{2}$ system can be solved for $x_{2}$.  The primary
advantage of this method is that no fill-in occurs in the off-diagonal blocks
($A_{12}$, $A_{13}$, and $A_{23}$).  This is particular critical for sparse
linear systems arising in SPICE-like circuit simulation
\cite{Kundert86,KundertSangiovanniVincentelli85,NagelPederson,Quarles:M89/42}.
Circuit matrices are typically permutable into block triangular form, with many
singletons (1-by-1 blocks).  They also often have a handful of rows and columns
with many nonzero entries, due to voltage and current sources.  These rows and
columns are pushed into the upper block triangular form, and related to the
singleton blocks (for example, $A_{33}$ in the above system is 1-by-1, and the
column in $A_{13}$ and $A_{23}$ has many nonzero entries).  Since these
nearly-dense rows and columns do not appear in the LU factorization of the
diagonal blocks, they cause no fill-in.

The structural rank of a matrix is based solely on the pattern of its entries,
not their numerical values.  With random entries, the two ranks are equal with
probability one.   The structural rank of any matrix is an upper bound on the
numerical rank.  The maximum transversal algorithm in the BTF package is useful
in determining if a matrix is structurally rank deficient, and if so, it
reveals a (non-unique) set of rows and columns that contribute to that rank
deficiency.  This is useful in determining what parts of a circuit are poorly
formulated (such as dangling components).

When ordered and factorized with KLU, very little fill-in occurs in the
resulting LU factors, which means that there is little scope for use of the
BLAS \cite{ACM679a}.  Sparse LU factorization methods that use the BLAS (such
as SuperLU \cite{SuperLU99} amd UMFPACK \cite{Davis03_algo,Davis03}) are slower
than KLU when applied to sparse matrices arising in circuit simulation.  Both
KLU and SuperLU are based on Gilbert and Peierl's left-looking method
\cite{GilbertPeierls88}.  SuperLU uses supernodes, but KLU does not; thus the
name {\em KLU} refers to a ``Clark Kent'' LU factorization algorithm (what
SuperLU was before it became Super).

For details of the permutation to block triangular form, left-looking sparse
LU factorization, and approximate minimum degree, refer to \cite{Davis06book}.
Concise versions of these methods can be found in the CSparse package.  KLU is
also the topic of a Master's thesis by Palamadai Natarajan \cite{Palamadai05};
a copy of
the thesis can be found in the {\tt KLU/Doc} directory.  It includes a
description of an earlier version of KLU; some of the function names and
parameter lists have changed in this version.  The descriptions of the methods
used still applies to the current version of KLU, however.

KLU appears as {\em Algorithm 907: KLU, a direct sparse solver for circuit
simulation problems}, ACM Transactions on Mathematical Software, vol 37, no 3,
2010.

%------------------------------------------------------------------------------
\section{Availability}
%------------------------------------------------------------------------------

KLU and its required ordering packages (BTF, COLAMD, AMD, and
SuiteSparse\_config) are
available at \newline {\tt http://www.suitesparse.com.}  In
addition, KLU can make use of any user-provided ordering function.  One such
function is included, which provides KLU with an interface to the ordering
methods used in CHOLMOD \cite{ChenDavisHagerRajamanickam06}, such as METIS, a
nested dissection method \cite{KarypisKumar98e}.  After permutation to block
triangular form, circuit matrices have very good node separators, and are thus
excellent candidates for nested dissection.  The METIS ordering takes much more
time to compute than the AMD ordering, but if the ordering is reused many times
(typical in circuit simulation) the better-quality ordering can pay off in
lower total simulation time.

To use KLU, you must obtain the KLU, BTF, SuiteSparse\_config,
AMD, and COLAMD packages
in the SuiteSparse suite of sparse matrix libraries.  See
{\tt http://www.suitesparse.com} for each of these packages.
They are also all contained within the single {\tt SuiteSparse.zip} or
{\tt SuiteSparse.tar.gz} distribution.

%------------------------------------------------------------------------------
\section{Using KLU and BTF in MATLAB}
%------------------------------------------------------------------------------

KLU has a single MATLAB interface which provides several options for factorizing
a matrix and/or using the factors to solve a linear system.  The following is
a synopsis of its use.  For more details, type {\tt help klu} in MATLAB.

{\footnotesize
\begin{verbatim}
   LU = klu (A)            factorizes R\A(p,q) into L*U+F, returning a struct
   x = klu (A,'\',b)       x = A\b
   x = klu (b,'/',A)       x = b/A
   x = klu (LU,'\',b)      x = A\b, where LU = klu(A)
   x = klu (b,'/',LU)      x = b/A, where LU = klu(A)
\end{verbatim}
}

With a single input {\tt klu(A)} returns a MATLAB struct containing the LU
factors.  The factorization is in the form \verb'L*U + F = R \ A(p,q)'
where {\tt L*U} is the LU factorization of just the diagonal blocks of the
block triangular form, {\tt F} is a sparse matrix containing the entries in
the off-diagonal blocks, {\tt R} is a diagonal matrix containing the row
scale factors, and {\tt p} and {\tt q} are permutation vectors.  The {\tt LU}
struct also contains a vector {\tt r} which describes the block boundaries
(the same as the third output parameter of {\tt dmperm}).  The {\tt k}th
block consists of rows and columns {\tt r(k)} to {\tt r(k+1)-1} in the
permuted matrix {\tt A(p,q)} and the factors {\tt L} and {\tt U}.

An optional final input argument ({\tt klu(A,opts)} for example) provides a
way of modifying KLU's user-definable parameters, including a partial pivoting
tolerance and ordering options.  A second output argument
({\tt [LU,info] = klu ( ... )}) provides statistics on the factorization.

The BTF package includes three user-callable MATLAB functions which replicate
most of features of the MATLAB built-in {\tt dmperm} function, and provide an
additional option which can significantly limit the worst-case time taken by
{\tt dmperm}.  For more details, type {\tt help btf}, {\tt help maxtrans},
and {\tt help strongcomp} in MATLAB.  Additional information about how
these functions work can be found in \cite{Davis06book}.

{\footnotesize
\begin{verbatim}
   [p,q,r] = btf (A)         similar to [p,q,r] = dmperm (A)
   q = maxtrans (A)          similar to q = dmperm (A')
   [p,r] = strongcomp (A)    similar to [p,q,r] = dmperm (A + speye(n))
\end{verbatim}
}

Both {\tt btf} and {\tt maxtrans} include a second option input, {\tt maxwork},
which limits the total work performed in the maximum transversal to
{\tt maxwork * nnz(A)}.  The worst-case time taken by the algorithm is
$O$ ({\tt n * nnz(A)}), where the matrix {\tt A} is {\tt n}-by-{\tt n}, but
this worst-case time is rarely reached in practice.

To use the KLU and BTF functions in MATLAB, you must first compile and install
them.  In the {\tt BTF/MATLAB} directory, type {\tt btf\_install}, and then
type {\tt klu\_install} in the {\tt KLU/MATLAB} directory.  Alternatively, if
you have the entire SuiteSparse, simply run the {\tt SuiteSparse\_install}
function in the {\tt SuiteSparse} directory.

After running the installation scripts, type {\tt pathtool} and save your path
for future MATLAB sessions.  If you cannot save your path because of file
permissions, edit your {\tt startup.m} by adding {\tt addpath} commands (type
{\tt doc startup} and {\tt doc addpath} for more information).

%------------------------------------------------------------------------------
\section{Using KLU and BTF in a C program}
\label{Cversion}
%------------------------------------------------------------------------------

KLU and BTF include the following C-callable functions.  Each function is
available in two or four versions: with {\tt int} or {\tt long} integers, and
(for functions that deal with numerical values), with {\tt float} or complex
{\tt float} values.  The {\tt long} integer is actually a {\tt SuiteSparse\_long},
which is typically a {\tt long}, defined with a {\tt \#define} statement.  It
becomes an {\tt \_\_int64} on Microsoft Windows 64, however.

The usage of real and complex, and {\tt int} and {\tt SuiteSparse\_long}, must not be
mixed, except that some functions can be used for both real and complex cases.

%------------------------------------------------------------------------------
\subsection{KLU Common object}
%------------------------------------------------------------------------------

The {\tt klu\_common} object ({\tt klu\_l\_common} for the {\tt SuiteSparse\_long}
version) contains user-definable parameters and statistics returned from
KLU functions.  This object appears in every KLU function as the last
parameter.  Details are given in the {\tt klu.h} include file, which also
appears below in Section~\ref{klu_include}.  Primary parameters and statistics
are summarized below.  The defaults are chosen specifically for circuit
simulation matrices.

\begin{itemize}
\item {\tt tol}: partial pivoting tolerance.  If the diagonal entry has a
magnitude greater than or equal to {\tt tol} times the largest magnitude
of entries in the pivot column, then the diagonal entry is chosen.
Default value: 0.001.

\item {\tt ordering}: which fill-reducing ordering to use: 0 for AMD,
1 for COLAMD, 2 for a user-provided permutation {\tt P} and {\tt Q}
(or a natural ordering if {\tt P} and {\tt Q} are NULL), or 3 for
the {\tt user\_order} function.  Default: 0 (AMD).

\item {\tt scale}: whether or not the matrix should be scaled.
If {\tt scale < 0}, then no scaling is performed and the input matrix
is not checked for errors.  If {\tt scale >= 0}, the input matrix is
check for errors.
If {\tt scale=0}, then no scaling is performed.
If {\tt scale=1}, then each row of {\tt A} is divided by the sum of
the absolute values in that row.
If {\tt scale=2}, then each row of {\tt A} is divided by the maximum
absolute value in that row.  Default: 2.

\item {\tt btf}:  if nonzero, then BTF is used to permute the input matrix
into block upper triangular form.  This step is skipped if {\tt Common.btf}
is zero.  Default: 1.

\item {\tt maxwork}: sets an upper limit on the amount of work performed in
{\tt btf\_maxtrans} to \newline {\tt maxwork*nnz(A)}.  If the limit is reached,
a partial zero-free diagonal might be found.  This has no effect on whether or
not the matrix can be factorized, since the matrix can be factorized with no
BTF pre-ordering at all.  This option provides a tradeoff between the
effectiveness of the BTF ordering and the cost to compute it.  A partial result
can result in fewer, and larger, blocks in the BTF form, resulting to more work
required to factorize the matrix.  No limit is enforced if {\tt maxwork <= 0}.
Default: 0.

\item {\tt user\_order}: a pointer to a function that can be provided by the
application that uses KLU, to redefine the fill-reducing ordering used by KLU
for each diagonal block.  The {\tt int} and {\tt SuiteSparse\_long} prototypes must be
as follows:

{\footnotesize
\begin{verbatim}
int my_ordering_function (int n, int Ap [ ], int Ai [ ], int Perm [ ], klu_common *Common) ;

SuiteSparse_long my_long_ordering_function (SuiteSparse_long n,
    SuiteSparse_long Ap [ ], SuiteSparse_long Ai [ ],
    SuiteSparse_long Perm [ ], klu_l_common *Common);
\end{verbatim}
}

The function should return 0 if an error occurred, and either -1 or a positive
(nonzero) value if no error occurred.  If greater than zero, then the return
value is interpreted by KLU as an estimate of the number of nonzeros in $L$ or
$U$ (whichever is greater), when the permuted matrix is factorized.  Only an
estimate is possible, since partial pivoting with row interchanges is performed
during numerical factorization.  The input matrix is provided to the function
in the parameters {\tt n}, {\tt Ap}, and {\tt Ai}, in compressed-column form.
The matrix {\tt A} is {\tt n}-by-{\tt n}.  The {\tt Ap} array is of size {\tt
n+1}; the {\tt j}th column of {\tt A} has row indices {\tt Ai[Ap[j] ...
Ap[j+1]-1]}.  The {\tt Ai} array is of size {\tt Ap[n]}.  The first column
pointer {\tt Ap[0]} is zero.  The row indices might not appear sorted in each
column, but no duplicates will appear.

The output permutation is to be passed back in the {\tt Perm} array, where
{\tt Perm[k]=j} means that row and column {\tt j} of {\tt A} will appear as
the {\tt k}th row and column of the permuted matrix factorized by KLU.  The
{\tt Perm} array is already allocated when it is passed to the user function.

The user function may use, and optionally modify, the contents of the {\tt
klu\_common Common} object.  In particular, prior to calling KLU, the user
application can set both {\tt Common.user\_order} and {\tt Common.user\_data}.
The latter is a {\tt void *} pointer that KLU does not use, except to pass to
the user ordering function pointed to by {\tt Common.user\_order}.  This is a
mechanism for passing additional arguments to the user function.

An example user function is provided in the {\tt KLU/User} directory, which
provides an interface to the ordering method in CHOLMOD.

\end{itemize}

%------------------------------------------------------------------------------
\subsection{KLU Symbolic object}
%------------------------------------------------------------------------------

KLU performs its sparse LU factorization in two steps.  The first is purely
symbolic, and does not depend on the numerical values.  This analysis returns a
{\tt klu\_symbolic} object ({\tt klu\_l\_symbolic} in the {\tt SuiteSparse\_long}
version).  The {\tt Symbolic} object contains a pre-ordering which combines the
block triangular form with the fill-reducing ordering, and an estimate of the
number of nonzeros in the factors of each block.  Its size is thus modest, only
proportional to {\tt n}, the dimension of {\tt A}.  It can be reused multiple
times for the factorization of a sequence of matrices with identical nonzero
pattern.  Note: a {\em nonzero} in this sense is an entry present in the data
structure of {\tt A}; such entries may in fact be numerically zero.

%------------------------------------------------------------------------------
\subsection{KLU Numeric object}
%------------------------------------------------------------------------------

The {\tt Numeric} object contains the numeric sparse LU factorization, including
the final pivot permutations.  To solve a linear system, both the {\tt Symbolic}
and {\tt Numeric} objects are required.

%------------------------------------------------------------------------------
\subsection{A sparse matrix in KLU}
%------------------------------------------------------------------------------

% From here on, only the {\tt int} version is described.  In the {\tt SuiteSparse\_long}
% version, the function names change slightly ({\tt klu\_factor} becomes
% {\tt klu\_l\_factor}, and the {\tt int}/complex version {\tt klu\_z\_factor}
% becomes {\tt klu\_zl\_factor}, for example).  For more details on the
% {\tt SuiteSparse\_long} version, refer to Section~\ref{klu_include}.

The input matrix provided to KLU is in sparse compressed-column form, which is
the same data structure used internally in MATLAB, except that the version used
here allows for the row indices to appear in any ordering, and this version
also allows explicit zero entries to appear in the data structure.  The same
data structure is used in CSparse, which is fully documented in
\cite{Davis06book}.  If you wish to use a simpler input data structure,
consider creating a triplet matrix in CSparse (or CXSparse if you use the long
and/or complex versions of KLU), and then convert this data structure into a
sparse compressed-column data structure, using the CSparse {\tt cs\_compress}
and {\tt cs\_dupl} functions.  Similar functions are available in CHOLMOD
{\tt cholmod\_triplet\_to\_sparse}.

The matrix is described with four parameters:

\begin{itemize}
\item {\tt n}: an integer scalar.  The matrix is {\tt n}-by-{\tt n}.  Note that
KLU only operates on square matrices.

\item {\tt Ap}: an integer array of size {\tt n+1}.  The first entry is {\tt
Ap[0]=0}, and the last entry {\tt nz=Ap[n]} is equal to the number of entries
in the matrix.

\item {\tt Ai}: an integer array of size {\tt nz = Ap[n]}.
The row indices of entries in column {\tt j} of {\tt A} are located in
{\tt Ai [Ap [j] ... Ap [j+1]-1]}.  The matrix is zero-based; row and column
indices are in the range 0 to {\tt n-1}.

\item {\tt Ax}: a {\tt float} array of size {\tt nz} for the real case, or
{\tt 2*nz} for the complex case.  For the complex case, the real and imaginary
parts are interleaved, compatible with Fortran and the ANSI C99 Complex data
type.  KLU does not rely on the ANSI C99 data type, however, for portability
reasons.  The numerical values in column {\tt j} of a real matrix are located
in {\tt Ax [Ap [j] ... Ap [j+1]-1]}.  For a complex matrix, they appear in {\tt
Ax [2*Ap [j] ... 2*Ap [j+1]-1]}, as real/imaginary pairs (the real part appears
first, followed by the imaginary part).

\end{itemize}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_defaults}: set default parameters}
%-------------------------------------------------------------------------------

This function sets the default parameters for KLU and clears the statistics.
It may be used for either the real or complex cases.  A value of 0 is returned
if an error occurs, 1 otherwise.  This function {\bf must} be called before
any other KLU function can be called.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok ;
    klu_common Common ;
    ok = klu_defaults (&Common) ;                                             /* real or complex */

    #include "klu.h"
    SuiteSparse_long ok ;
    klu_l_common Common ;
    ok = klu_l_defaults (&Common) ;                                           /* real or complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_analyze}: order and analyze a matrix}
%-------------------------------------------------------------------------------

The following usage returns a {\tt Symbolic} object that contains the
fill-reducing ordering needed to factorize the matrix {\tt A}.  A NULL pointer
is returned if a failure occurs.  The error status for this function, and all
others, is returned in {\tt Common.status}.  These functions may be used for
both real and complex cases.  The AMD ordering is used if {\tt Common.ordering
= 0}, COLAMD is used if it is 1, the natural ordering is used if it is 2, and
the user-provided {\tt Common.user\_ordering} is used if it is 3.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int n, Ap [n+1], Ai [nz] ;
    klu_symbolic *Symbolic ;
    klu_common Common ;
    Symbolic = klu_analyze (n, Ap, Ai, &Common) ;                             /* real or complex */

    #include "klu.h"
    SuiteSparse_long n, Ap [n+1], Ai [nz] ;
    klu_l_symbolic *Symbolic ;
    klu_l_common Common ;
    Symbolic = klu_l_analyze (n, Ap, Ai, &Common) ;                           /* real or complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_analyze\_given}: order and analyze a matrix}
%-------------------------------------------------------------------------------

In this routine, the fill-reducing ordering is provided by the user ({\tt
Common.ordering} is ignored).  Instead, the row permutation {\tt P} and column
permutation {\tt Q} are used.  These are integer arrays of size {\tt n}.  If
NULL, a natural ordering is used (so to provide just a column ordering, pass
{\tt Q} as non-NULL and {\tt P} as NULL).  A NULL pointer is returned if an
error occurs.  These functions may be used for both real and complex cases.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int n, Ap [n+1], Ai [nz], P [n], Q [n] ;
    klu_symbolic *Symbolic ;
    klu_common Common ;
    Symbolic = klu_analyze_given (n, Ap, Ai, P, Q, &Common) ;                 /* real or complex */

    #include "klu.h"
    SuiteSparse_long n, Ap [n+1], Ai [nz], P [n], Q [n] ;
    klu_l_symbolic *Symbolic ;
    klu_l_common Common ;
    Symbolic = klu_l_analyze_given (n, Ap, Ai, P, Q, &Common) ;               /* real or complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_factor}: numerical factorization}
%-------------------------------------------------------------------------------

The {\tt klu\_factor} function factorizes a matrix, using a sparse left-looking
method with threshold partial pivoting.  The inputs {\tt Ap} and {\tt Ai} must
be unchanged from the previous call to {\tt klu\_analyze} that created the {\tt
Symbolic} object.  A NULL pointer is returned if an error occurs.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int Ap [n+1], Ai [nz] ;
    float Ax [nz], Az [2*nz] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    Numeric = klu_factor (Ap, Ai, Ax, Symbolic, &Common) ;                            /* real */
    Numeric = klu_z_factor (Ap, Ai, Az, Symbolic, &Common) ;                          /* complex */

    #include "klu.h"
    SuiteSparse_long Ap [n+1], Ai [nz] ;
    float Ax [nz], Az [2*nz] ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    Numeric = klu_l_factor (Ap, Ai, Ax, Symbolic, &Common) ;                          /* real */
    Numeric = klu_zl_factor (Ap, Ai, Az, Symbolic, &Common) ;                         /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_solve}: solve a linear system}
%-------------------------------------------------------------------------------

Solves the linear system $Ax=b$, using the {\tt Symbolic} and  {\tt Numeric}
objects.  The right-hand side {\tt B} is overwritten with the solution on
output.  The array {\tt B} is stored in column major order, with a leading
dimension of {\tt ldim}, and {\tt nrhs} columns.  Thus, the real entry $b_{ij}$
is stored in {\tt B [i+j*ldim]}, where {\tt ldim >= n} must hold.  A complex
entry $b_{ij}$ is stored in {\tt B [2*(i+j*ldim)]} and {\tt B [2*(i+j*ldim)+1]}
(for the real and imaginary parts, respectively).  Returns 1 if successful,
0 if an error occurs.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ldim, nrhs, ok ;
    float B [ldim*nrhs], Bz [2*ldim*nrhs] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_solve (Symbolic, Numeric, ldim, nrhs, B, &Common) ;                      /* real */
    ok = klu_z_solve (Symbolic, Numeric, ldim, nrhs, Bz, &Common) ;                   /* complex */

    #include "klu.h"
    SuiteSparse_long ldim, nrhs, ok ;
    float B [ldim*nrhs], Bz [2*ldim*nrhs] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_l_solve (Symbolic, Numeric, ldim, nrhs, B, &Common) ;                    /* real */
    ok = klu_zl_solve (Symbolic, Numeric, ldim, nrhs, Bz, &Common) ;                  /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_tsolve}: solve a transposed linear system}
%-------------------------------------------------------------------------------

Solves the linear system $A^Tx=b$ or $A^Hx=b$.  The {\tt conj\_solve} input
is 0 for $A^Tx=b$, or nonzero for $A^Hx=b$.  Otherwise, the function is
identical to {\tt klu\_solve}.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ldim, nrhs, ok ;
    float B [ldim*nrhs], Bz [2*ldim*nrhs] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_tsolve (Symbolic, Numeric, ldim, nrhs, B, &Common) ;                     /* real */
    ok = klu_z_tsolve (Symbolic, Numeric, ldim, nrhs, Bz, conj_solve, &Common) ;      /* complex */

    #include "klu.h"
    SuiteSparse_long ldim, nrhs, ok ;
    float B [ldim*nrhs], Bz [2*ldim*nrhs] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_l_tsolve (Symbolic, Numeric, ldim, nrhs, B, &Common) ;                   /* real */
    ok = klu_zl_tsolve (Symbolic, Numeric, ldim, nrhs, Bz, conj_solve, &Common) ;     /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_refactor}: numerical refactorization}
%-------------------------------------------------------------------------------

The {\tt klu\_refactor} function takes as input the {\tt Numeric} object
created by {\tt klu\_factor} (or as modified by a previous call to {\tt
klu\_refactor}).  It factorizes a new matrix with the same nonzero pattern as
that given to the call to {\tt klu\_factor} which created it.  The same pivot
order is used.  Since this can lead to numeric instability, the use of {\tt
klu\_rcond}, {\tt klu\_rgrowth}, or {\tt klu\_condest} is recommended to check
the accuracy of the resulting factorization.  The inputs {\tt Ap} and {\tt Ai}
must be unmodified since the call to {\tt klu\_factor} that first created the
{\tt Numeric} object.  This is function is much faster than {\tt klu\_factor},
and requires no dynamic memory allocation.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok, Ap [n+1], Ai [nz] ;
    float Ax [nz], Az [2*nz] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_refactor (Ap, Ai, Ax, Symbolic, Numeric, &Common) ;                      /* real */
    ok = klu_z_refactor (Ap, Ai, Az, Symbolic, Numeric, &Common) ;                    /* complex */

    #include "klu.h"
    SuiteSparse_long ok, Ap [n+1], Ai [nz] ;
    float Ax [nz], Az [2*nz] ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_refactor (Ap, Ai, Ax, Symbolic, Numeric, &Common) ;                    /* real */
    ok = klu_zl_refactor (Ap, Ai, Az, Symbolic, Numeric, &Common) ;                   /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_free\_symbolic}: destroy the {\tt Symbolic} object}
%-------------------------------------------------------------------------------

It is the user's responsibility to destroy the {\tt Symbolic} object when it is
no longer needed, or else a memory leak will occur.  It is safe to pass a NULL
{\tt Symbolic} pointer.  These functions may be used for both real and complex
cases.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    klu_symbolic *Symbolic ;
    klu_common Common ;
    klu_free_symbolic (&Symbolic, &Common) ;                                  /* real or complex */

    #include "klu.h"
    klu_l_symbolic *Symbolic ;
    klu_l_common Common ;
    klu_l_free_symbolic (&Symbolic, &Common) ;                                /* real or complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_free\_numeric}: destroy the {\tt Numeric} object}
%-------------------------------------------------------------------------------

It is the user's responsibility to destroy the {\tt Numeric} object when it is
no longer needed, or else a memory leak will occur.  It is safe to pass a NULL
{\tt Numeric} pointer.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    klu_numeric *Numeric ;
    klu_common Common ;
    klu_free_numeric (&Numeric, &Common) ;                                            /* real */
    klu_z_free_numeric (&Numeric, &Common) ;                                          /* complex */

    #include "klu.h"
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    klu_l_free_numeric (&Numeric, &Common) ;                                          /* real */
    klu_zl_free_numeric (&Numeric, &Common) ;                                         /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_sort}: sort the columns of L and U}
%-------------------------------------------------------------------------------

The {\tt klu\_factor} function creates a {\tt Numeric} object with factors
{\tt L} and {\tt U} stored in a compressed-column form (not the same data
structure as {\tt A}, but similar).  The columns typically contain lists of
row indices in unsorted order.  This function sorts these indices, for two
purposes:  (1) to return {\tt L} and {\tt U} to MATLAB, which expects its
sparse matrices to have sorted columns, and (2) to slightly improve the
performance of subsequent calls to {\tt klu\_solve} and {\tt klu\_tsolve}.
Except within a MATLAB mexFunction (see {\tt KLU/MATLAB/klu\_mex.c}, the use
of this function is optional.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_sort (Symbolic, Numeric, &Common) ;                                      /* real */
    ok = klu_z_sort (Symbolic, Numeric, &Common) ;                                    /* complex */

    #include "klu.h"
    SuiteSparse_long ok ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_sort (Symbolic, Numeric, &Common) ;                                    /* real */
    ok = klu_zl_sort (Symbolic, Numeric, &Common) ;                                   /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_flops}: determine the flop count}
%-------------------------------------------------------------------------------

This function determines the number of floating-point operations performed
when the matrix was factorized by {\tt klu\_factor} or {\tt klu\_refactor}.
The result is returned in {\tt Common.flops}.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_flops (Symbolic, Numeric, &Common) ;                                     /* real */
    ok = klu_z_flops (Symbolic, Numeric, &Common) ;                                   /* complex */

    #include "klu.h"
    SuiteSparse_long ok ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_flops (Symbolic, Numeric, &Common) ;                                   /* real */
    ok = klu_zl_flops (Symbolic, Numeric, &Common) ;                                  /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_rgrowth}: determine the pivot growth}
%-------------------------------------------------------------------------------

Computes the reciprocal pivot growth,
$\mbox{\em rgrowth} = \min_j (( \max_i |c_{ij}| ) / ( \max_i |u_{ij}| ))$,
where $c_{ij}$ is a scaled entry in a diagonal block of the block triangular
form.  In MATLAB notation:
\begin{verbatim}
    rgrowth = min (max (abs (R\A(p,q) - F)) ./ max (abs (U)))
\end{verbatim}
where the factorization is \verb'L*U + F = R \ A(p,q)'.
This function returns 0 if an error occurred, 1 otherwise.  If {\tt rgrowth} is
very small, an inaccurate factorization may have been performed.  The inputs
{\tt Ap}, {\tt Ai}, and {\tt Ax}  ({\tt Az} in the complex case) must be
unchanged since the last call to {\tt klu\_factor} or {\tt klu\_refactor}.  The
result is returned in {\tt Common.rgrowth}.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok, Ap [n+1], Ai [nz] ;
    float Ax [nz], Az [2*nz] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_rgrowth (Ap, Ai, Ax, Symbolic, Numeric, &Common) ;                       /* real */
    ok = klu_z_rgrowth (Ap, Ai, Az, Symbolic, Numeric, &Common) ;                     /* complex */

    #include "klu.h"
    SuiteSparse_long ok, Ap [n+1], Ai [nz] ;
    float Ax [nz], Az [2*nz] ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_rgrowth (Ap, Ai, Ax, Symbolic, Numeric, &Common) ;                     /* real */
    ok = klu_zl_rgrowth (Ap, Ai, Az, Symbolic, Numeric, &Common) ;                    /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_condest}: accurate condition number estimation}
%-------------------------------------------------------------------------------

This function is essentially the same as the MATLAB {\tt condest} function.  It
computes an estimate of the 1-norm condition number, using Hager's method
\cite{Hager84} and the generalization by Higham and Tisseur
\cite{HighamTisseur00}.  The inputs {\tt Ap}, and {\tt Ax} ({\tt Az} in the
complex case) must be unchanged since the last call to {\tt klu\_factor} or
{\tt klu\_refactor}.  The result is returned in {\tt Common.condest}.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok, Ap [n+1] ;
    float Ax [nz], Az [2*nz] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_condest (Ap, Ax, Symbolic, Numeric, &Common) ;                           /* real */
    ok = klu_z_condest (Ap, Az, Symbolic, Numeric, &Common) ;                         /* complex */

    #include "klu.h"
    SuiteSparse_long ok, Ap [n+1] ;
    float Ax [nz], Az [2*nz] ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_condest (Ap, Ax, Symbolic, Numeric, &Common) ;                         /* real */
    ok = klu_zl_condest (Ap, Az, Symbolic, Numeric, &Common) ;                        /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_rcond}: cheap reciprocal condition number estimation}
%-------------------------------------------------------------------------------

This function returns the smallest diagonal entry of {\tt U} divided by the
largest, which is a very crude estimate of the reciprocal of the condition
number of the matrix {\tt A}.  It is very cheap to compute, however.
In MATLAB notation, {\tt rcond = min(abs(diag(U))) / max(abs(diag(U)))}.
If the matrix is singular, {\tt rcond} will be zero.  The result is returned
in {\tt Common.rcond}.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_rcond (Symbolic, Numeric, &Common) ;                                     /* real */
    ok = klu_z_rcond (Symbolic, Numeric, &Common) ;                                   /* complex */

    #include "klu.h"
    SuiteSparse_long ok ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_rcond (Symbolic, Numeric, &Common) ;                                   /* real */
    ok = klu_zl_rcond (Symbolic, Numeric, &Common) ;                                  /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_scale}: scale and check a sparse matrix}
%-------------------------------------------------------------------------------

This function computes the row scaling factors of a matrix and checks to see if
it is a valid sparse matrix.  It can perform two kinds of scaling, computing
either the largest magnitude in each row, or the sum of the magnitudes of the
entries each row.  KLU calls this function itself, depending upon the {\tt
Common.scale} parameter, where {\tt scale < 0} means no scaling, {\tt scale=1}
means the sum, and {\tt scale=2} means the maximum.  That is, in MATLAB
notation, {\tt Rs = sum(abs(A'))} or {\tt Rs = max(abs(A'))}.  KLU then divides
each row of {\tt A} by its corresponding scale factor.  The function returns 0
if the matrix is invalid, or 1 otherwise.  A valid sparse matrix must meet the
following conditions:

\begin{enumerate}
\item {\tt n > 0}.  Note that KLU does not handle empty (0-by-0) matrices.
\item {\tt Ap}, {\tt Ai}, and {\tt Ax} ({\tt Az} for the complex case) must not be NULL.
\item {\tt Ap[0]=0}, and {\tt Ap [j] <= Ap [j+1]} for all {\tt j} in the range 0 to {\tt n-1}.
\item The row indices in each column, {\tt Ai [Ap [j] ... Ap [j+1]-1]}, must be in
the range 0 to {\tt n-1}, and no duplicates can appear.  If the workspace {\tt W} is
NULL on input, the check for duplicate entries is skipped.
\end{enumerate}

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int scale, ok, n, Ap [n+1], Ai [nz], W [n] ;
    float Ax [nz], Az [2*nz], Rs [n] ;
    klu_common Common ;
    ok = klu_scale (scale, n, Ap, Ai, Ax, Symbolic, Numeric, &Common) ;               /* real */
    ok = klu_z_scale (scale, n, Ap, Ai, Az, Symbolic, Numeric, &Common) ;             /* complex */

    #include "klu.h"
    SuiteSparse_long scale, ok, n, Ap [n+1], Ai [nz], W [n] ;
    float Ax [nz], Az [2*nz], Rs [n] ;
    klu_l_common Common ;
    ok = klu_l_scale (scale, n, Ap, Ai, Ax, Symbolic, Numeric, &Common) ;             /* real */
    ok = klu_zl_scale (scale, n, Ap, Ai, Az, Symbolic, Numeric, &Common) ;            /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_extract}: extract the LU factorization}
%-------------------------------------------------------------------------------

This function extracts the LU factorization into a set of data structures
suitable for passing back to MATLAB, with matrices in conventional
compressed-column form.  The {\tt klu\_sort} function should be called first if
the row indices should be returned sorted.  The factorization is returned in
caller-provided arrays; if any of them are NULL, that part of the factorization
is not extracted (this is not an error).  Returns 1 if successful, 0 otherwise.

The sizes of {\tt Li}, {\tt Lx}, and {\tt Lz} are {\tt Numeric->lnz},
{\tt Ui}, {\tt Ux}, and {\tt Uz} are of size {\tt Numeric->unz}, and
{\tt Fi}, {\tt Fx}, and {\tt Fz} are of size {\tt Numeric->nzoff}.
Note that in the complex versions, the real and imaginary parts are returned
in separate arrays, to be compatible with how MATLAB stores complex matrices.

This function is not required to solve a linear system with KLU.  KLU does not
itself make use of the extracted LU factorization returned by this function.
It is only provided to simplify the MATLAB interface to KLU, and it may be of
use to the end user who wishes to examine the contents of the LU factors.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    int ok, Lp [n+1], Li [lnz], Up [n+1], Ui [unz], Fp [n+1], Fi [nzoff], P [n], Q [n], R [n] ;
    float Lx [lnz], Lz [lnz], Ux [unz], Uz [unz], Fx [nzoff], Fz [nzoff], Rs [n] ;
    klu_symbolic *Symbolic ;
    klu_numeric *Numeric ;
    klu_common Common ;
    ok = klu_extract (Numeric, Symbolic,
        Lp, Li, Lx, Up, Ui, Ux, Fp, Fi, Fx, P, Q, Rs, R, &Common) ;                   /* real */
    ok = klu_z_extract (Numeric, Symbolic,
        Lp, Li, Lx, Lz, Up, Ui, Ux, Uz, Fp, Fi, Fx, Fz, P, Q, Rs, R, &Common) ;       /* complex */

    #include "klu.h"
    SuiteSparse_long ok, Lp [n+1], Li [lnz], Up [n+1], Ui [unz], Fp [n+1],
        Fi [nzoff], P [n], Q [n], R [n] ;
    float Lx [lnz], Lz [lnz], Ux [unz], Uz [unz], Fx [nzoff], Fz [nzoff], Rs [n] ;
    klu_l_symbolic *Symbolic ;
    klu_l_numeric *Numeric ;
    klu_l_common Common ;
    ok = klu_l_extract (Numeric, Symbolic,
        Lp, Li, Lx, Up, Ui, Ux, Fp, Fi, Fx, P, Q, Rs, R, &Common) ;                   /* real */
    ok = klu_zl_extract (Numeric, Symbolic,
        Lp, Li, Lx, Lz, Up, Ui, Ux, Uz, Fp, Fi, Fx, Fz, P, Q, Rs, R, &Common) ;       /* complex */
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt klu\_malloc}, {\tt klu\_free}, {\tt klu\_realloc}:
memory management}
%-------------------------------------------------------------------------------

KLU uses a set of wrapper routines for {\tt malloc}, {\tt free}, and {\tt
realloc}.  By default, these wrapper routines call the ANSI C versions of these
functions.  However, pointers to functions in {\tt Common} can be modified
after calling {\tt klu\_defaults} to allow the use of other memory management
functions (such as the MATLAB {\tt mxMalloc}, {\tt mxFree}, and {\tt
mxRealloc}.  These wrapper functions keep track of the current and peak memory
usage of KLU.  They can be called by the user.

{\tt klu\_malloc} is essentially the same as {\tt p = malloc (n * sizeof
(size))}, {\tt klu\_free} is essentially the same as {\tt free(p)} except that
{\tt klu\_free} returns NULL which can then be assigned to {\tt p}.  {\tt
klu\_realloc} is similar to {\tt realloc}, except that if the reallocation
fails, {\tt p} is returned unchanged.  Failure conditions are returned in {\tt
Common.status}.

{\footnotesize
\begin{verbatim}
    #include "klu.h"
    size_t n, nnew, nold, size ;
    void *p ;
    klu_common Common ;
    p = klu_malloc (n, size, &Common) ;
    p = klu_free (p, n, size, &Common) ;
    p = klu_realloc (nnew, nold, size, p, &Common) ;

    #include "klu.h"
    size_t n, nnew, nold, size ;
    void *p ;
    klu_l_common Common ;
    p = klu_l_malloc (n, size, &Common) ;
    p = klu_l_free (p, n, size, &Common) ;
    p = klu_l_realloc (nnew, nold, size, p, &Common) ;
    \end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt btf\_maxtrans}: maximum transversal}
%-------------------------------------------------------------------------------

The BTF package includes three user-callable functions (each with {\tt int}
and {\tt SuiteSparse\_long} versions).  They do not need to be called directly by an
application that uses KLU.  KLU will call these functions to perform the
permutation into upper block triangular form.

The {\tt btf\_maxtrans} function finds a column permutation {\tt Q} that gives
{\tt A*Q} a zero-free diagonal, if one exists.  If row {\tt i} is matched to
column {\tt j}, then {\tt Match[i]=j}.  If the matrix is structurally singular,
there will be some unmatched rows.  If row {\tt i} is unmatched, then {\tt
Match[i]=-1}.  If the matrix is square and structurally non-singular, then {\tt
Q=Match} is the column permutation.  The {\tt btf\_maxtrans} function can
accept as input a rectangular matrix; it operates on the bipartite graph of
{\tt A}.  It returns the number of columns matched.  Unlike the KLU
user-callable functions, the BTF functions do not check its inputs at all; a
segmentation fault will occur if any input pointers are NULL, for example.

The function can require up to $O$({\tt n*nnz(A)}) time (excluding the {\em
cheap match} phase, which takes another $O$({\tt nnz(A)}) time.  If {\tt
maxwork > 0} on input, the work is limited to $O$({\tt maxwork*nnz(A)})
(excluding the cheap match), but the maximum transversal might not be found if
the limit is reached.

The {\tt Work} array is workspace required by the methods; its contents
are undefined on input and output.

{\footnotesize
\begin{verbatim}
    int nrow, ncol, Ap [ncol+1], Ai [nz], Match [nrow], Work [5*ncol], nmatch ;
    float maxwork, work ;
    nmatch = btf_maxtrans (nrow, ncol, Ap, Ai, maxwork, &work, Match, Work) ;

    SuiteSparse_long nrow, ncol, Ap [ncol+1], Ai [nz], Match [nrow], Work [5*ncol], nmatch ;
    float maxwork, work ;
    nmatch = btf_l_maxtrans (nrow, ncol, Ap, Ai, maxwork, &work, Match, Work) ;
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt btf\_strongcomp}: strongly connected components}
%-------------------------------------------------------------------------------

The {\tt btf\_strongcomp} function finds the strongly connected components of a
directed graph, returning a symmetric permutation {\tt P}.  The matrix {\tt A}
must be square.  The diagonal of {\tt A} (or {\tt A*Q} if a column permutation
is given on input) is ignored.  If {\tt Q} is NULL on input, the matrix
{\tt P*A*P'} is in upper block triangular form.  Otherwise, {\tt Q} is modified
on output so that {\tt P*A*Q} is in upper block triangular form.  The vector
{\tt R} gives the block boundaries, where the {\tt k}th block consists of
rows and columns {\tt R[k]} through {\tt R[k+1]-1} in the permuted matrix.
The function returns the number of strongly connected components found
(the diagonal blocks in the block triangular form).

{\footnotesize
\begin{verbatim}
    int n, Ap [n+1], Ai [nz], Q [n], P [n], R [n+1], Work [4*n], ncomp ;
    ncomp = btf_strongcomp (n, Ap, Ai, Q, P, R, Work) ;

    SuiteSparse_long n, Ap [n+1], Ai [nz], Q [n], P [n], R [n+1], Work [4*n], ncomp ;
    ncomp = btf_l_strongcomp (n, Ap, Ai, Q, P, R, Work) ;
\end{verbatim}
}

%-------------------------------------------------------------------------------
\subsection{{\tt btf\_order}: permutation to block triangular form}
%-------------------------------------------------------------------------------

The {\tt btf\_order} function combines the above two functions, first finding a
maximum transversal and then permuting the resulting matrix into upper block
triangular form.  Unlike {\tt dmperm} in MATLAB, it always reveals the maximum
matching along the diagonal, even if the matrix is structurally singular.

On output, {\tt P} and {\tt Q} are the row and column permutations, where
{\tt i = P[k]} if row {\tt i} of {\tt A} is the {\tt k}th row of {\tt P*A*Q},
and {\tt j = BTF\_UNFLIP(Q[k])} if column {\tt j} of {\tt A} is the {\tt k}th
column of {\tt P*A*Q}.  If {\tt Q[k] < 0}, then the {\tt (k,k)}th entry in
{\tt P*A*Q} is structurally zero.  The vector {\tt R}, and the return value,
are the same as {\tt btf\_strongcomp}.

{\footnotesize
\begin{verbatim}
    int n, Ap [n+1], Ai [nz], P [n], Q [n], R [n+1], nfound, Work [5*n], ncomp, nfound ;
    float maxwork, work ;
    ncomp = btf_order (n, Ap, Ai, maxwork, &work, P, Q, R, &nfound, Work) ;

    SuiteSparse_long n, Ap [n+1], Ai [nz], P [n], Q [n], R [n+1], nfound, Work [5*n], ncomp, nfound ;
    float maxwork, work ;
    ncomp = btf_l_order (n, Ap, Ai, maxwork, &work, P, Q, R, &nfound, Work) ;
\end{verbatim}
}

%------------------------------------------------------------------------------
\subsection{Sample C programs that use KLU}
%------------------------------------------------------------------------------

Here is a simple main program, {\tt klu\_simple.c}, that illustrates the basic
usage of KLU.  It uses KLU, and indirectly makes use of BTF and AMD.  COLAMD is
required to compile the demo, but it is not called by this example.  It uses
statically defined global variables for the sparse matrix {\tt A}, which would
not be typical of a complete application.  It just makes for a simpler example.

{\footnotesize
\input{klu_simple_c.tex}
}

The {\tt Ap}, {\tt Ai}, and {\tt Ax} arrays represent the matrix
\[
A = \left[
\begin{array}{ccccc}
 2 &  3 &  0 &  0 &  0 \\
 3 &  0 &  4 &  0 &  6 \\
 0 & -1 & -3 &  2 &  0 \\
 0 &  0 &  1 &  0 &  0 \\
 0 &  4 &  2 &  0 &  1 \\
\end{array}
\right].
\]
The solution to $Ax=b$ is $x = [1 \, 2 \, 3 \, 4 \, 5]^T$.  The program
uses default control settings (no scaling, permutation to block triangular
form, and the AMD ordering).  It ignores the error codes in the return values
and {\tt Common.status}.

The block triangular form found by {\tt btf\_order} for this matrix is
given below
\[
PAQ = \left[
\begin{array}{c|ccc|c}
2 & 0 & 0 & -1 & -3 \\
\hline
  & 2 & 0 & 3 & 0 \\
  & 3 & 6 & 0 & 4 \\
  & 0 & 1 & 4 & 1 \\
\hline
  &   &   &   & 1 \\
\end{array}
\right].
\]
This ordering is not modified by the AMD ordering because the 3-by-3 matrix
$A_{22} + A_{22}^T$ happens to be a dense matrix.  No partial pivoting happens
to occur during LU factorization; all pivots are selected along the diagonal of
each block.  The matrix contains two singletons, which are the original entries
$a_{34}=2$ and $a_{43}=1$, and one 3-by-3 diagonal block (in which a single
fill-in entry occurs during factorization: the $u_{23}$ entry of this 3-by-3
matrix).

For a more complete program that uses KLU, see {\tt KLU/Demo/kludemo.c} for an
{\tt int} version, and {\tt KLU/Demo/kluldemo.c} for a version that uses {\tt
SuiteSparse\_long} instead.  The top-level main routine uses CHOLMOD to read in a
compressed-column sparse matrix from a Matrix Market file, because KLU does not
include such a function.  Otherwise, no CHOLMOD functions are used.  Unlike
{\tt klu\_simple.c}, CHOLMOD is required to run the {\tt kludemo.c} and {\tt
kluldemo.c} programs.

%------------------------------------------------------------------------------
\section{Installation}
\label{Install}
%------------------------------------------------------------------------------

Installation of the C-callable interface requires the {\tt make} utility, in
Linux/Unix.  Alternatively, you can use the Cygwin {\tt make} in Windows.
The MATLAB installation in any platform, including Windows is simple; just
type {\tt klu\_install} to compile and install KLU, BTF, AMD, and COLAMD.

For {\tt make}, system-dependent configurations are in the {\tt
../SuiteSparse\_config/SuiteSparse\_config.mk} file.
You can edit that file to customize the
compilation, but it now automatically detecs your system (Linux, Mac, etc).
So it's not likely that you will need to edit that file.

To compile and install the C-callable KLU, BTF, AMD, and COLAMD libraries, go
to the {\tt KLU} directory and type {\tt make}.  The KLU and BTF libraries are
placed in {\tt KLU/Lib/libklu.*} and {\tt BTF/Lib/libbtf.*}.
Two KLU demo
programs will be compiled and tested in the {\tt KLU/Demo} directory.  You can
compare the output of {\tt make} with the results in the KLU distribution, {\tt
kludemo.out}.

Typing {\tt make clean} will remove all but the final compiled libraries and
demo programs.  Typing {\tt make purge} or {\tt make distclean} removes all
files not in the original distribution.  If you compile KLU or BTF and then
later change the {\tt ../SuiteSparse\_config/SuiteSparse\_config.mk}
file then you should type {\tt
make purge} and then {\tt make} to recompile.

When you compile your program that uses the C-callable KLU library, you need to
add the {\tt KLU/Lib/libklu.*}, {\tt BTF/Lib/libbtf.*}, {\tt AMD/Lib/libamd.*},
and {\tt COLAMD/Lib/libcolamd.*} libraries, and you need to tell your compiler to
look in the {\tt KLU/Include} and {\tt BTF/Include} directory for include
files.
Alternatively, do {\tt make install}, and KLU will be installed in
/usr/local/lib and /usr/local/include, and documentation is placed in
/usr/local/doc.  These installation locations can be changed;
see {\tt SuiteSparse/README.txt} for details.

If all you want to use is the KLU mexFunction in MATLAB, you can skip the use
of the {\tt make} command entirely.  Simply type {\tt klu\_install} in the
MATLAB command window while in the {\tt KLU/MATLAB} directory.  This works on
any system with MATLAB, including Windows.

%------------------------------------------------------------------------------
\newpage
\section{The KLU routines}
\label{klu_include}
%------------------------------------------------------------------------------

The file {\tt KLU/Include/klu.h} listed below describes each user-callable
routine in the C version of KLU, and gives details on their use.

{\footnotesize
\input{klu_h.tex}
}

%------------------------------------------------------------------------------
\newpage
\section{The BTF routines}
\label{btf_include}
%------------------------------------------------------------------------------

The file {\tt BTF/Include/btf.h} listed below describes each user-callable
routine in the C version of BTF, and gives details on their use.

{\footnotesize
\input{btf_h.tex}
}

%------------------------------------------------------------------------------
\newpage
% References
%------------------------------------------------------------------------------

\bibliographystyle{plain}
\bibliography{KLU_UserGuide}

\end{document}


\section{Support computation}

Support computation must be split into two parts: 1) extraction of
extensions from the extension array; 2) computation of support for
each extension;

\noindent\textbf{Extract extensions:} we extract extensions
segment-by-segment. We allocate array of size $|\allvlabels|\times
|\allelabels|$ and execute $|\extensionarray_i|$ threads that store
0/1 flags in this array that represents whether a particular
combination of $(\elabel, \vlabel)\in \allvlabels\times \allelabels$
exists in the $i$-th segment.

\noindent\textbf{Support computation of extensions:} We describe
computation in one segment of $\extensionarray$. Computing supports of
extensions in all segments in parallel is similar using segmented
reduce/scan operations. From the properties of the $\extensionarray$
described in Section~\ref{sec:computing-extension} follows that in the
segments are block of vertex ids belonging to one graph and there is
one such block per segment. Therefore, by creating array of 0/1
indices by putting 1 on the end of the block. Scanning this array
results in mapping stored in array $\mathbf{M}$ of the sparse database
graph ids into a continuous range. The support computation proceeds as
follows: let have $n$ extensions and the number of graphs in the
segment is $g$. We allocate array $\mathbf{S}$ of size $n\cdot g$
allocating for $i$-th extension $i$-th block of size $g$. We then
execute $n\cdot g$ threads each assigned to an extension. Each thread
stores 1 in a block corresponding to its assigned extension at a
position looked up in $\mathbf{M}$. The final supports are then
computed by executing parallel segmented reduction in array
$\mathbf{S}$ with segments of size $g$. This can be parallelized even
more, by processing many segments at once.

\textbf{There is missing a lot of notation that make the text more
  clear and more concise. Additionally, we need to explain why we need
  to collect values $\allvlabels\times \allelabels$.}

\subsection{Growing the embeddings}

After we know the frequent extensions, we have to transform the
information from the extension array into a new column $\embedcol$ or
process some existing column. We have to consider two cases:

\textbf{(1) Forward extensions: } Forward extensions introduce new
vertex into the pattern and therefore introduce new column. The column
is extracted directly from one segment of $\extensionarray$, by
copying the needed information to a new column $\embedcol$ and
\emph{add} the new column to the embeddings.

\textbf{(2) Backward extensions: } backward extensions do not
introduce new vertex into the pattern. The backward extensions can
only filter the last column in the embeddings. There are two reason:
we consider only simple graphs and 2) the principle of
monotonicity. The last column is filtered against the \texttt{row}
field of some elements in $\extensionarray$ in the following way:
allocate a vector $\mathbf{F}$ of the same size as is the size of last
column and store there 0. Execute $|\mathbf{F}|$ threads, thread with
index $i$ stores 1 in $\mathbf{F}[i]$ if there exists a backlink from
elements of $\extensionarray$. Then execute parallel scan in
$\mathbf{F}$, which gives the new positions in the filtered
column. Finally copy the information from the last column into new
column and \emph{replace} the last column in the embeddings while
\emph{retaining} the original last column.


\documentclass[11pt]{article}
\usepackage{url}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\setlength\topmargin{-0.6cm}
\setlength\textheight{23.4cm}
\setlength\textwidth{17.0cm}
\setlength\oddsidemargin{0cm}
\begin{document}
\title{Ling 572 Reading 1}
\author{Daniel Campos  \tt {dacampos@uw.edu}}
\date{01/29/2019}
\maketitle
\section{ Q1}
Let P(X=i) be the probability of getting an i
when rolling a dice (e.g., i=1, 2, …, 6). What is
the value of P(X=i) with the maximum entropy
if the following is true?
\begin{description}
\item [(a)] P(X=1) + P(X=2) = 1/2: \\ Since we want to have the most uniform distribution our values for P(X=3)+ P(X=4) +P(X=5) + P(X=6) = 1/2 and they are all equal so P(X=1) = 1/4, P(X=2) = 1/4, P(X=3) = 1/8, P(X=4) = 1/8, P(X=5) = 1/8, P(X=6) = 1/8.
\item [(b)] P(X=1) + P(X=2) = 1/2 and P(X=6) = 1/3 \\ Keeping with the logic mentioned above,P(X=3)+ P(X=4) +P(X=5) = 1-(1/2+1/3) = 1-5/6 = 1/6  so P(X=1) = 1/4, P(X=2) = 1/4, P(X=3) = 1/18, P(X=4) = 1/18, P(X=5) = 1/18, P(X=6) = 1/3.
\end{description}
\section{ Q2  How many feature functions?}
If there are C classes and V features then there are |V|*|C| feautre functions since each feature function is a binary-valued function on events. That being said not all features may be used in the final Maxent model.
\section{ Q3 Similarities and differences}
\subsection{Modeling}
Both methods are similair in the modeling of lingusitic task because they treach each word as a feature but Naive Bayes(NB) assumes feature independence while Maxent can model dependent feautres and assumes that the dataset is best representing by only modeling what is known. NB uses two parameters are the class priors(p(c)) and the conditional probability(P(F|c)) while Maxent parameters are a set of feautre functions of (feature, class)binary pairs. NB has |C| + |V|*|C| parameters(C being classes and V being the vocabulary of the training set, otherwise refered to as the features) while maxent has up to |V|*|C| parameters but will optimze for a set of those that maximize the entropy.
\subsection{Training}
This is where the methods differ most since NB calculates the class prior which is merely how often the class shows in the training data along with the conditional probability which is count(f, c)/counc(c) where f is feature of F and c is a classs of C.  Maxent has a more traditional Training stage since training is done via either GIS or ISS algorithms which optimize feautre usage and weights to represent the data distribution of training. Basically Maxent compares the empiracal expectaions to the model and modify the weights of feature and use of features to optimize for maximum data entropy. Both models are prone to overfitting and poor generalization on test data which can be optimized by use of smoothing mechanism. Additionally, since Maxent has a more traditional training mechanism, it uses various regularizations and early stopping to improve training time and optimize model performance. In short training for Maxent may be able to model dependent features better but is much slower to NB since it may need to look at all possible feature combination and feature weights can be infinite and the iterative trainer can take a long time to reach those values.
\subsection{Decoding}
Decoding is done in a similair way for both models. Using the features(words) present in the candidate document the probability of all classes is measured and the class with the highest probability is selected. In NB this is done by adding the logprob of all features(1-P of feature if the feature t in V is not in the document) while Maxent uses the logprob of all features present in the document.
\end{document}