\subsubsection{DataMining}
\label{subsubsec:DataMining}

Knowledge discovery in databases (KDD) is the process of discovering
 useful knowledge from a collection of data. This widely used data
mining technique is a process that includes data preparation and
selection, data cleansing, incorporating prior knowledge on data
sets and interpreting accurate solutions from the observed results.
Major KDD application areas include marketing, fraud detection,
telecommunication and manufacturing.

DataMining is the analysis step of the KDD process. The overall of
the data mining process is to extract information from a data set
and transform it into an understandable structure for further use.
The actual data mining task is the
automatic or semi-automatic analysis of large quantities of data
to extract previously unknown, interesting patterns such as groups
 of data records (cluster analysis), unusual records (anomaly
detection), and dependencies (association rule mining).
\\
%
In order to use the \textbf{DataMining} post-processor, the user
needs to set the attribute \xmlAttr{subType}: \\
\\
\xmlNode{PostProcessor \xmlAttr{subType}=
\xmlString{DataMining}}. \\
\\
The following is a list of acceptable sub-nodes:
\begin{itemize}
  \item \xmlNode{KDD} \xmlDesc{string,required field}, the subnodes specifies
  the necessary information for the algorithm to be used in the postprocessor.
  The \xmlNode{KDD} has the required attribute: \xmlAttr{lib}, the name of the
  library the algorithm belongs to. Current algorithms applied in the KDD model
  is based on SciKit-Learn library. Thus currently there is only one library:
  \begin{itemize}
    \item \xmlString{SciKitLearn}
  \end{itemize}
  The \xmlNode{KDD} has the optional attribute: \xmlAttr{labelFeature}, the name
  associated to labels or dimensions generated by the \textbf{DataMining}
  post-processor.
  The default name depends on the type of algorithm employed.
  For clustering and mixture models it is the name of the PostProcessor
  followed by ``Labels'' (e.g., if the name of a clustering PostProcessor is
  ``kMeans'' then the default name associated to the labels is ``kMeansLabels''
  if not specified in the attribute \xmlAttr{labelFeature}).
  For decomposition and manifold models, the default names are the name of the
  PostProcessor followed by ``Dimension'' and an integer identifier beginning
  with 1. (e.g., if the name of a dimensionality reduction PostProcessor is
  ``dr'' and the user specifies 3 components, then the output dataObject will
  have three new outputs named ``drDimension1,'' ``drDimension2,'' and
  ``drDimension3.'').
  \nb The ``Labels'' are automatically added to the output \textbf{DataObjects}. It
  is also accessible by the users using the variable name defined above.
\end{itemize}

\paragraph{SciKitLearn}
\xmlString{SciKitLearn} is based on algorithms in SciKit-Learn library, and it performs data mining over PointSet and HistorySet. Note that for HistorySet's \xmlString{SciKitLearn} performs the task given in \xmlNode{SKLType} (see below) for each time step, and so only synchronized HistorySet can be used as input to this model. For unsynchronized HistorySet, use \xmlString{HistorySetSync} method in \xmlString{Interfaced} post-processor to synchronize the input data before using \xmlString{SciKitLearn}. The rest of this subsection and following subsection is dedicated to the \xmlString{SciKitLearn} library.

The temporal variable for a HistorySet \xmlString{SciKitLearn} is specified in the \xmlNode{pivotParameter} node:
\begin{itemize}
  \item \xmlNode{pivotParameter}, \xmlDesc{string, optional parameter} specifies the pivot variable (e.g., time, etc) in the input HistorySet.
      \default{None}.
\end{itemize}

The algorithm for the dataMining is chosen by the subnode \xmlNode{SKLType} under the parent node
\xmlNode{KDD}. The format is same as in \ref{subsubsec:SciKitLearn}. However, for the completeness
sake, it is repeated here.

The data that are used in the training of the \textbf{DataMining}
postprocessor are suplied with subnode \xmlNode{Features} in the parent node
 \xmlNode{KDD}.

\begin{itemize}
  \item \xmlNode{SKLtype}, \xmlDesc{vertical bar (\texttt{|}) separated string,
  required field}, contains a string that represents the data mining algorithm
  to be used.
  %
  As mentioned, its format is:\\
  \xmlNode{SKLtype}\texttt{mainSKLclass|algorithm}\xmlNode{/SKLtype} where the
  first word (before the ``\texttt{|}'' symbol) represents the main class of
  algorithms, and the second word (after the ``\texttt{|}'' symbol) represents
  the specific algorithm.
  %
  \item \xmlNode{Features}, \xmlDesc{string, required field}, defines the data
  to be used for training the data mining algorithm. It can be:
  \begin{itemize}
	\item the name of the variable in the defined dataObject entity
	\item the location (i.e. input or output). In this case the data mining
        is applied to all the variables in the defined space.
  \end{itemize}
\end{itemize}

The \xmlNode{KDD} node can have either optional or required subnodes depending
 on the dataMining algorithm used. The possible subnodes will be described separately
 for each algorithm below. The time dependent clustering data mining algorithms have a \xmlNode{reOrderStep} option that will try and keep the same labels on the clusters.  The higher the number, the longer the history that the clustering algorithm will look through to maintain the same labeling between time steps.

All the available algorithms are described in the following sections.

\paragraph{Gaussian mixture models}
\label{paragraph:GMM}

A Gaussian mixture model is a probabilistic model that assumes all
 the data points are generated from a mixture of a finite number of
 Gaussian distributions with unknown parameters.
\\
Scikit-learn implements different classes to estimate Gaussian
mixture models, that correspond to different estimation strategies,
 detailed below.

\subparagraph{ GMM classifier} \hfill
\label{subparagraph:GMMClass}

The GMM object implements the expectation-maximization (EM)
algorithm for fitting mixture-of-Gaussian models. The GMM comes with different options
 to constrain the covariance of  the difference classes estimated: spherical, diagonal, tied or
 full covariance.

\skltype{Gaussian Mixture Model}{mixture|GMM}
\begin{itemize}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional
field} Number of mixture components. \default{1}
	\item \xmlNode{covariance\_type}, \xmlDesc{string, optional
field}, describes the type of covariance parameters to use.
Must be one of ‘spherical’, ‘tied’, ‘diag’, ‘full’. \default{diag}
	\item \xmlNode{random\_state}, \xmlDesc{integer seed or random
 number generator instance, optional field},  A random number
generator instance \default{0 or None}
	\item \xmlNode{min\_covar}, \xmlDesc{float, optional field},
 Floor on the diagonal of the covariance matrix to prevent overfitting.
 \default{1e-3}.
	\item \xmlNode{thresh}, \xmlDesc{float, optional field},
convergence threshold. \default{0.01}
	\item \xmlNode{n\_iter}, \xmlDesc{integer, optional field},
Number of EM iterations to perform. \default{100}
	\item \xmlNode{n\_init}, \xmlDesc{integer, optional},
Number of initializations to perform. the best results is kept.
\default{1}
	\item \xmlNode{params}, \xmlDesc{string, optional field},
 Controls which parameters are updated in the training process.
Can contain any combination of ‘w’ for weights, ‘m’ for means, and
‘c’ for covars. \default{‘wmc’}
	\item \xmlNode{init\_params}, \xmlDesc{string, optional
field}, Controls which parameters are updated in the initialization
process. Can contain any combination of ‘w’ for weights, ‘m’ for means,
 and ‘c’ for covars. \default{‘wmc’}.
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
      <PostProcessor name='PostProcessorName' subType='DataMining'>
          <KDD lib='SciKitLearn'>
              <Features>variableName</Features>
              <SKLtype>mixture|GMM</SKLtype>
              <n_components>2</n_components>
              <covariance_type>spherical</covariance_type>
          </KDD>
      </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}

\subparagraph{ Variational GMM Classifier (VBGMM)} \hfill
\label{subparagraph:VBGMM}

The VBGMM object implements a variant of the Gaussian mixture model
 with variational inference algorithms. The API is identical to GMM.

\skltype{Variational Gaussian Mixture Model}{mixture|VBGMM}
\begin{itemize}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional
field} Number of mixture components. \default{1}
	\item \xmlNode{covariance\_type}, \xmlDesc{string, optional
field}, describes the type of covariance parameters to use.
Must be one of ‘spherical’, ‘tied’, ‘diag’, ‘full’. \default{diag}
	\item \xmlNode{alpha}, \xmlDesc{float, optional field},
represents the concentration parameter of the dirichlet process.
Intuitively, the Dirichlet Process is as likely to start a new cluster
 for a point as it is to add that point to a cluster with alpha
 elements. A higher alpha means more clusters, as the expected
number of clusters is ${\alpha*log(N)}$. \default{1}.
\end{itemize}

\paragraph{ Clustering }
\label{paragraph:Clustering}

Clustering of unlabeled data can be performed with this subType of
 the DataMining PostProcessor.

An overwiev of the different clustering algorithms is given in
Table\ref{tab:clustering}.

\begin{table}[!htbp]
  \centering
  \caption{Overview of Clustering Methods}
  \label{tab:clustering}
  \begin{tabular}{| L{2.5cm} | L{2.5cm} | L{2.5cm} | L{3.5cm} | L{2.75cm} |} \hline
    {\bf Method name} & {\bf Parameters} & {\bf Scalability} & {\bf
Usecase} & {\bf Geometry (metric used)} \\ \hline
    K-Means  & number of clusters & Very large n\_samples, medium
n\_clusters with MiniBatch code & General-purpose, even cluster size,
flat geometry, not too many clusters & Distances between points
 \\ \hline
    Affinity propagation & damping, sample preference & Not scalable with
n\_samples & Many clusters, uneven cluster size, non-flat geometry &
Graph distance (e.g. nearest-neighbor graph)       \\ \hline
    Mean-shift & bandwidth & Not scalable with n\_samples & Many clusters,
 uneven cluster size, non-flat geometry & Distances between points \\ \hline
    Spectral clustering & number of clusters & Medium n\_samples, small
n\_clusters & Few clusters, even cluster size, non-flat geometry &
Graph distance (e.g. nearest-neighbor graph)       \\ \hline
    Ward hierarchical clustering & number of clusters & Large n\_samples
and n\_clusters & Many clusters, possibly connectivity constraints &
Distances between points       \\ \hline
    Agglomerative clustering & number of clusters, linkage type, distance
 & Large n\_samples and n\_clusters & Many clusters, possibly
connectivity constraints, non Euclidean distances & Any pairwise
distance       \\ \hline
    DBSCAN & neighborhood size & Very large n\_samples, medium n\_clusters
 & Non-flat geometry, uneven cluster sizes & Distances between nearest
 points       \\ \hline
    Gaussian mixtures & many & Not scalable & Flat geometry, good for
 density estimation & Mahalanobis distances to centers       \\ \hline
  \end{tabular}
\end{table}

\FloatBarrier

\subparagraph{K-Means Clustering} \hfill
\label{subparagraph:KMeans}

The KMeans algorithm clusters data by trying to separate samples in n groups
of equal variance, minimizing a criterion known as the inertia or within-cluster
sum-of-squares. This algorithm requires the number of clusters to be specified.
 It scales well to large number of samples and has been used across a large
range of application areas in many different fields

\skltype{ K-Means Clustering}{cluster|KMeans}
\begin{itemize}
	\item \xmlNode{n\_clusters}, \xmlDesc{integer, optional field}
The number of clusters to form as well as the number of centroids to
generate. \default{8}
	\item \xmlNode{max\_iter}, \xmlDesc{integer, optional field},
Maximum number of iterations of the k-means algorithm for a single run.
\default{300}
	\item \xmlNode{n\_init}, \xmlDesc{integer, optional field},
Number of time the k-means algorithm will be run with different centroid
 seeds. The final results will be the best output of n\_init consecutive
 runs in terms of inertia. \default{3}
	\item \xmlNode{init}, \xmlDesc{string, optional},
Method for initialization, k-means++’, ‘random’ or an ndarray:
		\begin{itemize}
			\item ‘k-means++’ : selects initial cluster
centers for k-mean clustering in a smart way to speed up convergence.
			\item ‘random’: choose k observations (rows) at
 random from data for the initial centroids.
			\item If an ndarray is passed, it should be of
 shape (n\_clusters, n\_features) and gives the initial centers.
		\end{itemize}
	\item \xmlNode{precompute\_distances}, \xmlDesc{boolean, optional
field}, Precompute distances (if true faster but takes more memory).
\default{true}
	\item \xmlNode{tol}, \xmlDesc{float, optional field}, Relative
tolerance with regards to inertia to declare convergence. \default{1e-4}
	\item \xmlNode{n\_jobs}, \xmlDesc{integer, optional field}, The number
of jobs to use for the computation. This works by breaking down the pairwise
 matrix into n jobs even slices and computing them in parallel. If -1 all CPUs
 are used. If 1 is given, no parallel computing code is used at all, which is
useful for debugging. For n\_jobs below -1, (n\_cpus + 1 + n\_jobs) are used. Thus
 for n\_jobs = -2, all CPUs but one are used. \default{1}
	\item \xmlNode{random\_state}, \xmlDesc{integer or numpy.RandomState,
 optional field} The generator used to initialize the centers. If an integer
 is given, it fixes the seed. \default{the global numpy random number generator}.
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
      <PostProcessor name='PostProcessorName' subType='DataMining'>
          <KDD lib='SciKitLearn'>
              <Features>variableName</Features>
              <SKLtype>cluster|KMeans</SKLtype>
              <n_clusters>2</n_clusters>
              <tol>0.0001</tol>
              <init>random</init>
          </KDD>
      </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}

\subparagraph{  Mini Batch K-Means } \hfill
\label{subparagraph:MiniBatch}

The MiniBatchKMeans is a variant of the KMeans algorithm which uses
mini-batches to reduce the computation time, while still attempting
 to optimise the same objective function. Mini-batches are subsets of
 the input data, randomly sampled in each training iteration.

MiniBatchKMeans converges faster than KMeans, but the quality of the
results is reduced. In practice this difference in quality can be
 quite small.

\skltype{ Mini Batch K-Means Clustering}{cluster|MiniBatchKMeans}
\begin{itemize}
	\item \xmlNode{n\_clusters}, \xmlDesc{integer, optional field}
The number of clusters to form as well as the number of centroids to
generate. \default{8}
	\item \xmlNode{max\_iter}, \xmlDesc{integer, optional field},
Maximum number of iterations of the k-means algorithm for a single run.
\default{100}
	\item \xmlNode{max\_no\_improvement}, \xmlDesc{integer, optional
firld}, Control early stopping based on the consecutive number of mini
 batches that does not yield an improvement on the smoothed inertia.
To disable convergence detection based on inertia, set
max\_no\_improvement to None. \default{10}
	\item \xmlNode{tol}, \xmlDesc{float, optional field}, Control
 early stopping based on the relative center changes as measured by a
smoothed, variance-normalized of the mean center squared position changes.
 This early stopping heuristics is closer to the one used for the batch
 variant of the algorithms but induces a slight computational and memory
overhead over the inertia heuristic. To disable convergence detection
based on normalized center change, set tol to 0.0 (default). \default{0.0}
	\item \xmlNode{batch\_size}, \xmlDesc{integer, optional field},
Size of the mini batches. \default{100}
	\item{init\_size}, \xmlDesc{integer, optional field}, Number of
samples to randomly sample for speeding up the initialization
 (sometimes at the expense of accuracy): the only algorithm is initialized
 by running a batch KMeans on a random subset of the data.
\textit{This needs to be larger than k.}, \default{3 * \xmlNode{batch\_size}}
	\item \xmlNode{init}, \xmlDesc{string, optional},
Method for initialization, k-means++’, ‘random’ or an ndarray:
		\begin{itemize}
			\item ‘k-means++’ : selects initial cluster
centers for k-mean clustering in a smart way to speed up convergence.
			\item ‘random’: choose k observations (rows) at
 random from data for the initial centroids.
			\item If an ndarray is passed, it should be of
 shape (n\_clusters, n\_features) and gives the initial centers.
		\end{itemize}
	\item \xmlNode{precompute\_distances}, \xmlDesc{boolean, optional
field}, Precompute distances (if true faster but takes more memory).
\default{true}
	\item \xmlNode{n\_init}, \xmlDesc{integer, optional field},
Number of time the k-means algorithm will be run with different centroid
 seeds. The final results will be the best output of n\_init consecutive
 runs in terms of inertia. \default{3}
	\item \xmlNode{compute\_labels}, \xmlDesc{boolean, optional field},
Compute label assignment and inertia for the complete dataset once the
 minibatch optimization has converged in fit. \default{True}
	\item \xmlNode{random\_state}, \xmlDesc{integer or numpy.RandomState,
 optional field} The generator used to initialize the centers. If an integer
 is given, it fixes the seed. \default{the global numpy random number generator}.
	\item{reassignment\_ratio}, \xmlNode{float, optional field}, Control
the fraction of the maximum number of counts for a center to be reassigned.
A higher value means that low count centers are more easily reassigned, which
 means that the model will take longer to converge, but should converge in a
better clustering. \default{0.01}
\end{itemize}

\subparagraph{Affinity Propagation} \hfill
\label{subparagraph:Affinity}

AffinityPropagation creates clusters by sending messages between pairs of
samples until convergence. A dataset is then described using a small number
of exemplars, which are identified as those most representative of other
samples. The messages sent between pairs represent the suitability for one
sample to be the exemplar of the other, which is updated in response to the
values from other pairs. This updating happens iteratively until convergence,
 at which point the final exemplars are chosen, and hence the final clustering
 is given.

\skltype{ AffinityPropogation Clustering}{cluster|AffinityPropogation}
\begin{itemize}
	\item \xmlNode{damping}, \xmlDesc{float, optional field}, Damping factor
 between 0.5 and 1. \default{0.5}
	\item \xmlNode{convergence\_iter}, \xmlDesc{integer, optional field},
Number of iterations with no change in the number of estimated clusters that
stops the convergence. \default{15}
	\item \xmlNode{max\_iter}, \xmlDesc{integer, optional field}, Maximum
 number of iterations. \default{200}
	\item \xmlNode{copy}, \xmlDesc{boolean, optional field}, Make a copy of
input data or not. \default{True}
	\item \xmlNode{preference}, \xmlDesc{array-like, shape (n\_samples,)
or float, optional field}, Preferences for each point - points with larger
values of preferences are more likely to be chosen as exemplars. The number
of exemplars, ie of clusters, is influenced by the input preferences value.
\default{If the preferences are not passed as arguments, they will be set to the median of
 the input similarities.}
	\item \xmlNode{affinity}, \xmlDesc{string, optional field},Which affinity to use.
 At the moment precomputed and euclidean are supported. euclidean uses the negative squared
euclidean distance between points. \default{``euclidean``}
	\item \xmlNode{verbose}, \xmlDesc{boolean, optional field}, Whether to be verbose.
\default{False}
\end{itemize}

\subparagraph{ Mean Shift } \hfill
\label{subparagraph:MeanShift}

MeanShift clustering aims to discover blobs in a smooth density of samples. It is
 a centroid based algorithm, which works by updating candidates for centroids to be
the mean of the points within a given region. These candidates are then filtered in
a post-processing stage to eliminate near-duplicates to form the final set of centroids.

\skltype{ Mean Shift Clustering}{cluster|MeanShift}
\begin{itemize}
	\item \xmlNode{bandwidth}, \xmlDesc{float, optional field}, Bandwidth used
in the RBF kernel. If not given, the bandwidth is estimated using
\textit{sklearn.cluster.estimate\_bandwidth}; see the documentation for that function for
 hints on scalability.
	\item \xmlNode{seeds}, \xmlDesc{array, shape=[n\_samples, n\_features],
optional field}, Seeds used to initialize kernels. If not set, the seeds are
calculated by \textit{clustering.get\_bin\_seeds} with bandwidth as the grid size and
 default values for other parameters.
	\item \xmlNode{bin\_seeding}, \xmlDesc{boolean, optional field}, If true,
 initial kernel locations are not locations of all points, but rather the
 location of the discretized version of points, where points are binned onto
 a grid whose coarseness corresponds to the bandwidth. Setting this option
to True will speed up the algorithm because fewer seeds will be initialized.
 \default{False} Ignored if seeds argument is not None.
	\item \xmlNode{min\_bin\_freq}, \xmlDesc{integer, optional field},
To speed up the algorithm, accept only those bins with at least min\_bin\_freq
 points as seeds. \default{1}.
	\item \xmlNode{cluster\_all}, \xmlDesc{boolean, optional field}, If true,
 then all points are clustered, even those orphans that are not within any
kernel. Orphans are assigned to the nearest kernel. If false, then orphans
are given cluster label -1. \default{True}
\end{itemize}

\subparagraph{Spectral clustering} \hfill
\label{subparagraph:Spectral}

SpectralClustering does a low-dimension embedding of the affinity matrix between
 samples, followed by a \textit{KMeans} in the low dimensional space. It is
especially efficient if the affinity matrix is sparse and the pyamg module is
installed.

\skltype{Spectral Clustering}{cluster|Spectral}
\begin{itemize}
	\item \xmlNode{n\_clusters}, \xmlDesc{integer, optional field},
	The dimension of the projection subspace.\default{8}
	%
	\item \xmlNode{affinity}, \xmlDesc{string, array-like or callable, optional
	  field}, If a string, this may be one of:
	\begin{itemize}
		\item ‘nearest\_neighbors’,
		\item ‘precomputed’,
		\item ‘rbf’ or
		\item one of the kernels supported by \textit{sklearn.metrics.pairwise\_kernels}.
	\end{itemize}
	Only kernels that produce similarity scores (non-negative values that increase
	with similarity) should be used. This property is not checked by the clustering
	 algorithm. \default{‘rbf’}
	%
	\item \xmlNode{gamma}, \xmlDesc{float, optional field}, Scaling factor of RBF,
	polynomial, exponential $chi^2$ and sigmoid affinity kernel.
	Ignored for $affinity='nearest\_neighbors'$. \default{1.0}
	%
	\item \xmlNode{degree}, \xmlDesc{float, optional field}, Degree of the polynomial
	 kernel. Ignored by other kernels. \default{3}
	%
	\item \xmlNode{coef0}, \xmlDesc{float, optional field}, Zero coefficient for
	polynomial and sigmoid kernels. Ignored by other kernels. \default{1}
	%
	\item \xmlNode{n\_neighbors}, \xmlDesc{integer, optional field}, Number of neighbors
	to use when constructing the affinity matrix using the nearest neighbors method.
	Ignored for affinity='rbf'. \default{10}
	%
	\item \xmlNode{eigen\_solver} \xmlDesc{string, optional field},  The eigenvalue
	decomposition strategy to use:
	\begin{itemize}
		\item None,
		\item ‘arpack’,
		\item ‘lobpcg’, or
		\item ‘amg’
	\end{itemize}
	\nb{AMG requires pyamg to be installed. It can be faster on very large, sparse
	problems, but may also lead to instabilities}
	%
	\item \xmlNode{random\_state}, \xmlDesc{integer seed, RandomState instance,
	 or None, optional field}, A pseudo random number generator used for the
	initialization of the lobpcg eigen vectors decomposition when $eigen_solver == ‘amg’$
	 and by the K-Means initialization. \default{None}
	%
	\item \xmlNode{n\_init}, \xmlDesc{integer, optional field}, Number of time the
	 k-means algorithm will be run with different centroid seeds. The final results
	 will be the best output of n\_init consecutive runs in terms of inertia.
	\default{10}
	%
	\item \xmlNode{eigen\_tol}, \xmlDesc{float, optional field}, Stopping criterion
	 for eigendecomposition of the Laplacian matrix when using arpack eigen\_solver.
	\default{0.0}
	%
	\item \xmlNode{assign\_labels}, \xmlDesc{string, optional field}, The strategy to
	use to assign labels in the embedding space. There are two ways to assign labels
	after the laplacian embedding:
	\begin{itemize}
		\item ‘kmeans’,
		\item ‘discretize’
	\end{itemize}
	 k-means can be applied and is a popular choice. But it can also be sensitive
	to initialization. Discretization is another approach which is less sensitive
	 to random initialization. \default{‘kmeans’}
	%
	\item \xmlNode{kernel\_params}, \xmlDesc{dictionary of string to any, optional
	 field}, Parameters (keyword arguments) and values for kernel passed as
	callable object. Ignored by other kernels. \default{None}
\end{itemize}

\textbf{Notes} \\
If you have an affinity matrix, such as a distance matrix, for which 0 means identical
elements, and high values means very dissimilar elements, it can be transformed in a
similarity matrix that is well suited for the algorithm by applying the Gaussian
 (RBF, heat) kernel:
\begin{equation}
np.exp(- X ** 2 / (2. * delta ** 2))
\end{equation}
Another alternative is to take a symmetric version of the k nearest neighbors
connectivity matrix of the points.
If the \textit{pyamg} package is installed, it is used: this greatly speeds
up computation.

\subparagraph{ DBSCAN Clustering } \hfill
\label{subparagraph:DBSCAN}

The Density-Based Spatial Clustering of Applications with Noise (DBSCAN)
 algorithm views clusters as areas of high density separated by
areas of low density. Due to this rather generic view, clusters found by
DBSCAN can be any shape, as opposed to k-means which assumes that clusters
 are convex shaped.

\skltype{DBSCAN Clustering}{cluster|DBSCAN}
\begin{itemize}
	\item \xmlNode{eps}, \xmlDesc{float, optional field}, The maximum
	distance between two samples for them to be considered as in the
	 same neighborhood. \default{0.5}
	%
	\item \xmlNode{min\_samples}, \xmlDesc{integer, optional field},
	The number of samples in a neighborhood for a point to be
	considered as a core point. \default{5}
	%
	\item \xmlNode{metric}, \xmlDesc{string, or callable, optional field}
	The metric to use when calculating distance between instances in
	 a feature array. If metric is a string or callable, it must be one
	of the options allowed by \textit{metrics.pairwise.calculate\_distance}
	 for its metric parameter. If metric is “precomputed”, X is assumed
	 to be a distance matrix and must be square. \default{'euclidean'}
	%
	\item \xmlNode{random\_state}, \xmlDesc{numpy.RandomState,
	 optional field}, The generator used to initialize the centers.
	\default{numpy.random}.
\end{itemize}

\subparagraph{Agglomerative Clustering } \hfill
\label{subparagraph:agglomerative}

Hierarchical clustering is a general family of clustering algorithms that build nested clusters by merging or splitting them successively.
This hierarchy of clusters is represented as a tree (or dendrogram).
The root of the tree is the unique cluster that gathers all of the samples, the leaves being the clusters with only one sample.
The AgglomerativeClustering object performs a hierarchical clustering using a bottom up approach: each observation starts in its own cluster,
and clusters are successively merged together. The linkage criteria determines the metric used for the merge strategy:
\begin{itemize}
  \item Ward: it minimizes the sum of squared differences within all clusters. It is a variance-minimizing approach and in this sense
is similar to the k-means objective function but tackled with an agglomerative hierarchical approach.
  \item Maximum or complete linkage: it minimizes the maximum distance between observations of pairs of clusters.
  \item Average linkage: it minimizes the average of the distances between all observations of pairs of clusters.
\end{itemize}

AgglomerativeClustering can also scale to large number of samples when it is used jointly with a connectivity matrix,
but is computationally expensive when no connectivity constraints are added between samples: it considers at each step all of the possible merges.

\skltype{Agglomerative Clustering}{cluster|Agglomerative}
\begin{itemize}
  \item \xmlNode{n\_clusters}, \xmlDesc{int, optional field}, The number of clusters to find. \default{2}
  \item \xmlNode{connectivity}, \xmlDesc{array like or callable, optional field}, Connectivity matrix. Defines for each sample the neighboring samples
   following a given structure of the data. This can be a connectivity matrix itself or a callable that transforms the data into a connectivity matrix,
   such as derived from kneighbors graph. Default is None, i.e, the hierarchical clustering algorithm is unstructured. \default{None}
  \item \xmlNode{affinity}, \xmlDesc{string or callable, optional field}, Metric used to compute the linkage. Can be ``euclidean'', ``$l1$'', ``$l2$'',
   ``manhattan'',``cosine'', or ``precomputed''. If linkage is ``ward'', only ``euclidean'' is accepted. \default{euclidean}
%  \item \xmlNode{memory}, \xmlDesc{Instance of joblib.Memory or string, optional field}, Used to cache the output of the computation of the tree.
%   By default, no caching is done. If a string is given, it is the path to the caching directory.
  \item \xmlNode{n\_components}, \xmlDesc{int, optional field}, Number of connected components. If None the number of connected components is estimated
  from the connectivity matrix. NOTE: This parameter is now directly determined from the connectivity matrix and will be removed in $0.18$.
%  \item \xmlNode{compute\_full\_tree}, \xmlDesc{bool or 'auto', optional field}, Stop early the construction of the tree at \xmlNode{n\_clusters}.
%  This is useful to decrease computation time if the number of clusters is not small compared to the number of samples. This option is useful only
%  when specifying a connectivity matrix. Note also that when varying the number of clusters and using caching, it may be advantageous to compute the full tree.
  \item \xmlNode{linkage}, \xmlDesc{{ward,complete,average}, optional field}, Which linkage criterion to use. The linkage criterion determines which distance
  to use between sets of observation. The algorithm will merge the pairs of cluster that minimize this criterion. Ward minimizes the variance of the clusters being merged.
  Average uses the average of the distances of each observation of the two sets. Complete or maximum linkage uses the maximum distances between all observations
  of the two sets.. \default{ward}
%  \item \xmlNode{pooling\_func}, \xmlDesc{callable, optional field}, This combines the values of agglomerated features into a single value, and
%  should accept an array of shape $[M, N]$ and the keyword argument axis=1, and reduce it to an array of size $[M]$. \default{np.mean}
\end{itemize}

\subparagraph{Clustering performance evaluation} \hfill
\label{subparagraph:ClusterPerformance}

Evaluating the performance of a clustering algorithm is not as trivial as
counting the number of errors or the precision and recall of a supervised
 classification algorithm. In particular any evaluation metric should not
 take the absolute values of the cluster labels into account but rather if
 this clustering define separations of the data similar to some ground truth
 set of classes or satisfying some assumption such that members belong to
the same class are more similar that members of different classes according
to some similarity metric.

If the ground truth labels are not known, evaluation must be performed using
 the model itself. The \textbf{Silhouette Coefficient} is an example of
such an evaluation, where a higher Silhouette Coefficient score relates to
 a model with better defined clusters. The Silhouette Coefficient is defined
 for each sample and is composed of two scores:
\begin{enumerate}
	\item The mean distance between a sample and all other points in the
	 same class.
	%
	\item The mean distance between a sample and all other points in the
	 next nearest cluster.
\end{enumerate}

The Silhoeutte Coefficient s for a single sample is then given as:
\begin{equation}
s = \frac{b - a}{max(a, b)}
\end{equation}
The Silhouette Coefficient for a set of samples is given as the mean of the
 Silhouette Coefficient for each sample. In normal usage, the Silhouette
 Coefficient is applied to the results of a cluster analysis.

\begin{description}
	\item[Advantages] \hfill \\
	\begin{itemize}
		\item The score is bounded between -1 for incorrect
		clustering and +1 for highly dense clustering. Scores around
		 zero indicate overlapping clusters.
		\item The score is higher when clusters are dense and well
		 separated, which relates to a standard concept of a cluster.
	\end{itemize}
	\item[Drawbacks] \hfill \\
	The Silhouette Coefficient is generally higher for convex clusters
	than other concepts of clusters, such as density based clusters like
	 those obtained through DBSCAN.
\end{description}

\paragraph{Decomposing signals in components (matrix factorization problems)}
\label{paragraph:Decomposing}
\subparagraph{Principal component analysis (PCA)}
\label{subparagraph:PCA}

\begin{itemize}
	\item \textbf{Exact PCA and probabilistic interpretation} \\
	Linear Dimensionality reduction using Singular Value Decomposition of
	the data and keeping only the most significant singular vectors to
	 project the data to a lower dimensional space.
	\skltype{Exact PCA}{decomposition|PCA}
	\begin{itemize}
		\item \xmlNode{n\_components}, \xmlDesc{integer, None or String,
		optional field}, Number of components to keep. if
		\item \xmlNode{n\_components} is not set all components are kept,
		\default{all components}
		\item \xmlNode{copy}, \xmlDesc{boolean, optional field}, If False,
		 data passed to fit are overwritten and running fit(X).transform(X)
 		will not yield the expected results, use fit\_transform(X) instead.
		\default{True}
		\item \xmlNode{whiten}, \xmlDesc{boolean, optional field}, When True
		the components\_ vectors are divided by n\_samples times singular
		 values to ensure uncorrelated outputs with unit component-wise
		variances. Whitening will remove some information from the transformed
		 signal (the relative variance scales of the components) but can
		sometime improve the predictive accuracy of the downstream estimators
		 by making there data respect some hard-wired assumptions. \default{False}
	\end{itemize}
\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
      <PostProcessor name='PostProcessorName' subType='DataMining'>
          <KDD lib='SciKitLearn'>
              <Features>variable1,variable2,variable3, variable4,variable5</Features>
              <SKLtype>decomposition|PCA</SKLtype>
              <n_components>2</n_components>
          </KDD>
      </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}

	\item \textbf{Randomized {(Approximate)} PCA} \\
	Linear Dimensionality reduction using Singular Value Decomposition of the data
	and keeping only the most significant singular vectors to project the data to a
	 lower dimensional space.
	\skltype{Randomized PCA}{decomposition|RandomizedPCA}
	\begin{itemize}
		\item \xmlNode{n\_components}, \xmlDesc{interger, None or String,
		optional field}, Number of components to keep. if n\_components is
		not set all components are kept.\default{all components}
		\item \xmlNode{copy}, \xmlDesc{boolean, optional field}, If False,
		 data passed to fit are overwritten and running fit(X).transform(X)
 		will not yield the expected results, use fit\_transform(X) instead.
		\default{True}
		\item \xmlNode{iterated\_power}, \xmlDesc{integer, optional field},
		Number of iterations for the power method. \default{3}
		\item \xmlNode{whiten}, \xmlDesc{boolean, optional field}, When True
		the components\_ vectors are divided by n\_samples times singular
		 values to ensure uncorrelated outputs with unit component-wise
		variances. Whitening will remove some information from the transformed
		 signal (the relative variance scales of the components) but can
		sometime improve the predictive accuracy of the downstream estimators
		 by making there data respect some hard-wired assumptions. \default{False}
		\item \xmlNode{random\_state}, \xmlDesc{int, or Random State instance
		or None, optional field}, Pseudo Random Number generator seed control.
		 If None, use the numpy.random singleton. \default{None}
	\end{itemize}
	\item \textbf{Kernel PCA} \\
	Non-linear dimensionality reduction through the use of kernels.
	\skltype{Kernel PCA}{decomposition|KernelPCA}
	\begin{itemize}
		\item \xmlNode{n\_components}, \xmlDesc{interger, None or String,
		optional field}, Number of components to keep. if n\_components is
		not set all components are kept.\default{all components}
		\item \xmlNode{kernel}, \xmlDesc{string, optional field}, name of
		the kernel to be used, options are:
		\begin{itemize}
			\item linear
			\item poly
			\item rbf
			\item sigmoid
			\item cosine
			\item precomputed
		\end{itemize}
		\default{linear}
		\xmlNode{degree}, \xmlDesc{integer, optional field}, Degree for poly
		 kernels, ignored by other kernels. \default{3}
		\xmlNode{gamma}, \xmlDesc{float, optional field}, Kernel coefficient
		 for rbf and poly kernels, ignored by other kernels. \default{1/n\_features}
		\item \xmlNode{coef0}, \xmlDesc{float, optional field}, independent term in
		 poly and sigmoig kernels, ignored by other kernels.
		\item \xmlNode{kernel\_params}, \xmlDesc{mapping of string to any, optional
		 field}, Parameters (keyword arguments) and values for kernel passed as
		callable object. Ignored by other kernels. \default{3}
		\item{alpha}, \xmlDesc{int, optional field}, Hyperparameter of the ridge
		regression that learns the inverse transform (when fit\_inverse\_transform=True).
		\default{1.0}
		\item \xmlNode{fit\_inverse\_transform}, \xmlDesc{bool, optional field},
		Learn the inverse transform for non-precomputed kernels. (i.e. learn to find
		 the pre-image of a point) \default{False}
		\item \xmlNode{eigen\_solver}, \xmlDesc{string, optional field}, Select eigensolver
		 to use. If n\_components is much less than the number of training samples,
		arpack may be more efficient than the dense eigensolver. Options are:
		\begin{itemize}
			\item auto
			\item dense
			\item arpack
		\end{itemize} \default{False}
		\item{tol}, \xmlDesc{float, optional field}, convergence tolerance for arpack.
		\default{0 (optimal value will be chosen by arpack)}
		\item{max\_iter}, \xmlDesc{int, optional field}, maximum number of iterations
		for arpack. \default{None (optimal value will be chosen by arpack)}
		\item \xmlNode{remove\_zero\_eig}, \xmlDesc{boolean, optional field}, If True,
		 then all components with zero eigenvalues are removed, so that the number of
		 components in the output may be < n\_components (and sometimes even zero due
		 to numerical instability). When n\_components is None, this parameter is
		 ignored and components with zero eigenvalues are removed regardless. \default{True}
	\end{itemize}
	\item \textbf{Sparse PCA} \\
	Finds the set of sparse components that can optimally reconstruct the data. The amount
	of sparseness is controllable by the coefficient of the L1 penalty, given by the
	parameter alpha.
	\skltype{Sparse PCA}{decomposition|SparsePCA}
	\begin{itemize}
		\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, Number of
		sparse atoms to extract. \default{None}
		\item \xmlNode{alpha}, \xmlDesc{float, optional field}, Sparsity controlling
		 parameter. Higher values lead to sparser components. \default{1.0}
		\item \xmlNode{ridge\_alpha}, \xmlDesc{float, optional field}, Amount of ridge
		 shrinkage to apply in order to improve conditioning when calling the transform
		 method. \default{0.01}
		\item \xmlNode{max\_iter}, \xmlDesc{float, optional field}, maximum number of
		iterations to perform. \default{1000}
		\item \xmlNode{tol}, \xmlDesc{float, optional field}, convergence tolerance.
		\default{1E-08}
		\item \xmlNode{method}, \xmlDesc{string, optional field}, method to use,
		options are:
		\begin{itemize}
			\item lars: uses the least angle regression method to solve the lasso
			 problem (linear\_model.lars\_path)
			\item cd: uses the coordinate descent method to compute the Lasso
			solution (linear\_model.Lasso)
		\end{itemize}
		Lars will be faster if the estimated components are sparse. \default{lars}
		\item \xmlNode{n\_jobs}, \xmlDesc{int, optional field}, number of parallel
		 runs to run. \default{1}
		\item \xmlNode{U\_init}, \xmlDesc{array of shape (n\_samples, n\_components)
		, optional field}, Initial values for the loadings for warm restart scenarios
		\default{None}
		\item \xmlNode{V\_init}, \xmlDesc{array of shape (n\_components, n\_features),
		 optional field}, Initial values for the components for warm restart scenarios
		\default{None}
		\item{verbose}, \xmlDesc{boolean, optional field}, Degree of verbosity of the
		 printed output. \default{False}
		\item{random\_state}, \xmlDesc{int or Random State, optional field}, Pseudo
		number generator state used for random sampling. \default{None}
	\end{itemize}
	\item \textbf{Mini Batch Sparse PCA} \\
	Finds the set of sparse components that can optimally reconstruct the data. The amount
	 of sparseness is controllable by the coefficient of the L1 penalty, given by the
	parameter alpha.
	\skltype{Mini Batch Sparse PCA}{decomposition|MiniBatchSparsePCA}
	\begin{itemize}
		\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, Number of
		 sparse atoms to extract. \default{None}
		\item \xmlNode{alpha}, \xmlDesc{float, optional field}, Sparsity controlling
		parameter. Higher values lead to sparser components. \default{1.0}
		\item \xmlNode{ridge\_alpha}, \xmlDesc{float, optional field}, Amount of ridge
		 shrinkage to apply in order to improve conditioning when calling the transform
		 method. \default{0.01}
		\item \xmlNode{n\_iter}, \xmlDesc{float, optional field}, number of iterations
		to perform per mini batch. \default{100}
		\item \xmlNode{callback}, \xmlDesc{callable, optional field}, callable that
		gets invoked every five iterations. \default{None}
		\item \xmlNode{batch\_size}, \xmlDesc{int, optional field}, the number of
		features to take in each mini batch. \default{3}
		\item \xmlNode{verbose}, \xmlDesc{boolean, optional field}, Degree of verbosity
		 of the printed output. \default{False}
		\item \xmlNode{shuffle}, \xmlDesc{boolean, optional field}, whether to shuffle
		the data before splitting it in batches. \default{True}
		\item \xmlNode{n\_jobs}, \xmlDesc{integer, optional field}, Parameters (keyword
		arguments) and values for kernel passed as callable object. Ignored by other
		kernels. \default{3}
		\item \xmlNode{metho}, \xmlDesc{string, optional field}, method to use,
		options are:
		\begin{itemize}
			\item lars: uses the least angle regression method to solve the lasso
			 problem (linear\_model.lars\_path),
			\item cd: uses the coordinate descent method to compute the Lasso solution
			 (linear\_model.Lasso)
		\end{itemize}
		Lars will be faster if the estimated components are sparse. \default{lars}
		\item \xmlNode{random\_state}, \xmlDesc{integer or Random State, optional field},
		 Pseudo number generator state used for random sampling. \default{None}
	\end{itemize}
\end{itemize}

\subparagraph{Truncated singular value decomposition} \hfil \\
\label{subparagraph:TruncatedSVD}
Dimensionality reduction using truncated SVD (aka LSA).
\skltype{Truncated SVD}{decomposition|TruncatedSVD}
\begin{itemize}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, Desired dimensionality
	of output data. Must be strictly less than the number of features. The default value is
	useful for visualisation. For LSA, a value of 100 is recommended. \default{2}
	\item \xmlNode{algorithm}, \xmlDesc{string, optional field}, SVD solver to use:
	\begin{itemize}
		\item Randomized: randomized algorithm
		\item Arpack: ARPACK wrapper in.
	\end{itemize}
	\default{Randomized}
	\item \xmlNode{n\_iter}, \xmlDesc{float, optional field}, number of iterations andomized
	SVD solver. Not used by ARPACK. \default{5}
	\item \xmlNode{random\_state}, \xmlDesc{int or Random State, optional field}, Pseudo number
	generator state used for random sampling. If not given, the numpy.random singleton is used.
	\default{None}
	\item \xmlNode{tol}, \xmlDesc{float, optional field}, Tolerance for ARPACK. 0 means machine
	precision. Ignored by randomized SVD solver. \default{0.0}
\end{itemize}

\subparagraph{Fast ICA} \hfil \\
\label{subparagraph:FastICA}
A fast algorithm for Independent Component Analysis.
\skltype{Fast ICA}{decomposition|FastICA}
\begin{itemize}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, Number of components to
	use. If none is passed, all are used. \default{None}
	\item \xmlNode{algorithm}, \xmlDesc{string, optional field}, algorithm used in FastICA:
	\begin{itemize}
		\item parallel,
		\item deflation.
	\end{itemize}
	\default{parallel}
	\item \xmlNode{fun}, \xmlDesc{string or function, optional field}, The functional form of
	the G function used in the approximation to neg-entropy. Could be either:
	\begin{itemize}
		\item logcosh,
		\item exp, or
		\item cube.
	\end{itemize}
	One can also provide own function. It should return a tuple containing the value of the
	 function, and of its derivative, in the point. \default{logcosh}
	\item \xmlNode{fun\_args}, \xmlDesc{dictionary, optional field}, Arguments to send to the
	functional form. If empty and if fun=’logcosh’, fun\_args will take value {‘alpha’ : 1.0}.
	\default{None}
	\item \xmlNode{max\_iter}, \xmlDesc{float, optional field}, maximum number of iterations
	 during fit. \default{200}
	\item \xmlNode{tol}, \xmlDesc{float, optional field}, Tolerance on update at each iteration.
	\default{0.0001}
	\item \xmlNode{w\_init}, \xmlDesc{None or an (n\_components, n\_components) ndarray,
	optional field}, The mixing matrix to be used to initialize the algorithm. \default{None}
	\item \xmlNode{randome\_state}, \xmlDesc{int or Random State, optional field}, Pseudo number
	 generator state used for random sampling. \default{None}
\end{itemize}

\paragraph{Manifold learning}
\label{paragraph:Manifold}
A manifold is a topological space that resembles a Euclidean space locally at each point. Manifold
learning is an approach to non-linear dimensionality reduction. It assumes that the data of interest
 lie on an embedded non-linear manifold within the higher-dimensional space. If this manifold is of
 low dimension, data can be visualized in the low-dimensional space. Algorithms for this task are
based on the idea that the dimensionality of many data sets is only artificially high.
\subparagraph{Isomap} \hfil \\
\label{subparagraph:Isomap}
Non-linear dimensionality reduction through Isometric Mapping (Isomap).
\skltype{Isometric Mapping}{manifold|Isomap}
\begin{itemize}
	\item \xmlNode{n\_neighbors}, \xmlDesc{integer, optional field}, Number of neighbors to
	consider for each point. \default{5}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, Number of coordinates to
	manifold. \default{2}
	\item \xmlNode{eigen\_solver}, \xmlDesc{string, optional field}, eigen solver to use:
	\begin{itemize}
		\item auto: Attempt to choose the most efficient solver for the given problem,
		\item arpack: Use Arnoldi decomposition to find the eigenvalues and eigenvectors
		\item dense: Use a direct solver (i.e. LAPACK) for the eigenvalue decomposition
	\end{itemize}
	\default{auto}
	\item \xmlNode{tol}, \xmlDesc{float, optional field}, Convergence tolerance passed to
	arpack or lobpcg. not used if eigen\_solver is ‘dense’. \default{0.0}
	\item \xmlNode{max\_iter}, \xmlDesc{float, optional field}, Maximum number of iterations
	for the arpack solver. not used if eigen\_solver == ‘dense’. \default{None}
	\item \xmlNode{path\_method}, \xmlDesc{string, optional field}, Method to use in finding
	 shortest path. Could be either:
	\begin{itemize}
		\item Auto: attempt to choose the best algorithm
		\item FW: Floyd-Warshall algorithm
		\item D: Dijkstra algorithm with Fibonacci Heaps
	\end{itemize}
	\default{auto}
	\item \xmlNode{neighbors\_algorithm}, \xmlDesc{string, optional field}, Algorithm to use
	 for nearest neighbors search, passed to neighbors.NearestNeighbors instance.
	\begin{itemize}
		\item auto,
		\item brute
		\item kd\_tree
		\item ball\_tree
	\end{itemize}
	\default{auto}
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
      <PostProcessor name='PostProcessorName' subType='DataMining'>
          <KDD lib='SciKitLearn'>
              <Features>input</Features>
              <SKLtype>manifold|Isomap</SKLtype>
              <n_neighbors>5</n_neighbors>
	      <n_components>3</n_components>
	      <eigen_solver>arpack</eigen_solver>
	      <neighbors_algorithm>kd_tree</neighbors_algorithm>
            </KDD>
      </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}

\subparagraph{Locally Linear Embedding} \hfil \\
\label{subparagraph:LLE}
\skltype{Locally Linear Embedding}{manifold|LocallyLinearEmbedding}
\begin{itemize}
	\item \xmlNode{n\_neighbors}, \xmlDesc{integer, optional field}, Number of neighbors to
	consider for each point. \default{5}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, Number of coordinates to
	 manifold. \default{2}
	\item \xmlNode{reg}, \xmlDesc{float, optional field}, regularization constant, multiplies
	the trace of the local covariance matrix of the distances. \default{0.01}
	\item \xmlNode{eigen\_solver}, \xmlDesc{string, optional field}, eigen solver to use:
	\begin{itemize}
		\item auto: Attempt to choose the most efficient solver for the given problem,
		\item arpack: use arnoldi iteration in shift-invert mode.
		\item dense: use standard dense matrix operations for the eigenvalue
	\end{itemize}
	\default{auto}
	\item \xmlNode{tol}, \xmlDesc{float, optional field}, Convergence tolerance passed to arpack.
	 not used if eigen\_solver is ‘dense’. \default{1E-06}
	\item \xmlNode{max\_iter}, \xmlDesc{int, optional field}, Maximum number of iterations for the
	 arpack solver. not used if eigen\_solver == ‘dense’. \default{100}
	\item \xmlNode{method}, \xmlDesc{string, optional field}, Method to use. Could be either:
	\begin{itemize}
		\item Standard: use the standard locally linear embedding algorithm
		\item hessian: use the Hessian eigenmap method
		\item itsa: use local tangent space alignment algorithm
	\end{itemize}
	\default{standard}
	\item \xmlNode{hessian\_tol}, \xmlDesc{float, optional field}, Tolerance for Hessian eigenmapping
	 method. Only used if method == 'hessian' \default{0.0001}
	\item \xmlNode{modified\_tol}, \xmlDesc{float, optional field}, Tolerance for modified LLE method.
	 Only used if method == 'modified' \default{0.0001}
	\item \xmlNode{neighbors\_algorithm}, \xmlDesc{string, optional field}, Algorithm to use for nearest
	 neighbors search, passed to neighbors.NearestNeighbors instance.
	\begin{itemize}
		\item auto,
		\item brute
		\item kd\_tree
		\item ball\_tree
	\end{itemize}
	\default{auto}
	\item \xmlNode{random\_state}, \xmlDesc{int or numpy random state, optional field}, the generator
	or seed used to determine the starting vector for arpack iterations. \default{None}
\end{itemize}
\subparagraph{Spectral Embedding} \hfil \\
\label{subparagraph:Spectral}
Spectral embedding for non-linear dimensionality reduction, it forms an affinity matrix given by the
 specified function and applies spectral decomposition to the corresponding graph laplacian. The resulting
 transformation is given by the value of the eigenvectors for each data point
\skltype{Spectral Embedding}{manifold|SpectralEmbedding}
\begin{itemize}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, the dimension of projected
	sub-space. \default{2}
	\item \xmlNode{eigen\_solver}, \xmlDesc{string, optional field}, the eigen value decomposition
	 strategy to use:
	\begin{itemize}
		\item none,
		\item arpack.
		\item lobpcg,
		\item amg
	\end{itemize}
	\default{none}
	\item \xmlNode{random\_state}, \xmlDesc{integer or numpy random state, optional field}, A
	pseudo random number generator used for the initialization of the lobpcg eigen vectors
	decomposition when eigen\_solver == ‘amg. \default{None}
	\item \xmlNode{affinity}, \xmlDesc{string or callable, optional field}, How to construct
	 the affinity matrix:
	\begin{itemize}
		\item \textit{nearest\_neighbors} : construct affinity matrix by knn graph
		\item \textit{rbf} : construct affinity matrix by rbf kernel
		\item \textit{precomputed} : interpret X as precomputed affinity matrix
		\item \textit{callable} : use passed in function as affinity the function takes
		 in data matrix (n\_samples, n\_features) and return affinity matrix (n\_samples, n\_samples).
	\end{itemize}
	\default{nearest\_neighbor}
	\item \xmlNode{gamma}, \xmlDesc{float, optional field}, Kernel coefficient for rbf kernel.
	\default{None}
	\item \xmlNode{n\_neighbors}, \xmlDesc{int, optional field}, Number of nearest neighbors for
	 nearest\_neighbors graph building. \default{None}
\end{itemize}

\subparagraph{Multi-dimensional Scaling (MDS)} \hfil \\
\label{subparagraph:MDS}

\skltype{Multi Dimensional Scaling}{manifold|MDS}
\begin{itemize}
	\item \xmlNode{metric}, \xmlDesc{boolean, optional field}, compute metric or nonmetric SMACOF
	 (Scaling by Majorizing a Complicated Function) algorithm \default{True}
	\item \xmlNode{n\_components}, \xmlDesc{integer, optional field}, number of dimension in
	which to immerse the similarities overridden if initial array is provided. \default{2}
	\item \xmlNode{n\_init}, \xmlDesc{integer, optional field}, Number of time the smacof
	algorithm will be run with different initialisation. The final results will be the best
	 output of the n\_init consecutive runs in terms of stress. \default{4}
	\item \xmlNode{max\_iter}, \xmlDesc{integer, optional field}, Maximum number of iterations
	of the SMACOF algorithm for a single run \default{300}
	\item \xmlNode{verbose}, \xmlDesc{integer, optional field}, level of verbosity \default{0}
	\item \xmlNode{eps}, \xmlDesc{float, optional field}, relative tolerance with respect
	to stress to declare converge \default{1E-06}
	\item \xmlNode{n\_jobs}, \xmlDesc{integer, optional field}, The number of jobs to use for
	the computation. This works by breaking down the pairwise matrix into n\_jobs even slices
	 and computing them in parallel. If -1 all CPUs are used. If 1 is given, no parallel
	 computing code is used at all, which is useful for debugging. For n\_jobs below -1,
	(n\_cpus + 1 + n\_jobs) are used. Thus for n\_jobs = -2, all CPUs but one are used.
	\default{1}
	\item \xmlNode{random\_state}, \xmlNode{integer or numpy random state, optional field},
	The generator used to initialize the centers. If an integer is given, it fixes the seed.
	Defaults to the global numpy random number generator. \default{None}
	\item \xmlNode{dissimilarity}, \xmlDesc{string, optional field}, Which dissimilarity
	measure to use. Supported are ‘euclidean’ and ‘precomputed’. \default{euclidean}
\end{itemize}

\paragraph{Scipy}
\xmlString{Scipy} provides a Hierarchical clustering that performs clustering over PointSet and HistorySet.
This algorithm also automatically generates a dendrogram in .pdf format (i.e., dendrogram.pdf).

\begin{itemize}
  \item \xmlNode{SCIPYtype}, \xmlDesc{string, required field}, SCIPY algorithm to be employed.
  \item \xmlNode{Features}, \xmlDesc{string, required field}, defines the data to be used for training the data mining algorithm. It can be:
    \begin{itemize}
      \item the name of the variable in the defined dataObject entity
      \item the location (i.e. input or output). In this case the data mining is applied to all the variables in the defined space.
    \end{itemize}
  \item \xmlNode{method},         \xmlDesc{string, required field}, The linkage algorithm to be used  \default{single, complete, weighted, centroids, median, ward}.
  \item \xmlNode{metric},         \xmlDesc{string, required field}, The distance metric to be used \default{ ‘braycurtis’, ‘canberra’, ‘chebyshev’, ‘cityblock’,
                                                                    ‘correlation’, ‘cosine’, ‘dice’, ‘euclidean’, ‘hamming’, ‘jaccard’, ‘kulsinski’,
                                                                    ‘mahalanobis’, ‘matching’, ‘minkowski’, ‘rogerstanimoto’, ‘russellrao’, ‘seuclidean’,
                                                                    ‘sokalmichener’, ‘sokalsneath’, ‘sqeuclidean’, ‘yule’}.
  \item \xmlNode{level},          \xmlDesc{float, required field},  Clustering distance level where actual clusters are formed.
  \item \xmlNode{criterion},      \xmlDesc{string, required field}, The criterion to use in forming flat clusters. This can be any of the following values:
    \begin{itemize}
      \item   ``inconsistent''     : If a cluster node and all its descendants have an inconsistent value less than or equal to `t` then all its leaf descendants
                                     belong to the same flat cluster. When no non-singleton cluster meets this criterion, every node is assigned to its own
                                     cluster. (Default)
      \item   ``distance''         : Forms flat clusters so that the original observations in each flat cluster have no greater a cophenetic distance than $t$.
      \item   ``maxclust''         : Finds a minimum threshold ``r'' so that the cophenetic distance between any two original observations in the same flat cluster
                                     is no more than ``r'' and no more than $t$ flat clusters are formed.
      \item   ``monocrit''         : Forms a flat cluster from a cluster node c with index i when $monocrit[j] <= t$.
       \item  ``maxclust\_monocrit'' : Forms a flat cluster from a non-singleton cluster node ``c'' when $monocrit[i] <= r$ for all cluster indices ``i''
                                       below and including ``c''. ``r'' is minimized such that no more than ``t'' flat clusters are formed. monocrit must be
                                       monotonic.
    \end{itemize}
  \item \xmlNode{dendrogram},     \xmlDesc{boolean, required field}, If True the dendrogram is actually created.
  \item \xmlNode{truncationMode}, \xmlDesc{string, required field}, The dendrogram can be hard to read when the original observation matrix from which the
                                                                    linkage is derived is large. Truncation is used to condense the dendrogram. There are several
                                                                    modes:
                                                                    \begin{itemize}
                                                                      \item ``None'': No truncation is performed (Default).
                                                                      \item ``lastp'': The last p non-singleton formed in the linkage are the only non-leaf nodes
                                                                       in the linkage; they correspond to rows $Z[n-p-2:end]$ in Z. All other non-singleton
                                                                       clusters are contracted into leaf nodes.
                                                                      \item ``level''/``mtica'': No more than p levels of the dendrogram tree are displayed.
                                                                      This corresponds to Mathematica behavior.
                                                                    \end{itemize}
  \item \xmlNode{p},              \xmlDesc{int, required field},     The $p$ parameter for truncationMode.
  \item \xmlNode{leafCounts},     \xmlDesc{boolean, required field}, When True the cardinality non singleton nodes contracted into a leaf node is indiacted in
                                                                     parenthesis.
  \item \xmlNode{showContracted}, \xmlDesc{boolean, required field}, When True the heights of non singleton nodes contracted into a leaf node are plotted as
                                                                     crosses along the link connecting that leaf node.
  \item \xmlNode{annotatedAbove}, \xmlDesc{float, required field},  Clustering level above which the branching level is annotated.
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={subType}]
<Simulation>
  ...
  <Models>
    ...
    <PostProcessor name="hierarchical" subType="DataMining" verbosity="quiet">
      <KDD lib="Scipy" labelFeature='labels'>
        <SCIPYtype>cluster|Hierarchical</SCIPYtype>
        <Features>output</Features>
        <method>single</method>
        <metric>euclidean</metric>
        <level>75</level>
        <criterion>distance</criterion>
        <dendrogram>true</dendrogram>
        <truncationMode>lastp</truncationMode>
        <p>20</p>
        <leafCounts>True</leafCounts>
        <showContracted>True</showContracted>
        <annotatedAbove>10</annotatedAbove>
      </KDD>
    </PostProcessor>
    ...
  <Models>
  ...
<Simulation>
\end{lstlisting}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  GEANT manual in LaTeX form                                     %
%                                                                 %
%  Michel Goossens (for translation into LaTeX)                   %
%  Version 1.00                                                   %
%  Last Mod. Jan 24 1991  1300   MG + IB                          %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Origin{G.Lynch}
\Documentation{F.Carminati}
\Submitted {23.08.93}   \Revised {16.12.93}
\Version{Geant 3.16}    \Routid{PHYS328}
\Makehead {Plural scattering}
\section{Subroutines}

\Shubr{GMCOUL}{(OMEGA,DIN*)}
\begin{DLtt}{MMMMMMMM}
\item[OMEGA] ({\tt REAL}) $\Omega_0$ of the Moli\`ere theory;
\item[DIN]  ({\tt REAL}) array of dimension 3 containing the
direction of the scattered particle with respect to the
incoming direction;
\end{DLtt}
Generates single scatters in small angle approximation returning
the new direction with respect to the old one.
It is called by \Rind{GMULTS} when $\Omega_0 \leq 20$.

\section{Method}

In the Moli\`{e}re theory the average number of Coulomb scatters
for a charged particle in a step is
expressed by the parameter $\Omega_0$ (see {\tt [PHYS325]}).
When
$\Omega_0 \leq 20$, the Moli\`{e}re theory is not applicable any
more, even if it has been noted \cite{bib-THEO} that it still gives reasonable
results down to its {\it mathematical limit} $\Omega_0 = e$.
The range $1<\Omega_0 \leq 20$ is called the {\it plural
scattering} regime.
An interesting study of this regime can be found in \cite{bib-KEIL}.

In {\tt GEANT}, when $\Omega_0 \leq 20$, a direct simulation method
is used for the scattering angle.
The number of scatters is distributed according to a Poissonian law
with average $\Omega = k \Omega_0 = e^{2\gamma-1} \Omega_0
\approx 1.167 \Omega_0$ with $\gamma = 0.57721\dots$ Euler's constant.
Using the customary notations for
the probability distribution function for small
angle ($\sin \theta \approx \theta$) single scattering, we
can write:

\begin{equation}
\label{eq:phys328-1}
f(\theta) \theta d\theta = \frac{d \sigma}{\theta d\theta} \frac{1}
{\sigma} \theta d\theta
\end{equation}

where $\sigma$ is the cross section for single elastic scattering.
We use as cross section the one reported by Moli\`{e}re \cite{bib-MOLI}
\cite{bib-MOL1}:

\[
\frac{d \sigma}{\theta d\theta} = 2 \pi \left ( \frac{2 Z Z_{inc}
e^2}{p v} \right ) ^2 \frac{1}{(\theta^2 + \chi_{\alpha}^2)^2}
\]

This is the classical Rutherford cross section corrected by the
screening angle $\chi_{\alpha}$. This angle is described by Moli\`{e}re
as a correction to the Born approximation used to derive the
quantum mechanical form of the Rutherford cross section. We have then:

\[
\sigma = \int_{0}^{\infty}{\frac{d \sigma}{\theta d\theta} \theta d\theta}
= 2 \pi \left ( \frac{2 Z Z_{inc}e^2}{p v} \right ) ^2
\int_{0}^{\infty}{\frac{\theta d \theta}{(\theta^2 + \chi_{\alpha}^2)^2}} =
\pi \left ( \frac{2 Z Z_{inc}e^2}{p v} \right ) ^2
\frac{1}{\chi_{\alpha}^2}
\]

and so equation (\ref{eq:phys328-1}) becomes:

\[
f(\theta) \theta d\theta = \chi_{\alpha}^2 \frac{2 \theta d \theta}
{(\theta^2 + \chi_{\alpha}^2)^2} = \frac{ 2 \Theta d \Theta}{(1+\Theta^2)^2}
\]

where we have set $\Theta = \theta / \chi_{\alpha}$. To sample from this
distribution we calculate the inverse of the distribution function:

\[
\eta = \int_{0}^{\Theta}{\frac{2 t d t}{(1+t^2)^2}}
 =  1 - \frac{1}{1+\Theta^2} \hspace{1cm} \Rightarrow \hspace{1cm}
\Theta = \sqrt{\frac{1}{1-\eta}-1}
\]

where $\eta$ is a number uniformly distributed between 0 and 1. If we
observe that also $1-\eta$ is uniformly distributed between 0 and 1 and
we remember the definition of $\Theta$, we obtain:

\[
\theta = \sqrt{\chi_{\alpha}^2 \left (\frac{1}{\eta}-1 \right)}
\]

To calculate $\chi_{\alpha}^2$ we observe that:

\begin{eqnarray*}
\Omega_0 & = & \frac{\chi_{c}^2}{k \chi_{\alpha}^2} \\
\chi_{\alpha}^2 & = & \frac{\chi_{c}^2}{k \Omega_0} =
\frac{\chi_{cc}^2 Z_{inc}^2 t/E^2\beta^4}{k b_{c} Z_{inc}^2 t /\beta^2} =
\frac{\chi_{cc}^2}{k b_{c} p^2 c^2}
\end{eqnarray*}

where we have used the notations of {\tt [PHYS325]} and
$k = 1.167$.


  \section{Introduction}

Since the early developments of population genetics, geography has been recognized as one of the major determinants of genetic variation in natural populations~\citep{wright1943isolation,malecot1948mathematics,kimura1964stepping,cavalli1994history,
epperson2003geographical}. For these populations, spatial patterns of genetic variation can be influenced by landscape barriers, geographical distances and by the processes of divergence and admixture resulting from the colonization of new areas. In additions, analyzing spatial patterns of genetic variation has been a long-standing goal of evolutionary biogeography, molecular ecology, landscape genetics, and conservation biology~\citep{segelbacher2010applications,manel2010perspectives}.

Statistical approaches to analyze spatial patterns of genetic variation often rely on the inference of population genetic structure from multi-locus genotype data, which is commonly performed using the Bayesian approach implemented in the computer program {\tt STRUCTURE}~\citep{pritchard2000inference}.  Assuming $K$ unobserved ancestral gene pools, {\tt STRUCTURE} computes allele frequencies in each pool, and estimates individual {\it ancestry coefficients} representing the proportion of an individual genome that originates from each pool. Using {\tt STRUCTURE}, ancestry coefficients are estimated without prior knowledge on geographic proximity among individuals.

The approach implemented in {\tt STRUCTURE} has been substantially improved by a number of approaches that include spatial proximity information based on individual geographic coordinates (reviewed by~\cite{franccois2010spatially}). Among those spatially explicit approaches, the computer program {\tt TESS} is one of the most frequently used algorithms~\citep{chen2007bayesian,franccois2006bayesian}. In the {\tt TESS} model, ancestry proportions are continuously distributed over geographic space, and the parameters that specify the shape of the clines are estimated from the genetic and the geographic data. Using geographic information, {\tt TESS} provides better estimates of ancestry coefficients than {\tt STRUCTURE} when the levels of ancestral population divergence are low (Durand et al. 2009).

The Bayesian approaches implemented in {\tt STRUCTURE} and {\tt TESS} rely on Markov Chain Monte Carlo algorithms. Monte Carlo algorithms are based on computer intensive stochastic simulations, and have the advantage of sampling the posterior distribution of the model parameters. However the application of stochastic algorithms can be difficult when data include more than a few hundreds of individuals or a few thousands of allelic markers. With the availability of next generation sequencing data, there is a need to analyze genotypic matrices that represent thousands of individuals and hundreds of thousands of markers. While fast versions of {\tt STRUCTURE} have already been proposed~\citep{raj2014faststructure, frichot2014fast, alexander2011linking, wollstein2015detecting}, developing fast and accurate estimation of ancestry coefficients in a geographic framework remains an important computational challenge.

In this study, we present a spatially explicit algorithm that provides fast estimation of ancestry coefficients with accuracy comparable to the one obtained with {\tt TESS} 2.3~\citep{durand2009spatial}. The new algorithms are based on least-squares optimization and on geographically constrained non-negative matrix factorization~\citep{cai2011graph,frichot2014fast}. These improvements of {\tt TESS} are implemented in the computer program {\tt TESS3}. We show that {\tt TESS3} is substantially faster than {\tt TESS} 2.3, with an increase in computational speed of several orders of magnitude. In addition, we show that ancestral allele frequencies are correctly estimated, and we illustrate the use of {\tt TESS3} program to perform genome scans for selection based on ancestral allele frequency differentiation. To illustrate our approach, {\tt TESS3} was applied to genomic data from European lines of the model species {\it Arabidopsis thaliana} for which an individual-based sampling design is available~\citep{atwell2010genome}.

%Cavalli-Sforza, L. L., Menozzi, P., Piazza, A. (1994). The history and geography of human genes. Princeton university press.

%Epperson, B. K. (2003). Geographical Genetics. Princeton University Press.

%Kimura, M.,  Weiss, G. H. (1964). The stepping stone model of population structure and the decrease of genetic correlation with distance. Genetics, 49(4), 561.

%Wright, S. (1943). Isolation by distance. Genetics, 28(2), 114.

%Segelbacher, G., Cushman, S. A., Epperson, B. K., Fortin, M. J., Francois, O., et al. (2010). Applications of landscape genetics in conservation biology: concepts and challenges. Conservation Genetics, 11(2), 375-385.

%Raj A, Stephens M, Pritchard JK: fastSTRUCTURE: Variational Inference of Population Structure in Large SNP Data Sets. Genetics 2014, 197:573-589.


\documentclass[a4paper]{article}

%% Language and font encodings
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{mlka_math}

%% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

%% Useful packages
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{neuralnetwork}
\usepackage{sidecap}

\newtheorem{theorem}{Theorem}[section]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}

\title{Cheatsheet for Fisher-Rao Metric, Geometry, and Complexity of Neural Networks}
\author{Sandro Braun, Leander Kurscheidt}

%%% --- commands ---

%network notation with nonlinearities
\include{definitions}

\begin{document}
\maketitle
\begin{abstract}
\end{abstract}
\section{Geometry of Deep Rectified Networks}

\subsection{Lemma 2.1}

\begin{lemma}[Structure in Gradients]
	\begin{align}
		\label{lemma_structure_in_gradient}
		\sum_{t=0}^{L} \sum_{i \in [k_t], j \in [k_{t+1}]} \frac{\partial O^{L+1}}{\partial W^{t_{ij}}} W^t_{ij} = (L+1)O^{L+1}(x) = \langle \nabla_\theta f_\theta (x), \theta \rangle
	\end{align}
\end{lemma}

\subsubsection{Example for Lemma 2.1 with single output}

Take a simple feed forward neural network with only two inputs and one output. Remember that by definition $O^0 = x$.
Illustrated: \\

\begin{neuralnetwork}[height=2]
	\newcommand{\nodetextx}[2]{$x_#2$}
	\newcommand{\nodetexty}[2]{$y_#2$}
	\inputlayer[count=2, bias=false, title=$O^0$, text=\nodetextx]
	\outputlayer[count=1, title=$O^1$, text=\nodetexty] \linklayers
\end{neuralnetwork}

The simple example can then be written with:

\begin{align}
	\label{eq_simple_nn}
	O^0 = 	\begin{bmatrix}
				O^0_1 \\
				O^0_2
			\end{bmatrix} &,
	W^0 = 	\begin{bmatrix}
				W^0_1 \\
				W^0_2
			\end{bmatrix} \\
			O^1 &= \sigma({O^0}^T W^0) = \sigma(
											\underbrace{O^0_1 W^0_1 + O^0_2 W^0_2}_z
											).
\end{align}
Substituting the term in brackets will simplify notation.

%\begin{minipage}{.5\textwidth}
Now calculate the partial derivatives of \eqref{eq_simple_nn} with respect to $W^0_i$.
\begin{align*}
	\frac{\partial O^1}{\partial W^0_1} &= \sigma'(z) O^0_1 \\
	\frac{\partial O^1}{\partial W^0_2} &= \sigma'(z) O^0_2
\end{align*}
Summing up the $\frac{\partial O^1}{\partial W_j} W_j$ reveals,

\begin{align*}
	\sum_{j=1}^{2} \frac{\partial O^1}{\partial W^0_j} W^0_j &= \sigma'(z) O^0_1  W^0_1 + \sigma'(z) O^0_2  W^0_2 = \sigma'(z)(\underbrace{O^0_1 W^0_1 + O^0_2 W^0_2}_\text{z}) \\
														&= \sigma'(z) z
\end{align*}
Note that this is equivalent to calculating $\langle \nabla_{W^0} O^1, W^0 \rangle$:
\begin{align*}
	\nabla_{W^0} O^1 &=
		\begin{bmatrix}
			\frac{\partial O^1}{\partial W^0_1} \\
			\frac{\partial O^1}{\partial W^0_2}
		\end{bmatrix} \\
	\langle \nabla_{W^0} O^1, W^0 \rangle &= \sum_{j=1}^{2} \frac{\partial O^1}{\partial W^0_j} W^0_j = \dots = \sigma'(z) z
\end{align*}
Using the relation $\sigma(z) = \sigma'(z)(z)$ reveals \eqref{eq_simple_nn}. Then reading left to right reveals \eqref{lemma_structure_in_gradient} for $L=0$, $\theta= W^0$ and $f_\theta(x) = O^1(x)$ which completes the example.

\begin{align*}
		\sum_{t=0}^{L=0} \sum_{j=1}^{2} \frac{\partial O^1}{\partial W^t_j} W^t_j
		=  \langle \nabla_{W^0} O^1, W^0 \rangle
		= \sigma'(z) z
		= \sigma(z) = O^1
\end{align*}

%\end{minipage}%

\subsubsection{Applying Lemma 2.1 to the network structure}

Lemma 2.1 applied to the network notation leads to the following notation:

\begin{align}
	\label{eq:lemma2_1}
	\text{Lemma 2.1} : \sigma(z) &= \sigma'(z) z \\
	\label{eq:basicNN}
	\text{Network notation} : f_\theta(x) &= \basicNN \\
	\label{eq:structuredNN}
	\text{combined} : f_\theta(x) &= \structuredNN
\end{align}

\subsubsection{Example for Lemma 2.1 with multiple outputs}
	Take again a simple neural network consisting of a single layer, hence $L=0$. We will now start from \eqref{eq:basicNN} for $L=0$, then use \eqref{eq:lemma2_1} to get \eqref{eq:structuredNN}.
	Now assume, w.l.o.g. the network has two inputs and two outputs. We then have
	\begin{align}
	O^0 &= 	\begin{bmatrix}
				O^0_1 \\
				O^0_2
			\end{bmatrix}, ~
	O^1 = 	\begin{bmatrix}
				O^1_1 \\
				O^1_2
			\end{bmatrix}, ~
	W^0 = 	\begin{bmatrix}
				W^0_{1,1}  & W^0_{1,2} \\
				W^0_{2,1}  & W^0_{2,2}
			\end{bmatrix},	\\
	\label{eq:2_output_example}
	O^1 &=
			\sigma(O^{0^T} W^0) \\
		&=
			\sigma(
			\begin{bmatrix}
				O^0_1 W^0_{1,1}  + O^0_2 W^0_{1,2} \\
				O^0_1 W^0_{2,1}  + O^0_2 W^0_{2,2}
			\end{bmatrix})  \\
		&=
			\begin{bmatrix}
				\sigma(O^0_1 W^0_{1,1}  + O^0_2 W^0_{1,2}) \\
				\sigma(O^0_1 W^0_{2,1}  + O^0_2 W^0_{2,2})
			\end{bmatrix} \\
		&=
			\begin{bmatrix}
				\sigma(N^1_1) \\
				\sigma(N^1_2)
			\end{bmatrix} \\
	\end{align}
	For simplicity, we can replace the sum within each vector element with $N^1_i$. Now remember \eqref{eq:lemma2_1} and apply it to \eqref{eq:2_output_example}.
	\begin{align}
	O^1 &=
			\begin{bmatrix}
				\sigma(N^1_1) \\
				\sigma(N^1_2)
			\end{bmatrix} \\
		&=
			\begin{bmatrix}
				\sigma'(N^1_1) N^1_1 \\
				\sigma'(N^1_2) N^1_2
			\end{bmatrix} \\
		&=
			\begin{bmatrix}
				N^1_1 & N^1_2
			\end{bmatrix}
			\underbrace{
				\begin{bmatrix}
					\sigma'(N^1_1)	&	0\\
					0				&	\sigma'(N^1_2)
				\end{bmatrix}
			}_{D^1(x)} \\
		&=
			N^1 D^1(x)
	\end{align}
	The last step is to remember $N^{i+1} = O^{i^\TT}W^1$, then we have
	\begin{align}
		O^1 &=
				{O^0}^\TT W^0 D^1(x) \\
			&=
				{x}^\TT W^0 D^1(x)
	\end{align}
	which is equivalent to inserting $L=0$ to \eqref{eq:structuredNN}. This completes the example.

\subsection{Corollary 2.1}

\subsubsection{Notes}
\begin{enumerate}
	\item
	\begin{proof}
		We want to show $\frac{\partial l(f,Y)}{\partial f} = -y \Leftrightarrow yf < 1$. So,
		\begin{alignat*}{2}
			                && 1-y_if_i &> 0 \\
			\Leftrightarrow && l &=1-y_if_i \\
			\Leftrightarrow && \frac{\partial l}{\partial f} &=-y_i
		\end{alignat*}
	  \end{proof}
	\item
	\begin{proof}
		We want to show $\frac{\partial l(f,Y)}{\partial f} = 0 \Leftrightarrow yf > 1$. So,
		\begin{alignat*}{2}
							&& 1-y_if_i &< 0 \\
			\Leftrightarrow && l &=0 \\
			\Leftrightarrow && \frac{\partial l}{\partial f} &=0
		\end{alignat*}
	  \end{proof}
\end{enumerate}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/01.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/02.jpg}
\end{figure}
\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/03.jpg}
\end{figure}
\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/04.jpg}
\end{figure}
\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/05.jpg}
\end{figure}
\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/3_1_proof.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/3_1_proof_cont.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/07.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
\includegraphics[width=\textwidth]{whiteboard_notes/08.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/09.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/10.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/11.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/12.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/13.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/14.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/15.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/16.jpg}
\end{figure}

\begin{figure}[htb]
	\centering
	\includegraphics[width=\textwidth]{whiteboard_notes/17.jpg}
\end{figure}

\include{section4}

\bibliographystyle{alpha}
\bibliography{sample}

\end{document}


\documentclass[12pt]{article}
\usepackage{fancyhdr}
\usepackage{color}
\usepackage{multicol}
\usepackage{fancyvrb}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{sectsty}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{array}
\newcommand{\sectionbreak}{\clearpage}

\usepackage{tikz}
\usepackage{tkz-euclide}
\usetkzobj{all}

\allsectionsfont{\centering}

\usepackage{draftwatermark}
	\SetWatermarkText{\copyright wolf-math.com}
	\SetWatermarkScale{4}
	\SetWatermarkLightness{1}

\usepackage[margin=1in, headsep=0pt]{geometry}
\setlength{\parindent}{0cm}
\pagestyle{empty}

\begin{document}

Mr. Wolf \\ www.wolf-math.com

\section*{Linear Relations}

\subsection*{Goals}

\textbf{SWBAT} determine the differences between linear relationships and non-linear relationships.\\

\textbf{SWBAT} determine the slope of a line by using the slope formula and by using a graph.\\

\textbf{SWBAT} graph linear functions by using slope intercept form.\\

\textbf{SWBAT} create an equation given either 2 points, or a point and slope.\\

\textbf{SWBAT} convert from point-slope form to slope-intercept form.

\subsection*{Standards}

\textbf{Reasoning with Equations and Inequalities \hfill	A-REI}\\

 Solve equations and inequalities in one variable\\

3.	 Solve linear equations and inequalities in one variable, including
equations with coefficients represented by letters.\\

\subsection*{Connections}

\textbf{Now} we are learning how to create, graph, and solve linear equations.\\

\textbf{Later} we will learn what happens when two linear equations are multiplied, and it turns into a quadratic.

\pagebreak

\section*{Linear Functions}

\textbf{What is a Linear Function?} \\

\vspace{1in}

\textbf{How do we know something is a linear function?}\\

\begin{enumerate}
\item there is 1 $x$ without an exponent

\item this $x$ is not in the denominator of a fraction

\item there is only 1 variable \\

\end{enumerate}
\begin{center}

\begin{tabular}{ l | c }
  \textbf{Linear equations} & \textbf{Nonlinear equations}  \\ \hline
  $4x-5y=16$ & $2x+6y^2=-25$  \\
  $x=10$ & $y=\sqrt{x}+2$  \\
  $y=\frac{1}{2}x$ & $y=\frac{1}{x}$\\
\end{tabular}
\end{center}

\begin{Large}
\textbf{You Try}
\end{Large}

\textbf{State Whether each Function is a Linear Function} \textit{and why}

\begin{enumerate}
\item $f(x)=8-\frac{3}{4}x$\\

\vspace{1in}

\item $g(x)=\frac{2}{x}$\\

\vspace{1in}

\item $h(x)=3x-4$\\
\end{enumerate}

\pagebreak

\section*{Slope}

\textbf{What is Slope?} -- (Not the definition you see below)
\vspace{1in}

\textbf{Definition}\\
$$Slope=\frac{\text{change\, in\,}y}{\text{change\, in\,} x}$$
\indent What does "change in" mean?\\

\vspace{1in}

If we have 2 points we can find the slope between them\\

\begin{multicols}{2}
\begin{enumerate}
\item $(3,2),(8,12)$ \\

\item $(-1,4),(3,-8)$\\

\item $(-2,-5),(-7,10)$\\

\item $(4,4), (10,12)$\\
\end{enumerate}
\end{multicols}

\begin{Large}
\textbf{You Try}
\end{Large}

\begin{enumerate}
\item Determine the rate of change of the graph\\

%\includegraphics[scale=.5]{rateofchange.jpg}\\

\begin{tikzpicture}[scale=.3]

   \tkzInit[xmax=10,ymax=10,xmin=-2,ymin=-2]
   \tkzGrid
 %  \tkzAxeXY

\draw[line width=1.5pt,<->](0,10)--(0,-2);
\draw[line width=1.5pt,<->](-2,0)--(10,0);
\draw[red,line width=1pt,<->](-2,-1.3333)--(10,6.33333);

\end{tikzpicture}

\item $(8,-3),(6,-2)$\\

\item $(-2,-5),(8,-15)$\\
\end{enumerate}

\pagebreak

\section*{Types of Slope}

\textbf{There are $4$ types of slope.}

\begin{enumerate}
\begin{multicols}{2}
	\item Positive\\ \includegraphics[scale=.6]{positive.jpg}\\

	\item Negative\\ \includegraphics[scale=.6]{negative.jpg}\\

	\item Zero\\ \includegraphics[scale=.6]{0.jpg}\\

	\item undefined\\ \includegraphics[scale=.6]{undefined.jpg}\\
\end{multicols}
\end{enumerate}

We can see how this works out in the equation of a graph.\\

\begin{enumerate}
\item $y=3x+6$\\

\item $y=-\frac{4}{3}x+2$\\

\item $x=10$\\

\item $y=10$\\
\end{enumerate}

\pagebreak

\section*{Graphs of Linear Functions}

\textbf{$y=mx+b$} is called \textit{slope-intercept form}. This is the form that is most commonly used for graphing linear equations, although we'll see at least one other later on. \\

We can use a t-chart to graph the equation. $y=3x-7$\\

\begin{center}

\includegraphics[scale=.9]{graphchart.jpg}

\end{center}

The y-intercept is the $b$ value. We just take this and graph it on the vertical axis. The slope is the $m$ value. To graph we begin by finding the intercept, then graph the rest by using the slope.\\

Graph $y=-\frac{1}{2}x+1$

\begin{center}
\includegraphics[scale=1]{graph.jpg}
\end{center}

\pagebreak

\section*{Point-Slope Form}

How to find the equation of a line that passes through two different points.\\

This is a job for \textit{Point-Slope Form}!\\

\begin{Large}
$$y-y_1=m(x-x_1)$$\\
\end{Large}

All that is needed is 2 points, or 1 point and a slope. Either way the slope is going to be discovered. $y_1$, and $x_1$ are from the point $(x_1,y_1)$. \\

If I need a line to pass through the point $(3,5)$ and has slope of $4$, then I just need to plug my pieces into the form.\\

$x_1=3$\\
$y_1=5$\\
$m=4$\\

so my equation looks like...\\

$$y-5=4(x-3)$$

That's it!\\

\textbf{You Try:} Write an equation that goes through the indicated point, with the indicated slope, then graph.

\begin{multicols}{2}

$(-1,3)$ slope$=\frac{1}{2}$\\

\includegraphics[scale=.9]{graphing.jpg}

$(3,-4)$ slope$=\frac{4}{3}$\\

\includegraphics[scale=.9]{graphing.jpg}

\end{multicols}

\pagebreak

\subsection*{With 2 Points}

The process is exactly the same with 2 points, except that the slope needs to be calculated. \\

\textbf{Step 1:} Calculate the slope $\frac{y_1-y_2}{x_1-x_2}$\\

\textbf{Step 2:} Pick which point you like better.\\

\textbf{Step 3:} Use slope from step 1, your favorite point, then plug  in to point-slope form.\\

\hrulefill

Find the equation of the line passing through the points $(1,1)$, and $(-3,5)$\\

\textbf{Step 1:} $\frac{1- -3}{1-5}=\frac{2}{-4}=-\frac{1}{2}$\\

\textbf{Step 2:} I think $(1,1)$ is a fine looking point, so I'm going to use it!\\

\textbf{Step 3:} $(y-1)=\frac{1}{2}(x-1)$\\

\hrulefill

\textbf{You Try:} Write the equation of the line passing through the points, the graph.

\begin{enumerate}
\begin{multicols}{2}

\item $(2,3)$ and $(4,4)$\\
\includegraphics[scale=.7]{graphing.jpg}

\item $(-1,-5)$ and $(1,-3)$\\
\includegraphics[scale=.7]{graphing.jpg}

\item $(-12,-20)$ and $(-30,-40)$\\
\includegraphics[scale=.7]{graphing.jpg}

\item $(-14,-5)$ and $(-8,3)$\\
\includegraphics[scale=.7]{graphing.jpg}
\end{multicols}
\end{enumerate}

\pagebreak

\section*{Converting Forms}

\noindent $ax+by=c$ is called standard form.\\

\noindent Convert  $2y+3x=6$ to slope-intercept form. Do it by solving for $y$.\\

$2y=6-3x$\\

$y=\frac{6-3x}{2}$\\

$y=3-\frac{3}{2}x$\\

$y=-\frac{3}{2}x+3$\\

\noindent \textbf{You Try} Convert $4y+3x=12$ to slope-intercept form.\\

\vspace{23mm}

\noindent \textbf{Converting to slope-intercept form.} \\

\noindent Convert $y-7=\frac{3}{4}(x+20)$ to slope-intercept form\\

$y-7=\frac{3}{4}x+15$ \textbf{**distribute the $\frac{3}{4}$**}\\

$y=\frac{3}{4}x+22$\\

\begin{enumerate}
\item $(y-1)=\frac{1}{2}(x-4)$\\

\item $(y-3)=\frac{10}{9}(x-9)$\\

\item $(y+1)=4(x+1)$\\

\item $(y-8)=\frac{5}{9}(x-18)$\\

\end{enumerate}

\pagebreak

\section*{Review of Linear Equations}

\textbf{Identify the function as linear} or nah -- Circle the linear equations

\begin{enumerate}
\begin{multicols}{2}

\item $y=2x-1$ \\

\item $3x+6y=4$\\

\item $y=x^2$\\

\item $y=\frac{1}{x}$\\

\item \includegraphics[scale=.5]{linear.jpg}

\item $xy=13$\\

\item $\frac{x}{y}=10$\\

\item $\frac{y}{x}=10$\\

\item $3x-y=10$\\

\item \includegraphics[scale=.5]{nonlinear.jpg}

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Calculate the slope $\frac{y_1-y_2}{x_1-x_2}$}

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $(-2,7),(8,13)$\\

\item $(0,0),(0,10)$\\

\item $(-6,-2),(-3,-4)$\\

\item $(5,2),(9,2)$\\

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Graph the lines}

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $y=\frac{3}{2}x-4$\\

\includegraphics[scale=.7]{graphing.jpg}

\item $y=\frac{-1}{5}x+3$\\

\includegraphics[scale=.7]{graphing.jpg}

\end{multicols}
\end{enumerate}

\pagebreak

\textbf{Write an equation in Point-Slope Form} $y-y_1=m(x-x_1)$\\

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $(2,3) m=9$\\

\item $(-1,-6) m=\frac{4}{5}$\\

\item $(-3,-9), (2, 1)$\\

\item $(7,0),(9,2)$\\

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Convert from Point-Slope to Slope-Intercept Form}\\

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $y-3=\frac{1}{2}(x-6)$\\

\item $y+16=\frac{8}{3}(x-3)$\\

\item $y-10=\frac{3}{4}(x+20)$\\

\item $y+212=\frac{9}{5}(x-100)$\\

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Pre-assessment:}  This section for extra credit on quiz\\

Solve for $x$ and $y$. Use substitution, elimination, or graphing.\\

$$2x-3y=-2$$

$$4x+y=24$$\\

\pagebreak

\hfill NAME:\underline{\hspace{3in}}\\

\section*{Linear Equations Test}

\textbf{Identify the function as linear} or not -- Circle the linear equations

\begin{enumerate}
\begin{multicols}{2}

\item $xy=12$ \\

\item $\frac{x}{y}=11$\\

\item $3x-y=9$\\

\item \includegraphics[scale=.5]{nonlinear.jpg}

\item $y=2x-3$\\

\item $x+6y=5$\\

\item $y=x^3$\\

\item \includegraphics[scale=.5]{linear.jpg}

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Calculate the slope $\frac{y_1-y_2}{x_1-x_2}$}

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $(-3,5),(7,12)$\\

\item $(-1,-1),(-1,9)$\\

\item $(-5,-1),(-2,-3)$\\

\item $(6,3),(10,3)$\\

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Graph the lines}

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $y=\frac{3}{2}x-4$\\

\includegraphics[scale=.7]{graphing.jpg}

\item $y=\frac{-2}{5}x+5$\\

\includegraphics[scale=.7]{graphing.jpg}

\end{multicols}
\end{enumerate}

\pagebreak

\textbf{Write an equation in Point-Slope Form} $y-y_1=m(x-x_1)$\\

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $(7,3) m=3$\\

\item $(-2,-4) m=\frac{6}{11}$\\

\item $(-4,-9), (1, 1)$\\

\item $(-8,3),(-3,8)$\\

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Convert from Point-Slope to Slope-Intercept Form}\\

\begin{enumerate}[resume]
\begin{multicols}{2}

\item $y-5=\frac{1}{2}(x-5)$\\

\item $y+10=\frac{9}{5}(x-5)$\\

\item $y-12=\frac{4}{3}(x+15)$\\

\item $y-20=\frac{5}{8}(x-32)$\\

\end{multicols}
\end{enumerate}

\hrulefill

\textbf{Extra Credit:} Freezing is at $32^{\circ}F$, which is also $0^{\circ}C$. Boiling is at $212^{\circ}F$ or $100^{\circ}C$. Using this information, write a linear equation that converts between the two units of temperature.

\end{document}