Throughout this paper, bold capital letters denote matrices (e.g., $\mathbf{X}$) and bold lower-case letters denote column vectors (e.g., $\mathbf{x}$). $\norm{\mathbf{X}}_2 =
(\mathbf{X}^T\mathbf{X})^{1/2}$ and $\norm{\mathbf{X}}_1 = \sum_i |\mathbf{x}_i|$
denote the $l_2$ and $l_1$ norms, respectively, with $T$ indicating the
matrix transpose. We also denote $\norm{\mathbf{X}}_F =
(\text{Tr}(\mathbf{X}^T\mathbf{X}))^{1/2}$ as the Frobenius norm, where
Tr indicate the trace of a matrix, i.e.,  $\text{Tr}(X)\equiv \sum_{i=1}^n \mathbf{x}_{ii}$.

This section contains the theory to implement a model such as the model presented in section \ref{sec:fab}. We first present a brief overview of the mathematical background behind Optimization, Machine Learning and Artificial Neural Networks (ANN) in sections \ref{sec:opt},\ref{sec:ml} and \ref{sec:ann} respectively. This theory leads up to all the necessary details involved with Sparse Coding, which is presented in \ref{sec:sc} with two examples in computer vision and speech recognition to give the reader a better understanding of the concept. We later present an underlying concept called Non-Negative Matrix-Factorization in section \ref{sec:nnsc} and the proof behind it. This technique is commonly used when dealing with tasks involving only positive values.

%Lastly we present techniques for sparse approximations in section \ref{sec:regularization}.

\label{sec:theoretical}

\subsection{Optimization}
\label{sec:opt}
A problem that consists of finding the best solution from a set of feasible solutions. In mathematics and computer science, we refer this as a optimization problem. The standard form of a optimization problem is defined as

\begin{align*}
& \underbrace{\text{min}}_{x} \qquad f(x) \\
& \text{subject to} \quad g_i(x) \leq 0, \ i = 0,\dots,n
\end{align*}
where we define $f(x)$ as the objective function to be minimized with respect to $x$ and $g_i(x)$ as the $i$:th constraint. By convention, the standard form is to minimize the objective function, we can maximize by negating the expression above \cite{convex}.

\subsubsection{Cost function}

An objective function that is of standard form is often referred to as a cost or loss function that maps events or values of variables to some value that represents the "cost" involving that particular event. In classification, the cost is usually portrayed as "penalty" involving an incorrect classification.

Supervised learning tasks, described in \ref{sec:ml}, such as regression or classification for parameter estimation can be formulated as a loss function over a training set. The goal is to find the models that represent the input well; and the loss function quantifies the amount of deviation of the prediction from the true values \cite{convex}.

\subsection{Machine Learning}
\label{sec:ml}
Machine learning can be considered a sub-field of computer science and statistics which can be described as the study of algorithms that can learn from data. Mostly Machine learning is employed on computational tasks where designing and programming explicit, rule-based algorithms is infeasible. The common applications include spam filtering, optical character recognition (OCR), search engines and computer vision. It also has ties to both artificial intelligence and optimization.

Machine learning tasks are typically classified into three broad categories, depending on the nature of the learning "signal" or "feedback" available to a learning system. One category is Supervised learning; where the computer is presented with example inputs and their desired outputs, given by the user, and the goal is to learn a general rule that maps inputs to outputs. The second category is Unsupervised learning, where no labels are given to the learning algorithm. This way the algorithm has to find its own structure in the input, this algorithm can be run to do stand-alone unsupervised learning (discover hidden patterns) or a means towards another type of end. Sparse Coding, which forms the basis of this thesis is a neural network model for unsupervised learning. Lastly we have reinforcement learning where a computer program interacts with a dynamic environment in which it must perform a certain goal (such as driving a vehicle), without a teacher explicitly telling it whether it has come close to its goal or not. Another example is learning to play a game by playing against an opponent \cite{ainorvig}.

A core objective of a learner is to generalize from its experience. Generalization in this context is the ability of a learning machine to perform accurately on new, unseen examples/tasks after having experienced a learning data set \cite{bishop, bengio}. The training examples usually come from some unknown probability distribution and the learner has to build a general model so as to produce sufficiently accurate predictions from incoming new examples.

In machine learning one can simplify the inputs by mapping them into a lower-dimensional space through dimensionality reduction, described in section \ref{sec:dim}.

\subsubsection{Dimensionalty Reduction}
\label{sec:dim}

Representing an object as a vector of $n$ elements, we say that the vector is in $n$-dimensional space. Dimensionalty reduction refers to a process of representing the object of $n$-dimensional vector to an $m$-dimensional vector, where $m < n$. By refining the data in this way, we may lose information that might be valuable but we can represent it using less dimensions and in some cases we can even make a better prediction or analysis using this subspace. The common linear dimensionality reduction is called Principal Component Analysis (PCA), which find "internal axes" of a dataset, called components and sort by importance. It performs a linear mapping of the data to a lower-dimensional space in such a way that the variance of the data in the low-dimensional representation is maximized. The original space is not retained, i.e. we have lost some information but keep the most important variance to the space spanned by a few eigenvectors. The first $m$ components are then used as the new basis. Each of these components may be thought of as a high-level feature, describing data vectors better than original axes \cite{dr}.

Dimensionality reduction can be divided into feature selection and feature extraction. Feature selection approaches try to find a subset of the original variables, while feature extraction transforms the data in high-dimensional space to that of a fewer dimensional space. The data transformation may be linear, as in PCA, but many nonlinear dimensionality reduction techniques also exist \cite{samet}.

A different approach to nonlinear dimensionality reduction is through the use of autoencoders, a special kind of feed-forward neural networks with a bottle-neck hidden layer, which is presented in-depth in section \ref{sec:autoencoders}.

\subsubsection{Deep Learning}
\label{sec:deeplearning}

A branch of machine learning based on algorithms that try to model high-level abstractions in data by using complex structures or multiple non-linear transformations is referred to deep learning \cite{deep, deep2}. Deep learning focuses on learning representations of data, where it has maybe come to replacing handcrafted features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction \cite{deep3}.

Some representation are based on interpreting information processsing in a nervous system inspired by advances in neuroscience, such as neural coding which attempts to define a relationship between the stimulus and the neuronal responses and the relationship among the electrical activity of the neurons in the brain \cite{nervous}, see section \ref{sec:scnn} for more information.

\subsection{Artificial Neural Networks}
\label{sec:ann}

In machine learning, a family of statistical learning algorithms called artificial neural networks (ANN) that were inspired by the work of McCulloch, Warren; Walter Pitts as early as 1943 to reflect a central nervous systems of animals \cite{1943}. Generally ANN is a network with connected nodes and edges that form a artificial "biological neural network" which compute values from inputs provided by the edges connected to the nodes, even though the relation between the model and the brain is debated to what degree it really represents the brain \cite{brain}.

ANN models are essentially mathematical functions defining a function
\begin{equation}
f : X \rightarrow Y
\end{equation}

but sometimes models are also associated with a particular learning algorithm, like the perceptron presented in the section \ref{sec:perceptron} below. The learning output is obtained by connection weights, parameters and specific architecture by the learning algorithm. Two frameworks where ANN have made a great contribution is computer vision and speech recognition tasks, where rule-based programming have been unsuccessful at detecting patterns \cite{perceptron}. The connection between neural networks and Sparse Coding is explained in section \ref{sec:scnn}.

There are two main ways to "feed" the network with information. One being that of a feedforward neural network, the term “feedforward” indicates that the network has links that extend in only one direction. Except during training, there are no backward links in a feedforward network; all links proceed from input nodes toward output nodes. Eventually, despite the apprehensions of earlier workers, a powerful algorithm for apportioning error responsibility through a multi-layer network was formulated in the form of the backpropagation algorithm \cite{backpro}. The effects of error in the output nodes are propagated backward through the network after each training case. The essential idea of backpropagation is to combine a non-linear multi-layer perceptron-like system capable of making decisions with the objective error function of the Delta Rule \cite{backpro}.
% checked

\pagebreak
\subsubsection{Perceptron}
\label{sec:perceptron}
The basic concept of a single layer perceptron was introduced by Rosenblatt in 1958 \cite{perceptron}. It computes a single output by forming a linear combination of real-valued inputs and weights to possibly giving it through some non-linear function. This can be written as

\begin{equation}
 y = \phi( \sum\limits_{i=1}^n a_i x_i + b ) = \phi( \mathbf{a}^T \mathbf{x}+ b )
\end{equation}

where  $\mathbf{a}$ denotes the vector of weights,  $ \mathbf{x}$ is the vector of inputs, $ b$ is the bias and $ \phi$ is the activation function. Usually in multilayer networks, the activation function is often chosen to be the logistic sigmoid $ 1 / (1 + e^{-x})$ or the hyperbolic tangent $ \tanh(x)$. They are convenient as they are close to linear near the origin, while they converge to a value when leaving the origin. This allows perceptron networks to model well both strongly and mildly nonlinear mappings \cite{nonlinear}. Perceptrons were a popular machine learning solution in the 1980s, but since the 1990s faced strong competition from the much simpler support vector machines \cite{svm}. More recently, there has been some renewed interest in backpropagation networks, such as perceptrons due to the successes of deep learning, see section \ref{sec:deeplearning} for more detail.

A typical perceptron layer network consists of source nodes forming the first layer. Following with one or more hidden layers, and an output layer of nodes, in the case where we have three or more layers it is usually called a multilayer perceptron (MLP). The input signal propagates through the network layer-by-layer. The signal-flow of such a network with one hidden layer can be seen in figure \ref{fig:autoencoder} in section \ref{sec:autoencoders}.

The computations performed by such a feedforward network with a single hidden layer with nonlinear activation functions and a linear output layer can be written mathematically as

\begin{equation}
\mathbf{y}= \mathbf{f}(\mathbf{x}) = \mathbf{B}\boldsymbol{\phi}( \mathbf{A}\mathbf{x}+ \mathbf{a} ) + \mathbf{b}
\label{eq:nonlinear}
\end{equation}

where  $\mathbf{x}$ is a vector of inputs and  $\mathbf{y}$ a vector of outputs.  $ \mathbf{A}$ is the matrix of weights of the first layer,  $ \mathbf{a}$ is the bias vector of the first layer.  $ \mathbf{B}$ and  $ \mathbf{b}$ are, respectively, the weight matrix and the bias vector of the second layer.

MLP networks are typically used in supervised learning problems. Here the training set of input-output is pairs and the network must learn to model the dependency between them. The training here means adapting all the weights and biases ( $ \mathbf{A},
\mathbf{B}, \mathbf{a}$ and  $ \mathbf{b}$ in equation \ref{eq:nonlinear} to their optimal values for the given pairs  $ (\mathbf{x}(t), \mathbf{y}(t))$. The criterion to be optimised is typically the squared reconstruction error.
%to read more on regression see section \ref{sec:regularization}

\begin{equation}
\label{eq:error}
\sum_t \vert\vert\mathbf{f}(\mathbf{x}(t)) - \mathbf{y}(t)\vert\vert^2.
\end{equation}

By setting the same values for the inputs as well as the outputs of the network, MLP networks can be used for unsupervised learning. The values of the hidden neurons extract the sources, this approach however is rather computationally intensive. \cite{hidden}

\subsubsection{Autoencoder}
\label{sec:autoencoders}

Autoencoder is a simple 3-layer neural network where output units (Layer $L_3$) are directly connected back to input units (Layer $L_1$). E.g. in a network presented in the figure below:

\begin{figure}[H]
	\centering
	\includegraphics[scale=0.3]{./figures/autoencoder}
	\caption[Caption for LOF]{As a concrete example, suppose the inputs $x$ are the pixel intensity values from a $10 \times 10$ image (100 pixels) so  $n=100$, and there are $s_2=50$ hidden units in layer $L_2$.	\protect\footnotemark}
	\label{fig:autoencoder}
\end{figure}
\footnotetext{Stanford, \texttt{http://ufldl.stanford.edu/wiki/index.php/Autoencoders\_and\_Sparsity},7 April 2013, (accesed 10 July 2015)}
Typically in an autoencoder, the number of hidden units are much less than number of input and output. As a result, it first compresses (encodes) the input vector to "fit" in a smaller representation, and then tries to reconstruct (decode) it back. Here is where Sparse Coding can be said to be an extension of autoencoders with the constraint that the hidden layer must mostly be unused nodes (sparse), for more information on their similarities see the end of section \ref{sec:sc}. Autoencoders simple form can be written as
%\[ D(d(e(x;\theta^r); \theta^d), x) \]

\begin{equation}
\label{eq:auto}
\norm{\mathbf{A}\sigma(\mathbf{A}^T \mathbf{x}) - \mathbf{x}}^2
\end{equation}

where $\sigma$ is a nonlinear function such as the logistic sigmoid, and $\mathbf{A}$ is the activation density of the nodes. Once a deep network is pretrained, input vectors are transformed to a better representation. \cite{autoencoder2}

\subsubsection{Sparse Coding and the connection to Neural Networks}
\label{sec:scnn}

Information retrieved is presented in the brain by the pattern of activations of the nerual connections formed, which we say form a neural code. This defines the pattern at which the neural activity corresponds to each presented information.
% - rewrite
One property of the neural code is the fraction at which the neurons are active at any time. If a set of $N$ neurons, which can be active in the region $\in [0,1/2]$ corresponding to low activity to strong activity, the expected value of this fraction is the density of the code. If the average fraction is above 1/2 we can replace each active neuron with an inactive one causing the fraction activity to get below 1/2 without loss of information and vice versa. Sparse coding is a neural code, which is of a relatively small set of neurons but with strong activity. For each set of information, a different subset is triggered of all available neurons.\cite{scprimate}

\subsubsection{Local Codes}
Low activity of neurons are local codes, where an item is represented by a small set of neurons or a separate neuron, this way one can ensure that there is no overlap between the representations of two items. To understand this we make an analogy that envolves the characters on a computer keyboard, where each key encodes a single character. This scheme has the advantage that it is simple and is also easy to decode, due to local codes only representing a finite number of combinations. More generalization is essential and a widely observed behavior. \cite{mclaren}

\subsubsection{Dense Distributed Codes}

% - rewrite
The opposite of local codes are dense codes, where the average activity ratio is $\geq 0.5$, the item is represented by activities of all the neurons, which implies a representational capacity of $2^N$. Given the billions of neurons in a human brain, $2^N$, as the number of neurons the representational capacity of a dense code in the brain is immense, therefore its greatest feature is dealing with redundancy. Dense codes limit the number of memories that can be stored in an associative memory by simple learning rules. On the contrast, dense codes may facilitate good generalization performance and high redundancy.

\subsubsection{Sparse Codes}
% - rewrite

These neural codes come to a favorable compromise between dense and local codes by having a small average activity ratio, called sparse codes \cite{scprimate}. We can redeem the capacity of local codes by a modest fraction of active units per pattern, thus interference by items represented simultaneously will be less likely as capacity grows exponentially with average activity ratio. It is more likely that a single layer network with a sparse representation as input can learn to generate a target output \cite{willshaw}. Due to linear discriminant functions being able to map higher proportions, see Perceptrons for linear separability in section \ref{sec:perceptron}. Single layer networks for learning is therefore simpler, faster and substantially more plausible as a way of a biological implementation in the brain, as the redundancy for fault tolerance can be chosen by controlling the sparseness.

For learning various tasks a neural code can therefore contain codewords of varying sparseness. This implies that we want to maximize sparseness while having a high representational capacity. One plausible way would be to assign sparse codes for items of high probability while having distributed codes for lower probability items.
A code with a given average sparseness can contain codewords of varying sparseness. If the goal is to maximize sparseness while keeping representational capacity high, a sensible strategy is to assign sparse codewords to high probability items and more distributed codewords to lower probability items. However, if we would only store identities of active units, the code would have short average discription length \cite{cover}. Some perceptual learning could be explained by prediction of the sparseness of the encoded items with high probability.

Below is an example where the input signal is an image, the basis vectors represent the sparse coding method. In the example below we present an explicit visualization of the sparse coding.

\begin{ex}{}
\label{ex:scimage}
An image reconstruction usage of sparse coding. The basis vectors are visualized in Figure \ref{fig:basex}, which have been trained from natural images. The basis vectors are then used to represent different parts of a picture using the activation matrix.

\begin{figure}[H]
\centering
        \includegraphics[scale=0.20]{./figures/basisexample.jpg}
        \caption{Sparse basis functions learned from images.}
        \label{fig:basex}
\end{figure}

These basis vectors are used with activations to represent an image. The activation matrix is best represented in the figure below, where a part of an image uses the activated vectors (non-black) to represent the corresponding image.

\begin{figure}[H]
\centering
\includegraphics[scale=0.30]{./figures/Sparsejen_black_v2.jpg}
\caption{An image, encoded with the basis functions from figure \ref{fig:basex} and reconstructed in the right plot using certain subsets of activations and basis functions for each patch. The red square represents a patch where we have encoded the activations for some basis functions, shown as the middle plot, to reconstruct the patch of the decoded image to the right. Notice, among the entire set of basis functions, only a fair amount is used and the rest is black indicated that they are not being used, i.e. we have a "sparse" representation of the image. \protect\footnotemark
}
\label{fig:recex}
\end{figure}

\end{ex}
\footnotetext{Peter Foldiak and Dominik Endres ,Scholarpedia, \texttt{http://www.scholarpedia.org/article/Sparse\_coding}, 2008, (accessed 10 June 2015)}
\newpage

\subsubsection{Sparse Coding}

\label{sec:sc}
Sparse Coding is similar to Principal Component Analysis (PCA) in that we want to find a small number of basis functions to represent an input signal as a linear combination presented in equation \ref{eq:lincomb} but with a constraint that the learned basis functions need to be sparse and of a higher dimension than the input data. Here we present the general theory behind Sparse Coding and an example used for signal processing in example \ref{ex:sc} \cite{olshausen}.

\begin{equation}
\mathbf{x} \approx \mathbf{BA}
\label{eq:lincomb}
\end{equation}

Consider a linear system of equations $\mathbf{x} = \sum_{i=1}^ka_i \phi_i$, the vector coefficients $a_i$ are no longer uniquely determined by the input vector $\mathbf{x}$, where $\sum_{i=1}^ka_i = \mathbf{B}$ is an
underdetermined $m\times p$ matrix $(m \ll p)$. $\mathbf{B}$, is called the dictionary or sometimes design matrix. The problem is to estimate the signal $\alpha$, subject to the
constraint that it is sparse. The underlying motivation for sparse
decomposition problems is that even though the observed values are in
high-dimensional ($m$) space, the actual signal is organized in some
lower-dimensional subspace $(k \ll m)$. This implies that $\mathbf{x}$ can be decomposed as a linear combination of only a few $m \times 1$ vectors in $\mathbf{B}$, called atoms \cite{sc}.

Regular PCA allows us to learn a complete set of basis vector while the Sparse Coding wishes to learn an \textbf{over-complete}
basis to recognize patterns and structures inherent in the input data.
Although we now can recognize patterns in the data, we have coefficients of the columns
$\alpha_i$ that are no longer uniquely determined by the input vector
$\mathbf{x} \in \mathbb{R}^2$. This is why we introduce a criterion called
\textbf{sparsity} to resolve the degeneracy introduced by
over-completeness, where sparsity is defined as having few non-zero
components. The definition of the sparse coding cost function on a set of $m$
input vectors is presented in equation \ref{eq:sccost}. In artificial
neural networks, the cost function represents a function to return a number representing how well the neural network performed to map training examples to correct output \cite{olshausen}.

%\begin{equation}
%\mathbf{x} = \mathbf{BA}
%\label{eq:lincomb}
%\end{equation}
%
%Consider a linear system of equations $\mathbf{x} = \mathbf{BA}$, where $\mathbf{B}$ is an
%under-determined $m\times p$ matrix $(m \ll p)$. $\mathbf{B}$, is called the dictionary or the design matrix. The problem is to estimate the signal $\mathbf{A}$, subject to the
%constraint that it is sparse. The underlying motivation for sparse
%decomposition problems is that even though the observed values are in
%high-dimensional ($m$) space, the actual signal is organized in some
%lower-dimensional subspace $(k \ll m)$. This implies that $\mathbf{x}$ can be decomposed as a linear combination of only a few $m \times 1$ vectors in $\mathbf{B}$, called atoms.
%
%Regular PCA allow us to learn a complete set of basis vector while the Sparse Coding wish to learn a \textbf{over-complete}
%basis to recognize patterns and structures inherent in the input data.
%Although we now can recognize patterns in the data, we have coefficients of the columns
%$\mathbf{B}^(j)$ that are no longer uniquely determined by the input vector
%$\mathbf{x} \in \mathbb{R}^2$. This is why we introduce a criterion called
%\textbf{sparsity} to resolve the degeneracy introduced by
%over-completeness, where sparsity is defined as having few non-zero
%components. The definition of the sparse coding cost function on a set of $m$
%input vectors is presented in equation \ref{eq:sccost}. In artificial
%neural networks, the cost function represents a function to return a number representing how well the neural network performed to map training examples to correct output.

\begin{equation}
\min_{a_i^{(j)},\phi_i}\sum_{j=1}^m
\ \underbrace{\norm{\mathbf{x}^{(j)}-\sum_{i=1}^ka_i^{(j)}\phi_i}^2}_{\text{reconstruction term}}+\underbrace{\lambda \sum_{i=1}^k
S(a_i)^{(j)}}_{\text{sparsity penalty}}
\label{eq:sccost}
\end{equation}

where $S(a_i^{(j)})$ is a sparsity cost function which penalize $a_i$ for being far from zero.
%and $\norm{\mathbf{x}^{(j)}-\sum_{i=1}^ka_i^{(j)}\phi_i}^2 = \sum_{ij} [\mathbf{x}_{ij}-(a \phi)_{ij} ]^2$.
The first term in equation \ref{eq:sccost} is a reconstruction term that forces the algorithm to provide a good representation of $\mathbf{x}$ and the second as a sparsity penalty which force the representation to be sparse, while $\lambda$ is a scale to determine the relative importance between the two contributions. Note that if we are given $S(a_i^{(j)})$, estimation of $\phi_i$ is easy via least squares. In the beginning, we do not have $S(a_i^{(j)})$ however. Yet, many algorithms exist that can solve the objective above with respect to $S(a_i^{(j)})$.

%Actually, this is how we do inference: we need to solve an optimisation problem if we want to know the h belonging to an unseen $\mathbf{x}$.
% timo banned it

The most direct approach to determine sparsity is through the "$L_0$" norm $S(a_i) = \mathbf{1}(|a_i| > 0 )$, which is non-differentiable and difficult to optimize. The more common choices for sparsity cost penalty $S(a_i)$ are the $L_1$, $S(a_i) = |a_i|_1$ and the log penalty $S(a_i)=\log(1+a^2_i)$. To prevent empirical scaling of $a_i$ and $\phi_i$ to make the sparsity penalty arbitrarily small, we constrain $\norm{\phi}^2$ to be less than some constant $C$. Including the constraint demand we get the full sparse coding cost function \cite{olshausen}

\begin{align}
\begin{split}
\min_{a_i^{(j)},\phi_i} \quad & \sum_{j=1}^m
\norm{\mathbf{x}^{(j)}-\sum_{i=1}^ka_i^{(j)}\phi_i}^2+\lambda \sum_{i=1}^kS(a_i)^{(j)}
\\
\text{subject to} & \quad \norm{\phi_i}^2 \leq C, \forall \ i = 1,\dots,k
\end{split}
\end{align}

Below we present another example of sparse coding but used in a context of signal processing of a one dimensional signal.
\begin{ex}{}
\label{ex:sc}
Say, we have an infinite $1-D$ time-series signal. We can represent this signal in the Fourier domain, where we get a few coefficients representing the whole signal in a different domain.
\begin{equation*}
\mathbf{x} = \sum_{i=1}^k a_i \phi_i \approx \mathbf{x}'
\end{equation*}

We want to find these few coefficients ($a_i$,the basis) of an input signal in the alternative domain and reconstruct your signal with these few coefficients. Once we have found the coefficients, we determine how close is the reconstructed signal to our original input signal by the error. That is to minimize our representation: $|\mathbf{x}- \mathbf{x}'|$.

The least number of basis functions of the input signal that minimize the above error, is the best basis representation of our input signal. We then could use the $L_2$ norm for the error, which is what we are most familiar with and it computes the Euclidean, square difference, between basis functions. Basically, $L_0$ norm looks like a Dirac Delta Function, $L_1$ norm looks like a diamond and $L_2$ norm looks like a circle and are the other types of norms which can be used in this context.

\end{ex}

To end this section we would like to review the difference between Sparse Coding, Autoencoders and Sparse-PCA, as it is somewhat missleading at times.

\begin{itemize}
	\item{Autoencoders do not encourage sparsity in their general form.}
	\item{An autoencoder uses a model for finding the codes, while sparse coding does so by means of optimisation.}
\end{itemize}

Note that Sparse Coding, looks almost the same as Autoencoder as in equation \ref{eq:auto} in section \ref{sec:autoencoders} Autoencoders, once we set $\mathbf{B}=\sigma (\mathbf{A}^T\mathbf{x})$. For natural image data, regularized autoencoders and sparse coding tend to yield very similar $\mathbf{B}$. However, auto encoders are much more efficient and are easily generalized to much more complicated models. E.g. the decoder can be highly nonlinear, e.g. a deep neural network. Therefore, Sparse coding can be seen as a modification of the sparse autoencoder method in which we try to learn the set of features for some data "directly".

In Sparse-PCA one also wants to represent a collection of vectors as a linear combination of basis vectors (a.k.a. principal components). Here the focus, as in traditional PCA, is on choosing a small $n \ll M$ number of basis vectors that together "explain as much variance" as possible, i.e. represent the original data as well as possible. And the sparsity is enforced not on the mapping bases $\rightarrow$data, but on the mapping data$\rightarrow$bases, because the idea is to have PCs that are linear combinations of only small subsets of original features/vectors (to ease the interpretation), as explained in Zou, Hastie, and Tibshirani, 2006 \cite{zou}.

\subsubsection{Non-Negative Sparse Coding}

\label{sec:nnsc}
In standard Sparse Coding, described above, the data is described as a combination of elementary features involving both additive and subtractive interactions. The fact that features can ‘cancel each other out’ using subtraction is contrary to the intuitive notion of combining parts to form a whole \cite{hoyer}. Arguments for non-negative representations come from biological modeling, where such constraints are related to the non-negativity of neural firing rates. These non-negative representations assume that the input data $\mathbf{X}$, the basis $\mathbf{B}$, and the hidden components $\mathbf{A}$ are all non-negative. Since energy consumption is an inherently non-negative quantity, this representation is beneficial is reasonable for modeling energy usage. \\
Non-negative matrix factorization (NMF) can be performed by the minimization of the following objective function:

\begin{equation}
\label{eq:nmf}
C ( \mathbf{A,B} ) = \frac{1}{2} \norm{ \mathbf{X} - \mathbf{BA} }^2
\end{equation}

Here Hoyer \cite{hoyer} take $ \norm{ \mathbf{X} - \mathbf{BA} }^2 = \sum_{ij} [ \mathbf{X}_{ij} - \mathbf{BA}_{ij} ]^2$. Denoting a general matrix norm by

\begin{equation}
\| A \|_{p,q}  =  \left[\sum_{j=1}^n \left( \sum_{i=1}^m |a_{ij}|^p \right)^{q/p}\right]^{1/q}
\end{equation}

Using $p=2,q = 2$ we get the Frobenius norm and we conclude that equation \ref{eq:nmf} is using the Frobenius norm, this is an insurance for later use in the Discriminative Disaggregation via Sparse Coding model \ref{alg:ddsc}.
\[ \|A\|_F^2 =\left(\sqrt{\sum_{i=1}^m\sum_{j=1}^n |a_{ij}|^2}\right)^2 = \sum_{i,j}[a_{ij}]^2\]

\textbf{Definition 1.} Non-negative sparse coding (NNSC) of a non-negative data matrix $\mathbf{X}$ (i.e. $\forall \ i,j \ : \ X_{ij} \geq 0$) is given by the minimization of
\begin{equation}
\label{eq:nnsc}
C ( \mathbf{A,B} ) = \frac{1}{2} \norm{ \mathbf{X} - \mathbf{BA} }^2 + \lambda \sum_{ij} \mathbf{A}_{ij}
\end{equation}
subject under the constraints $\forall \ i,j \ : \ B_{ij} \geq 0, \ A_{ij} \geq 0$ and $\forall \ i \ : \ \| \mathbf{B}_i \| = 1$, where $\mathbf{B}_i$ denotes the i:th column of $\mathbf{B}$. It is also assumed that the constant $\lambda \geq 0$.
~\\

\pagebreak
\textbf{Theorem 1.} The equation \ref{eq:nmf} is non-increasing under the update rule:
\begin{equation}
\label{eq:update}
\mathbf{A}^{t+1}=\mathbf{A}^t.*(\mathbf{B}^T\mathbf{X})./(\mathbf{B}^T\mathbf{B}\mathbf{A}^t+\lambda)
\end{equation}
where $.*$ and $./$ denote element-wise multiplication and division (respectively), and the addition of the scalar $\lambda$ is done to every element of the matrix  $\mathbf{B}^T\mathbf{B}\mathbf{A}^t$.
~\\

The proof is seen below in \ref{proof:noninc}. As each element of $\mathbf{A}$ is updated by simply multiplying with some non-negative factor, it is guaranteed that the elements of $\mathbf{A}$ stay non-negative under this update rule. As long as the initial values of $\mathbf{A}$ are all chosen strictly positive, iteration of this update rule is in practice guaranteed to reach the global minimum to any required precision.

\begin{proof2}{Proof of Theorem 1}
To prove Theorem 1, first note that the equation \ref{eq:nnsc} in definition 1 is separable in the columns of $\mathbf{A}$ so that each column can be optimized without considering the others. We may thus consider the problem for the case of a single column, denoted s. The corresponding column of $\mathbf{X}$ is denoted $x$, giving the objective
\begin{equation}
F(\mathbf{a}) = \frac{1}{2}\norm{\mathbf{X}-\mathbf{B}\mathbf{a}}^2 + \lambda \sum_i a_i
\end{equation}
We need an iliary function $G(\mathbf{a},\mathbf{a}^t)$ with the properties that $G(\mathbf{a},\mathbf{a}) = F(\mathbf{a})$ and $G(\mathbf{a},\mathbf{a}^t) \geq F(\mathbf{a})$. We will then show that the multiplicative update rule corresponds to setting, at each iteration, the new state vector to the values that minimize the auxiliary function:
\begin{equation}
\mathbf{a}^{t+1} = \arg\!\min_{a} G(\mathbf{a},\mathbf{a}^t).
\end{equation}
This is guaranteed not to increase the objective function $F$, as
\begin{equation}
\label{eq:proof}
F(\mathbf{a}^{t+1}) \leq G(\mathbf{a}^{t+1},\mathbf{a}^t) \leq G(\mathbf{a}^t,\mathbf{a}^t) = F(\mathbf{a}^t).
\end{equation}
We define the function $G$ as
\begin{equation}
G(\mathbf{a},\mathbf{a}^t) = F(\mathbf{a}^t) + (\mathbf{a}-\mathbf{a}^t)^T \nabla F(\mathbf{a}^t) + \frac{1}{2} (\mathbf{a}-\mathbf{a}^t)^T\mathbf{K}(\mathbf{a}^t)(\mathbf{a}-\mathbf{a}^t)
\end{equation}
where the diagonal matrix $\mathbf{K}(\mathbf{a}^t)$ is defined by elementwise division as
\begin{equation}
K_{ij}(\mathbf{a}^t) = \delta_{ij}\frac{(\mathbf{B}^T\mathbf{B}\mathbf{a}^t)_i+\lambda}{\mathbf{a}^t_i},
\end{equation}
where $i$ denotes the $i$:th column. Inserting $\mathbf{a}$ in function $G$ we get the result from equation \ref{eq:proof}, $G(\mathbf{a},\mathbf{a}) = F(\mathbf{a})$. Writing out
\begin{equation}
F(\mathbf{a}) = F(\mathbf{a}^t) + (\mathbf{a}-\mathbf{a}^t)^T \nabla F(\mathbf{a}^t)+\frac{1}{2}(\mathbf{a}-\mathbf{a}^t)^T(\mathbf{B}^T\mathbf{B})(\mathbf{a}-\mathbf{a}^t),
\end{equation}
we see that the second property, $G(\mathbf{a},\mathbf{a}') \geq F(\mathbf{a}),$ is satisfied if
\begin{equation}
0 \leq (\mathbf{a}-\mathbf{a}^t)^T[\mathbf{K}(\mathbf{a}^t) - \mathbf{B}^T\mathbf{B}](\mathbf{a}-\mathbf{a}^t).
\end{equation}
Hoyer proved this positive semidefiniteness for the case of $\lambda \geq 0$ \cite{hoyer}. He concludes that as a non-negative diagonal matrix is positive semidefinite, and the sum of two positive semidefinite matrices is also positive semidefinite, the proof for $\lambda = 0$, in his paper also holds for $\lambda \geq 0$.
It remains to be shown that the update rule in equation \ref{eq:update} selects the minimum of $G$. This minimum is easily found by taking the gradient and equating it to zero:
\begin{equation}
\nabla_\mathbf{a} G(\mathbf{a},\mathbf{a}) = \mathbf{B}^T(\mathbf{B}\mathbf{a}^t-\mathbf{x}) + \lambda\mathbf{c} + \mathbf{K}(\mathbf{s}^t)(\mathbf{a}-\mathbf{a}^t)=0,
\end{equation}
where $\mathbf{c}$ is a vector with all ones. Solving for $\mathbf{a}$, this gives
\begin{align}
\mathbf{a} & = \mathbf{a}^t - \mathbf{K}^{-1}(\mathbf{a}^t)(\mathbf{B}^t\mathbf{B}\mathbf{a}^t-\mathbf{B}^T\mathbf{x}+\lambda \mathbf{c}) \\
& = \mathbf{a}^t-(\mathbf{a}^t./(\mathbf{B}^T\mathbf{B}\mathbf{a}^t+\lambda\mathbf{c})).* (\mathbf{B}^T\mathbf{B}\mathbf{a}^t - \mathbf{B}^T\mathbf{x}+\lambda \mathbf{c}) \\
& = \mathbf{a}^t.\times (\mathbf{B}^T\mathbf{x}./(\mathbf{B}^T\mathbf{B}\mathbf{a}^t+\lambda\mathbf{c}))
\end{align}
which is the desired update rule \ref{eq:update}.
\label{proof:noninc}
\end{proof2}

%\subsubsection{Tranformation Algorithms and variations for sparse approximation}
%%
%\label{sec:regularization}
%\subsubsection{Ridge Regression}
%
%A famous method of regularization called Tikhonov regularization, in statistics this method is known as ridge regression, an algorithm for non-linear least-squares problems. When x is non-existent, we call it an ill-posed problem such as below
%
%\begin{equation}
%A\mathbf{x}=\mathbf{b},
%\end{equation}
%
%a standard approach to the problem is ordinary least squares, which often leads to an underdetermined system of equations. Real-world problems often operate as a low-pass filter, meaning that the direction of the mapping where $A$ maps $\mathbf{x}$ to $\mathbf{b}$. By the inverse-mapping, we operate as a high-pass filter, which is often the case in mathematical formulations, it has an undesirable tendency of amplifying noise. \cite{ill-posed} Ordinary least squares (OLS) nullifies every element of the reconstructed elements of $\mathbf{x}$, that is in the null-space of $A$. The ridge regression uses the $l_2$-norm as a means of regression to the residuals and can be mathematically written as the following formulation
%\begin{equation}
%\label{eq:ridge}
%\beta^{ridge} = \arg \! \operatorname*{\min}_\beta \sum_{i=1}^n (\mathbf{y}_i - ( \beta_0 + \beta^T \mathbf{x}_i))^2 + \lambda \norm{\beta}^2_2
%\end{equation}
%There are other techniques which are based on the assumption that the $l_2$-norm does not capture the features to the respective dimensions and therefore "LASSO" has been used as a means of using the $l_1$-norm instead.
%
%\subsubsection{LASSO}
%
%"LASSO" In the context of regression and statistics, is fittingly being used as a metaphor of $L_1$ constraint applied to linear model. Coincidently, LASSO is also the initials for Least Absolute Shrinkage and Selection Operator. LASSO minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant. LASSO not only helps to improve the prediction accuracy when dealing with multicolinearity data, but also carries several nice properties such as interpretability and numerical stability. Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models \cite{lasso}. In a Bayesian context, this is equivalent to placing a zero-mean Laplace prior distribution on the parameter vector \cite{bayes_lasso}. The optimization problem may be solved using quadratic programming or more general convex optimization methods. The formulation of LASSO regression can mathematically be written as
%
%\begin{equation}
%\label{eq:lasso}
%\beta^{lasso} = \arg \! \operatorname*{\min}_\beta \sum_{i=1}^n (\mathbf{y}_i - ( \beta_0 + \beta^T \mathbf{x}_i))^2 + \lambda \norm{\beta}_1
%\end{equation}
%
%The difference between LASSO and ridge regression is that the penalty is increased and all parameters remain non-zero while still being reduced, using ridge regression. While increasing the penalty using LASSO will cause more and more of the parameters to be driven to zero. This makes LASSO an advantage for practical and model evaluation reasoning as we have to deal with less parameters to the model. As a result, LASSO selects more relevant features and discards the others, whereas ridge regression never fully discards any features. \cite{bayes_lasso}
%
%\subsubsection{Gradient Descent}
%
%The gradient descent method gives a way to find a local minimum of a given general function. We initialize the algorithm by a guess to the solution and compute the gradient for the function at the initial point. Next, we start the process of stepping through the negative direction of the gradient and repeat the process until the algorithm has converged at which the gradient is zero or until we reach a point at which we believe that there is no local minimum. The algorithm is called a first-order algorithm as it only takes the first derivative of the function into account.
%
%Here we will walk through the process of a gradient descent algorithm for finding the solution to the minimum of some function $f(x)$. Initializing with some initial value $x_0$ for $x$, we can change its value proportional to the dimension of x: with only one dimension; we can make it higher or lower. The best direction at $x_0$ to minimize $f$, we take the gradient $\nabla f$ along every dimension of $x$. Intuitively, the gradient will give the curve at that which $x$ will point to an increase in the function. Therefore we change $x$ in the opposite direction to lower the function value:
%\begin{equation}
%x_{k+1} = x_k - \lambda \nabla f(x_k)
%\end{equation}
%where $\lambda$ represents the stepsize in the discretization of the function we represent. This can be a fixed value or change depending on which implementation one decides to chose. Often $\lambda>0$ is a small number that forces the algorithm to make small jumps, even for the most minimal $\lambda$ it has been shown to converge to a minimum. As a result, the algorithm will be stable and its optimal value depends on the function, given stable conditions (and a certain choice of $\lambda$), it is guaranteed that $f(x_{k+1}) \leq f(x_k)$ \cite{gradient}.
%
%\subsubsection{Coordinate Descent}
%
%Instead of relying on the gradient of a function, one approach called the coordinate descent can minimize a function by minimizing it along one direction at a time. By the given regression model we can minimize the function $f(x)$.
%\begin{equation}
%y = X \theta,
%\end{equation}
% the function to minimise for a least squares regression is
% \begin{equation}
% f(\theta) = | y- X \theta |^2.
% \end{equation}
% Minimising over $\theta$ is achieved by
% \begin{equation}
% 0 = X_{i}^\top (X \theta - y) = X_{i}^\top (X_i \theta_i + X_{-i} \theta_{-i} -y).
% \end{equation}
% So the next step in the coordinate descent update is
% \begin{equation}
% \theta_{i} = \frac{X_{i}^\top(y- X_{-i}\theta_{-i})}{X_{i}^\top X_{i}}.
% \end{equation}
%Regardless the learning rate of the gradient descent procedure (which could indeed speed up convergence), the comparison between the two is fair at least in terms of complexity.
%Coordinate descent needs to perform $O(n)$ operations for each coordinate update (n operations to compute residuals $r = (y- X_{-i}\theta_{-i})$ and n to compute $X_{i}^\top r$ . Each cycle of this type is performed p times, where p is the number of covariates.
%Gradient descent performs the same number of operations $O(np)$.
%


%%%lecture 10

\begin{beispiel}
	$E = L^2([0,1])$ and $f,g \in E$ with
	\[
		\skal{f}{g}_{L^2} = \int_{0}^{1} f(x) \overline{g(x)} \,\mathrm{d}x, \qquad \norm{f}_{L^2} = \left( \int_{0}^{1} \abs{f(x)}^2 \,\mathrm{d}x \right)^{\frac{1}{2}}.
	\]
	Set $h \in C([0,1] \times [0,1])$ and for $f \in L^2([0,1])$
	\[
		A(f)(x) = \int_{0}^{1} h(x,y)f(y) \,\mathrm{d}y, \qquad x \in [0,1].
	\]
	Then
	\[
		\norm{A} \leq \left( \int_{0}^{1} \left( \int_{0}^{1} \abs{h(x,y)}^2 \,\mathrm{d}y \right) \,\mathrm{d}x \right)^{\frac{1}{2}} < \infty.
	\]
\end{beispiel}
\begin{beispiel}
	Let $(E, \norm{.})$ be a normed space. Then there are no $A,B \in B(E,E)$ such that
	\[
		AB - BA = I
	\]
	where $I$ is the identity ($I(x)=x$ for $x \in E$).
	\begin{bemerkung}
		Consider $ f \in E = C^{\infty}([0,1])$ and
		\[
			A = \diffd{}{x}, \qquad B=x.
		\]Then
		\[
			(AB - BA)(f)(x) = \diffd{}{x}(x(f(x))) - x \diffd{}{x}f(x) = f(x).
		\]
	\end{bemerkung}
	Argue by contradiction. \\
	Assume $A,B \in B(E,E)$ with $AB-BA = I$. \\
	Hint: Consider $A^nB-BA^n$ for $n = 1,2,\dots$. For $n =2$ we have
	\begin{align*}
		A^2B-BA^2 &= A^2B-ABA + ABA - BA^2 \\
		&= A(AB-BA) + (AB-BA)A \\
		&= 2A.
	\end{align*}
	For $n=3$ we have
	\begin{align*}
		A^3B-BA^3 &= A^3B-A^2BA + A^2BA - BA^3 \\
		&=A^2(AB-BA) + (A^2B-BA^2)A \\
		&= 3A^2.
	\end{align*}
	In general
	\[
		A^nB-BA^n = nA^{n-1}, \qquad n=2,3,4,\dots \qquad (*)
	\]
	Check using an induction argument. We obtain
	\[
		n \norm{A^{n-1}} = \norm{A^nB-BA^n} \leq \norm{A^nB} + \norm{BA^n} \leq  2 \norm{A^{n-1}} \norm{A} \norm{B}
	\]
	Hence \[
		(2 \norm{A} \norm{B} -n) \norm{A^{n-1}} \geq 0, \qquad  \forall\,  n = 2,3,\dots.
	\]
	We conclude that $\norm{A^{n-1}}= 0$ for $n$ large enough. Clearly the same for $\norm{A^n}$. This yields $A^n=0$ for $n$ large enough. Repeated use of $(*)$ gives $A=0$. This contradicts $AB-BA= I$ so the implication in the example is proven.
\end{beispiel}

Recall a important theorem:

\begin{theorem}[Riesz representation theorem]
	$(E, \skal{.}{.})$ Hilbert space $f \in B(E,\mathbb{C})$. $f$ is bounded linear functional on $E$. This yields
	\[
		\exists\,! x_f \in E: \qquad f(x) = \skal{x}{x_f}, \qquad \forall\, x \in E.
	\]
	Also it holds
	\[
		\underset{\substack{\text{operator norm} \\ \text{of $f$}}}{\underbrace{\norm{f}}} = \underset{\substack{norm of} \\ \text{$x_f$ in $E$}}{\underbrace{\norm{x_f}}}.
	\]
\end{theorem}

\begin{definition}
	$\varphi: E \times E \to \mathbb{C}$ is called:
	\begin{itemize}
		\item Bilinear, if for scalars $\alpha$ and $\beta$ it holds
		\begin{align*}
			\varphi( \alpha x+ \beta y,z) &= \alpha \varphi(x,z)+ \beta \varphi(y,z) \qquad \forall\, x,y,z \in E \\
			\varphi(x,\alpha y + \beta z) &= \bar{\alpha} \varphi(x,z) + \overline{\beta} \varphi(y,z) \qquad \forall\, x,y,z \in E.
		\end{align*}
		\item Bounded, if there exists $M>0$ such that
		\[
			\abs{\varphi(x,y)} \leq M \norm{x}\norm{y}, \qquad \forall\, x,y \in E.
		\]
		\item Coercive, if there exists $K>0$ such that
		\[
			\varphi(x,x) \geq K \norm{x}^2, \qquad \forall\, x \in E.
		\]
	\end{itemize}
\end{definition}
Clearly $\skal{.}{.}$ in $E$ is a bilinear, bounded and coercive functional in $E$ (with $M=K=1$). \\
We will now introduce a Generalization of the Riesz representation theorem.

\begin{theorem}[Lax-Milgram]
	$(E, \skal{.}{.})$ Hilbert space. Let $\varphi: E \times E \to \mathbb{C}$ be a bilinear, bounded and coercive functional. $f: E \to \mathbb{C}$ bounded linear functional in $E$. Then there exists an unique $x_f \in E$ such that
	\[
		f(x) = \varphi(x,x_f), \qquad  \forall\, x \in E.
	\]
\end{theorem}

\begin{beweis}
	\begin{enumerate}[Step 1:]
		\item $\exists\,!$ $A \in B(E,E)$ with
		\[
			\varphi(x,y)= \skal{x}{A(y)}, \qquad \forall\, x,y \in E.
		\]
		\item $A$ is injective and surjective.
		\item Apply RRT with $ \tilde x_f = A^{-1}(x_f)$
		\begin{align*}
			f(x) &= \skal{x}{x_f} \\ &= \skal{x}{A(A^{-1}(x_f))} \\ &= \varphi(x, \tilde x_f), \qquad \forall\, x \in E.
		\end{align*}
	\end{enumerate}
	\begin{description}
		\item[Step 1:] Fix $y \in E$ and consider for $x \in E$
		\[
			x \stackrel{f_y}{\mapsto } \varphi(x,y) \in \mathbb{C}.
		\]
		\textbf{Claim:} \text{    }$f_y: E \to \mathbb{C}$ is a bounded linear functional. \\
		For $x,y,z \in E$ and $\alpha,\beta$ scalars we have
		\begin{align*}
			f_y(\alpha x+ \beta z) &= \varphi(\alpha x + \beta z,y) \\
			&= \alpha \varphi(x,y) + \beta \varphi(z,y) \\
			&= \alpha f_y(x) + \beta f_y(z).
		\end{align*}
		Hence $f_y$ is linear. It is bounded because of
		\[
			\abs{f_y(x)} = \abs{\varphi(x,y)} \leq (M \norm{y})\norm{x}, \qquad \forall\, x \in E.
		\]
		So $f_y$ is bounded. \\
		RRT implies $f_y(x) = \skal{x}{A(y)}$ for all $x \in E$ for some $A(y) \in E$. \\ Now we have $A : E \to E$.
		\textbf{Claim:} \text{    }$A \in B(E,E)$. \\
		For $x,y,z \in E$ and scalars $\alpha, \beta$ we have
		\begin{align*}
			\skal{x}{A( \alpha y + \beta z)} &= \varphi(x, \alpha y + \beta z) \\
			&= \bar{\alpha} \varphi(x,y)+ \bar{\beta} \varphi(x,z) \\
			&= \bar{\alpha} \skal{x}{A(y)} + \bar{\beta} \skal{x}{A(z)} \\
			&= \skal{x}{\alpha A(y)}+ \skal{x}{\beta A(z)}. \\
		\end{align*}
		This is equivalent to
		\[
			\skal{x}{A(\alpha y + \beta z)- \alpha A(y) - \beta A(z)} = 0, \qquad  x \in E.
		\]
		This implies
		\[
			\norm{A(\alpha y + \beta z) - \alpha A(y) - \beta A(z)} = 0.
		\]
		So \[
			A( \alpha y+ \beta z) = \alpha A(y) + \beta A(z) \qquad \forall\, y,z \in E \text{ and scalars }\beta,\alpha.
		\]
		Hence, $A$ is linear. We will now show that $A$ is bounded: \\
		We know because $\varphi$ is continuous that for all $x,y \in E$
		\[
			\abs{\skal{x}{A(y)}} = \abs{\varphi(x,y)} \leq M \norm{x} \norm{y}.
		\]
		Take $x = A(y)$ and get
		\[
			\norm{A(y)}^2 \leq  M \norm{A(y)} \norm{y} \qquad \forall\, y \in E
		\]
		which implies
		\[
			\norm{A(y)} \leq M \norm{y} \qquad \forall\, y \in E.
		\]
		Hence $\norm{A} \leq M < \infty$.
		\item[Step 2:] Note $\varphi(x,y) = \skal{x}{A(y)}$ for alle $x,y \in E$. \\
		\textbf{Claim:} \text{    }$A$ is injective, i.e.
		\[
			A(x_1) = A(x_2) \qquad \Rightarrow \qquad x_1 = x_2.
 		\]
		$\varphi$ is coercive so
		\[
			\norm{x}^2 \leq \frac{\varphi(x,x)}{K} = \frac{1}{K} \underset{>0}{\underbrace{\abs{\skal{x}{A(x)}}}} \leq \frac{1}{K} \norm{x} \norm{A(x)} \qquad \forall\,  x \in E.
		\]
		Hence \[
			\norm{x} \leq  \frac{1}{K} \norm{A(x)}, \qquad \forall\, x \in E.
		\]
		If $A(x_1) = A(x_2)$ we have $A(x_1-x_2) = 0 \in E$ then
		\[
			\norm{x_1 - x_2} \leq \frac{1}{K} \norm{A(x_1 -x_2)} = 0.
		\]
		We get $x_1 = x_2$. \\
		\textbf{Claim:} \text{    }$A$ is surjective, i.e. the image of $A$ is $E$: \[
			\mathcal{R}(A) = \set[A(x)]{x \in E} = E.
		\]
		We first show that $\mathcal{R}(A)$ is a closed subspace of $E$.
		\begin{itemize}
			\item $\mathcal{R}(A)$ is a subspace in $E$ since $A$ is linear.
			\item $\mathcal{R}(A)$ is closed
			\[
				y_n \to y \qquad \text{in }(E, \norm{.}) \qquad \Rightarrow y \in \mathcal{R}(A).
			\]
		\end{itemize}
		$\mathcal{R}(A)$ is linear. Take $y_1,y_2 \in \mathcal{R}(A)$ with preimages $x_1,x_2$ and yield
		\[
			\alpha_1 y_1 + \alpha_2 y_2 = \alpha_1 A(x_1) + \alpha_2 A(x_2) = A(\alpha_1 x_1 + \alpha_2 x_2).
		\]
		So \[
			\alpha_1 y_1 + \alpha_2 y_2 \in \mathcal{R}(A).
		\]
		Assume \[
			y_n \to y \qquad \text{in }(E, \norm{.}).
		\]
		For $n=1,2,\dots$ there are $x_1,x_2,\dots$ such that $y_n= A(x_n)$ for $n=1,2,\dots$. \\
		\textbf{Claim:} \text{    }$(x_n)_{n \in \mathbb{N}}$ is a Cauchy sequence in $E$ since
		\begin{align*}
			\norm{x_n-x_m} &\leq \frac{1}{K} \norm{A(x_n-x_m)} \\
			&= \frac{1}{K} \norm{A(x_n)- A(x_m)} \\
			&= \frac{1}{K} \norm{y_n -y_m} \to 0, \qquad n,m \to \infty
		\end{align*}
		since $(y_n)_{n \in \mathbb{N}}$ converges. \\
		Since $(E, \norm{.})$ is a Banach space $(x_n)_{n \in \mathbb{N}}$ converges in $(E, \norm{.})$. Call the limit $x \in E$. Hence
		\[
			A(x_n) \to y
		\]
		since $A$ is bounded, continuos and linear. So $y = A(x)$ and we get $y \in \mathcal{R}(A)$. \\
		Secondly $A$ is surjective, i.e. $\mathcal{R}(A)=E$. \\
		Assume that this is not true. The Orthogonal decomposition theorem gives
		\[
			E = \mathcal{R}(A) \oplus \mathcal{R}(A)^{\perp}.
		\]
		The first one is a closed subspace in $E$ and the second one is not empty by assumption. Fix $z \in \mathcal{R}(A)^{\perp} \setminus \set{0}$. Note
		\[
			\varphi(x,y) = \skal{x}{A(y)} \qquad x,y \in E
		\]
		With $x = y = z$ we get
		\[
			\varphi(z,z) = \skal{z}{A(z)} = 0
		\]
		and
		\[
			\varphi(z,z) \geq K \norm{z}^2 \geq 0 \qquad \Rightarrow  \,z=0.
		\]
		This is a contradiction. \\
		The Conclusion is
		\[
			\mathcal{R}(A)^{\perp} = \set{0} \qquad \Rightarrow \qquad \mathcal{R}(A) = E.
		\]
		We have $\varphi(x,y) = \skal{x}{A(y)}$ for all $x,y \in E$ and $A \in B(E,E)$ surjective.
		\item[Step 3:] see above.
	\end{description}
\end{beweis}

\subsection{Adjoint operator}
\label{sub:adjoint_operator}

$(E,\skal{.}{.})$ Hilbert space and $A \in B(E,E)$ with adjoint $A^{*}$, i.e.
\[
	\skal{A(x)}{y} = \skal{x}{A^*(y)} , \qquad \forall\, x,y \in E.
\]
Fix $y \in E$ and consider
\[
	x \stackrel{f_y}{\mapsto } \skal{A(x)}{y} \in \mathbb{C}.
\]
\textbf{Claim:} \text{    }$f_y$ is a bounded linear functional on $E$
\begin{itemize}
	\item linear since $A$ is linear.
	\item bounded since $A$ is bounded with
	\[
		\abs{f_y(x)} \leq (\norm{A}\norm{y})\norm{x}, \qquad x \in E.
	\]
\end{itemize}
	RRT implies
	\[
		f_y(x) = \skal{x}{A^*(y)}, \qquad x \in E.
	\]
	We have $A^*: E \to E$ such that
	\[
		\skal{A(x)}{y} = \skal{x}{A^*(y)} , \qquad \forall\, x,y \in E.
	\]
\begin{proposition}
	$A \in B(E,E)$. Then $A^* \in B(E,E) $ and $\norm{A^*} = \norm{A}$.
\end{proposition}
\begin{beweis}
	$A^*$ linear:
	\[
		\skal{x}{A^* ( \alpha y + \beta z)} = \skal{x}{\alpha A^*(y) + \beta A^*(z)} \qquad \forall\, x,y \in E.
	\]
	$A^*$ bounded: \\ Take $x= A^*(y)$ and get
	\begin{align*}
		\norm{A^*(y)}^2 &= \abs{\skal{A(A^*(y))}{y}} \\
		&\leq \norm{A(A^*(y))} \norm{y} \\
		&\leq \norm{A} \norm{A^*(y)} \norm{y}, \qquad y \in E.
	\end{align*}
	We get
	\[
		\norm{A^*(y)} \leq \norm{A} \norm{y}, \qquad y \in E.
	\]
	Conclucion: $A^* \in B(E,E)$. We also get
	\[
		\norm{A^*} \leq \norm{A}.
	\]
	But we also know that $A^{**} = A$ since
	\begin{align*}
		\skal{x}{A^{**}(y)} &= \skal{A^*(x)}{y} \\
		&= \overline{\skal{y}{A^*(x)}} \\
		&= \overline{\skal{A(y)}{x}} \\
		&= \skal{x}{A(y)}, \qquad x,y \in E.
	\end{align*}
	So \[
		\norm{A} = \norm{A^{**}} \leq \norm{A^*}
	\]
	which impllies
	\[
		\norm{A} = \norm{A^*}.
	\]
\end{beweis}
\begin{bemerkung}
	$A,B \in B(E,E)$ then
	\begin{align*}
		(A+B)^* &= A^* + B^* \\
		(AB)^* &= B^* A^* \\
		(\alpha A)^* &= \bar{\alpha}A^* \\
		A^{**} &= A \\
		I^* &= I.
	\end{align*}
\end{bemerkung}
\begin{beispiel}
	Continuity of the example above: For $f \in L^2([0,1])$ consider
	\[
		A(f)(x) = \int_{0}^{1}h(x,y)f(y) \,\mathrm{d}y, \qquad x \in [0,1].
	\]
	For $g \in L^2([0,1])$ it holds
	\begin{align*}
		\skal{A(f)}{g}_{L^2} &= \int_{0}^{1} A(f)(x)\overline{g(x)} \,\mathrm{d}x \\
		&= \int_{0}^{1}\int_{0}^{1} h(x,y)f(y) \,\mathrm{d}x \overline{g(x)} \,\mathrm{d}x \\
		&= \int_{0}^{1}f(y) \cdot \int_{0}^{1}h(x,y) \overline{g(x)} \,\mathrm{d}x \,\mathrm{d}y \\
		&= \int_{0}^{1} f(y) \cdot \overline{\int_{0}^{1} \overline{h(x,y)} g(x) \,\mathrm{d}x} \,\mathrm{d}y \\
		&= \skal{f}{A^*(g)}_{L^2}.
	\end{align*}
	This gives us
	\[
		A^*(f)(x) = \int_{0}^{1} \overline{h(y,x)}f(y) \,\mathrm{d}y, \qquad x \in [0,1].
	\]
\end{beispiel}

\begin{beispiel}
	$A \in B(E,E)$. It follows
	\[
		\mathcal{R}(A)^{\perp} = N(A^*) = \set[x \in E]{A^*(x) = 0}
	\]
	since $x \in \mathcal{R}(A)^{\perp}$. It is equivalent that
	\[
		\skal{x}{A(y)} = 0, \qquad \forall\, y \in E
	\]
	\[
		\Leftrightarrow \qquad \skal{A^*(x)}{y} = 0, \qquad \forall\, y \in E
	\]
	\[
		\Rightarrow \qquad A^*(x) = 0 \qquad \Leftrightarrow \qquad x \in N(A^*).
	\]
	We get
	\[
		N(A^*)^{\perp} = \overline{\mathcal{R}(A)}
	\]
	since
	\[
		N(A^*)^{\perp} = \left( R(A)^{\perp} \right)^{\perp} = \overline{\spn(\mathcal{R}(A))} = \overline{\mathcal{R}(A)}.
	\]
\end{beispiel}
\begin{bemerkung}
	$A \in B(E,E)$ is called self adjoint if $A^* = A$.
\end{bemerkung}
For $A \in B(E,E)$ we have
\[
	\norm{A} =\sup\limits_{\substack{\norm{x} = 1  \\ \norm{y}=1}} \abs{\skal{A(x)}{y}}
\]
since
\[
	\norm{\skal{A(x)}{y}} \leq \underset{\leq \norm{A}\norm{x}}{\underbrace{\norm{A(x)}}} \leq \norm{A}, \qquad \text{for }\norm{x}=\norm{y}=1.
\]
If $A(x) = 0$ for all $x \in E$ then $\norm{A}=0$ and also
\[
	\sup\limits_{\substack{\norm{x} = 1  \\ \norm{y}=1}} \abs{\skal{A(x)}{y}} = 0.
\]
For $x$ with $A(x) \neq 0$ then it is
\[
	A \left( \frac{1}{\norm{x}}x \right) \neq 0.
\]
For such an $x$ with $\norm{x}=1$ we have
\[
	\abs{\skal{A(x)}{\frac{1}{\norm{A(x)}}A(x)}} = \frac{1}{\norm{A(x)}} \norm{A(x)}^2 = \norm{A(x)}
\]
and
\[
	\norm{A} \leq \sup_{\norm{x}=1} \norm{A(x)} \leq \sup\limits_{\substack{\norm{x} = 1  \\ \norm{y}=1}} \abs{\skal{A(x)}{y}} \leq \norm{A}.
\]
\begin{proposition}
	Let $A \in B(E,E)$ be self-adjoint. Then
	\[
		\norm{A} = \sup_{\norm{x}=1} \abs{\skal{A(x)}{x}}.
	\]
\end{proposition}
\begin{beweis}
	Set
	\[
		M = \sup_{\norm{x}=1} \abs{\skal{A(x)}{x}}.
	\]
	For $\norm{x}=1$ we have
	\[
		\abs{\skal{A(x)}{x}} \leq \norm{A(x)} \norm{x} \leq \norm{A}.
	\]
	Furthermore
	\[
		M \leq \norm{A}.
	\]
	It remains to prove: $\norm{A} \leq M$. \\
	For $x,z \in E$ consider:
	\begin{align*}
		\skal{A(x+z)}{x+z} - \skal{A(x-z)}{x-z} &= 2 \skal{A(x)}{z} + 2 \skal{A(z)}{x} \\
		&= 2 \left( \skal{A(x)}{z} + \skal{z}{A^*(x)} \right) \\
		&= 2 (\skal{A(x)}{z} + \skal{z}{A(x)}) \\
		&= 4 \re( \skal{A(x)}{z}).
	\end{align*}
	Assume now $A(x) \neq 0$ and set
	\[
		z = \frac{1}{\norm{A(x)}} A(x).
	\]
	Hence
	\[
		\norm{A(x)} = \frac{1}{4} \left( \skal{A(x+\frac{1}{\norm{A(x)}} A(x))}{x + \frac{1}{\norm{A(x)}} A(x)}
		- \skal{A(x-\frac{1}{\norm{A(x)}} A(x))}{x- \frac{1}{\norm{A(x)}} A(x)} \right).
	\]
	Note \[
		\abs{\skal{A(y)}{y}} = \norm{y}^2 \abs{\skal{A(\frac{1}{\norm{y}}y)}{ \frac{1}{\norm{y}}y}} \leq M \norm{y}^2.
	\]
	We now obtain
	\begin{align*}
		\norm{A(x)} &\leq \frac{1}{4} \left( M \norm{x+\frac{1}{\norm{A(x)}} A(x)}^2 + M \norm{x - \frac{1}{\norm{A(x)}} A(x)}^2 \right) \\
		&= \frac{M}{4} 2 \left( \norm{x}^2 + \norm{\frac{1}{\norm{A(x)}} A(x)}^2 \right) \\
		&= \frac{M}{2} (\norm{x}^2 +1).
	\end{align*}
	So
	\[
		\norm{A} = \sup_{\norm{x}=1} \norm{A(x)} \leq M
	\]
	and this yields
	\[
		\norm{A}= M.
	\]
\end{beweis}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{parskip}% http://ctan.org/pkg/parskip
%for degrees
\usepackage{gensymb}
%for bmatrix
\usepackage{amsmath}
% for Sarrus
\usepackage{tikz}
\usetikzlibrary{calc,matrix}

\begin{document}

\section{Algebra}
	\subsection{Exponent Properties}
		\begin{equation}
			\frac{a^n}{a^m} = a^{n-m}
		\end{equation}
		\begin{equation}
			x^a y^a = \left( {xy} \right)^a
		\end{equation}
		\begin{equation}
			x^{\left( {\frac{a}{b}} \right)} = \sqrt[b]{{x^a }}
		\end{equation}
	\subsection{Properties of radicals}
		\begin{equation}
			\sqrt[n]{a} = a^{\frac{1}{n}}
		\end{equation}
		\begin{equation}
			\sqrt[n]{ab} = \sqrt[n]{a}\sqrt[n]{b}
		\end{equation}
		\begin{equation}
			\sqrt[m]{\sqrt[n]{a}} = \sqrt[nm]{a}
		\end{equation}
		\begin{equation}
			\sqrt[n]{\frac{a}{b}} = \frac{\sqrt[n]{a}}{\sqrt[n]{b}}
		\end{equation}
		\begin{equation}
			\sqrt[n]{a^n} = |a|, \ \ \mbox{if $n$ is even}
		\end{equation}
	\subsection{Complex numbers}
		\begin{equation}
			(a+bi)(c+di) =  ac-bd+(ad+bc)i
		\end{equation}
		\begin{equation}
			(a+bi)(a-bi) =  a^2 + b^2
		\end{equation}
		\begin{equation}
			|a + bi| = \sqrt{a^2+b^2} \ \ \mbox{Complex Modulus}
		\end{equation}
		\begin{equation}
			\overline{(a+bi)}=a-bi
		\end{equation}
		\begin{equation}
			z_1 \cdot z_2 = r_1 \cdot r_2 e^{i(\theta_1 + \theta_2)}
		\end{equation}
		\begin{equation}
			z^{\frac{1}{n}} = \sqrt[n] r \cdot e^{i(\frac{\phi}{n} + \frac{2k\pi}{n})}; \ \ k = 0,1,..,n-1
		\end{equation}
		\begin{equation}
			e^{ni\theta}=\cos{n\theta} + \sin{n\theta}  \ \ \mbox{De Moivre’s Formula}
		\end{equation}
	\subsection{Logarithms}
		\begin{equation}
			\log _b b = 1
		\end{equation}
		\begin{equation}
			\log _b 1 = 0
		\end{equation}
		\begin{equation}
			\log _b (x^r) = r \log _b x
		\end{equation}
		\begin{equation}
			\log _b (xy) = \log _b (x) + \log _b (y)
		\end{equation}
		\begin{equation}
			\log _b \left( \frac{x}{y} \right) = \log _b (x) - \log _b (y)
		\end{equation}
		\begin{equation}
			\log _b \left( x \right) = \log _b \left( c \right)\log _c \left( x \right) = \frac{{\log _c \left( x \right)}}{{\log _c \left( b \right)}}
		\end{equation}
	\subsection{Quadratic Formula}
		\begin{equation}
			x = \frac{{ - b \pm \sqrt {b^2 - 4ac} }}{{2a}} \ \ \mbox{when $ax^2 + bx + c = 0$}
		\end{equation}
\section{Linear Algebra}
	Matrix addition: one by one. (commutative, associative)

	Scalar multiplication: all.

	Matrix "multiplication of rows into columns". Multiplication is not commutative ($AB\neq BA$).

	\begin{equation}
		c_{jk} = \sum_{i=1}^{n} a_{ji}b_{ik}
	\end{equation}

	Inner or dot product of Vectors
	\begin{equation}
		\langle a,b \rangle = \mathbf{a} \bullet \mathbf{b} = \textbf{a}^T\mathbf{b}
	\end{equation}

	Matrix to the power $A^0=I$

	Inverse:
	\begin{equation}
		\begin{bmatrix}
			a & b \\ c & d \\
		\end{bmatrix}^{-1} =
		\frac{1}{\det(\mathbf{A})}
		\begin{bmatrix}
			\,\,\,d & \!\!-b \\ -c & \,a \\
		\end{bmatrix} =
		\frac{1}{ad - bc}
		\begin{bmatrix}
			\,\,\,d & \!\!-b \\ -c & \,a \\
		\end{bmatrix}
	\end{equation}
	Identities
	\begin{equation}
		(AB)^T = B^TA^T
	\end{equation}
	\begin{equation}
		(A+B)^T = A^T+B^T
	\end{equation}
	\begin{equation}
		(AB)^{-1} = B^{-1}+A^{-1}
	\end{equation}
	\begin{equation}
		A^kB^l = A^{k+l}
	\end{equation}
	Conjungate transpose / adjugate
	\begin{equation}
		A^* = (\overline{A})^\mathrm{T} = \overline{A^\mathrm{T}}
	\end{equation}
	Determinants
		\begin{equation}
			\det(\mathbf{A}) = \sum_{\sigma \in S_n} \mbox{sgn}(\sigma) \prod_{i=1}^n A_{i,\sigma_i}
		\end{equation}
		For 3$\times$3 matrices (Sarrus rule)
		\begin{equation}
			\begin{tikzpicture}[>=stealth]
			\matrix [%
			  matrix of math nodes,
			  column sep=1em,
			  row sep=1em
			] (sarrus) {%
			  a_{11} & a_{12} & a_{13} & a_{11} & a_{12} \\
			  a_{21} & a_{22} & a_{23} & a_{21} & a_{22} \\
			  a_{31} & a_{32} & a_{33} & a_{31} & a_{32} \\
			};

			\path ($(sarrus-1-1.north west)-(0.5em,0)$) edge ($(sarrus-3-1.south west)-(0.5em,0)$)
				  ($(sarrus-1-3.north east)+(0.5em,0)$) edge ($(sarrus-3-3.south east)+(0.5em,0)$)
				  (sarrus-1-1)                          edge            (sarrus-2-2)
				  (sarrus-2-2)                          edge[->]        (sarrus-3-3)
				  (sarrus-1-2)                          edge            (sarrus-2-3)
				  (sarrus-2-3)                          edge[->]        (sarrus-3-4)
				  (sarrus-1-3)                          edge            (sarrus-2-4)
				  (sarrus-2-4)                          edge[->]        (sarrus-3-5)
				  (sarrus-3-1)                          edge[dashed]    (sarrus-2-2)
				  (sarrus-2-2)                          edge[->,dashed] (sarrus-1-3)
				  (sarrus-3-2)                          edge[dashed]    (sarrus-2-3)
				  (sarrus-2-3)                          edge[->,dashed] (sarrus-1-4)
				  (sarrus-3-3)                          edge[dashed]    (sarrus-2-4)
				  (sarrus-2-4)                          edge[->,dashed] (sarrus-1-5);

			\foreach \c in {1,2,3} {\node[anchor=south] at (sarrus-1-\c.north) {$+$};};
			\foreach \c in {1,2,3} {\node[anchor=north] at (sarrus-3-\c.south) {$-$};};
		  \end{tikzpicture}
		\end{equation}
		\begin{equation}
			\det (A \cdot B) = \det (A) \cdot \det (B)
		\end{equation}
		\begin{equation}
			\det (A^{-1}) = \det (A)^{-1}
		\end{equation}
		\begin{equation}
			\det\left(rA\right) = r^n\det(A)\ \ \ \ \mbox{for all $A^{n\times n}$ and scalars $r$}
		\end{equation}

		The determinant of a triangular matrix equals the product of the diagonal entries. Since for any triangular matrix A the matrix ${\displaystyle \lambda I-A}$, whose determinant is the characteristic polynomial of A, is also triangular, the diagonal entries of A in fact give the multiset of eigenvalues of A (an eigenvalue with multiplicity m occurs exactly m times as diagonal entry)

		\subsection{Transpose}
			\begin{equation}
				[A^\mathrm{T}]_{ij} = [A]_{ji}
			\end{equation}
			\begin{equation}
				(A^T)^T = A
			\end{equation}
			\begin{equation}
				(AB)^T = B^TA^T %validated..
			\end{equation}
			\begin{equation}
				det(A^T) = det(A)
			\end{equation}
			\begin{equation}
				(A^T)^{-1} = (A^{-1})^T
			\end{equation}
\section{Trigonometry}
	\subsection{Definitions}
		\begin{equation}
			\sin \theta = \frac{{{\rm{opposite}}}}{{{\rm{hypotenuse}}}}
		\end{equation}
		\begin{equation}
			\tan \theta = \frac{{{\rm{opposite}}}}{{{\rm{adjacent}}}}
		\end{equation}
	\bgroup
	\def\arraystretch{2}
		\begin{table}[h]
		\centering
		\begin{tabular}{cccccc}
			\hline
				$\Theta$	& $0\degree$ & $30\degree$ & $45\degree$ & $60\degree$ & $90\degree$ \\
				$sin \Theta$	& $0$ & $\frac{1}{2}$ & $\frac{1}{\sqrt{2}}$ & $\frac{\sqrt{3}}{2}$ & 1	\\
				$cos \Theta$	& $1$ & $\frac{\sqrt{3}}{2}$ & $\frac{1}{\sqrt{2}}$ & $\frac{1}{2}$	& 0 \\
				$tan \Theta$	& $0$ & $\frac{1}{\sqrt{3}}$ & $1$ & $\sqrt{3}$	& / \\
			\hline
		\end{tabular}
  		\caption{Trigonometric functions standard values}
  		\label{tab:standard-values}
		\end{table}
	\egroup
	\subsection{Formulas and Identities}
		\begin{equation}
			\tan \theta = \frac{\sin \theta}{\cos \theta}
		\end{equation}
		Pythagorean identities
		\begin{equation}
			\sin^2 \theta + \cos^2 \theta = 1
		\end{equation}
		Odd/Even formulas
		\begin{equation}
			\sin(-\theta) = - \sin \theta
		\end{equation}
		\begin{equation}
			\cos(-\theta) = \cos \theta
		\end{equation}
		Sum and difference formulas
		\begin{equation}
			\sin \left( {\alpha \pm \beta} \right) = \sin \alpha \cos \beta \pm \cos \alpha \sin \beta
		\end{equation}
		\begin{equation}
			\cos \left( {\alpha \pm \beta } \right) = \cos \alpha \cos \beta \mp \sin \alpha \sin \beta
		\end{equation}
		Double angle formulas
		\begin{equation}
			\sin 2\theta = 2\sin \theta \cos \theta
		\end{equation}
		\begin{equation}
			\cos 2\theta = \cos ^2 \theta - \sin ^2 \theta = 2\cos ^2 \theta - 1
		\end{equation}
		Half angle formulas
		\begin{equation}
			\tan \frac{\theta}{2} = \pm \sqrt{\frac{1-cos(\theta)}{1+cos(\theta)}}
		\end{equation}
		Euler's theorem
		\begin{equation}
			e^{ \pm i\theta } = \cos \theta \pm i\sin \theta
		\end{equation}
		\begin{equation}
			\cos \theta = \frac{1}{2} (e^{i\theta} + e^{-i\theta})
		\end{equation}
		\begin{equation}
			\sin \theta = \frac{1}{2i} (e^{i\theta} - e^{-i\theta})
		\end{equation}
		Miscellaneous formulas
		\begin{equation}
			\sin ^2 \theta = \frac{1-cos(2\theta)}{2}
		\end{equation}
		\begin{equation}
			\cos ^2 \theta = \frac{1+sin(2\theta)}{2}
		\end{equation}
\section{Calculus111}
	\subsection{Limits}
		\subsubsection{Properties}
			\begin{equation}
				\mathop {\lim }\limits_{x \to a} \left[ cf(x) \right] = c\mathop{\lim }\limits_{x \to a} f(x)
			\end{equation}
			L'Hopital's Rule
			\begin{equation}
				\mathop {\lim }\limits_{x \to c} \frac{{f\left( x \right)}}{{g\left( x \right)}} = \mathop {\lim }\limits_{x \to c} \frac{{f'\left( x \right)}}{{g'\left( x \right)}}
			\end{equation}
		\subsubsection{Evaluations}
			\begin{equation}
				\mathop {\lim }\limits_{x \to 0} \frac{{\sin x}}{x} = 1
			\end{equation}
			\begin{equation}
				\mathop {\lim }\limits_{x \to - \infty } e^x = 0
			\end{equation}
	\subsection{Derivatives}
		\subsubsection{Definition}
			\begin{equation}
				\frac{d}{{dx}}f\left( x \right) = \mathop {\lim }\limits_{h \to 0} \frac{{f\left( {x + h } \right) - f\left( x \right)}}{h }
			\end{equation}
		\subsubsection{Properties}
			Product rule
			\begin{equation}
				\left(fg\right)'=f'g+fg'
			\end{equation}
			Chain rule
			\begin{equation}
				\frac{d}{{dx}}\left[ {f\left( u \right)} \right] = \frac{d}{{du}}\left[ {f\left( u \right)} \right]\frac{{du}}{{dx}} \ \ \ \mbox{or} \ \ \ \left(f(g(x)\right)'=f'\left(g(x)\right)g'(x) \ \ \ \mbox{or} \ \ \ (f\circ g)' = (f'\circ g) \cdot g'
			\end{equation}
			Quotient Rule
			\begin{equation}
				\left[\frac{f(x)}{g(x)}\right]'=\frac{g(x)f'(x)-f(x)g'(x)}{[g(x)]^2}.\
			\end{equation}
		\subsubsection{Common Derivatives}
			\begin{equation}
				\frac{d}{{dx}}\left(a^x\right)=a^x\ln(a)
			\end{equation}
			\begin{equation}
				\frac{d}{{dx}}\ln \left( x \right) = \frac{1}{x}, \ \ \ x > 0
			\end{equation}
			Power rule
			\begin{equation}
				\frac{d}{{dx}}x^n = nx^{\left( {n - 1} \right)}
	       \end{equation}
			\begin{equation}
				\frac{d}{{dx}}\cos x = -\sin x
			\end{equation}
			\begin{equation}
				\frac{d}{{dx}}\sin x = \cos x
			\end{equation}
	\subsection{Integrals}
		\subsubsection{Fundamental Theorem of Calculus}
			\begin{equation}
			\int\limits_a^b {\frac{d}{{dx}}F\left( x \right)dx} = F\left( b \right) - F\left( a \right)
			\end{equation}
		\subsubsection{Properties}
			\begin{equation}
				\int_{}^{}k dx = kx + C
			\end{equation}
		\subsubsection{Common Integrals}
			\begin{equation}
				\int_{}^{}k dx = kx + C
			\end{equation}
			\begin{equation}
				\int_{}^{}x^n dx = \frac{1}{n+1} x^{n+1} + C, \ \ \ \mbox{n $\neq$ -1}
			\end{equation}
			\begin{equation}
				\int_{}^{}\frac{1}{x} dx = \ln|x| + C
			\end{equation}
			\begin{equation}
				\int_{}^{}\ln u = u\ln(u)-u+C
			\end{equation}
			\begin{equation}
				\int_{}^{}e^x dx = e^x+C
			\end{equation}
			\begin{equation}
				\int_{}^{}\sin ax = -\frac{1}{a}\cos ax+C
			\end{equation}
			\begin{equation}
				\int_{}^{}\cos x = \sin x+C
			\end{equation}
			Per partes
			\begin{equation}
				\int {u\frac{{dv}}{{dx}}} dx = uv - \int {\frac{{du}}{{dx}}} vdx
			\end{equation}
			Substitution Rule
			\begin{equation}
				\int {f(u)\frac{du}{dx}} dx = f(u)du
			\end{equation}

	\subsection{Laplace transforms}
		\subsubsection{Definition}
			\begin{equation}
				X(s) = \int\limits_0^\infty {x(t)e^{ - st} dt}
			\end{equation}
		\subsubsection{Properties}
			\begin{equation}
				1 \Leftrightarrow \frac{1}{{s}}
			\end{equation}
			Kroeneker delta function
			\begin{equation}
				\delta (t) \Leftrightarrow 1
			\end{equation}
			\begin{equation}
				Ke^{ - at} u(t) \Leftrightarrow \frac{K}{{s + a}}
			\end{equation}
			\begin{equation}
				t^n u(t) \Leftrightarrow \frac{{n!}}{{s^{n + 1} }}
			\end{equation}
			\begin{equation}
				\sin (\alpha t)u(t) \Leftrightarrow \frac{\alpha }{{(s^2 + \alpha ^2 )}}
			\end{equation}
			\begin{equation}
				\cos (\alpha t)u(t) \Leftrightarrow \frac{s}{{(s^2 + \alpha ^2 )}}
			\end{equation}
			\begin{equation}
			e^{ - at} \sin (\Omega t)u(t) \Leftrightarrow \frac{\Omega }{{(s + a)^2 + \Omega ^2 }}
			\end{equation}
			\begin{equation}
				e^{ - at} \cos (\Omega t)u(t) \Leftrightarrow \frac{{s + a}}{{(s + a)^2 + \Omega ^2 }}
			\end{equation}

			\begin{equation}
				e^{at} x(t) \Leftrightarrow X(s-a)
			\end{equation}

			Time domain scaling
			\begin{equation}
				x(at)u(t) \Leftrightarrow \frac{1}{a}X\left( {\frac{s}{a}} \right)
			\end{equation}
			Time domain shifting
			\begin{equation}
				x(t - a)u(t - a) \Leftrightarrow e^{ - as} X(s + a)
			\end{equation}
			Derivative
			\begin{equation}
				\frac{{d^n x(t)}}{{dt^n }} \Leftrightarrow s^n X(s) \ \ or \ \ \mathcal{L}[\dot{x}] = sX(s)-x(0+)
			\end{equation}
			Integral
			\begin{equation}
				\int{x(t)dt} \Leftrightarrow \frac{X(s)}{s}
			\end{equation}

			Convolution
			\begin{equation}
			\int\limits_0^\infty {x_1 (\tau )x_2 (t - \tau )d\tau } \Leftrightarrow X_1 (s)X_2 (s)
			\end{equation}
\section{Greek letters}

\bgroup
\def\arraystretch{2}
\begin{table}[hbp]
  \centering
  \begin{tabular}{cccc}
  \hline
  	$\alpha A$				& Alpha		& $\nu N$		& Nu	\\
  	$\beta B$				& Beta		& $\xi \Xi$	&  Xi 		\\
  	$\gamma \Gamma$			& Gamma		& $o O$	& Omicron		\\
	$\delta \Delta$			& Delta 	& $\pi \Pi$	& Pi		\\
	$\epsilon \varepsilon $	& Epsilon	& $\rho\varrho P$ & Rho   \\
	$\zeta Z$				& Zeta		& $\sigma \Sigma$& Sigma  	\\
	$\eta H$				& Eta		& $\tau T$	& Tau  		\\
	$\theta \vartheta \Theta$	& Theta & $\upsilon \Upsilon$ & Upsilon  \\
	$\iota I$				& Iota		& $\phi \varphi \Phi$ & Phi    \\
	$\kappa K$				& Kappa		& $\chi X$	& Chi  		\\
	$\lambda \Lambda$		& Lambda	& $\psi \Psi$	& Psi  	\\
	$\mu M$					& Mu 		& $\omega \Omega$ & Omega   \\

  \hline
  \end{tabular}
  \caption{Greek letters}
  \label{tab:symbol-variants}
\end{table}
\egroup

\end{document}


%\input{../macros.tex}
%\begin{document}
\begin{answer}

\subsection*{Intro}
We seek to maximise $like(\mu_0, \mu_1, \Sigma, \phi) = p(x, y | \mu_0, \mu_1, \Sigma, \phi) = \Pi p(y_i | \phi) p(x_i | \mu_0, \mu_1, \Sigma, y_i)$, using simple formula $p(x_i, y_i) = p(y_i)p(x_i | y_i)$, and that  $p(y|\phi, \mu_0, ...) = p(y | \phi)$. Then we can separately maximise $\Pi p(y_i | \phi)$ and $\Pi p(x_i | \mu_0, \mu_1, \Sigma, y_i))$, denoting each (not sure if this is standard) $like(\phi), like(\mu_0, \mu_1, \Sigma)$.

\subsection*{MLE of $\phi$}
$like(\phi) = p(y | \phi) = \Pi_{i=1}^n \phi^{y_i} (1-\phi)^{1- y_i}$, where $y_i = 0,1$. Then to maximise log likelihood, $l(\phi) = \sum y_i \log(\phi) + (1 - y_i) \log(1 - \phi)$ we seek $\frac{d l(\phi)}{d \phi} = 0$, i.e. $ \sum \frac{y_i}{\phi} + \frac{y_i - 1}{1 - \phi} = 0 = \sum \frac{y_i - \phi}{\phi (1- \phi)}$ which coincides with $\phi = \bar{y}$.

\subsection*{MLE of $\mu_0, \mu_1$}
$like(\mu_0, \mu_1, \Sigma) = \Pi p(x_i | \mu_0, \mu_1, \Sigma, y_i))$, so we'll maximise over log likelihood i.e. over
\begin{equation*} -\frac{nd}{2} \log(2\pi) - \frac{n}{2}\log(|\Sigma|) -\frac{1}{2} \Sigma_{y_i = 0} (x - \mu_0)^T \Sigma^{-1} (x - \mu_0) -\frac{1}{2} \Sigma_{y_i = 1} (x - \mu_1)^T \Sigma^{-1} (x - \mu_1)
\end{equation*}
Then the derivative of the log likelihood with respect to $\mu_0$ is $\Sigma_{y_i = 0} \Sigma^{-1}(x - \mu_0)$ i.e. is maximised with $\mu_0 = \bar x \text{ averaged over } x_i : y_i = 0$, and similarly for $\mu_1$.

\subsection*{MLE of $\Sigma$}
To maximise over $\Sigma$ seems pretty involved. We'll require a formula for $\frac{\partial}{\partial A_{ij}} \det(A)$. We'll use the matrix minors formulas: $\delta_{ik} |A| = \sum_{j=1}^n |A_{ji}| A_{jk}$ Where $A_{ji}$ is the minor matrix, and $|A_{ji}|$ is its determinant. This gives a form of the inverse for $A$ of $K_{ij} = \frac{(-1)^{i+j}}{|A|} |A_{ji}|$ as this gives $K_{ij} A_{jk} = \sum_{j=1}^n \frac{(-1)^{i+j}}{|A|} |A_{ji}| A_{jk} = \delta_{ij}$. Then $\frac{\partial}{\partial A_{ab}} |A| = \frac{\partial}{\partial A_{ab}} \left( \sum_{j=1}^n (-1)^{b+j} |A_{jb}| A_{jb} \right)$. The sum only has one term with a $A_{ab}$ component, where $j=a$, so we get $\frac{\partial}{\partial A_{ab}} |A| = \frac{\partial}{\partial A_{ab}} (-1)^{a+b} |A_{ab}| A_{ab} = (-1)^{a+b} |A_{ab}| = A^{-1}_{ba} |A|$ (by using the minor form of matrix inverse). This gives us the formula $\nabla_A |A| = |A| (A^{-1})^T$.

So seeking a solution of

\begin{equation*}
\nabla_A \left(
	\frac{n}{2}\log(|A|) -\frac{1}{2} \Sigma_{y_i = 0} (x - \mu_0)^T A (x - \mu_0) -\frac{1}{2} \Sigma_{y_i = 1} (x - \mu_1)^T A (x - \mu_1)
\right) = 0
\end{equation*}

where we let $A = \Sigma^{-1}$ and use $|\Sigma| = 1/|A|$, as well as assuming $\mu_0, \mu_1$ are already maximised (i.e. $\mu_0 = \hat{\mu_0}_{MLE}$ etc.). Then we get $\frac{n}{2} A^{-1} - \frac{1}{2} \Sigma_{i=1}^n (x - \mu_{y_i})(x - \mu_{y_i})^T = 0$ which gives the desired MLE equation, where we used that $\Sigma$, and so $A$ too, are symmetric so $(A^{-1})^T = A^{-1}$, and that for any matrix $A$, $\frac{\partial}{\partial A} z^T A z = z z^T$.

\end{answer}
%\end{document}


\section{Parametric smoother}
These are what you have seen already. We force a function defined by
``few'' parameters on the data and use something like least squares to
find the ``best'' estimates for the parameters.

For example, a regression line computed with least squares can be
thought of as a smoother. In this case $S[\by|\bx](x_0) =
(1 \, x_0) \, (\bX'\bX)^{-1}\bX'\by$ with $\bX$ a design matrix containing a
column of 1's and $\bx$ ({\tt cbind(1,x)}).

The lack of flexibility of these types of smoother can make them
provide misleading results.

\begin{figure}[htp]
\caption{\label{f2.2} CD4 cell count since seroconversion for HIV infected men.}
\centerline{\epsfig{figure=Plots/plot-02-02.ps,angle=270,width=.8\textwidth}}
\end{figure}


\appendix
\chapter{Analytical Modelling of Buckled Beam Mechanism}\label{chap:appendixA}
The bistable mechanism consists of an initially flat beam which, when compressed by a distance $\Delta l$, buckles and forms a structure that exists in two stable positions. In the case of the actuator, which consists of a pair of SMA coils and the buckled beam itself, is considered to require an input torque $M_\mathrm{in}$ at the input pivot to switch between its two stable states.

As the entire kinematic stage is comprised of flexure-based mechanisms, as shown in \cref{fig:bistable-mechanism}, the pivots that support the buckled beam present with an inherent angular stiffness, $K_\mathrm{in}$ and $K_\mathrm{out}$ at the input and output pivot, respectively. The buckled beam is considered to have a flexural rigidity of $EI$ and an initial length before compression of $L$. The distance between the centre of the flexural pivots and the beam is considered to be offset by a distance of $p$, as shown in \cref{fig:buckled-beam-schematic}.

Based on the hypothesis described in the work by \cite{tivot2021} and the Euler-Bernoulli beam theory, the beam deflection can be described using the following equation:
\begin{equation}\label{eq:deflection_A}
  y(x) = \left(A\sin{kx}+B(\cos{kx}-1)+C\frac{x}{l}\right) l {\theta }_\textrm{in}
\end{equation}
with $k=\sqrt{P/(EI)}$ and the boundary conditions of the supported beam are as follows
\[y(0)=0\]
\[y'(0)\cong{\theta }_\textrm{out}\]
\[M_0\cong K_\textrm{out}\theta_\textrm{out}+Vp-Pp\theta_\textrm{out}\]
\[y(l)\cong -p(\theta_\textrm{out}+\theta_\textrm{in})\]
\[y'(l)\cong \theta_\textrm{in}\]
Furthermore, the deflection parameters of \cref{eq:deflection_A} are given by
\begin{equation}\label{A_norm}
A =    \frac{(1+2\overline{p})kl+{\varepsilon }_0\left(\overline{p}  \sin{kl}-\frac{\cos{kl}-1}{kl}\right)}
{kl\left( kl \cos{kl}-\sin{kl}-\left({\overline{p}}^2+\overline{p}\right){(kl)}^2\sin{kl}+{\varepsilon }_0\left(\sin{kl}+2\frac{\cos{kl}-1}{kl}\right)\right)}
\end{equation}
\begin{equation} \label{B_norm}
B = \frac{  \overline{p}(1+2\overline{p}){(kl)}^2+{\varepsilon }_0 \left(\overline{p} \left(\cos{kl}-1\right) + \frac{\sin{kl}}{kl} -1\right)}
{kl\left( kl \cos{kl}-\sin{kl}-\left({\overline{p}}^2+\overline{p}\right){(kl)}^2\sin{kl}+{\varepsilon }_0\left(\sin{kl}+2\frac{\cos{kl}-1}{kl}\right)\right)}
\end{equation}
\begin{equation} \label{C_norm}
C = \frac{  {\overline{p}}^2{(kl)}^2\sin{kl} -2\overline{p} kl\cos{kl} -\sin{kl} - {\varepsilon }_0\left(\overline{p}\sin{kl}-\frac{\cos{kl}-1}{kl}\right)}
{ kl \cos{kl}-\sin{kl}-\left({\overline{p}}^2+\overline{p}\right){(kl)}^2\sin{kl}+{\varepsilon }_0\left(\sin{kl}+2\frac{\cos{kl}-1}{kl}\right)}
\end{equation}
where $\overline{p} = p/l $ and $\varepsilon_0=K_\textrm{out}/(EI/l)$. When considering the beam’s arc length as constant, the end-shorting, $\Delta l$, can be approximated using the following expression
\begin{equation}\label{eq:delta_l}
 \Delta l\cong \frac{p}{2}({\theta }^2_\textrm{in}+\theta ^2_\textrm{out})+\int^l_0{\frac{y'(x)^2}{2}dx}=H l{\theta }^2_\textrm{in}
\end{equation}
Here, the coefficient $H$ is expressed as
\begin{multline}\label{eq:H-smabb}
 H = \frac{\left({A}^2+{B}^2\right){\left(kl\right)}^2}{4} + \frac{\left({A}^2-{B}^2\right)kl\sin{2kl} }{8} + \frac{AB kl\left(\cos{2kl}-1\right)}{4}\\
 +AC\sin{kl} + BC\left(\cos{kl}-1\right) + \frac{{C}^2}{2} +\frac{\overline{p}}{2}\left({\left( Akl+C\right)}^2+1\right)
\end{multline}

By rearranging \cref{eq:delta_l}, the input angle can be expressed as
\begin{equation}\label{eq:theta_in_A}
 {\theta}_\textrm{in}=\pm \sqrt{\frac{\Delta l}{l}}\sqrt{\frac{1}{H}}
\end{equation}

Finally, the input moment can be described as the following \cref{eq:M_in_A}
\begin{equation}
\begin{split}
 M_\textrm{in} &\cong M_l+K_\textrm{in}{\theta }_\textrm{in}+Vp-Pp{\theta}_\textrm{in}\\
 &=\frac{EI}{l}\left({\left(kl\right)}^2\left(\overline{p}\left(C-1\right)-A\sin{kl}-B\cos{kl}\right)+{\varepsilon }_l\right){\theta}_\textrm{in}
 \label{eq:M_in_A}
\end{split}
\end{equation}
\noindent{where ${\varepsilon }_l=K_\textrm{in}/(EI/l)$.}
These equations, as developed by Loic Tissot-Daguette, are used to obtain the moment and angular stroke requirements of the bistable element when sizing the SMA elements for the bistable gripper.


\section{Other Notes}

\subsection{Convergence of Value Iteration}

let $T^\pi(v) = R^\pi + \gamma P^\pi v$

\begin{equation}
	\begin{aligned}
		||T^\pi(u) - T^\pi(v)||_{\infty} &= ||(R^\pi + \gamma P^\pi u) - (R^\pi + \gamma P^\pi v) ||_\infty \\
		&= ||\gamma P^\pi (u-v) ||_\infty \\
		&\leq || \gamma P^\pi ||u-v ||_\infty ||_\infty \\
		&\leq \gamma ||u-v||_\infty
	\end{aligned}
\end{equation}

So $T$ will converge to fixed point at linear rate $\gamma$ according to \emph{contraction mapping theorem}.


\section{Computational Complexities of the Algorithms}

\lettrine[nindent=0em,lines=3]{I}n this chapter we will briefly discuss each one of the algorithms intended to be used in the development of the Intelligent HVAC System. Along with the description of such algorithms we will present a very brief analysis on their computational complexity.

\subsection{Artificial Neural Network}

Artificial Neural Networks (ANNs) are computing systems inspired by the biological networks that constitute the brain. Such systems learn (progressively improve performance) to do tasks by considering examples, generally without task-specific programming.

An ANN is based on a collection of connected units called artificial neurons, (analogous to axons in a biological brain). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving (postsynaptic) neuron can process the signal(s) and then signal downstream neurons connected to it. Neurons may have state, generally represented by real numbers, typically between 0 and 1. Neurons and synapses may also have a weight that varies as learning proceeds, which can increase or decrease the strength of the signal that it sends downstream. Further, they may have a threshold such that only if the aggregate signal is below (or above) that level is the downstream signal sent. Typically, neurons are organized in layers. Different layers may perform different kinds of transformations on their inputs. Signals travel from the first (input), to the last (output) layer, possibly after traversing the layers multiple times. \cite{ann_wikipedia}

The multilayer perceptron is an artificial neural network structure and is a nonparametric estimator that can be used for classification and regression. It is a supervised learning algorithm that learns a function $f(\cdot): R^m \rightarrow R^o$ by training on a dataset, where $m$ is the number of dimensions for input and $o$ is the number of dimensions for output. Given a set of features $X = {x_1, x_2, ..., x_m}$ and a target $y$, it can learn a non-linear function approximator for either classification or regression. It is different from logistic regression, in that between the input and the output layer, there can be one or more non-linear layers, called hidden layers. Figure \ref{Fig:mlp_example} shows a one hidden layer MLP with scalar output. \cite{ann_scikit}

\begin{figure}[H]
\centering \includegraphics[width = 60mm, height = 60mm]{imgs/multilayerperceptron_network.png}
\caption{Multilayer perceptron}
\label{Fig:mlp_example}
\end{figure}

The leftmost layer, known as the input layer, consists of a set of neurons $\{x_i | x_1, x_2, ..., x_m\}$ representing the input features. Each neuron in the hidden layer transforms the values from the previous layer with a weighted linear summation $w_1x_1 + w_2x_2 + ... + w_mx_m$, followed by a non-linear activation function $g(\cdot):R \rightarrow R$ - like the hyperbolic tan function. The output layer receives the values from the last hidden layer and transforms them into output values.

The advantages of multilayer perceptrons are:
 \begin{itemize}
 \item Capability to learn non-linear models
 \item Capability to learn models in real-time (on-line learning)
 \end{itemize}

 The disadvantages of multilayer perceptron (MLP) include:
 \begin{itemize}
 \item MLP with hidden layers have non-convex cost function where there exists more than one local minimum. Therefore different random weight initializations can lead to different validation accuracy.
 \item MLP requires the tuning number of of hyperparameters such as the \textit{number of hidden neurons, layers and iterations}.
 \item MLP is sensitive to feature scaling
 \end{itemize}

 \subsubsection{Computational Complexity}

Suppose there are $n$ training samples, $m$ features, $k$ hidden layers, each containing $h$ neurons - for simplicity, and $o$ output neurons. The time complexity of the feedforward-backpropagation algorithm is

\begin{equation}
O(n\cdot m \cdot h^k \cdot o)
\label{eq:ann_complexity1}
\end{equation}

Also for the sake of simplicity lets assume that $m>h>o$, hence the time complexity for feedforward-backpropgation algorithm is

\begin{equation}
O(n\cdot m^{k+2})
\label{eq:ann_complexity2}
\end{equation}

Additional to the computation of the gradients using the backpropagation algorithm we need to consider the complexity (convergence ratio) of the algorithm used for solving the minimization problem inherent to the neural network, recall that at each iteration a minimization problem is solved. For the best case, this is superlinear (L-BFGS) and for the worst case it is linear (Stochastic Gradient Descent).

\subsection{Anomaly Detection}
\label{sec:anomaly_detection_alg}

Assume we have a sample $\mathbb{M}$ drawn from the same distribution. An outlier, novelty, or anomaly is an instance that is very much different from other instances in the sample. An outlier may indicate an abnormal behavior of the system; for example, in a dataset of credit card transactions, it may indicate fraud; in an image, outliers may indicate anomalies, for example, tumors. \cite{intro_to_machine_learning}

Outlier detection is not generally cast as a supervised, two-class clas- sification problem of seperating typical instances and outliers, because generally there are very few instances that can be labeled as outliers and they do not fit a consistent pattern that can be easily captured by a two- class classifier. Instead, it is the typical instances that are modeled; this is sometimes called one-class classification. Once we model the typical instances, any instance that does not fit the model (and this may occur in many different ways) is an anomaly.

Outlier detection basically implies spotting what does not normally happen; that is, it is density estimation followed by checking for in- stances with too small probability under the estimated density. As usual, the fitted model can be parametric, semiparametric, or nonparametric. In the parametric case, for example, we can fit a Gaussian to the whole data and any instance having a low probability, or equally, with high Mahalanobis distance to the mean, is a candidate for being an outlier. \cite{anomaly_scikit}

Figure \ref{Fig:noveltyDetection_example} shows how novelty detection can separate data into regular observations and abnormal observations.

\begin{figure}[H]
\centering \includegraphics[width = 90mm, height = 40mm]{imgs/noveltyDetection.png}
\caption{Novelty detection}
\label{Fig:noveltyDetection_example}
\end{figure}

One common way of performing outlier detection is to assume that the regular data come from a known distribution (e.g. data are Gaussian distributed). From this assumption, we generally try to define the “shape” of the data, and can define outlying observations as observations which stand far enough from the fit shape. The process for defining the "shape" of the data is as follows and assumes that all of the samples come from a known distribution, say Gaussian for the purposes of this explanation.

Assume a training set $T = \{x^{(1)}, \cdots, x^(m) \}$ where $x \in \mathbb{R}^n$. Suppose that each feature $x_i from x$ is distributed according to some Gaussian distribution $x_i \sim \mathcal{N}(\mu_i, \sigma_i^2)$. Hence $p(x) = \prod_{j = 1}^n p(x_j; \mu_j, \sigma_j^2)$. The task is now to find the corresponding means $\mu_i$ for $i = 1,\cdots,n$ and standard deviations $\sigma_i^2$ for  $i = 1,\cdots,n$ of the $n$ different distributions. This can be easily done for the Gaussian distribution as follows

\begin{equation}
\mu_j = \frac{1}{m} \sum_{i=1}^{m} x_j^{(i)}
\label{eq:mean_eq_gaussian}
\end{equation}

and

\begin{equation}
\sigma_j^2 = \frac{1}{m} \sum_{i=1}^{m} (x_j^{(i)} - \mu_j)^2
\label{eq:std_eq_gaussian}
\end{equation}

We can now describe the simplest anomaly detection algorithm

\begin{enumerate}
\item Choose features $x_i$ that might be indicative/sensitive of/to anomalies.
\item Fit parameters $\mu_1, \cdots, \mu_n$ and $\sigma_1^2,\cdots, \sigma_n^2$ using equations \eqref{eq:mean_eq_gaussian} and \eqref{eq:std_eq_gaussian} respectively.
\item Given a new example $x$, compute $p(x) = \prod_{j = 1}^n p(x_j; \mu_j, \sigma_j^2)$
\item Tag the new sample $x$ as an anomaly if $p(x) < \epsilon$, where $epsilon$ is a hyperparameter defined by the user.
\end{enumerate}

 \subsubsection{Computational Complexity}

Usually the complexity of this algorithm is negligible compared to the complexity of the feedforward-backpropagation algorithm. As can be seen from the Algorithm in Section \ref{sec:anomaly_detection_alg} the asymptotic complexity of such algorithm is $O(n)$ where $n$ is the number of choosen features.