\chapter[ICA on the Sphere]{Independent Component Analysis on the Sphere}
\label{ch_mrs_ica}

\section{Introduction}
\index{blind source separation}
\index{ICA}

Blind Source Separation (BSS) is a problem that occurs in multi-dimensional data processing. The overall goal is to recover
unobserved signals, images or \emph{sources} $S$ from mixtures of these sources $X$ observed typically at the output of an
array of sensors. The simplest mixture model takes the form:
\begin{equation}\label{model0}
X = A S
\end{equation}
where $X$ and $S$ are random vectors of respective sizes $m \times 1$, $n \times 1$ and $A$ is an $m \times n$ matrix. The
entries of $S$ are assumed to be independent random variables. Multiplying $S$ by $A$ linearly mixes the $n$ sources into
$m$ observed processes. \\

Independent Component Analysis methods were developed to solve the BSS problem, \emph{i.e.} given a batch of $T$ observed
samples of $X$, estimate the mixing matrix $A$ and reconstruct the corresponding $T$ samples of the source vector $S$, relying
mostly on the statistical independence of the source processes. Note that with the above model, the independent sources can
only be recovered up to a multiplication by a \emph{non-mixing} matrix \emph{i.e.} up to a permutation and a scaling of the
entries of $S$. Although independence is a strong assumption, it is in many cases physically plausible. The point is that
it goes beyond the simple second order decorrelation obtained for instance using Principal Component Analysis (PCA) : decorrelation
is not enough to recover the source processes since any rotation of a white random vector remains a white random vector.\\

Algorithms for blind component separation and mixing matrix estimation depend on the model used for the probability distribution
of the sources~\cite{ica:3easy}. In a first set of techniques, source separation is achieved in a noise-less setting, based on the
non-Gaussianity of all but possibly one of the components. Most mainstream ICA techniques belong to this category : JADE~\cite{ica:jade},
FastICA, Infomax~\cite{ica:icabook}. In a second set of blind techniques, the components are modeled as Gaussian processes, either
stationary or non stationary and, in a given representation, separation requires that the sources have diverse, \emph{i.e.} non
proportional, variance profiles. The Spectral Matching ICA method (SMICA) ~\cite{ica:Del2003}, considers in this sense the case of
mixed stationary Gaussian components and goes further than the above model (Eq.~\ref{model0}) by taking into account additive
\emph{instrumental } noise $N$:
\begin{equation}\label{model1}
X = A S + N
\end{equation}
Moving to a Fourier representation, the idea is that colored components can be separated based on the diversity of their power spectra.\\

The next two sections give a short overview of two significant ICA methods mentioned above and implemented in the \mrs package:
JADE and SMICA. A shorter description of FastICA ~\cite{ica:icabook} is also given. This is followed by a description of ways to
combine wavelets and ICA techniques. Some useful properties of wavelet transforms can indeed come enhance the performance of ICA
methods in several situations.

\section{JADE}
\index{ICA!jade}

The Joint Approximate Diagonalization of Eigenmatrices method (JADE) assumes the observed data $X$ follows the noiseless mixture
model~(\ref{model0}) where the independent sources $S$ are non-Gaussian \emph{i.i.d.}\footnote{The letters \emph{i.i.d.} stand for
independently and identically distributed meaning that each entries of $X$ at a given time $t$ are independent of $X$ at any other
time $t'$ and that the distribution of $X$ does not depend on time. } random processes. The mixing matrix is assumed to be square
and invertible so that (de)mixing is actually just a change of basis.

As mentioned above, second order statistics do not retain enough information for source separation in this context: finding a change of
basis in which the data covariance matrix is diagonal will not in general enable to identify the independent sources properly. Nevertheless,
decorrelation is \emph{half the job}~\cite{ica:tutorial} and one may seek the basis in which the data is represented by maximally independent
processes among those bases in which the data is decorrelated. This leads to so-called orthogonal algorithms: after a proper whitening of
the data by multiplication with the inverse of a square root of the covariance matrix of the data $W$, one is then seeking a rotation $R$
(which leaves things white) so that $\hat{ S}$ defined by
\begin{equation}
\hat{ S} = W^{-1} \, Y =  W^{-1}\, R \, X_{\textrm{white}}  = W^{-1}\, R \, W \, X
\end{equation}
and $\hat{B} = \widehat{A^{-1}} =  W^{-1}\, R \, W$ are estimations of the sources and of the inverse of the mixing matrix.\\

JADE is such an orthogonal ICA method and, like most mainstream ICA techniques, it exploits higher order statistics so as to achieve some
sort of \emph{ non linear decorrelation}. Precisely, in the case of JADE, statistical independence is assessed using fourth order cross cumulants :
\begin{eqnarray}  \nonumber
F_{ijkl} & = & \textrm{cum}( y_i, y_j, y_k, y_l )   \nonumber    \\
  & =& \mathcal{E} (y_i y_j y_k y_l) - \mathcal{E} (y_i y_j)\mathcal{E} (y_k y_l)\nonumber \\
  & & -\mathcal{E} (y_iy_l)\mathcal{E} ( y_j y_k)-\mathcal{E} (y_iy_k)\mathcal{E} (y_j y_k)
\end{eqnarray}
where $\mathcal{E}$ stands for statistical expectation and the $y_i$'s are the entries of vector $Y$ modeled as random variables,
and the correct change of basis (\emph{i. e.} rotation) is found by somehow \emph{diagonalizing} the fourth order cumulant tensor.
Indeed, if the $y_i$'s were independent, all the cumulants with at least two different indices would be zero. As a consequence of
the independence assumption of the source processes $S$ and of the \emph{whiteness} of $Y$ for all rotations $R$, the fourth order
tensor $F$ is well structured: JADE was precisely devised to take advantage of the algebraic properties of $F$. JADE's objective
function is given by
\begin{eqnarray}  \nonumber
%\mathcal{J}_{\textrm{jade}}( R )   &=& \sum _{ijkl \ne ijkk}  \textrm{cum}(  y_i, y_j, y_k, y_l )^2  \nonumber    \\
  \mathcal{J}_{\textrm{jade}}( R ) & =&  \sum _{ij}   \sum_{k \ne l} \textrm{cum}(  y_i, y_j, y_k, y_l )^2
\end{eqnarray}
which can be interpreted as a joint diagonalization criterion. Fast and robust algorithms are available for the minimization
of $\mathcal{J}_{\textrm{jade}}( R )$ with respect to $R$ based on Jacobi's method for matrix diagonalization~\cite{ica:pham2001}.
More details on JADE can be found in~\cite{ica:jade,ica:tutorial,ica:icabook}.

\subsubsection{JADE for spherical maps}

Applying JADE on multichannel data mapped to the sphere does not require any particular modification of the algorithm. Indeed, JADE estimates
the fourth order cumulant tensor from the available data samples assuming an \emph{i.i.d.} random field. Hence, given a pixelization scheme on
the sphere such as provided by the Healpix package, JADE can be directly applied to the multichannel spherical data pixels.

\section{FastICA}
\index{ICA!fastica}

FastICA is by now a standard technique in ICA. Like JADE, it is meant for the analysis of mixtures of independent non-Gaussian sources in
a noise-less setting. A complete description of this method can be found in \cite{ica:icabook} and references therein. Many papers on this
algorithm are available at \emph{http://www.cs.helsinki.fi/ u/ahyvarin/papers/fastica.shtml}. We give here a brief and simplified account
of the algorithm. FastICA, again like JADE, is a so-called orthogonal ICA method: the independent components are sought by maximizing a
measure of non-Gaussianity under the constraint that they are decorrelated. Intuitively, one should understand that mixtures of independent
non-Gaussian random variables tend to \emph{ look more Gaussian}. An enlightening view on the relation between mutual information, which is
a natural measure of independence, decorrelation and non-Gaussianity can be found in~\cite{ica:3easy,ica:geomindep}. Non-Gaussianity is assessed
in FastICA using a contrast function $G$ based on a non-linear approximation to \emph{negentropy}~\cite{ica:icabook}. In practice, depending
on the application, different approximations or non-linear (non-quadratic) functions should be experimented with. In a simple deflation scheme,
for sphered data, the directions are found sequentially : a direction $r$ of maximal non-Gaussianity is sought by maximizing
\begin{equation}
J_G(r) = \Big( \mathcal{E} \{ G(r^T x_{\textrm{white}}  ) \} - \mathcal{E} \{ G(\nu ) \} \Big)^2
\end{equation}
where $\nu$ stands for centered unit variance Gaussian variable, under the constraint that $r$ has unit norm and that $r$ is orthogonal
to the directions found previously. The contrast function $G$ can for instance be chosen among the following~\cite{ica:icabook}:
\begin{eqnarray}
G_0 (u)  & = &  \frac{1}{a} \textrm{log}\,\textrm{cosh} (a u ) \nonumber    \\
G_1 (u)  & = &  -\frac{1}{a} \textrm{exp}(- a u^2 / 2 )   \nonumber	   \\
G_2 (u)  & = &  \frac{1}{4} u^4 \nonumber    \\
\end{eqnarray}
where $a$ is a constant to be determined depending on the application. It can be shown that the maxima of $J_G$ occur at certain maxima
of $\mathcal{E} \{ G(r^T x_{\textrm{white}} ) \} $. These are obtained for $r$ solution to :
\begin{equation}
\mathcal{E} \{ x_{\textrm{white}} g(r^T x_{\textrm{white}}  ) \} - \lambda r = 0
\end{equation}
where $\lambda$ is a constant easily expressed in terms of the optimal direction $r_0$, and $g$ is the derivative of $G$. Solving this
equation using Newton's method, and a few approximations, a \emph{fixed-point} algorithm is derived which consists in repeating the
following two steps until convergence :
\begin{eqnarray}
r  & \leftarrow & \mathcal{E} \{ x_{\textrm{white}} g(r^T x_{\textrm{white}}  ) \} - \mathcal{E} \{ g'(r^T x_{\textrm{white}}  ) \} r     \nonumber  \\
r  & \leftarrow  &  \frac{r}{\| r \|}   \nonumber	   \\
\end{eqnarray}
A simple implementation of this algorithm is included in the present package. It is largely based on the $\textbf{Matlab}^{TM}$ code
available at \emph{www.cis.hut.fi/projects/ica/fastica/}.

%\section{SMICA}
%\index{smica}
%\index{ICA!smica}
%
%Spectral Matching ICA (SMICA) was designed to address some of the general problems raised by Cosmic Microwave Background data analysis
%where the major component of interest (CMB itself) is well modeled by an isotropic stationary Gaussian random field. Although standard
%ICA methods may be used in this context, they are not expected to perform as well as methods based on Gaussian model especially in the
%presence of additive Gaussian instrumental noise as in (\ref{model1}).
%
%SMICA belongs to a set of blind source separation techniques where the components are modeled in a given representation as \emph{locally i.i.d.}
%centered Gaussian processes. Independent Gaussian sources can then be separated based on their statistical independence (which obviously reduces
%\emph{locally} to decorrelation) provided they have diverse (\emph{i.e.} non proportional) variance profiles \emph{i.e.} energy distributions in
%that representation. SMICA considers in this sense the case of mixed stationary Gaussian components in a noisy context as in model~(\ref{model1}) :
%moving to a Fourier representation, colored components can be separated based on the diversity of their power spectra.
%
%\subsection{SMICA's objective function}
%
%In order to derive the Spectral Matching ICA criterion, we assume that, in the Fourier domain, the \emph{locally i.i.d.} Gaussian sources $S$
%and noise $N$ processes in (\ref{model1}) actually have constant spectral covariance matrices $R_f^S(q) \in \mathbb{R}^{n\times n}$ and $R_f^N(q)
%\in \mathbb{R}^{m \times m} $ on each of a set of $Q$ frequency bands. Clearly, the appropriate notion of frequency should be used, depending on
%whether the data $X$ is a set of time series, a set of 2D maps, etc. The assumption of statistical independence between the components $S$ implies
%that the $R_f^S(q)$ are diagonal matrices. With a similar assumption regarding  the noise processes in the $m$ different channels, the $R_f^N(q)$
%also are diagonal matrices. Applying a Fourier transform on (\ref{model1}) does not affect the mixing matrix $A$, so that the model covariance
%matrix of the observations $X$ in the $q^{\textrm{th}}$ frequency band is structured as
%\begin{equation}
%\label{structure}
%R_f^X(q) = A R_f^S(q) A^{\dagger} +   R_f^N(q)
%\end{equation}
%where we assumed that the instrumental noise is independent of the sources. Then, provided estimates $\widehat{R}_f^X(q) $ of $R_f^X(q) $ can be
%obtained from the available data (\emph{e.g.} empirical covariance estimator), SMICA consists in minimizing
%\begin{equation}\label{Cost_fourier}
% \Phi_f (\theta) =  \sum _{q=1}^{Q}  \alpha_q \mathcal{D} \left( \widehat{R}_f^X(q) \ , A R_f^S(q) A^{\dagger} +   R_f^N(q) \right)
%\end{equation}
%for some sensible choice of the weights $\alpha_q$ and of the matrix mismatch measure $\mathcal{D}$, with respect to the full set of parameters
%$\theta = (A,R_f^S(q), R_f^N(q) )$ or a subset thereof. As discussed in ~\cite{ica:Del2003}, a good choice for $\mathcal{D}$ is
%\begin{equation}\label{eq:kl}
%  \mathcal{D}_{KL} (R_1, R_2 )  =  \frac{1}{2} \Big( \mathrm{tr} (R_1R_2^{-1}) - \log\det (R_1R_2^{-1}) - m  \Big)
%\end{equation}
%which is the Kullback-Leibler divergence between two $m$-variate zero-mean Gaussian distributions with covariance matrices $R_1$ and $R_2$.
%With this mismatch measure, the SMICA criterion is shown to be related to the likelihood of the data in a Gaussian model, so that we can resort
%to the EM algorithm to minimize~(\ref{Cost_fourier}). The weights $\alpha_q$ should be chosen to reflect the variability of the estimate of the
%corresponding covariance matrix. Following the derivation in~\cite{ica:Del2003}, these are taken to be the number of Fourier modes in each band $q$. \\
%
%\subsection{Source map estimation}\label{sect:mapesti}
%
%As a result of applying SMICA, power densities in each frequency band are estimated for the sources and detector noise along with
%the estimated mixing matrix. These may be used in reconstructing the source maps \emph{via} for instance Wiener filtering in each band:
%a Fourier mode $X(\nu)$ in frequency band $q$ is used to reconstruct the maps according to
%\begin{equation}
%  \widehat{S}(\nu)
%  =
%  (\widehat{A}\adj \widehat{R}_f^N(q)^{-1} \widehat{A} + \widehat{R}_f^S(q)^{-1})\inv
%  \widehat{A}\adj \widehat{R}_f^N(q)^{-1} X(\nu)
%  \label{Wiener}
%\end{equation}
%In the limiting case where noise is small compared to signal components, this filter reduces to
%\begin{equation}
% \widehat{S}(\nu)  =
%  (\widehat{A}\adj \widehat{R}_f^N(q)^{-1} \widehat{A} )\inv
%  \widehat{A}\adj \widehat{R}_f^N(q)^{-1} X(\nu)
%  \label{Wiener1}
%\end{equation}
%Clearly, the above Wiener filter is optimal only in front of stationary Gaussian processes. For non Gaussian maps, such as given by
%the Sunyaev Zel'dovich effect, better reconstruction can be expected from non linear methods.
%
%\subsection{SMICA for spherical maps}\label{sect:smicas}
%
%In the linear mixture model~(\ref{model1}), $X$ now stands for an array of observed spherical maps, $S$ is now an array of spherical source maps
%to be recovered and $N$ is an array of spherical noise maps. The mixing matrix $A$ achieves a pixelwise linear mixing of the source maps, in the
%Healpix scheme for instance.
%
%Extending SMICA to deal with multichannel data mapped to the sphere is straightforward~\cite{ica:Del2003}. The idea is simply to substitute the
%spherical harmonics transform to the Fourier transform used in the above description of SMICA. Then, data covariance matrices are estimated in
%this representation over $Q$ intervals in multipole number $l$, assuming the components are stationary and isotropic over the sphere. These $Q$
%covariance matrices are still structured according to~(\ref{structure}) and source separation can be achieved by minimizing the spectral matching
%criterion~(\ref{Cost_fourier}). Source maps reconstruction follows as in section~\ref{sect:mapesti}.\\
%
%SMICA has already been applied in astrophysical data analysis, showing significant success for CMB spectral estimation in multidetector
%experiments~\cite{ica:Del2003,ica:patanchon}. Working in the frequency domain does offer several benefits such as easy handling of detector
%dependent point spread functions. However, the non locality of the Fourier or the spherical harmonics transform will have some undesired
%effects when dealing with non-stationary components or noise, or with incomplete data maps. The latter is a common issue in astrophysical
%data analysis : either the instrument scanned only a fraction of the sky or some regions of the sky were masked due to localized strong
%astrophysical sources of contamination (compact radio-sources or galaxies, strong emitting regions in the galactic plane). A simple way
%to overcome these effects is to move instead to a wavelet representation so as to benefit from the localization property of wavelet filters.
%This leads to WSMICA~\cite{starck:yassir05}, an extension of SMICA which is reviewed in section~\ref{sect:wsmica} below.

\section{ICA and Wavelets}
\index{wavelet!ICA}
\index{ICA!wavelet}

Several properties of wavelets have been recognized as particularly useful in multichannel data processing : bringing wavelets and
independent component analysis together has proven quite profitable. Extensions WJADE and WSMICA of the two ICA methods described
previously are discussed in this section.

Wavelets are remarkable at data compression meaning data that is structured in the initial representation requires fewer significant
coefficients in a wavelet representation. In imprecise and general terms, wavelets grab the coherence between coefficients of the
structured data and produces a smaller set of significant coefficients which are then less coherent and which have a sparser statistical
distribution. Then, the super-Gaussian\footnote{A super-Gaussian distribution is also called a lepto-kurtic distribution, referring to a
distribution with a narrow central peak and heavy tails. A typical example is the Laplacian distribution.} \emph{i.i.d.} statistical model
which appears in most standard ICA methods may better suit the wavelet coefficients of the data than the data samples in the initial representation.

Wavelets have been developed for the analysis of non-stationary and singular data in order to overcome certain difficulties attached to
the Fourier transform. Wavelets are widely used to reveal variations in the spectral content of time series or images as they permit to
single out regions in direct space while retaining localization in the frequency domain. Astrophysical data analysis has much to gain in
avoiding the assumption of stationarity underlying Fourier analysis. Moreover, observed data maps are commonly imperfectly shaped and
incomplete with missing or masked patches due to experimental settings, scanning strategies, etc. This will impair direct application of
the Spectral Matching ICA method described previously. One might consider resorting to wavelets.

%In dealing with non stationary data or incomplete data, an attractive feature of  wavelet filters over the spherical harmonic transform is that they are well localized in the initial representation.

%Dealing with non stationary data or incomplete data is made simpler due to the good localization of wavelet coefficients in the initial representation.In the smaller scales, most of the samples in the filtered signal will  be unaffected by the presence of gaps and only these samples should be used to estimate the data covariance matrices on each wavelet scale.

\subsection{WJADE}\label{sec:wjade}
\index{ICA!wjade}
\index{jade!wavelet}

Wavelets come into play as a sparsifying transform. Applying a wavelet transform on both sides of~(\ref{model0}) does not affect the
mixing matrix and the model structure is preserved. Also, moving the data to a wavelet representation does not affect its information
content. However, the statistical distribution of the data coefficients in the new representation is different: wavelets are known to
lead to sparse \emph{i.i.d.} representations of structured data. Further, the \emph{local} (coefficient wise) signal to noise ratio
depends on the choice of a representation. A wavelet transform tends to grab the informative coherence between pixels while averaging
the noise contributions, thus enhancing structures in the data. Although the standard ICA model~(\ref{model0}) is for a noiseless setting,
the derived methods can be applied to real data. Performance will depend on the detectability of significant coefficients \emph{i.e.} on
the sparsity of the statistical distribution of the coefficients. Moving to a wavelet representation will often lead to more robustness to noise.

Once the data has been transformed to a proper representation (\emph{e.g.} wavelets but also ridgelets and curvelets in the case of strongly
anisotropic 2D or 3D data), WJADE consists in applying the standard JADE method to the new multichannel coefficients. Once the mixing matrix
is estimated, the initial source maps are obtained using the adequate inverse transform after some non linear denoising or thresholding of
the coefficients if necessary.

\subsection{Covariance matching in wavelet space : WSMICA}\label{sect:wsmica}
\index{ICA!wsmica}
\index{smica!wavelet}
\index{wavelet!transform}

Let us consider the case of spherical maps as in \ref{sect:smicas} but possibly incomplete, partly masked or non-stationary. As a model case,
we actually consider incomplete data maps in which the positions of the missing pixels are known in advance. Moving to a wavelet representation,
it is possible to keep track of the missing pixels on each scale so that we can derive a covariance matching ICA criterion, WSMICA, robust to
gaps in the data. Indeed, an attractive feature of wavelet filters over the spherical harmonic transform is that they are well localized in the
initial representation. Provided the wavelet filter response on scale $j$ is short enough compared to data size and gap widths, most of the
samples in the filtered signal will then be unaffected by the presence of gaps. Using exclusively these samples yields an estimated covariance
matrix $\widehat{R}_w^X(j)$ which is not biased by the missing data. The price to pay is a possibly slight increase in variance which depends on scale $j$.

With the Isotropic Undecimated Wavelet Transform on the Sphere (UWTS) decribed in section~\ref{sect_wts}, the multichannel data $X$ is decomposed
into $J$ detail maps $X_{j}^w$ and a smooth approximation map $X_{J+1}^w$ over a dyadic resolution scale which simply sum back as:
\begin{equation}
X(\vartheta, \varphi) = X_{J+1}^w (\vartheta, \varphi) + \sum_{j=1}^{J} X_{j}^w(\vartheta, \varphi)
\end{equation}
Denoting $l_j$ the size of the set $\mathcal{M}_j$ of wavelet coefficients unaffected by the gaps at scale $j$, the wavelet covariances are empirically estimated using
\begin{equation}
\widehat{R}_w^X (j) = \frac{1}{l_j }  \sum_{t \in  \mathcal{M}_j } X_j^w( \vartheta_t, \varphi_t )X_j^w( \vartheta_t, \varphi_t ) ^\dagger
\end{equation}

Clearly, applying the above UWTS on both sides of~(\ref{model1}) does not affect the mixing matrix $A$ so that
the model covariance matrix of the observations at scale $j$, is still structured as
\begin{equation}
R_w^X(j) = A R_w^S(j) A^{\dagger} +   R_w^N(j)
\end{equation}
where $R_w^S(j)$ and $R_w^N(j)$ are the model diagonal spectral covariance matrices in the wavelet representation of $S$ and $N$ respectively
at scale $j$. Given an estimation of $R_w^X(j)$ from the data, $\widehat{R}_w^X(j)$, source separation follows from  minimizing the following
covariance matching criterion WSMICA-S in this \emph{spherical wavelets} representation:
\begin{equation}\label{Cost_wavelet}
 \Phi (\theta) =  \sum _{j=1}^{J+1}  \alpha_j \mathcal{D} \left( \widehat{R}_w^X(j), \,
    A R_w^S(j) A^{\dagger} + R_w^N(j) \right)
\end{equation}
with respect to the full set of parameters $\theta = (A,R_w^S(j), R_w^N(j) )$ or a subset thereof. Again, a good choice for $\mathcal{D}$ is
the Kullback-Leibler divergence given in equation~\ref{eq:kl}. With this mismatch measure, we can again resort to the EM algorithm to minimize~(\ref{Cost_wavelet}).

The weights in the covariance mismatch~(\ref{Cost_wavelet}) should be chosen to reflect the variability of the estimate of the corresponding
covariance matrix. Since WSMICA-S uses wavelet filters with only limited overlap, in the case of complete data maps we follow the derivation
in~\cite{ica:Del2003} and take $\alpha_j$ to be proportional to the number of spherical harmonic modes in the spectral domain covered at
scale $j$. In the case of data with gaps, we must further take into account that only a fraction $\beta_j$ of the wavelet coefficients are
unaffected so that the $\alpha_j$ should be modified in the same ratio.

Source maps may be reconstructed outside the possible gaps by Wiener filtering on each scale prior to inverting the wavelet transform, following
the procedure described in section~\ref{sect:mapesti}. Non linear filtering techniques may yield better results in the case of non-Gaussian components.

\subsubsection{WSMICA for \emph{flat} maps}

WSMICA can be easily implemented in the case of flat 2D data maps by substituting the 2D isotropic undecimated \emph{\`a trous} algorithm with
the cubic box-spline~\cite{starck:book02} as scaling function to the UWTS used in the previous section to deal with incomplete spherical maps.
This transform has several favorable properties for astrophysical data analysis. In particular, it is a shift invariant transform, the wavelet
coefficient maps on each scale are the same size as the initial image, and the wavelet and scaling functions have small compact supports in the
initial representation. As in the case of spherical maps, these properties allow  missing patches in the data maps to be handled easily.

WSMICA was used in~\cite{starck:yassir05} to process realistic CMB multichannel data as expected from the Planck experiment however on small enough
maps so that curvature could be neglected. The reported numerical experiments clearly confirm the benefits of correctly processing existing gaps.
Wavelets are able to correctly grab the spectral content of partly masked data maps and from there allow for better component separation.

\section{Applications}

\subsection{CMB data analysis}\label{sect:NUMEXP}
\index{CMB}
\index{CMB!ICA}
\index{ICA!CMB}

As an application of WSMICA on the sphere, we consider here the problem of CMB data analysis but in the special case where the use of a galactic
mask is a cause of non-stationarity which impairs the use of the spherical harmonics transform.

The simulated CMB, galactic dust and Sunyaev Zel'dovich (SZ) maps used, shown on the left-hand side of figure~\ref{components}, were obtained as
described in~\cite{ica:Del2003}. The problem of instrumental point spread functions is not addressed here, and all maps are assumed to have the
same resolution.The high level foreground emissions from the galactic plane region were discarded using the $Kp2$ mask from the WMAP team
website\footnote{\emph{http://lambda.gsfc.nasa.gov/product/map/intensity\_mask.cfm}}. These three \emph{incomplete} maps were mixed using the matrix
in table~\ref{MatrixA}, in order to simulate observations in the six channels of the Planck high frequency instrument~(HFI).

\begin{table}[!h]
  \begin{center}
   \footnotesize {
    \begin{tabular}{@{} cccc|c @{}}
      CMB & DUST & SZ &  & channel \\
      & & & &\\
      \hline
      & & & &\\
      $\quad1.0\quad$  &  $ 1.0 $    		&  $\quad-1.51\quad$	 	&& 100~GHz\\
      $\quad1.0 \quad$ & $ 2.20$   		& $\quad-1.05\quad$  		&& 143~GHz\\
      $\quad1.0 \quad$ &  $ 7.16 $  		& $\quad0.0\quad $ 			&& 217~GHz\\
      $\quad1.0 \quad$ &   $56.96 $		&  $\quad 2.22\quad$ 	&&353~GHz\\
      $\quad1.0 \quad$ &  $1.1\times10^{3}$ 		&   $\quad5.56\quad$  	&&545~GHz\\
      $\quad1.0\quad $ &  $1.47\times10^{5}$  	& $ \quad11.03\quad $	&& 857~GHz\\
    \end{tabular}}
    \caption{Entries of $A$, the mixing matrix used in our simulations.}\label{MatrixA}
  \end{center}
\end{table}

Gaussian \emph{instrumental} noise was added in each channel according to model~(\ref{model0}). The relative noise standard deviations between channels
were set according to the nominal values of the Planck HFI given in table~\ref{NoiseScale}.

\begin{table}[!h]
  \begin{center}
     \footnotesize {
    \begin{tabular}{@{} cccccc|c @{}}
      100&143&217&353&545&857&channel \\
          & & & & & &\\
  \hline
          & & & & & &\\
    $ 2.65\!\!\times\!\!10^{-6}$&$2.33\!\!\times\!\!10^{-6}$&$3.44\!\!\times\!\!10^{-6}$&$1.05\!\!\times\!\!10^{-5}$&$1.07\!\!\times\!\!10^{-4}$& $4.84\!\!\times\!\!10^{-3}$ &noise std\\
          & & & & & &\\
  % \hline
   %    & & & \\
   %   353~GHz&545~GHz  &857~GHz  &   channel\\
       %   & & &  \\
  %\hline
  %    & & & \\
   %$1.05\times10^{-5}$&$1.07\times10^{-4}$ & $4.84\times10^{-3}$ &   noise std\\
     % & & &  \\
      \end{tabular}}
    \caption{Nominal noise standard deviations in the six channels of the Planck HFI.}\label{NoiseScale}
  \end{center}
\end{table}

The synthetic observations were decomposed into six scales using the isotropic UWTS and WSMICA was used to obtain estimates of the mixing matrix
and of the initial source templates. The resulting component maps estimated using WSMICA, for nominal noise levels, are shown on the right-hand
side of figure~\ref{components} where the quality of reconstruction can be visually assessed by comparison to the initial components. The component
separation was also performed with SMICA based on Fourier statistics computed in the same six dyadic bands imposed by our choice of wavelet transform,
and with JADE. In figure~\ref{resultats}, the performances of SMICA, WSMICA and JADE, are compared in the particular case of CMB map estimation,
in terms of the relative standard deviation of the reconstruction error, $MQE$, defined by
 \begin{equation}
  MQE  = \frac{\mathbf{std}  ( CMB(\vartheta, \varphi)  - \alpha \times \widehat{CMB}(\vartheta, \varphi)  )}{\mathbf{std}  ( CMB(\vartheta, \varphi)   ) }
  \label{MQE}
\end{equation}
\index{jade}
\index{smica}
\index{smica!wavelet}
\index{jade!wavelet}
where $\mathbf{std}$ stands for empirical standard deviation  obviously computed outside the masked regions), and $\alpha$ is a linear regression
coefficient estimated in the least squares sense. As expected, since it is meant to be used in a noiseless setting, JADE performs well when noise
is very low. However, as the noise level increases, its performance degrades quite rapidly compared to the covariance matching methods. Further,
these results clearly show that using wavelet-based covariance matrices provides a simple and efficient way to cancel the bad impact that gaps
have on the performance of source separation using statistics based on the non local Fourier representation.

\begin{figure*}[htb]
% \begin{center}
% \includegraphics[width=7cm]{CMB_map}
% \includegraphics[width= 7cm]{CMB_w0db}
% \includegraphics[width= 7cm]{hist_equal_dust}
% \includegraphics[width= 7cm]{hist_equal_dust_w_0db}
% \includegraphics[width= 7cm]{sz_map3}
% \includegraphics[width= 7cm]{sz_w_0db_bis}
% \caption{%%
% The maps on the left are the templates for CMB, galactic dust and SZ used in the experiment
  % described in section~\ref{sec:NUMEXP}. The maps on the right were estimated using wSMICA
  %  and scalewise Wiener filtering. (The different maps are drawn here in different color scales
   % in order to enhance structures and ease visual comparisons).  }
% \label{components}
% \end{center}
\vbox{
\centerline{
\hbox{
% \psfig{figure=CMB_w0db.ps,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
\psfig{figure=CMB_map.pdf,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
\psfig{figure=CMB_w0db.pdf,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
}}
\centerline{
\hbox{
\psfig{figure=hist_equal_dust.pdf,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
\psfig{figure=hist_equal_dust_w_0db.pdf,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
}}
\centerline{
\hbox{
\psfig{figure=sz_map3.pdf,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
\psfig{figure=sz_w_0db_bis.pdf,bbllx=0.5cm,bblly=7.5cm,bburx=21.5cm,bbury=20cm,width=7cm,clip=}
}}}
\caption{The maps on the left are zero mean templates for CMB ($\sigma =  4.17\times 10^{-5}$), galactic dust ($\sigma = 8.61\times 10^{-6}$)
and SZ ($\sigma = 3.32\times 10^{-6}$) used in the experiment described in section~\ref{sect:NUMEXP}. The standard deviations given are for
the region outside the galactic mask. The maps on the right were estimated using WSMICA and scalewise Wiener filtering as is explained in Appendix~2.
Obviously, map reconstruction using Wiener filtering is optimal only in front of stationary Gaussian processes. For non Gaussian maps, such as
given by the Sunyaev Zel'dovich effect, better reconstruction can be expected from non linear methods. The different maps are drawn here in
different color scales in order to enhance structures and ease visual comparisons. }
\label{components}
\end{figure*}

\begin{figure}
\begin{center}
\includegraphics[width=8cm]{resultats_aa}
\caption{Relative reconstruction error defined by~(\ref{MQE}) of the CMB component map using SMICA, WSMICA and JADE as a function of the instrumental noise level in dB relative to the nominal values in table~\ref{NoiseScale}.}
\label{resultats}
\end{center}
\end{figure}

\subsection{Sunyaev-Zeldovich cluster detection}

Another application of component separation techniques in astrophysics and cosmology is in the reconstruction of Sunyaev-Zel'dovich (SZ)
galaxy clusters in future SZ-survey experiments such as Olimpo, APEX, or Planck which will use multiband bolometer cameras. The goal is
to optimize SZ-Cluster extraction from the multichannel observed noisy maps. Resorting to blind methods is again attractive.

A complete description of the method we developed is given in~\cite{cluster:sz_cluster}. Before the actual detection of the SZ clusters,
the multichannel data maps are combined and filtered to produce a clean map of the SZ component, with greater signal to noise ratio. We used
an ICA approach to estimate the mixing matrix and perform the separation. A non linear filtering technique in the wavelet domain was used
for the purpose of denoising. Finally a detection algorithm extracts the SZ clusters candidates from the restored SZ map. This is a new
application of ICA to multichannel astrophysical data analysis.

In the 100 to 600~GHz range, the brightest components of the sky are the Cosmic Microwave Background (CMB), the Infrared Point Sources,
the Galactic dust emission and, swamped in the previous ones, the SZ clusters. It follows that the true sky map $X_{\nu}(\vartheta, \varphi)$,
in a given optic band centered on $\nu$, can be modeled as a sum of distinct astrophysical radiations as in
\begin{equation} \label{SkyBandModel1}
	X_{\nu}(\vartheta, \varphi)  =  CMB_{\nu}(\vartheta, \varphi)  + IR_{\nu}(\vartheta, \varphi)  + Gal_{\nu}(\vartheta, \varphi)  + SZ_{\nu}(\vartheta, \varphi)
\end{equation}
where $\vartheta, \varphi$ denote spatial or angular indexes on 2D or spherical maps. Assuming again that the radiative properties of the sources
are completely isotropic in the sense that they do not depend on the direction of observation, the above model can be rewritten in the following factored form:
\begin{equation} \label{SkyBandModel2}
X_{\nu}(\vartheta, \varphi) =  \sum_{i} a_{\nu, i} S_{i}(\vartheta, \varphi)  \,\,+\,\, N_{\nu}(\vartheta, \varphi)
\end{equation}
where $S_{i}$ is the spatial template and $a_{\nu, i}$  the emission law of the $i\,{\textrm{th}}$ astrophysical component. Although this is
a coarse approximation in the case of Infrared Point Sources, it is mostly valid for the other three components. With observations available
in $m$ channels, assuming the beam varies only slightly as a function of $\nu$, equation~(\ref{SkyBandModel2}) can be written in matrix form :
\begin{equation} \label{SkyBandModel3}
X(\vartheta, \varphi) = A\,\, S(\vartheta, \varphi)  \,\,+\,\, N(\vartheta, \varphi)
\end{equation}
where $X(\vartheta, \varphi)$ is a vector in $\mathbb{R}^m$, $A$ is an $m\times n$ matrix, n is the number of contributing astrophysical components,
$S(\vartheta, \varphi)$ is now a vector in  $\mathbb{R}^n$ and $N(\vartheta, \varphi)$ in $\mathbb{R}^m$. Equation~(\ref{SkyBandModel3}) expresses
that the observations consist of linear mixtures of astrophysical components with different weights and additive noise.

Simulations of the major contributions in the frequency range considered here were generated according to the procedure described
in~\cite{cluster:sz_cluster}. Figure~\ref{RawMaps} shows such typical simulations. These four physical components are linearly
combined into a ``true" sky maps which are then convolved  with the experimental beam. Then instrumental noise is added. The resulting
noisy mixture maps, as shown on figure~\ref{NoisyMaps} would be what the analysis team would recover from the data, after pointing
reconstruction, outlier removal, de-correlation of instrumental systematics in the data, and map-making.

\begin{figure*}[htp]
\vbox{
\centerline{
\hbox{
\psfig{figure=CMB.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.5cm,width=6.5cm,clip=}
\hspace{0.2cm}
\psfig{figure=SZOld.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.5cm,width=6.5cm,clip=}
}}
\vspace{0.3cm}
\centerline{
\hbox{
\psfig{figure=IR.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.5cm,width=6.5cm,clip=}
\hspace{0.2cm}
\psfig{figure=TheGalax.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.5cm,width=6.5cm,clip=}
}}
}
\caption{The 4 physical components of the sky included in our simulation:
{\bf a)} is a map of the CMB's anisotropies in unit of $\mu$K,
{\bf b)} is the SZ Cluster map, in unit of y Compton,
{\bf c)} is the IR point source map, convolved with a beam of 2 arcmin, in Jy at 350GHz, finally
{\bf d)} is the Galactic dust map in unit of MJy/st at 100 $\mu$m.}
\label{RawMaps}
\end{figure*}

%
%
%
\begin{figure*}[htp]
\vbox{
\centerline{
\hbox{
\psfig{figure=c143GHz.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.0cm,width=6.0cm,clip=}
\hspace{0.2cm}
\psfig{figure=c217GHz.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.0cm,width=6.0cm,clip=}
}}
\vspace{0.2cm}
\centerline{
\hbox{
\psfig{figure=c385GHz.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.0cm,width=6.0cm,clip=}
\hspace{0.2cm}
\psfig{figure=c600GHz.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.7cm,height=7.0cm,width=6.0cm,clip=}
}}
}
\caption{Simulated maps in Olimpo's four frequency bands. {\bf upper right} is the 147 GHz Band, {\bf upper left)} is the 217 GHz Band, {\bf lower right} is
the 385 GHz Band: CMB anisotropies, IR point sources and Galactic Dust blend in this band {\bf lower left} 500 GHz: IR point sources and Galactic Dust are
the dominant features at high frequencies. SZ cluster signal is dominated by other astrophysical sources at all frequencies.}
\label{NoisyMaps}
\end{figure*}
%
%
%

Any mainstream ICA algorithm for sparse sources, based on maximizing non-Gaussianity, would probably be successful at separating the SZ component map.
We chose to use JADE in the wavelet domain, as described in section~\ref{sec:wjade}. Wavelets come into play as a sparsifying transform : data is sparse
on a basis when this basis allows to describe that signal with a small number of coefficients. This is a highly desirable property, since noise is not
expected to be sparse at the same time on such a basis. Choosing a sparsifying basis thus allows to enhance signal to noise ratio. Moving the data to
a wavelet representation does not affect its information content and applying a wavelet transform on both sides of~(\ref{model0}) does not affect the
mixing matrix and the model structure is preserved. However, the statistical distribution of the data coefficients in the new representation is different:
wavelets are known to lead to sparse approximately \emph{i.i.d.} representations of structured data. Further, the \emph{local} (coefficient wise) signal
to noise ratio depends on the choice of a representation. A wavelet transform tends to grab the informative coherence between pixels while averaging
the noise contributions, thus enhancing structures in the data. Although the standard ICA model is for a noiseless setting, the derived methods can be
applied to real data. Performance will depend on the detectability of significant coefficients \emph{i.e.} on the sparsity of the statistical distribution
of the coefficients. Moving to a wavelet representation will then often lead to more robustness to noise. We noted that the estimation of the mixing matrix
could be slightly enhanced by prefiltering the data using a Gaussian with the same width as the optical beam. The resulting SZ component map is shown
on figure~\ref{JADEMaps}. Different filtering techniques were applied on this map and the results of a quantitative comparison are given in~\cite{cluster:sz_cluster}.

\begin{figure}[htp]
 \centerline{
\hbox{
%\psfig{figure=SZJadeG1_69w.ps}
\psfig{figure=SZJadeG1_69w.pdf,bbllx=3.cm,bblly=6.cm,bburx=19.1cm,bbury=21.8cm,height=6.5cm,width=6.5cm,clip=}
}
}
\caption{SZ component map extracted by JADE from the four observed noisy maps. The SZ cluster signal, subdominant at all observed frequencies,
now appears clearly. No obvious leftovers from other astrophysical sources are seen. \textbf{Remaining noise is small, because we prefiltered
data before JADE processing, and we simulated the nominal noise levels of an ambitious project: Olimpo.}}
\label{JADEMaps}
\end{figure}


\documentclass{article}

\usepackage{lipsum}
\usepackage[margin=1.5in]{geometry}
\usepackage{titlesec}
\usepackage{graphicx}
\usepackage{amsmath}

\usepackage{mathtools, amssymb, nccmath}
\usepackage{bigstrut, changepage, lipsum}
\usepackage{mathtools}

\newcommand{\code}{\texttt}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\usepackage{siunitx} % Required for alignment

% Specify images directory
\graphicspath{ {./report-images/} }

% Header and Footer stuff
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhead{}
\fancyfoot{}
\fancyfoot[R]{ \thepage\ }
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrulewidth}{0pt}
\newcommand{\sectionbreak}{\clearpage}
\setlength{\parindent}{0pt}

%

\begin{document}

%----------------------------------------------------------------------------------------
%	TITLE PAGE
%----------------------------------------------------------------------------------------

\begin{titlepage} % Suppresses displaying the page number on the title page and the subsequent page counts as page 1
	\newcommand{\HRule}{\rule{\linewidth}{0.5mm}}% Defines a new command for horizontal lines, change thickness here

	\center % Centre everything on the page

	%------------------------------------------------
	%	Headings
	%------------------------------------------------

	\textsc{\Large Basic orthogonal and periodic functions}\\[0.5cm] % Major heading such as course name

	\textsc{\large Exercise 3}\\[0.5cm] % Minor heading such as course title

	%------------------------------------------------
	%	Title
	%------------------------------------------------

	\HRule\\[0.6cm]

	{\huge\bfseries Least-squares Approximation of cos(2x) and cos(4x) Using Chebyshev Polynomials}\\[0.25cm] % Title of your document

	\HRule\\[1.5cm]

	%------------------------------------------------
	%	Author(s)
	%------------------------------------------------

	\begin{minipage}{0.4\textwidth}
		\begin{flushleft}
			\large
			\textit{Author}\\
			\textsc{Cesare De Cal} % Your name
		\end{flushleft}
	\end{minipage}
	~
	\begin{minipage}{0.4\textwidth}
		\begin{flushright}
			\large
			\textit{Professor}\\
			\textsc{Annie Cuyt}\\ % Supervisor's name
			[0.25cm]
			\textit{Assistant Professor}\\
			\textsc{Ferre Knaepkens} % Supervisor's name

		\end{flushright}
	\end{minipage}

	\vfill\vfill\vfill

	{\large\today}

	\vfill

\end{titlepage}

%----------------------------------------------- Introduction ------------------------------------------------------
\section{Introduction}\label{sec:intro}
This exercise asks to compute the Chebyshev approximation
$$t(x)=\sum^n_{j=0}{a_jT_j(x)}$$

for the functions $f(x)=\cos(2x)$ and $f(x)=cos(4x)$ over the interval $[-\pi,\pi]$ for $n = 6$ and then plot the original functions with the approximations and draw conclusions on the results.\\

As we've seen in class, Chebyshev polynomials are a set of orthogonal polynomials that can be used to approximate to a least squares fit. For this exercise, I'll first compute the roots of the polynomial (also called Chebyshev nodes), then calculate its coefficients for both the functions. Finally, I'll calculate the y values of the approximation to see how well it approximates the original function. During this process, I'll make sure to choose the correct data points and the correct number of data points to exploit the properties of these basis functions.

%---------------------------------- Tools ---------------------------------------------------------------------------
\section{Tools}
To solve this problem, I've used MATLAB as requested by the exercise. To make the computation more efficient, I wrote my own function which calculates the Chebyshev polynomial instead of using the built-in MATLAB function $\code{chebyshevT}$ which was noticeably slower. Writing my own function was just a matter of using the compact closed-form expression for the Chebyshev polynomials:

$$T_i(x)=\cos{(i\arccos(x))}$$

Built-in MATLAB functions have been used such as $\code{cos(x)}$ and $\code{acos(x)}$ to calculate the cosine and the inverse cosine function. $\code{linspace(start, end, nrOfPoints)}$ was used to create an array of equidistant points for plotting. The functions $\code{plot(x, y)}$, $\code{legend()}$, $\code{axis([xmin xmax ymin ymax])}$, $\code{ylabel(label)}$, and $\code{xlabel(label)}$ were also used for plotting.

\section{Computation}
Given $n=6$, I first calculate the nodes of the Chebyshev polynomial by using the formula:

$$x_j=\cos \Big( \frac{2j-1}{2n}\pi \Big), j=1,\dots,n+1$$

The resulting zeros are:

$$
\begin{bmatrix}
9.749279121818236e-01\\
7.818314824680298e-01\\
4.338837391175582e-01\\
6.123233995736766e-17\\
-4.338837391175581e-01\\
-7.818314824680295e-01\\
-9.749279121818237e-01\\
\end{bmatrix}
$$

Let's first analyze the function $\cos(2x)$. We've seen in class we can calculate the coefficients with the formula given by:

$$c_j=\frac{2}{N}\sum_{k=0}^{n}f(x_j)T_j(x_k)$$

To better exploit the properties of Chebyshev polynomials, I rescale the interval from $[-\pi,\pi]$ to $[-1,1]$. This is done in the function calculation $f(x_j)$ by multiplying the argument by $\pi$. I find the following coefficients:

$$
\begin{bmatrix}
4.407675118300712e-01\\
3.489272363107635e-16\\
5.739892333362830e-01\\
-2.537652627714643e-16\\
6.516380108719009e-01\\
-1.554312234475219e-15\\
-7.019674665493458e-01\\
\end{bmatrix}
$$

The polynomial for the first function $\cos{2x}$ looks like this:

\begin{equation}
  \begin{array}{l}
    t(x) = 4.407675118300712e-01 \times T_0(x)+ 3.489272363107635e-16 \times T_1(x) + \\
    5.739892333362830e-01 \times T_2(x) + \dots -7.019674665493458e-01 \times T_6(x)
  \end{array}
\end{equation}

To plot these coefficients in the graph, I need to calculate their associated y values:

$$
\begin{bmatrix}
9.876173836808355e-01\\
1.986723718905843e-01\\
-9.149466098687946e-01\\
9.999999999999993e-01\\
-9.149466098687926e-01\\
1.986723718905768e-01\\
9.876173836808371e-01\\
\end{bmatrix}
$$

Calculating the coefficients for the $\cos(4x)$ function now only requires minor changes in the code. The following are the coefficients of the polynomial:
$$
\begin{bmatrix}
6.879841220704104e-01\\
2.537652627714643e-16\\
-1.535576743556636e-01\\
-1.459150260935920e-15\\
1.012919200612446e+00\\
-1.094362695701940e-15\\
5.104689360033142e-01
\end{bmatrix}
$$

The polynomial for the second function $\cos{4x}$ looks like this:

\begin{equation}
  \begin{array}{l}
    t(x) = 6.879841220704104e-01 \times T_0(x) + 2.537652627714643e-16 \times T_1(x) \\
    -1.535576743556636e-01 \times T_2(x) + \dots + 5.104689360033142e-01 \times T_6(x)
  \end{array}
\end{equation}

The associated y values for the given roots are the following:

$$
\begin{bmatrix}
9.507761930971577e-01\\
-9.210585772947382e-01\\
6.742545978208001e-01\\
1.000000000000000e+00\\
6.742545978207990e-01\\
-9.210585772947411e-01\\
9.507761930971639e-01\\
\end{bmatrix}
$$

In order to properly calculate the associated y values, I had to subtract half of the first coefficient from the calculation. This is done because the formula I used to calculate the coefficients isn't correct for the first coefficient. The plots for both functions have been added to the next section.\\

\section{Plots}
Function $\cos(2x)$ and $\cos(4x)$ respectively:\\
\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{cos2x.jpg}\\
\includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{cos4x.jpg}
\section{Observations}
I first calculated the zeros of the polynomial so I could use them as nodes in polynomial interpolation because the resulting interpolation polynomial minimizes the effect of Runge's phenomenon. The functions, however, still display a little bit of Runge's phenomenon. \\

The Chebyshev approximation for the function $\cos(2x)$ is pretty accurate, whereas the approximation for $\cos(4x)$ not so much as it doesn't take into account some of the spikes of the original function.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%% DYMOLA INTERFACE  %%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Dymola Interface}
Modelica is "a non-proprietary, object-oriented, equation-based language to conveniently model complex physical systems containing, e.g., mechanical, electrical, electronic, hydraulic,
thermal, control, electric power or process-oriented subcomponents."\footnote{\url{http://www.modelica.org}}.  Modelica models (with a file extension of .mo) are built, translated (compiled), and simulated in Dymola (http://www.modelon.com/p-
roducts/dymola/), which is a commercial modeling and simulation environment based on the Modelica modeling language.
A standard Modelica example called BouncingBall, which simulates the trajectory of an object falling in one dimension from a height, is shown as an example:
\begin{lstlisting}
model BouncingBall
  parameter Real e=0.7 "coefficient of restitution";
  parameter Real g=9.81 "gravity acceleration";
  parameter Real hstart = 10 "height of ball at time zero";
  parameter Real vstart = 0 "velocity of ball at time zero";
  Real h(start=hstart,fixed=true) "height of ball";
  Real v(start=vstart,fixed=true) "velocity of ball";
  Boolean flying(start=true) "true, if ball is flying";
  Boolean impact;
  Real v_new;
  Integer foo;

equation
  impact = h <= 0.0;
  foo = if impact then 1 else 2;
  der(v) = if flying then -g else 0;
  der(h) = v;

  when {h <= 0.0 and v <= 0.0,impact} then
    v_new = if edge(impact) then -e*pre(v) else 0;
    flying = v_new > 0;
    reinit(v, v_new);
  end when;

  annotation (uses(Modelica(version="3.2.1")),
    experiment(StopTime=10, Interval=0.1),
    __Dymola_experimentSetupOutput);

end BouncingBall;
\end{lstlisting}

\subsubsection{Files}
When a modelica model, e.g., BouncingBall model, is implemented in Dymola, the platform dependent C-code from a Modelica model and the corresponding executable code
(i.e., by default dymosim.exe on the Windows operating system) are generated for simulation.  After the executable is generated, it may be run multiple times (with Dymola license).
A separate TEXT file (by default dsin.txt) containing model parameters and initial conditions are also generated as part of the build process.  The RAVEN Dymola interface
modifies input parameters by changing copies of this file.  Both the executable and TEXT parameter file (or simulation initialization file) names must be provided to RAVEN. The TEXT parameter file must be of type 'DymolaInitialisation'.  In the case of
the BouncingBall model previously mentioned on the Windows operating system, the \textless Files\textgreater  specification would look like:
\begin{lstlisting}[style=XML]
<Files>
  <Input name='dsin.txt' type='DymolaInitialisation'>dsin.txt</Input>
</Files>
\end{lstlisting}

The Dymola interface can only pass scalar values into the TEXT parameter file. If the user wants to pass vector information to Dymola, he can do so by providing an optional TEXT vector file to Dymola. This file must have the type 'DymolaVectors'. This additional file can then be read by the Dymola model. If vecor data is passed from RAVEN to the Dymola interface and the TEXT vector file is not specified, the interface will display an error and stop the Dymola execution. If the TEXT vector file is specified (and vector data is passed to the interface), the interface will write the datd into the specified file, but also display a warning, saying that the Dymola interface found vector data to be passed and if this data is supposed to go into the simulation initialisation file of type 'DymolaInitialisation' the array must be split into scalars. The \textless Files\textgreater specification for the vector data look as follows:
\begin{lstlisting}[style=XML]
<Files>
  <Input name='timeSeriesData.txt' type='DymolaVectors'>timeSeriesData.txt</Input>
</Files>
\end{lstlisting}

\subsubsection{Models}
An executable (dymosim.exe) and a simulation initialization file (dsin.txt) can be generated after either translating or simulating the
Modelica model (BouncingBall.mo) using the Dymola Graphical User Interface (GUI) or Dymola Application Programming Interface (API)-routines.
To generate an executable and a simulation initialization file, use the Dymola API-routines (or Dymola GUI) to translate the model as follows:
\lstset{
    frame=single,
    breaklines=true,
    postbreak=\raisebox{0ex}[0ex][0ex]{\ensuremath{\color{red}\hookrightarrow\space}}
}
\begin{enumerate}
\item Change to the directory containing the .mo file to generate an executable.  In Dymola GUI, this corresponds to File/Change Directory in menus:
\begin{lstlisting}
>> cd("C:/msys64/home/KIMJ/projects/raven/framework/CodeInterfaces/Dymola");
C:/msys64/home/KIMJ/projects/raven/framework/CodeInterfaces/Dymola
 = true
\end{lstlisting}
\item Reads the specified file and displays its window.  In Dymola GUI, this corresponds to File/Open in the menus:
\begin{lstlisting}
>> openModel("BouncingBall.mo")
 = true
\end{lstlisting}
\item Compile the model (with current settings), and create the model executable and the corresponding simulation initialization file.  In Dymola GUI, this corresponds to Translate Model in the menus:
\begin{lstlisting}
>> translateModel("BouncingBall");
 = true
\end{lstlisting}
At this point the model executable and the simulation initialization file should have been created in the same directory as the original model file.
Additionally, they could be created by simulating the model.  The following command corresponds to Simulate in the menus in Dymola GUI:
\begin{lstlisting}
>> simulateModel("BouncingBall", stopTime=10, numberOfIntervals=0, outputInterval=0.1, method="dassl", resultFile="BouncingBall");
 = true
\end{lstlisting}
The file extension (.mat) is automatically added to a output file (resultFile), e.g., BouncingBall.mat.  If the generated executable code is triggered directly from a
command prompt, the output file is always named as "dsres.mat".
\end{enumerate}
The model executable is specified to RAVEN using the \textless Models\textgreater  section of the input file as follows:
\begin{lstlisting}[style=XML]
<Simulation>
    ...
  <Models>

      <executable>dymosim.exe</executable>

  </Models>
    ...
</Simulation>
\end{lstlisting}
RAVEN works best with Comma-Separated Value (CSV) files.  Therefore, the default
.mat output type needs to be converted to .csv output.
The Dymola interface will automatically convert the .mat output to human-readable
forms, i.e., .csv output, through its implementation of the finalizeCodeOutput function.
\\In order to speed up the reading and conversion of the .mat file, the user can specify
the list of variables (in addition to the Time variable) that need to be imported and
converted into a csv file minimizing
the IO memory usage as much as possible. Within the \xmlNode{Code} the following
XML
node (in addition ot the \xmlNode{executable} one) can be inputted:

\begin{itemize}
   \item \xmlNode{outputVariablesToLoad}, \xmlDesc{space separated list, optional
   parameter}, a space separated list of variables that need be exported from the .mat
   file (in addition to the Time variable). \default{all the variables in the .mat file}.
\end{itemize}
For example:
\begin{lstlisting}[style=XML]
<Simulation>
    ...
  <Models>

      <executable>dymosim.exe</executable>
      <outputVariablesToLoad>var1 var2 var3</outputVariablesToLoad>

  </Models>
    ...
</Simulation>
\end{lstlisting}


% JuliaCon Proceedings
% Template with instructions: https://www.overleaf.com/read/dqjbrhqxjpwq

\documentclass{juliacon}
\setcounter{page}{1}
\usepackage{url}
\usepackage{amsmath}
\usepackage[]{algorithm2e}
\newcommand\EatDot[1]{}

\begin{document}

\input{header}

\maketitle

\begin{abstract}

\small{\texttt{StateSpaceModels.jl}} is an open-source Julia package for modeling, forecasting and simulating time series in a state-space framework. The package represents a straightforward tool that can be useful for a wide range of applications that deal with time series. In addition, it contains features that are not present in related commercial software, such as Monte Carlo simulation and the possibility of setting any user-defined linear model.

\end{abstract}

\section{Motivation} \label{sec:introduction}

State-space modeling is a classical framework in control engineering that represents a system through the definition of input, state, and output variables \cite{zadeh2008linear}. Input variables are external entities that are inserted into the system and can serve as control inputs or noise. State variables represent unobserved components that evolve through time following a given state equation and also depending on the values of the input variables. Finally, output variables result from the realization of the state plus noise factors and represent the observable outcome of the system. In general, state-space models make use of the Kalman filter \cite{kalman1960new, welch1995introduction} to obtain predictive estimates for the state.

Due to its comprehensive form and wide range of potential applications, state-space models found a niche in time-series modeling, forecasting, and simulation, representing a flexible framework for time-series analysis with time-varying parameters \cite{durbin2012time}. The ability to conveniently define the evolution of a time series allows the characterization of stochastic components, such as trend and seasonality, that are non-trivial to model in other frameworks.

There are several packages focused on state-space models for time-series analysis in other languages, such as \texttt{KFAS} in R \cite{kfas} and \texttt{Statsmodels} in Python \cite{seabold2010statsmodels}, as well as commercial software such as \texttt{STAMP} \cite{koopman2000stamp}. Among related Julia packages, we highlight:
\begin{itemize}
    \item ~\texttt{Kalman.jl} \cite{kalmanjl}, which implements Kalman filtering and smoothing.
    \item ~\texttt{StateSpace.jl} \cite{statespacejl}, a control-oriented package with several versions of the Kalman filter.
    \item ~\texttt{ControlSystems.jl} \cite{controlsystemsjl}, a package aimed at control applications with numerous features, including state-space modeling.
\end{itemize}

The contributions brought by these packages notwithstanding, there are currently no state-space model packages aimed at time-series analysis in Julia. Furthermore, some have not been updated for a long time and are not functional in Julia 1.0. Thus, the main objectives of \texttt{StateSpaceModels.jl} \cite{statespacemodels} are:

\begin{enumerate}
    \item to fill this gap by implementing a general and intuitive framework in Julia for modeling, estimating, forecasting and simulating time series with state-space models;
    \item to provide an open-source package that is capable of performing the same functions as related commercial software such as \texttt{STAMP} and more;
    \item to develop a package for time-series analysis with state-space models that is fully implemented in Julia, contrary to related packages in other languages that depend on C or Fortran \mbox{routines} \cite{kfas}.
\end{enumerate}

The remainder of this paper is organized as follows. In Section \ref{sec:statespace}, the Gaussian state-space framework is introduced. In Section \ref{sec:modeling}, the procedures for model specification are presented. Section \ref{sec:estimation} explains the filtering, estimation and smoothing processes. Section \ref{sec:diagnostics} discusses the available diagnostics to evaluate the model specification and estimation. In Section \ref{sec:forecasting}, the forecasting and simulation procedures are presented. Section \ref{sec:applications} contains examples of applications that illustrate the use of the package. Finally, conclusions and future work are provided in Section \ref{sec:conclusion}.

\section{Gaussian state-space framework} \label{sec:statespace}

For the sake of consistency and readability, we utilize in the package the same notation as \cite{durbin2012time}. Suppose we have a series of observations $y_{1}, \dots, y_{n}$. In a state-space framework, it is assumed that the observations depend on a set of unobserved components denominated states, or simply the state, and denoted by $\alpha_{1}, \dots, \alpha_{n}$. The state components often have physical interpretations, such as trend and seasonality in a time-series context or position and speed in a control setting.

The main idea behind state-space modeling is to define the evolution of the state and its relation with the observed variables. This is done through the following two equations, denoted the observation equation and the state equation, respectively:
%
\begin{align}
    y_{t} &= Z_{t} \alpha_{t} + \varepsilon_{t}, \hspace{0.08\columnwidth} \varepsilon_{t} \sim N(0, H_{t}), \label{eq:obs_equation} \\
    \alpha_{t+1} &= T_{t} \alpha_{t} + R_{t} \eta_{t}, \quad \eta_{t} \sim N(0, Q_{t}), \label{eq:state_equation}
\end{align}
%
\noindent where $y_{t}$ is a $p \times 1$ vector of observations and $\alpha_{t}$ is an unobserved $m \times 1$ vector representing the state at instant $t$. Note that the behavior of the system over time is determined by $\alpha_{t}$ as defined in \mbox{Eq. \eqref{eq:state_equation}}, but $\alpha_{t}$ cannot be directly observed, contrary to $y_{t}$.

Matrices $Z_{t}$, $T_{t}$, and $R_{t}$ are the ones that define how the observations relate to the state and how the state evolves over time, and are generally assumed to be known. The error terms $\varepsilon_{t}$ and $\eta_{t}$ are supposed to be serially independent and independent of each other. Finally, $H_{t}$ and $Q_{t}$ are the covariance matrices of the error terms. In this package, it is assumed that $T$, $R$, $H$, and $Q$ are not time-varying, as this is the case for the vast majority of practical applications. Conversely, $Z_{t}$ is allowed to vary over time.

Next, for illustration purposes, we present an example model and show how it can be inserted into the state-space framework. Consider the following model, called the linear trend model:
%
\begin{align}
    y_{t} &= \mu_{t} + \varepsilon_{t}, \hspace{0.115\columnwidth} \varepsilon_{t} \sim N(0, \sigma^{2}_{\varepsilon}), \label{eq:lineartrend1} \\
    \mu_{t+1} &= \mu_{t} + \nu_{t} + \xi_{t}, \quad \xi_{t} \sim N(0, \sigma^{2}_{\xi}), \label{eq:lineartrend2} \\
    \nu_{t+1} &= \nu_{t} + \zeta_{t}, \hspace{0.12\columnwidth} \zeta_{t} \sim N(0, \sigma^{2}_{\zeta}), \label{eq:lineartrend3}
\end{align}
%
\noindent where $\mu_{t}$ represents the trend component and $\nu_{t}$ the slope component. Note that the we can rewrite this model in the framework of Eq. \eqref{eq:obs_equation}--\eqref{eq:state_equation} in the following manner:
%
\begin{align}
    y_{t} &= \begin{bmatrix} 1 & 0 \end{bmatrix} \alpha_{t} + \varepsilon_{t}, \hspace{0.15\columnwidth} \varepsilon_{t} \sim N(0, \sigma^{2}_{\varepsilon}), \\
    \alpha_{t+1} &= \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \alpha_{t} + \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\!\eta_{t}, \quad \eta_{t} \sim N \bigg(\!\begin{bmatrix} 0 \\ 0 \end{bmatrix}\!, \begin{bmatrix} \sigma^{2}_{\xi} & 0 \\ 0 & \sigma^{2}_{\zeta} \end{bmatrix}\!\bigg).
\end{align}
%
Similarly, more complex and sophisticated models can also be modeled in the state-space framework.

\section{Model specification} \label{sec:modeling}

Model specification is the step wherein the user defines the state-space model they want to consider. This is done through the creation of a \texttt{StateSpaceModel} structure that contains the observations $y_{t}$ as well as matrices $Z_{t}$, $T$, and $R$.

\subsection{Predefined models} \label{subsec:predefined}

A set of predefined classical models is available, namely the local level model, the linear trend model, the basic structural model, and the structural model with exogenous variables. These models can be conveniently defined by calling a function and providing the observations $y_{t}$. Note that all models automatically extend to the multivariate case.

\subsubsection{Local level model}

The local level model consists of a stochastic level component that is defined by a random walk:
%
\begin{align}
    y_{t} &= \mu_{t} + \varepsilon_{t}, \quad \varepsilon_{t} \sim N(0, \sigma^{2}_\varepsilon), \label{eq:locallevel1} \\
    \mu_{t+1} &= \mu_{t} + \xi_{t}, \quad \xi_{t} \sim N(0, \sigma^{2}_\xi). \label{eq:locallevel2}
\end{align}
%
A local level \texttt{StateSpaceModel} can be created using:
%
\begin{lstlisting}[language = Julia]
model = local_level(y)
\end{lstlisting}
%
\noindent where \texttt{y} is an observation vector in the univariate case or a matrix in column-wise fashion in the multivariate case, i.e., each column representing a variable and each line representing a time period.
%

\subsubsection{Linear trend model}

The linear trend model consists of a local level model with the addition of a stochastic slope term, as seen in Eq. \eqref{eq:lineartrend1}--\eqref{eq:lineartrend3}. A linear trend \texttt{StateSpaceModel} can be created using:
%
\begin{lstlisting}[language = Julia]
model = linear_trend(y)
\end{lstlisting}

\subsubsection{Structural model}

The basic structural model consists of three stochastic components: trend, slope, and seasonality, defined in the following manner:
%
\begin{align}
    y_{t} &= \mu_{t} + \gamma_{t} + \varepsilon_{t}, \hspace{0.09\columnwidth} \varepsilon_{t} \sim N(0, \sigma^{2}_\varepsilon), \label{eq:structural1} \\
    \mu_{t+1} &= \mu_{t} + \nu_{t} + \xi_{t}, \hspace{0.095\columnwidth} \xi_{t} \sim N(0, \sigma^{2}_\xi), \label{eq:structural2} \\
    \nu_{t+1} &= \nu_{t} + \zeta_{t}, \hspace{0.18\columnwidth} \zeta_{t} \sim N(0, \sigma^{2}_\zeta), \label{eq:structural3} \\
    \gamma_{t+1} &= \sum_{j=1}^{s-1} \gamma_{t+1-j} + \omega_{t}, \quad \omega_{t} \sim N(0, \sigma^{2}_\omega). \label{eq:structural4}
\end{align}
%
In this model, there is a stochastic trend that contains a slope component, similarly to the linear trend model. Furthermore, a stochastic seasonal component is modeled so that its sum over the seasonality period $s$ is equal to zero except for a noise factor. A basic structural \texttt{StateSpaceModel} can be created using:
%
\begin{lstlisting}[language = Julia]
model = structural(y, s)
\end{lstlisting}

Additionally, a structural model can contain exogenous variables, also known as explanatory variables. Exogenous variables represent external factors that are correlated to the phenomenon in question and can, thus, be used to improve the estimation and forecasting by increasing the available information. This model is identical to \eqref{eq:structural1}--\eqref{eq:structural4} except for the observation equation, which has an additional exogenous factor:
%
\begin{align}
    y_{t} &= \mu_{t} + \gamma_{t} + \theta^{\top} X_{t} + \varepsilon_{t}, \quad \varepsilon_{t} \sim N(0, \sigma^{2}_\varepsilon) \label{eq:structural_exogenous}
\end{align}
%
A structural model with exogenous variables can be created with:
%
\begin{lstlisting}[language = Julia]
model = structural(y, s; X = X)
\end{lstlisting}
%
\noindent where \texttt{X} is a matrix containing the exogenous variables observations in column-wise fashion.

\subsection{User-defined models}

Apart from the well-known predefined models presented in Section \ref{subsec:predefined}, the package allows the input of any user-defined linear model through the definition of matrices $Z_{t}$, $T$, and $R$ through the following constructor:
%
\begin{lstlisting}[language = Julia]
model = StateSpaceModel{Float64}(y, Z, T, R)
\end{lstlisting}

In the simpler case where $Z_{t}$ is constant, the argument \texttt{Z} can be input as an \texttt{Array\{Float64, 2\}}. Conversely, if $Z_{t}$ is time-varying, it must be input as an \texttt{Array\{Float64, 3\}} where the third dimension represents the time periods. Any desired float precision can be employed within the model.

\section{Filtering, estimation, and smoothing} \label{sec:estimation}

Next, the model needs to be estimated. In this step, values for the covariance matrices $H$ and $Q$, initially assumed to be unknown, are obtained via maximum likelihood estimation \cite{casella2002statistical}. In order to do that, the Kalman filter needs to be utilized. Additionally, a smoother is applied in order to obtain the so-called smoothed estimates for the state.

After specifying the model, the estimation step can be called with the function \texttt{statespace}:
%
\begin{lstlisting}[language = Julia]
ss = statespace(model; verbose = 1)
\end{lstlisting}

In this step, the Kalman filtering, the maximum likelihood estimation of the fixed parameters, and the smoothing are conducted. The optional argument \texttt{verbose}, which defaults to 1, specifies the verbosity: 0 for no output, 1 for some progress information, 2 for progress information and summary of the optimization result and 3 for progress information as well as the full optimization log.

An example of the estimation log with default verbosity can be seen below.
%
\begin{lstlisting}[language = Julia]
julia> ss = statespace(model)
==============================================================
                  StateSpaceModels.jl v0.2.0
 (c) Raphael Saavedra, Guilherme Bodin, and Mario Souto, 2019
--------------------------------------------------------------
            Starting state-space model estimation.
    Initiating maximum likelihood estimation with 3 seeds.
--------------------------------------------------------------
             Seed 0 is aimed at degenerate cases.
--------------------------------------------------------------
||    seed    |     log-likelihood      |      time (s)     ||
||       0    |        -16217.4939      |          0.21     ||
||       1    |         -1350.4763      |          2.65     ||
||       2    |         -1350.4763      |          4.08     ||
||       3    |         -1350.4763      |          5.69     ||
--------------------------------------------------------------
           Maximum likelihood estimation complete.
                  Log-likelihood: -1350.4763
             End of state-space model estimation.
==============================================================
\end{lstlisting}

\subsection{Filtering}

The filtering step derives the predictive and filtered states as well as their covariance matrices at every time period via the Kalman filter. The predictive state and its variance are given by
%
\begin{align}
    a_{t+1} &= \mathbb{E}[\alpha_{t+1} | Y_{t}], \\
    P_{t+1} &= \mathbb{V}[\alpha_{t+1} | Y_{t}],
\end{align}
%
while the filtered state and its variance are given by
%
\begin{align}
    a_{t|t} &= \mathbb{E}[\alpha_{t} | Y_{t}], \\
    P_{t|t} &= \mathbb{V}[\alpha_{t} | Y_{t}],
\end{align}
%
where $Y_{t}$ denotes the set of time periods $y_{1}, \dots, y_{t}$.

Furthermore, the filter also computes the innovations, or prediction errors, as well as their covariance matrix at every time period:
%
\begin{align}
    v_{t} &= y_{t} - Z a_{t}, \\
    F_{t} &= \mathbb{V}[v_{t}].
\end{align}
%

The package contains an implementation of the standard Kalman filter and the square-root Kalman filter \cite{durbin2012time}. The latter is a variant which utilizes Cholesky decomposition to ensure that the computed covariance matrices are positive semidefinite, thus avoiding numerical errors.

Additionally, in order to provide a flexible framework, we allow the utilization of any user-implemented variant of the Kalman filter through an abstract type called \texttt{AbstractFilter}:
%
\begin{lstlisting}[language = Julia]
struct MyKalmanFilter <: AbstractFilter
\end{lstlisting}

The choice of the filter type must be done in the estimation step as follows:
%
\begin{lstlisting}[language = Julia]
ss = statespace(model; filter_type = SquareRootFilter{Float64})
\end{lstlisting}

Filters allow any desired float precision. If no \texttt{filter\_type} is provided, the standard type \texttt{KalmanFilter\{Float64\}} is utilized as a default option. Further information on Kalman filtering can be found in Chapter 4.3 of \cite{durbin2012time}.

\subsection{Estimation}

The Kalman filter is responsible for deriving estimates for the predictive state and for its covariance matrix at each time period, but it is dependent on the values of the noise covariances $H$ and $Q$. Thus, these constant parameters need to be estimated via maximum likelihood \cite{casella2002statistical}. The log-likelihood function is given by
%
\begin{align}
    \ell(Y_{n}) = -\frac{np}{2} \log 2 \pi - \frac{1}{2} \sum_{t=1}^{n} (\log |F_{t}| + v_{t}^{\top} F_{t}^{-1} v_{t}),
\end{align}
%
which depends on the innovations and their covariance matrix at each time period. Therefore, at each optimization iteration, the Kalman filter needs to be executed.

The implemented optimization method \texttt{RandomSeedsLBFGS} generates random initial values for the parameters and then uses the L-BFGS algorithm \cite{liu1989limited} to maximize the log-likelihood function via the unconstrained optimization package \texttt{Optim.jl} \cite{mogensen2018optim}.

Additionally, similar to the filtering step, users are able to define any desired optimization method through abstract type \texttt{AbstractOptimizationMethod}:
%
\begin{lstlisting}[language = Julia]
struct MyOptimizationMethod <: AbstractOptimizationMethod
\end{lstlisting}

As with the filter type, the choice of optimization method must be done in the estimation step:
%
\begin{lstlisting}[language = Julia]
ss = statespace(model; optimization_method = RandomSeedsLBFGS())
\end{lstlisting}

More advanced parameters, such as optimizer tolerances and the number of seeds, can be set within the \texttt{RandomSeedsLBFGS} constructor.

\subsection{Smoothing}

The smoothing step is conducted after estimation is complete. It consists in obtaining the so-called smoothed state and its covariance matrix at each time period, i.e.,
%
\begin{align}
    \hat{\alpha}_{t} &= \mathbb{E}[\alpha_{t} | Y_{n}], \\
    V_{t} &= \mathbb{V}[\alpha_{t} | Y_{n}].
\end{align}
%
Similar to the filtering step, the smoothing is conducted through an iterative process. The procedure is done backwards, starting from the final time period $n$. The smoothed state estimates are useful for analyzing the behavior of the state components. Further information on state smoothing can be found on Chapter 4.4 of \cite{durbin2012time}.

\subsection{Missing observations}

One interesting property of the Kalman filter and smoother is their ability to treat missing observations. Suppose we have a time series spanning time periods $t = 1, \dots, n$, but for some periods $\tau, \dots, \tau^{*}$ we have no observations available. This represents an obstacle for estimating a model in several frameworks. Nonetheless, the Kalman filter and smoother allow the derivation of minimum variance linear unbiased estimates for the missing observations so that a completion of the time series is possible. Conveniently, the only necessary modification in the filtering and smoothing equations is considering $Z_{t} = 0$ for the missing period.

In the package, this is implemented so that any \texttt{NaN} values in the observations are considered missing observations and automatically treated in the filtering and smoothing steps. In this way, the estimation is successfully conducted even when the time series of interest is incomplete.

Consider the following simulated time series composed by a growing trend plus a Gaussian noise. Additionally, we remove observations 10 through 20 to treat them as missing:
%
\begin{lstlisting}[language = Julia]
# Create growing trend series with Gaussian noise
y = collect(1:0.25:20) + 0.5*randn(77)
# Remove observations 10 through 20
y[10:20] .= NaN
# Specify the state-space model and estimate it
model = linear_trend(y)
ss = statespace(model)
\end{lstlisting}
%
The results of the estimation step are presented in Fig. \ref{fig:missing_observations}. The predictive, filtered, and smoothed state estimates can be accessed in \texttt{ss.filter.a}, \texttt{ss.filter.att}, and \texttt{ss.smoother.alpha}, \mbox{respectively}. It is clear in Fig. \ref{fig:missing_observations} that the filter and smoother are able to effectively capture the growing linear trend of the series even in the period where observations are missing.
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/missing_obs.pdf}
	\caption{Automatic completion of missing observations in the filtering and smoothing.}
	\label{fig:missing_observations}
\end{figure}

\subsection{Estimation outputs}

At the end of the filtering, estimation, and smoothing routine executed by the \texttt{statespace} function, a \texttt{StateSpace} structure is returned. It contains six fields:

\begin{itemize}
    \item ~\texttt{model}: the previously defined \texttt{StateSpaceModel}.
    \item ~\texttt{filter}: a \texttt{FilterOutput} structure representing the result of the Kalman filter. It contains the predictive and filtered states as well as the innovations and their covariance matrices at each time period. Furthermore, there is a flag indicating if steady state was attained and the time period when it was attained.
    \item ~\texttt{smoother}: a \texttt{SmoothedState} structure representing the result of the smoothing procedure. It contains the smoothed state as well as its covariance matrix at each time period.
    \item ~\texttt{covariance}: a \texttt{StateSpaceCovariance} structure which contains the covariance matrices of the observation and state noises.
    \item ~\texttt{filter\_type}: the defined filter type.
    \item ~\texttt{optimization\_method}: the defined optimization method.
\end{itemize}

\section{Diagnostics} \label{sec:diagnostics}

As a way of evaluating the model specification and estimation, the package contains statistical diagnostics that can be conducted over the residuals. The available diagnostics are the Jarque-Bera normality test \cite{jarque1980efficient}, the Ljung-Box independence test \cite{box1970distribution}, and an homoscedasticity test \cite{durbin2012time}. The diagnostics can be run through function \texttt{diagnostics}.

An example of the diagnostics log can be seen below.
%
\begin{lstlisting}[language = Julia]
julia> diagnostics(ss)
==============================================================
                    Running diagnostics...
                Jarque-Bera: p-value = 0.89683
                  Ljung-Box: p-value = 0.95855
           Homoscedasticity: p-value = 0.30752
==============================================================
\end{lstlisting}

\section{Forecasting and Simulation} \label{sec:forecasting}

Finally, after the model has been estimated, forecasting and Monte Carlo simulation can be conducted. In the case of forecasting, estimates for the future values of the time series are computed along with their probability distributions. Alternatively, it is possible to simulate an arbitrary number of future scenarios of the series, which is useful for a wide range of applications such as stochastic optimization.

The minimum mean square error forecasts for $y_{t}$ can be obtained by treating future values of $y_{t}$ as missing values. Forecasting is conducted with the function \texttt{forecast}, which receives a \texttt{StateSpace} structure coming from the estimation step and the number of time periods ahead to be forecast, and outputs the minimum square error forecasts and the predictive distributions of $y_{t}$ at each time period:
%
\begin{lstlisting}[language = Julia]
pred, dist = forecast(ss, N)
\end{lstlisting}
%
Alternatively, another powerful tool is the simulation of future scenarios, since these are used as input by several applications, such as certain stochastic optimization models. Performing Monte Carlo simulation in a time-series state-space framework involves sampling several scenarios for the observation and state errors utilizing the estimated variances $H$ and $Q$ at each time period, and then computing the state-space recursions for each scenario.

Monte Carlo simulation can be conducted with the use of the function \texttt{simulate}, which receives a \texttt{StateSpace} structure coming from the estimation step, the number of time periods ahead $N$, and the number of scenarios $S$ to be simulated, and outputs an $N \times S$ matrix of scenarios for $y_{t}$:
%
\begin{lstlisting}[language = Julia]
scenarios = simulate(ss, N, S)
\end{lstlisting}

\section{Applications} \label{sec:applications}

In this section, we present several practical applications which can be addressed via state-space modeling. We utilize \texttt{StateSpaceModels.jl} to tackle the presented problems and provide the results. Furthermore, for the sake of reproducibility, the code of some of the examples is in the folder \texttt{examples} in the \texttt{StateSpaceModels.jl} repository.

\subsection{Airline passengers}

As a first example, let us use the classical monthly airline passengers time series. In order to avoid multiplicative effects, we use the well-known approach of taking the log of the series. Fig. \ref{fig:log_airline_forecast} shows the log-airline passengers time series. We can estimate a structural \texttt{StateSpaceModel} as follows.
%
\begin{lstlisting}[language = Julia]
# Load the AirPassengers dataset
AP = CSV.read("AirPassengers.csv")
# Take the log of the series
logAP = log.(Vector{Float64}(AP[:Passengers]))
# Specify the state-space model
model = structural(logAP, 12)
# Estimate the state-space model
ss = statespace(model)
\end{lstlisting}

By estimating a structural model, we can analyzed the individual components of the series, such as trend and seasonality. These components are presented in Fig. \ref{fig:log_airline_components} and represent dimensions 1 and 3 of the smoothed state, respectively.

We can also forecast the following two years of the time series by using the \texttt{forecast} function. The result is displayed in Fig. \ref{fig:log_airline_forecast}.
%
\begin{lstlisting}[language = Julia]
# Number of months ahead to be forecast
N = 24
# Perform forecasting
pred, dist = forecast(ss, N)
\end{lstlisting}
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/log_airline_forecast.pdf}
	\caption{Log-airline passengers historical data and forecasting.}
	\label{fig:log_airline_forecast}
\end{figure}
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/log_airline_components.pdf}
	\caption{Smoothed trend and seasonal components of the log-airline passengers time series.}
	\label{fig:log_airline_components}
\end{figure}

\subsection{Electricity consumption}

Another practical example is the study of monthly electricity consumption in a given area. In this example, we will use real data from a Brazilian distribution company. In addition, we will consider a temperature series, which is highly correlated to electricity consumption in Brazil, as an exogenous variable. A similar study using \texttt{StateSpaceModels.jl} was conducted in \cite{saavedra2018simulating}.

In this case, our objective is to simulate a large set of future scenarios that can be used as inputs in a stochastic optimization problem with the goal of obtaining the best contracting strategy for a distribution company. To this end, the function \texttt{simulate} will be used to perform Monte Carlo simulation. We will also use the square-root Kalman filter for this application.
%
\begin{lstlisting}[language = Julia]
# Specify the state-space model
model = structural(consumption, 12; X = temperature)
# Estimate the state-space model
ss = statespace(model; filter_type = SquareRootFilter{Float64})
# Number of months ahead to be simulated
N = 24
# Number of scenarios to be simulated
S = 1000
# Perform simulation
sim = simulate(ss, N, S)
\end{lstlisting}

The time series and the resulting simulation can be seen in Fig. \ref{fig:energy_simulation}. The future scenarios are graphically represented by their mean, which is identical to the forecast at each time period, as well as the 5\% and 95\% quantiles.
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/energy_simulation.pdf}
	\caption{Monthly energy consumption historical data and simulation of future scenarios.}
	\label{fig:energy_simulation}
\end{figure}

\subsection{Vehicle tracking}

Finally, in order to illustrate one application that does not fall into any of the predefined models, thus requiring a user-defined model, let us consider an example from control theory. More precisely, we are going to use \texttt{StateSpaceModels.jl} to track a vehicle from noisy sensor data. In this case, $y_t$ is a $2 \times 1$ observation vector representing the corrupted measurements of the vehicle's position on the two-dimensional plane in instant $t$. Since sensors collect the observations with the presence of additive Gaussian noise, we need to filter the observation in order to obtain a better estimate of the vehicle's position.

The position and speed in each dimension compose the state of the vehicle. Let us refer to $x_t^{(d)}$ as the position on the axis $d$ and to $\dot{x}^{(d)}_t$ as the speed on the axis $d$ in instant $t$. Additionally, let $\eta^{(d)}_t$ be the input drive force on the axis $d$, which acts as state noise. For a single dimension, we can describe the vehicle dynamics as
\begin{equation}
\begin{aligned}
    & x_{t+1}^{(d)} = x_t^{(d)} + \Big( 1 - \frac{\rho \Delta_t}{2} \Big) \Delta_t \dot{x}^{(d)}_t + \frac{\Delta^2_t}{2} \eta_t^{(d)}, \\
    & \dot{x}^{(d)}_{t+1} = (1 - \rho) \dot{x}^{(d)}_{t} + \Delta_t \eta^{(d)}_t,
\end{aligned}\label{eq_control}
\end{equation}
where $\Delta_t$ is the time step and $\rho$ is a known damping effect on speed.

We can cast the dynamical system (\ref{eq_control}) as a state-space model in the following manner:
\begin{align*}
    y_t &= \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \alpha_{t+1} + \varepsilon_t, \\
    \alpha_{t+1} &= \begin{bmatrix} 1 & (1 - \tfrac{\rho \Delta_t}{2}) \Delta_t & 0 & 0 \\ 0 & (1 - \rho) & 0 & 0 \\ 0 & 0 & 1 & (1 - \tfrac{\rho \Delta_t}{2}) \\ 0 & 0 & 0 & (1 - \rho) \end{bmatrix} \alpha_{t} + \begin{bmatrix} \tfrac{\Delta^2_t}{2} & 0 \\ \Delta_t & 0 \\ 0 & \tfrac{\Delta^2_t}{2} \\ 0 & \Delta_t \end{bmatrix} \eta_{t},
\end{align*}
where $\alpha_t = (x_t^{(1)}, \dot{x}^{(1)}_{t}, x_t^{(2)}, \dot{x}^{(2)}_{t})^{\top}$ and $\eta_t = (\eta^{(1)}_t, \eta^{(2)}_t)^{\top}$.

\vspace{0.3cm}

We can formulate the vehicle tracking problem in the \texttt{StateSpaceModels.jl} framework as:

\begin{lstlisting}[language = Julia]
# State transition matrix
T = kron(Matrix{Float64}(I, p, p), [1 (1 - ρ * Δ / 2) * Δ; 0 (1 - ρ * Δ)])
# Input matrix
R = kron(Matrix{Float64}(I, p, p), [.5 * Δ^2; Δ])
# Output (measurement) matrix
Z = kron(Matrix{Float64}(I, p, p), [1 0])
# User defined model
model = StateSpaceModel{Float64}(y, Z, T, R)
# Estimate vehicle speed and position
ss = statespace(model)
\end{lstlisting}

In this example, we define the noise variances $H$ and $Q$, generate the noises and simulate a random vehicle trajectory using the state-space equations.
%
\begin{lstlisting}[language = Julia]
# Generate random actuators
Q = .5 * Matrix{Float64}(I, q, q)
η = MvNormal(zeros(q), Q)
# Generate random measurement noise
H = 2. * Matrix{Float64}(I, p, p)
ε = MvNormal(zeros(p), H)
# Simulate vehicle trajectory
α = Matrix{Float64}(undef, n + 1, m)
y = Matrix{Float64}(undef, n, p)
for t in 1:n
    y[t, :] = Z * α[t, :] + rand(ε)
    α[t + 1, :] = T * α[t, :] + R * rand(η)
end
\end{lstlisting}

An illustration of the results can be seen in Fig. \ref{fig:vehicle_tracking}. It can be seen that the measurements are reasonably noisy when compared to the true position. Furthermore, the estimated positions, represented by the smoothed state, effectively estimate the true positions with small inaccuracies.
%
\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{Figures/vehicle_tracking.pdf}
	\caption{Vehicle tracking using a state-space model.}
	\label{fig:vehicle_tracking}
\end{figure}

\section{Conclusion} \label{sec:conclusion}

\texttt{StateSpaceModels.jl} is a flexible package for time-series modeling, forecasting, and simulating that is fully implemented in Julia. The package contains an implementation of the Kalman filter and smoother as well as their square-root variants. Additionally, users have the ability to use any implemented filter or optimization method. Missing observations in the form of \texttt{NaN} values are automatically treated in the filtering and smoothing steps.

Besides comprising several predefined classical models, it is also possible to define any linear model with the \texttt{StateSpaceModel} constructor. Forecasting and Monte Carlo simulation of future scenarios are also available. Finally, the package documentation contains a manual with straightforward examples that are simple to reproduce.

\input{bib.tex}

\end{document}