\chapter{Frequency Selective Filters in DT}

Recall the response of stable DT LTI systems to periodic inputs. Given a stable LTI system with frequency response $H\left(e^{j\omega}\right)$ the input-output relationship is

\[
x[n] = \sum\limits_{k = N_0}^{N_0 + N-1} a_k e^{jk\omega_0n} \longrightarrow y[n] = \sum\limits_{k = N_0}^{N_0 + N-1} a_k H\left(e^{jk\omega_0}\right) e^{jk\omega_0n}
\]

Note the output is equivalent to a signal with Fourier series coefficients $b_k = a_k H\left(e^{jk\omega_0}\right)$. That is the Fourier coefficients are scaled by the frequency response at the harmonic frequency $k\omega_0$.

Similarly for aperiodic signals, given a stable LTI system with frequency response $H\left(e^{j\omega}\right)$ the input-output relationship is
\[
x[n] = \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right) e^{j\omega n} \; d\omega \longrightarrow y[n] = \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right)H\left(e^{j\omega}\right) e^{j\omega n} \; d\omega
\]
Note the output is equivalent to a signal with DT Fourier Transform $Y\left(e^{j\omega}\right) = X\left(e^{j\omega}\right) H\left(e^{j\omega}\right)$. That is the Fourier transform at each continuous frequency $\omega$ is scaled by the frequency response at that frequency.

As in CT, we can design the frequency response to modify the values of $a_k$ or $X\left(e^{j\omega}\right)$ selectively, passing them unmodified, increasing (amplifying) them, or decreasing (attenuating) them. Such systems are called DT filters (or more colloquially digital filters). As in CT there are 4 basic types:
\begin{itemize}
\item Low-pass Filters attenuate high frequencies while passing through lower frequencies. They are often used to reduce the effects of high-frequency noise in a signal.
\item High-pass Filters attenuate lower frequencies while passing through higher frequencies. They are used, for example, to select high-frequency audio components in high-end audio systems.
\item Bandpass Filters attenuate frequencies outside a band of frequencies. They can be viewed as a combination of a high-pass and low-pass filter. They are commonly used to select a range of frequencies for further processing and are central to many communication technologies.
\item Notch or Bandstop Filters attenuate frequencies inside an often narrow band of frequencies. Common applications are the removal of one or more corrupting signals mixed into another signal.
\end{itemize}

While the design of such filters is outside the scope of this course, you are now equipped to understand and apply them based on your knowledge of the Fourier methods covered over the past several weeks. This is similar to CT filtering, with the important exception that the frequency domain is periodic in $2\pi$ for DT systems, so the filter frequency responses are periodic as well, with all the work being done in a range of frequencies from $(0,2\pi)$ or equivalently from $(-\pi,\pi)$.

Digital filters have a number of advantages over CT filters and are widely used now in place of CT filters in audio, communication, and control applications. Audio (and video) in particular is now almost exclusively processed, stored, and transmitted digitally, converting to CT only at the point of an amplifier and speaker.

\section{Ideal Filters}

The above filter types each have an ideal form.

Low-pass filters remove frequency content above a threshold, $\omega_c \in [0,\pi]$, called the \emph{cutoff frequency}. They have an ideal frequency response ,for any integer multiple $k$, given by:
\[
H(j\omega) = \left\{ \begin{array}{lc}
  1 & 2\pi k -\omega_c < \omega < 2\pi k + \omega_c\\
  0 & \text{else}
\end{array}
\right.
\]
with magnitude and phase plot
\begin{center}
  \includegraphics[scale=1]{graphics/lowpass-ideal-dt.pdf}
\end{center}
As in CT filters, the range of frequencies $-\omega_c \leq \omega \leq \omega_c$ are called the pass-band. The range of frequencies outside the pass-band are called the stop-band.

High-pass filters remove frequency content below the cutoff frequency $\omega_c\in [0,\pi]$. They have an ideal frequency response
\[
H(j\omega) = \left\{ \begin{array}{lc}
  0 & 2\pi k -\omega_c < \omega < 2\pi k + \omega_c\\
  1 & \text{else}
\end{array}
\right.
\]
with magnitude and phase plot
\begin{center}
  \includegraphics[scale=1]{graphics/highpass-ideal-dt.pdf}
\end{center}

Bandpass filters remove frequency content outside a band of frequencies called the pass-band. They have an ideal frequency response
\[
H(j\omega) = \left\{ \begin{array}{lc}
  1 & 2\pi k -\omega_0 - \frac{B}{2} < \omega < 2\pi k -\omega_0+\frac{B}{2}\\[1em]
  1 & 2\pi k + \omega_0 -\frac{B}{2} < \omega < 2\pi k + \omega_0+\frac{B}{2}\\[1em]
  0 & \text{else}
\end{array}
\right.
\]
where $\omega_0\in [0,\pi]$ is the \emph{center frequency} and $B < \pi - \omega_0$ is the \emph{bandwidth}. The magnitude and phase plot looks like
\begin{center}
  \includegraphics[scale=1]{graphics/bandpass-ideal-dt.pdf}
\end{center}

Finally, notch or bandstop filters remove frequency content inside a band of frequencies (the stop band) defined by the center frequency $\omega_0\in [0,\pi]$ and bandwidth $B < \pi-\omega_0$. The ideal frequency response is
\[
H(j\omega) = \left\{ \begin{array}{lc}
  0 & -\omega_0 - \frac{B}{2} < \omega < -\omega_0+\frac{B}{2}\\[1em]
  0 & \omega_0 -\frac{B}{2} < \omega < \omega_0+\frac{B}{2}\\[1em]
  1 & \text{else}
\end{array}
\right.
\]
with magnitude and phase plot
\begin{center}
  \includegraphics[scale=1]{graphics/bandstop-ideal-dt.pdf}
\end{center}
Often the bandstop filter has a very narrow bandwidth, thus it "notches" out a frequency component of the input signal.

\section{Practical Filters}

While ideal CT filters cannot be implemented in practice because they are non-causal, this more nuanced in DT systems. We have to make a distinction between \emph{off-line} and \emph{real-time} DT filters. Off-line DT filters, which we will discuss next time, can store as many samples as needed to arbitrarily approximate a non-causal filter, leading to an output that is delayed relative to the input by a significant amount. Real-time filters on the other hand must produce an output $y[n]$ for every input $x[n]$ with no delay. The distinction is important in some applications, controls in particular, and less so in other areas like audio or video where a delay is not noticeable. In the remainder of this lecture we assume a real-time filter implementation, which should be causal.

Practical filters are described by a frequency response that is a ratio of two polynomials in $e^{j\omega}$, i.e.
\[
H\left(e^{j\omega}\right) = \frac{K \cdot\left(e^{j\omega} + b_1\right)\cdot\left(e^{j\omega} + b_2\right)\cdots \left(je^{j\omega} + b_M\right)}{\left(e^{j\omega} + a_1\right)\cdot\left(e^{j\omega} + a_2\right)\cdots \left(e^{j\omega} + a_N\right)}
\]
where $K$ is a constant that controls the gain at DC, and the zero or more complex coefficients $b_k$ and the one or more complex coefficients $a_k$ are called the \emph{zeros} and \emph{poles} of the filter respectively. Such systems correspond to difference equations as we have covered before and are realizable in real arithmetic if all poles and zeros are real or come in conjugate pairs. The processes of designing DT filters consists of choosing the poles and zeros, or equivalently choosing the coefficients of the numerator and dominator polynomials. This is covered in ECE 3704, ECE 4624, and other upper-level courses.

The general DT frequency response corresponds to a difference equation that when written in recursive form looks like
\[
y[n] = \underbrace{-\frac{c_{N+1}}{c_1} y[n-N] -\frac{c_{N}}{c_1}  y[n-N+1] + \cdots -\frac{c_{2}}{c_1} y[n-1]}_{\text{auto-regressive}} + \underbrace{\frac{d_{N+1}}{c_1} x[n-N] + \frac{d_{N}}{c_1} x[n-N+1] + \cdots + \frac{d_1}{c_1} x[n]}_{\text{moving-average}}
\]
The terms corresponding to the weighted sums of previous outputs are called the \emph{auto-regressive} portion of the filter. The terms corresponding to the weighted sums of previous inputs are called the \emph{moving-average} portion of the filter.

Filters without auto-regressive terms ($c_i = 0$ for $i > 1$) are called \emph{finite impulse response} (FIR) filters, because their impulse response has only a finite number of non-zero values. Filters with auto-regressive terms are called \emph{infinite impulse response} (IIR) filters, because their impulse response is non-zero for $n > 0$ (although they do approach zero as $n\rightarrow \infty$).

Practical (real-time) DT filters differ from the ideal in that they cannot be zero over any finite range of frequencies and cannot transition discontinuously between stop and pass bands. Similar to CT filters, they must vary over the bands and transition smoothly, with a degree of variation and sharpness that is a function of the order of the filter and the exact form of the frequency response polynomials. Thus practical filters are described by additional parameters that define the stop and pass-bands.

The overall gain of the filter is the magnitude of the frequency response at a frequency that depends on the filter type, zero for a low-pass filter and the center frequency for a band-pass filter. The pass-band is defined by the frequency at which the magnitude of the frequency response drops below the overall gain, often -3dB = $\frac{\sqrt{2}}{2}$. The stop-band is defined similarly, as the frequency at which the magnitude of the frequency response drops further below the overall gain, often -20dB = 0.1 or -40dB = 0.01. The \emph{transition bandwidth} is defined as the difference in the stop-band and pass-band frequencies. The \emph{pass-band ripple} is defined as the maximum deviation from the overall gain, over the pass-band. For DT filters the frequencies are specified in radians per sample. After we discuss sampling we will see how to convert this to an equivalent CT frequency using the sample time.

\section{First-order and second-order systems as filters}

Given the equivalence of stable LTI systems and linear, constant-coefficient difference equations, block diagrams, impulse responses, and frequency responses, filters can be represented in any of these ways. We have covered extensively first-order and second-order DT systems and seen how they can be represented variously as circuits, difference equations, block diagrams, and as frequency responses. We now see how they can describe simple filters and serve as building blocks for higher-order filters.

\begin{example} Consider a first-order DT system
\[
y[n+1] + ay[n] = b x[n+1]
\]
It can be represented as a block diagram
\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
      minimum height=2em, minimum width=2em]
  \begin{tikzpicture}[auto]
    \node [input, name=input] at (0,0) {};
    \node [left of=input] {$x[n]$};
    \node [sum, right of=input, node distance=5em] (sum) {$\Sigma$};
    \node[block] at (3,-1) (block2) {$D$};
    \node [shape=coordinate, name=conn] at (4,0) {};
    \node[shape=coordinate] at (5,0) (output) {};
    \node [right of=output] {$y[n]$};

    \draw [->] (input) -- node {$b$} (sum);
    \draw (sum) -- (conn);
    \draw [->] (conn) -- (output);
    \draw [->] (conn) |- (block2);
    \draw [->] (block2) -| node {$-a$} (sum);
  \end{tikzpicture}
\end{center}
or as a recursive difference equation
\[
y[n] = -ay[n-1] + bx[n]
\]
or as an impulse response
\[
h[n] = b(-a)^n u[n]
\]
or as a frequency response if $|a| < 1$
\[
H\left(e^{j\omega}\right) = \frac{b}{1+ae^{-j\omega}} = \frac{be^{j\omega}}{e^{j\omega}+a}
\]

Let us examine two cases, where $a = \tfrac{1}{2}, b=1-a = \tfrac{1}{2}$ and $a = -\tfrac{1}{2}, b=1+a = \tfrac{3}{2}$. If we plot the frequency response when $a = \tfrac{1}{2}, b=\tfrac{1}{2}$ we see the DC gain is about -9.5 dB, and the response passes through -3dB at $\pm 2.4$ rad/sample. Thus this corresponds approximately to a high-pass filter.
\begin{center}
  \includegraphics[scale=0.7]{graphics/lecture25_1.png}
\end{center}
If we plot the frequency response when $a = -\tfrac{1}{2}, b=\tfrac{3}{2}$ we see the DC gain is 0 dB, and the response passes through -3dB at $\pm 0.7$ rad/sample. Thus this corresponds approximately to a low-pass filter.
\begin{center}
  \includegraphics[scale=0.7]{graphics/lecture25_2.png}
\end{center}

\end{example}

\begin{example} As with CT filters we can increase the sharpness of the filter by increasing the order. Consider a second-order DT system
\[
c_1 y[n+2] + c_2y[n+1] + c_3y[n] = d_1x[n+2] + d_2x[n+1] + d_3x[n]
\]
It can be represented as a block diagram
\begin{center}
  \tikzstyle{block} = [draw, fill=gray!20, rectangle,
      minimum height=2em, minimum width=2em]
  \begin{tikzpicture}[auto]
    \node [input, name=input] at (-2,0) {};
    \node [left of=input] {$x[n]$};
    \node [shape=coordinate, name=conn0] at (0,0) {};
    \node [sum] at (2,0) (sum1) {$\Sigma$};
    \node [block] at (4,-1) (block1) {$D$};
    \node [block] at (4,-3) (block2) {$D$};
    \node [block] at (0,-1) (block3) {$D$};
    \node [block] at (0,-3) (block4) {$D$};
    \node [shape=coordinate, name=conn1] at (4,-2) {};
    \node [shape=coordinate, name=conn2] at (4,-4) {};
    \node [sum] at (2,-2) (sum2) {$\Sigma$};
    \node [sum] at (2,-4) (sum3) {$\Sigma$};
    \node [shape=coordinate, name=conn] at (4,0) {};
    \node [shape=coordinate] at (5,0) (output) {};
    \node [right of=output] {$y[n]$};
    \node at (3,-3.7) {$-\frac{c_3}{c_1}$};
    \node at (3,-1.7) {$-\frac{c_2}{c_1}$};
    \node at (1,-1.7) {$\frac{d_2}{c_1}$};
    \node at (1,-3.7) {$\frac{d_3}{c_1}$};
    \node at (1,0.3) {$\frac{d_1}{c_1}$};

    \draw (input) -- (conn0);
    \draw [->] (conn0) -- (sum1);
    \draw [->] (conn0) -- (block3);
    \draw [->] (block3) -- (block4);
    \draw [->] (block3) |- (sum2);
    \draw [->] (block4) |- (sum3);
    \draw (sum1) -- (conn);
    \draw [->] (conn) -- (output);
    \draw [->] (conn) -- (block1);
    \draw (block1) -- (conn1);
    \draw [->] (conn1) -- (sum2);
    \draw [->] (conn1) -- (block2);
    \draw [->] (block2) |- (sum3);
    \draw [->] (sum3) -- (sum2);
    \draw [->] (sum2) -- (sum1);
  \end{tikzpicture}
\end{center}
or as a recursive difference equation
\[
y[n] = -\frac{c_3}{c_1} y[n-2] - \frac{c_2}{c_1} y[n-1] + \frac{d_3}{c_1} x[n-2] + \frac{d_2}{c_1} x[n-1] + \frac{d_1}{c_1} x[n]
\]
or as a frequency response if $|\gamma_i| < 1$ for all $i$ where $\gamma_i$ are the roots of the characteristic equation\\ $c_1 E^2 + c_2 E + c_3 = 0$.
\[
H\left(e^{j\omega}\right) = \frac{d_1 e^{j2\omega} + d_2 e^{j\omega} + d_3}{c_1 e^{j2\omega} + c_2 e^{j\omega} + c_3}
\]
As a concrete example, consider this system when c = [$c_1 = 1$, $c_2 = 0$, $c_3 = 0.1716$, $d_1 = 0.2929$, $d_2 = 0.5858$, and $d_3 = 0.2929$. If we plot the frequency response we see the filter gain is 0 dB at DC and passes through -3 dB at approximately $\pm 1.57$ rad/sample. Thus it corresponds to a low-pass filter.
\begin{center}
  \includegraphics[scale=0.7]{graphics/lecture25_3.png}
\end{center}
One thing to note is the attenuation drops off quickly after the passband. This is an advantage of DT filters; they can have small transition bands. Comparing them directly to an equivalent CT filter will have to be deferred until we discuss sampling.

Note the autoregressive part of the filter is to the right of the summations, while the moving average part is to the left. The output of the delay blocks multiplied by non-zero coefficients are called \emph{filter taps} in signal processing parlance. This repeating structure can be taken advantage of in the creation of general-purpose digital signal processing hardware.
\end{example}

\section{Higher-Order Filters}

We can continue to increase the steepness of the passband to stop-band transitions by increasing the order of the filter. While this can be accomplished using a serial connection of stages as in CT filters, since the implementation of DT filters requires just memory (delay blocks) and adders/multipliers it is common to just implement the filter with a larger number of delay blocks.

\newpage
Matlab code for plotting the first-order frequency response plot:
\begin{verbatim}
a = 1/2;
w = -pi:0.01:pi;
H = a./(exp(j*w) + a);

% Create a nice FR plot
hFig = figure();
hold on;

subplot(2,1,1);
hm = plot(w,20*log10(abs(H)));
grid on;
axis tight;
hTitle  = title ('Frequency Response - first order a = 0.5');
hYLabel1 = ylabel('Magnitude (dB)');
set(gca, 'FontSize', 14, 'YTick', -20:5:5, 'LineWidth', 2);

subplot(2,1,2);
hp = plot(w,unwrap(angle(H)));
grid on;
axis tight;
hYLabel2 = ylabel('Phase (radians)');
hXLabel = xlabel('Frequency (rad/sample)');
set(gca, 'FontSize', 14, 'LineWidth', 2);

set(hm, 'linewidth', 2);
set(hp, 'linewidth', 2);

set([hXLabel, hYLabel1, hYLabel2]  , ...
     'FontSize'   , 14          );
set( hTitle                    , ...
     'FontSize'   , 14          , ...
     'FontWeight' , 'bold'      );
\end{verbatim}

\newpage
Matlab code for plotting the second-order frequency response plot:
\begin{verbatim}
c = [1,0,0.1716];
d = [0.2929, 0.5858, 0.2929];
w = -pi:0.01:pi;
H = (d(1)*exp(j*2*w) + d(2)*exp(j*w) + d(3))./(c(1)*exp(j*2*w) + c(2)*exp(j*w) + c(3));

% Create a nice FR plot
hFig = figure();
hold on;

subplot(2,1,1);
hm = plot(w,20*log10(abs(H)));
grid on;
axis tight;
hTitle  = title ('Frequency Response - second order');
hYLabel1 = ylabel('Magnitude (dB)');
set(gca, 'FontSize', 14, 'YTick', -100:20:0, 'LineWidth', 2);

subplot(2,1,2);
hp = plot(w,unwrap(angle(H)));
grid on;
axis tight;
hYLabel2 = ylabel('Phase (radians)');
hXLabel = xlabel('Frequency (rad/sample)');
set(gca, 'FontSize', 14, 'LineWidth', 2);

set(hm, 'linewidth', 2);
set(hp, 'linewidth', 2);

set([hXLabel, hYLabel1, hYLabel2]  , ...
     'FontSize'   , 14          );
set( hTitle                    , ...
     'FontSize'   , 14          , ...
     'FontWeight' , 'bold'      );
\end{verbatim}


\subsection{Decomposition of U}
\SecLabel{u}

Let $\alpha\colon \field{4} \to \field{4}$ be given by
$$
\alpha(x) \eqdef U_{x}(0)
$$
and let $U'_{\hex{0}},\ldots,U'_{\hex{f}}$ be permutations of $\field{4}$ given by
$$
U'_k(x) \eqdef U_k(x) \oplus \alpha(k).
$$
It follows that for all $k \in \field{4}$, $U'_k(0) = 0$. The codebook of $U'$ is given in \FigRef{up}.

\FigTex{up-m.tex}

In~\cite{LinAffEQ}, Biryukov~\etal{} propose efficient algorithms for checking affine and linear equivalence of permutations. Applying these algorithms to the permutations $U'_i$ shows that all $U'_i$ are pairwise linear equivalent. Furthermore, they differ only by a linear layer in the output. Formally, let $M_{\hex{0}}, \ldots, M_{\hex{f}}\colon \field{4} \to \field{4}$ be given by
$$
M_k(x) \eqdef U'_k(x) \circ U'^{-1}_{\hex{0}}(x).
$$
Then, each $M_i$ is linear. The codebook of $M$ is given in \FigRef{m}.

\begin{remark}
In the hindsight, it could be trivially checked that $U_i \circ U_j^{-1}$ is linear for all $i,j \in \field{4}$. However, it is not always clear which properties or relations can be expected. For this reason, the linear/affine equivalence algorithms from~\cite{LinAffEQ} and their improved variants by Dinur~\cite{LinAffEQ2} are very useful tools for S-Box reverse-engineering.
\end{remark}

The next step is to observe that the functions $M_i$ have two interesting properties:
\begin{enumerate}
    \item the functions $M_i$ have orders $1,3,5,15$; those with order $15$ generate all $M_i$;
    \item the functions $M_i$ are linearly related: they are contained in linear subspace of dimension 4;
\end{enumerate}
These properties point towards a finite field structure. Let $b \eqdef M_{\hex{0}} \oplus M_{\hex{5}}$. Then $b$ is \emph{linear-similar} to the multiplication by $X$ in the finite field
$$
\fielde{4} \simeq \field{}[X]/(X^4 + X^3 + 1).
$$
By ``linear-similar'' it is meant that
$$
b = l \circ (\cdot~\fmult X) \circ l^{-1}
$$
for some linear bijection $l \in \linbij{4}$, where $(\cdot~\fmult X)$ denotes the multiplication in the finite field by $X$. In this case, $l = l' = \swaplsb$, where
$$
\swaplsb\colon \field{4} \to \field{4}, \swaplsb(x_1,x_2,x_3,x_4) \eqdef (x_1,x_2,x_4,x_3).
$$
Among all choices of $b$ and the field defining polynomial, this choice results in the simplest mapping $l$.

Note that similarity is preserved for powers, i.e. $b^i = \swaplsb \circ (\cdot~\fmult X^i) \circ \swaplsb$. It follows that for all $k \in \field{4}$, $\swaplsb \circ M_k \circ \swaplsb$ is the finite field multiplication by the power of $X$ equal to the discrete logarithm of $M_k$ base $b$. More precisely, let $\gamma\colon \field{4} \to \field{4}$ be such that
$$
\gamma(k) \eqdef (\swaplsb \circ M_k \circ \swaplsb)(1).
$$
Furthermore, let $\beta\colon \field{4} \to \field{4}$ be given by
$$
\beta(x) \eqdef U'_{\hex{0}}(\swaplsb(x)) \oplus \hex{c}.
$$
Then $U$ can be decomposed as follows:
$$
U_k(x) = \alpha(k) \oplus \beta(\gamma(k) \fmult \swaplsb(x)) \oplus \hex{c}.
$$

Note that $\alpha$ is affine such that $\alpha \oplus \hex{c}$ is linear:
$$
\alpha(x_1,x_2,x_3,x_4) = (1, x_2 \oplus 1, x_1, 0).
$$
The constant part of $\alpha$ cancels with the constant from $\beta$ and the linear part can be merged with the outer linear encoding $L$.
The graphical representation of the final decomposition of $U$ and the codebooks of $\alpha, \beta, \gamma$ and $\swaplsb$ are given in \FigRef{final-u}.

\FigTex{final-u.tex}


\section{Kernels
\hfill\normalfont $\color{section-text-color}\widehat{=} \text{ \sffamily scalar prod. in feature space } \phi$}

\textbf{K. trick:}\;
$\bm{x}_i\!^\top \bm{x}_j \overset{\text{Mercer}}{\rightsquigarrow}$
\highlight*{$ k(\bm{x}_i, \bm{x}_j) = \phi(\bm{x}_i)\!^\top \phi(\bm{x}_j)$}

% ===
\emph{Properties of kernel}

$k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ is an inn. prod. (symm., pos.-def.).

Need: \enskip \highlight{$K \succeq 0 \;\forall \bm{x}_i$}, \enskip where $K_{i,j} = k(\bm{x}_i,\bm{x}_j)$

Hence:
\enskip $\bullet$ check pos. eigenvalues \textit{or better} \par
\enskip $\bullet$ $vKv^\top = \sum_i\sum_j \alpha_i\alpha_j\, k(\bm{x}_i,\bm{x}_j) \geq 0$ \par
\enskip $\to$ check for $i\!=\!j\!=\!1$ for \textit{counter example}

% ===
\emph{Important kernels}
{\setstretch{0.9}
\begin{tabular}{@{}>{\normalcolor}l l}
	Constant:	& $k(x,y)=c \text{ \normalcolor with } c \geq 0$\\
	Linear: 	& $k(x,y)=x^\top y$\\
	Polynomial:	& $k(x,y)=(x^\top y + 1)^d$\\
	Gaussian:	& $k(x,y) = \exp(-\norm{x-y}_2^2/h^2)$\\
	Laplacian:	& $k(x,y) = \exp(-\norm{x-y}_1/h)$
\end{tabular}
}

% ===
\emph{Composition rules}\\
$\circ$ $k=k_1+k_2$
\enskip
$\circ$ $k=c\cdot k${\footnotesize $\color{gray},\; c>0$}
\enskip
$\circ$ $k=k_1\!\cdot\! k_2$

$\circ$ $k=f(k_1)${\footnotesize \color{gray}, $f$: exp. \textit{or} polyn. with \underline{all} pos. coeff.}

% ===
\emph{Kernelized Perceptron / \color{OrangeRed} SVM}
\vspace{-1pt}

\textbf{Ansatz:} $w^* \in \operatorname{span}(X) \Rightarrow$ \highlight*{$w = \sum_{j=1}^n \alpha_j y_j x_j$}

\vspace{-1pt}
{\small $\alpha^\ast {=}
\arg\min\limits_\alpha \frac{1}{n}\sum_n \max\{ 0, {\color{OrangeRed}1}{-}y_i\alpha^\top k_i \} {\color{OrangeRed}+ \lambda\alpha^\top \bm{D}_y\bm{K}\bm{D}_y \alpha}$}

$\color{gray} \text{with } k_i\!=\![.., y_j\,k(x_i,x_j), ..] \text{ and } D_y\!=\!\operatorname{diag}(y_i)$

\textbf{Predict:} $\hat y = \operatorname{sign}(\sum_{i=1}^n \alpha_iy_ik(x_i, x))$

% ===
\emph{Kernelized linear regression (KLR)}

\textbf{Ansatz:} \highlight*{$w=\sum_{j=1}^n \alpha_j x_j$} $= \sum_{j=1}^n \alpha_j\phi(x_j)$

$\alpha^\ast =
\arg\min\limits_\alpha \frac{1}{n} \norm{\alpha^\top K-y}_2^2 + \lambda\alpha^\top K\alpha$
\vspace{-1pt}

\textbf{closed form:} $\alpha^\ast = (K+\lambda I)^{-1}y$

\textbf{Predict:} $\hat y = \sum_{i=1}^n \alpha_i k(x_i,x)$

% ===
\emph{Kernelized LogReg}

$\hat\alpha \!=\! \arg\!\min\limits_\alpha \sum_{i=1}^n \log(1\!+\!\exp(-y_i\alpha^\top\! K_i)) \!+\! \lambda\alpha^\top\! K\alpha$\vspace{-3pt}\\
$P(y\vert x,\hat\alpha) = \big( 1+\exp(-y\sum_{j=1}^n\alpha_j k(x_j,x)) \big)^{-1}$

% ===
\emph{Semi-parametric kernel}

additive combination of linear and non-linear kernel fct's, e.g. $x\leftrightarrow \sin(x\!\cdot\!\gamma)$ {\small\color{gray} ``periodic'' kernel}


\section{$n$-step Bootstrapping}
$n$-step methods allow us to observe multiple time-steps of returns before updating a state with the observed data and a bootstrapped estimate of the value of the $n$th succeeding state.

\subsection{$n$-step TD Prediction}
Define the $n$-step return
\begin{equation}
    G_{t:t+n} \doteq \sum_{i=t}^{t+n-1}\gamma^{i-t}R_{i+1} + \gamma^n V_{t+n-1}(S_{t+n})
\end{equation}
where $n \geq 1$, $0 \leq t < T - n$ and $V_i$ is the estimated state-value function as of time $i$. If $t + n > T$ then $G_{t+n} \equiv G_t$, the standard return. The $n$-step return is the target for \emph{$n$-step TD} methods, note that $n-1$ rewards are observed and the succeeding value is bootstrapped with the latest estimate of the value function. The corresponding update for state-values is
\begin{equation}
    V_{t+n}(S_t) = V_{t+n - 1}(S_t) + \alpha [G_{t:t+n} - V_{t+n-1}(S_{t})]  \quad\quad 0 \leq t < T.
\end{equation}
Note that Monte-Carlo can be thought of as TD($\infty$)Pseudocode for $n$-step TD is given in the box below.\\

\includegraphics[width=\textwidth]{\NotesImages/n_step_td_state_values.png}\\

The $n$-step return obeys the \emph{error-reduction property}, and because of this $n$-step TD can be shown to converge to correct predictions (given a policy) under appropriate technical conditions. This property states that the $n$-step return is a better estimate than $V_{t+n-1}$ in the sense that the error on the worst prediction is always smaller
\begin{equation}
    \max_s \left|\Epi[G_{t:t+n}|S_t=s] - v_\pi(s)\right| \leq \gamma^n \max_s \left|V_{t+n-1}(s) - v_\pi(s)\right|
\end{equation}

\subsection{$n$-step Sarsa}
\subsubsection*{Sarsa}
We develop $n$-step methods for control. We generalise Sarsa to $n$-step Sarsa, or Sarsa($n$). This is done in much the same way as above, but with action-values as opposed to state-values. The $n$-step return in this case is defined as
\begin{equation}
    G_{t:t+n} \doteq \sum_{i=t}^{t+n-1}\gamma^{i-t}R_{i+1} + \gamma^n Q_{t+n-1}(S_{t+n}, A_{t+n})
\end{equation}
where $n \geq 1$, $0 \leq t < T - n$ and $Q_i$ is the estimated action-value function as of time $i$. If $t + n > T$ then $G_{t+n} \equiv G_t$, the standard return. The corresponding update is
\begin{equation}
    Q_{t+n}(S_t, A_t) = Q_{t+n-1}(S_t, A_t) + \alpha [G_{t:t+n} - Q_{t+n-1}(S_{t}, A_{t})]  \quad\quad 0 \leq t < T.
\end{equation}

\subsubsection*{Expected Sarsa}
We define $n$-step expected Sarsa similarly
\begin{equation}
    G_{t:t+n} \doteq \sum_{i=t}^{t+n-1}\gamma^{i-t}R_{i+1} + \gamma^n \bar{V}_{t+n-1}(S_{t+n})
\end{equation}
where $n \geq 1$, $0 \leq t < T - n$ and $\bar{V}_i$ is the \emph{expected approximate value} of state $s$
\begin{equation}
    \bar{V}_i(s) \doteq \sum_a \pi(a|s)Q_i(s, a).
\end{equation}
As always, if $t + n > T$ then $G_{t+n} \equiv G_t$, the standard return. The corresponding update is formally the same as above\\

\includegraphics[width=\textwidth]{\NotesImages/n_step_sarsa.png}\\

\subsection{$n$-step Off-policy Learning}
We can learn with $n$-step methods off-policy using the importance sampling ratio (target policy $\pi$ and behaviour policy $b$)
\[
    \rho_{t:h} \doteq \prod_{k=t}^{\text{min}(h, T-1)}\frac{\pi(A_k|S_k)}{b(A_k|S_k)}.
\]
For state-values we have
\[
    V_{t+n}(S_t) \doteq V_{t+n-1}(S_t) + \alpha \rho_{t:t+n-1}[G_{t:t+n} - V_{t+n-1}(S_t)]
\]
and for action-values we have
\[
    Q_{t+n}(S_t, A_t) = Q_{t+n-1}(S_t, A_t) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_t, A_t)]
\]
note that for action values the importance sampling ratio starts one time-step later, because we are attempting to discriminate between actions at time $t$.\\

\includegraphics[width=\textwidth]{\NotesImages/off_policy_n_step_sarsa.png}\\

\subsection{*Per-decision Methods with Control Variates}
We have the standard recursion relation for the $n$-step return
\[
    G_{t:h} = R_{t+1} + \gamma G_{t+1:h}.
\]
For an off-policy algorithm, one would be tempted to simply weight this target by the importance sampling ratio. This method, however, shrinks the estimated value functions when the importance sampling ratio is 0, hence increasing variance. We thus introduce the \emph{control-variate} $(1 - \rho_t)V_{h-1}(S_t)$, giving an off-policy update of
\[
    G_{t:h} = \rho_t (R_{t+1} + \gamma G_{t+1:h}) + (1 - \rho_t)V_{h-1}(S_t)
\]
where $G_{h:h} = V_{h-1}(S_h)$. Note that the control-variate has expected value 0, since the factors are uncorrelated and the expected value of the importance sampling ratio is 1.\\

We can do a similar thing for action-values
\[
    G_{t:h} \doteq R_{t+1} + \gamma \rho_{t+1:h}\left( G_{t+1:h} - Q_{h-1}(S_{t+1}, A_{t+1}) \right) - \gamma \bar{V}_{h-1}(S_{t+1}),
\]
where once again the importance sampling ratio starts one time-step later.

\subsubsection*{Control Variates in General}
Suppose we want to estimate $\mu$ and assume we have an unbiased estimator for $\mu$ in $m$. Suppose we calculate another statistic $t$ such that $\mathbb{E}\left[t\right]=\tau$ is a known value. Then
\[
    m^\star = m + c\left(t-\tau\right)
\]
is also an unbiased estimator for $\mu$ for any $c$, with variance
\[
    \textrm{Var}\left(m^{\star}\right)=\textrm{Var}\left(m\right) + c^2\,\textrm{Var}\left(t\right) + 2c\,\textrm{Cov}\left(m,t\right).
\]

It is easy to see that taking

\[
    c = - \frac{\textrm{Cov}\left(m,t\right)}{\textrm{Var}\left(t\right)}
\]
minimizes the variance of $m^{\star}$. With this choice

\begin{align}
\textrm{Var}(m^{\star}) & =\textrm{Var}(m) - \frac{\left[\textrm{Cov}(m,t)\right]^2}{\textrm{Var}(t)} \\
& = (1-\rho_{m,t}^2)\textrm{Var}(m)
\end{align}
where $\rho_{m,t}=\textrm{Corr}\left(m,t\right) $ is the Pearson correlation coefficient of $m$ and $t$. The greater the value of $|\rho_{m,t}|$, the greater the variance reduction achieved.

\subsection{Off-policy Learning Without Importance Sampling: The $n$-step Tree Backup Algorithm}
We introduce the $n$-step \emph{tree-backup algorithm} algorithm using the return
\begin{equation}
    G_{t:t+n} \doteq R_{t+1} + \gamma \sum_{a \neq A_{t+1}} \pi(a|S_{t+1})Q_{t+n-1}(S_{t+1}, a) + \gamma \pi(A_{t+1}|S_{t+1})G_{t+1:t+n}
\end{equation}
for $t < T-1$, $n > 1$ and with $G_{i:i} = 0$ and $G_{T-1:t+n} = R_T$. This algorithm updates $S_t$ with bootstrapped, probability weighted action-values of \emph{all} actions that were not taken all along the trajectory and recursively includes the rewards realised, weighted by the probability of their preceding actions under the policy. Pseudocode given below.\\

\includegraphics[width=\textwidth]{\NotesImages/n_step_tree_backup.png}\\

\subsection{*A Unifying Algorithm: $n$-step $Q(\sigma)$}
We introduce an algorithm which, at each time step, can choose to either take an action as a sample as in Sarsa or to take an expectation over all possible actions as in tree-backup. \\

Define a sequence $\sigma_t \in [0, 1]$ that at each time step chooses a proportion of sampling vs. expectation. This generalises Sarsa and tree-backup by allowing each update to be a linear combination of the two ideas. The corresponding return (off-policy) is
\begin{align}
    G_{t:h} \doteq R_{t+1} &+ \gamma \left(\sigma_{t+1}\rho_{t+1}  (1- \sigma_{t+1})\pi(A_{t+1}\vert S_{t+1})\right) \left( G_{t+1:h} - Q_{h-1}(S_{t+1}, A_{t+1}) \right) \\
                           &+ \gamma \bar{V}_{h-1}(S_{t+1}),
\end{align}
for $t < h< T$, with $G_{h:h} \doteq Q_{h-1}(S_h, A_h)$ if $h<T$ and $G_{T-1:T} \doteq R_t$ if $h=T$. Pseudocode given below.\\

\includegraphics[width=\textwidth]{\NotesImages/off_policy_n_step_Q_sigma.png}\\


\graphicspath{{Chapter2/Figs/}}

\section{MOFA Model description} \label{mofa:model_description}

\subsection{Mathematical formulation}

Multi-Omics Factor Analysis (MOFA) is a multi-view generalisation of conventional Factor Analysis to an arbitrary number of $M$ data modalities (or views). It is inspired from the Group Factor Analysis framework discussed in \Cref{section:gfa}.

The input data consists of $M$ views $\bfY^m \in \R^{N \times D_m}$ with non-overlapping features. Views, or data modalities, often represent different assays, but there is flexibility in their definition. Formally, the input data is factorised as:
\begin{equation} \label{mofa_master_equation}
	\mathbf{Y}^m = \mathbf{Z}(\mathbf{W}^{m})^{T} + \bepsilon^m
\end{equation}
where $\bfZ \in \R^{N \times K}$ is a matrix that contains the factor values and $\bfW^m \in \R^{D_m \times K}$ are a set of $M$ matrices (one per view) that contain the feature weights. Finally, $\bepsilon^m \in \R^{D_m}$ captures the residuals, or the noise, which is assumed to be normally distributed and heteroskedastic:
\begin{equation}
	p(\epsilon^{m}_{d}) = \Ndist{\epsilon^{m}_{d}}{0,(\tau_{d}^{m})^{-1}}
\end{equation}
where $\tau$ corresponds to the precision (inverse of the variance). Altogether, this results in the following likelihood:
\begin{equation}
	p(\bfY|\bfW,\bfZ,\bTau) = \prod_{m=1}^{M} \prod_{d=1}^{D_m} \prod_{n=1}^{N} \Ndist{y_{nd}^m}{\bfz_{n}^T\bfw_{d}^{m},(\tau_d^m)^{-1}}
	% p(y_{nd}^m) = \Ndist{y_{nd}^m}{\bfz_{n,:}\bfw_{d,:}^{mT},1/\tau_d^m},
\end{equation}

Non-Gaussian noise models can also be defined (see \Cref{section:mofa_ngaussian}), but unless otherwise stated, I will always assume Gaussian residuals.

\subsubsection{Prior distributions for the factors}  \label{section:mofa_factors}

For the factors, we can define an isotropic Gaussian prior, as commonly done in most factor analysis models:
\begin{equation}
	p(z_{nk}) = \Ndist{z_{nk}}{0,1}
\end{equation}
This effectively  assumes (1) a continuous latent space and (2) independence between samples and factors.

\subsubsection{Prior distributions for the weights}  \label{section:mofa_weights}

The key determinant to ensure that the model is interpretable lies on the regularization structure imposed on the weights. Here we encode two levels of sparsity on their prior distributions, (1) a view- and factor-wise ARD prior \cite{Mackay1996} and (2) a feature-wise spike-slab prior \cite{Mitchell1988}:
\begin{equation*}
	p(w_{dk}^{m}) = (1-\theta_{k}^{m}) \mathds{1}_0(w_{dk}^{m}) + \theta_{k}^{m} \Ndist{w_{dk}^{m}}{0, 1 / \alpha_{k}^{m}}
\end{equation*}
The aim of the ARD prior is to disentangle the activity of factors to the different views, such that the weight vector $\bfw_{:,k}^m$ is shrunk to zero if the factor $k$ does not explain any variation in view $m$. The spike-and-slab prior encourages zero values within active factors at the level of individual features.

However, the standard formulation of the spike-and-slab prior contains a Dirac delta function, which is incompatible with the variational inference scheme. To solve this we adopt a re-parametrization of the weights $w$ as a product of a Gaussian random variable $\hat{w}$ and a Bernoulli random variable $s$, \cite{Titsias2011} resulting in the following prior distribution:
\begin{equation}
	p(\hat{w}_{dk}^{m},s_{dk}^{m}) = \Ndist{\hat{w}_{dk}^{m}}{0, \frac{1}{\alpha_k^m}}  \text{Ber}(s_{dk}^m \,|\,\theta_k^m)
\end{equation}
In this formulation $\alpha_k^m$ controls the activity of factor $k$ in view $m$ and $\theta_k^m$ controls the corresponding fraction of non-zero weights (i.e. the sparsity levels).\\

Finally, we define conjugate priors for $\theta$ and $\alpha$:
\begin{align}
	p(\theta_k^m) &= \Bdist{\theta_k^m}{a_0^\theta,b_0^\theta}\\
	p(\alpha_k^m) &= \Gdist{\alpha_k^m}{a_0^\alpha, b_0^\alpha}
\end{align}
with hyper-parameters $a_0^\theta,b_0^\theta =1$ and $a_0^\alpha, b_0^\alpha=1e^{-5}$ to get uninformative priors. Posterior values of $\theta_k^m$ close to $0$ implies that most of the weights of factor $k$ in view $m$ are shrunk to $0$ (sparse factor). In contrast, a value of $\theta_k^m$ close to $1$ implies that most of the weights are non-zero (non-sparse factor). A small value of $\alpha_k^m$ implies that factor $k$ is active in view $m$. In contrast, a large value of $\alpha_k^m$ implies that factor $k$ is inactive in view $m$.

All together, the joint probability density function of the model is given by
\begin{align}
	\begin{split}
	p(\bfY,\hat{\bfW},\bfS,\bfZ,\btheta, \balpha, \btau)  = &\prod_{m=1}^{M} \prod_{n=1}^{N} \prod_{d=1}^{D_m} \Ndist{y_{nd}^m}{\sum_{k=1}^{K} s_{dk}^m \hat{w}_{dk}^m z_{nk},1/\tau_d} \\
	& \prod_{m=1}^{M}\prod_{d=1}^{D_m} \prod_{k=1}^{K} \Ndist{\hat{w}_{dk}^m}{0,1/\alpha_k^m} \text{Ber}(s_{d,k}^m|\theta_k^m) \\
	& \prod_{n=1}^{N} \prod_{k=1}^{K} \Ndist{z_{nk}}{0,1} \\
	& \prod_{m=1}^{M} \prod_{k=1}^{K} \Bdist{\theta_k^m}{a_0^\theta,b_0^\theta} \\
	& \prod_{m=1}^{M} \prod_{k=1}^{K} \Gdist{\alpha_k^m}{a_0^\alpha, b_0^\alpha} \\
	& \prod_{m=1}^{M} \prod_{d=1}^{D_m} \Gdist{\tau_d^m}{a_0^\tau,b_0^\tau}.
	\label{likelihood}
	\end{split}
\end{align}
and the corresponding graphical model is shown below:

\begin{figure}[H]
 	\begin{center}
	\input{graphical_models/mofa}
	% \label{fig:MOFA_graphical_model}
	\caption{\textbf{Graphical model for MOFA}. Grey circles represent the observed variables whereas white circles represent hidden variables that are infered by the model. Each plate repsents a dimension of the model: $M$ for the number of views, $N$ for the number of samples, $K$ for the number of factors and $D_m$ for the number of features in the $m$-th view. The use of transparency in the top left nodes is intentional and becomes clear in Chapter 5.}
	\end{center}
\end{figure}

This completes the definition of the MOFA model.

\subsubsection{Inference}

To make the model scalable to large datasets we adopt a Variational inference framework with a structured mean field approximation. A detailed overview is given in \Cref{section:variational_inference}, and details on the variational updates for the MOFA model are given in Appendix~\ref{appendix:mofa}. To enable efficient inference for non-Gaussian likelihoods we employ local bounds \cite{Jaakkola2000,Seeger2012}. This is described in detail in \Cref{section:mofa_ngaussian}.

\subsubsection{Missing values} \label{section:mofa_missing_values}

The probabilistic formulation naturally accounts for incomplete data matrices, as missing observations do not intervene in the likelihood. In practice, we implement this using memory-efficient binary masks $\mathcal{O}^m \in \mathbb{R}^{N\times D_m}$ for each view $m$, such that $\mathcal{O}_{n,d} = 1$ when feature $d$ is observed for sample $n$, 0 otherwise.

\subsection{Downstream analysis} \label{mofa:downstream}

Once trained, the MOFA model can be queried for a set of downstream analysis (\Cref{fig:MOFA}):
\begin{itemize}
	\item \textbf{Variance decomposition}: calculate the variance explained ($R^2$) by each factor in each view. This is the first and arguably the most important plot to be inspected once the model is trained, as it summarises the variation (i.e. the signal) in a complex multi-view data set using a simple heatmap. With a quick visual inspection, this plot can be used to determine which factors are shared between multiple data modalities and which ones are exclusive to a single data modality.

	\item \textbf{Visualisation of the samples on the latent space}: the samples can be visualised in the latent space using beeswarm plots for individual factors or scatterplots for combinations of factors.

	\item \textbf{Inspection of weights}: the feature weights can be interpreted as an importance score for each feature on each factor. Inspecting the top weights for a given factor can reveal the molecular signatures that underlie each factor.

	\item \textbf{Association analysis between factors and external covariates}: multi-omic datasets typically consist of a large set of molecular readouts that are used for model training, and a small set of additionals covariates or response variables such as clinical outcome measurements. The external covariates are not used for model training but they can be linked to the factors \textit{a posteriori} using a simple association analysis.

	\item \textbf{Imputation}: the latent factors capture a condensed low-dimensional representation of the data that can be used to generate (denoised) reconstructions of the input data. This can be valuable for the inspection of very sparse datasets.

	\item \textbf{Feature set enrichment analysis}: when a factor is difficult to characterise based only on the inspection of the top weights, one can compute a statistical test for enrichment of biological pathways using predefined gene-set annotations.

\end{itemize}

\begin{figure}[H]
	\begin{center}
		\includegraphics[width=1.0\textwidth]{MOFA}
		\caption{\textbf{MOFA overview}.\\
		(a) The MOFA model takes as input $M$ data matrices ($\bfY^1, \cdots, \bfY^M$), each one representing a separate view or data modality. Samples must be co-occurrent but the features are not necessarily related across data modalities. MOFA performs a multi-view matrix decomposition that results in a matrix of factors ($\bfZ$) and $M$ matrices of feature weights, one for each data modality ($\bfW^1, \cdots, \bfW^M$).\\
		(b) The trained MOFA model can be queried for different downstream analyses.}
		\label{fig:MOFA}
	\end{center}
\end{figure}

\subsubsection{Interpretation of the factors} \label{section:interpretation_factors}

The interpretation of the factor values is intuitively similar to that of principal components in PCA. Each factor sorts cells along a one-dimensional axis with a mean of zero. Samples with different signs indicate opposite \textit{effects} along this source of variation, with higher absolute value indicating a stronger effect.

For example, if the $k$-th factor captures the variability associated with commitment to cell type X, we could expect cells that belong to cell type X to be at one end of the factor (irrespective of the sign, only the relative positioning being of importance). In contrast, cells that do not belong to cell type X are expected to be at the other end of the factor.

% mention prior

\subsubsection{Interpretation of the weights} \label{section:interpretation_weights}

The weights provide a score for each feature on each factor. Features with no association with the factor are expected to have values close to zero (as specified by the prior distributions). In contrast, features with strong association with the factor are expected to have large absolute values. The sign of the weight indicates the direction of the effect such that a positive weight indicates that the feature is positively associated with the factor values.

Following the example above, genes that are upregulated cell type X are expected to have large positive weights, whereas genes that are downregulated in cell type X (or, equivalently, upregulated in the other cell types) are expected to have large negative weights. Genes that do not change in expression between the cell types are expected to have a value of zero.

\subsubsection{Variance decomposition} \label{section:variance_decomposition}

The first step in the downstream analysis is to calculate the percentage of variance explained for each factor $k$ in each view $m$ ($R^{2}_{m,k}$), which can be visualised using a heatmap (see \Cref{fig:MOFA}b). This is done by adapting the coefficient of determination statistic that is traditionally used for linear regression analysis:
\begin{equation*}
	R^{2}_{m,k} = 100 \frac{ (\sum_{n=1}^{N} \sum_{d=1}^{D_m} y_{n,d}^{m} - z_{nk} w_{dk}^{m})^2 }{ (\sum_{n=1}^{N} \sum_{d=1}^{D_m} y_{n,d}^{m})^2 }
\end{equation*}

\subsection{Model selection and consistency across random initilizations} \label{section:mofa_robustness}

The optimisation problem in MOFA is not convex and the resulting posterior distributions depend on the initialisation of the model. Thus, when doing random initialisation of the parameters and/or expectations it becomes mandatory to perform model selection and assess the consistency of the factors across different trials. The strategy we adopted in this work is to train several MOFA models under different parameter initialisations, where the expectation of each node is randomly sampled from its underlying distribution. After fitting, we select the model with the highest ELBO for downstream analysis. In addition, we evaluate the robustness of the factors by plotting the Pearson correlations between factors across all trials (\Cref{fig:MOFA_robustness}).

\begin{figure}[H]
	\centering
	\includegraphics[width=1.0\textwidth]{MOFA_robustness}
	\caption{ \textbf{Model selection and robustness analysis in MOFA}.\\
	The left plot the log ELBO (y-axis) for 25 model instances (x-axis). The arrow indicates the model with the highest ELBO that would be selected for downstream analysis. The right plot displays the absolute value of the Pearson correlation coefficient between pairwise combinations of all factors across the 25 model instances. A block-diagonal matrix indicates that factors are robustly estimated regardless of the initialisation.}
	\label{fig:MOFA_robustness}
\end{figure}

\subsection{Learning the number of factors} \label{section:mofa_nfactors}

As described in \Cref{section:hierarchical_priors}, the use of an ARD prior allows factors to be actively pruned by the model if their variance explained is negligible. In the implementation we control the pruning of factors by a hyperparameter that defines a threshold on the minimum fraction of variance explained by a factor (across all views). Additionally, because of the non-convexity of the optimisation problem, different model instances can potentially yield solutions with different number of active factors. Thus, the optimal number of factors can be selected by the model selection strategy outlined in \Cref{section:mofa_robustness}.

\subsection{Monitoring convergence}

An attractive property of Variational inference is that the objective function (the ELBO) increases monotonically at every iteration. This provides a simple way of monitoring convergence:
\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{elbo_convergence}
	\caption{\textbf{Training curve for two different instances of MOFA with random initialisations}. The y-axis displays the log of the ELBO, with higher values indicating a better fit. The x-axis displays the iteration number. The horizontal dash lines mark the value of the ELBO upon convergence. }
	\label{fig:elbo_convergence}
\end{figure}

Training is stopped when the change in the lower bound becomes smaller than a predefined threshold.

% COPIED
%\subsection{Guidelines for the selection of data modalities} \label{section:guidelines_views}
%Data modalities typically correspond to different molecular layers, but the user can also explore data-driven modalities that do not necessarily correspond to different molecular readouts (see for example Figure 3). Analogous to the number of samples per group, the size of the data modality can have an influence on the latent space, such that larger data modalities can contribute more to the latent space than small data modalities, simply because they have larger amounts of variation. The signal that can be extracted from small data modalities will depend on the degree of structure within the dataset, the levels of noise and on how strong the sample imbalance is between data modalities. Hence, in the case of a strong feature imbalance, we recommend the user to subset highly variable features in the large data modalities to maintain the number of features within the same order of magnitude.

\subsection{Modelling and inference with non-Gaussian data} \label{section:mofa_ngaussian}

Gaussian likelihoods are sufficient to model the residuals of most types of continuous data. Thus, when possible, we advise the user to apply data transformations (i.e. $log$ transformation for example) and use the Gaussian likelihood model. However, there are cases where Gaussian likelihoods are not appropriate, even after data transformations, namely binary and (low) count data. However, non-Gaussian likelihoods are problematic because they are not conjugated with prior distributions, and this prevents the use of the efficient variational inference scheme for Gaussian likelihoods (see Appendix~\ref{appendix:mofa}).

To implement efficient variational inference in conjunction with a non-Gaussian likelihood we adapt prior work from \cite{Seeger2012}, where the full derivation can be found. The idea is to approximate non-Gaussian observations by a normally-distributed \textit{pseudo-data} that is constructed using second-order Taylor expansions. This defines a lower bound that can be improved by adjusting parameters at each iteration. Denoting the parameters in the MOFA model as $\bfX= (\bfZ,\bfW,\balpha,\btau,\btheta)$, recall that the variational framework approximates the posterior $p(\bfX | \bfY )$ with a distribution $q(\bfX)$, which is indirectly optimised by optimising a lower bound of the log model evidence. The resulting optimization problem can be re-written as
\begin{equation*}
\min_{q(\bfX)} (-\Lagr(\bfX)) =  \min_{q(\bfX)} \E_q \big[ -\log p(\bfY|\bfX) \big] + \KL[q(\bfX)||p(\bfX)].
\end{equation*}
Let's now assume a general likelihood function $p(\bfY|\bfX)=f(\bfY|\bfC)$ with $\bfC = \bfZ\bfW^{T}$ that we can write as
\begin{equation*}
-\log p(\bfY|\bfX) = \sum_{n=1}^{N} \sum_{d=1}^{D} f_{nd} (c_{nd})
\end{equation*}
with $f_{nd}(c_{nd}) = -\log p(y_{nd}|c_{nd})$. For simplicity in the derivation I will assume a single view and thus drop the index $m$ for clarity.

Extending \cite{Seeger2012} to our heteroscedastic noise model, we require $f_{nd}(c_{nd})$ to be twice differentiable and upper bounded by a constant $\kappa_d$. I do not prove this here, but this property holds true in many important models as for example the Bernoulli and Poisson likelihoods, as demonstrated in \cite{Seeger2012}. Under this assumption a lower bound on the log likelihood can be defined using Taylor expansion:
\begin{equation*}
f_{nd}(c_{nd}) \leq \frac{\kappa_d}{2} (c_{nd} - \zeta_{nd})^2 + f'(\zeta_{nd})(c_{nd} - \zeta_{nd}) + f_{nd}(\zeta_{nd}) := q_{nd}(c_{nd},\zeta_{nd}),
\end{equation*}
where $\zeta_{nd}$ are additional variational parameters that determine the location of the Taylor expansion and have to be optimised to make the lower bound as tight as possible. Plugging the bounds into the optimization problem above, we obtain:
\begin{equation*}
\min_{q(\bfX),\bZeta} \quad \sum_{d=1}^{D}\sum_{n=1}^{N} \E_q [ q_{nd}(c_{nd},\zeta_{nd})] + \KL[q(\bfX)||p(\bfX)]
\end{equation*}

The new objective function has two class of parameters to optimise: the parameters associated with the local bounds $\bZeta$, and the parameters associated with the model $(\bfX)$.
The algorithm proposed in \cite{Seeger2012} alternates between updates of $\bZeta$ and $\mathrm{q}(\bTheta)$. The update for $\bZeta$ is given by
\begin{equation*}
\zeta \leftarrow \E[\bfZ]\E[\bfW]^{T}
\end{equation*}
where the expectations are taken with respect to the corresponding $q$ distributions.\\
% In order to find the updates for $q(\bTheta)$ we bring the taylor approximation of $q(f_{nd})$ in q audratic form:
% \begin{equation*}
% q(f_{nd},\zeta_{nd}) \propto \frac{\kappa_d}{2}(f_{nd} - (zeta_{nd} - g(\zeta_{nd})/\kappa_d))^2
% \end{equation*}
% and note that this is proportional to the log of a Gaussian distribution $-log \Normal (\hat{y}_{nd}|f_{ng},\frac{1}{ng})$ where $\hat{y}_{nd} = zeta_{nd} - g'(\zeta_{nd})/\kappa_d$ is defined as a pseudodata based on the zero-inflated observations.
% Consequently, for fixed $\zeta_{nd}$, the updates of the variational distributions $Q(X)$ and $Q(W)$ are equivalent to the ones derived in X, but with pseudodata $\hat{Y}$ and precision $\kappa_g$
On the other hand, the updates for $q(\bfX)$ are identical to the standard variational Bayesian updates with Gaussian likelihoods, but with the observed data $\bfY$ replaced by the \textit{pseudo-data} $\hat{\bfY}$ and where the precisions $\tau_{nd}$ (which were treated as random variables) are replaced by the constant terms $\kappa_d$ introduced above. This might seem a minor change, but it is very important. In the Gaussian case the model infers a variance parameter for each feature which means that MOFA explicitly models heteroscedastic noise, but when using non-Gaussian likelihoods this is no longer possible.

Finally, the general formula pseudodata is given by (derived in \cite{Seeger2012}):
\begin{equation*}
\hat{y}_{nd} = \zeta_{nd} - f'(\zeta_{nd})/\kappa_d
\end{equation*}
where different log likelihood functions $f(\cdot)$ yield different $\kappa_d$ values.

In MOFA we implemented a Bernoulli likelihood model for binary data and a Poisson likelihood model for (low) count data.

\subsubsection*{Bernoulli likelihood for binary data}

When the observations are binary, $y \in \{0,1\}$, they can be modelled using a Bernoulli likelihood:
\begin{equation*}
	p(y|c) = \frac{e^{yc}}{1+e^c}
\end{equation*}
The second derivative of the log likelihood is bounded by:
\begin{equation*}
	f''(c) = \sigma(c)\sigma(-c) \leq 1/4 := \kappa
\end{equation*}
where $\sigma$ is the sigmoid function $f(c) = 1/(1+e^{-c})$.

The pseudodata updates are given by
\begin{equation*}
	\hat{y}_{nd} = \zeta_{nd} - 4*(\sigma(\zeta_{nd}) - y_{nd})
\end{equation*}

% \begin{equation*}
% \bfY|\bfZ,\bfW \sim \text{Ber}(\sigma(\bfZ\bfW^T)),
% \end{equation*}
% where $\sigma(a)=(1+e^{-a})^{-1}$ is the logistic link function and $\bfZ$ and $\bfW$ are the latent factors and weights, respectively.

% As an approximation, here we adopt the Jaakkola upper bound \cite{Jaakkola2000} on the negative log-likelihood. Denoting $c_{nd}=(\bfZ\bfW^T)_{nd}$ this is given by:
% \begin{align*} \label{jaakkola}
% 	\begin{split}
% 		-\log\left(p(y_{nd}|c_{nd})\right) &= -\log\left(\sigma\left((2y_{nd}-1)  c_{nd}\right)\right)\\
% 		& \leq -\log(\zeta_{nd})-\frac{(2y_{nd}-1)c_{nd}-\zeta_{nd})}{2} +\lambda(\zeta_{nd})\left(c_{nd}^2 -\zeta_{nd}^2 \right)\\
% 		& =: b_J(\zeta_{nd}, c_{nd},y_{nd} )
% 	\end{split}
% \end{align*}
% with $\lambda$ given by $\lambda(\zeta)=\frac{1}{4\zeta}\tanh\left(\frac{\zeta}{2}\right)$.\\
% This can easily be derived from a first-order Taylor expansion on the function $f(x) = - \log(e^{\frac{x}{2}}+e^{-\frac{x}{2}}) = \frac{x}{2}-\log(\sigma(x))$ in $x^2$ and by the convexity of
% $f$ in $x^2$ this bound is global as discussed in \cite{Jaakkola2000}.\\
% In order to make use of this tighter bound but still be able to re-use the variational updates from the Gaussian case we re-formulate the bound as a Gaussian likelihood on pseudo-data $\hat{\bfY}$.\\
% As above we can plug this bound on the negative log-likelihood into the variational optimization problem to obtain  \begin{equation*}
% \min_{q(\bfX),\bZeta} \quad \sum_{d=1}^{D}\sum_{n=1}^{N} \mathbb{E}_q b_J(\zeta_{nd}, c_{nd},y_{nd} ) + \KL[q(\bfX)||p(\bfX)].
% \end{equation*}
% This is minimized iteratively in the variational parameter $\zeta_{nd}$ and the variational distribution of Z,W:\\
% Minimizing in the variational parameter $\zeta$ this leads to the updates given by
% \begin{equation*}
% 	\zeta_{nd}^2 = \mathbb{E}[c_{nd}^2]
% \end{equation*}
% as described in \cite{Jaakkola2000}, \cite{Bishop2006}.\\
% For the variational distribution $q(\bfZ,\bfW)$ we observe that the Jaakkola bound can be re-written as
% \begin{equation*}
% 	b_J(\zeta_{nd}, c_{nd},y_{nd} ) = -\log\left(\varphi\left(\hat{y}_{nd}; c_{nd}, \frac{1}{2\lambda(\zeta_{nd})}\right)\right) + \gamma(\zeta_{nd}),
% \end{equation*}
% where $\varphi(\cdot; \mu, \sigma^2)$ denotes the density function of a normal distribution with mean $\mu$ and variance $\sigma^2$ and $\gamma$ is a term only depending on $\bZeta$. This allows us to re-use the updates for $\bfZ$ and $\bfW$ from a setting with Gaussian likelihood by considering the Gaussian pseudo-data
% \begin{equation*}
% 	\hat{y}_{nd}= \frac{2y_{nd}-1}{4 \lambda(\zeta_{nd})}
% \end{equation*}
% updating the data precision as $\tau_{nd} = 2\lambda(\zeta_{nd})$ using  updates generalized for sample- and feature-wise precision parameters on the data.

\subsubsection*{Poisson likelihood for count data}

When observations are natural numbers, such as count data $y \in \N = \{0,1,\cdots\}$, they can be modelled using a Poisson likelihood:
\begin{equation*}
	p(y|c) = \lambda(c)^y e^{-\lambda(c)}
\end{equation*}
where $\lambda(c)>0$ is a convex rate function. As done in \cite{Seeger2012}, here we adopt the rate function $\lambda(c)=\log(1+e^c)$, which yields the following upper bound of the second derivative of the log-likelihood:
\begin{equation*}
	f''_{nd}(c_{nd}) \leq \kappa_d = 1/4 + 0.17*\max(\bfy_{:,d}).
\end{equation*}
%The bound degrades with the presence of entries with large values. Thus, we follow common practice and clip overly large counts.\\
The pseudodata updates are given by
\begin{equation*}
	\hat{y}_{nd} = \zeta_{nd} - \frac{\mathrm{S}(\zeta_{nd})(1-y_{nd}/\lambda(\zeta_{nd}))}{\kappa_d}.
\end{equation*}

\pagebreak

\subsection{Theoretical comparison with published methods}

A variety of latent variable models exist with the aim of perfoming multi-view data integration, most of them inspired by the Group Factor Analysis formulation. A summary is provided in the table below. MOFA is the only method that scales to large datasets (employs Variational Bayes inference instead of MCMC-based approaches), has a combination of ARD and spike-slab regularisation on the weights, and is also capable of handling non-gaussian modalities and missing values.

\begin{table}[H]
	\begin{tabular}{@{}lllllll}
		\toprule
		{\textbf{Publication}} & {\textbf{Inference}} & {\textbf{\parbox{2.1cm}{View-wise\\ sparsity}}} & {\textbf{\parbox{2.4cm}{Feature-wise\\ sparsity}}} & {\textbf{\parbox{1.3cm}{Missing\\ values}}} & {\textbf{Likelihood}} &  \parbox{1.5cm}{{\textbf{Noise\\ model}}} \\ \toprule
		\parbox{2.2cm}{Shen2009} & \parbox{2.1cm}{EM,\\ grid search} & \parbox{2.1cm}{$L_1$- penalties} & $L_1$-penalty & No & Gaussian & \parbox{1.5cm}{Hetero-\\scedastic} \\\midrule
		\parbox{2.2cm}{Mo2013} &  \parbox{2.1cm}{EM,\\ grid search} & \parbox{2.1cm}{$L_1$- penalties} & $L_1$-penalty & No &\parbox{2cm}{Gaussian,\\Poisson,\\Bernoulli}  & \parbox{1.5cm}{Hetero-\\scedastic} \\\midrule
		\parbox{2.2cm}{Virtanen2012} & VB & ARD & None & No & Gaussian & \parbox{1.5cm}{Homo-\\scedastic} \\\midrule
		\parbox{2cm}{Klami2014} & VB & ARD & None & No & Gaussian & \parbox{1.5cm}{Homo-\\scedastic} \\ \midrule
		\parbox{2cm}{Bunte2016}  & Gibbs & ARD & Spike-Slab & No & Gaussian & \parbox{1.5cm}{Homo-\\scedastic} \\ \midrule
		\parbox{2cm}{Hore2016}  & VB  & None & Spike-Slab & Yes  & Gaussian & \parbox{1.5cm}{Hetero-\\scedastic}  \\ \midrule
		\parbox{2cm}{Remes2016}  & VB & ARD & None & No & Gaussian & \parbox{1.5cm}{Homo-\\scedastic} \\ \midrule
		\parbox{1.7cm}{Zhao2015} & Gibbs & ARD & \parbox{2.5cm}{Three-parameter\\ beta prior} & No & Gaussian & \parbox{1.5cm}{Hetero-\\scedastic}
		\\ \midrule
		\parbox{2.6cm}{Leppaaho2017} & {Gibbs} & {ARD} & {Spike-Slab} & {Yes} & {Gaussian} & \parbox{2cm}{{Homo-}\\{scedastic}} \\ \midrule
		MOFA  & VB & ARD & Spike-Slab & Yes & \parbox{2cm}{Gaussian,\\Poisson,\\Bernoulli} & \parbox{2cm}{Hetero-\\scedastic} \\  \bottomrule
	\end{tabular}
	\caption{\textbf{Overview of latent variable methods for multi-view data integration.} Abbreviations used: VB (variational Bayes inference), Gibbs (Gibbs sampling based inference), ARD(Automatic Relevance Determination)}
	\label{GFAtable}

\end{table}

% \subsection{A note on the implementation}
% The core of MOFA is implemented in Python, and the downstream analysis and visualisations are implemented in R. To facilitate adoption of the method, we deployed MOFA as open-source software available from Bioconductor\footnote{\url{https://www.bioconductor.org/packages/release/bioc/html/MOFA.html}} with multiple tutorials and a web-based analysis workbench\footnote{\url{http://www.ebi.ac.uk/shiny/mofa/}}.


\documentclass[../research.tex]{subfile}

\subsection{Topological Manifolds}
\label{sec:3.1}

  \begin{prop}
    We say a topological space $X$ is \textnormal{second-countable} if the topology has a countable basis.

    $\R^n$ is second-countable.
  \end{prop}

  \begin{proof}
    The set of open balls
    $$S=\{B_r(q):r\in\mathbb{Q},q\in\mathbb{Q}^n\}$$
    forms a basis for the Euclidean topology, since any real open ball $B_\epsilon(x)$, where
    $\epsilon\in\R$ and $x\in\R^n$, is a countable union of open balls $(B_{r_i}(q_i))_{i\in\N}\in
    S$, where each $B_{r_i}(q_i)\subset B_\epsilon(x)$.
  \end{proof}

  \begin{definition}[Chart]
    Let $M$ be a second-countable topological space. A \textit{chart}, $(U,E,\varphi)$, is an ordered
    tuple of open subsets $U\subset M,E\subset\R^n$ and a homeomorphism $\varphi:U\to E$.
  \end{definition}

  \begin{definition}[Atlas and manifold]
    An \textit{atlas} of $M$ is a collection of charts $\{(U_i,E_i,\varphi_i):i\in I\}$ such that the
    $U_i$ cover $M$. I.e.,
    $$\bigcup_{i\in I}{U_i}=M.$$
    Then $M$ is said to be an \textit{$n$-manifold} which, colloquialy, is said to be locally homeomorphic
    to $\R^n$.
  \end{definition}

  \begin{example}
    The 2-sphere,
    $$S^2=\{x^2+y^2+z^2=r^2:x,y,z\in\R^3,r>0\},$$
    can be contructed as follows: \\

    We define an equivalence relation, $\sim$ , on the 2-disk,
    $$D^2=\{x^2+y^2\leq r^2:x,y\in\R^2,r>0\},$$
    by $x\sim y\text{ if and only if }x^2+y^2=r^2.$
    Then the quotient space $D^2/\sim$ is homeomorphic to $S^2$.

    This amounts to ``gluing'' together the boundary points of $D^2$. $S^2$ is 2-manifold, that is,
    locally homeomorphic to $\R^2$.
  \end{example}

  \begin{definition}[Transition map]
    Given two charts $(U_1,E_1,\varphi_1),(U_2,E_2,\varphi_2)$ such that $U_1\cap U_2\neq\varnothing$,
    we define the \textit{transition map} between them by
    $$\tau_{1,2}:\varphi_1(U_1\cap U_2)\to\varphi_2(U_1\cap U_2).$$

    Clearly $\tau_{1,2}$ is a homeomorphism, since $\tau_{1,2}=\varphi_2\circ\varphi_1^{-1}$. Hence
    $\tau_{1,2}^{-1}=\varphi_1\circ\varphi_2^{-1}:=\tau_{2,1}$ is also a homeomorphism. Transition
    maps give us a way of ``moving'' throughout an $n$-manifold in way that resembles
    moving through $n$-dimensional Euclidean space.
  \end{definition}

\subsection{Differentiable Manifolds}
\label{sec:3.2}

  \begin{definition}[Differentiable manifold]
    We say a topological $n$-manifold $M$ is \textit{differentiable} if, for every pair of charts
    $(U_i,E_i,\varphi_i),(U_j,E_j,\varphi_j)$ such that $U_i\cap U_j\neq\varnothing$, the transition
    maps $\tau_{ij},\tau_{ji}$ are differentiable in $\R^n$. That is, all their partial derivatives
    exist and are continuous.
    % We call $\tau_{ij}$ and $\tau_{ji}$ \textit{diffeomorphisms}, and say
    % that $\varphi_i(U_i\cap U_j)$ and $\varphi_j(U_i\cap U_j)$ are \textit{diffeomorphic}.
  \end{definition}

  \begin{definition}[Differentiable function]
    Let $M$ be a differentiable $m$-manifold, $N$ a differentiable $n$-manifold, and $f:M\to N$ a continuous function. Fix charts
    $(U_1,E_1,\varphi_1)$, $(U_2,E_2,\varphi_2)$ be charts for $M$, $N$ respectively. We say that
    $f$ is \textit{differentiable} if the function
    $$\varphi_2\circ f\circ\varphi_1^{-1}:\R^m\to\R^n$$
    is differentiable.

    If $f$ is a homeomorphism, and $\varphi_1\circ f\circ\varphi_2^{-1}:\R^n\to\R^m$ differentiable, then
    we say $f$ is a \textit{diffeomorphism}, and $M$ and $N$ are \textit{diffeomorphic}.

    Note this requires $m=n$, so that the Jacobian matrix of $\varphi_2\circ f\circ\varphi_1^{-1}$ is
    invertible.
  \end{definition}

  \begin{example}
    If we let $N=\R^n$, then $f:M\to\R^n$ is differentiable if the function $f\circ\varphi^{-1}:\R^m
    \to\R^n$ is differentiable, for all charts $(U,E,\varphi)$ of $M$.
  \end{example}

  Given an $n$-dimensional differentiable manifold $M$, we can attach to every point of $M$ a
  ``tangent space'' - a vector space isomorphic to $\R^n$.

  \begin{definition}[Tangent space]
    Fix $x\in M$, and choose any chart $(U,E,\varphi)$ with $x\in U$. Let $\Gamma$ be the set of
    continuous and injective curves passing through $x$. That is,
    $$\Gamma=\{\gamma_i:(-1,1)\to M:i\in I\text{ s.t. }\gamma_i(0)=x\}.$$
    We then define an equivalence relation, $\sim$ , on $\Gamma$ by:
    $$\gamma_1\sim\gamma_2\text{ if and only if }(\varphi\circ\gamma_1)'(0)=(\varphi\circ\gamma_2)'(0).$$
    Then the \textit{tangent space}, $T_xM$, is equal to the quotient space $\Gamma/\sim$. We call
    the equivalence classes in $T_xM$ \textit{tangent vectors}, which we denote $\gamma_i':=[\gamma_i]$.
  \end{definition}

  \textbf{Remark:} The definition of $\Gamma$ is somewhat arbitrary. More generally, we can define
  $$\Gamma=\{\gamma_i:(a,b)\to M:i\in I\text{ s.t. }\gamma_i(c)=x\},$$
  where $a,b\in\R,a<b$, and $c\in(a,b)$. Then $\gamma_1\sim\gamma_2\text{ if and only if }
  (\varphi\circ\gamma_1)'(c)=(\varphi\circ\gamma_2)'(c),$ and we denote the tangent vectors $\gamma_i'(c)$.

  \begin{prop}
    If we define vector addition and scalar multiplication in $T_xM$ by
    $$\gamma_1'+\lambda\gamma_2':=\{\gamma:(\varphi\circ\gamma)'(0)=(\varphi\circ\gamma_1)'(0)
    +\lambda(\varphi\circ\gamma_2)'(0),\text{ where }\gamma_1\in\gamma_1',\gamma_2\in\gamma_2',\lambda\in\R\},$$
    the identity element in $T_xM$ as the equivalence class of constant curves:
    \begin{align*}
      \gamma_0'&=\{\gamma:\gamma(t)=c\in U,\,\forall t\in (-1,1)\} \\
               &=\{\gamma:(\varphi\circ\gamma)'(0)=0\},
    \end{align*}
    and the inverse of $\gamma'$ by
    $$-\gamma':=\{-\gamma:(\varphi\circ(-\gamma))'(0)=-(\varphi\circ\gamma)'(0),\text{ where }\gamma\in\gamma'\}.$$
    Then $T_xM\cong\R^n$.
  \end{prop}

  \begin{definition}[Directional derivative]
    Given a differentiable $n$-manifold $M$ and a differentiable function $f:M\to\R^n$, we define
    the \textit{directional derivative} of $f$ in the direction $\gamma'$ as
    $$D_\gamma(f):=(f\circ\gamma)'(0),\,\forall\gamma\in\gamma'.$$
  \end{definition}

  \begin{definition}[Pushforward]
    If $\varphi:M\to N$ is a differentiable function of differentiable manifolds, then we define the
    \textit{pushforward} of $\varphi$ at $x$ as the linear map
    $$d\varphi_x:T_xM\to T_{\varphi(x)}N,$$
    defined by $d\varphi_x(\gamma')=(\varphi\circ\gamma)'(0)\,\,\forall\gamma\in\gamma'$.
  \end{definition}

  \subsubsection{Lie groups}
  \label{sec:3.2.1}

  \begin{definition}[Lie group]
  \end{definition}


%!TEX root=report.tex
\subsection{Phase and Amplitude}

The $\hat{\beta}$ parameter in the OLS problem contains the coefficients to the linear combinations of $\cos(\omega_i t)$ and $\sin(\omega_i t)$, there is are used to predict $Y$.
Since there is a cosine and sine for each frequency in the design matrix $X$, one will end up with a linear combination of frequency function pairs. This makes interpretation difficult.
Instead one should convert the $\beta$ parameters for the trigonometric functions into amplitudes and phases corresponding to each pair of periodic functions such that only a cosine  $A_i \cos(\omega_i t - \phi_i)$ is needed.
This is done by using Ptolemy's theorem
\begin{equation}
A_i \cos(\omega_i t - \phi_i) = A_i \cos(\phi_i) \cos(\omega_i t) + A_i \sin(\phi_i) \sin(\omega_i t).
\end{equation}

Comparing with the linear combination from OLS
\begin{align}
\hat{Y} = \cdots + \beta_{c,i} \cos(\omega_i t) + \beta_{s,i} \sin(\omega_i t) + \cdots
\end{align}
It is seen that
\begin{align}
\beta_{c,i} = A_i \cos(\phi_i) && \text{ and } && \beta_{s,i} = A_i \sin(\phi_i).
\end{align}

By dividing these two equations with each other, $\phi_i$ can be calculated as
\begin{equation}
\frac{A_i \sin(\phi_i)}{A_i \cos(\phi_i)} = \frac{\beta_{s,i}}{\beta_{c,i}} \Rightarrow \phi_i = \arctan\left(\frac{\beta_{s,i}}{\beta_{c,i}}\right).
\end{equation}

To isolate $A_i$, square both equations and add them together
\begin{equation}
A_i^2 \cos(\phi_i)^2 + A_i^2 \sin(\phi_i)^2 = \beta_{c,i}^2 + \beta_{s,i}^2 \Rightarrow A_i = \sqrt{\beta_{c,i}^2 + \beta_{s,i}^2}.
\end{equation}

The result can be plotted with a circular color scale, using $\phi_i$ (the hue) and $A_i$ (the intensity).


\addcontentsline{toc}{chapter}{Appendices}

% The \appendix command resets the chapter counter, and changes the chapter numbering scheme to capital letters.
%\chapter{Appendices}
\appendix
\chapter{Derivations}
\label{ap:derivations}
% \section{Classical Limit of Nuclear TDSE \label{ap:polar_X}}
% The time-dependent nuclear Schr\"odinger equation:
% \[i \ \hbar \frac{\delta}{\delta t} \chi (\textbf{R}, t) = \left( \sum_{\nu = 1}^{N_{n}} \frac{[-i \ \hbar\nabla_{\nu} + \textbf{A}_{\nu}(\textbf{R}, t)]^2}{2 M_{\nu}} + \epsilon(\textbf{R}, t)\right) \chi (\textbf{R}, t)\]
% Substituting the polar form $\chi(\textbf{R}, t) = |\chi(\textbf{R}, t)|e^{\frac{i}{\hbar}S(\textbf{R}, t)}$
% \[i \ \hbar \frac{\delta}{\delta t} |\chi(\textbf{R}, t)|e^{\frac{i}{\hbar}S(\textbf{R}, t)} = \left( \sum_{\nu = 1}^{N_{n}} \frac{[-i \ \hbar\nabla_{\nu} + \textbf{A}_{\nu}(\textbf{R}, t)]^2}{2 M_{\nu}} + \epsilon(\textbf{R}, t)\right) |\chi(\textbf{R}, t)|e^{\frac{i}{\hbar}S(\textbf{R}, t)}\]
% We can remove the dependencies to neaten the equations up:
% \[i \ \hbar \frac{\delta}{\delta t} |\chi|e^{\frac{i}{\hbar}S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{[-i \ \hbar\nabla_{\nu} + \textbf{A}_{\nu}]^2}{2 M_{\nu}} + \epsilon \right) |\chi |e^{\frac{i}{\hbar}S}\]
% Using chain rule to expand the time-derivative:
% \[i \ \hbar (|\dot{\chi}|e^{\frac{i}{\hbar}S} + |\chi|\dot{e^{\frac{i}{\hbar}S}}) = \left( \sum_{\nu = 1}^{N_{n}} \frac{[-i \ \hbar\nabla_{\nu} + \textbf{A}_{\nu}]^2}{2 M_{\nu}} + \epsilon \right) |\chi |e^{\frac{i}{\hbar}S}\]
% ...
% \[i \ \hbar (|\dot{\chi}|e^{\frac{i}{\hbar}S} + |\chi|\frac{i}{\hbar}\dot{S} e^{\frac{i}{\hbar}S}) = \left( \sum_{\nu = 1}^{N_{n}} \frac{[-i \ \hbar\nabla_{\nu} + \textbf{A}_{\nu}]^2}{2 M_{\nu}} + \epsilon \right) |\chi |e^{\frac{i}{\hbar}S}\]
% Tidying up the LHS a bit and removing totally imaginary parts:
% \[\cancel{i \ \hbar |\dot{\chi}|e^{\frac{i}{\hbar}S}} - |\chi|\dot{S} e^{\frac{i}{\hbar}S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{[-i \ \hbar\nabla_{\nu} + \textbf{A}_{\nu}]^2}{2 M_{\nu}} + \epsilon \right) |\chi |e^{\frac{i}{\hbar}S}\]
% Expanding out the squared bracket in the RHS:
% \[- |\chi| \dot{S} e^{\frac{i}{\hbar}S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{(-i \ \hbar\nabla_{\nu})^2 + \textbf{A}_{\nu}^2 - i \ \hbar (\nabla_{\nu}\textbf{A}_{\nu}) -i \ \hbar \textbf{A}_{\nu}\nabla_{\nu}) }{2 M_{\nu}} + \epsilon \right) |\chi |e^{\frac{i}{\hbar}S}\]
% Treating just the RHS (multiplying out the bracket):
% \begin{dmath*}
%   - |\chi| \dot{S} e^{\frac{i}{\hbar}S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{1}{2 M_{\nu}}\left(
%   \underbrace{(- \hbar^2 \nabla_{\nu}^2 |\chi |e^{\frac{i}{\hbar}S})}_{(1)}
%   + \underbrace{(\textbf{A}_{\nu}^2|\chi |e^{\frac{i}{\hbar}S})}_{(2)}
%   - \underbrace{(2 i \ \hbar \nabla_{\nu}\textbf{A}_{\nu} |\chi |e^{\frac{i}{\hbar}}S)}_{(3)}
%  \right) + \epsilon |\chi |e^{\frac{i}{\hbar}S} \right)
% \end{dmath*}
%
% \noindent Treating term 1 and 3 seperately:
%
% \subsection{Term 1 \label{ap:polar_X_4}}
% \[TERM \ 1 = -\hbar^2\nabla^2_{\nu}|\chi |e^{\frac{i}{\hbar}S}\]
% Taking a single derivative:
% \[TERM \ 1 = -\hbar^2\nabla_{\nu}\left[ \nabla_{\nu}(|\chi |)e^{\frac{i}{\hbar}S} + |\chi |\nabla_{\nu}(e^{\frac{i}{\hbar}S}) \right]\]
% Using chain rule:
% \[TERM \ 1 = -\hbar^2\nabla_{\nu}\left[ \nabla_{\nu}(|\chi |)e^{\frac{i}{\hbar}S} + |\chi|\frac{i}{\hbar}e^{\frac{i}{\hbar}S} \nabla_{\nu}(S)  \right]\]
% Taking the derivative again:
% \[TERM \ 1 = -\hbar^2\left[(\nabla_{\nu}^2|\chi |)e^{\frac{i}{\hbar}S} + (\nabla_{\nu}|\chi |)\frac{i}{\hbar}e^{\frac{i}{\hbar}S} (\nabla_{\nu} S) + (\nabla_{\nu}|\chi|)\frac{i}{\hbar}e^{\frac{i}{\hbar}S} (\nabla_{\nu} S) + |\chi|\frac{-1}{\hbar^2}e^{\frac{i}{\hbar}S} (\nabla_{\nu}^2 S) \right] \]
% Tidying up (taking the $e^{\frac{i}{\hbar}S}$ outside the bracket, gathering like terms and removing imaginary terms):
% \[TERM \ 1 = -\hbar^2\left[\nabla_{\nu}^2|\chi | + \cancel{\frac{2i}{\hbar} (\nabla_{\nu}|\chi |\nabla_{\nu} S)} - \frac{|\chi|}{\hbar^2}(\nabla_{\nu}^2 S) \right] e^{\frac{i}{\hbar}S} \]
% \[TERM \ 1 = -\hbar^2\left[\nabla_{\nu}^2|\chi | - \frac{|\chi|}{\hbar^2}  (\nabla_{\nu}^2 S)\right] e^{\frac{i}{\hbar}S}\]
%
% \subsection{Term 3 \label{ap:polar_X_2}}
% \[TERM \ 3  = 2 i \ \hbar \nabla_{\nu}\textbf{A}_{\nu} |\chi |e^{\frac{i}{\hbar}}S\]
% Using chain rule (and cancelling imaginary terms)
% \[TERM \ 3  = 2 i \ \hbar \left[ \cancel{\nabla_{\nu}\textbf{A}_{\nu} |\chi |e^{\frac{i}{\hbar}}S} + \cancel{\textbf{A}_{\nu} \nabla_{\nu}|\chi |e^{\frac{i}{\hbar}}S} + \textbf{A}_{\nu} |\chi |\nabla_{\nu}e^{\frac{i}{\hbar}}S\right]\]
% ...
% \[TERM \ 3 = 2 i \ \hbar \textbf{A}_{\nu} |\chi |\frac{i}{\hbar}e^{\frac{i}{\hbar}S} \nabla_{\nu}S \]
% Tidying up:
% \[TERM \ 3 = -2 |\chi |e^{\frac{i}{\hbar}S} \textbf{A}_{\nu} \nabla_{\nu}S \]
% \subsection{Putting it all together \label{ap:Polar_X_final}}
% \begin{dmath*}
%   - |\chi| \dot{S} e^{\frac{i}{\hbar}S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{1}{2 M_{\nu}}\left(
%   -\hbar^2\left[\nabla_{\nu}^2|\chi | - \frac{|\chi|}{\hbar^2}  (\nabla_{\nu}^2 S)\right] e^{\frac{i}{\hbar}S}
%   + (\textbf{A}_{\nu}^2|\chi |e^{\frac{i}{\hbar}S})
%   + 2 |\chi |e^{\frac{i}{\hbar}S} \textbf{A}_{\nu} \nabla_{\nu}S
%  \right) + \epsilon |\chi |e^{\frac{i}{\hbar}S} \right)
% \end{dmath*}
%
% Dividing through by $-|\chi |e^{\frac{i}{\hbar}S}$:
% \begin{dmath*}
%   \dot{S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{1}{2 M_{\nu}}\left(
%   \hbar^2\left[\frac{\nabla_{\nu}^2|\chi |}{|\chi|} - \frac{1}{\hbar^2}  (\nabla_{\nu}^2 S)\right]
%   - \textbf{A}_{\nu}^2
%   - 2 \textbf{A}_{\nu} \nabla_{\nu}S
%  \right) - \epsilon \right)
% \end{dmath*}
% Tidying up:
% \begin{dmath*}
%   \dot{S} = \left( \sum_{\nu = 1}^{N_{n}} \frac{1}{2 M_{\nu}}\left(
%   \hbar^2\frac{\nabla_{\nu}^2|\chi |}{|\chi|} - \left( \nabla_{\nu}^2 S
%   + \textbf{A}_{\nu}^2
%   + 2 \textbf{A}_{\nu} \nabla_{\nu}S\right)
%  \right) - \epsilon \right)
% \end{dmath*}
% Factorising and more tidying:
% \begin{dmath*}
%   \dot{S} = \left( \underbrace{\hbar^2 \sum_{\nu = 1}^{N_{n}} \frac{1}{2 M_{\nu}}
%   \frac{\nabla_{\nu}^2|\chi |}{|\chi|}}_{\text{Quantum Potential}}  - \epsilon - \sum_{\nu = 1}^{N_{n}} \frac{(\nabla_{\nu}S
%   + \textbf{A}_{\nu})^2}{2 M_{\nu}} \right)
% \end{dmath*}

\section{Preservation of the Norm \label{ap:Norm_Pres}}
\subsection{Ehrenfest \label{ap:Norm_Pres_Eh}}
The statement of the conservation of norm is:
\[ \sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l}^{(I)} \vert^2 = 0\]
Using chain rule (and assuming this hold for each replica) we can write this as:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = \left(\frac{d}{dt} C_{l}^{*}\right)C_{l} + C_{l}^*\left(\frac{d}{dt} C_{l}\right)\]
If we write, $C_l = (a + bi)$ and $C_l^* = (a - bi)$ we can see the following relation holds:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = \sum_{l}^{N_{states}} \left(\frac{d}{dt} C_{l}^{*}\right)C_{l} + C_{l}^*\left(\frac{d}{dt} C_{l}\right) = \sum_{l}^{N_{states}} 2\mathcal{R} \left[ C_{l}^*\left(\frac{d}{dt} C_{l}\right) \right]\]
We have an expression for the time-derivative of the adiabatic expansion coefficient, under Ehrenfest (equation \eqref{eq:adiab_elec}). Inserting this above we get:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = \sum_{l}^{N_{states}}2 \mathcal{R}\left[ C_{l}^* \frac{-i}{\hbar} C_{l} \epsilon_{l} - \sum_{k} C_{l}^{*} C_{k} d_{lk}^{ad} \right]\]
The first term is imaginary so we can remove it, as we're only interested in the real components:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = - 2 \sum_{l,k}^{N_{states}} \mathcal{R}\left[  C_{l}^{*} C_{k} d_{lk}^{ad} \right]\]
The next term is exactly zero due to the anti-symmetry of the NACE that is the equation above can be written as:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = -2 \sum_{k=2}^{N_{states}} \sum_{l<k} \mathcal{R}\left[(C_{l}^{*}C_{k} - C_{k}^{*}C_{l})d_{lk}\right] = 0\]

\subsection{CTMQC \label{ap:CTMQC}}
The proof of conservation of the norm in CTMQC is similar to Ehrnfest. Again we write:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = \sum_{l}^{N_{states}} 2\mathcal{R} \left[ C_{l}^*\left(\frac{d}{dt} C_{l}\right) \right]\]
This time, the propagtion equation is slightly different:
\[\sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = \sum_{l}^{N_{states}}2 \mathcal{R}\left[ C_{l}^* \frac{-i}{\hbar} C_{l} \epsilon_{l} - \sum_{k} C_{l}^{*} C_{k} d_{lk}^{ad} - \sum_{\nu=1}^{N_n}\sum_{k}\frac{\mathcal{Q}_{lk,\nu}^{(I)}}{\hbar M_{\nu}}\cdot \left(\textbf{f}_{k,\nu}^{(I)} - \textbf{f}_{l,\nu}^{(I)}\right)|C_k|^2 C_l^{(I)}C_l^{* \ (I)} \right]\]

We have seen that the Ehrenfest part of the above equation conserves the norm, so we can remove that. we can also combine the 2 $C_l$ terms on the end:

\[ \sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = -2 \sum_{l,k}^{N_{states}} \mathcal{R}\left[ \sum_{\nu=1}^{N_n}\frac{\mathcal{Q}_{lk,\nu}^{(I)}}{\hbar M_{\nu}}\cdot \left(\textbf{f}_{k,\nu}^{(I)} - \textbf{f}_{l,\nu}^{(I)}\right)|C_k|^2 |C_l^{(I)}|^2 \right] \]

Because $\mathcal{Q}_{lk,\nu}^{(I)} = \mathcal{Q}_{kl,\nu}^{(I)}$ (and the diagonal is undefined) we can re-write the above equation as:
\[ \sum_{l}^{N_{states}} \frac{d}{dt} \vert C_{l} \vert^2 = -2 \sum_{l}^{N_{states}} \sum_{k<l} \mathcal{R}\left[ \sum_{\nu=1}^{N_n}\frac{\mathcal{Q}_{lk,\nu}^{(I)}}{\hbar M_{\nu}}\cdot \left[ \left(\textbf{f}_{k,\nu}^{(I)} - \textbf{f}_{l,\nu}^{(I)}\right) + \left(\textbf{f}_{l,\nu}^{(I)} - \textbf{f}_{k,\nu}^{(I)}\right)\right]|C_k|^2 |C_l^{(I)}|^2 \right] = 0 \]

So the norm should be conserved for each replica in CTMQC.

\section{Rabi Oscillation \label{ap:Rabi}}
By only allowing one parameter to vary in the propagation one can isolate and test that. For the electronic propagation I held the nuclear positions constant resulting in rabi oscillation. This is due  to the Schr\"odinger equation changing from a partial differential equation to, an analytically solvable, ordinary differential equation  i.e:
\[i \hbar \frac{\delta}{\delta t} \Phi(\textbf{R}(t), t) = \hat{H}(\textbf{R}(t), t) \Phi(\textbf{R}(t), t)\]
\[\downarrow\]
\[i \hbar \frac{d}{d t} \Phi(t) = \hat{H}(t) \Phi( t)\]
Which has the general solution:
\[\Phi(t) = e^{i \hbar \hat{H}t} \Phi(0)\]

% Side note:
%http://tex.stackexchange.com/questions/1319/showcase-of-beautiful-typography-done-in-tex-friends


\setcounter{section}{6}
\setcounter{subsection}{0}
\subsection{Functors}

Universal contruction is about to pick the ``best'' embodiment of an idea. \eg.
there is a lot of product in the universe, so we pick the ``best'' one, the one
that every product gets factorized into. Thus comes the ``universal''
construction.

Functors are just mapping between categories.

Why so important? We look for structures/patterns in categories and try to
emphasize/extract the pattern from them. Functor is used to find these patterns
and map them into a category when we can recognize these patterns.

``Category is the definition of structure''. In orther words, ``to recognize a
certain structure in a category'' is the same as defining the pattern as a
category. Being able to recognize a category inside another category is just
doing pattern recognition.

The mappings we are interested in are those which preserve structures. A
function is a mapping from set to set, which does not have a structure. A single
set represented in a category is just a category of bunches of objects and with
no arrows except the identities. No structure. (Discrete category)
\\

\begin{definition}{Functor}
  $F$ is a map between categories that preserves the structure.
  \ie. it maps arrow to arrow.
  For categories $C$, $D$, a functor is that for every
  $f \in C(a, b)$, $F$ maps it into $F~f \in D(F~a, F~b)$ that preserves the
  structure.

\[
\begin{tikzcd}[sep=huge]
  a \rar{F_a} \dar[red]{f} & F~a \dar[red]{F~f} \\
  b \rar{F_b} & F~b
\end{tikzcd}
\]

  Since a hom-set is a set, a functor just defines this ``huge'' function.

  Preserving the structure means for $g \circ f \in C$, $F (g \circ f) = (F g)
  \circ (F f) \in D$.

  Naturally one have to also make sure
  $\forall a \in C, F(id_a) = id_{F a} \in D$.
\end{definition}

\subsubsection{Functor properties}

\begin{definition}{faithful/full}
Functors that don't collapse structure is called ``faithful''. A faithful
functor is injective on \emph{hom-sets}. Correspondingly, A functor is ``full'' when it
is surjective. Functors that are ``full'' or ``faithful'' are only about
injective/surjective on the arrows, not about objects. Functor could map two
distinct objects
$a, b\in C$ into some $c \in D$, as long as the arrows between $a$ and $b$
doesn't map to the same arrow in $D$, this functor could still be faithful.
\end{definition}

\newcommand{\singletoncat}{%
\begin{tikzcd}[cramped]
  \cdot \ar[loop]{r}{id}
\end{tikzcd}%
}

\subsubsection{Interesting functors}

\begin{definition}{Selecting functor}
The possible functor mapping from category 1 (singleton category) to
another category $C$ is unique (up to iso), that is, to map $\id \in \mathrm{Arr}(1)$ arrow
to the $\id$ arrow on some
object in $D$. This process is like ``selecting'' an element in $D$.
\end{definition}
\hfill
\begin{definition}{Constant functor}
Another important functor mapping from $C$ to $D$ is called constant functor, which maps
all arrows in $C$ into the $\id$ arrow of an object $c \in D$. Constant functor
on object $c$ is denoted as $\Delta_c$.
\end{definition}
\hfill
\begin{definition}{Endofunctor}
  An endofunctor is a functor that maps from $C$ to itself, $F: C \to C$. Haskell
  functors are all endofunctors mapping from and to the $\Hask$ category.
  Haskell functors consists of two parts:
  \begin{enumerate}
  \item Mapping between types (types are objects in $\Hask$), \ie. Type constructors.
  \item Mapping between functions, \ie. \verb+fmap+.
  \end{enumerate}
\end{definition}
\hfill
\begin{remark}
  Take this example,
  \begin{lstlisting}
fmap :: (a -> b) -> (Maybe a -> Maybe b)
fmap f Nothing = Nothing *'\label{nothing}'*
fmap f (Just x) = Just (f x)
  \end{lstlisting}

  To haskell, which uses parametrically polymorphism, line \ref{nothing}
  is the only possible implementation for \lstinline{fmap f Nothing}. However,
  it's not true to math. For languages which doesn't use parametrically
  polymorphism, it might be possible to return other values. \eg. Return
  \lstinline{Just 0} for if type \lstinline{a} is \lstinline{Int}. Haskell is
  imposing a stronger condition to be a functor.

  \newcommand{\Maybe}{\lstinline{Maybe}}
  Let's check if \Maybe~really is a functor. Questions to ask:
  \begin{enumerate}
  \item Does \lstinline{fmap} preserves identity?
  \item Does \lstinline{fmap} preserves composition?
  \end{enumerate}

  Surely it does.
\end{remark}


\section{Methodology} \label{methods}

%This section introduce the methods that applied to this project.
%Since both two methods are only work on square-free polynomials, the first step
%of this project is applying square free decomposition to original polynomials to
%avoid repeat roots. After square free decomposition, methods based on Budan's
%Theorem and Continued Fraction will be applied. They both based on
%Descartes' rule of sign to check how many real roots in a interval. The continued fraction
%method also uses Mobius transformation, which will be introduce in Continued Fraction
%subsection.

\subsection{Square Free Decomposition}
In mathematics, a square-free polynomial is a polynomial defined over a field
that does not have a divisor any square of a non-constant
polynomial\cite{Yuns}. Usually, a square-free polynomial refers to the
polynomials with no repeated roots. This project applied Yun's
algorithm to perform square-free decomposition. It's based one the
succession of Greatest Common Divisor(GCD).

\subsubsection{Greatest Common Divisor}

In algebra, the greatest common divisor of two polynomials is a polynomial,
of the highest possible degree, that is a factor of both the two original
polynomials. This concept is similar to the GCD of two integers. Ideally, both
polynomials divided by their greatest common divisor should have no remainder.

This project uses Euclidean algorithm to compute the GCD of two polynomials.
In Algorithm~\ref{alg1}, $rem(a,b)$ refers to the remainder of Euclidean division of
polynomial $a$ and polynomial $b$.

\begin{algorithm}[H]
\label{alg1}
\SetAlgoLined
  \SetKwInOut{Input}{input}
  \SetKwInOut{Output}{output}

  \Input{$P1$: a univariate polynomial\newline
         $P2$: a univariate polynomial}
  \Output{Greatest common divisor of $P1$ and $P2$}

  $r_0 = P1$\;
  $r_1 = P2$\;
  \For{$i=1; r_i \neq 0; i=i+1$}{
    $r_{i+1} = rem(r_{i-1},r_i)$
  }

  \Return{$r_{i-1}$}

\caption{GCD of two polynomials}
\end{algorithm}

\subsubsection{Yun's Algorithm}

Based on the success of GCD, Yun developed a square-free decomposition
algorithm for univariate polynomials.

Given a primitive polynomial $P$, assume $P=P_1P_2^2...P_n^n$ is desired
factorization. Yun's algorithm will compute square-free polynomials $P_i$ and
the subscript refers to the times this square-free polynomial appears. Which
means $P_i$ appears $i$ times in original polynomial. This process is described in
Algorithm~\ref{alg2} formally.

\begin{algorithm}[H]
\label{alg2}
\SetAlgoLined
  \SetKwInOut{Input}{input}
  \SetKwInOut{Output}{output}

  \Input{Primitive polynomial $P$}
  \Output{List of square-free polynomials}

  $G = GCD(P, dP/dx)$\;
  $C_1=P/G$\;
  $D_1 = (dP/dx)/G - dC_1/dx$\;

  \For{$i=1,C_i\neq 0;i=i+1$}{
    $P_i= GCD(C_i,D_i)$\;
    $C_{i+1}= C_i/P_i$\;
    $D_{i+1}= D_i/P_i - dC_{i+1}/dx$\;
  }

  \Return{$P_1, P_2...P_n$}

\caption{Yun's Square-free Decomposition Algorithm}
\end{algorithm}

\subsection{Budan's Theorem}

Budan's theorem is a theorem used for bounding the number of real roots in a
given interval. Given a univariate polynomial $P$, we denote
$\#_{l,r}(P)$ as the number of real roots of $P$ in half-open interval $(l,r]$.
Then we denote $v_h(P)$ as the number of sign changes in coefficients of
polynomial $P_h$, where $P_h(x) = P(x+h)$.

Budan's theorem states that $v_l(h) - v_r(h) - \#_{l,r}(P)$ is a nonnegative
even integer. From this statement, we can know that if $v_l(h) - v_r(h) = 1
\text{ or } 0$, there is only one or zero real root in interval $(l,r]$.

Based on this theorem, this project combines bisection method with Budan's
theorem to isolate the real roots.  This process described in
Algorithm~\ref{alg3}.

\begin{algorithm}[H]
\label{alg3}
\SetAlgoLined
  \SetKwInOut{Input}{input}
  \SetKwInOut{Output}{output}

  \Input{A square-free polynomial $P$}
  \Output{List of intervals contains only one real root}

  $ret = []$\;
  $up\_bound = Upper(P)$\;
  $low\_bound = - up\_bound$\;
  $search = [(low\_bound, up\_bound)]$\;

  \While{$search$ not empty}{
    $l, r = pop(search)$\;
    $vl = sign\_change(P(x+l))$\;
    $vr = sign\_change(P(x+r))$\;
    \If{$vl - vr = 1$}{
      $ret.append([l,r])$\;
    }
    \ElseIf {$vr - vl > 1$}{
      \If{$l - r \geq MINIMAL\_RANGE$} {
        $mid = l + (r-l)/2$\;
        $search.append(mid, r)$\;
        $search.append(l, mid)$\;
      }
      \ElseIf{$(vl - vr) \%2 = 1$} {
        $ret.append([l,r])$\;
      }
    }
  }

  \Return{ret}\;
\caption{Real-root isolation based on Budan's Theorem}
\end{algorithm}

%\begin{algorithm}[H]
%\label{alg3}
%\SetAlgoLined
%  \SetKwInOut{Input}{input}
%  \SetKwInOut{Output}{output}

%  \Input{A square-free polynomial $P$}
%  \Output{List of intervals contains only one real root}

%  $ret = []$\;
%  $up\_bound = Upper(P)$\;
%  $low\_bound = - up\_bound$\;
%  $search = [(low\_bound, up\_bound)]$\;

%  \While{$search$ not empty}{
%    $l, r = pop(search)$\;
%    $vl = sign\_change(P(x+l))$\;
%    $vr = sign\_change(P(x+r))$\;
%    \If{$vl - vr = 1$}{
%      $ret.append([l,r])$\;
%    }
%    \ElseIf {$r - l \geq MINIMAL\_RANGE $}{
%      $mid = l + (r-l)/2$\;
%      $search.append(mid, r)$\;
%      $search.append(l, mid)$\;
%    }
%  }

%  \Return{ret}\;
%\caption{Real-root isolation based on Budan's Theorem}
%\end{algorithm}

In Algorithm~\ref{alg3}, $Upper(P)$ returns the upper bound of real roots of
$P$. This project uses Lagrange's bound\cite{Lag} in this project. Assuming $P =
a_0 + a_1x + ... + a_nx^n$, Lagrange's bound is $max\{1,
\sum_{i=0}^{n-1}|\frac{a_i}{a_n}|\}$.

\subsection{Continued Fraction}

Let's first introduce some notations used in this algorithm. Let $M(x)$
represent a Mobius transformation, which maps $x$ to $\frac{ax+b}{cx+d}$. So
that the number of positive roots of $P(M(x))$ equals the number of roots in
interval $(\frac{b}{d}, \frac{a}{c}]$ of $P$. Denote the sign changes of
coefficients of $P$ as $s$.

We use $\{a,b,c,d,p, s\}$ to represent an interval. Where, $ad-bc \neq 0$ and the
roots of original polynomial $P$ in interval $(\frac{b}{d},\frac{a}{c}]$ are
images of positive roots of $p$.

Continued fraction method can be formalized as Algorithm~\ref{alg4}.

\begin{algorithm}[H]
\label{alg4}
\SetAlgoLined
  \SetKwInOut{Input}{input}
  \SetKwInOut{Output}{output}

  \Input{A square-free polynomial with no zero root $P$}
  \Output{List of intervals contains only one positive real root}

  $s = sign\_change(P)$\;
  \If{$s=0$}{
    \Return{[]};
  }
  \ElseIf{$s=1$}{
    \Return{$[(0,\infty)]$};
  }

  $ret = []$\;
  $intervals = [\{1,0,0,1,P,s\}]$\;

  \While{$intervals$ not empty}{
    $\{a,b,c,d,p,s\} = pop(intervals)$\;
    \If{$s = 0$}{
      continue;
    }
    $p' = p(x+1)$\;

    \If{$p'(0)=0$}{
      $ret.append([\frac{a+b}{c+d},\frac{a+b}{c+d}])$\;
      $p' = p'/x$\;
    }

    $s' = sign\_change(p')$\;
      $intervals.append(\{a, a+b, c, c+d, p', s' \})$\;

    \If{$s-s' = 1$}{
      $ret.append([\frac{b}{d},\frac{a+b}{c+d}])$\;
    }
    \ElseIf{$s-s' > 1$} {
      % add A(b/(1 + x)),  M(b/(1 + x)) to L
      $p'' = (x+1)^m p(1/(1+x))$\;
      $intervals.append(\{b, a+b, d, c+d, p'', sign\_change(p'') \})$\;

    }
  }

  \Return{ret}\;
\caption{Real-root isolation based on Continued Fraction}
\end{algorithm}

Algorithm~\ref{alg4} can only returns positive real roots, therefore,
Algorithm~\ref{alg4} will be applied both on $P(x)$ and $P(-x)$. Zero roots
should be handled before using Algorithm~\ref{alg4}.


\documentclass{article}
\usepackage{amsmath}
\usepackage[margin=1.0in]{geometry}
\usepackage{xcolor}

\begin{document}

\noindent
Does $\displaystyle \sum_{n=1}^\infty \frac{1}{n(n+1)}$
diverge, converge absolutely, or converge conditionally?

\subsection*{Solution 1}

Rewrite $\frac1{n(n+1)}$ using partial fraction decomposition:
\[ \frac1{n(n+1)} = \frac{A}{n} + \frac{B}{n+1}\]
So
\[ 1 = A(n+1) + Bn\]
Using $n=0$, we get $A  = 1$. Using $n=-1$, we get $B=-1$. So
\[ \frac1{n(n+1)} = \frac{1}{n} - \frac{1}{n+1}\]
Therefore, our original series can be rewritten:
$\displaystyle \sum_{n=1}^\infty \frac{1}{n(n+1)} =  \displaystyle \sum_{n=1}^\infty \left( \frac{1}{n} - \frac{1}{n+1} \right)$

\vskip20pt
\noindent
We appear to have a telescoping series. After cancellations, the $n$th term of the sequence of partial sums is:
\[ s_n = \frac{1}{1} - \frac{1}{n+1}\]
Thus,
\begin{align*}
\lim_{n \to \infty} s_n
&=  \lim_{n \to \infty} \left( \frac{1}{1} - \frac{1}{n+1}\right) \\
&=  \frac11 - 0\\
&= 1
\end{align*}
So, (by definition), the series $\displaystyle \sum_{n=1}^\infty \frac{1}{n(n+1)}$ converges, and we furthermore know that the sum is $1$. Recall that it is unusual that we get to know the value of a convergent series.

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty \frac{1}{n(n+1)}$ converges absolutely.

\subsection*{Solution 2}

Note that $n(n+1) \geq n^2$ for all $n \geq 1$. So
\[ \frac1{n(n+1)} \leq \frac{1}{n^2}\]
The series $\displaystyle \sum_{n=1}^\infty  \frac{1}{n^2}$ converges by the $p$-test. Therefore, the series  $\displaystyle \sum_{n=1}^\infty \frac{1}{n(n+1)}$ converges by the Direct Comparison Test.

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty \frac{1}{n(n+1)}$ converges absolutely.

\end{document}%%%%%%%%%%%%%%%%%

\begin{align*}
L&=\lim_{n \to \infty} \sqrt[n]{|a_n|}\\
&= \lim_{n \to \infty} \sqrt[n]{\left| \right|}\\
\end{align*}

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty AAAAAAAAAAAAAA$ converges absolutely.

Since $|r| < 1$, the series ...  converges by the Geometric Series Test.

Since $|r| \geq 1$, the series ...  diverges by the Geometric Series Test.

The function $f(x)=\frac{}{}$ is continuous, positive, and decreasing on $[1,\infty)$.

\subsection*{Solution}


\section*{Exercises}

\begin{ex}
  Use elementary operations to solve the system of equations
  \begin{equation*}
    \begin{array}{c}
      3x+y=3 \\
      x+2y=1.
    \end{array}
  \end{equation*}

  \begin{sol}
    $(x,y) = (1,0)$.
  \end{sol}
\end{ex}

\begin{ex}
  Use elementary operations to find the point $(x,y)$
  that lies on both lines $x+3y=1$ and $4x-y=3$.
  \begin{sol}
    $
    \begin{array}{c}
      x+3y=1 \\
      4x-y=3
    \end{array}
    $, Solution is: $(x,y) = \paren{\frac{10}{13},\frac{1}{13}}$
  \end{sol}
\end{ex}

\begin{ex}
  Use elementary operations to determine whether the three lines
  $x+2y=1$, $2x-y=1$, and $4x+3y=3$ have a common point of intersection.
  If so, find the point, and if not, tell why they don't have such a
  common point of intersection.
  \begin{sol}
    $
    \begin{array}{c}
      x+2y=1 \\
      2x-y=1 \\
      4x+3y=3
    \end{array}
    $, Solution is: $(x,y) = \paren{\frac{3}{5},\frac{1}{5}}$
  \end{sol}
\end{ex}

\begin{ex}
  Do the three planes, $x+y-3z=2$, $2x+y+z=1$, and $3x+2y-2z=0$ have a
  common point of intersection? If so, find one and if not, tell why
  there is no such point.
  \begin{sol}
    No solution exists. $
    \begin{array}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
      x &+& y &-& 3z &=& 2 \\
      2x &+& y &+& z &=& 1 \\
      3x &+& 2y &-& 2z &=& 0
    \end{array}
    $, after elementary operations:
    $
    \begin{array}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
      x & &   &+& 4z &=& 0 \\
        & & y &-& 7z &=& 0 \\
        & &   & & 0 &=& 1
    \end{array}$.
    Thus one of the equations says $0=1$ in an equivalent system of
    equations.
  \end{sol}
\end{ex}

\begin{ex}
  Solve the following system of equations by back substitution.
  \begin{equation*}
    \begin{array}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
      x &+& 3y &-& 2z &=& 5 \\
        & & y  &+& 3z &=& 4 \\
        & &    & & z  &=& 1.
    \end{array}
  \end{equation*}

  \begin{sol}
    $(x,y,z) = (4,1,1)$.
  \end{sol}
\end{ex}

\begin{ex}
  Write the following system of linear equations as an augmented
  matrix. Caution: you first have to simplify and rearrange the
  equations so that ``like'' variables are lined up in columns. Write
  the variables in the order $x$, $y$, $z$.
  \begin{equation*}
    \begin{array}{c}
      x - 3z + 2y = 5 \\
      6 - x = 4 + y - z \\
      2x + 3 = x + 3y.
    \end{array}
  \end{equation*}

  \begin{sol}
    $
    \begin{array}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
      x &+& 2y &-& 3z &=& 5 \\
      -x &-& y &+& z &=& -2 \\
      x &-& 3y && &=& -3. \\
    \end{array}
    $
    As an augmented matrix:
    $
    \begin{mymatrix}{rrr|r}
      1 & 2 & -3 & 5 \\
      -1 & -1 & 1 & -2 \\
      1 & -3 & 0 & -3
    \end{mymatrix}.
    $
  \end{sol}
\end{ex}

\begin{ex}
  Four times the weight of Gaston is 150kg more than the weight of
  Ichabod. Four times the weight of Ichabod is 660kg less than seventeen
  times the weight of Gaston. Four times the weight of Gaston plus the weight
  of Siegfried equals 290kg. Brunhilde would balance all three of the
  others. Find the weights of the four people.
  \begin{sol}
    $
    \begin{array}{c}
      4g-I=150 \\
      4I-17g=-660 \\
      4g+s=290 \\
      g+I+s-b=0
    \end{array}
    $, Solution is : $\set{g=60,I=90,b=200,s=50}$
  \end{sol}
\end{ex}


\section{Robust Regression}
\subsection{Singular values decomposition (SVD)}
After applying the SVD to both $\mathbf X$ and $\mathbf X_{noise}$ data, the Fig.\ref{fig:1_6_a} illustrates the singular values. For the original data $\mathbf X$, there are only three singular values which corresponding to its rank. Due to the randomness and independences of noise, the matrix $\mathbf X_{noise}$ is full rank which has ten singular values. However, there are three significant singular values which represents the dimension of signal subspace. The magnitude of other non-zero singular values are approximate half of the signal singular values, whose difference can be used to detect the signal and noise subspace. Nevertheless, if the noise power is large, it becomes hard to identify the rank of $\mathbf X_{noise}$.
\begin{figure}[htbp]
    \centering
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/16/16a1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/16/16a2.eps}
     \end{subfigure}
    \caption{SVD of $\mathbf X$ and $\mathbf X_{noise}$}
    \label{fig:1_6_a}
\end{figure}
\subsection{Low rank approximation}
The SVD algorithm could be used to recover the original matrix. Since the noise power is much less than the signal power, the recovered matrix is noiseless if only $k$ significant values are concerned, where $k$ is the rank of the original matrix. Fig.\ref{fig:1_6_b} shows the error between the noise matrix and recovered noiseless matrix. As to the noiseless error curve, when the number of $k$ is equal to the actual rank, the error reaches the bottom point. In this experiment, the minimum point at rank 3 is 27.07, while it is 49.34 of noise error over all rank. Therefore, the rank and the dimension of signal subspace are determined.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.4\textwidth]{fig/16/16b.eps}
    \caption{Error in low rank approximation}
    \label{fig:1_6_b}
\end{figure}
\subsection{OLS vs PCR}
The parameter matrix $\mathbf B$ is calculated based on the OLS and PCR method. Meanwhile, the square error between the actual $\mathbf Y$ and estimated output on training and testing data are illustrated in Fig.\ref{fig:1_6_c}. When the number of significant components $k$ is larger than 3, the PCR has the same performance with the OLS method. However, when testing the parameter matrix via $\mathbf X_{test}$, the estimated error of the PCR increases with $k<3$, while the error of OLS decreases.
\begin{figure}[htbp]
     \centering
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/16/16c1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/16/16c2.eps}
     \end{subfigure}
        \caption{Training and testing error of OLS \& PCR}
        \label{fig:1_6_c}
\end{figure}
\subsection{Realisations of OLS and PCR}
Totally 50 realisations were simulated by OLS and PCR methods and its average error with test data are plotted in Fig.\ref{fig:1_6_d}. Compared with the testing error shown in Fig\ref{fig:1_6_c}, the average errors of both PCR and OLS algorithms are reduced to a large extent. The largest error of PCR is reduced from 85 to 65, while the OLS decreases to 37.81.
\vspace*{3in}
\begin{figure}[t!]
    \centering
    \includegraphics[width=0.4\textwidth]{fig/16/16d.eps}
    \caption{Estimated error of realisations}
    \label{fig:1_6_d}
\end{figure}