\documentclass[12pt, a4paper]{article}

\usepackage{geometry}
\geometry{verbose,a4paper,tmargin=2cm,bmargin=2cm,lmargin=2cm,rmargin=2cm}

\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{cases}
\usepackage{hyperref}
\usepackage{longtable}
\usepackage{graphicx}
\graphicspath{ {../Media/} }

\title{SIR models for the spread of COVID-19}
\date{\today}
\author{
	Maksim Gordienko \\
	Alexander Sysoev
}

\begin{document}
	\maketitle

	\section{Introduction}
	In this article we are going to show how Kermack and McKendrick SIR models can be used for epidemic process simulation. Then we add vital dynamics to SIR model. Next we solve simple models in analytical way. Finally, we introduce sophisticated SEIRS-like model especially for COVID-19 conditions, close to real life.

	\section{Mathematical model of epidemics}
	Let's consider a group of $N$ people and classify them into these types:
	\begin{itemize}
		\item \textbf{S} -- Susceptible
		\item \textbf{I} -- Infected
		\item \textbf{R} -- Recovered
	\end{itemize}

	This is called the SIR model for the spread of epidemic diseases that describes the changes in numbers of the three types of individuals. We denote the number of susceptible persons by $s(t)$, the number of infected individuals by $i(t)$ and the number of recovered people by $r(t)$. The time $t$ is measured in days. Also we suppose that each person is in contact with $m$ persons per day average. Hence the number of contacts between the susceptible and infected people becomes $\frac{m}{N} i(t) s(t)$. If we set the probability of infection for each contact as $p$ then the number of newly infected individuals within $\Delta t$ days becomes
	\begin{equation}
		\frac{m}{N} i(t) s(t) p \Delta t
	\end{equation}
	in total. Let $\beta = mp$, the number of non-infected people \textbf{(S)} from the $t$-th day to $ (t + \Delta t) $-th day changes as
	\begin{equation}
		s(t + \Delta t) - s(t) = - \frac{\beta}{N} s(t) i(t) \Delta t
	\end{equation}
	When $\Delta t \rightarrow 0 $, we can rewrite it as differential equation
	\begin{equation}
		\frac{ds(t)}{dt} = - \frac{\beta}{N} s(t) i(t)
	\end{equation}
	Meanwhile, the infected individuals recover at a removal rate $ \gamma $ per day. Subsequently, increase in the number of the recovered persons becomes
	\begin{equation}
		\frac{dr(t)}{dt} = \gamma i(t)
	\end{equation}
	Respectively, $ \dfrac{1}{\gamma} $ is the expected duration of infection. Also the number of recovered people includes the amount of deceased persons because they cannot possibly infect others.

	\newpage

	As well as the total number of individuals is $N$, we can express $i(t)$
	\begin{equation}
		i(t) = N - s(t) - r(t)
	\end{equation}
	The change in the number of infected people can be written as
	\begin{equation}
		\frac{di(t)}{dt} = - \frac{ds(t)}{dt} - \frac{dr(t)}{dt} = \frac{\beta}{N} s(t) i(t) - \gamma i(t) = \frac{\beta}{N} (s(t) - \gamma) i(t)
	\end{equation}
	Gathering the equations above, we have
	\begin{numcases}{}
		\frac{ds(t)}{dt} = - \frac{\beta}{N} s(t) i(t) \label{eq:dsdt} \\
		\frac{di(t)}{dt} = (\frac{\beta}{N} s(t) - \gamma) i(t) \label{eq:didt} \\
		\frac{dr(t)}{dt} = \gamma i(t) \label{eq:drdt}
	\end{numcases}
	As initial conditions, we use
	\begin{equation} \label{eq:init_cond}
		\begin{dcases}
			s(0) = N_1 \\
			i(0) = N_2 \\
			r(0) = 0
		\end{dcases}
	\end{equation}
	If we introduce the following transformations of variables
	\[
		\tilde{s}(t) = \frac{s(t)}{N}, \tilde{i}(t) = \frac{i(t)}{N}, \tilde{r}(t) = \frac{r(t)}{N}, \tilde{t} = \beta t
	\]
	Then system of the equations becomes
	\begin{equation}
		\begin{dcases}
			\frac{d \tilde{s} (t)}{d \tilde{t}} = - \tilde{s}(t) \tilde{i}(t) \\
			\frac{d \tilde{i}(t)}{d \tilde{t}} = (\tilde{s}(t) - \frac{1}{R_0}) \tilde{i}(t) \\
			\frac{d \tilde{r}(t)}{d \tilde{t}} = \frac{1}{R_0} \tilde{i}(t)
		\end{dcases}
	\end{equation}
	The number $R_0 = \dfrac{\beta}{\gamma}$ is known as the basic reproduction number. The number of infected people increases when $R_0 > 1$ and decreases when $R_0 < 1$.

	\section{Analytical solution of SIR model}
	When model is defined, we can solve the system of equations \eqref{eq:dsdt} -- \eqref{eq:drdt}. For simplicity forget about function arguments. Firstly, rewrite the equation \eqref{eq:dsdt} as
	\begin{equation} \label{eq:dsdt_rewritten}
		i = - \frac{1}{\tilde{\beta}} (\frac{s'}{s})
	\end{equation}
	where $\tilde{\beta} = \dfrac{\beta}{N}$. Then differentiate both sides
	\begin{equation} \label{eq:di_1}
		i' = - \frac{1}{\tilde{\beta}} (- \frac{{s'}^2}{s^2} + \frac{s''}{s})
	\end{equation}
	Next insert the equation \eqref{eq:dsdt_rewritten} into \eqref{eq:didt}
	\begin{equation} \label{eq:di_2}
		i' = -(\tilde{\beta} s - \gamma) \frac{1}{\tilde{\beta}} (\frac{s'}{s})
	\end{equation}

	\newpage

	Comparing equations \eqref{eq:di_1} and \eqref{eq:di_2} we have
	\begin{equation} \label{eq:dsdt_combined}
		s \frac{d^2 s}{dt^2} - (\frac{ds}{dt})^2 + (\gamma - \tilde{\beta} s) s \frac{ds}{dt} = 0
	\end{equation}
	Now we introduce the following function
	\begin{equation} \label{eq:phi_intro}
		\phi = \frac{dt}{ds}
	\end{equation}
	Afterward \eqref{eq:dsdt_combined} becomes
	\begin{equation}
		\frac{d \phi}{ds} + \frac{\phi}{s} = (\gamma - \tilde{\beta} s)\phi^2
	\end{equation}
	This is a Bernoulli differential equation. Divide both parts by $ \phi^2 $
	\begin{equation}
		\frac{\phi'}{\phi^2} + \frac{1}{\phi s} = \gamma - \tilde{\beta} s
	\end{equation}
	Then make a substitution
	\begin{equation}
		z = \frac{1}{\phi}, z' = -\frac{\phi'}{\phi^2}
	\end{equation}
	Now solve the first-order linear ordinary equation
	\begin{equation}
		-z' + \frac{z}{s} = \gamma - \tilde{\beta} s
	\end{equation}
	with general solution where $C$ is constant
	\begin{equation}
		z = -\gamma s \ln{s} + \tilde{\beta} s^2 + Cs
	\end{equation}
	Returning back to $\phi$
	\begin{equation} \label{eq:phi_res}
		\phi = \frac{1}{s(C - \gamma \ln{s} + \tilde{\beta} s)}
	\end{equation}
	From the relation of inverse function in equation \eqref{eq:phi_intro}, we have
	\begin{equation}
		\frac{1}{\phi} = \frac{ds}{dt}
	\end{equation}
	Using equation \eqref{eq:dsdt}, we obtain
	\begin{equation} \label{eq:it_def}
		i(t) = -\frac{1}{\tilde{\beta}} (C - \gamma \ln{s(t)} + \tilde{\beta} s(t))
	\end{equation}
	Moreover, from equations \eqref{eq:dsdt} and \eqref{eq:drdt}, we get
	\begin{equation}
		\frac{dr}{dt} = -\frac{\gamma}{\tilde{\beta}} (\frac{s'}{s})
	\end{equation}
	Subsequently, the relation between s(t) and r(t) becomes
	\begin{equation} \label{eq:rt_def}
		r(t) = -\frac{\gamma}{\tilde{\beta}} \ln{\frac{s(t)}{C_1}}
	\end{equation}
	where $C_1$ is a constant. According to our initial conditions \eqref{eq:init_cond}
	\begin{equation}
		C_1 = N_1
	\end{equation}

	\newpage

	From the relation $s(0) + i(0) + r(0) = N$, we have
	\begin{equation}
		C = -\tilde{\beta}N + \gamma \ln{N_1}
	\end{equation}
	If we substitute $C$ into equation \eqref{eq:phi_res}, we obtain
	\begin{equation}
		\frac{dt}{ds} = \frac{1}{s(-\tilde{\beta}N - \gamma \ln{\dfrac{s}{N_1} + \tilde{\beta}s})}
	\end{equation}
	Integrate this and express $t$ as a function of $s$
	\begin{equation} \label{eq:t_eps}
		t = \int_{s(0)}^{s(t)} \frac{d \varepsilon}{\varepsilon(-\tilde{\beta}N - \gamma \ln{\dfrac{\varepsilon}{N_1}} + \tilde{\beta} \varepsilon)}
	\end{equation}
	For convenience, we change a variable
	\begin{equation}
		\xi = \frac{\varepsilon}{N_1}
	\end{equation}
	Rewrite the equation \eqref{eq:t_eps}
	\begin{equation}
		t(s) = \int_{1}^{\frac{s(t)}{N_1}} \frac{d \xi}{\xi(-\beta - \gamma \ln{\xi} + \beta \xi \frac{N_1}{N})}
	\end{equation}
	Now we can calculate $t(s)$ using numerical integration with a small step size and $s(t)$ as a parameter like that
	\begin{equation}
		\int_{a}^{b} f(\xi) \,d\xi \simeq \sum_{i=0}^{n-1} f(a + ih)h,~ \text{where} ~ h = \frac{b-a}{n}
	\end{equation}
	If $t(s)$ is obtained, then $i(t)$ and $r(t)$ can be calculated from equations \eqref{eq:it_def} and \eqref{eq:rt_def} respectively.

	\newpage

	\section{SIR Model with natural deaths and births}

	Let's rewrite our system of equations \eqref{eq:dsdt} -- \eqref{eq:drdt} with an addition of the death and birth processes. Consider these rates are equal to each other, we introduce new variable $D$:
	\begin{numcases}{}
		\frac{ds(t)}{dt} = - \frac{\beta}{N} s(t) i(t) + D(N - s(t)) \label{eq:death_s} \\
		\frac{di(t)}{dt} = (\frac{\beta}{N} s(t) - \gamma - D) i(t) \label{eq:death_i} \\
		\frac{dr(t)}{dt} = \gamma i(t) - Dr(t) \label{eq:death_r}
	\end{numcases}
	As we did before, we rewrite it into one equation:
	\begin{equation} \label{eq:re_death_s}
		i = \frac{s' - D(N- s)}{-\tilde{\beta}s}
	\end{equation}
	Again we differentiate both sides (with $\tilde{\beta} = \frac{\beta}{N}$ substitution):
	\begin{equation}
		i' = -\frac{1}{\tilde{\beta}}(\frac{(s'' + D s')s - (s' + D s - D N)s'}{s^2}),
	\end{equation}
	\begin{equation} \label{eq:re_death_s_diff}
		i' = -\frac{1}{\tilde{\beta}}(\frac{s''}{s} - \frac{s'^2 - D N s'}{s^2})
	\end{equation}
	Now we can insert \eqref{eq:re_death_s} into \eqref{eq:death_i}:
	\begin{equation}
		i' = (\tilde{\beta} s - \gamma - D)(\frac{s' - D(N- s)}{-\tilde{\beta}s})
	\end{equation}
	\begin{equation} \label{eq:re_death_i_sub}
		i' = -\frac{1}{\tilde{\beta}s}(\tilde{\beta}s - \gamma - D)(s' + D s - D N)
	\end{equation}
	And comparing \eqref{eq:re_death_i_sub} and \eqref{eq:re_death_s_diff} we get next equation:
	\begin{equation}
		s s'' - s'^2 +(-\tilde{\beta}s^2 + (\gamma + D)s + D N)s' - \tilde{\beta}D s^3 - (\tilde{\beta}D N + D(\gamma + D))s^2 - (\gamma + D)D N s = 0
	\end{equation}
	We can see that coefficients at $s^3, s^2$ and $s$ are $consts$ so we substitute them with \\
	$A = -\tilde{\beta}D, B = - (\tilde{\beta}D N + D(\gamma + D)),  C = - (\gamma + D)D N$:
	\begin{equation} \label{eq:death_main}
		s s'' - s'^2 +(-\tilde{\beta}s^2 + (\gamma + D)s + D N)s' +A s^3 + B s^2 + C s = 0
	\end{equation}
	Now we introduce a function $\phi = \frac{dt}{ds}$, so the equation will be:
	\begin{equation} \label{eq:death_main_phi}
		-\phi' - \frac{\phi}{s} +(-\tilde{\beta}s^2 + (\gamma + D)s + D N)\frac{\phi^2}{s} + (A s^2 + B s + C)\phi^3 = 0
	\end{equation}
	And by substitution $v = \phi s, v' = \phi's + \phi$ the following equation occur:
	\begin{equation} \label{eq:death_main_v}
		v' +(-\tilde{\beta} + \frac{(\gamma + D)}{s} + \frac{D N}{s^2})v^2 + (A + \frac{B}{s} + \frac{C}{s^2})v^3 = 0
	\end{equation}

	\newpage
	Then we assume these three substitutions:
	\begin{numcases}{}
		r(s) = \frac{1}{v(s)}, \\
		\xi(s) = (\tilde{\beta} - \frac{(\gamma + D)}{s} - \frac{D N}{s^2}), \\
		\eta(s) = -(A + \frac{B}{s} + \frac{C}{s^2})
	\end{numcases}
	and we get Abel-type first-order second-kind equation:
	\begin{equation}
		r r' = \xi(s)r + \eta(s)
	\end{equation}

	\newpage

	\section{Modeling COVID-19 pandemic}

	\subsection{Definitions}

	In order to create lifelike COVID-19 infection model we need to introduce more parameters in our model and consider different spread cases.
	\begin{itemize}
		\item \textbf{$S$} -- Susceptible. Includes all individuals who are not infected but are susceptible to contract the disease.
		\item \textbf{$E$} -- Exposed. Includes all persons who are exposed to infection, but not yet infections. Some of them might fall ill and some may not. We introduced that component due to incubation period of COVID-19 pandemic (approximated 5-6 days by WHO).
		\item \textbf{$I_s$} -- Symptomatic infected. It includes all the individuals who were symptomatic and infectious. They have approached some health-care facility, but have not yet been quarantined. This compartment was brought into the picture considering concerning news from worst-affected counties like Italy and the U.S. of the hospitals and health-care centers getting filled up very fast during this pandemic. A significant portion of the infected individuals may not be quarantined in case of the health-care system collapses.
		\item \textbf{$I_{as}$} -- Asymptomatic infected. It includes all those people who are affected but asymptomatic before they either recover or die or get permanently disabled. This compartment was formalized keeping after reports of various authentic studies claiming that around 30-40 percent of the infected individuals remain
		asymptomatic.
		\item \textbf{$Q$} -- Quarantined. Includes all persons who are currently under quarantine in healh-care facility.
		\item \textbf{$Q'$} -- In intensive care unit. Includes the quarantined patients who had to be moved to ICU after their condition worsened.

		\item \textbf{$C$} -- Carrier. This compartment includes individuals who have left quarantine after being tested negative but actually have not fully healed. So they can possibly infetct other susceptible individuals. They eventually either fall sick again or recover from disease. We introduced it due to multiple cases of re-infection being reported from countries like South Korea, China and Japan. It is safe to assume that one of the following happened:
		\begin{enumerate}
			\item Medical inefficiency (inaccurate test result)
			\item Loss of immunity after recovery and subsequent re-infection
		\end{enumerate}
		For the first possibility we introduced the $C$ compartment. To account for the second possibility we have kept the transition from $R$ to $S$ compartment respectively.
		\item $R_{wd}$ -- Recovered without disability. It includes all individuals who have recovered from infection without any disability and they can no longer infect any other individual
		\item $D$ -- Deceased. Includes all deaths during the pandemic.
		\item $R_d$ -- Recovered with disability. In this compartment we have kept all people who have recovered from the infection, can no longer infect anyone else but have been permanently disabled post recovery.
	\end{itemize}

	\newpage

	\subsection{Formualting differential equations}

	Now we can define differential equations corresponding to described model above.

	\begin{equation}
		\frac{dS}{dt} = -\alpha \frac{S(I_s + I_{as} + C)}{N} + g R_{wd}
	\end{equation}
	Where $\alpha$ is the disease transmission rate (same as $\beta = mp$ in SIR model) and $g$ is the rate at which a fraction of recovered individuals lose their immunity.

	\begin{equation}
		\frac{dE}{dt} = \alpha \frac{S(I_s + I_{as} + C)}{N} - \mu E
	\end{equation}
	If average incubation period is taken to be an exponential distribution $\mu$.

	\begin{equation}
		\frac{d I_s}{dt} = r \mu E - \varepsilon I_s + fC - \zeta_1 I_s - \eta_1 I_s
	\end{equation}
	Where $f$ is rate at which a fraction of carriers gets re-infected. Here $0 \le r \le 1$ is a number which shows how many individuals in the exposed compartment moves to $I_s$ compartment rather than $I_{as}$ one. And $\varepsilon, \zeta_1, \eta_1$ are the rates at which infected individuals get quarantined, deceased, disabled respectively.

	\begin{equation}
		\frac{d I_{as}}{dt} = (1 - r)\mu E - \beta_3 I_{as} - \zeta_3 I_{as} - \eta_3 I_{as}
	\end{equation}
	Where $\beta_3, \zeta_3$ and $\eta_3$ are the recovery rate, death rate and disability rate of asymptomatic individuals respectively.

	\begin{equation}
		\frac{dQ}{dt} = \varepsilon I_s - \beta_1 Q - vQ - pQ - \zeta_2 Q - \eta_2 Q
	\end{equation}
	Where $v, p, \zeta_2$ and $\eta_2$ are the rates at which the quarantined individuals go to carrier state, ICU, deceased and disabled compartments respectively. $\beta_1$ is the recovery rate for quarantined people.

	\begin{equation}
		\frac{d Q'}{dt} = pQ - \beta_4 Q' - \zeta_5 Q' - \eta_5 Q'
	\end{equation}
	Where $\beta_4, \zeta_5$ and $\eta_5$ are the rate with which individuals in ICU recover, die and get disabled respectively.

	\begin{equation}
		\frac{dC}{dt} = vQ - fC - \beta_2 C - \zeta_4 C - \eta_4 C
	\end{equation}
	Where $\beta_2, \zeta_4$ and $\eta_4$ are the rate with which carrier individuals recover, die and get disabled silently, respectively.

	\begin{equation}
		\frac{dR_{wd}}{dt} = \beta_1 Q + \beta_3 I_{as} + \beta_2 C - gR_{wd}
	\end{equation}

	\begin{equation}
		\frac{dD}{dt} = \zeta_1 I_s + \zeta_2 Q + \zeta_3 I_{as} + \zeta_4 C + \zeta_5 Q'
	\end{equation}

	\begin{equation}
		\frac{dR_d}{dt} = \eta_1 I_s + \eta_2 Q + \eta_3 I_{as} + \eta_4 C + \eta_5 Q'
	\end{equation}

	System of these equations can be solved using numerical methods.

	\newpage

	\subsection{Parameters and initial conditions selection}

	\begin{longtable}[c]{| c | c | c | c | c | c |}
		\caption{Initial condtions and rates}\\
		\hline
		Initial condition & Value & Rate & Value & Rate & Value\\
		\hline
		$N$ & 83783945 & $\alpha$ & 0.42 & $v$ & 0.05 \\
		$S$ & 83783935 & $g$ & 0.0001 & $p$ & 0.01\\
		$E$ & 0 & $\mu$ & 7 & $\zeta_2$ & 0.2\\
		$I_s$ & 10 & $f$ & 0.2 & $\eta_2$ & 0.1 \\
		$I_{as}$ & 0 & $r$ & 0.3 & $\beta_1$ & 0.5\\
		$Q$ & 0 &$\varepsilon$ & 0.5 & $\beta_4$ & 0.05\\
		$Q'$ & 0 &$\zeta_1$ & 0.025 & $\zeta_5$ & 0.2 \\
		$C$ & 0 &$\eta_1$ & 0.1 & $\eta_5$ & 0.1\\
		$R_{wd}$ & 0 & $\beta_3$ & 0.1458 & $\beta_2$ & 0.1458\\
		$D$ & 0 & $\zeta_3$ & 0.1 & $\zeta_4$ & 0.05\\
		$R_d$ & 0 & $\eta_3$ & 0.05 & $\eta_4$ & 0.01\\
		\hline

	\end{longtable}

	We decided to use early data from Germany for modeling epidemics. Data sources links provided at the end of article. Primarily, we relied on dataset by \textit{Our World in Data}. Rates were calculated using correlation with recent observations, research news and publications. These values were actual on February 2, 2020. We take into account further statistics to fine tune the parameters closer to real life.

	\subsection{Obtained results}

	\begin{figure}[h!]
		\centering
		\includegraphics[width=12cm]{seirs.png}
		\caption{SEIRS model with fixed disease transmission rate $\alpha = 0.42$.}
	\end{figure}

	\newpage

	\begin{figure}[h!]
		\centering
		\includegraphics[width=12cm]{seirs_2.png}
		\caption{SEIRS model with fixed disease transmission rate $\alpha = 0.42$. Zoomed to outbreak.}
	\end{figure}

	\begin{figure}[h!]
		\centering
		\includegraphics[width=15cm]{germany_stats.png}
		\caption{Official Germany COVID-19 Statistics}
	\end{figure}

	Comparing official data and our model, we can see, that the start of the outbreak is pretty much the same, but due to fixed disease transmission rate in the model, which indicates almost complete disregard of security measures, we get more dead and  infected people, whereas the real disease spread rate begins to slow down because of the lockdown and other measures such as improved tests.

	\

	Let's see how conditions getting worse if people simply do not maintain social distance and neglet personal protective equipment like masks. Potentially number of contacts and probability of getiing infection increases as well as our $\alpha$ rate.

	\begin{figure}[h!]
		\centering
		\includegraphics[width=12cm]{seirs_bigalpha.png}
		\caption{SEIRS model with increased $\alpha = 0.55$.}
	\end{figure}

	\newpage

	\begin{figure}[h!]
		\centering
		\includegraphics[width=12cm]{seirs_bigalpha_2.png}
		\caption{SEIRS model with increased $\alpha = 0.55$. Zoomed to outbreak.}
	\end{figure}

	We observe that expected outbreak occured earlier. Number of asymptomatic infected people reaches 5 millon, comparing with about 1 millon before. Meanwhile, the amount of deaths has almost doubled. Rapidly decreasing susceptible people means the high rate of infection. Thus, the situation has greately worsened. It means a heavy burden on medical institutions and will affect every aspect of our life.

	\section{Conclusion}

	We have created two models, which can help students understand basics of the disease spread process, and one more accurate, which can give more valuable information about epidemics, specifically about consequences of "covid-dissidence" and ignorance of the safety measurements.

	\section{References and data sources}

	\begin{itemize}
		\item Exact analytical solutions of the Susceptible-Infected-Recovered (SIR) epidemic
		model and of the SIR model with equal death and birth rates. Tiberiu Harko, Francisco S. N. Lobo, M. K. Mak.

		\item A Mathematical Model of Epidemics—A Tutorial
		for Students. Yutaka Okabe and Akira Shudo. Department of Physics, Tokyo Metropolitan University, Hachioji, Tokyo 192-0397, Japan.

		\item Modelling and simulation of COVID-19 propagation in a large population with specific reference to India.
		Ashish Menon, Nithin K Rajendran, Anish Chandrachud, Girish Setlur. Department of Physics, Indian Institute of Technology Guwahati.

		\item Statistics of the COVID-19 pandemic in Germany: \url{https://en.wikipedia.org/wiki/Statistics_of_the_COVID-19_pandemic_in_Germany}

		\item Complete COVID-19 dataset maintained by Our World in Data: \url{https://github.com/owid/covid-19-data/tree/master/public/data/}
	\end{itemize}
\end{document}


% -*- root: Main.tex -*-
\section{Principle Component Analysis}
$\mathbf{X} \in \mathbb{R}^{D \times N}$. $N$ observations, $K$ rank.\\
1. Empirical Mean: $\overline{\mathbf{x}} = \frac{1}{N} \sum_{n=1}^N \mathbf{x}_n$.\\
2. Center Data: $\overline{\mathbf{X}} = \mathbf{X} - [\overline{\mathbf{x}}, \ldots, \overline{\mathbf{x}}] = \mathbf{X} - \mathbf{M}$.\\
3. Cov.: $\boldsymbol{\Sigma} = \frac{1}{N	} \sum_{n=1}^N (\mathbf{x}_n - \overline{\mathbf{x}}) (\mathbf{x}_n - \overline{\mathbf{x}})^\top = \frac{1}{N} \overline{\mathbf{X}}\overline{\mathbf{X}}^\top$.\\
4. Eigenvalue Decomposition: $\boldsymbol{\Sigma} = \mathbf{U} \boldsymbol{\Lambda} \mathbf{U}^\top$.\\
5. Select $K < D$, inly keep $\mathbf{U}_K, \boldsymbol{\lambda}_K$.\\
6. Transform data onto new Basis: $\overline{\mathbf{Z}}_K = \mathbf{U}_K^\top \overline{\mathbf{X}}$.\\
7. Reconstruct to original Basis: $\tilde{\overline{\mathbf{X}}} = \mathbf{U}_k \overline{\mathbf{Z}}_K$.\\
8. Reverse centering: $\tilde{\mathbf{X}} = \tilde{\overline{\mathbf{X}}} + \mathbf{M}$.\\
For compression save $\mathbf{U}_k, \overline{\mathbf{Z}}_K, \overline{\mathbf{x}}$.\\
$\mathbf{U}_k \in \mathbb{R}^{D \times K}, \boldsymbol{\Sigma} \in \mathbb{R}^{D \times D}, \overline{\mathbf{Z}}_K \in \mathbb{R}^{K \times N}, \overline{\mathbf{X}} \in \mathbb{R}^{D \times N}$

\subsection*{Iterative View}
Residual $r_i$: $x_i - \tilde{x}_i = I - uu^T  x_i$\\
Cov of $r$:  $\frac{1}{n} \sum_{i=1}^n (I-uu^T)x_i x_i^T (I-uu^T)^T =$ \\
$(I-uu^T) \Sigma (I-uu^T)^T = \Sigma - 2\Sigma u u^T + u u^T \Sigma u u ^T = \Sigma - \lambda uu^T$ \\
1. Find principal eigenvector of $(\Sigma - \lambda u u^T)$\\
2. which is the second eigenvector of $\Sigma$\\
3. iterating to get $d$ principal eigenvector of $\Sigma$

\subsection*{Power Method \textit{- good for small k, easy, robust}}
Power iteration: $v_{t+1} = \frac{Av_t}{||Av_t||}$, $\lim_{t \rightarrow \infty} v_t = u_1$\\
Assuming $\langle u_1, v_0 \rangle \not = 0$ and $|\lambda_1| > |\lambda_j| (\forall j \geq 2)$.
\textbf{SVD:} + good for mid-sized problems, numerical techniques (QR decomposition),..


\section{Evaluating Dirichlet Integral}

Not every integral can be evaluated by \textbf{Newton-Leibniz formula}. Some of
them require certain tricks, and this integral is a perfect example:

\begin{equation}
	\int_0^\infty{\sin(x)\over x}dx
	\label{int-sinc}
\end{equation}

The function inside the integral of (\ref{int-sinc}) is called the
\textit{sinc} function, which is extremely useful in the field of signal and
image processing. To integrate this function, we could consider finding the
Laplace transform of this function:

\begin{equation}
	\int_0^\infty{\sin(x)\over x}dx=\mathcal{L}
	\left\{\sin(x)\over x\right\}(0)
	\label{sinc-laplace}
\end{equation}

Therefore, the integration of \textit{sinc} is divided into two steps:

\begin{itemize}
	\item Find the Laplace transform of \textit{sinc}
	\item Evaluate the Laplace transform at $s=0$
\end{itemize}

\subsection{Laplace transform of \textit{sinc}}

We denote the Laplace transform of \textit{sinc} as $F(s)$, conveniencing the
evaluation.

$$F(s)=\int_0^\infty{\sin(x)\over x}e^{-sx}dx$$
$$
\begin{aligned}
	F'(s)&={d\over ds}\int_0^\infty{\sin(x)\over x}e^{-sx}dx \\
	&=\int_0^\infty{\partial\over\partial s}{\sin(x)\over x}e^{-sx}dx \\
	&=-\int_0^\infty\sin(x)e^{-sx}dx=-\mathcal{L}\{\sin(x)\}
\end{aligned}
$$

Now we are to find the Laplace transform of sine function. It will take two
iterations of integration by parts if we evaluate the integral directly.
However, if we complexify it, turning $\sin(x)$ to $\Im\left(e^{ix}\right)$ in
other words, we can make use of the properties of exponential function so that
the process becomes easier.

$$
\begin{aligned}
	\mathcal{L}\{\sin(x)\}&=\int_0^\infty\Im\left(e^{ix}\right)e^{-sx}dx \\
	&=\Im\left(\int_0^\infty e^{ix-sx}dx\right)
	=\Im\left(\int_0^\infty e^{(i-s)x}dx\right) \\
	&=\Im\left(\left.e^{(i-s)x}\over i-s\right|_0^\infty\right) \\
	&=\Im\left(\lim_{x\to\infty}{e^{(i-s)x}\over i-s}-{1\over i-s}\right) \\
	&=\Im\left(1\over s-i\right)=\Im\left(s+i\over(s+i)(s-i)\right) \\
	&=\Im\left(s+i\over s^2+1\right)={1\over s^2+1}
\end{aligned}
$$

After plugging the result of this function, we simplify the expression of
$F'(s)$:

\begin{equation}
	F'(s)=-{1\over s^2+1}
	\label{sinc-deriv}
\end{equation}

Before trying to find the exact $F(s)$, we also need to consider its property,
especially its horizontal assymptote.

$$
\lim_{s\to\infty}F(s)=\int_0^\infty{\sin(x)\over x}
\lim_{s\to\infty}\left(e^{-sx}\right)dx=0
$$

Therefore, $F(s)$ can be represented by what was concluded in equation
(\ref{sinc-deriv}) via expressing it as a subtraction:

$$
\begin{aligned}
	F(s)&=F(s)-\lim_{b\to\infty}F(b) \\
	&=\int_\infty^sF'(t)dt=-\int_s^\infty F'(t)dt \\
	&=-\int_s^\infty{-1\over t^2+1}dt=\left.\arctan(t)\right|_s^\infty \\
	&={\pi\over 2}-\arctan(s)
\end{aligned}
$$

\subsection{Evaluate the Laplace transform of \textit{sinc} at $s=0$}

Since we have all the necessary equations calculated, we can now take down the
integral by simply plugging in that sacred zero.

$$
\begin{aligned}
	\int_0^\infty{\sin(x)\over x}dx&=\mathcal{L}\left\{\sin(x)\over x\right\}(0) \\
	&=F(0)={\pi\over2}-0={\pi\over2}
\end{aligned}
$$

As it turns out, the integral of \textit{sinc} from zero to infinity is half of
$\pi$.


% !TEX root = main.tex

%-------------------------------------------------
\section{Conditional probability}\label{sec:cond_prob}

Recall from elementary probability that the conditional probability of $A$ given $B$ is defined by
\[
\prob(A|B) = \frac{\prob(A\cap B)}{\prob(B)}.
\]
provided that $P(B)>0$. The following theorem shows that this defines a probability measure on subsets of $\Omega$.

\begin{theorem}
Let $\prob$ be a probability measure on subsets of $\Omega$ and let $B$ be an event with $\prob(B)>0$. Then the function
\[
\mathbb{Q}(A) = \prob(A|B)
\]
is also probability measure on subsets of $\Omega$.

\begin{proof}
To show that $\mathbb{Q}$ is a probability measure, we need to verify that $\mathbb{Q}(\emptyset) = 1$, $\mathbb{Q}(\Omega) = 1$ and that $\mathbb{Q}$ is countably additive. Firstly,
\begin{align*}
\mathbb{Q}(\emptyset)
	& = \prob(\emptyset|B) = \prob(\emptyset\cap B)/\prob(B) = \prob(\emptyset)/\prob(B) = 0; \\
\mathbb{Q}(\Omega)
	& = \prob(\Omega|B) = \prob(\Omega\cap B)/\prob(B) = \prob(B)/\prob(B) = 1.
\end{align*}
To prove that $\mathbb{Q}$ is countable additive, let $A_1,A_2,\ldots$ be pairwise disjoint subsets of $\Omega$. Then using the fact that $\prob$ is countably additive,
\begin{align*}
\mathbb{Q}(\medcup_{i=1}^{\infty} A_i)
	= \prob(\medcup_{i=1}^{\infty} A_i\,|\,B)
	& = \frac{\prob\big[(\medcup_{i=1}^{\infty} A_i) \cap B\big]}{\prob(B)} \\
	& = \frac{\prob\big[\medcup_{i=1}^{\infty} (A_i \cap B)\big]}{\prob(B)} \quad\text{(because intersection is distributive over union)},\\
	& = \frac{\sum_{i=1}^{\infty} \prob(A_i \cap B)}{\prob(B)} \quad\text{(because the sets $A_i\cap B$ are disjoint)}, \\
	& = \sum_{i=1}^{\infty}\frac{\prob(A_i \cap B)}{\prob(B)}
	  = \sum_{i=1}^{\infty}\prob(A_i|B)
		= \sum_{i=1}^{\infty} \mathbb{Q}(A_i)\quad\text{as required.}
\end{align*}
\end{proof}
\end{theorem}


\section{Maximising the generalised Rayleigh quotient}

In the following, we assume two symmetric matrices $\A \in \reals^{N \cross
N}$ and $\B \in \reals^{N \cross N}$ (i.e. $\A = \A^T$ and $\B =
\B^T$).

\subsection{The generalised Rayleigh quotient}

The generalised Rayleigh quotient of a non-zero vector $\w \in \reals^N$ and
the ordered matrix pair $(\A,\B)$ is the scalar $r(\w)$ defined as:
%
\begin{equation}
\label{eq:Rayleigh}
r(\w) = \frac{\w^T \A \w}
             {\w^T \B \w}
\end{equation}

\subsection{The generalised eigenvalue decomposition}

The generalised eigenvalue problem for the ordered matrix pair $(\A,\B)$
consists of finding the scalars $\lambda_i$ and corresponding vectors $\w_i$
for which the following holds:
%
\begin{equation}
\label{eq:GEVD}
\A \w_i = \lambda_i \B \w_i
\end{equation}

We can gather all generalised eigenvectors $\w_i$ as columns of a matrix
$\W$, and gather the corresponding generalised eigenvalues $\lambda_i$ in a
diagonal matrix $\Lambda$. The generalised eigenvalue decomposition (GEVD) of
$(\A,\B)$ is then:
\[
\A \W = \Lambda \B \W
\]

\subsection{Theorem}

The first generalised eigenvector $\w_1$ of $(\A,\B)$ (which corresponds to
the largest generalised eigenvalue $\lambda_1$) is also the vector $\what$
that maximises the generalised Rayleigh quotient $r(\w)$.

\subsection{Proof}

As a first step, we will show that if $\what$ is the maximum of $r(\w)$, that
it is indeed an eigenvector of $(\A,\B)$. In the second step, we will show
that the largest eigenvalue $\lambda_1$ of $(\A,\B)$ corresponds to the
maximum of $r(\w)$.

If $\what$ is a maximum of $r(\w)$, then
\begin{equation}
\label{eq:critical}
\grad{r(\what) = \vb{0}}.
\end{equation}

To calculate the gradient of $r(\w)$, we first expand the inner product
\begin{equation*}
\begin{split}
\w^T \A \w = & \; w_1 w_1 a_{11} + w_1 w_2 a_{12} + \tdots \\
                 & +  w_2 w_1 a_{21} + w_2 w_2 a_{22} + \tdots \\
                 & +  \tdots
\end{split}
\end{equation*}
%
The partial derivative with respect to say, the second element of $\w$ then
becomes:
\begin{align*}
\pdv{w_2}(\w^T \A \w) &= \sum_{i=1}^N w_i a_{2i}
                             + \sum_{j=1}^N w_j a_{j2} \\
                             %
                          &= 2 \; \sum_{i=1}^N w_i a_{2i},
\end{align*}
where the two sums are equal because $\A$ is symmetric. Note that
$\sum_{i=1}^N w_i a_{2i}$ is the second element of the vector $\A \w$.
Because analogous expressions hold for partial derivatives with respect to
the other elements of $\w$, we have:
\[
\grad(\w^T \A \w) = 2 \A \w
\]

Similarly, we have:
\begin{align*}
\pdv{w_2}(\frac{1}{\w^T \B \w})
    &= \frac{-1}{\qty(\w^T \B \w)^2} \; \pdv{w_2}(\w^T \B \w) \\
    &= \frac{-1}{\qty(\w^T \B \w)^2} \; 2 \sum_{i=1}^N w_i b_{2i},
\end{align*}
and
\[
\grad(\frac{1}{\w^T \B \w})
    = \frac{-1}{\qty(\w^T \B \w)^2} \; 2 \B \w
\]

We can now calculate
\begin{align*}
\grad{r(\w)} &= \frac{1}{\w^T \B \w} \grad(\w^T \A \w)
                + \qty(\w^T \A \w) \;
                  \grad(\frac{1}{\w^T \B \w})  \\[1em]
             % &= \frac{1}{\w^T \B \w} 2 \A \w
             %    + \w^T \A \w
             %      \frac{-1}{\qty(\w^T \B \w)^2} 2 \B \w  \\
             &= \frac{2 \A \w \qty(\w^T \B \w)
                      - 2 \B \w \qty(\w^T \A \w)}
                     {\qty(\w^T \B \w)^2}
\end{align*}

With \cref{eq:critical}, we now have the following condition for our
maximising vector $\what$:
\[
2 \A \what \qty(\what^T \B \what)
    &= 2 \B \what \qty(\what^T \A \what)
\]
or
\begin{align*}
\A \what &= \frac{\what^T \A \what}
                 {\what^T \B \what} \; \B \what  \\[1em]
\A \what &= r(\what) \; \B \what
\end{align*}
This is the generalised eigenvalue/eigenvector definition (\cref{eq:GEVD})
for $\w_i = \what$ and $\lambda_i = r(\what)$.

We have thus shown that if $\what$ is a maximum of $r(\w)$, that it is a
eigenvector of $(\A,\B)$, with $r(\what)$ its corresponding eigenvalue.

As the second step, we now show that $r(\what)$ is the \emph{largest}
eigenvalue of $(\A,\B)$. We follow the reasoning of \cite{Trefethen1997} (p.
204), who prove a related result for the ordinary Rayleigh quotient.

We will rewrite the generalised Rayleigh quotient $r(\w)$ by writing the
arbitrary vector $\w$ as a linear combination of the generalised eigenvectors
$\w_i$ of $(\A,\B)$: $\w = \sum_i c_i \w_i$. Then:
\begin{align*}
r(\w) &= \frac{\qty(\sum_i c_i \w_i)^T \A \qty(\sum_i c_i \w_i)}
              {\qty(\sum_i c_i \w_i)^T \B \qty(\sum_i c_i \w_i)} \\[1em]
              %
      &= \frac{\sum_i c_i^2 \w_i^T \A \w_i}
              {\sum_i c_i^2 \w_i^T \B \w_i} \\[1em]
              %
      &= \frac{\sum_i c_i^2 \lambda_i \w_i^T \B \w_i}
              {\sum_i c_i^2 \w_i^T \B \w_i}
\end{align*}

Generalised eigenvectors are defined up to a scaling factor. We may therefore
define our $\w_i$ to be scaled such that $\w_i^T \B \w_i = 1$. We then have:
\[
r(\w) = \frac{\sum_i c_i^2 \lambda_i}{\sum_i c_i^2}.
\]
%
Each generalised Rayleigh quotient is thus a convex combination of
generalised eigenvalues $\lambda_i$. The maximum of a convex combination of
one-dimensional points is obtained in the largest of these points. If
$\lambda_1$ is thus the largest generalised eigenvalue of $(\A,\B)$, then
$\max{r(\w)} = \lambda_1$.

We have thus shown that $\what = \argmax r(\w) = \w_1$, where $\w_1$ is an
eigenvector of $(\A,\B)$, and that its corresponding eigenvalue $\lambda_1 =
\max r(\w) = r(\what)$ is the largest of the eigenvalues of $(\A,\B)$.

% \qed


\lab{Algorithms}{Reduced Row Echelon Form}{Reduced Row Echelon Form}
\objective{Write your own code to put a matrix in Reduced Row Echelon Form}

Recall that there are three elementary row operations:

\begin{description}
\item[Type I]  Swap two rows: $R_j \longleftrightarrow R_k$
\item[Type II] Multiply a row by a constant:  $a* R_j \longrightarrow R_j$
\item[Type III] Add a multiple of one row to another: $ a R_k + R_j \longrightarrow R_j$
\end{description}
By combining these operations, we can perform row reduction. Type
the following three m-files.  Be sure the path is set to the correct
directory and that the file names correspond to the function names.

\begin{verbatim}
function A = type1(A,j,k)
%
% This function swaps the j-th and k-th rows
%

temp = A(j,:); A(j,:) = A(k,:); A(k,:) = temp;
\end{verbatim}

\begin{verbatim}
function A = type2(A,j,a)
%
% This function multiplies the j-th row by the constant a
%

A(j,:) = a*A(j,:);
\end{verbatim}

\begin{verbatim}
function A = type3(A,j,k,a)
%
% This function multiplies the constant a times
% the k-th row and adds it to the j-th row
%

A(j,:) = a*A(k,:) + A(j,:);
\end{verbatim}
Now test your functions by running the following:
\begin{verbatim}
>> A = [4 5 6 3; 2 4 6 4; 7 8 0 5]

>> B = type1(A,1,2)
\end{verbatim}
This swaps the first two rows. Now continue with the other two
functions:
\begin{verbatim}
>> C = type2(B,1,0.5)

>> D = type3(C,2,1,-4)

>> E = type3(D,3,1,-7)
\end{verbatim}
At this point we've reduced the first column of the system.  Let's
continue with the second column:
\begin{verbatim}
>> F = type2(E,2,-1/3)

>> G = type3(F,1,2,-2)

>> H = type3(G,3,2,6)
\end{verbatim}
We're almost done.  Let's finish the reduce the third column.
\begin{verbatim}
>> I = type2(H,3,-1/9)

>> J = type3(I,2,3,-2)

>> K = type3(J,1,3,1)
\end{verbatim}
We have successfully row reduced the matrix.  We can check our work
by typing {\tt rref(A)}.

\section*{Programming Row Reduction}

When we do row reduction by hand, we usually take extra steps to
avoid fractions.  However, computers aren't slowed down by decimal
computations.  Below we will re-do the above reduction, but a little
more efficiently.  In addition, rather than defining a new matrix in
every step, we will redefine the matrix $A$ to be the output from
the last calculation.
\begin{verbatim}
>> A = [4 5 6 3; 2 4 6 4; 7 8 0 5]

>> A = type3(A,2,1,-A(2,1)/A(1,1))

>> A = type3(A,3,1,-A(3,1)/A(1,1))
\end{verbatim}
Notice that we didn't row swap or multiply a row by a constant to
make the leading entry a one.  Let's continue:
\begin{verbatim}
>> A = type3(A,3,2,-A(3,2)/A(2,2))

>> A = type3(A,1,2,-A(1,2)/A(2,2))
\end{verbatim}
The second column is reduced.  Now for the third column:
\begin{verbatim}
>> A = type3(A,2,3,-A(2,3)/A(3,3))

>> A = type3(A,1,3,-A(1,3)/A(3,3))
\end{verbatim}
The final step is to divide each row by its leading coefficient:
\begin{verbatim}
>> A = type2(A,1,1/A(1,1))

>> A = type2(A,2,1/A(2,2))

>> A = type2(A,3,1/A(3,3))
\end{verbatim}
We are finished.  Compare your answer with what you get from {\tt
rref}.  We can generalize this process by writing a function:
\begin{verbatim}
function A = rref_3x4(A)
%
% Reduces a 3 x 4 augmented matrix into rref
%
A = type3(A,2,1,-A(2,1)/A(1,1)); A = type3(A,3,1,-A(3,1)/A(1,1)); A
= type3(A,3,2,-A(3,2)/A(2,2)); A = type3(A,1,2,-A(1,2)/A(2,2)); A =
type3(A,2,3,-A(2,3)/A(3,3)); A = type3(A,1,3,-A(1,3)/A(3,3)); A =
type2(A,1,1/A(1,1)); A = type2(A,2,1/A(2,2)); A =
type2(A,3,1/A(3,3));

\end{verbatim}
Now test your function and compare with {\tt rref}.
\begin{verbatim}
>> A = randn(3,4)

>> rref_3x4(A)

>> ref(A)
\end{verbatim}

\section*{Remarks}

\begin{itemize}
\item The above algorithm won't work on every matrix because there is
no logic for dealing with under-determined systems. As a result,
it's a good idea to test this algorithm with random matrices. Why is
that?
\item Fast computer algorithms don't really compute the RREF this
way. They actually compute the REF and then use back-substitution to
finish.
\item The above algorithm is actually called ``Row Reduction Without
Pivoting''. It turns out that the swap operation is important in
dealing with certain matrices that are ``almost'' under determined.
\end{itemize}

\begin{problem}
Write a MATLAB function called {\tt myrref}, which takes as
input an $n\times (n+1)$ matrix and performs the above naive row
reduction. Compare your answers with MATLAB's {\tt rref}. HINT:
Write a nested {\tt for} loop and type {\tt help continue} to get a
good idea of what to do when $j=k$.
\end{problem}


% Source: waynedisonitau123

\subsubsection{Kirchhoff's Theorem}
Denote $L$ be a $n \times n$ matrix as the Laplacian matrix of graph $G$, where $L_{ii} = d(i)$, $L_{ij} = -c$ where $c$ is the number of edge $(i, j)$ in $G$.
\begin{itemize}
    \item The number of undirected spanning in $G$ is $\lvert \det(\tilde{L}_{11}) \rvert$.
    \item The number of directed spanning tree rooted at $r$ in $G$ is $\lvert \det(\tilde{L}_{rr}) \rvert$.
\end{itemize}

\subsubsection{Tutte's Matrix}
Let $D$ be a $n \times n$ matrix, where $d_{ij} = x_{ij}$ ($x_{ij}$ is chosen uniformly at random) if $i < j$ and $(i, j) \in E$, otherwise $d_{ij} = -d_{ji}$. $\frac{rank(D)}{2}$ is the maximum matching on $G$.

\subsubsection{Cayley's Formula}
\begin{itemize}
  \item Given a degree sequence $d_1, d_2, \ldots, d_n$ for each \textit{labeled} vertices, there are $$\frac{(n - 2)!}{(d_1 - 1)!(d_2 - 1)!\cdots(d_n - 1)!}$$ spanning trees.
  \item Let $T_{n, k}$ be the number of \textit{labeled} forests on $n$ vertices with $k$ components, such that vertex $1, 2, \ldots, k$ belong to different components. Then $T_{n, k} = kn^{n - k - 1}$.
\end{itemize}

\subsubsection{Erdős–Gallai Theorem}
A sequence of non-negative integers $d_1 \geq d_2 \geq \ldots \geq d_n$ can be represented as the degree sequence of a finite simple graph on $n$ vertices if and only if $d_1 + d_2 + \ldots + d_n$ is even and
$$ \sum_{i = 1}^{k}d_i \leq k(k - 1) + \sum_{i = k + 1}^{n}\min(d_i, k) $$
holds for all $1 \leq k \leq n$.

\subsubsection{Burnside's Lemma}
Let $X$ be a set and $G$ be a group that acts on $X$.
For $g \in G$, denote by $X^g$ the elements fixed by $g$:
\[
X^g = \{ x \in X \mid gx \in X \}
\]
Then
\[
|X/G| = \frac{1}{|G|} \sum_{g \in G} |X^g|.
\]


\documentclass{article}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}

\newtheorem{theorem}{Theorem}

\title{From Mass-Spring Systems to Spectral Graph Neural Networks}
\author{Nguyen Ngoc Khanh}

\begin{document}
    \maketitle

    \section{Mass-Spring System}

    \subsection{The two particles system}

    Consider a spring follows Hook's law. Let two particles $i$ and $j$ connected by a spring located at $x_i$ and $x_j$ respectively and $e_{i j} = \frac{x_j - x_i}{||x_j - x_i||_2}$ be the direction from $x_i$ to $x_j$ then the force that $i$ affects $j$ can be represented as:

    \begin{equation}
        F_{i j} = - k (||x_j - x_i)||_2 - L) e_{i j} = - k (x_j - x_i) + k L e_{i j}
    \end{equation}

    where $k$ is a positive real number, the characteristic of the spring and $L$ is the initial length the of spring. The magnitude of the force is proportional to the displacement from the initial distance between two particles.

    Let two particles connected by a spring sit in an Euclidean space such that the particles can freely move on a particular $z$ axis. At the initial condition, the two particles are located at $x_i$ and $x_j$ and the spring is at its length ($||x_j - x_i||_2 = L$) (no force).

    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.8\textwidth]{fig1.png}
        \caption{subspace of $e_{i j}$ and $z$}
        \label{fig:fig1}
    \end{figure}

    Since, $i$ and $j$ can only move in the $z$ axis, we can rewrite

    \begin{gather*}
        x_i = x_i^{(0)} + z_i z \\
        x_j = x_j^{(0)} + z_j z \\
    \end{gather*}

    Where $x_i^{(0)}$ and $x_j^{(0)}$ are the initial positions of $i$ and $j$, $z_i$ and $z_j$ are the displacements on the $z$ axis. Hence, the projected force on the $z$ axis can be written as

    \begin{equation}
        F_{i j} \cdot z = (- k (x_j - x_i) + k L e_{i j}) \cdot z = (- k (x_j^{(0)} - x_i^{(0)}) + k L e_{i j}) \cdot z + (- k (z_j - z_i) z) \cdot z
    \end{equation}

    The first term is the dot product of the initial force with the $z$ direction which is essential zero since there is no force at the beginning. Hence, the projected force on the $z$ axis can be written as

    \begin{equation}
        F_{i j} \cdot z = - k (z_j - z_i)
    \end{equation}

    The projected force on the $z$ axis linearly depends on the corresponding displacement.

    \subsection{The n particles system}

    Let $n$ particles with the same weight $m$ on an an Euclidean space that can freely move on a particular $z$ axis. Some of them are connected by springs of the same characteristic $k$ which is denoted by a undirected unweighted graph $G = (V, E)$. A particular node $i$ is affected by all of its neighbours where the projected force on $i$ is

    \begin{figure}[h!]
        \centering
        \includegraphics[width=0.8\textwidth]{fig2.png}
        \caption{4 particles system}
        \label{fig:fig2}
    \end{figure}

    \begin{equation}
        F_i \cdot z = - k \sum_{e_{j i} \in E} z_i - z_j = - k (d_i z_i - \sum_{e_{j i} \in E} z_j)
    \end{equation}

    Where $d_i$ denotes the degree of the node $i$. Newton's Second Law of Motion:

    \begin{gather*}
        a_i \cdot z = \frac{F_i}{m} \cdot z \\
        \ddot{z_i} = - \frac{k}{m} (d_i z_i - \sum_{e_{j i} \in E} z_j)
    \end{gather*}

    We can rewrite in the matrix form

    \begin{equation}
        \ddot{z} =  - \frac{k}{m} (D - A) z = - \frac{k}{m} L z
        \label{eq:newton_2nd}
    \end{equation}

    Where $z = (z_1, z_2, ..., z_n)^T$, $A$ is the adjacency matrix of $G$ and $D$ is the degree matrix of $A$ (diagonal matrix where each entry equals to the corresponding degree of the node). $L = D - A$ is called Laplacian matrix of $A$.

    We are seeking the mode of oscillation of the system. A mode of oscillation is a particular frequency where all particles oscillate at the same frequency. At that frequency, the differential equation for each particle must be in the form:

    \begin{equation}
        \ddot{z_i} = - \omega^2 z_i
    \end{equation}

    Where $\omega$ is the frequency. In the matrix form:

    \begin{equation}
        \ddot{z} = - \omega^2 z
        \label{eq:oscillation_mode}
    \end{equation}

    From \ref{eq:newton_2nd} and \ref{eq:oscillation_mode}, we have

    \begin{equation}
        L z = \frac{m}{k} \omega^2 z
        \label{eq:oscillation_mode_freq}
    \end{equation}

    From \ref{eq:oscillation_mode_freq}, the oscillation mode frequencies are equivalent to the eigenvalues of the Laplacian matrix, and the initial condition to achieve each of the frequencies is the corresponding eigenvector.

    Since $L$ is real symmetric, by the Spectral Theorem, it has an eigenbasis. Furthermore, $L$ is positive semi definite, then all of its eigenvalues are positive, hence the frequencies make sense.

    For an arbitrary initial condition, since the system is linear, we can decompose the displacement $z$ into the eigenbasis of the Laplacian matrix then solve each of the component individually.

    \section{Graph Laplacian Basis}

    Recall that, the eigen decomposition of $L$ is as follow:

    \begin{equation}
        L = U \Lambda U^T = U \Lambda U^{-1}
    \end{equation}

    Where each column vector in $U$ is a normalized eigenvector.

    Analogous to Fourier Transform, the eigenvalues of Laplacian matrix can serve as the frequency and the eigenbasis is corresponding to the Fourier basis.

    Let $x \in \mathbb{R}^n$ be a graph signal on $G = (V, E)$ where each component of $x$ is a real number corresponding to a node in $G$.

    The convolution in spatial domain is equivalent to multiplication in spectral domain. Define the convolution operation as:

    \begin{equation}
        y(x) =  U ( U^T w \odot U^T x )
        \label{eq:conv_vec}
    \end{equation}

    Where $w \in \mathbb{R}^n$ is called filter or kernel. Define $W = diag(U^T w)$ be the diagonal matrix whose entries are the entries of $U^T w$, we can rewrite \ref{eq:conv_vec} as

    \begin{equation}
        y(x) =  (U W U^T) x
    \end{equation}

    \subsection{ChebNet}

    Let $\mathcal{L}$ be the normalized laplacian matrix.

    \begin{equation}
        \mathcal{L} = D^{-\frac{1}{2}} L D^{-\frac{1}{2}} = I - D^{-\frac{1}{2}} A D^{-\frac{1}{2}}
    \end{equation}

    The decomposition of $\mathcal{L}$

    \begin{equation}
        \mathcal{L} = U \Lambda U^T = U \Lambda U^{-1}
    \end{equation}

    \begin{theorem}(Chung \cite{chung1997spectral})
        All eigenvalues of $\mathcal{L}$ are in the interval $[0, 2]$.
    \end{theorem}

    ChebNet \cite{defferrard2016convolutional} approximate the diagonal matrix $W$ using Chebyshev polynomials as the orthogonal basis in the polynomial subspace of the vector space of all functions $f: [-1, +1] \to \mathbb{R}$ with respect to the inner product.

    \begin{equation}
        \int_{-1}^{+1} f(x) g(x) \frac{dx}{\sqrt{1-x^2}}
    \end{equation}

    Chebyshev polynomials of the first kind:

    \begin{equation}
        T_{n+1}(x) = 2x T_n(x) - T_{n-1}(x)
    \end{equation}

    Where $x \in [-1, +1]$, $T_0(x) = 1$ and $T_1(x) = x$. Let $f_i: [-1, +1] \to \mathbb{R}$ is an arbitrary function such that $f_i(\Tilde{\lambda_i}) = w_i$ where $\Tilde{\lambda_i} = \frac{2 \lambda_i}{\lambda_{\max}} - 1 \in [-1, +1]$, $\lambda_{\max}$ is the largest eigenvalue. We want to project $f_i$ into the subspace with the orthogonal basis of the first $K$ terms of Chebyshev polynomials of the first kind. We can write $f_i$ as

    \begin{equation}
        \hat{f}_i(t) = \sum_{k=0}^{K-1} \theta_{k i} T_k(t)
    \end{equation}

    Hence, $w_i$ is approximated as

    \begin{equation}
        \hat{w_i} = \hat{f}_i(\Tilde{\lambda_i}) = \sum_{k=0}^{K-1} \theta_{k i} T_k(\Tilde{\lambda_i})
    \end{equation}

    Matrix form of the approximation on $W$:

    \begin{equation}
        \hat{W} = \sum_{k=0}^{K-1} \theta_k T_k(\Tilde{\Lambda})
    \end{equation}

    Where $\Tilde{\Lambda}$ is the diagonal matrix of $\Tilde{\lambda_i}$. Moreover,

    \begin{equation}
        U \hat{W} U^T = \sum_{k=0}^{K-1} \theta_k U T_k(\Tilde{\Lambda}) U^T = \sum_{k=0}^{K-1} \theta_k T_k(\Tilde{\mathcal{L}})
    \end{equation}

    Where $\Tilde{\mathcal{L}} = \frac{2 \mathcal{L}}{\lambda_{\max}} - I$. It is a great exercise to prove the Chebyshev recurrence for $\Tilde{\mathcal{L}}$:

    \begin{equation}
        T_{n+1}(\Tilde{\mathcal{L}}) = 2 \Tilde{\mathcal{L}} T_n(\Tilde{\mathcal{L}}) - T_{n-1}(\Tilde{\mathcal{L}})
    \end{equation}

    Finally, The convolution operation is

    \begin{equation}
        y(x) = \sum_{k=0}^{K-1} \theta_k T_k(\Tilde{\mathcal{L}}) x
    \end{equation}

    The construction of ChebNet avoids decomposing the matrix $L$ as compare to \ref{eq:conv_vec}.

    \subsection{Graph Convolutional Network}

    Similar to ChebNet, GCN \cite{kipf2016semi} limits $K = 2$ and sets $\lambda_{\max} = 2$ hence $\Tilde{\mathcal{L}} = \mathcal{L} - I$.

    \begin{gather*}
        U \hat{W} U^T = \theta_0 T_0(\Tilde{\mathcal{L}}) + \theta_1 T_1(\Tilde{\mathcal{L}}) \\
        = \theta_0 I + \theta_1 \Tilde{\mathcal{L}} \\
        = \theta_0 I - \theta_1 D^{-\frac{1}{2}} A D^{-\frac{1}{2}}
    \end{gather*}

    The convolution operation is

    \begin{equation}
        y(x) = \theta_0 x - \theta_1 D^{-\frac{1}{2}} A D^{-\frac{1}{2}} x
    \end{equation}

    The notes here is greatly inspired by \cite{chen2020note}.

    \bibliographystyle{plain}
    \bibliography{references}
\end{document}


\begin{frame}
	\frametitle{Random variables}
	A variable quantity whose possible values depend, in random manner, on a set of random outcomes events\footnote{\href{https://en.wikipedia.org/wiki/Random_variable}{Wikipedia}}\vspace{1em}

	Every random variables is defined over \emph{probability space}: $(\Omega, \mathcal{F}, \mathcal{P})$ \footnote{Additional information can be found, among others, in \href{http://vfu.bg/en/e-Learning/Math--Bertsekas_Tsitsiklis_Introduction_to_probability.pdf}{D.P. Bertsekas and J.N. Tsitsiklis. Introduction to Probability}}\vspace{1em}

	Consider the toss of a coin
	\begin{itemize}
		\item $\Omega=\{H, T\}$ is the set of possible outcomes. In this case, head or tail
		\item $\mathcal{F}=\{\{\}, \{H\}, \{T\}, \{H,T\}\}$ is the set of events we consider
		\item $\mathcal{P}$ probability function. It associates elements of $\mathcal{F}$ with a probability value. For example $$\mathcal{P}(\{\})=0,\quad\mathcal{P}(\{H\})=0.5, \quad\mathcal{P}(\{T\})=0.5, \quad \mathcal{P}(\{H, T\})=1$$
	\end{itemize}
\end{frame}

\begin{frame}
	\frametitle{Moments}
	$X$ is a random variable , taking values in $\mathbb{R}$, and having probability density function $f(x)$
	\begin{itemize}
		\item Mean: $\mu=E[X]=\int_\mathbb{R} sf(s)ds$
		\item $n^{th}$ moment: $\int_\mathbb{R} s^nf(s)ds$
		\item $n^{th}$ central moment $E[(X-E[X])^n]=\int_\mathbb{R} (s-\mu)^nf(s)ds$
		\begin{itemize}
			\item Variance: second central moment $E[(X-E[X])^2]=\int_\mathbb{R} (s-\mu)^2f(s)ds$
		\end{itemize}
	\end{itemize}

	\vspace{1em}
	Some variables have moments on infinite value. For example \emph{heavy tailed distribution} \footnote{\href{https://en.wikipedia.org/wiki/Heavy-tailed_distribution}{\small{ Wikipedia. Heavy-tailed distribution}}}
	$$f(x)=\left\{\begin{aligned}
					\frac{1}{x^2} && x \geq 1\\
					0&&\textrm{otherwise}
				   \end{aligned}
	\right.$$
\end{frame}

\begin{frame}
	\frametitle{Mean and mode (most likely outcome)}
	\emph{Mean} and \emph{mode} are different concept\vspace{0.5em}
	\begin{itemize}
		\item Mean: weighted sum of all of the possible outcomes
		\begin{itemize}
			\item Mean value could lie outside the set of possible outcomes
		\end{itemize}
		\item Mode: An outcome with the highest probability value
	\end{itemize}
	\vspace{0.5em}
	\begin{columns}
		\column{0.5\textwidth}
		$X$ is a random variable taking values
		\begin{itemize}
			\item 0 with probability 0.2
			\item 1 with probability 0.8
		\end{itemize}
		\vspace*{0.5em}

	    Mean value: $\mu=0.2\cdot 0 + 0.8 \cdot 1=0.8$\\\vspace{0.5em}

		Mode: $\argmax_{x\in\{0,1\}} p(x) \;=1$

		\column{0.4\textwidth}
		\begin{block}{Probability of $X$}
			\centering
			\begin{tikzpicture}
			%\draw[help lines, color=gray!30, dashed] (-4.9,-4.9) grid (4.9,4.9);
			\draw[->, thick] (-0.5,0)--(1.5,0) node[right]{$x$};
			\draw[->, thick] (-0.5,0)--(-0.5,1) node[above]{$p(x)$};
			\draw (0, 0) -- (0, 0.2) node [above] {$0.2$};
			\draw (1, 0) -- (1, 0.8) node [above] {$0.8$};
			\node [below] at (0,0) {0};
			\node [below] at (1,0) {1};
			\filldraw[fill=black!40, draw=black](0,0.2) circle (0.05cm);
			\filldraw[fill=black!40, draw=black](1,0.8) circle (0.05cm);
			\end{tikzpicture}
		\end{block}
	\end{columns}

	%Under certain assumptions (e.g., Ergodicity) the average of

	\takeaway{\bf{The mean value is not even an element of the possible outcomes}}
\end{frame}
\begin{frame}
	\frametitle{Sum of independent random variable}
	\onslide<1->The distribution of the sum of two random variables must always be carefully computed\vspace{0.5em}
	\begin{columns}\onslide<1->
		\column{0.5\textwidth}
		\begin{itemize}
			\item <1->$X$ uniformly distributed between 0 and 1%, $X\sim\mathcal{U}(0, 1)$
			\item <1->$Y$ uniformly distributed between 0 and 1
			\item <1->$Z=X+Y$ \emph{is not} uniformly distributed between 0 and 1
			\item <2-> The distribution of $Z$ depends on the joint distribution of $X$ and $Y$
			\item <3-> If $X$ and $Y$ are independent variables, then $Z$ has a triangular distribution\\
			\begin{itemize}
				\item $X$ and $Y$ are independent if
				for all $x$ and $y$ $$P(X\leq x,Y\leq y)=P(X\leq x)\cdot P(Y \leq y)$$
			\end{itemize}
		\end{itemize}

		\column{0.46\textwidth}
		\begin{block}{Probability density functions}
			\onslide<1->{
			\begin{tikzpicture}
				\draw[->] (-0.5,0)--(1.5,0) node[right]{$x$};
				\draw[->] (-0.5,0)--(-0.5,1.5) node[above]{$p(x)$};
				\draw[very thin, dashed] (-0.5, 1)node[left] {1} -- (0, 1);
				\draw[thin](0, 0) node[below] {0}-- (0, 1);
				\draw[thick] (0, 1) -- (1, 1);
				\draw[thin] (1, 0)node[below] {1} -- (1, 1);
			\end{tikzpicture}}
			\onslide<3->{
			\begin{tikzpicture}
			\draw[->] (-0.5,0)--(2.5,0) node[right]{$z$};
			\draw[->] (-0.5,0)--(-0.5,1.5) node[above]{$p(z)$};
			\draw[very thin, dashed] (-0.5, 1)node[left] {1} -- (1, 1);
			\draw[thick](0, 0) node[below] {0}-- (1, 1);
			\draw[very thin, dashed] (1, 0)node[below]{1} -- (1, 1);
			\draw[thick] (2, 0)node[below] {2} -- (1, 1);
			\end{tikzpicture}}
		\end{block}
	\end{columns}
\end{frame}
\begin{frame}
	\frametitle{Sum of variables normally distributed random variables}
	The Gaussian distribution is also called \emph{Normal} distribution and is denoted with the symbol $\mathcal{N}$\vspace{0.5em}

	\begin{itemize}
		\item The probability density function of a normally distributed random variable $X$ is
		$$f_X(x;\mu,\sigma)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^s}{2\sigma^2}}$$
		\item <2-> If $X\sim\mathcal{N}(\mu_x, \sigma_x^2)$ and $Y\sim\mathcal{N}(\mu_y, \sigma_y^2)$ and are independent, then
		\item <2-> $Z=X+Y$ is also normally distributed;\vspace{1em} $Z\sim\mathcal{N}(\mu_x+\mu_y, \sigma_x^2+\sigma_y^2)$
		%\item <2-> If $\alpha$ is a non-zero constant, then $\alpha Z$ is normally distributed
		%\begin{itemize}
		%	\item mean: $\alpha \mu_z$, variance: $\alpha^2\sigma_z^2$
		%\end{itemize}

		\item <3-> If $\bm{Z}$  has a multivariate normal distribution with mean $\bm{\mu}_z$ and covariance $L=E[(Z-\bm{\mu}_z)(Z-\bm{\mu}_z)^T]$
		and $A$ is a matrix, then
		\item <3-> The variable $\bm{S}=A\bm{Z}$ is normally distributed with mean $A \bm{\mu}_z$ and covariance matrix is $ALA^T$
	\end{itemize}
\onslide<2>{
\takeaway{\large Note however that $X\cdot Y$ is \emph{not} normally distributed}}
\end{frame}

\section{Random variables and dynamic systems}
\separatorslide

\begin{frame}
	\frametitle{Gaussian distributions and linear systems}
	Assume
	\begin{columns}
		\column{0.6\textwidth}
		\begin{itemize}
			\item $X(0)$ is normally distributed, $X(0)\sim\mathcal{N}(\mu_0, \sigma_0^2)$
			\item $W(k)$ is normally distributed, $W(k)\sim\mathcal{N}(\mu_{w,k}, \sigma_{w,k}^2)$
			\item $X(0)$ and $W(k)$ are independent for all $k$
			\item for $k\geq0$ the following recursive equation holds: $X(k+1) =X(k) + W(k)$
		\end{itemize}
		\column{0.4\textwidth}
		\begin{block}{Graphical representation}
			\begin{tikzpicture}
			\node(X0){$X(0)$};
			\node[above=1em of X0](W0){$W(0)$};
			\node[right=1em of X0](sum1){$+$};
			\node[right=1em of sum1](X1){$X(1)$};
			\node[above=1.2em of X1](W1){$W(1)$};
			\node[right=1em of X1](sum2){$+$};
			\node[right=1em of sum2](X3){$\cdots$};
			\draw[->] (W0) to (sum1);
			\draw[->] (X0) to (sum1);
			\draw[->] (sum1) to (X1);
			\draw[->] (W1) to (sum2);
			\draw[->] (X1) to (sum2);
			\draw[->] (sum2) to (X3);
			\end{tikzpicture}
		\end{block}
	\end{columns}

	\vspace*{0.5em}

	\onslide<1-> What is the distribution of $X(1)$ ?
	\begin{itemize}\onslide<2->
		\item $X(1)$ is normally distributed (sum of two independent Gaussian variables)
		\item Mean $\mu_1=\mu_0+\mu_{w,0}$, variance $\sigma_1^2 = \sigma_0^2 + \sigma_{w,0}^2$
	\end{itemize}

	\vspace*{0.5em}
	\onslide<2-> What is the distribution of $X(k)$ ?
	\begin{itemize}\onslide<3->
		\item $X(k)$ is Gaussian distributed% (sum of independent Gaussian variables)
		\item Mean $\mu_k=\mu_0+\sum_{j=0}^{k-1}\mu_{w,j}$, variance $\sigma_k^2=\sigma_0^2 + \sum_{j=0}^{k-1}\sigma_{w,j}^2$
	\end{itemize}
\end{frame}

\begin{frame}
	\frametitle{Non Gaussian distributions and linear systems}
	Assume
	\begin{columns}
		\column{0.6\textwidth}
		\begin{itemize}
			\item $X(0)$ is uniformly distributed, $X(0)\sim\mathcal{U}(0, 1)$
			\item $W(k)$ is uniformly distributed, $W(k)\sim\mathcal{U}(0, 1)$
			\item $X(0)$ and $W(k)$ are independent for all $k$
			\item for $k\geq0$ the following recursive equation holds: $X(k+1) =X(k) + W(k)$
		\end{itemize}
		\column{0.4\textwidth}
		\begin{block}{Graphical representation}
			\begin{tikzpicture}
			\node(X0){$X(0)$};
			\node[above=1em of X0](W0){$W(0)$};
			\node[right=1em of X0](sum1){$+$};
			\node[right=1em of sum1](X1){$X(1)$};
			\node[above=1.2em of X1](W1){$W(1)$};
			\node[right=1em of X1](sum2){$+$};
			\node[right=1em of sum2](X3){$\cdots$};
			\draw[->] (W0) to (sum1);
			\draw[->] (X0) to (sum1);
			\draw[->] (sum1) to (X1);
			\draw[->] (W1) to (sum2);
			\draw[->] (X1) to (sum2);
			\draw[->] (sum2) to (X3);
			\end{tikzpicture}
		\end{block}
	\end{columns}

	\vspace*{0.5em}

	\onslide<2-> What is the distribution of $X(1)$ ?
	\begin{itemize}\onslide<3->
		\item $X(1)$ has a triangular distribution
		\item Mean $\mu_1=\mu_0+\mu_{w,0}$, variance $\sigma_1^2 = \sigma_0^2 + \sigma_{w,0}^2\;$\footnote{\href{http://eli.thegreenplace.net/2009/01/07/variance-of-the-sum-of-independent-variables}{See also here}}
	\end{itemize}

	\vspace*{0.5em}
	\onslide<4-> What is the distribution of $X(k)$ ?
	\begin{itemize}\onslide<4->
		\item The distribution of $X(k)$ depends on the distribution of $X(k-1)$ and of $W(k-1)$
		\item Mean $\mu_k=\mu_0+\sum_{j=0}^{k-1}\mu_{w,j}$, variance $\sigma_k^2=\sigma_0^2 + \sum_{j=0}^{k-1}\sigma_{w,j}^2$

	\end{itemize}
\end{frame}

\begin{frame}
	\frametitle{Gaussian distributions and non linear systems}
	Assume
	\begin{columns}
		\column{0.6\textwidth}
		\begin{itemize}
			\item $X(0)$ is normally distributed $X(0)\sim\mathcal{N}(0, 1)$
			\item $W(k)$ is normally distributed $W(k)\sim\mathcal{N}(0, 1)$
			\item $X(0)$ and $W(k)$ are independent for all $k$
			\item for $k\geq0$ we have $X(k+1) =X^2(k) + W(k)$
		\end{itemize}
		\column{0.4\textwidth}
		\begin{block}{Graphical representation}
			\begin{tikzpicture}
			\node(X0){$X(0)$};
			\node[above=1em of X0](W0){$W(0)$};
			\node[right=1em of X0](sum1){$+$};
			\node[right=1em of sum1](X1){$X(1)$};
			\node[above=1.2em of X1](W1){$W(1)$};
			\node[right=1em of X1](sum2){$+$};
			\node[right=1em of sum2](X3){$\cdots$};
			\draw[->] (W0) to (sum1);
			\draw[->] (X0) to (sum1);
			\draw[->] (sum1) to (X1);
			\draw[->] (W1) to (sum2);
			\draw[->] (X1) to (sum2);
			\draw[->] (sum2) to (X3);
			\end{tikzpicture}
		\end{block}
	\end{columns}

	\vspace*{0.5em}

	\onslide<2-> What is the distribution of $X(1)$ ?
	\begin{itemize}\onslide<3->
		\item $X(1)$ is not normally distributed.
		\item Mean $\mu_1=E[X(0)^2]+\mu_{w,0}$
		\item Variance $\sigma_1^2=var(X(0))+\sigma_{w,0}^2$
	\end{itemize}
\end{frame}

\begin{frame}
	\frametitle{Few remarks}
	\begin{itemize}
		\setlength\itemsep{2em}
		\item Mean of the sum is always equal to the sum of the means (linear operator)
		\item If two random variables are independent, then the variance of the sum is equal to the sum of the variances
		%\item Variables $X(k)$ are normally distributed for all $k\geq0$ \emph{only} in the case of linear systems and if the variables $w(k)$ are normally distributed
		\item The knowledge of mean and variance rarely completely characterize the distribution of a random variable
	\end{itemize}
\end{frame}

\section{Marginal and conditional density functions}
\separatorslide
\begin{frame}
	\frametitle{Marginal and conditional density functions}
	Assume $X$ and $Y$ are two random variables taking values in the interval $[0, 1]$\vspace{0.5em}\\
	and $f_{X,Y}(x,y)$ is the joint probability density function of $X$ and $Y$\vspace{0.5em}
	%The following probabilities can be computed from the joint distribution of $X$ and $Y$
 	\begin{itemize}%\setlength\itemsep{0.5em}
 		\item <1-> Marginal probability density functions are given by
 		$$f_{X}(x)=\int_0^1f_{X,Y}(x,z) dz\;, \qquad f_{Y}(y)=\int_0^1f_{X,Y}(z,y) dz$$

 		%Probability of $X$ taking value $x$ independently of the value taken by $Y$
 		%\item Marginal probability density function
 		%Probability of $X$ taking value $x$ independently of the value taken by $Y$
 		%\item <1-> %Probability of $Y$ taking value $y$ independently of the value taken by $X$
 		%$P(Y\leq y)=\int_0^y\int_0^1 P(X=z, Y=y) dz$ (Marginal distribution)
 		%\item <2-> %Probability of $Y$ taking value $y$ once we know that $X$ has value $x$
 		% $P(Y\leq y|X=x)=\int_0^y \frac{P(X=x, Y=y)}{P(X=x)}$ (Conditional distribution)
 		%\item <2->Probability of $X$ taking value $x$ once we know that $Y$ has value $y$ %$P(X=x|Y=y)=\frac{P(X=x, Y=y)}{P(Y=y)}$ (Conditional distribution)

 		\item <2-> Conditional probability density function: $f_{X|Y}(x;y)=\frac{f_{X,Y}(x,y)}{f_{Y}(y)},$ if $f_Y(x)>0$
 		\item <3-> Conditional mean and variance of $X$ given $Y$
		$$E[X|Y=y]\!=\int_0^1\!\! zf_{X|Y}(z;y)dz,\quad var(X|Y=y)\!=\int_0^1 \!\!(z-E[X|Y=y])^2f_{X|Y}(z;y)dz$$
 	\end{itemize}

\end{frame}

\begin{frame}
	\frametitle{Conditional distribution and expected conditional risk}
	Consider the case of estimating the value of a variable $X$ from its measurement $Y$
	\begin{itemize}
		\item For example $Y=X+W$, where $W$ is random noise independent of $X$
		\item The conditional distribution of $X|Y$ provides information about the probability of different values of $X$ given that $Y=y$
		\item <3-> For selecting a best value, we need a cost function, e.g., the \emph{expected conditional risk}
		$$R(t,y)=E[c(t-X)|Y=y]=\int_{\mathbb{R}}c(t-z)f_{X|Y}(z;y)dz$$
	\end{itemize}

	\onslide <3->If
	\begin{itemize}
		\item $c(t-x)$ is symmetric, e.g., $c(z-x)=(z-x)^2$ and
		\item the conditional density function $f_{X|Y}(z;y)$ is symmetric around $E(X|Y=y)$,
	\end{itemize}
	then
	\begin{itemize}
		\item the minimum of $R(t,y)$ does not depends on the specific cost function $c(t-X)$
		\item \emph{the minimum of $R(t,y)$ is given by $E[X|Y=y]$}
	\end{itemize}
%\onslide<2->
%\takeaway{Under these assumption, the best estimate of $X$ given $Y=y$ is $E[X|Y=y]$}
\onslide<2>\takeaway{\bf{How could we select a best \emph{value} for $X$?}}
\end{frame}

\begin{frame}
	\frametitle{The Maximum a posteriori estimator}
	Another approach for selecting a best value for the estimation of $X$ given $Y=y$ could be to take the maximum of $f_{X|Y}(z;y)$ \vspace{2em}
	\begin{itemize}
		\item We can define $$\hat{x}(y)=\argmax_{z\in\mathbb{R}}f_{X|Y}(z;y)$$
		\item This estimator is called \emph{Maximum a posteriori} (MAP)
	\end{itemize}
	\begin{itemize}
		\item For unimodal and symmetric distributions, e.g., Gaussian distribution, then MAP estimator and the expected conditional value estimator coincide
	\end{itemize}
\takeaway{In general, MAP and the expected conditional value are different}
\end{frame}

%\begin{frame}
%	\frametitle{Some properties of the conditional expectation}
%	Assumptions:
%	\begin{itemize}
%		\item $c(z-x)$ is symmetric, e.g., $c(z-x)=(z-x)^2$
%		\item the conditional distribution of $(X|Y=y)$ is symmetric around $E(X|Y=y)$
%	\end{itemize}\vspace{1em}
%	then
%	\begin{itemize}
%		\item the minimum of $R(z,y)$ does not depends on the specific cost function $c(z-X)$
%		\item the minimum of $R(z,y)$ is given by $E[X|Y=y]$
%	\end{itemize}\vspace{1.5 em}
%
%	\takeaway{Under these assumptions mean and mode of the conditional distribution coincide}
%\end{frame}


\subsection{Ordinary differential equations}

An ordinary differential equation is one with only one independent variable. For example:

\(\dfrac{dy}{dx}=f(x)\)

The order of a differential equation is the number of differentials of \(y\) included. For example one with the second derivative of \(y\) is of order \(2\).

Ordinary equations can can either implicit or explicit. An explicit function shows the highest order derivative as a function of other terms.

An implicit function is one which is not explicit.

A linear ODE is an explicit ODE where the derivative terms of \(y\) do not multiply together, that is, in the form:

\(y^{(n)}=\sum_ia_i(x)y^{(i)}+r(x)\)

\subsubsection{First-order ODEs}

We have an evolution:

\(\dfrac{dy}{dt}=f(t,y)\)

And a starting condition:

\(y_0=f(t_0)\)

We now discuss various ways to solve these.


\section{Generalized Linear Models}
We relax the assumption that $\mu$ is linear. Instead, we assume that g $\circ \mu$ is linear, for some function $g$:\\

$g(\mu (\mathbf x)) = \mathbf x^ T \beta$

The function $g$ is assumed to be known, and is referred to as the link function. It maps the domain of the dependent variable to the entire real Line.

it has to be strictly increasing,

it has to be continuously differentiable and

its range is all of $\mathbb{R}$

\subsection{The Exponential Family}

A family of distribution $\, \{ \mathbf{P}_{{\boldsymbol \theta }}: {\boldsymbol \theta }\in \Theta \} ,\,$  where the parameter space $\Theta \subset \mathbb {R}^ k\,$ is -$k$ dimensional, is called a $k$-parameter exponential family on $\mathbb{R}^1$ if the pmf or pdf $\, f_{\boldsymbol \theta }:\mathbb {R}^ q\to \mathbb {R}\,$ of $\, \mathbf{P}_{{\boldsymbol \theta }}\,$ can be written in the form:\\

$\displaystyle  \displaystyle f_{\boldsymbol \theta }(\mathbf{y})=h(\mathbf{y})\, \exp \left({\boldsymbol \eta }({\boldsymbol \theta })\cdot \mathbf{T}(\mathbf{y})-B({\boldsymbol \theta })\right)\qquad \text {where } \\ \begin{cases}  {\boldsymbol \eta }({\boldsymbol \theta })=\begin{pmatrix} \eta _1({\boldsymbol \theta })\\ \vdots \\ \eta _ k({\boldsymbol \theta })\end{pmatrix}& :\mathbb {R}^ k\to \mathbb {R}^ k\\ \mathbf{T}(\mathbf{y})=\begin{pmatrix} T_1(\mathbf{y})\\ \vdots \\ T_ k(\mathbf{y})\end{pmatrix}& :\mathbb {R}^ q\to \mathbb {R}^ k\\ B({\boldsymbol \theta })& :\mathbb {R}^ k\to \mathbb {R}\\ h(\mathbf{y})& :\mathbb {R}^ q\to \mathbb {R}.\\ \end{cases}$\\

if $k=1$ it reduces to:\\

$\displaystyle  \displaystyle f_\theta (y)=h(y)\, \exp \left(\eta (\theta ) T(y)-B(\theta )\right)$


\section{The CLs upper limit}
\label{sec:CLs}

For a signal hypothesized value $\mu$, one can compute the probability that this hypothesis (called S+B hypothesis) gives a \textbf{greater} test statistic value than the observed one $q_{obs}$ as:
\begin{equation}
    p_{s+b} = \int_{q_{obs}}^{\infty} f(q_{\mu}|\mu) d q_{\mu}
\end{equation}
In the meantime, the probability that the background-only hypothesis gives a \textbf{smaller} test statistic than observed data can also be calculated as:
\begin{equation}
    1 - p_{b} = \int_{-\infty}^{q_{obs}} f(q_{\mu}|0) d q_{\mu}
\end{equation}
Then we define the CLs~\cite{Read_2002} of a hypothesized value $\mu$ as:
\begin{equation}
    CLs = \frac{p_{s+b}}{1-p_{b}}
\end{equation}
For purpose of excluding a signal hypothesis, a threshold CLs of 0.05 is often used.
For this reason, usually under the circumstance that no significant derivation between data and background-only hypothesis is found,
one would like to find the value of hypothesized signal strength $\mu$ by requiring its $CLs = 0.05$ (called 95\% CLs upper limit) for exclusion.

The sensitivity of an experiment to exclude a new signal process is quantified by \textit{median upper limit}~\cite{Bellos:2725027},
which is obtained using ``Asimov dataset".
The Asimov dataset is defined such that when one uses it to evaluate the estimators for all parameters, one obtains the true parameter values.
Moreover, it is useful to use Asimov dataset to compute how much the sensitivity is expected to vary, given the expected fluctuations in the data.
The $\hat{\mu}$ is assumed to follow a Gaussian distribution with a mean value of $\mu '$ and the standard deviation of $\sigma$.
First of all, the test statistic from profile likelihood ratio can be approximated as~\cite{Cowan:2010js}:
\begin{equation}
	-2 ln \lambda(\mu) = \frac{(\mu - \hat{\mu})^2}{\sigma^2} + \mathcal{O}(1/\sqrt{N})
\end{equation}
Given that the Asimov dataset corresponding to a signal strength $\mu'$, one finds:
\begin{equation}
	-2 ln \lambda_{A}(\mu) \approx \frac{(\mu - \mu')^2}{\sigma^2} = q_{\mu,A}
\end{equation}
where $q_{\mu,A} = -2ln\lambda_{A}(\mu)$ is the observed test statistic of Asimov dataset.
Then the standard derivation can be computed as:
\begin{equation}
	\sigma_A^2 = \frac{(\mu - \hat{\mu})^2}{q_{\mu,A}}
\end{equation}
In a special situation where one wants to find the median exclusion significance for the hypothesis $\mu$ assuming that there is no signal ($\mu' = 0$),
one gets:
\begin{equation}
        \sigma_A^2 = \frac{\hat{\mu}^2}{q_{0,A}}
\end{equation}


\subsection{An iterative feacture extraction scheme}
One key idea in this paper is that we consider different iterative
	processes to approximately solve \eqref{Auf} and relate them to
	many existing popular CNN models. Here, let us assume that
	the feature-data mapping \eqref{Auf}  is given as a linear form \eqref{linearA}.
	We next propose some iterative schemes to solve \eqref{Auf}
	for an appropriately chosen $u^0$.
	\begin{itemize}
		\item Residual correction method,
		\begin{equation}\label{eq:smoothB}
		u^{i} = u^{i-1} + B^{i}(f- A(u^{i-1})), \quad i=1:\nu.
		\end{equation}
		Here $B^i$ can be chosen as linear like $B^{i}(f) = \eta^i \ast f$ or nonlinear
		like \eqref{extractor}. The reason why $B^{i}$ is taken
		the nonlinear form as in \eqref{extractor} will be discussed later based on our
		main discovery about the relationship
		between MgNet and iResNet as discussed in \S~\ref{sec:CNNs} and \S~\ref{sec:relation}.
		We refer to \cite{xu1992iterative}
		for more discussion on iterative schemes in the form of \eqref{eq:smoothB}.
		\item Chebyshev semi-iterative method,
		\begin{equation}\label{eq:chebysev-semi}
		u^{i} = \omega^i\left(u^{i-1} + B^{i}\left(f- A(u^{i-1})\right)\right)+ (1- \omega^i) u^{i-2},\quad i=1:\nu.
		\end{equation}
		The above scheme can be obtained from the above semi-iterative form
		by applying the Chebyshev polynomial theory \cite{hackbusch1994iterative, golub2012matrix}.
		Similar to the previous case, considering the iterative form of the residual $r^j = f - A(u^{j})$,
		\eqref{eq:chebysev-semi} implies that
		\begin{equation}
		r^{i} = \omega^i r^{i-1} + (1-\omega^i)r^{i-2} - AB^i r^{i-1}.
		\end{equation}
		This scheme corresponds to the LM-ResNet in \cite{lu2018beyond}
		which was obtained as a linear multi-step scheme for some underlying ODEs.
	\end{itemize}


\section*{Two-state System}

\begin{center}
  \begin{tabular}{>{$\textstyle}c<{$}*{2}{>{$\textstyle}r<{$}}}
    \toprule %
    \multicolumn{1}{c}{$\mathcal{O}$} %
    &\multicolumn{1}{c}{$\lambda$} %
    &\multicolumn{1}{c}{$\ket{\psi^{\text{(eigen)}}}$} %
    \\ \midrule %
    \arrayrulecolor{lightgray} %
    \sigma_x \equiv X %
    &\num{+1} %
    & \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix} %
    \\%
    \begin{pmatrix} 0 & 1\\ 1 & 0\end{pmatrix} %
      & \num{-1} %
      & \frac{1}{\sqrt{2}}\begin{pmatrix} -1 \\ 1 \end{pmatrix} %
      \\ \hline %
      \sigma_y\equiv Y %
      & \num{+1} %
      & \frac{1}{\sqrt{2}}\begin{pmatrix} -1 \\ -i \end{pmatrix} %
      \\ %
      \begin{pmatrix} 0 & -i\\ i & 0\end{pmatrix} %
        & \num{-1} %
        & \frac{1}{\sqrt{2}}\begin{pmatrix} -1 \\ i \end{pmatrix} %
        \\ \hline %
    \sigma_z\equiv Z %
    & \num{+1} %
    & \begin{pmatrix} 1 \\ 0 \end{pmatrix} %
    \\ %
    \begin{pmatrix} 1 & 0\\ 0 & -1 \end{pmatrix} %
        & \num{-1} %
    & \begin{pmatrix} 0 \\ 1 \end{pmatrix} %
    \\
\bottomrule
\end{tabular}
\end{center}

\begin{align*}
\ket{\psi} &= \cos(\theta/2)\ket{0} + e^{i\phi}\sin(\theta/2)\ket{1} %
\\ %
(x, y, z) &= (\sin\theta \cos\phi, \sin\theta \sin\phi, \cos\theta)
\end{align*}

\begin{center}
  \begin{tabular}{>{$\textstyle}c<{$}*{3}{>{$\textstyle}r<{$}}}
    \toprule %
    \multicolumn{1}{c}{$\mathcal{O}$} %
    &\multicolumn{1}{c}{$\lambda$} %
    &\multicolumn{1}{c}{$\ket{\psi^{\text{(eigen)}}}$} %
    &\multicolumn{1}{c}{$(\theta, \phi)$}%
    \\ \midrule %
    \arrayrulecolor{lightgray} %
    \sigma_x \equiv X %
    &\num{+1} %
    & \frac{1}{\sqrt{2}} \left(\ket{0} + \ket{1} \right) %
    & (\frac{\pi}{2},0) %
    \\%
    \dyad{0}{1} + \dyad{1}{0} %
    & \num{-1} %
    & \frac{1}{\sqrt{2}} \left(\ket{0} - \ket{1} \right) %
    & (\frac{\pi}{2},-\pi) %
    \\ \hline %

    \sigma_y\equiv Y %
    & \num{+1} %
    & \frac{1}{\sqrt{2}} \left( \ket{0} + i \ket{1} \right) %
    & (\frac{\pi}{2},\frac{\pi}{2}) %
    \\ %
    -i\left( \dyad{0}{1} - \dyad{1}{0} \right) %
      & \num{-1} %
      & \frac{1}{\sqrt{2}} \left(\ket{0} - i \ket{1} \right) %
    & (\frac{\pi}{2},-\frac{\pi}{2}) %
      \\ \hline %

    \sigma_z\equiv Z %
    & \num{+1} %
    & \ket{0} %
    & (0,0) %
    \\ %
    \dyad{0}{0} - \dyad{1}{1} %
    & \num{-1} %
    & \ket{1} %
    & (\pi, 0)
    \\
    \arrayrulecolor{black} %
\bottomrule
\end{tabular}
\end{center}

% Matches the work in QIS-Exercises
%\begin{center}
%  \begin{tabular}{>{$\textstyle}c<{$}*{3}{>{$\textstyle}r<{$}}}
%    \hline %
%    \mathcal{O} & \lambda & \psi^{\text{(eigen)}} & (\theta, \phi)%
%    \\ \hline %
%    \sigma_x \equiv X %
%    &\num{+1} %
%    & \frac{1}{\sqrt{2}} \left( +\ket{0} + \ket{1} \right) %
%    & (\frac{\pi}{2},0) %
%    \\%
%    \dyad{0}{1} + \dyad{1}{0} %
%    & \num{-1} %
%    & \frac{1}{\sqrt{2}} \left( -\ket{0} + \ket{1} \right) %
%    & (\frac{\pi}{2},\pi) %
%    \\ \hline %
%
%    \sigma_y\equiv Y %
%    & \num{+1} %
%    & \frac{1}{\sqrt{2}} \left( -\ket{0} - i \ket{1} \right) %
%    & (\frac{\pi}{2},\frac{\pi}{2}) %
%    \\ %
%    -i\left( \dyad{0}{1} - \dyad{1}{0} \right) %
%      & \num{-1} %
%      & \frac{1}{\sqrt{2}} \left( -\ket{0} + i \ket{1} \right) %
%    & (\frac{\pi}{2},-\frac{\pi}{2}) %
%      \\ \hline %
%
%    \sigma_z\equiv Z %
%    & \num{+1} %
%    & \ket{0} %
%    & (0,0) %
%    \\ %
%    \dyad{0}{0} - \dyad{1}{1} %
%    & \num{-1} %
%    & \ket{1} %
%    & (\pi, 0)
%    \\
%\hline
%\end{tabular}
%\end{center}

\subsection*{Commutators for Two-State System}


\chapter{Stochastic Neural Networks}
Neurons fire in a probabilistic manner. The activation function of a stochastic neuron is non-deterministic, but probabilistic.

\section{Boltzmann Machine}
Boltzmann machine is a generalization of Hopfield network. Boltzmann-Gibbs distribution:
$$P(1|u)=\frac{1}{1+e^{-u/T}}$$
\begin{center}where T is the temperature of transitions \end{center}

\section{Stochastic Recurrent Network}
Consider transition from ($u_1$, $u_2$) to ($u_1'$, $u_2$). Number of state $N$ is 2.
\begin{equation*}
\begin{split}
P(u_1\ change\ to\ u_1') &= P(first\ neuron\ to\ fire) \times P(u_1' | u_1) \\
&= \frac{P(u_1' | u_1)}{N}
\end{split}
\end{equation*}
If $u_1$' is 1 and $u_1$ is 0, then:
$$P(u_1\ change\ to\ u_1') = \frac{P(1| u_1)}{N}$$
And the numerator can be found using Boltzmann equation and u is 0

\section{Network Entropy}
Entrophy gives a measure of randomness of the network at a given state.
$$H_i=-\sum{p(i,j)log_e p(i,j)}$$

\section{State-Transition Matrix}
$$
S =
\begin{bmatrix}
p(0,0) & p(1,0) & \cdots & p(m-1, 0) \\
p(0,1) & p(1,1) & \cdots & p(m-1, 1) \\
\vdots & \vdots & \ddots & \vdots \\
p(0,m-1) & p(1,m-1) & \cdots & p(m-1,m-1)
\end{bmatrix}
$$
Columns: current states \\
Rows: next states \\
$p(i|j)$: probability of transition from state i to state j

\section{Markov Chain}
$$\boldsymbol{X}(0) \rightarrow \boldsymbol{X}(1) \rightarrow \ldots \ \boldsymbol{X}(k)$$
$\boldsymbol{X}(k)$ is the vector of state probabilities at time $k$ \\
$X_0(k)$ is probability of the network being at state $S_i$ at time $k$

$$\boldsymbol{X}(k+1) = \boldsymbol{S}\boldsymbol{X}(k)$$

$$\boldsymbol{X}(k) = \boldsymbol{S}^{k} \boldsymbol{X}(0)$$

\section{Simulated Annealing}
Simulated Annealing provides a \emph{scheme} that allows a network to jump out of a local minima towards the \emph{global minima}. Simulated Annealing begins at high temperature to have sufficient randomness for the neurons to jump out of local minima and gradually lowers the temperature to facilitate settling down at global minima.


\section{Exact couples}
Let us begin with a conceptual discussion of exact couples. As a special case,
we will recover the construction of the spectral sequence associated to a
filtered chain complex (Theorem-Definition \ref{filtered-sseq}).
\begin{definition}
    An \emph{exact couple} is a diagram of (possilby (bi)graded) abelian
    groups
    \begin{equation*}
	\xymatrix{
	    A\ar[rr]^i & & A\ar[dl]_j\\
	    & E\ar[ul]_k &
	    }
    \end{equation*}
    which is exact at each joint.
\end{definition}
As $jkjk = 0$, the map $E\xar{jk} E$ is a differential, denoted $d$. An exact
couple determines a ``derived couple'':
\begin{equation}\label{derived-couple}
    \xymatrix{
	A^\prime\ar[rr]^{i^\prime = i|_{\img i}} & &
	A^\prime\ar[dl]_{j^\prime}\\
	& E^\prime\ar[ul]_{k^\prime} &
        }
\end{equation}
where $A^\prime = \img(i)$ and $E^\prime = H_\ast(E,d)$. Iterating this
procedure, we get exact sequences
\begin{equation*}
    \xymatrix{
	A^r\ar[rr]^{i_r} & & A^r\ar[dl]_{j_r}\\
	& E^r\ar[ul]_{k_r} &
        }
\end{equation*}
where the next exact couple is the derived couple of the preceding exact
couple.

It remains to define the maps in the above diagram. Define $j^\prime(ia) = ja$.
\emph{A priori}, it is not clear that this well-defined. For one, we need $[ja]
\in E^\prime$; for this, we must check that $dja = 0$, but $d=jk$, and $jkja =
0$ so this follows. We also need to check that $j^\prime$ is well-defined
modulo boundaries. To see this, suppose $ia = 0$. We then need to know that
$ja$ is a boundary. But if $ia = 0$, then $a = ke$ for some $e$, so $ja = jke =
de$, as desired.

Define $k^\prime:H(E,d) \to \img i$ via $k^\prime([e])\mapsto ke$. As before,
we need to check that this is well-defined. For instance, we have to check that
$ke\in\img i$. Since $de = 0$ and $d = jk$, we learn that $jke = 0$. Thus $ke$
is killed by $j$, and therefore, by exactness, is in the image of $i$. We also
need to check that $k^\prime$ is independent of the choice of representative of
the homology class. Say $e = de^\prime$. Then $kd = kd e^\prime = kjke^\prime =
0$.
\begin{exercise}
    Check that these maps indeed make diagram \eqref{derived-couple} into an
    exact couple.
\end{exercise}
It follows that we obtain a spectral sequence, in the sense of
Theorem-Definition \ref{filtered-sseq}.
\begin{exercise}\label{explicit-er}
    By construction,
    $$A^r = \img(i^{r}|_{A}) = i^{r} A.$$
    Show, by induction, that
    $$E^r = \frac{k^{-1}(i^{r}A)}{j(\ker i^{r})}$$
    and that
    \begin{equation*}
	i_r(a) = ia,\ j_r(i^r a) = [ja],\ k_r(e) = ke.
    \end{equation*}
\end{exercise}
Intuitively: an element of $E^1$ will survive to $E^r$ if its image in $A^1$
can be pulled back under $i^{r-1}$. The differential $d^r$ is obtained by
the homology class of the pushforward of this preimage via $j$ to $E^1$.
\begin{remark}\label{bigraded}
    In general, the groups in consideration will be bigraded. It is clear by
    construction that $\deg(i') = \deg(i)$, $\deg(k') = \deg(k)$, and $\deg(j')
    = \deg(j)-\deg(i)$. It follows by an easy inductive argument that
    $$\deg(d^r) = \deg(j)+\deg(k)-(r-1)\deg(i).$$
\end{remark}
The canonical example of an exact couple is that of a filtered complex; the
resulting spectral sequence is precisely the spectral sequence of
Theorem-Definition \ref{filtered-sseq}. If $C_\ast$ is a filtered chain
complex, we let $A_{s,t} = H_{s+t}(F_s C_\ast)$, and $E^1_{s,t} = E_{s,t} =
H_{s+t}(\gr_s C_\ast)$. The exact couple is precisely that which arises from
the long exact sequence in homology associated to the short exact sequence of
chain complexes
$$0\to F_{s-1} C_\ast\to F_s C_\ast \to \gr_s C_\ast\to 0.$$
Note that in this case, the exact couple is one of bigraded groups, so Remark
\ref{bigraded} dictates the bidegrees of the differentials.

We will conclude this section with a brief discussion of the convergence of the
spectral sequence constructed above.  Assume that $i:A\to A$ satisfies the
property that
$$\ker(i)\cap\bigcap i^r A = 0.$$
Let $\wt{A}$ be the colimit of the directed system
$$A\xar{i} A\xar{i} A\to\cdots$$
There is a natural filtration on $\wt{A}$. Let $I$ denote the image of the map
$A\to \wt{A}$; the kernel of this map is $\bigcup \ker(i^r)$. The groups $i^r
I$ give an exhaustive filtration of $\wt{A}$, and the quotients $i^r I/i^{r+1}
I$ are all isomorphic to $I/i I$ (since $i$ is an isomorphism on $\wt{A}$).
Then we have an isomorphism
\begin{equation}\label{einfty}
    E^\infty \simeq I/iI.
\end{equation}
Indeed, we know from Exercise \ref{explicit-er} that
$$E^\infty \simeq \frac{k^{-1}\left(\bigcap i^r A\right)}{j\left(\bigcup \ker
i^{r}\right)};$$
by our assumption on $i$, this is
$$\frac{\ker(k)}{j\left(\bigcup \ker i^{r}\right)} \simeq
\frac{j(A)}{j\left(\bigcup \ker i^r\right)}.$$
But there is an isomorphism $A/iA\to j(A)$ which clearly sends $iA + \bigcup
\ker i^r$ to $j\left(\bigcup \ker i^r\right)$. By our discussion above,
$A/\bigcup \ker i^r \simeq I$, and $iA/\bigcup \ker i^r \simeq iI$. Modding out
by $iI$ on both sides, we get \eqref{einfty}.

%Our starting point is that you have a chain complex $F_\ast C$.
%We have a short exact sequence $0\to F_{s-1} C\to F_s C \to \gr_s C \to 0$, giving a lexseq in homology:
%$$
%\cdots\to H_{s+t}(F_{s-1}) \to H_{s+t}(F_s) \to H_{s+t}(\gr_s) \to H_{s+t-1}(F_{s-1}) \to \cdots
%$$
%We let $A^1_{s,t} = H_{s+t}(F_{s})$ and $E^1_{s,t} = H_{s+t}(\gr_s)$.
%\textbf{Note: }Weibel uses this notation for something else.
%His $D$ is my $A$, but with different indexing.
%
%Anyway, our diagram can now be rewritten as:
%$$
%\cdots\to A^1_{s-1,t+1} \xar{i} A^1_{s,t} \xar{j} E^1_{s,t} \xar{k} A_{s-1, t} \to \cdots
%$$
%Define:
%$$
%A^r_{s,t} := \img(A^1_{s,t} \xar{i^{r-1}} A^1_{s+r - 1, t-r + 1}) = A^1_{s,t}/\ker(i^{r-1})
%$$
%One thing you can do is look at:
%$$
%\cdots \xar{i} \to A_{s-1} \xar{i} A_s \xar{i} A_{s+1} \xar{i} A_{s+2} \xar{i} \cdots
%$$
%We also have a surjection $\img(i^{r-1}) \to \img(i^r)$ and an injection $i:\img(i^r) \to \img(i^{r-1})$, giving a map $\img(i^{r-1}) \to \img(i^{r-1})$, given by $i$.
%There's another thing you can do: you also have a map $\img(i^r) \to \img(i^{r-1}) \to \img(i^r)$, whose composite is $i$ itself.
%Explicitly, we can write:
%\begin{equation*}
%    \xymatrix{
%	& A^r_{s,t} \ar@{->>}[dr]\ar[rr]^i & & A^r_{s+1,t-1}\ar@{->>}[dr] & \\
%	A^{r+1}_{s-1,t+1}\ar@{>->}[ur]\ar[rr]_i & & A^{r+1}_{s,t}\ar@{>->}[ur]\ar[rr]_i & & A^{r+1}_{s+1,t-1}
%    }
%\end{equation*}
%Great, so in our lexseq, we now have:
%\begin{equation*}
%    \xymatrix{
%	\cdots\ar[r] & A^1_{s-1,t+1}\ar@{->>}[d] \ar[r]^i & A^1_{s,t} \ar@{->>}[d] \ar[r]^j & E^1_{s,t} \ar[r]^k & A^1_{s-1,t}\ar[r]^i & A^1_{s,t-1} \ar[r] & \cdots\\
%	& A^2_{s-1,t+1}\ar@{->>}[d]\ar[r]^i & A^2_{s,t}\ar@{->>}[d] & & A^2_{s-2,t+1}\ar[r]^i \ar@{>->}[u] & A^2_{s-1,t}\ar@{>->}[u] & \\
%	& A^3_{s-1,t+1}\ar[r]^i\ar@{->>}[d] & A^3_{s,t}\ar@{->>}[d] & & A^3_{s-3,t+2}\ar@{>->}[u]\ar[r]^i & A^3_{s-2,t+1}\ar@{>->}[u] &\\
%	& \vdots & \vdots & & \vdots \ar@{>->}[u] & \vdots\ar@{>->}[u] &
%    }
%\end{equation*}
%Recall that for a filtered complex, we really have a surjection:
%$$
%A^r_{s,t} = \img(H_{s+t}(F_s) \to H_{s+t}(F_{s+r-1}) \to \img(H_{s+t}(F_s) \to H_{s+1}(C)) = F_s H_{s+t}(C)
%$$
%In particular, all the vertical surjections in our big diagram maps down surjectively to the filtration.
%More precisely:
%\begin{equation*}
%    \xymatrix{
%	\cdots\ar[r] & A^1_{s-1,t+1}\ar@{->>}[d] \ar[r]^i & A^1_{s,t} \ar@{->>}[d] \ar[r]^j & E^1_{s,t} \ar[r]^k & A^1_{s-1,t}\ar[r]^i & A^1_{s,t-1} \ar[r] & \cdots\\
%	& A^2_{s-1,t+1}\ar@{->>}[d]\ar[r]^i & A^2_{s,t}\ar@{->>}[d] & & A^2_{s-2,t+1}\ar[r]^i \ar@{>->}[u] & A^2_{s-1,t}\ar@{>->}[u] & \\
%	& A^3_{s-1,t+1}\ar[r]^i\ar@{->>}[d] & A^3_{s,t}\ar@{->>}[d] & & A^3_{s-3,t+2}\ar@{>->}[u]\ar[r]^i & A^3_{s-2,t+1}\ar@{>->}[u] &\\
%	& \vdots\ar@{->>}[d] & \vdots\ar@{->>}[d] & & \vdots \ar@{>->}[u] & \vdots\ar@{>->}[u] &\\
%	0\ar[r] & F_{s-1} H_{s+t}(C) \ar[r]^i & F_s H_{s+t}(C) \ar[r] & \gr_s H_{s+t}(C)\ar[r] & 0 & & &
%    }
%\end{equation*}
%Note that if $F_\ast C$ is exhaustive, then this filtration on homology is exhaustive, and, well, what it says is that $F_s H_{s+t}(C) = \colim A^r_{s,t}$.
%Also, if $F_\ast C$ is bounded below ($F_{-1}(C) = 0$), then $A^1_{s,t} = 0$ for $s\leq -1$.
%Eventually, the groups in the vertical injections will be $0$, so that's good.
%So, see, all I need to do now is fill in the missing column beneath $E^1_{s,t}$.
%In particular:
%\begin{equation*}
%    \xymatrix{
%	\cdots\ar[r] & A^1_{s-1,t+1}\ar@{->>}[d] \ar[r]^i & A^1_{s,t} \ar@{->>}[d] \ar[r]^j & E^1_{s,t} \ar[r]^k & A^1_{s-1,t}\ar[r]^i & A^1_{s,t-1} \ar[r] & \cdots\\
%	& A^2_{s-1,t+1}\ar@{->>}[d]\ar[r]^i & A^2_{s,t}\ar@{->>}[d]\ar[r] & E^2_{s,t}\ar[r] & A^2_{s-2,t+1}\ar[r]^i \ar@{>->}[u] & A^2_{s-1,t}\ar@{>->}[u] & \\
%	& A^3_{s-1,t+1}\ar[r]^i\ar@{->>}[d] & A^3_{s,t}\ar@{->>}[d] \ar[r] & E^3_{s,t}\ar[r] & A^3_{s-3,t+2}\ar@{>->}[u]\ar[r]^i & A^3_{s-2,t+1}\ar@{>->}[u] &\\
%	& \vdots\ar@{->>}[d] & \vdots\ar@{->>}[d] & \vdots & \vdots \ar@{>->}[u] & \vdots\ar@{>->}[u] &\\
%	0\ar[r] & F_{s-1} H_{s+t}(C) \ar[r]^i & F_s H_{s+t}(C) \ar[r] & \gr_s H_{s+t}(C)\ar[r] & 0 & 0 & 0 &
%    }
%\end{equation*}
%We want to construct in the $E^r$ groups.
%One way to do this is by exact couples, which is the easiest approach.
%This is what I'll do.
%
%On Wednesday, we'll do examples.
%Today was the guts, and the applications will come next week.


\section*{Exercises}

\begin{ex}
  Which of the following linear transformations $T:\C^2\to\C^2$ is unitary?
  \begin{enumerate}
  \item $T\paren{\begin{mymatrix}{c} x \\ y \end{mymatrix}}
    = \begin{mymatrix}{c} x \\ iy \end{mymatrix}$.
  \item $T\paren{\begin{mymatrix}{c} x \\ y \end{mymatrix}}
    = \begin{mymatrix}{c} iy \\ -ix \end{mymatrix}$.
  \item $\def\arraystretch{1.4}
    T\paren{\begin{mymatrix}{c} x \\ y \end{mymatrix}}
    = \begin{mymatrix}{c}
      x + iy \\
      ix + y \\
    \end{mymatrix}$.
  \item $\def\arraystretch{1.3}
    T\paren{\begin{mymatrix}{c} x \\ y \end{mymatrix}}
    = \begin{mymatrix}{c}
      \frac{1}{\sqrt{2}} x + \frac{i}{\sqrt{2}} y \\
      \frac{1}{\sqrt{2}} x - \frac{i}{\sqrt{2}} y \\
    \end{mymatrix}$.
  \end{enumerate}
  \begin{sol}
    (a) Yes. (b) Yes. (c) No. (d) Yes.
  \end{sol}
\end{ex}

\begin{ex}
  Determine which of the following matrices are unitary.
  \begin{equation*}
    A = \begin{mymatrix}{cc}
      0 & i \\
      i & 0 \\
    \end{mymatrix},
    \quad
    B = \frac{1}{5}
    \begin{mymatrix}{cc}
      4 & 3i \\
      3 & 4i \\
    \end{mymatrix},
    \quad
    C = \frac{1}{\sqrt{5}}
    \begin{mymatrix}{cc}
      1 & 2i \\
      2i & 1 \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    (a) Yes. (b) No. (c) Yes.
  \end{sol}
\end{ex}

\begin{ex}
  Determine which of the following matrices are unitary.
  \begin{equation*}
    A = \frac{1}{2}
    \begin{mymatrix}{ccc}
      1 & i & 0 \\
      0 & 1 & i \\
      i & 0 & 1 \\
    \end{mymatrix},
    \quad
    B = \frac{1}{3}
    \begin{mymatrix}{ccc}
      1   & 2  & 2i \\
      -2i & 2i & 1  \\
      2i  & i  & 2  \\
    \end{mymatrix},
    \quad
    C = \frac{1}{2}
    \begin{mymatrix}{cccc}
      1 &  1 &  1 &  1 \\
      1 &  i & -1 & -i \\
      1 & -1 &  1 & -1 \\
      1 & -i & -1 &  i \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    (a) No. (b) Yes. (c) Yes.
  \end{sol}
\end{ex}

\begin{ex}
  Assume $A$ and $B$ are unitary $3\times 3$-matrices. Then which of
  the following matrices are unitary?
  \begin{equation*}
    (a)~ AB\quad
    (b)~ A+B\quad
    (c)~ iA\quad
    (d)~ -A\quad
    (e)~ 2A\quad
    (f)~ B^{-1}\quad
    (g)~ A^T\quad
    (h)~ A^{\adjoint}\quad
    (i)~ A^2\quad
    (j)~ ABA^{-1}
  \end{equation*}
  \begin{sol}
    All except (b) and (e) are unitary.
  \end{sol}
\end{ex}

\begin{ex}
  Assume $A$ and $B$ are $3\times 3$-matrices, and assume $A$ is
  unitary and $B$ is invertible. Then which of the following matrices
  are unitary?
  \begin{equation*}
    (a)~ AB\quad
    (b)~ A+B\quad
    (f)~ BAB^{-1}\quad
    (g)~ ABA^{-1}\quad
    (h)~ BB^{\adjoint}\quad
    (i)~ BB^{-1}
  \end{equation*}
  \begin{sol}
    Only (f) is unitary.
  \end{sol}
\end{ex}

\begin{ex}
  Which of the following matrices are hermitian? Which ones are
  symmetric?
  \begin{equation*}
    A = \begin{mymatrix}{cc} 0 & 1 \\ 1 & 0 \end{mymatrix}, \quad
    B = \begin{mymatrix}{cc} i & 2 \\ 2 & -i \end{mymatrix}, \quad
    C = \begin{mymatrix}{cc} 0 & i \\ -i & 0 \end{mymatrix}, \quad
    D = \begin{mymatrix}{cc} 1 & 1+i \\ 1-i & 2 \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  Which of the following matrices are hermitian? Which ones are
  symmetric?
  \begin{equation*}
    A = \begin{mymatrix}{ccc}
      0   & -2 & 1+i \\
      -2  &  3 & i   \\
      1-i &  i & 2   \\
    \end{mymatrix}, \quad
    B = \begin{mymatrix}{ccc}
      3     & i    & -1+2i \\
      -i    & 0    & -1+i  \\
      -1-2i & -1-i & 2     \\
    \end{mymatrix}, \quad
    C = \begin{mymatrix}{ccc}
      i & i & i \\
      i & i & i \\
      i & i & i \\
    \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  Assume $A$ and $B$ are hermitian $3\times 3$-matrices, and $A$ is
  invertible. Then which of the following matrices are hermitian?
  \begin{equation*}
    (a)~ AB\quad
    (b)~ A+B\quad
    (c)~ iA\quad
    (d)~ -A\quad
    (e)~ 2A\quad
    (f)~ A^{\adjoint}BA\quad
    (g)~ A^{-1}BA\quad
    (h)~ BB^{\adjoint}\quad
    (i)~ A^2
  \end{equation*}
  \begin{sol}
    All except (a), (c), and (d) are hermitian.
  \end{sol}
\end{ex}

\begin{ex}
  Unitarily diagonalize the hermitian matrix
  $A=\begin{mymatrix}{cc}
    2  & 1+i \\
    1-i & 3 \\
  \end{mymatrix}$.
  \begin{sol}
    Eigenvalues: $\eigenvar_1=1$ and $\eigenvar_2=4$. Normalized
    eigenvectors:
    $\vect{v}_1 = \frac{1}{\sqrt{3}}\begin{mymatrix}{c} 1+i \\ -1 \end{mymatrix}$,
    $\vect{v}_2 = \frac{1}{\sqrt{3}}\begin{mymatrix}{c} 1 \\ 1-i \end{mymatrix}$.
    \begin{equation*}
      D = \begin{mymatrix}{cc} 1 & 0 \\ 0 & 4 \end{mymatrix}, \quad
      P = \frac{1}{\sqrt{3}}\begin{mymatrix}{cc} 1+i & 1 \\ -1 & 1-i \end{mymatrix}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Unitarily diagonalize the hermitian matrix
  $A=\begin{mymatrix}{cc}
    0  & -1+2i \\
    -1-2i & 4 \\
  \end{mymatrix}$.
\end{ex}

\begin{ex}
  Unitarily diagonalize the hermitian matrix
  $A=\begin{mymatrix}{ccc}
    2   & 1+i & 1-i  \\
    1-i & 3   & -2i  \\
    1+i & 2i  &   3  \\
  \end{mymatrix}$.
\end{ex}


%----------------------------------------------------------------------------------------
%   LAGRANGIAN POINTS
%----------------------------------------------------------------------------------------

\section{Computation of the Lagrangian points}
\subsection{Introduction}
Given a set of forces which apply to an object in the plane, our goal is to compute the equilibrium positions of this object. We will restrain ourselves to three kind of interactions: elastic, centrifugal and gravitational forces. We will need the previously written Newton-Raphson method, which will be applied to the resultant force. As a result, we will obtain the roots of this three-dimensional function, corresponding to the equilibrium positions.

\subsection{Forces}
Before explaining and analysing the method, we need to express both the general form of the forces and their Jacobian matrices. Each force will be parameterized with an integer - the intensity - and their origin.

We will start with the elastic force, which represent a spring action on an object. In a one dimensional space, this force is expressed by the following formula:
\[\vec{f}_c = -k (l - l_0)\vec{x},\]
with $l$ the spring length and $l_0$ its free length.

In the plane $(O, \vec{x}, \vec{y})$, this force can be reduced using orthogonals projections to the following form:
\[f_e:\begin{pmatrix}x\\y\end{pmatrix}\rightarrow \begin{pmatrix}\frac{-k(x-x_0)}{\sqrt[]{(x-x_0)^2+(y-y_0)^2}}\\\frac{-k(y-y_0)}{\sqrt[]{(x-x_0)^2+(y-y_0)^2}}\end{pmatrix},\]
where $\begin{pmatrix}x_0\\y_0\end{pmatrix}$ is the origin coordinate and $k$ the intensity.

We can now express the $f_e$ Jacobian matrix:
\[H_{f_e}:\begin{pmatrix}x\\y\end{pmatrix}\rightarrow \begin{pmatrix}
\frac{(y_0-y)^2}{((x_0-x)^2+(y_0-y)^2)^\frac{3}{2}} &
\frac{(x_0-x)(y_0-y)}{((x_0-x)^2+(y_0-y)^2)^\frac{3}{2}}\\
-\frac{(x_0-x)(y_0-y)}{((x_0-x)^2+(y_0-y)^2)^\frac{3}{2}} &
\frac{(x_0-x)^2}{((x_0-x)^2+(y_0-y)^2)^\frac{3}{2}}
\end{pmatrix}\]

Since the force expressions have already been given in the subject, we will only provide the forms of the Jacobian matrices for the centrifugal and gravitational forces:
\[H_{f_c}:\begin{pmatrix}x\\y\end{pmatrix}\rightarrow \begin{pmatrix}
k & 0\\
0 & k
\end{pmatrix}\]

\[H_{f_g}:\begin{pmatrix}x\\y\end{pmatrix}\rightarrow \begin{pmatrix}
\frac{k(2x_0^2 - y_0^2 - 4x_0x + 2x^2 + 2y_0y - y^2)}{((x_0 - x)^2 + (y_0 - y)^2)^\frac{5}{2}} &
\frac{3k(x_0 - x)(y_0 - y)}{((x_0 - x)^2 + (y_0 - y)^2)^\frac{5}{2}}\\
\frac{3k(x_0 - x)(y_0 - y)}{((x_0 - x)^2 + (y_0 - y)^2)^\frac{5}{2}} &
-\frac{c (x_0^2 - 2 y_0^2 - 2 x_0 x + x^2 + 4 y_0 y - 2 y^2)}{((x_0 - x)^2 + (y_0 - y)^2)^\frac{5}{2}}
\end{pmatrix}\]

\paragraph{Implementation} The implementation use a functionnal programming approach, which means that we use a kind of functions which will build and return a function as expression. This allow us to be more versatile by storing and using the function itself rather than using a general function, and to give it as a parameter to the Newton-Raphson method for example.

\paragraph{Tests} The tests check in particular cases the validity of the result for one force. In particular, we check for the gravitational force that the norm is invariant by rotation around the origin. Moreover, these tests rose the issue of dividing by a norm equal to zero. We handle this case by setting the value to the asymptotic value of the function in this point (either infinite or a constant). % Bring examples...

\subsection{Results}
\subsubsection{First approach}
We consider the following case:
\begin{itemize}
  \item Two gravitational forces with coefficients $1$ (resp. $0.01$) and originating from $\begin{pmatrix}0\\0\end{pmatrix}$ (resp. $\begin{pmatrix}1\\0\end{pmatrix}$).
  \item A centrifugal force with coefficient $1$ at the barycenter of the two masses, i.e., at $\begin{pmatrix}\frac{0.01}{1.01}\\0\end{pmatrix}$.
\end{itemize}

We can easily plot the situation using the norm of the resultant force. So we get the Figure~\vref{fig:resultant_force_norm}. From above, the graph is more explicit, as shown in the Figure~\vref{fig:equilibrium}. The situation is similar to the Earth rotation around the Sun. Centrifugal force and gravitational attraction by the Sun are applied to the Earth. The centrifugal force applied to the barycenter of the two bodies is characteristic of a two-body interaction.

\begin{figure}[ht]
\centering
\subfloat[3D shape]{\includegraphics[width=.6\columnwidth]{resultant_force_norm}\label{fig:resultant_force_norm}}
\subfloat[Equilibrium]{\includegraphics[width=.45\columnwidth]{equilibrium}\label{fig:equilibrium}}
\caption[Resultant force norm]{Resultant force norm}
\end{figure}

\subsubsection{Lagrangian points}
The main issue is that the Newton-Raphson method can only compute one root while we seem to have an infinite number of roots. We can call the algorithm by specifying a step over the whole grid. We will see that in this kind of interaction, the equilibrium points are particular and well-known as the Lagrangian points.

It is not obvious with the last example that only five points correspond to an equilibrium situation. However, we can modify a bit the datas to make this phenomenon more clear and spread the values with the logarithm, as in Figure~\vref{fig:lagrange_points}.

\begin{figure}[ht]
  \centering
  \includegraphics[width=0.8\columnwidth]{lagrange_points}
  \caption[Lagrange points]{Lagrangian points.}
  \label{fig:lagrange_points}
\end{figure}

As the Newton-Raphson algorithm follows the slope of the curve, we can call it on equireparted points on the grid, distant by a fixed step we will note $\tau$. The algorithm must work in a closed domain. Moreover, at each iteration, the algorithm will output a position corresponding to a root. Assuming that we will perform enough iterations to find at least all the roots, we must be able to determine a maximal precision on a root coordinate. The minimal distance between two distinct roots, noted $\varepsilon$, will be another parameter of this algorithm.