\documentclass[main.tex]{subfiles}
\begin{document}

\marginpar{Monday\\ 2020-3-30, \\ compiled \\ \today}

% Last time we discussed: how can we distinguish what is a GW and what is not?

% We do this by separating them by frequency.
% We cannot really measure the nature of tensor perturbation of GWs, since we are only measuring integrated effects.

% We can precisely map the effect of the GW in time, by sampling with a frequency which is much higher than the one of the GW.

So, let us do this formally: we consider \(g = \overline{g} + h\) where \(\overline{g}\) only contains low frequency components while \(h\) contains high frequency ones, we expand the Ricci tensor in powers of \(h_{\mu \nu }\) up to second order and rewrite the Einstein equations as
%
\begin{align}
\frac{8 \pi G}{c^{4}} \qty(T_{\mu \nu } - \frac{1}{2} g_{\mu \nu }T)
= \overline{R}_{\mu \nu }
+ R^{(1)}_{\mu \nu }
+ R^{(2)}_{\mu \nu }
\,,
\end{align}
%
where the second order in \(h_{\mu \nu }\) term, \(R^{(2)}_{\mu \nu }\), has both high and low frequency components, while the term \(\overline{R}_{\mu \nu }\) is exclusively low frequency and \(R_{\mu \nu }^{(1)}\) is exclusively high frequency.

Let us give a heuristic argument for this statement: the gravitational waves contained in \(h_{\mu \nu } \) will have several high frequencies; let us call two of them \(\omega_1  \) and \(\omega_2 \), so as we compute a term which is quadratic in \(h\) we will get
%
\begin{align}
\qty(\sin(\omega_{1} t) + \sin(\omega_2 t))^2
= \\ =
\frac{1}{2} \left(-\cos \left(2 t \omega _1\right)-\cos \left(2 t \omega _2\right)-2 \cos \left(t \omega _1+t \omega _2\right)+2 \cos \left(t \omega _1-t \omega _2\right)+2\right)
\,,
\end{align}
%
where we used the prostapheresis formulas: we get terms oscillating with \(\omega_1 + \omega_2 \) as well as \(\omega_1 - \omega_2 \), one of which is high frequency while the other is low frequency, since we are considering a high-frequency wavepacket.

So, what we want to do is to separate the field equation into its low and high frequency parts:
%
\begin{align} \label{eq:low-frequency-ricci-tensor-contribution}
\overline{R}_{\mu \nu } &= - \qty(R_{\mu \nu }^{(2)})^{\text{low}} + \frac{8 \pi G}{c^{4}} \qty(T_{\mu \nu } - \frac{1}{2} g_{\mu \nu } T)^{\text{low}} \\
R_{\mu \nu }^{(1)} &= - \qty(R_{\mu \nu }^{(2)})^{\text{high}} + \frac{8 \pi G}{c^{4}} \qty(T_{\mu \nu } - \frac{1}{2} g_{\mu \nu } T)^{\text{high}}
\,.
\end{align}

How might we define these formally?
Recall, we expand in two parameters: \(h\) and \(\lambda / L_B \sim  f_B / f\). Then, in terms of orders of magnitude we have\footnote{The fact that the derivatives correspond to divisions by lengths is justified since we are looking at harmonic expansions, the metric will be a sum of sinusoids around a characteristic frequency.}
%
\begin{align}
\overline{R}_{\mu \nu } \sim \partial^2 \overline{g}_{\mu \nu } \sim \frac{1}{L_B^2}
\,,
\end{align}
%
while
%
\begin{align}
\qty[R^{(2)}_{\mu \nu }]^{\text{low}} \sim \qty(\partial h)^2
 \sim \qty(\frac{h}{\lambda })^2
\,.
\end{align}

See Maggiore \cite[pag.\ 31]{maggioreGravitationalWavesVolume2007} for more details.
So, inserting these order-of-magnitude estimates inside equation \eqref{eq:low-frequency-ricci-tensor-contribution} we get
%
\begin{align}
\frac{1}{L_B^2} \sim \frac{h^2}{\lambda^2} + \qty(T_{\mu \nu } \text{ contribution})
\,.
\end{align}

If we are in a vacuum (\(T_{\mu \nu }=0\)) then
%
\begin{align}
\frac{1}{L_B^2} \sim \qty(\frac{h}{\lambda })^2
\implies h \sim \frac{\lambda }{L_B}
\,,
\end{align}
%
while if the stress-energy tensor is nonvanishing we get
%
\begin{align}
\frac{1}{L_B^2} \sim \qty(\frac{h}{\lambda })^2 + (T_{\mu \nu } \text{ contribution}) \gg \qty(\frac{h}{\lambda })^2
\,,
\end{align}
%
which means \(h \ll \lambda / L_B\). This shows that \textbf{the linearized expansion cannot be extended beyond linear order} starting from a flat background: if we start from a flat metric then \(1/L_B = 0\), so any \(h > 0\) violates the condition \(h \lesssim \lambda / L_B\).
We can also see that the notion of a gravitational wave only makes sense as long as \(h\) is small: if \(h\) were close to 1, then we would have \(\lambda \sim L_B\), but our only way to distinguish GW from background is through their wavelength!

\todo[inline]{Unclear what the problem is: we cannot use the flat metric, but the real world's metric is not flat anyways! Why can we not go to second order with a curved background?}

The solution to this problem is to take averages on a scale \(\ell\) such that \(\lambda_{GW} \ll \ell \ll L_B\), so that the gravitational wave is fully averaged out, while the background metric is approximately constant across the integration volume.
This low-frequency projection reads:
%
\begin{align}
\overline{R}_{\mu \nu } =
- \expval{R^{(2)}_{\mu \nu }} + \frac{8 \pi G}{c^{4}} \expval{T_{\mu \nu } - \frac{1}{2} g_{\mu \nu } T}
\,.
\end{align}

After some math (see Maggiore \cite[eqs.\ 1.122-1.123]{maggioreGravitationalWavesVolume2007}) we can define a stress tensor of the GW, which looks like
%
\begin{align}
t_{\mu \nu } = - \frac{c^{4}}{8 \pi G} \expval{R^{(2)}_{\mu \nu } - \frac{1}{2} \overline{g}_{\mu \nu } R^{(2)}}
\,,
\end{align}
%
where the Ricci scalar is computed using the smoothed metric: \(R^{(2)} = \overline{g}^{\mu \nu } R_{\mu \nu }^{(2)}\).
Also, we define a  ``smoothed out'' stress-energy tensor of matter:
%
\begin{align}
\expval{T_{\mu \nu } - \frac{1}{2} g_{\mu \nu } T}
\overset{\text{def}}{=}
\overline{T}_{\mu \nu } - \frac{1}{2} \overline{g}_{\mu \nu } \overline{T}
\,,
\end{align}
%
so the equation which will hold is the equivalence of the smoothed Einstein tensor with the sum of the smoothed and GW stress-energy tensors:
%
\boxalign{
\begin{align}
\overline{R}_{\mu \nu } - \frac{1}{2} \overline{g}_{\mu \nu } \overline{R} =
\frac{8 \pi G}{c^{4}} \qty(\overline{T}_{\mu \nu } + t_{\mu \nu })
\,.
\end{align}}

Do note that if we work with these, the stress-energy tensor which is conserved is \(\overline{T}_{\mu \nu } + t_{\mu \nu }\): if we take the divergence of the equation we get, by the Bianchi identities,
%
\begin{align}
\nabla^{\mu } \qty(\overline{R}_{\mu \nu } - \frac{1}{2} \overline{g}_{\mu \nu }\overline{R}) = 0 = \nabla^{\mu } \qty(\overline{T}_{\mu \nu } + t_{\mu \nu })
\,.
\end{align}

This means that gravitational waves and matter can exchange energy and momentum. If the stress-energy tensor is slowly-varying enough, as often happens practically, we can just use \(\overline{T}^{\mu \nu } \approx T^{\mu \nu }\).

How does this look like far from the source?
There, we can approximate \(\overline{g} \approx \eta_{\mu \nu }\) and \(\nabla_{\mu } \approx \partial_{\mu }\).
This \(t_{\mu \nu }\) only has 2 physical degrees of freedom (those of the gravitational wave): we need to gauge the others away.

The Lorentz gauge plus \(h=0\) eliminates \(5\) degrees of freedom.

When we have terms like \(h \partial \partial h\), we can integrate by parts on a sufficiently large volume to turn them into \(\partial(h \partial ) - \partial h \partial h\).
Using the facts \(\partial^{\mu }h_{\mu \nu } = h = \square h_{\mu \nu } = 0\) we can also simplify several terms: in the end we get
%
\boxalign{
\begin{align}
t_{\mu \nu } = \frac{c^{4}}{32 \pi G} \expval{\partial_{\mu } h_{\alpha \beta } \partial_{\nu } h^{\alpha \beta }}
\,.
\end{align}}

This is invariant under the residual gauge transformations, and coordinate independent: we can compute it in any frame we like.

\subsubsection{Explicit TT-gauge expression}

Let us compute this in the simplest case: a gravitational wave travelling along the \(z\) axis, described in the TT-gauge.
The 00 component will read:
%
\begin{align}
t^{00 } = \frac{c^{4}}{32 \pi G} \expval{\partial^{0} h_{\alpha \beta } \partial^{0} h^{\alpha \beta }} = \frac{c^2}{16 \pi G} \expval{\dot{h}^2_{+} + \dot{h}^2_{ \times }}
\,,
\end{align}
%
since:
%
\begin{align}
\partial^{0} h_{\alpha \beta } \partial^{0} h^{\alpha \beta } =
\frac{\dot{h}^{TT}_{ij}}{c} \frac{\dot{h}^{TT}_{ij}}{c}
= \frac{1}{c^2}\sum _{i, j =1}^{2} \qty(\dot{h}_{ij}^{TT})^2
= \frac{2}{c^2} \qty(\dot{h}_{+}^2 + \dot{h}_{ \times }^2)
\,.
\end{align}

As for the other components, we will have \(t^{01} = t^{02} =0 \) by symmetry (they would represent momentum transfer in a direction orthogonal to the propagation, also formally \(\partial_{1,2} h_{ij}^{TT} = 0\) since it only depends on \(t\) and \(z\)), and also \(t^{03}= t^{00}\), since the perturbation is a function of \((t - z/c)\): so, we have
%
\begin{align}
\partial_{3} h_{ij}^{TT} = - \partial_{0} h_{ij}^{TT} = + \partial^{0} h_{ij}^{TT}
\,.
\end{align}

\subsection{Energy and momentum flux far from the source}

If we are far enough away from the source, we can compute the energy crossing a surface \(\dd{A}\) in a time \(\dd{t}\) as the spacetime density contained in a volume \(\dd{A} c \dd{t}\): if we are considering a specific direction (say, the flux coming the way of the Earth) then we can say that the wave's propagation is aligned with the \(z\) axis and so we can write
%
\begin{align}
\dd{E} &= \dd{A} c \dd{t} \frac{c^2}{32 \pi G} \expval{\dot{h}^{TT}_{ij}\dot{h}^{TT}_{ij}} \\
\frac{ \dd{E}}{ \dd{t} \dd{A}} &= \frac{c^3}{16 \pi G} \expval{\dot{h}^2_{+} + \dot{h}^2_{ \times }}
\,.
\end{align}

In order to get the total power \(\dv*{E}{t}\) which is emitted by the source we can integrate this expression in \(R^2 \dd{\Omega }\); however we will need to use the general expression \(\expval{\dot{h}^{TT}_{ij}\dot{h}^{TT}_{ij}}\) since we cannot have alignment with the \(z\) axis across the whole sphere.

To get the momentum density the reasoning is similar: we start from \(\dd{P^{k}} = \dd{A} c \dd{t} t^{0k}/c\) to get
%
\begin{align}
\frac{ \dd{P^{k}}}{ \dd{A} \dd{t}} = \frac{c^{4}}{32 \pi G} \expval{ \partial^{0} h_{\alpha \beta } \partial^{ k} h^{\alpha \beta }}
= - \frac{c^3}{32 \pi G} \expval{\dot{h}^{TT}_{ij} \partial^{k} h_{ij}^{TT}}
\,,
\end{align}
%
where we used the fact that \(\partial^{0} = -\partial_0  =-  \pdv*{}{(ct)}\).

So, the general expressions for the energy and momentum density emitted at a distance will be
%
\begin{align}
\dv{E}{t} = \frac{c^3}{32 \pi G} r^2 \int
\expval{\dot{h}^{TT}_{ij}\dot{h}^{TT}_{ij}} \dd{\Omega }
\qquad \text{and} \qquad
\dv{P^{k}}{t} = - \frac{c^3}{32 \pi G} r^2
\int \expval{\dot{h}^{TT}_{ij} \partial^{k} h_{ij}^{TT}} \dd{\Omega }
\,.
\end{align}

We want to express this explicitly in terms of the quadrupole moment. We start from the expression of the TT-gauge amplitude in terms of the quadrupole \eqref{eq:traceless-transverse-amplitude-from-quadrupole}: its derivative reads
%
\begin{align}
\dot{h}_{ij}^{TT} (t, \vec{x}) = \frac{1}{r} \frac{2G}{c^{4}} \Lambda_{ij, kl} \dot{\ddot{Q}}^{kl} (t - r/c)
\,.
\end{align}

Inserting this into the expression we get
%
\begin{subequations}
\begin{align}
\dv{E}{t} &= \frac{r^2c^{3}}{32 \pi G} \int \dd{\Omega }
\expval{
\frac{1}{r} \frac{2G}{c^{4}} \Lambda_{ij,kl} \dot{\ddot{Q}}^{kl}
\frac{1}{r} \frac{2G}{c^{4}} \Lambda_{ij,mn} \dot{\ddot{Q}}^{mn}
}  \\
&=\frac{G}{8 \pi c^{5}} \int \dd{\Omega }
\Lambda_{ij, kl} \expval{\dot{\ddot{Q}}^{ij} \dot{\ddot{Q}}^{kl}}
\marginnote{\(\Lambda \) is idempotent and symmetric under swaps of index pairs: so \(\Lambda_{ij, kl} \Lambda_{ij, mn} = \Lambda_{kl, mn}\).}
\,,
\end{align}
\end{subequations}
%
where the only expression depending on the angle is \(\Lambda_{ij, kl} \), since the quadrupole only depends on the source, not on an observer's position.

\begin{claim}
The explicit expression for the projection tensor \(\Lambda_{ij,kl}(\hat{n})\) in terms of the unit vector \(\hat{n}\) is:
%
\begin{align}
\Lambda_{ij, kl} (\hat{n}) = \delta_{ik} \delta_{jl} - \frac{1}{2} \delta_{ij} \delta_{kl} - n_j n_l \delta_{ik} - n_i n_k \delta_{jl}
+ \frac{1}{2} n_k n_l \delta_{ij}
+ \frac{1}{2} n_i n_j \delta_{kl}
+ \frac{1}{2} n_i n_j n_k n_l
\,,
\end{align}
%
and its integral in \(\dd{\Omega }\) is given by
%
\begin{align}
\int \dd{\Omega } \Lambda_{ij, kl} = \frac{4 \pi }{30}
\qty(11 \delta_{ik} \delta_{jl} - 4 \delta_{ij} \delta_{kl} + \delta_{il} \delta_{jk} )
\,.
\end{align}
\end{claim}

With this result, we can write
%
\begin{align}
\dv{E}{t} &= \frac{G}{8 \pi c^{5}} \frac{2\pi}{15} \qty(11 \delta_{ik} \delta_{jl} - 4 \delta_{ij} \delta_{kl} + \delta_{il} \delta_{jk} )
\expval{\dot{\ddot{Q}}^{ij} \dot{\ddot{Q}}^{kl}}
\,,
\end{align}
%
but since the quadrupole moment derivatives \(\dot{\ddot{Q}}^{ij}\) are both traceless and symmetric the first and third delta combinations are equal (so the factor multiplying them will be 12), while the second combination will vanish. So, we will get
%
\boxalign{
\begin{align}
\label{eq:radiated-power-GW}
\dv{E}{t} = \frac{G}{8 \pi c^{5}} \frac{2 \pi }{15} 12
\expval{\dot{\ddot{Q}}^{ij} \dot{\ddot{Q}}^{ij}}
= \frac{G }{5 c^{5}}
\expval{\dot{\ddot{Q}}^{ij} \dot{\ddot{Q}}^{ij}}
= \frac{G}{5 c^{5}}
\expval{\dot{\ddot{M}}_{ij} \dot{\ddot{M}}_{ij} - \frac{1}{3} \dot{\ddot{M}}_{kk}^{2} }
\,.
\end{align}}
%

When we do the same from the \textbf{momentum loss}, we get an integral in the form
%
\begin{align}
\dv{P^{k}}{t} \propto \int \dd{\Omega } \dot{\ddot{Q}}^{TT}_{ij} \partial^{k} \ddot{Q}^{TT}_{ij}
\,,
\end{align}
%
which is odd under spatial inversion because of the spatial derivative, so there is no contribution!

This is not true if we go beyond the quadrupole approximation: full GR calculations/simulations show that there can be kicks at the merger.

In linearized gravity, we can calculate the power loss by energy conservation:
%
\begin{align}
\dv{E _{\text{source}}}{t} = - \dv{E _{\text{far-field}}}{t} = - \frac{G}{5c^{5}} \expval{\dot{\ddot{Q}}^{ij} \dot{\ddot{Q}}^{ij}}
\,,
\end{align}
%
although this is not exact moment-by-moment in full GR, since there some of the emission can be delayed.
In our approximation, the two sides of the equation should be calculated at the same retarded time: the power loss by the source at \(t - r/c\) will be the power detected a distance \(r\) away at a time \(t\).

\subsection{Angular momentum loss from GW}

\subsubsection{Effective back-action force}

% We can calculate the angular distribution in a relatively simple way, since it is easy to go to TT gauge at a point.

We can model the back-action from GW as a force applied on the source \cite[sec.\ 3.3.4]{maggioreGravitationalWavesVolume2007}, whose power is the average of \(\vec{F} \cdot \vec{v}\) as in classical mechanics; since we consider a continuous medium we have:
%
\begin{align}
- \frac{G}{5 c^{5}} \expval{\dv[3]{Q_{ij}}{t} \dv[3]{Q_{ij}}{t}}
= \dv{E _{\text{source}}}{t} =
\expval{\int  \dd[3]{x} \dv{F_i}{V} \dot{x}_{i}}
\,.
\end{align}

We can integrate by parts twice to write the two third derivatives as
%
\begin{align}
\expval{\dv[3]{Q_{ij}}{t} \dv[3]{Q_{ij}}{t}}
= \expval{\dv{Q_{ij}}{t} \dv[5]{Q_{ij}}{t}}
\,.
\end{align}

Also, the 0th component of the conservation of the stress-energy tensor \(\partial_{\mu } T^{0 \mu }\) for a classical source reads \(\partial_{t} \rho + \partial_{i} \qty(\rho v_i) = 0 \). Using this fact, we can rewrite the first derivative of \(Q_{ij}\) as
%
\begin{align}
\dv{Q_{ij}}{t} &= \dv{}{t} \int  \dd[3]{x}
\rho \bigg( x_{i} x_{j} - \underbrace{\frac{1}{3} r^2 \delta_{ij}}_{\mathclap{\text{contracted with traceless \(\dot{\ddot{\ddot{Q}}}_{ij}\)}}}\bigg)
= - \int \dd[3]{x} \partial_{k} \qty(\rho v_k)
 x_{i} x_{j}  \\
&= + \int \dd[3]{x} \rho v_k \partial_{k} \qty(x_i x_j) =
2 \int \dd[3]{x} \rho v_{(i } x_{j)}
\,.
\end{align}
%
\todo[inline]{Missing symmetrization in the slides! Although it is implicit when contracting with \(Q\)\dots}

Then our equation reads:
%
\begin{align}
\expval{\int \dd[3]{x} \dv{F_i}{V} \dot{x}_{i}} = - \frac{2G}{5c^{5}}\expval{\int \dd[3]{x} \dv[5]{Q_{ij}}{t} \rho \dot{x}_{i} x_{j}}
\,,
\end{align}
%
so we can identify the terms to get
%
\begin{align}
\dv{F_{i}}{V} = - \frac{2G}{5c^{5}} \dv[5]{Q_{ij}}{t} \rho (t, \vec{x}) x_{j}
\,,
\end{align}
%
so finally, since the only position-dependent terms on the right-hand side are the density and the position we get that the effective force is:
%
\boxalign{
\begin{align}
F_{i} = - \frac{2G}{5c^{5}} \dv[5]{Q_{ij}}{t} m \overline{x}_{j}
\,,
\end{align}}
%
where \(\overline{x}_{j} \) is the center-of-mass coordinate.

\subsubsection{Angular momentum}

With this effective force we can calculate the torque explicitly: in general it is given by \(T_i = \epsilon_{ijk} x_j F_k\), which we can make into a local relation by substituting \(T_i\) and \(F_k\) with their densities. With this we can write
%
\begin{align}
\dv{T_i}{V} = \epsilon_{ijk} x_j  \dv{F_k}{V} = -
\epsilon_{ijk} x_j  \frac{2G}{5 c^{5}} \dv[5]{Q_{kl}}{t} \rho (t, \vec{x}) x_l
\,,
\end{align}
%
so the total torque is given by
%
\begin{align}
T_{i} &= - \frac{2G}{5 c^{5}} \epsilon_{ijk} \dv[5]{Q_{kl}}{t}
\int \dd[3]{x} \rho (t, \vec{x}) x_l x_j \underbrace{- \frac{1}{3} r^2 \delta_{lj}}_{\mathclap{\text{does not contribute, since \(Q\) is traceless}}} \\
&= - \frac{2G}{5 c^{5}} \epsilon_{ijk} \dv[5]{Q_{kl}}{t} Q_{lj}
\,,
\end{align}
%
so if we take the average, substituting the derivative of the angular momentum for the torque and integrating by parts twice to get a second and a third derivative, we find:
%
\begin{align}
\expval{\dv{L_{i}}{t}} = -\frac{2G}{5c^{5}} \epsilon_{ijk}
\expval{\ddot{Q}_{jl} \dot{\ddot{Q}}_{kl}}
\,.
\end{align}

\section{Back-reaction and the evolution of binary systems}

\subsection{Compact circular inspiral}

We use the reduced mass formalism for a two-body problem: we define the total mass \(M = m_1 +m_2 \), the reduced mass \(\mu = m_1 m_2 / M\), the relative coordinate \(\vec{x} = \vec{x}_1 - \vec{x}_2\) whose modulus is \(R = \abs{\vec{x}}\).
The angular velocity of the circular motion is \(\omega_{s}\), satisfying Kepler's law \(\omega_{s}^2 R^3 = GM\).

As we saw, the amplitude of the emitted gravitational waves is given by
%
\begin{align}
A = \frac{4 G^{5/3} \omega_{s}^{2/3} \mu M^{2/3}}{r c^{4}}
\,,
\end{align}
%
and the amplitudes of the two polarizations in time are given in terms of \(A\) by
%
\begin{subequations}
\begin{align}
h_{+} &= A \frac{1 - \cos^2\theta   }{2} \cos(2 \omega_{s} t _{\text{ret}} + 2 \varphi ) \\
h_{\times } &= A \cos \theta  \sin(2 \omega_{s} t _{\text{ret}} + 2 \varphi )
\,.
\end{align}
\end{subequations}

In order to make these expressions easier to interpret, we define the chirp mass:
%
\begin{align}
M_c = \mu ^{3/5} M^{2/5} = \frac{\qty(m_1 m_2 )^{3/5}}{(m_1 + m_2 )^{1/5}}
\,.
\end{align}

Also, since the frequency of the emitted GW is double that of the binary we define \(f_{GW} = 2 f_s = 2 \omega_{s} / (2 \pi )\), and \(\omega_{GW} = 2 \pi f_{GW}\).
The reduced wavelength corresponding to this frequency is \(\lambdabar = c / \omega_{GW}\); also we can define a Schwarzschild radius corresponding to the chirp mass: \(R_C = 2GM_c / c^2\).

With these definitions we can rewrite \(A\) as:
%
\begin{align}
A = \frac{4}{r} \qty( \frac{G M_C}{c^2})^{5/3} \qty( \frac{\pi f_{GW}}{c})^{2/3}
= \frac{1}{\sqrt[3]{2}} \frac{R_C}{r} \qty( \frac{R_C}{\lambdabar})^{2/3}
\,,
\end{align}

As we have shown, in the quadrupole approximation the radiated power is:
%
\begin{align}
\dv{E}{t} = \frac{G}{5 c^{5}} \expval{\dot{\ddot{M}}_{ij} \dot{\ddot{M}}_{ij} - \frac{1}{3} \qty(\dot{\ddot{M}}_{kk})^2}
\,,
\end{align}
%
which we can calculate explicitly for our binary.
We have already derived expressions for the second time derivatives of the second mass moment, \(\ddot{M}_{ij}\) for motion in the \(xy\) plane \eqref{eq:second-mass-moment-xy-components}; taking their derivative we get
%
\begin{align}
\dot{\ddot{M}}_{11} = - \dot{\ddot{M}}_{22} =
- 4 \mu R^2 \omega_{s}^3 \sin(2 \omega_{s}t )
\qquad \text{and} \qquad
\dot{\ddot{M}}_{12} = 4 \mu R^2 \omega_{s}^3 \cos(\omega_{s}t)
\,,
\end{align}
%
so the term \(\qty(\dot{\ddot{M}}_{ij})^2\) vanishes (\(\dot{\ddot{M}}_{ij}\) is traceless) and \(\dot{\ddot{M}}_{11}^2 = \dot{\ddot{M}}_{22}^2\), so the contribution we get is
%
\begin{align}
\dv{E}{t} &= \frac{G}{5 c^{5}} \expval{2 \dot{\ddot{M}}_{11}^2 + 2 \dot{\ddot{M}}_{12}^2 }  \\
&= \frac{G}{5 c^{5}} 2 \qty(4 \mu R^2 \omega_{s}^3)^2 \underbrace{\qty(\expval{\sin^2(2 \omega_{s}t)} + \expval{\cos^2(2\omega_{s}t)})}_{= 1/2 + 1/2 = 1}
\,,
\end{align}
%
since the average of the squared sine or cosine is \(1/2\).
Now, we can simplify this expression by using \(R^3 = GM / \omega_{s}^2\) and substituting in the GW angular velocity:
%
\begin{align}
\dv{E_{GW}}{t} &= \frac{32G}{5c^{5}} \mu^2 R^{4} \omega_{s}^{6}  \\
&= \frac{32G}{5c^{5}} \mu^2 \qty( \frac{GM}{\omega_{s}})^{4/3} \omega_{s}^{6}  \\
&= \frac{32}{5c^{5}} \mu^2 G^{7/3} M^{4/3} \omega_{s}^{10/3} \\
&= \frac{32}{5} \frac{c^{5}}{G} \qty( \frac{G M_c \omega_{GW}}{2 c^{3}})^{10/3} \label{eq:gravitational-wave-total-power-emitted}
\,.
\end{align}

We can apply a similar reasoning for the angular momentum loss (here \(L = \abs{\vec{L}}\)):
%
\begin{align}
\dv{L}{t} =
\frac{32}{5} \frac{c^{5}}{G} \qty( \frac{G M_c \omega_{GW}}{2 c^{3}})^{10/3} \frac{2}{\omega_{GW}}
= \frac{32}{5} \frac{c^{5}}{G} \qty(\frac{G M_c}{c^{3}})^{10/3} \qty(\frac{\omega_{GW}}{2})^{7/3}
\,.
\end{align}

If we have masses in circular orbit, then by the virial theorem their total energy is given by half of the potential energy:
%
\begin{align}
E = - \frac{1}{2} \frac{G m_1 m_2  }{R}
\,,
\end{align}
%
which we can differentiate with respect to time to get \(\dot{E} = G m_1 m_2 / (2 R^2) \dot{R}\); this can be also written as
%
\begin{align} \label{eq:radius-evolution-binary}
\dot{R} = - \frac{2R^2}{G m_1 m_2 } \dot{E}_{GW}
\,,
\end{align}
%
as long as the orbital energy is only lost through gravitational wave emission.

This means that, as the GW carry away energy, the radius shrinks; this corresponds to an increase in frequency, and a corresponding increase in gravitational wave emission.

In all our calculations we assumed the orbits to be circular!
This is fine, as long as they are almost-circular: the condition to require is that the variation of the radius across a single orbit is very small:
%
\begin{align}
\abs{\frac{\dot{R} T}{R}} \ll 1
\,,
\end{align}
%
where \(T\) is the period.
In order to express this conditions in terms of the orbital angular velocity \(\omega_{s}\) we use Kepler's law: if we differentiate \(R = \qty(GM / \omega_{s}^2)^{1/3}\) we get
%
\begin{align} \label{eq:derivative-radius-evolution-binary}
\dot{R} = - \frac{2}{3} \omega_{s}^{-5/3} \qty(GM)^{1/3} \dot{\omega}_{s}
= - \frac{2}{3} \frac{R}{\omega_{s}} \dot{\omega}_{s}
\,.
\end{align}

This equality can also be written in terms of logarithmic derivatives as
%
\begin{align} \label{eq:log-derivatives-radius-angular-velocity-binary}
\frac{\dot{R}}{R} + \frac{2}{3} \frac{\dot{\omega}_{s}}{\omega_{s}} = 0
\,.
\end{align}

Starting from this, and using \(T = 2 \pi / \omega_{s}\) we can write
%
\begin{align}
\abs{\frac{\dot{R}T}{R}} = \frac{2}{3} \frac{R}{\omega_{s}} \dot{\omega}_{s} \frac{2\pi}{\omega_{s}} \frac{1}{R} = \frac{4 \pi }{3} \frac{\dot{\omega}_{s}}{\omega_{s}^2}
\,,
\end{align}
%
which means that we need to require \(\dot{\omega}_{s} \ll \omega_{s}^2\).

In practice, this condition is quite well satisfied for most of the inspiral, right up until the merger phase.
Motion satisfying this condition is called \textbf{quasi-circular}.

\end{document}


\chapter{Exploring The Hopfield Model}
% This is where we describe our experiments. We insert plots, images, ....

We started out the analysis by experimenting with converge of patterns. We shall present two experiments, that provide some interesting insight for the reader into the Hopfield Model.

\section{Facial Recognition for Police criminal records}

This experiment describes how the Hopfield network might be used by a Police Office, in order to identify criminals. Since the police has a big database that contains facial images of criminals, the network can perform identify, given an image, which criminal is depicted.

Before being sent to the Hopfield model, the image suffers a few normalising transformations:
\begin{enumerate}
 \item the background is completely removed
 \item the image is scaled to 25x25 pixels.
 \item conversion first to grayscale, and then to only black or white pixels takes place
\end{enumerate}

In figure \ref{fig:criminal}, we present the convergence series for the input image of a criminal. The network used for this experiment had 400 neurons, running on images of 20x20 pixels. It clearly illustrates the how efficient the networks can be in these scenarios. However, the network might also convert to to spurious patterns, which might create problems in certain situations. See section \ref{spurious_patterns} for a more detailed description.

\begin{figure}[h]
  \centering
\includegraphics[scale=2]{images/convergence/all.png}
\caption{Convergence sequence for a criminal in the police database}
\label{fig:criminal}
\end{figure}

Furthermore, we have developed a GUI, powered by the underlying recognition core, to provide a friendly user experience, in addition to providing interactive facilities (figure \ref{fig:gui}). An overview of our the recognition system can be found in figure ~\ref{fig:system}.

\begin{figure}[h]
  \centering
\includegraphics[scale=0.3]{screenshots-small/gui1.jpg}
\caption{GUI for a criminal record recognition system.}
\label{fig:gui}
\end{figure}

\begin{figure}[h]
  \centering
\includegraphics[scale=0.3]{recognition.jpg}
\caption{Outline of the Recognition System demonstrating how the various system components interact}
\label{fig:system}
\end{figure}

\section{Further analysis}

Our initial aim was to explore various properties of the attractors in a Hopfield Network. We are interested in studying, testing or challenging several ideas about the neural network:
\begin{itemize}
 \item Clusters of attractors: We are interested to see how the Hopfield network behaves when learning similarly correlated patterns. This implies training the network with a cluster of attractors, which will lower the capacity of the network. We believe that basin sizes will get smaller as the patterns get closer to each other.
 \item Super-Attractors: Find out what happens if the network is trained several times with the same pattern. We believe this super-attractor will have a larger basin size compared to the other attractors.
 \item Convergence: We plan to train the network with different types of attractors and find out to which ones patterns tend to converge.
\end{itemize}

\section{Basins of Attraction}
%I think we should move the measurements for the basins of attraction in here.
A basin of attraction for a particular attractor \(\alpha\) is defined as the set of all states that will eventually converge to \(\alpha\), under repeated update. Here, we are particularly interested in finding a way to measure the size of such a basin of attraction. A large basin size would provide stability for the attractor, since the states in the neighbourhood would converge to it. A small basin size for an attractor would mean that the network might never recall the pattern corresponding to that attractor.

\subsection{Measuring the Basin Size using the Storkey-Valabregue technique}
\label{storkey_basin_size}

 A large part of our experiments were thus dedicated to measuring basin sizes of different attractors, clusters of attractors and a newly introduced concept of \emph{Super-Attractors}, in section ~\ref{super_attractors}.

 A recognised scientific method for calculating the basin size is the Storkey-Valabregue measurement. This can be computed as follows:

 \begin{enumerate}
  \item Initially n = 1
  \item Choose an initial fixed point corresponding to a stored pattern \(\mu\)
  \item \label{itm:choose radius} Choose some initial normalised Hamming radius \(r=r_{0}\)
  \item Let the set A be all the states Hamming-distant \( nr \) from the fixed point
  \item Sample 100 states from A
  \item Calculate how many of these states are attracted to the fixed point. Denote this number \( t_{\mu}(r) \)
  \item If \( t_{\mu}(r) \) is smaller than 90, stop and return \(nr\)
  \item Increment \(n\) by a suitable amount and repeat from (\ref{itm:choose radius})
  \item Repeat for each attractor.
 \end{enumerate}

% TODO: This algorithm is implemented in the Haskell function ...

\section{Generating Clusters of Attractors}

There are various ways of generating clusters of attractors (i.e. attractors that have a low Hamming distance between each other). We shall present two different methodologies that can be used to generate them.

One way is to start with a root pattern and then reverse each bit with a probability p. The new patterns can be interpreted as noisy versions of the root pattern. We shall denote this procedure T1.

\subsection{Generating Gaussian-Distributed Clusters}

Another objective of this work was to analyse patterns that are extracted according to a Gaussian distribution with a specific mean and variance.

Since the state space is \( 2^N \) for a network of N neurons, the Gaussian Distribution will have to be defined over a huge set of states. This is highly non-trivial, since we are dealing with binary patterns, that cannot be ordered in an easy way.

In this case, we will use Mancinelli's method for sampling Gaussian distributed patterns \cite[p.~33]{federico}. We tackle this issue using the simplest possible approach: to draw numbers from a Normal Distribution and then translate them into patterns that will preserve the distance between them. Formally, if \(x\) and \(y\) were drawn and \( |x-y|=\delta\), then the Hamming distance between the encoded patterns \(x'\) and \(y'\) would also be \( d(x',y')=\delta\).

\subsubsection{Encoding of the patterns}

In order to encode a number into a binary pattern, we first round the number to the nearest integer. Let k be the integer obtained as such. Now, from left to right, we set to 1 all the bits in locations 1..k of our pattern. The remaining bits are set to -1.

For example, if the pattern size is 7 and the integer obtained is 4, we get the following pattern: [1,1,1,1,-1,-1,-1].

\subsubsection{Limitations of the encoding}

Although the method has a big advantage for simplicity, we can only obtain N different patterns out of all \( 2^N\) patterns available in the state space. However, we can regain capacity if we start flipping bits, while at the same time keeping constant the hamming distance to some certain mean pattern \(\mu\).

Another idea would be to extend our distribution to a multi-variate distribution. In this case, the capacity of the network would increase from \(N\) to \(\frac{n}{k}^k\), where k is the number of dimensions.

Unfortunately, we did not have enough time to implement these ideas, so we only experimented with the naive encoding of patterns described above.

\section{First experiments with basin sizes}
\label{sec:fexp}

In this section we shall introduce the experiments that we used to analyse various properties of the Hopfield Network. We remind the reader about the two methods that we are going to use for generating clusters of attractors:
\begin{itemize}
 \item T1: We start with a root pattern \(\mu\), and generate patterns by flipping each bit from \(\mu\) with probability \(p\).
 \item T2: This method is generating Gaussian-distributed patterns. We sample several numbers from a normal distribution with a certain mean and variance, and then encode each number into patterns of the form [1,1,1,-1,-1].
\end{itemize}

\subsection{Basin Size for One Cluster using T1}

Our first experiment is showing us the basin sizes for increasing values of p, the probability of flipping a bit. Method T1 is used, for a Hopfield Network of N neurons.

\begin{easylist}[enumerate]
\ListProperties(Style2*=,Numbers=a,Numbers1=R,FinalMark=.)
& We generate a random pattern \(\mu\)

& For all values of probability p from 0 to 0.5

    && Starting from \(\mu\) we generate P patterns using T1, and give them to the Hopfield network in order to be learned according to Hebb or Storkey rule.

    && We measure the basin size for all the patterns learned using the Storkey-Valabregue measurement and take their mean.
\end{easylist}

\begin{figure}[h]
  \centering
  \input{plot-T1-onecluster-100}
\caption{Average Basin size of patterns belonging to one cluster generated using T1}
\label{fig:plot-T1-onecluster}
\end{figure}

\subsubsection{Comments}

The results are displayed in figure \ref{fig:plot-T1-onecluster}. This initial experiment is confirming not only what Mancinelli obtained \cite{federico}, but also what we already know from theory: A Hopfield network that is trained with similar patterns loses capacity. Although we are not measuring capacity here, the basin size is strongly correlated to that.

Initially, when p is zero, we observe a huge value of 24 for the basin size. This is easily explained, since all the patterns generated are the the same as the root pattern. Subsequently, the network is trained with the same patterns that have the effect of creating a super-attractor with a huge basin size.

When p is between 0 and a critical value of 0.35, the basin size is 0, since the patterns are too close to each other and don't have enough space to fit basins of attraction between them. As soon as p goes over 0.35, the basins start to increase until a maximum size of 11.

\subsubsection{Further explanation of the results}

The results can also be easily understood and visualised in figure \ref{fig:basins}. For small p values, the attractors formed in the network are too close to each other to allow room for any basins of attraction (apart from the attractors lying at the edge of the cluster). When the p-value is high, the attractors become more disperse, leaving room for larger basins. It is also worth mentioning that the basin shapes are not necessarily hyper-spherical.

\begin{figure}[h]
  \centering
\includegraphics[scale=0.25]{images/basins_of_attraction.png}
\caption{Illustration of clusters of attractors generated using T1, with different p-values. }
\label{fig:basins}
\end{figure}

\subsection{Basin Size for Two Clusters using T1}

This experiment is similar to the previous one, however it contains two clusters this time. The value \(p_{1}\) for one cluster will stay the same (fixed at 0.45), while \( p_{2}\), corresponding to the second cluster, will vary in the range [0-0.5]. Method T1 is used, for a Hopfield Network of N neurons. The procedure is given below:
\newline
\begin{easylist}[enumerate]
\ListProperties(Style2*=,Numbers=a,Numbers1=R,FinalMark=.)
& We generate 2 random patterns \(\mu_{1}\) and \(\mu_{2}\), corresponding to clusters \( C_{1} \) and \( C_{2} \).

& We generate P patterns for \( C_{1} \), using T1 with associated probability \( p_{1}=0.45\).

& For all values of probability \( p_{2} \) from 0 to 0.5

    && Starting from \(\mu_{2}\) we generate P patterns using T1 with associated probability \( p_{2} \), and give them to the Hopfield network in order to be learned according to Hebb or Storkey rule.

    && For both sets of patterns, we measure the mean basin size using the Storkey-Valabregue measurement and plot the values on the graph.
\end{easylist}

We will be interested to observe how can the probability p influences the basin sizes. For this reason, we have run two experiments, in which we are outputting the basin sizes for cluster 1, in which p varies. The

\subsubsection{Comments}

\begin{figure}[h]
  \centering
  \input{plot-T1-twocluster-100}
\caption{Average Basin size of patterns belonging to Cluster 2, generated using T1}
\label{fig:plot-T1-twocluster}
\end{figure}

This latter experiment seems to suggest that the probability $p$ of flipping bits in the pattern doesn't actually influence the basin sizes of attractors in Cluster 1. The same behaviour has been observed in the previous graph, and this is the case especially when the root images, that are randomly generated, are far away from each other.

\subsubsection{Link to attachment theory}
A network having been trained with two clusters of patterns can be interpreted, in attachment theory, with an infant that has been exposed to two different caregivers. The patterns can be for example, visual or auditory memories describing the 2 caregivers.

\section{Experiments with Gaussian-distributed patterns}

In this section we are testing clusters of patterns that have been generated using a normal distribution. We are expecting to get similar results to the T1 method, since by the Central Limit Theorem, the Binomial distribution ~ B(N, p) is nicely approximated by a Gaussian distribution with mean N(Np, Np(1-p)). Since in the previous experiments, we used to increase the probability p of flipping a bit, this now translates to increasing the standard deviation of the normal distribution.

\subsubsection{Comments}

\begin{figure}[h]
  \centering
  \input{plot-T2-onecluster-100}
\caption{Average basin sizes of one cluster generated using T2}
\label{fig:plot-T2-onecluster}
\end{figure}

The results here are quite surprising. The experiment with Gaussian-distributed patterns shows us that the basin sizes decrease exponentially as standard deviation increases. This might be the case because at low standard deviation, the patterns would repeat themselves and create small super-attractors that have big basin sizes.
This is in contradiction with what we obtained previously, when using method T1, and further details and possible explanations are given in section \ref{inconsistencies}.

\subsection{Basin Size for Two Clusters using T2}

\subsubsection{Comments}

\begin{figure}[h]
  \centering
  \input{plot-T2-twocluster-100}
\caption{Average basin size for two clusters generated using T2}
\label{fig:plot-T2-twocluster}
\end{figure}

This experiment has been run on a network of 100 nodes, with 2 clusters of patterns:C1, with fixed standard deviation $\sigma_{1}$, and C2, with varying standard deviation $\sigma_{2}$. We have run two different trials, with $\sigma_{1}$ = 5 or 10.

As it can be easily noticed in the graph, the standard deviation does not have any effect on the basin size of the clusters. This seems to suggest that a more disperse cluster will not be affected by neighbouring attractors.

\section{Super Attractors}
\label{super_attractors}
In the context of learning models such as the Hopfield model, we define a super attractor as an attractor resulting from training a model with multiple occurrences or instances of some stored pattern. The degree of a super attractor denotes the number of occurrences of its corresponding pattern in the training set.

In the context of modelling Attachment theory, a super attractor may represent repeated interactions with the primary care giver. Clearly, it is of interest to investigate the properties of such an attractor; in particular establishing the existence and, if existent, the type of relationship between the degree of the super attractor and the extent of its dominance, or stability, over the space of patterns.

\subsection{Single super attractor}

% Define variables
\newcommand{\psuper}{$p_{super}$}
\newcommand{\prandom}{$\overrightarrow{p}_{random}$}

\begin{enumerate}

\item Fix N, the number of neurons.

\item \label{itm:choose pattern} Choose a random pattern \psuper, which signifies the primary care giver.

\item Choose a number of random patterns \prandom, such that the Hamming distance between \psuper and each of \prandom is between 25\% and 75\%.

The range forms a ball centred at 50\% Hamming distance\footnote{The percentage Hamming distance is simply the Hamming distance divided by the number of bits N} with an arbitrary radius, chosen such that the probability of a \prandom falling into \psuper's basin of attraction is small. This is done to avoid forming clusters of attractors, which we deal with separately in Recall that the Hopfield network is sign blind, and as a result the inverse of \psuper, $p_{super}^{-1}$, forms a symmetric super attractor. It is for this reason that a symmetric range about 50\% is chosen.

\item Choose a degree $d$ for \psuper and train a Hopfield network using $\overrightarrow{p}^d_{super}$ ($d$ instances of \psuper) and \prandom.

\item Measure the basin of attraction of \psuper using the Storkey-Valabregue method.

\item Repeat from (\ref{itm:choose pattern}) for various values of degree $d$.

\end{enumerate}

In our experiment 100 neurons are used, and the network is trained with 16 random patterns in addition to the super attractor. It is run with degree values {[}1, 2, 4, 8, 16, 32{]}. The entire procedure is repeated 800 times for various randomly chosen \psuper and \prandom. The average results obtained are summarised in ~\ref{fig:one super plot}.

This experiment can be replicated by running \texttt{Experiment.hs}.

\begin{figure}[h]
  \centering
\input{plot-one-super}
\caption{Average basin of attraction for a super attractor with varying degrees.}
\label{fig:one super plot}
\end{figure}

The results show that as the super attractor's degree is increased, its basin of attraction also increases. Also note that this increase appears to approach the singularity at 50\% Hamming distance, which is consistent with our knowledge of \psuper's symmetrical counterpart. The very fast initial growth also indicates that super attractors in general very stable and demonstrate strong dominance over the space of patterns.

Linking back to the Attachment theory model, this may be interpreted as depicting the strong influence of repeated and consistent interaction with the child, represented by a super attractor of increasing degree. In particular, it exhibits its dominance over distantly scattered, unrelated influences, which are represented by the random patterns.

\subsection{Two super attractors}

% Define variables
\newcommand{\poriginsuper}{$p_{origin}$}
\newcommand{\pnewsuper}{$p_{new}$}
\newcommand{\dorigin}{$d_{origin}$}
\newcommand{\dnew}{$d_{new}$}

\begin{enumerate}

\item Fix N, the number of neurons.

\item Fix a degree \dorigin, for all chosen \poriginsuper from (\ref{itm:choose two super}).

\item \label{itm:choose two super} Choose a random pattern \poriginsuper, which signifies the primary care giver.

\item Choose a random pattern \pnewsuper, such that the Hamming distance between \poriginsuper and each of \pnewsuper is between 25\% and 75\%. This could symbolise the training due to either an alternate and influential care giver, or that resulting from a retraining period.

\item Choose a number of random patterns \prandom, such that the Hamming distance between \poriginsuper and each of \prandom is between 25\% and 75\%.

Note that while this allows for the possibility of forming a cluster in the vicinity of \pnewsuper, in practice the probability of this occurring is negligible.

\item Choose a degree \dnew for \pnewsuper and train a Hopfield network using \prandom, \dorigin instances of \poriginsuper, and \dnew instances of \pnewsuper

\item Measure the basins of attraction for each of \poriginsuper and \pnewsuper using the Storkey-Valabregue method.

\item Repeat from (\ref{itm:choose two super}) for different value of degree \dnew.

\end{enumerate}

In our experiment 100 neurons are used, and the network was trained with 8 random patterns in addition to the original and new super attractors, \poriginsuper and \pnewsuper, respectively. \poriginsuper is given a fixed degree, \dorigin, of 8. This value was chosen as being the smallest point of stability from the previous experiment ~\ref{fig:one super plot}, beyond which increasing the degree further does not greatly impact the basin of attraction. It is run with \dnew degree values {[}1, 2, 4, 8, 16, 32{]}. The entire procedure is repeated 634 times for various randomly chosen \poriginsuper, \pnewsuper and \prandom. The average results obtained are summarised in ~\ref{fig:two super plot}.

This experiment can be replicated by running \texttt{Experiment2SuperAttractors.hs}.

\begin{figure}[h]
  \centering
\input{plot-two-super}
\caption{Average basin of attraction for the two super attractors, \poriginsuper having a fixed degree, \dorigin, of 8, and varying the degree, \dnew, of \pnewsuper.}
\label{fig:two super plot}
\end{figure}

The results clearly reveal that increasing the degree of the new pattern increases its own basin of attraction, and decreases that of the original pattern. This relationship is symmetrical, as can be observed by the shape of the graph, in addition to the intersection point, which occurs (roughly) near the point (8 , 8) where both patterns have the same degree of 8. By random choice of original pattern, and random choice of new pattern (subject to a Hamming distance of 25\% to 75\% from the original pattern), we can claim that the basin of attraction depends solely on the relative degrees of the two patterns, and is independent of the actual patterns themselves. This is also consistent with the symmetry observed.

This also fits in nicely nicely with in the Attachment theory interpretation. The original pattern as before represents the consistent interactions with a primary care giver. The new pattern may represent the consistent, yet significantly different, interactions with an alternate and influential care giver, perhaps the other parent of the child or a relative with whom they have developed a close relationship. Intuitively, the care giver which has more interactions would exert a stronger influence, undermining the dominance of the other care giver, and indeed the impact of any other interactions.

In relation to retraining, the new pattern may also be taken to represent the influences arising due to the undertaken retraining phase. We find that, similar to the above, increasing the number of interactions due to retraining further ingrains its dominance, and dilutes that of the previous interactions due to the primary care giver or otherwise.

% undefined variables
\let\psuper\undefined
\let\poriginsuper\undefined
\let\pnewsuper\undefined
\let\dorigin\undefined
\let\dnew\undefined
\let\prandom\undefined


\documentclass[10pt,conference,compsocconf]{IEEEtran}

\usepackage{amsmath}    % math
\usepackage{graphicx}   % For figure environment
\usepackage{todonotes}
\usepackage{subcaption}

\usepackage{hyperref}
\begin{document}
\title{Mini-project II}

\author{
  Ada Pozo P\'{e}rez, Luc\'{i}a Montero Sanchis, Milica Novakovic\\
  \textit{Deep Learning 2018, EPFL Lausanne, Switzerland}
}

\maketitle

\begin{abstract}
  This report describes the implementation of a simple deep learning framework. Several different optimizers, activation functions, layers and loss functions are implemented, including fully connected, dropout, \textit{ReLU}, \textit{Tanh}, sequential and \textit{MSE} loss among others. This framework is used for testing different architectures on a toy dataset, using different optimizers, in order to determine the effects that different activation functions and optimizers have in the training and accuracy.
\end{abstract}

\section{Introduction}
	In this project we implement a deep learning framework using only \textit{Pytorch} tensors and \textit{numpy}.

	The framework is built with two base classes: \textit{Module} and \textit{Optimizer}. The following containers are implemented as subclasses for them:

	\begin{itemize}
	    \item Layers: \textit{Linear}, \textit{Sequential} and \textit{Dropout}.
	    \item Activation functions: \textit{ReLU}, \textit{Leaky ReLU}, \textit{Sigmoid} and \textit{Tanh}.
	    \item Loss functions: \textit{Mean Squared Error (MSE)} and \textit{Cross-entropy loss}.
	    \item Optimizers: \textit{Stochastic Gradient Descent} and \textit{Adam}.
	\end{itemize}

    To test our framework we generate a toy dataset and run several experiments on it, comparing the performance of different combinations of modules, activations, optimizers and loss functions.

    The rest of this report is organized as follows: Section \ref{sec:implementation} describes how the different containers were implemented while Section \ref{sec:experiments} shows the results of the experiment to test the framework. Conclusions are drawn in Section \ref{sec:summary}.

\section{Implementation} % (fold)
\label{sec:implementation}
    We have defined two generic classes, \textit{Module} and \textit{Optimizer}, that are used as parent classes for the implementations. The optimizers are based on the structure of \textit{Optimizer}, whereas the remaining implementations are based on \textit{Module}.

    \textit{Optimizer} has two methods:
    \begin{itemize}
        \item \textit{step}: implements the update of the parameters.
        \item \textit{adaptive\_lr}: returns a generator that decays the learning rate by a factor with every optimization step.
    \end{itemize}

    In \textit{Module} we can find the methods:
    \begin{itemize}
        \item \textit{forward} and  \textit{backward}: implement the forward and backward passes of backpropagation, respectively.
        \item \textit{update}: updates the values of the parameters and resets the computed gradients to zero.
        \item \textit{params}: returns the values and the last computed gradients of the parameters of the layer.
    \end{itemize}

  \subsection{Layers} % (fold)
  \label{sub:modules}
    % Not sure about having ``modules'' as a title here
    \subsubsection{Linear} % (fold)
    \label{ssub:linear}
        The fully connected layer is implemented in the \textit{Linear} module. It is created specifying the number of input and output units. It is also possible to specify the default learning rate, which will be used to update the parameters. Nevertheless, this learning rate is only used when no learning rate is specified for the \textit{update} function.

        Unless a specific standard deviation (std) is specified during initialization, the weights are initialized with the Xavier \cite{Glorot10understandingthe} initialization adapted for ReLU \cite{DBLP:journals/corr/HeZR015} -- following a zero-mean Gaussian distribution with standard deviation given by $\text{std} = \sqrt{2/n}$,
        where $n$ is the number of input units.
        If a value for parameter \textit{std\_w} is specified, then the weights are initialized according to the zero-mean Gaussian distribution whose std value is \textit{std\_w}. For instance, when using $\tanh$ or sigmoid activation functions we recommend to specify a value for the parameter equal to $\text{std} = 1\sqrt{n}$.

        In addition to this, in the initialization it is also specified whether there is a bias -- in which case the bias values would be initialized following a Gaussian distribution with zero mean and standard deviation given by the parameter \textit{std\_b}. If the value of parameter \textit{std\_b} is not specified, they are initialized to 0. If there is no bias, in the backpropagation this parameter is ignored.

        During the forward pass, the input to the layer is saved for the subsequent gradient computation. The output of the layer $s^{(l)}$ is computed with the equation \ref{eq:forward}, where $w^{(l)}$ and $b^{(l)}$ are the values of the weights and the bias respectively and $x^{(l-1)}$ is the input to the layer. The bias term is considered if it was specified to do so, otherwise it is ignored.
        \begin{equation}
      	    \label{eq:forward}
            s^{(l)} = w^{(l)}x^{(l-1)} + b^{(l)}
        \end{equation}

        The backward pass function receives as an argument the gradient of the loss with respect to the output of the layer ($\partial l/\partial s^{(l)}$) and returns the gradient of the loss with respect to the input of the layer, which is computed as given by equation \ref{eq:backward_x}. The gradient of the loss with respect to the weights and with respect to the bias are computed with equations \ref{eq:backward_w} and \ref{eq:backward_s}, respectively:
         \begin{equation}
      	    \label{eq:backward_x}
            \frac{\partial l}{\partial x^{(l-1)}} =\frac{\partial l}{\partial s^{(l)}}  w^{(l)}
         \end{equation}
         \begin{equation}
      	    \label{eq:backward_w}
            \frac{\partial l}{\partial w^{(l)}} =\left(\frac{\partial l}{\partial s^{(l)}}\right)^T x^{(l-1)}
         \end{equation}
         \begin{equation}
      	    \label{eq:backward_s}
            \frac{\partial l}{\partial b^{(l)}} =\frac{\partial l}{\partial s^{(l)}}
         \end{equation}

        The update of the parameters is performed when the method \textit{update} is called. If a learning rate is passed as a parameter it is used for the update, otherwise the one specified upon initialization is used. As will be explained in more detail in the optimizers section \ref{sub:optimizers}, the update can be done directly on the derivatives of the bias and weights (for instance, when using SGD), or in the case when the parameter \textit{values} is specified it will use these values instead (if Adam is used, the optimizer computes the parameter \textit{values} that should be used). The equations used for the updates are therefore given by \ref{eq:update_w} and \ref{eq:update_b}, where $\gamma$ is the learning rate:
        \begin{equation}
            \label{eq:update_w}
            w^{(l)}\leftarrow
            \begin{cases}
              w^{(l)} -  \gamma \cdot \partial l/\partial w^{(l)}, & \text{if SGD is used} \\
              w^{(l)} -  \gamma \cdot \text{value}_w, & \text{if Adam is used}
            \end{cases}
        \end{equation}

        \begin{equation}
            \label{eq:update_b}
            b^{(l)}\leftarrow
            \begin{cases}
              b^{(l)} -  \gamma \cdot \partial l/\partial w^{(l)}, & \text{if SGD is used} \\
              b^{(l)} -  \gamma \cdot \text{value}_b, & \text{if Adam is used}
            \end{cases}
        \end{equation}

        When using \textit{Adam}, $\text{value}_w$ and $\text{value}_b$ are computed as given by equation \ref{eq:valueadam}.

        Lastly, the \textit{Linear} module's method \textit{params} returns a list of tuples containing a tuple with the weights and the gradient and, if bias is activated, a second tuple with the values of the bias and the gradient.
    % subsubsection linear (end)

  	\subsubsection{Sequential} % (fold)
  	\label{ssub:sequential}
        This class implements a container for a list of layers. Its parameters are the list of layers and a variable that indicates whether the model is in training or testing mode. In the forward pass, it simply calls the \textit{forward} method of each layer, propagating the output through the layers. In the backward pass, it propagates the gradient calling the corresponding method of each layer iterating through them in reverse order.

        Likewise, the \textit{params} method calls the \textit{params} method of each layer and combines their outputs, returning a list of tuples containing the values and gradient for each parameter.

        The \textit{update} method iterates over all the layers calling each layer's corresponding method. If a learning rate $\gamma$ is specified, then it is used to update all parameters. It is also possible to specify the \textit{values} parameter that should be used for the update, instead of using the gradient computed during the forward pass. As previously mentioned, this is useful for implementing the Adam optimizer and will be explained later.
   	% subsubsection sequential (end)

  	\subsubsection{Dropout} % (fold)
  	\label{ssub:dropout}
        We implement a dropout layer by defining a class \textit{Dropout}. This layer is created taking as parameters the probability of dropping a hidden unit ($p_{drop}$) and whether the model is training. If it is not training, the forward and backward passes simply return their respective inputs.

        The forward pass is implemented in the \textit{forward} method. It creates a mask of hidden units to drop and saves it for the backward pass. The mask is used to shut down the corresponding units of the input data. Lastly, the result is normalized by $1-p_{drop}$. The \textit{backward} method carries out the backward pass, using the mask saved in the forward pass to shut down the same units of the gradient of the next layer. The result is again normalized by $1-p_{drop}$.
  	% subsubsection dropout (end)
  % subsection modules (end)

  \subsection{Activation functions} % (fold)
  \label{sub:activation_functions}
    The following activation functions are implemented in our framework:
  	\subsubsection{ReLU} % (fold)
  	\label{ssub:relu}
        In the forward pass this activation layer applies the non-linear function $f(x) = max(0,x)$ to the input \cite{Nair:2010:RLU:3104322.3104425}. The backward pass is computed with the derivative:
      	\begin{equation*}
            f'(x)=
            \begin{cases}
              1, & \text{if}\ x>0 \\
              0, & \text{otherwise}
            \end{cases}
        \end{equation*}
  	% subsubsection relu (end)

  	\subsubsection{Leaky ReLU} % (fold)
  	\label{ssub:leakyrelu}
        We implement a parametric Leaky ReLU, which takes takes a parameter $\alpha$ and computes the forward pass as:
      	\begin{equation}
            f(x)=
            \begin{cases}
              x, & \text{if}\ x>0 \\
              \alpha\cdot x, & \text{otherwise}
            \end{cases}
        \end{equation}

        In the backward pass, the gradient is given by:
      	\begin{equation}
            f'(x)=
            \begin{cases}
              1, & \text{if}\ x>0 \\
               \alpha, & \text{otherwise}
            \end{cases}
        \end{equation}

  	% subsubsection leakyrelu (end)
  	\subsubsection{Tanh} % (fold)
  	\label{ssub:tanh}
        We implement the hyperbolic tangent activation function in the class \textit{Tanh}. This class computes in the \textit{forward} method the $\tanh$ of the input, which is saved for the next step. In the backward pass step, we compute the gradient as $1-\tanh^2{x}$
  	% subsubsection tanh (end)
  	\subsubsection{Sigmoid} % (fold)
  	\label{ssub:sigmoid}
        We implement the forward pass of this function using the $\tanh$ function to ensure that there is no overflow, according to the formula $\sigma(x) = 0.5 + 0.5\cdot\tanh{(x/2)}$. We store the obtained value and then compute the derivative as $\sigma(x) \cdot (1-\sigma(x))$.
  	% subsubsection sigmoid (end)
  % subsection activation_functions (end)
  \subsection{Loss functions} % (fold)
  \label{sub:loss_functions}

  Two loss functions are implemented in this framework: the Mean Squared Error (MSE) and the Cross Entropy Loss.
  	\subsubsection{Mean Squared Error (MSE)} % (fold)
  	\label{ssub:mean_squared_error}
        In the forward pass, this module computes the difference between the output of the network $f(x,w)$ and the ground truth $y$ and saves it for the backward pass. The loss returned is computed as:
      	\begin{equation}
      	Loss = \sum_{i=1}^{N}(y_i - f(x_i,w))^2
      	\end{equation}

      	The backward pass returns the gradient of the loss, which is computed using the stored difference between the output of the network and the ground truth:
      	\begin{equation}
      	    \nabla_{Loss} = 2 (y - f(x,w))
      	\end{equation}
  	% subsubsection mean_squared_error (end)
  	\subsubsection{Cross-entropy loss} % (fold)
  	\label{ssub:cross_entropy_loss}
  	This module computes the loss of the network in the forward pass as:
  	\begin{equation}
  	    Loss = -\sum_{i=1}^{N} y_i\log (f(x_i,w)) + (1 - y_i)\log(1 - f(x_i,w))
  	\end{equation}

  	In the backward pass the gradient of the loss is computed with the following equation:
  	\begin{equation}
  	    \nabla_{Loss} =  y - f(x,w)
  	\end{equation}
  	% subsubsection cross_entropy_loss (end)
  % subsection loss_functions (end)
  \subsection{Optimizers} % (fold)
  \label{sub:optimizers}
  As explained in the beginning of Section \ref{sec:implementation}, the optimizers are defined using a class with a \textit{step} method that updates the parameters and a method \textit{adaptive\_lr} that implements a generator that yields decaying learning rates. The decay is controlled by parameters $\kappa$ and $\eta$ according to:
    \begin{equation}
    \label{eq:lr}
  	    \gamma^{(t)} =  \eta\cdot t^ {-\kappa}
  	\end{equation}
  	where $t$ is incremented every time the generator is called.
  	\subsubsection{Stochastic Gradient Descent (SGD)} % (fold)
  	\label{sub:stochastic_gradient_descent_}
        The parameter update using Stochastic Gradient Descent calls the \textit{update} method of the model (explained in Sections \ref{ssub:linear} and \ref{ssub:sequential}). This update can be done either with a fixed learning rate if $\kappa$ and $\eta$ are not passed as parameters when creating the optimizer, or with the described decaying learning rate if they are passed. In the second case, each time \textit{step} is called the next learning rate is generated and passed to the \textit{update} method of the model.
  	% subsection stochastic_gradient_descent_ (end)
  	\subsubsection{Adam} % (fold)
  	\label{ssub:adam}
        This class is implemented as described in \cite{kingma2014adam}.  Given the parameters of the optimizer settings $\beta_1$, $\beta_2$ and $\epsilon$, at each timestep $t$ we first calculate the gradient with respect to the stochastic objective $g_t$ and then the first and second moment vectors are computed according to equations \ref{eq:fmv} and \ref{eq:smv} respectively:
  		\begin{equation} \label{eq:fmv}
            m_t=\beta_1\cdot m_{t-1}+(1-\beta_1)\cdot g_t
        \end{equation}
  		\begin{equation}
  		    \label{eq:smv}
            v_t=\beta_2\cdot v_{t-1}+(1-\beta_2)\cdot g_t^2
        \end{equation}

        With the previously obtained results we compute the \textit{value} argument that is used to update the parameter, as given by:
        \begin{equation}
  		    \label{eq:valueadam}
            \text{value} = \frac{m_t}{1 - \beta_1^{t}}\cdot \frac{1}{\epsilon+\sqrt{v_t/(1-\beta_2^t)}}
        \end{equation}
        Then, each parameter $\theta$ is updated as $\theta_{t}=\theta_{t-1}-lr\cdot\text{value}$.\\
        It is worth mentioning that this optimizer has a \textit{restart} method that resets the value of $t$ and the moments to $0$ (which are also the values with which the optimizer is initialized).

        The default values for the parameters are the ones used in   \cite{kingma2014adam}, that is, a learning rate of $10^{-3}$, and $\beta_1 = 0.9$ and $\beta_2 = 0.999$.
  	% subsubsection adam (end)
  % subsection optimizers (end)
% section implementation (end)

\section{Experiments} % (fold)

    \label{sec:experiments}
    \begin{figure}[t]
        \centering
        \includegraphics[width=0.45\textwidth]{figures/architecture_p2.png}
        \caption{Architecture of neural network}
        \label{fig:archi}
    \end{figure}

    \subsection{Datasets} % (fold)
    \label{sub:dataset}
    The train and test datasets are each composed of 1000 randomly generated samples. Each sample consists on two values chosen from a uniform distribution between 0 and 1, corresponding to a point in the 2D space. The samples that fall inside the disk of radius $1/\sqrt{2\pi}$ centered in $(0.5, 0.5)$ are labelled as 1's, whereas the rest are labelled as 0's.

    \subsection{Architecture} % (fold)
    \label{sub:architecture}

    The architecture of the system in the experiments can be found in Figure \ref{fig:archi}. It has 2 input and 2 output units and 3 fully connected layers with 25 units each. The fully connected layers are followed by a dropout layer, whose drop probability is set on the experiments. The two first hidden layers use the same activation, while the last one may use a different one. These activation functions vary depending on the experiment.

    \subsection{Results} % (fold)
    \label{sub:results}
        \begin{figure*}[t]
            \centering
            \begin{subfigure}{0.45\linewidth}
                \centering
                \includegraphics[width=\linewidth]{figures/res_circle_tr.png}
                \caption{Training set - 98.4\% accuracy.}
                \label{fig:rescircletr}
            \end{subfigure}%
            \begin{subfigure}{0.45\linewidth}
            \centering
            \includegraphics[width=\linewidth]{figures/res_circle_te.png}
            \caption{Test set - 97.5\% accuracy.}
            \label{fig:rescirclete}
            \end{subfigure}
            \caption{Classification results. Misclassified samples are shown in red, whereas correctly classified ones are shown in different gray levels depending on the class.}
            \label{fig:rescircle}
        \end{figure*}

        \begin{figure*}[t]
        \centering
        \begin{subfigure}{0.45\linewidth}
            \centering
            \includegraphics[width=\textwidth]{figures/adamsgdloss.png}
            \caption{Loss.}
            \label{fig:adamsgdloss}
             \end{subfigure}%
            \begin{subfigure}{0.45\linewidth}
            \centering
            \includegraphics[width=\textwidth]{figures/adamsgdaccuracy.png}
            \caption{Accuracy.}
            \label{fig:adamsgdaccuracy}
             \end{subfigure}
             \caption{Performance per epoch using SGD and Adam.}
            \label{fig:perf_optim}

        \end{figure*}
        To test our framework, we evaluate the performance with three different experiments:

        \begin{enumerate}
            \item General performance: this experiment aims to compare the performance on the train and test sets, checking whether there is any overfitting and if dropout is needed.
            \item Comparison of the optimizers: this experiment compares the performance in terms of speed and accuracy when using \textit{Adam} or \textit{SGD} as optimizers.
            \item Comparison of the activation functions: similarly to the previous experiment, this one compares the performance with different activation functions (\textit{Tanh}, \textit{ReLU} and \textit{Leaky ReLU}) for the first two layers.
        \end{enumerate}

        All the experiments use a a batch size of 50 samples and train the model for 50 epochs.

        \textbf{General performance.} For the first experiment, the model uses \textit{ReLU} as the activation function for the first two fully connected layers and  \textit{Sigmoid}  for the last one. No dropout is used - i.e. it has  0 drop probability. The optimizer used is \textit{Adam} with a learning rate of $10^{-3}$, and $\beta_1 = 0.9$ and $\beta_2 = 0.999$.  The classification results obtained are shown in Figures \ref{fig:rescircletr} and \ref{fig:rescirclete}, for training and test respectively, with the misclassified samples in red.

        The accuracy obtained for the test set is 97.5\%, very similar to the 98.4\% obtained for the training set. Hence, the model does not overfit, as could be expected since both sets are sampled from the same distribution. Therefore, we do not use dropout in any of the following experiments carried out.  It is also worth noting that all misclassified points are located in the border area.

        \textbf{Comparison of the optimizers.} In the second experiment we use the same architecture as in the previous one, but we train the model both with \textit{Adam} and \textit{SGD}. Adam uses the same parameters as before, while for \textit{SGD} we use the adaptive learning rate given by equation \ref{eq:lr}, with $\kappa=0.6$ and $\eta=0.035$. In Figure \ref{fig:perf_optim} can be found the loss and accuracy for the optimizers per epoch. From these figures one can observe that during the first 20 epochs, the training with SGD is faster, with a steeper slope than the one found using \textit{Adam}. However, this changes in the following epochs and  by epoch 50 both accuracy and loss are slightly better when using \textit{Adam}. Similarly to the previous experiment, there is no overfitting.

        \textbf{Comparison of the activation functions.}
        Lastly, we compare the results obtained using different activation functions (\textit{Tanh}, \textit{ReLU} and \textit{Leaky ReLU}) for the first two hidden layers using \textit{Adam} as optimizer. The results obtained can be observed in Figure \ref{fig:activ1comp}. In this plot we have omitted the test accuracy curves for simplicity since they were very close to their corresponding training accuracies. The accuracies obtained after 50 epochs are quite similar and do not seem to be strongly affected by the activation functions chosen for the first two activation layers. However, it is worth mentioning that \textit{Tanh} learns the fastest at first, while \textit{Leaky ReLU} seems to be the slowest.
        \begin{figure}[htp!]
            \centering
            \includegraphics[width=0.45\textwidth]{figures/activ1comp.png}
            \caption{Adam's accuracy with different activation functions.}
            \label{fig:activ1comp}
        \end{figure}

\section{Conclusions} % (fold)
\label{sec:summary}
	In this project we have implemented and used a simple deep learning framework for classifying the points that fall inside and outside a disk. The performance obtained with the chosen architecture was higher than 95\% in both train and test sets after 50 epochs. Since both sets were sampled from the same distribution, there was no overfitting in the train data. The misclassified points are located in the border area, which was to be expected because a small variation in these points results in a change of their label -- whereas small changes in points that are not in the border would not modify the labels. It should also be noted that by using a well-chosen learning rate -- in this case a decaying one -- \textit{SGD} achieved a similar result to the one obtained with \textit{Adam} width default parameters.
% section summary (end)

\bibliographystyle{IEEEtran}
\bibliography{literature}

\end{document}


\documentclass[jou]{apa6}

\usepackage[american]{babel}
\usepackage{hyperref}
\usepackage{amsthm}
\usepackage{thmtools}

\usepackage{csquotes}
\usepackage[style=apa,sortcites=true,sorting=nyt,backend=biber]{biblatex}
\DeclareLanguageMapping{american}{american-apa}
\addbibresource{bibliography.bib}

\title{2019-12-14 Class Summary: Chinese Remainder Theorem}

\author{Kalvis Aps\={\i}tis, {\small \tt kalvis.apsitis}{\small \tt @gmail.com}}
\affiliation{Riga Business School (RBS), University of Latvia (LU)}

\leftheader{NMS Selection Training in Number Theory: 2019-12-14 Class Summary}

\declaretheoremstyle[headfont=\normalfont\bfseries,notefont=\mdseries\bfseries,bodyfont = \normalfont,headpunct={:}]{normalhead}
\declaretheorem[name={Example}, style=normalhead,numberwithin=section]{problem}

\setcounter{section}{2}

\abstract{This document lists the key results from the December 14, 2019 class in
Number Theory. This training for competition math is aimed at 16\textendash{}18 year
olds (typically, Grades 10\textendash{}12).
}

\keywords{Chinese remainder theorem, Bezout's identity, Modular arithmetic.}

\begin{document}
\maketitle

\section{Examples}

These are {\bf not} the homework problems; it is just a supplementary study material
with examples taken from the lecture; most of them were analyzed during the lecture.
In case you have forgotten something, hints for these examples are given at the end of this document.
Full notes and solutions in Latvian: \url{http://linen-tracer-682.appspot.com/numtheory-tales/tale-numtheory-jun03-crt/content.html#/section}

\begin{problem}
Find integers $x,y$ such that $18x + 42y = 6$.
(Here we have chosen $6 = \mathit{gcd}(18,42)$.)
\end{problem}

\begin{problem}
Prove that the sequence $1,11,111,\ldots$ contains
an infinite subsequence such that any two members of that subsequence
are mutually prime.
\end{problem}

{\bf Blankinship Algorithm:} Blankinship Algorithm can be
used to find solutions for the
Bezout's identity: the integers $x,y$ such that $ax+by=d$.
It is explained here: \url{http://mathworld.wolfram.com/BlankinshipAlgorithm.html}.
It applies Gaussian row operations to a $2 \times 3$ matrix: you
should know how to subtract one row from another.

{\bf Inverse Congruence Class:} For a congruence class $a$
that is mutually prime with modulo $m$, denote by $a^{-1}$
a congruence class such that $a^{-1}\cdot a \equiv 1$ modulo $m$.
(In other words: Given a number $a < m$, find some number $b$
such that $a \cdot b$ gives remainder $1$ when divided by $m$.)

\begin{problem}
Find inverses $1^{-1}$, $3^{-1}$, $5^{-1}$, $7^{-1}$, $9^{-1}$, $11^{-1}$, $13^{-1}$, $15^{-1}$
(all modulo $16$).
\end{problem}

{\bf Chinese Remainder Theorem:}
For multiple mutually prime modulos $m_1,m_2,\ldots,m_k$ one can find $x$ that is
congruent to any numbers $a_1,a_2,\ldots,a_k$ with respect to those modulos.

\begin{problem}
Find a natural number $x$ that is a solution to this system of congruences:
$$\left\{ \begin{array}{l}
x \equiv 1\;(\mathit{mod}\,3)\\
x \equiv 2\;(\mathit{mod}\,5)\\
x \equiv 3\;(\mathit{mod}\,7)
\end{array} \right.$$
\end{problem}

\begin{problem}
Assume that you want to find number $x$ that satisfies both congruences:
$$\left\{ \begin{array}{l}
x \equiv 4\;(\mathit{mod}\,5)\\
x \equiv 6\;(\mathit{mod}\,11)
\end{array} \right.$$
You can solve this "graphically" - build a $5 \times 11$ table representing
all possible pairs of remainders, when you divide numbers by $5$ and by $11$.
Fill in this table by choosing $x=1,2,3,\ldots$ until you find the necessary
combination of remainders $(4;6)$.
\end{problem}

\begin{problem}
Find the smallest positive integer $n$, such that
numbers $\sqrt[5]{5n}$, $\sqrt[6]{6n}$, $\sqrt[7]{7n}$ are all positive integers.\\
(From {\em Vilniaus universiteto Matematikos ir informatikos fakulteto olimpiadas} -
a Lithuanian olympiad for high school students by Vilnius university; 2016,
Grade 10, P3.)
\end{problem}

\begin{problem}
Prove that for each positive integer n, there are pairwise relatively prime integers
$k_0, k_1, \ldots, k_n$ , all strictly greater than $1$,
such that $k_0 k_1 \ldots k_n - 1$ is the product of
two consecutive integers.
\end{problem}

\begin{problem}
Prove that for every positive integer $n$, there exist integers $a$ and $b$ such that $4a^2 + 9b^2 - 1$ is divisible by $n$.\\
({\em Math Prize for Girls Olympiad, 2010, P2}).
\end{problem}

\begin{problem}
Are there infinitely many Fibonacci numbers that give the following remainders when divided by $1001$:\\
{\bf (a)} remainder $0$; {\bf (b)} remainder $900$; {\bf (c)} remainder $1000$.
\end{problem}

\begin{problem}
Prove or disprove the following hypotheses.\\
{\bf (a)} For all $k \geq  2,$ each sequence of $k$ consecutive positive integers contains a number that is not divisible by any prime number less than $k$.\\
{\bf (a)} For all $k\geq 2,$ each sequence of $k$ consecutive positive integers contains a number that is relatively prime to all other members of the sequence.\\
({\em Baltic Way, 2016, P2}).
\end{problem}

\newpage

\section{Hints for Some Examples}

{\bf Hint 2.1:} You can find this by trial and error for small numbers.
Or you can run Euclidean Algorithm to find the GCD (greatest common divisor)
of numbers $18$ and $42$. It will tell you, how many times you should add
or subtract $18$ and $42$ to get number $6$. (Blaninship's method is essentially
the same thing.)

{\bf Hint 2.2:} If you build the following sequence of mutual primes:
$2,3,7,43,\ldots$ (every next number equals the product of all the previous ones plus $1$),
then the corresponding numbers $11$, $111$, $1111111$, and so on will give remainder $1$
every time you divide them one by another.

{\bf Hint 2.3:} In order to find, say, $9^{-1}$ (modulo $16$), you can
try out all the odd remainders ($1,3,\ldots,13,15$). Or you can solve the
Bezout's identity $9x - 16y = 1$. Blankinship's algorithm again.

{\bf Hint 2.4:} One can build such a number step by step -
first write all the numbers congruent to $1$ (modulo $3$):
$1,4,7,\ldots$ until you find one that gives remainder $2$ when
divided by $5$, and so on. (Since $3,5,7$ are mutually prime,
Chinese remainder theorem promises that you will succeed.)

{\bf Hint 2.5:} Solution is shown in the table: \url{https://bit.ly/2MCzMmf}.
Try to locate numbers
$0,1,2,3,\ldots$ in this table and see the sequence how they fill up the table.
Similar ideas are used by problems that ask you to "Measure exactly $4$ liters
of water, given two jugs with volumes $5L$ and $11L$ respectively".

{\bf Hint 2.6:} Search for $n$ in the form $n = 2^a3^b5^c7^d$.
Then write the necessary conditions (as modular congruences) for all
the unknown powers $a,b,c,d$.

{\bf Hint 2.7:} Look at the polynomial $F(t) = t^2 + t + 1$ (it is
a product of two consecutive numbers $t$ and $t+1$ plus $1$).\\
Note that all the remainders it gives, when divided by $2$, by $3$, etc.
are periodic. And if $F(t)$ sometimes is divisible by a prime $p_1$ and
sometimes by a prime $p_2$, then eventually $F(t)$ (for some special
arguments $t$) will be divisible by them both: $p_1 \cdot p_2$.\\
Now, all you need to show that there are infinitely many primes that
sometimes divide the values of $F(t)$. At this point remember
the proof that there are infinitely many primes. Assume that this is not true -
i.e. $F(t)$ is divisible by only finitely many primes. Then plug into
$F(t)=t^2 + t + 1$ the number
$t = p_1\cdot{}p_2\ldots{}\cdot{}p_k+1$ their product plus $1$.

{\bf Hint 2.8:} Use Chinese remainder theorem to avoid looking at {\em all} possible
$n$. Just look at the prime powers $p^k$ (and all the remaining $n$ can be
obtained by combining the solutions for $p^k$ in a certain way).\\
Next, consider two separate cases: $n = 2^k$ (you can now pick $b$
so that it is inverse of $3$ modulo $2^k$ - so that the term $9b^2 - 1$
is congruent to $0$). On the other hand, if $n=p^k$ for some other $p \neq 2$,
then pick $a$ equal to inverse of $2$ modulo $p^k$ for similar reasons.

{\bf Hint 2.9:} The remainders of Fibonacci numbers when divided by any fixed $d$
are periodic (because the pairs of neighboring remainders eventually
start to repeat). What is more interesting: All the remainders of Fibonacci sequence
are "clean periodic" (not just "eventually periodic") - every remainder belongs to
the period. If $F_0 = 0$ (divisible by any $d$), then it means that infinitely
often $F_n$ will be divisible by that $d$.\\
{\bf (a)} is simple - since $F_0$ is divisible by $1001$, then the remainder $0$
is clearly in the period (modulo $1001$).\\
For {\bf (b)} and {\bf (c)} you need to factorize $1001$ as a product of three prime
factors and search for the combinations of remainders (as per Chinese remainder
theorem).

{\bf Hint 2.10:} Statement {\bf (a)} is clearly false. Just start from $2$ and you
will find a sequence, where every member is divisible by some small prime.\\
For {\bf (b)} you need to express some segment of $k$ subsequent numbers as an overlap
of several arithmetic progressions with prime differences $d < k$
(so that every progression contains at least two members among these $k$ subsequent
numbers and all the $k$ numbers are covered at least by one sequence).\\
This is doable when $k=17$. See \url{https://bit.ly/2Q2XASA}. Finally - use
Chinese remainder theorem to find an actual value $N$ such that the numbers
from $N$ to $N+16$ (inclusive) give the remainders you need.

\end{document}


%!TEX root = ../thesis.tex

\chapter{Adaptivity}
\label{adp_sec:main}
\section{Introduction}
\input{adaptivity/intro.tex}

\section{Error indicator}
\label{adap_sec:error_indicator}
\input{adaptivity/error_indicator.tex}

\section{Machine learning}
\input{adaptivity/machine_learning.tex}

\section{Merging triangle}
\label{adap_merge_triangle}
\input{adaptivity/merge_tri.tex}

\section{Matrix representation of NURBS Curves}
\label{adap_sec_mrep2d}
\input{adaptivity/mrep.tex}

\section{Numerical examples}
\input{adaptivity/ex_short_cantilever.tex}
\input{adaptivity/ex_chole_adap.tex}
\input{adaptivity/ex_bracket_adap.tex}

\section{Conclusions}
\paragraph{}
In this chapter, the machine learning algorithm is adopted to develop an extensible and flexible error indicator.
Any other error estimators can be added to the existing framework and their effects can be detected based on the performance indicators in machine learning.
A MLP trained error estimator that concludes expressions related to the eigenvalues of the SBFEM formulation and some key geometric properties of the scaled boundary finite element gives a higher convergence rate compared to the uniform refinement.
In order to improve the learning effectiveness of the MLP, regularization methods including bagging and dropout are utilized.
Due to the lack of the eigenvalue error indicator in the first order triangular element, method that eliminates these situation is developed.
A matrix representation of the NURBS curves is presented to achieve a higher efficiency and stability in calculating the intersections between edge of the element and the NURBS curve.
Stress analysis is conducted on 2D linear elasticity problems.


\documentclass[main.tex]{subfiles}
\begin{document}

\section{Electron scattering}

\subsection{Compton scattering onto an electron at rest}

\marginpar{Sunday\\ 2020-8-23, \\ compiled \\ \today}

We need to account for the fact that light has a quantum nature, which is not addressed in the classical treatment of scattering.
Also, now we will account for the momentum of the photon.

We start of with an electron at rest, and a photon with energy \(h \nu \) and momentum \(h \nu / c\) impinging on it.
After the scattering, the photon will have energy \(h \nu '\) and momentum \(h \nu ' / c\), while the electron will have momentum \(m v \gamma \).
Let us call the angle between the direction of the incoming photon and the direction of the outgoing one \(\theta \).

In terms of four-vectors, we can express the momenta of the photon before and after as
%
\begin{align}
k^{\mu } = \frac{\epsilon}{c} \left[\begin{array}{c}
1 \\
\vec{\Omega}
\end{array}\right]
\qquad \text{and} \qquad
k^{\prime \mu } = \frac{\epsilon'}{c} \left[\begin{array}{c}
1 \\
\vec{\Omega}'
\end{array}\right]
\,,
\end{align}
%
where \(\vec{\Omega}\) and \(\vec{\Omega}'\) are unit vectors defining the propagation directions, such that \(\vec{\Omega} \cdot \vec{\Omega}' = \cos \theta \).
On the other hand, the momenta of the electron will be
%
\begin{align}
p^{\mu }  = \left[\begin{array}{c}
mc \\
\vec{0}
\end{array}\right]
\qquad \text{and} \qquad
p^{\prime \mu } = \gamma \left[\begin{array}{c}
mc \\
m \vec{v}
\end{array}\right]
\,.
\end{align}

Since the particles are unchanged after the scattering, both the incoming and outgoing momenta must satisfy \(p^{\mu } p_{\mu } = - m^2 c^2\) and \(k^{\mu } k_{\mu } = 0\) (in any frame: they are Lorentz scalars).
Because of momentum conservation, we can also impose the four equations
%
\begin{align}
p^{\mu } + k^{\mu } = p^{\prime \mu } + k^{\prime \mu }
\,.
\end{align}

Solving the system of equations yields
%
\begin{align}
\epsilon' = \frac{\epsilon }{1 + \frac{\epsilon }{mc^2} \qty(1 - \vec{\Omega} \cdot \vec{\Omega}')}
\,,
\end{align}
%
implying that we must have \(\epsilon' \leq \epsilon \): the photon will lose energy in the scattering.
The calculation which yields the differential cross section of the scattering is quite complicated and requires the full machinery of QED; here we just give the result:
%
\begin{align}
\frac{ \dd{\sigma }  }{ \dd{\vec{\Omega} }'}
= \frac{r_0^2}{2 } \qty(\frac{\epsilon '}{\epsilon })^2
\qty[ \frac{\epsilon}{\epsilon '} + \frac{\epsilon'}{\epsilon } - (1 - \vec{\Omega} \cdot \vec{\Omega}')^2]
\,,
\end{align}
%
where \(\epsilon \) is the energy divided by \(m c^2\) and \(r_0 \) is the classical electron radius.
\todo[inline]{Missing square for the \(\sin \theta \) term in the KN cross section! see \cite[eq.\ 7.4]{rybickiRadiativeProcessesAstrophysics1979}.}

If we substitute in our formula for the energy, using \(\xi = \cos \theta = \vec{\Omega} \cdot \vec{\Omega}'\), we find
%
\begin{align}
\frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }'}
= \frac{r_0^2}{2} \frac{1 + \xi^2}{\qty(1 + \epsilon (1 - \xi ))^2}
\qty[1 + \frac{\epsilon^2 (1 - \xi )^2}{(1 + \xi )^2 (1 + \epsilon (1 - \xi ))}]
\delta \qty(\epsilon ' - \frac{\epsilon }{1 + \epsilon (1 - \xi )})
\,.
\end{align}

We have inserted a \(\dd{\epsilon '}\) differential for the outgoing photon energy, we are considering a density in one more variable but it is singular there, whence the \(\delta \) function, which describes the distribution in \(\epsilon '\) space --- concentrated at the only value allowed by 4-momentum conservation.
Note that this makes sense dimensionally, as the dimension of a \(\delta \) function is the inverse of the dimension of its argument.

In the low energy limit, \(h \nu \ll m_e c^2 \) or \(\epsilon \to 0\), we find
%
\begin{align}
\frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }'}
=
\frac{r_0^2}{2} (1 + \xi^2) \delta (\epsilon ' - \epsilon )
\,,
\end{align}
%
which is the Thomson cross section.

Let us now define the \textbf{Compton scattering kernel} \(\sigma \): it is the differential cross section times the electron density,
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \xi ) = n_e \frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }'}
\,.
\end{align}

We can integrate this in order to find the total cross section presented by the electrons to photons of an energy \(\epsilon \):
%
\begin{align}
\sigma (\epsilon ) &= \int \dd{\Omega '} \dd{\epsilon '} \sigma (\epsilon \to \epsilon ', \xi ) \\
&= \frac{3}{4} n_e \sigma_{T} \qty[ \qty(\frac{1 + \epsilon }{\epsilon^3})
\qty(\frac{2 \epsilon (1 + \epsilon )}{1 + 2 \epsilon }- \log \qty(1 + 2 \epsilon ))
+
\frac{1}{2 \epsilon } \log \qty(1 + 2 \epsilon )
-
\frac{1 + 3 \epsilon }{(1 + 2 \epsilon )^2}
]
\,.
\end{align}

How does this differ from the Thomson cross section? For \(\log \epsilon \lesssim -1 \) we have \(\sigma \approx \sigma_T\), while as \(\epsilon \) increases the cross section goes to zero.

\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{figures/compton-sigma.pdf}
\caption{Compton cross section as a function of \(\epsilon \), photon energy by \(m_e c^2\). }
\label{fig:compton-sigma}
\end{figure}

In the low energy limit we have the expansion
%
\begin{align}
\sigma (\epsilon )\approx \sigma_T \qty(1 -2 \epsilon + \frac{26}{5} \epsilon^2)
\,.
\end{align}

The introduction of these nonconservative aspects complicates the radiative transfer equation for scattering.
The absorption term is
%
\begin{align}
- \alpha _\nu ^{(s)} I _\nu =
- I (\epsilon , \Omega ) n_e \int \dd{\Omega }' \dd{\epsilon }' \frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }' } =     - I (\epsilon , \Omega  ) \sigma (\epsilon )
\,,
\end{align}
%
while for the emission term the intensity must go inside the integral, so we have
%
\begin{align}
j_\nu^{(s)} =
n_e \int \dd{\Omega }' \dd{\epsilon '} I(\epsilon ', \Omega ') \frac{ \dd{\sigma }}{ \dd{\Omega }' \dd{\epsilon }' } = \int \dd{\Omega }' \dd{\epsilon }'
I(\epsilon ', \Omega ')
 \sigma (\epsilon \to \epsilon ', \xi )
\,,
\end{align}
%
which cannot be expressed in terms of the integrated kernel \(\sigma (\epsilon )\).

For Thomson scattering the absorption term is similar, with \(n_e \sigma_T\) instead of \(\sigma (\epsilon )\).
The emission term, on the other hand, can now be evaluated to yield
%
\begin{align}
\int \dd{\Omega }' \dd{\epsilon }' \sigma (\epsilon \to \epsilon ', \xi )
= \frac{r_0^2}{2} n_e \int \dd{\Omega '} \dd{\epsilon '} I(\epsilon ', \Omega ') (1 + \xi^2) \delta (\epsilon ' - \epsilon ) \sim n_e \sigma_T J(\epsilon )
\,,
\end{align}
%
as long as we neglect the angular dependence of the intensity.

We are always restricting ourselves to electrons which are initially at rest. For them Thomson scattering is a good approximation; photons more energetic than a few tens of \SI{}{keV} (hard X-rays) hardly scatter, since the Klein-Nishina cross section drops at high energies.

So, by using the Thomson limit we do not get it wrong by much.

The main conclusions we can draw from this is that we expect no spectral modifications (since Thomson scattering is conservative) but we do expect angular redistribution (since Thomson scattering is essentially isotropic).

\subsection{Scattering in plasmas}

Electrons are not at rest in any realistic astrophysical setting. In order to have free electrons we need a plasma, which is made of (at least partially) ionized gas.
So, we need a way to describe the fractional ionization.
Let us consider only hydrogen for simplicity: we define the collisional ionization fraction \(x\) (ionized atoms divided by total atoms), which can be expressed in terms of the temperature as
%
\begin{align}
x = \frac{F}{1 + F}
\qquad \text{where} \qquad
F = 2T \exp(-\frac{\SI{1.58e5}{K}}{T})
\,.
\end{align}

The way \(x\) looks as a function of temperature is a kind of sigmoid: it is close to zero, then at \(T\) around \SI{e4}{K} it quickly rises, and becomes close to 1 at a few times \SI{e4}{K}. At temperatures larger than \SI{e5}{K} the plasma is basically fully ionized (which makes sense: the first ionization energy of hydrogen is \(\SI{13.6}{eV} \approx \SI{1.6e5}{K}\)).

At these temperatures, the electrons will be moving quite a lot because of thermal motion.

\subsection{Inverse Compton scattering}

Scattering onto moving electrons is often called \textbf{inverse Compton scattering}, since in this case the photon might gain energy instead of losing it.
We start off with a photon with momentum \(k^{\mu } = (\epsilon /c) (1, \vec{\Omega})\) and an electron with momentum \(p^{\mu } = \gamma (mc, m \vec{v})\); the outgoing electron and momentum momenta, energies and velocities will be denoted with a prime.

We know how to deal with scattering off a stationary electron, and stationarity is relative: if we boost to the electron's rest frame we can apply the results from regular Compton scattering, and then we will need to boost back to our frame.

We shall denote quantities calculated in the ERF (electron rest frame) with a pedix \(e\): for example the energy of the photon in the ERF before the scattering will be \(\epsilon_e\).
This means that in the ERF we will have the equality
%
\begin{align}
\epsilon_e'
= \frac{\epsilon_e}{1 + \frac{\epsilon_e}{mc^2} (1 - \cos \Theta )}
\approx \epsilon_e \qty(1 - \frac{\epsilon_e}{mc^2} (1 - \cos \Theta ))
\approx \epsilon_e
\qquad \text{if } \epsilon_e \ll m c^2
\,.
\end{align}

We are saying that in the rest frame the scattering will basically be conservative, this is for sure an approximation but, because of what we discussed earlier about the Klein-Nishina cross section dropping off at high energies, not a large one.

The Lorentz transform from the LAB frame to the ERF for the energy of the photon is:
%
\begin{align}
\epsilon_e = \gamma \epsilon \qty(1 - \beta \cos \theta )
\,,
\end{align}
%
while the transform back from  the ERF to the LAB frame is
%
\begin{align}
\epsilon' = \gamma \epsilon_e' \qty(1 + \beta \cos \theta _e )
\,.
\end{align}

Note that the quantities \(\beta \) and \(\gamma \) refer to the velocity of the electron: since we are approximating the scattering as conservative in the ERF calculating them before or after the scattering is the same (the direction of the electron can change: this is accounted for by letting \(\theta\) and \(\theta '\) be different).
So, we can just insert these two multiplicative factors one after another to find the total change in energy of the photon:
%
\begin{align}
\epsilon ' &\approx \gamma^2 \epsilon \qty(1 - \beta \cos \theta ) \qty(1 + \beta \cos \theta'_e )
\,.
\end{align}

\todo[inline]{Inaccuracy in the slides: the first formula does not really make sense, since the approximation of the nonconservativeness factor being equal to one has been made in part of it (\(\beta \) and \(\gamma \) being the same before and after), so it does not make sense to write it out.}

Let us neglect the angular terms for now: generally they are of order unity. Instead, the main factor in the formula is \(\gamma^2>1\): this means that in general the energy of the scattered photon is of the order \(\epsilon ' = \gamma^2 \epsilon > \epsilon  \).
The boost of the energy of the photon depends on how relativistic the electron is; if the electron is very relativistic the boost in energy for the photon can be quite large.

In order to make predictions about the effect of this for a population of electrons and photons, we can Lorentz transform the Compton Scattering Kernel: the final result of the manipulation is
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \xi ) = \frac{D}{D'} \sigma_e \qty(\epsilon _e \to \epsilon _e', \xi _e)
\,,
\end{align}
%
where \(D = 1 - \vec{\Omega} \cdot \vec{v} / c = 1 - \beta \cos \theta \) and \(D' = 1 - \vec{\Omega}' \cdot \vec{v}' / c \approx 1 + \beta \cos \theta '_e\) are the factors in the Lorentz transforms, which allow us to write \(\epsilon _e = \gamma D \epsilon \) and \(\epsilon _e' = \gamma D' \epsilon '\).

The transformation law for \(\xi = \vec{\Omega} \cdot \vec{\Omega}\) is
%
\begin{align}
1 - \xi = \frac{1 - \xi _e}{\gamma^2 D D'}
\,,
\end{align}
%
while the electron energy density transforms as \(n = \gamma n_e\).

This allows us to write down an explicit expression for \(\sigma (\epsilon \to \epsilon ', \vec{\Omega}, \vec{\Omega}')\) in the lab frame for a population of single-speed electrons:
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \vec{\Omega}, \vec{\Omega}', v)
=
\frac{n r_0^2}{2 \epsilon \nu \gamma }
\qty[1 + \qty(1 + \frac{1 - \xi }{\gamma^2 D D'})^2
+ \frac{\epsilon \epsilon ' (1 - \xi )^2}{\gamma^2 D D'}]
\delta \qty(\xi -1 + \frac{\gamma D}{\epsilon'} - \frac{\gamma D'}{\epsilon })
\,,
\end{align}
%
but we must also consider the fact that the electrons are distributed with different velocities: if their distribution is isotropic, so that \(\dd{n} = n f(v) \dd[3]{v}\) then we can integrate across all velocity space, getting an expression like
%
\begin{align}
\sigma (\epsilon \to \epsilon ', \xi )
= \int \dd{v} \sin \theta_v \dd{\theta _v} \dd{\phi _v}
 v^2 n f(v)
\sigma (\epsilon \to \epsilon ', \vec{\Omega}, \vec{\Omega}', v)
\,.
\end{align}

\end{document}


\chapter{Related work}\label{relatedwork}
State-space dimensionality reduction has been a topic of interest in RL for several years. In their 2015 paper, Curran et al. \cite{mario} used PCA to reduce the dimensionality of the state-space in a Super Mario environment. They found that with the right number of principal components, an agent using PCA was able to converge to a better policy and needed less episodes than an agent using the full observation. Curran et al. corrobarated these results in more environments in later research in 2016 \cite{pca_curran}. Similar results were found by Shah et al. in 2011 in the context of neuroscience \cite{pca_neural}. Bitzer et al. also found similar results in 2010. \cite{pca_bitzer}. Reducing the dimensionality using PCA led to better policies and quicker convergences; this was interpreted as ``[...] due to the higher dimensionality, learning in the full 4D state space requires far more exploration to cover the same proportion of space" \cite{pca_bitzer}.

Unlike the Starcraft II environment used in our research having image/grid based observations, the states and observations used in these papers are comprised of non-spatial related variables, i.e. linear states; to, for instance, denote the presence of one or more enemies within one grid-cell distance of Mario's position, Curran et al. use one variable with $2^8$ possible values ($0$-$255$). Hence we are using PCA in a different setting, with an observation format more commonly used in modern RL. Furthermore they use the Q-learning algorithm (mentioned in section \ref{pl-dqn}) which is not a state-of-the-art learning algorithm anymore and handles larger state-spaces badly due to using a lookup-table instead of a neural network for the Q-function.

Research has also been done with regards to using autoencoders for state-space-dimensionality reduction in RL. Lange and Riedmiller in 2010 \cite{AE_2010} used visual data as observations and found that the agent was able to find an optimal policy using lower dimensional data from the autoencoder. They did not compare the performance of this agent with any other agent. Furthermore, their system only has $31$ possible states, which is very limited; in contrast, our Starcaft II environment has a total of $731.187$ possible states.

In 2016, Finn et al. successfully showed the possibility of training an RL agent on a latent representation given by an autoencoder on visual information \cite{ae_visual}. The same year, Van Hoof et al. \cite{AE_2016} used an autoencoder to project noisy sensor data unto a lower dimensional space. They found that the agent using the autoencoder was far better able to find a good policy than the agent using full observations. This was explained as the agent being too sensitive to noisy data, hence being unable to train even decently. Our research in contrast explores an environment where the baseline agent is able to train to a good policy.

Ha and Schmidhuber use a variational autoencoder (VAE) \cite{vae} in their 2018 paper to train an agent in image based environments \cite{rl_vae}. Through this method, their agent outperformed baseline agents. They did find that the VAE produced mixed results for the latent representation; in an environment where details are important, they were lost in the latent representation. Differently from our research, they do not compare different reduction methods. Furthermore they do not use deep reinforcement learning, but train the agent using \textit{evolution strategies} \cite{es}. Lee et al. also train an agent on a variational autoencoder and found similar results \cite{rl_vaetwo}.

Khan et al. used an autoencoder to project visual data to a latent space and were able to train a self-driving car in the CARLA environment \cite{rl_carla}\cite{carla}. However, no comparison to a baseline agent (or other reduction methods) was given.

In 2019, Prakash et al. \cite{AE_2019} also used an autoencoder on visual data to project the observation data onto a lower dimensional space. They found that an agent using the autoencoder far outperformed the baseline agent. The baseline agent did not find a decent policy, though the authors claim that with enough episodes it would have.

Another paper in 2019 by Gelada et al. \cite{deepmdp} compared using a DeepMDP with using an autoencoder for state-space dimensionality reduction. They found that in simpler environments a DeepMDP can find better representations on lower dimensionality space than an autoencoder. Simultaneously though, they also find that in more complex environments, like Atari games, DeepMDPs can have trouble finding a good representation and that its loss can be difficult to optimize. They also find that a DeepMDP agent generally outperforms a baseline agent.

%project_matrix_2020
%dimred_overview_2008