\section{Results}
\label{section:results}

In order to demonstrate the performance of our method, we conducted three sets
of tests.
In the first we simulated observables from a set of stellar parameters for a
few hundred stars using the MIST stellar evolution models and the
gyrochronology model of equation \ref{eqn:gyro}.
The ages predicted with our model were compared to the true parameters used to
generate the data.
In the second we tested our model by measuring the ages of individual stars in
the NGC 6819 open cluster, and in the third we tested our model on \kepler\
asteroseismic stars.

\subsection{Test 1: simulated stars}
For the first test we drew masses, ages, bulk metallicities, distances and
extinctions at random for 1000 stars from the following uniform distributions:
\begin{eqnarray}
& \mathrm{EEP} \sim U(198, 480) \\
% & M \sim U(0.5, 1.5)~[M_\odot] \\
& t \sim U(0.5, 14)\mathrm{~[Gyr]} \\
& [Fe/H] \sim U(-0.2, 0.2) \\
& D \sim U(10, 1000)~\mathrm{[pc]} \\
& A_V \sim U(0, 0.1).
\end{eqnarray}
\teff, \logg, \fhat, parallax, and apparent magnitudes $B$, $V$, $J$, $H$, $K$,
\gaia\ $G$, $G_{BP}$ and $G_{RP}$ were generated from these
stellar parameters using the MIST stellar evolution models.
We added a small amount of noise to the `observed' stellar properties in order
to reflect optimistic observational uncertainties for isochrone-dating.
We added Gaussian noise with a standard deviation of 25 K to \teff, 0.01 dex
to \feh\ and \logg, and 10 mmags to $B$, $V$, $J$, $H$, and $K$ magnitudes.
These are just one choice of uncertainties that we could have adopted and are
extremely optimistic.
The uncertainties on predicted ages {\it will} depend strongly on all
observational uncertainties, however, since this analysis is designed to show
the relative improvement in stellar age precision when {\it rotation periods}
are included, we chose to use best-case spectroscopic parameters.
The noise added to Gaia $G$-band photometry ranged from
0.3 mmag for stars brighter than 13th magnitude, to 10 mmag for stars
around 20th magnitude \citep{evans2017, brown2018}.
Noise added to \gaia\ $G_{BP}$ and $G_{RP}$ bands ranged from 2 mmag for stars
brighter than 13th magnitude to 200 mmag for stars fainter than 17th.
Unphysical combinations of stellar parameters were discarded, resulting in a
final sample size of 841 simulated stars.
Figure \ref{fig:CMD_age} shows the position of these stars on an HRD
(with \logg\ on the y-axis instead of luminosity to improve the visibility of
the MS), colored by their age.
Rotation periods for these stars were generated using the gyrochronology
relation described in equation \ref{eqn:gyro}.
We added 5\% Gaussian noise to all stellar rotation periods to represent
realistic measurement uncertainties of 5\%.
The median uncertainty on rotation periods calculated from \kepler\ light
curves, provided in the \citet{mcquillan2014} catalog is 1\%.
However, the \citet{aigrain2015} injection and recovery study showed that true
rotation period uncertainties are often slightly larger than this, and noise
distribution of rotation periods can be highly non-Gaussian
\citep[\eg][]{aigrain2015, angus2018}.
\begin{figure}
  \caption{
      The simulated star sample plotted on an HRD, colored by age
    (top panel) and rotation period (bottom panel).
    HRD positions were calculated using MIST isochrones via the {\tt
    isochrones.py} {\it Python} package and rotation periods were generated
    using equation \ref{eqn:gyro}.
    This figure was generated in a Jupyter notebook available at
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Simulate_data.ipynb}
}
  \centering
    \includegraphics[width=1\textwidth]{simulated_CMD}
\label{fig:CMD_age}
\end{figure}
Figure \ref{fig:rotation_model} shows the rotation periods of 841
stars generated from the gyrochronology model.
\begin{figure}
  \caption{
Data simulated from the rotation period model.
    Late F, GK and early M dwarfs (stars with 0.56 $<$ \gcolor\ $<$ 2.7 follow
    the Praesepe-calibrated gyrochronology relation (dashed gray lines), with
    the exception of old, slowly
    rotating stars with large Rossby numbers whose rotation periods are fixed
    at 2$\times$ their convective overturn time.
    The rotation periods of early F (\gcolor $<$ 0.56), late M dwarfs (\gcolor
    $<$ 2.7) and subgiants (EEP $\gtrsim$ 420) were generated
    from a log-normal distribution with standard deviation given by equation
    \ref{eqn:gyro}.
% The top panel shows the rotation periods vs. B-V colors of simulated stars,
The top panel shows the rotation periods vs. \gcolor\ colors of simulated stars,
    colored by their age and the bottom panel shows the same stars colored
    by their equivalent evolutionary phase (EEP).
    % 9, 11, and 13 (rotation periods rise with age).
    The gray lines describe the mean gyrochronology model at ages 1,
    3, 5, 7, 9, 11, and 13 (rotation periods rise with age).
    This figure was generated in a Jupyter Notebook, available at
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Simulate_data.ipynb}
}
  \centering
    \includegraphics[width=1.\textwidth]{rotation_model_praesepe}
\label{fig:rotation_model}
\end{figure}

We took two approaches to inferring the ages of these simulated stars:
firstly using isochrone fitting {\it only}, and secondly using isochrone
fitting {\it combined with} a gyrochronology model (\sd).
Since the posterior PDFs of stars are often multimodal, we found that the
choice of initial positions of the {\tt emcee} walkers influenced the final
outcome because walkers occasionally got stuck in local minima.
We found that the following set of initial parameters worked well, though not
perfectly: EEP = 330, $t = 9.56$ Gyr, $[M/H] = -0.05$, $D = 269$ pc and $A_V =
0.0$\footnote{These are the default initial parameters provided in \sd.}.
Figure \ref{fig:simulation_results} shows the results of combining
gyrochronology with isochrone fitting for the simulated sample.
The stars' true ages are plotted against their predicted ages, with ages
inferred with gyrochronology and isochrone fitting
in color, and ages inferred using isochrone fitting only plotted in light
grey.
The different panels show the results for different types of stars: FGK
dwarfs that are still undergoing magnetic braking, FGK dwarfs that have ceased
magnetic braking (their Rossby number is around 2), M dwarfs, and evolved
stars.
The selection criteria for these groups are in the panel headings.
The FGK dwarfs with low Rossby numbers showed the largest improvement: the
median age precision for this group (defined as the standard deviation of the
posterior as a percentage of the median age) was 8\% when using a combination
of isochrones and gyrochronology, and 22\% using isochrone fitting alone.
This equates to an almost 3$\times$ improvement in age precision.
The age RMS of this group was 0.8 Gyr using isochrones and gyrochronology, and
2 Gyr using isochrones only.
Despite the fact that stars with Rossby numbers of 2 have stopped spinning
down and their rotation periods no longer evolve with age, the ages of these
stars can still be relatively precisely constrained with isochrone fitting.
The median age precision for stars with large Rossby numbers was 11\% with
gyrochronology and isochrone fitting, and 13\% with isochrone fitting only.
% This is because their rotation periods are still directly related to their
% age, but also {\it indirectly} related to their age via EEP and metallicity,
% as shown in equation \ref{eqn:gyro}.
% When including rotation periods to infer the ages of stars with large Rossby
% numbers, age precision improves from 21\% to 11\% and the age RMS improves
% from 2.5 to 0.9 Gyr.
% Age precision also increases for F dwarfs when incorporating their rotation
% periods, even though they are not directly related to their ages.
% This is because rotation period relates to color via the variance model in
% equation \ref{eqn:gyro}.
The precision of M dwarf ages did improve overall when their rotation periods
were included, but this improvement was entirely driven by the early M dwarfs.
The precision of this group improved overall from 33\% to 22\% and RMS from
5.4 Gyr to 4.6 Gyr.
The precision of ages inferred for evolved stars changed very little when
rotation periods were included in the inference process.
This is because the variance of the rotation-age relation was inflated by a
large amount in the gyrochronology model (equation \ref{eqn:gyro}), making
rotation periods almost entirely uninformative for this group.
The median age precision of subgiants from both gyrochronology and isochrone
fitting and isochrone fitting alone was 7\%.

\begin{figure}
  \caption{
The true vs. predicted ages of simulated stars.
    Ages calculated by combining gyrochronology
    and isochrone fitting with \sd\ are shown in color and ages calculated with
    isochrone fitting only are shown in gray.
The different panels show the results for stars
    with \gcolor\ $<$ 2.2 (FGK dwarfs) that are still braking magnetically
    ($Ro < 2$), stars with \gcolor\ $<$ 2.2 that have stopped braking
    magnetically ($Ro \geq 2$),
    stars with 2.2 $<$ \gcolor\ (M dwarfs), and evolved stars (EEP $>$ 420).
Gyrochronology is highly effective for FGK stars and ages inferred with both
    gyrochronology and isochrone fitting are more accurate and precise than
    ages inferred via isochrone fitting only for this group.
Neither gyrochronology nor isochrone fitting can provide precise ages for
    M dwarfs, so the ages of these stars are imprecise regardless of
    age-dating method.
    This figure was generated in a Jupyter Notebook available at
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Results_plots.ipynb}.
}
  \centering
    \includegraphics[width=1\textwidth]{simulation_results}
\label{fig:simulation_results}
\end{figure}

An important caveat associated with these results is that they strongly depend
on the uncertainties adopted for all observables used in the analysis: \teff,
\feh, \logg, $G$, $G_BP$, $G_RP$, $J$, $H$, $K$, $B$, $V$, and rotation
period.
Changing the uncertainties on these observables will affect the uncertainties
on inferred ages in different ways.
This demonstration is not intended to reflect the typical age uncertainties
that will result for all stars in reality, it merely exemplifies the increase
in stellar age precision that results from one specific choice of
uncertainties.
Needless to say, estimating the uncertainties on observables accurately can be
as important as accurately measuring the observables themselves.

% Figure \ref{fig:all_iso_gyr} shows the inferred vs. true ages of all simulated
% stars, colored by their \gaia\ \gcolor\ age.
% The ages of FGK stars (\gcolor\ $<$ 2.2) were more precisely recovered with
% gyrochronology and isochrone fitting than with isochrone fitting alone.
% In both cases M dwarfs ages were not precisely recovered.
% \begin{figure}
%   \caption{
%       Inferred vs. true ages for all simulated stars using isochrones plus
%       gyrochronology via the \sd\ code (left) and isochrones only (right).
%       Stellar ages are colored by \gaia\ \gcolor\ color.
%       The ages of FGK stars (\gcolor\ $<$ 2.2)
%       were more precisely recovered with \sd\ than with isochrones only.
%       In both cases M dwarfs ages were not precisely recovered.
%       Points in the lower right of the left panel are either M dwarfs or
%       evolved (EEP $>$ 400).
%       In both panels, the ages of FGK stars were slightly underestimated.
% }
%   \centering
%     \includegraphics[width=1\textwidth]{all_iso_gyro}
% \label{fig:all_iso_gyro}
% \end{figure}

% Figure \ref{fig:eep} shows the inferred equivalent evolutionary phase (EEP)
% for the simulated star sample.
% EEP is a dimensionless number that describes evolutionary stage and, combined
% with age and metallicity, determines stellar mass.
% An interesting result of inferring stellar properties with both isochrones
% {\it and} rotation periods is an increase in EEP precision, .
% The reason for this is partly that EEP and age are correlated, so shrinking
% the age posterior also shrinks the EEP posterior, and partly because rotation
% period is a function of HRD position.
% As shown in equation \ref{eqn:gyro}, rotation period does not just depend on
% color and age, but also on EEP and [Fe/H].
% \begin{figure}
%   \caption{
%     Inferred EEP for the simulated star sample with isochrones plus
%     gyrochronology (left panel) and isochrone fitting only (right panel).
%     Rotation periods provide information about position on the HRD via
% equation \ref{eqn:gyro} and therefore improve all stellar parameters, not
% just age.
%     This figure was generated in a Jupyter Notebook available at
%     \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Results_plots.ipynb}.
% }
%   \centering
%     \includegraphics[width=1\textwidth]{eep}
% \label{fig:eep}
% \end{figure}
% In general, the posterior PDFs of stellar properties, inferred using
% isochrones or stellar evolution models, are multimodal.
% The stars with under-predicted EEP in both panels of figure \ref{fig:eep} are
% slightly evolved, with EEPs $>$ 400.
% These slightly evolved stars have two high probability modes: one at their
% true EEP and age values, and another at a lower EEP and corresponding lower
% age.
% In other words, these are subgiants being mistaken for dwarfs.
% Rotation periods do not carry information for subgiants, so including the
% rotation periods in the analysis of these stars doesn't break the degeneracy,
% and this is why stars with under-predicted EEPs are present in both panels.
% Stars with over-predicted EEPs in the right-hand panel of figure \ref{fig:eep}
% are usually dwarfs being confused for subgiants.
% The rotation periods of dwarfs are informative, so when the rotation periods
% are included in the analysis for these stars, the degeneracy is broken, the
% amplitude of the secondary mode is reduced, and their EEP is correctly
% inferred.

% Figure \ref{fig:precision} shows the simulated stars on an HRD, with
% points colored by the precision of their predicted ages
% (defined as the standard deviation of the age
% posterior PDF, as a percentage of the median age).
% The top panel shows the precision of ages calculated using both gyrochronology
% and isochrone fitting via \sd\ and the bottom panel shows the precision of
% ages calculated with isochrone fitting only.
% Although these are empirical uncertainties, calculated via MCMC which only
% approximates the age posterior PDFs, they nevertheless show that combining
% gyrochronology and isochrone fitting improves age precision on the MS,
% particularly at low masses.
% \begin{figure}
%   \caption{
% Simulated stars on an HRD, colored by their relative age precision
%     using gyrochronology and isochrone fitting via \sd\ (top panel) and
%     isochrone fitting only (bottom panel).
% Combining gyrochronology with isochrone fitting significantly improves stellar
%     age precision on the MS.
% Isochrone fitting provides precise ages for hot stars and subgiants and the
%     rotation periods of these stars are relatively uninformative, so
%     gyrochronology does not significantly improve their age precision.
% The ages of late M dwarfs are highly imprecise because their ages are not well
%     determined by either their rotation periods or their position on the HRD
%     or CMD.
%     This figure was generated in a Jupyter Notebook available at
%     \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Results_plots.ipynb}.
% }
%   \centering
%     \includegraphics[width=1\textwidth]{precision_plot}
% \label{fig:precision}
% \end{figure}

This simulation experiment was designed to show the theoretical improvement in
age precision when gyrochronology is incorporated into isochrone fitting.
However, it does not demonstrate the accuracy of this method because the test
data were simulated from the same model used to infer ages.
The results of this experiment are therefore extremely accurate by design.
When applying this method to real data, the results will only be accurate if
the model is accurate.
In other words, \sd, like any age-dating method, provides model-dependent
ages.
Stellar ages calculated with \sd\ depend on both the accuracy of the MIST
models {\it and} the accuracy of the gyrochronology model (equation
\ref{eqn:gyro}).
In order to test the accuracy of \sd, we applied it to real data, as described
in the following section.

\subsection{Test 2: Open clusters}
In order to test our model on real stars with known ages, we selected a sample
of stars in the 2.5 Gyr NGC 6819 cluster.
We compiled \kepler-based rotation periods \citep{meibom2015}, \Gaia\
photometry and \gaia\ parallaxes for members of the NGC 6819 cluster.
Figure \ref{fig:NGC6819} shows the period-color relation of this
cluster and figure \ref{fig:NGC6819_results} shows the results of inferring
the ages of individual cluster members using a combination of gyrochronology
and isochrone fitting (via \sd) and isochrone fitting alone.
The ages of F stars in this cluster (\gcolor\ $\sim$ 5.5-6.5) were relatively
precisely constrained by isochrone fitting alone because, at 2.5 Gyr, they are
approaching the MS turnoff.
For these hot stars, ages inferred with gyrochronology and isochrones were
similar to ages inferred with isochrones and similarly precise, showing that
isochrones provide a lot of age information for these stars and rotation
periods do not add significantly more information.
The G and early K stars in this cluster (\gcolor\ $\lesssim$ .65) were not
precisely recovered from isochrone fitting alone -- the isochrone-only age
posteriors tend towards the prior which is a uniform distribution between 0
and 13.8 Gyrs.
The median age of stars in the cluster was 4.27 $\pm$ 0.48 Gyr when only
isochrone fitting was used.
In contrast, including gyrochronology when inferring the ages of G and K stars
in this cluster significantly improved age precision.
The median age of stars in this cluster was 2.63 $\pm$ 0.16 Gyr when ages were
inferred with a combination of isochrone fitting and gyrochronology, using the
newly calibrated Praesepe-based gyrochronology model.
The previously-calibrated \citet{angus2015} model resulted in a median stellar
age of 2.66 $\pm$ 0.21 Gyr, which is still consistent with the established
cluster age of 2.5 Gyr.
The median age of stars in the cluster using uncorrected photometry was
slightly underestimated at 1.86 $\pm$ 0.22 Gyr.
This suggests that, despite the fact that V-band extinction is marginalized
over during the inference process, correcting for extinction {\it before} ages
are estimated will reduce bias introduced by dust.

\begin{figure}
  \caption{
    The \kepler-based rotation periods of members of the 2.5 Gyr NGC 6819 open
    cluster.
    The raw \gcolor\ colors are shown in red and the dust-corrected colors are
    shown in black.
    The dashed line shows a gyrochronology model that was fit to the Praesepe
    cluster and the Sun in this work, interpolated to 2.5 Gyrs.
    The solid blue line shows a previously calibrated gyrochronology model
    \citep{angus2015}.
    This figure was generated in a Jupyter Notebook available at:
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/NGC6819.ipynb}
  }
  \centering
    \includegraphics[width=1\textwidth]{NGC6819}
\label{fig:NGC6819}
\end{figure}

\begin{figure}
  \caption{
    The inferred ages of members of the NGC 6819 open cluster as a function of
    their \gcolor\ color.
    Ages of stars inferred using a combination of isochrone fitting and
    gyrochronology (Praesepe and Sun calibration) with dereddened \gaia\ $G$,
    $G_{BP}$, and $G_{RP}$ photometry (black circles) and uncorrected, raw,
    photometry (red squares).
    % Black circles and red squares show the ages of stars inferred using a
    % combination of isochrone fitting and gyrochronology, with a gyrochronology
    % relation that was calibrated to Praesepe and the Sun.  Black circles show
    % ages inferred with dereddened \gaia\ $G$, $G_{BP}$ and
    % $G_{RP}$ photometry and red squares show ages inferred with uncorrected,
    % raw, photometry.
    Even though V-band extinction is marginalized over in the inference
    process, reddening can still bias ages.
    Blue triangles, pointing up, show ages inferred using isochrone fitting
    and gyrochronology, with the \citet{angus2015} gyrochronology model.
    Orange triangles, pointing down, show ages inferred using isochrone
    fitting only.
    The ages of F stars (stars bluer than 0.7) were precisely constrained by
    isochrones and including gyrochronology makes little difference to their
    inferred ages.
    The age precision of G and K dwarfs (stars redder than 0.7) was improved
    by including gyrochronology.
    The median age of stars inferred using the gyrochronology model
    calibrated to Praesepe and the Sun (black circles) was 2.65 $\pm$ 0.13
    which is consistent with the established cluster age (2.5 Gyr).
    This figure was generated in a Jupyter Notebook available at:
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/NGC6819.ipynb}
}
  \centering
    \includegraphics[width=1\textwidth]{NGC6819_results}
\label{fig:NGC6819_results}
\end{figure}

We found that 5\% rotation period uncertainties resulted in the most accurate
ages for NGC 6819.
The uncertainties on the measured rotation periods, provided in
\citet{meibom2015} and shown in figure \ref{fig:NGC6819}, were likely
underestimated for some stars.
Underestimated rotation period uncertainties can result in inaccurate age
estimates.
This raises the question: how should uncertainties on rotation periods be
estimated?
The likelihood is weighted by the inverse variance, so uncertainties on the
rotation period control the relative information provided by gyrochronology,
isochrones, and the prior.
If rotation period uncertainties are either too large or too small, the
resulting age estimate will be imprecise and/or inaccurate.
It is difficult to measure uncertainties on rotation periods directly:
standard techniques such as Lomb-Scargle periodograms and autocorrelation
functions do not provide them.
% In general, formal rotation period uncertainties should probably be calculated
% empirically via simulations as in the \citet{aigrain2015} study.
\racomment{Ideally, rotation period uncertainties should capture {\it both} the
measurement precision, {\it and} the physical uncertainty introduced by the
latitudinal movement of star spots on the surface of a differentially rotating
star.
For example, \citet{donahue1996} demonstrated that the seasonal variation in
measurements of G and K star rotation periods is a function of period, $\Delta
\mathrm{P_{rot}} \propto \mathrm{P_{rot}}^{1.3\pm0.1}$.
This variation is presumably caused by a latitudinal drift in the dominant
active regions, which traces the stellar cycle over several years, in
combination with latitudinal differential rotation.
This suggests that latitudinal spot drifting does not significantly
affect stellar rotation periods when stars are young, for example the scatter
of rotation periods about the mean gyrochronology model in Praesepe is only
around 5\%.
However it is likely that this effect will become more important at older
ages.
% The impact of this phenomenon on gyrochronology remain to be seen, and an
% in-depth study is warranted.
A thorough exploration of how rotation period uncertainties, from both
measurement uncertainty and physical variation, affect stellar ages via
gyrochronology is key to understanding the power of gyrochronology as an
age-dating method.
For now, we leave this exploration for a future study.}

\subsection{Test 3: Kepler asteroseismic stars}
\racomment{In order to test our method in the regime where both isochrone
fitting and gyrochronology become important, we recovered the ages of the 21
asteroseismic stars analysed in \citet{vansaders2016}.
These 21 stars were observed in \kepler's short cadence mode and are a mixture
of dwarfs and subgiants.
Their asteroseismic ages were calculated from the analysis of the frequencies
of individual oscillation modes \citep{mathur2012, metcalfe2014,
silvaaguirre2015, ceillier2016}, and their rotation periods from their
\kepler\ light curves \citep{garcia2014}.
We crossmatched these stars with the \Gaia\ catalog to obtain parallaxes and
apparant magnitudes in the \Gaia\ G, $\mathrm{G_{BP}}$ and $\mathrm{G_{RP}}$
band passes.
We also added J, H and K 2MASS magnitudes from the Kepler input catalog
\citep{brown2011},
% asteroseismic parameters $\Delta_\nu$ and $\nu_{\mathrm{max}}$ from
% \citet{ceillier2017} and \citet{serenelli2017},
and used spectroscopic effective temperatures, spectroscopic metallicities and
rotation periods reported in table 1 of \citet{vansaders2016}.
The \gaia\ photometry is extremely precise, and we found that artificially
inflating the uncertainties on \gaia\ apparent magnitudes by a factor of 10
substantially improved the quality of fit, both in terms of MCMC convergence
and agreement with asteroseismic age measurements.
In figure \ref{fig:astero} we show the ages of these 21 stars inferred using
isochrone fitting and gyrochronology, against their asteroseismic ages,
calculated using the Asteroseismic Modeling Portal (AMP) \citep{metcalfe2009,
metcalfe2012, metcalfe2014}.
In this figure, the colored symbols show ages inferred using a combination of
gyrochronology and isochrone fitting, implemented with the \sd\ {\it Python}
package.
The black and grey triangles show ages inferred from gyrochronology only,
where the mass and color of the stars were not inferred, but fixed to be the
asteroseismic mass and the \gaia\ \gcolor\ color.
These gyrochronal ages were calculated with age as the only free parameter,
without marginalizing over stellar mass, \gcolor\ color or extinction.
As a result, these gyrochronal ages are {\it not the same} as the ages given
by the maximum of the gyrochronal likelihood function used in the combined
age model, they simply represent an approximation to the gyrochronal age.
Grey triangles are shown for stars where gyrochronology is not applicable
because the stars are either too metal poor, too metal rich, or too evolved.
Black triangles are shown for stars where gyrochronology {\it is} applicable.
The white circles show ages inferred from isochrones only.
Dashed lines connect the three different age measurements for the same
stars.
}

\racomment{
Although the ages of all 21 stars shown in figure \ref{fig:astero} were
inferred with a joint isochronal and gyrochronal model, most (all but 8) were
either too evolved, too metal poor, or too metal rich for gyrochronology to
contribute any information to the ages.
These metal poor/rich or evolved stars lie in a regime where the variance on
their rotation period was artificially inflated because the gyrochronology
relations are not thoroughly understood or well calibrated.
The rotation periods of the remaining eight stars {\it did} contribute to
their inferred ages to some degree, however isochrones still dominated the age
information for some of them because these stars are relatively old and/or
relatively hot.
% Isochrones are information-rich for these hot and old stars, and this is
% exactly where gyrochronology is at its most information-poor.
}

\racomment{
In general, there is relatively poor agreement between the asteroseismic ages
and the ages inferred using \sd.
Much of this discrepancy is driven by differences in the isochronal ages,
which is likely attributable to differences between the MIST stellar evolution
models and those used in the AMP analysis: a combination of the Aarhus stellar
evolution code \citep[ASTEC][]{christensen-dalsgaard2008a} and the adiabatic
pulsation code \citep[ADIPLS][]{christensen-dalsgaard2008b}.
We compared non-rotating, Solar-metallicity MIST isochrones for middle-aged
stars with Solar-metallicity BaSTI isochrones \citep{pietrinferni2004,
hidalgo2018} and found that, for stars between 4 and 8 Gyrs, in the same
effective temperature range as the asteroseismic stars, the age discrepancy
between the two sets of models can be as large as 1-2 billion years.
The MIST isochrones lie above the BaSTI isochrones on the HR diagram, leading
to a systematic underprediction of ages.
}

\racomment{
The gyrochronal ages, where gyrochronology is applicable, do not show
excellent agreement with the asteroseismic ages either.
The four hot stars to the left in figure \ref{fig:astero} are rotating more
slowly than predicted by the Praesepe-based gyrochronology models, and as a
result their gyro-ages are older than their asteroseismic ones.
For these hot stars, two out of four have ages that are still consistent, or
close to consistent, with their asteroseismic ages.
The third star from the left is an anomalously slow rotator for its age and
mass and \citet{vansaders2016} also found this star to be surprisingly slowly
rotating.
In contrast, the star with a black triangle symbol (indicating that
gyrochronology is applicable) furthest to the right is {\it rapidly} rotating
for its mass and age, even when weakened braking is taken into account.
% Both stars are around Solar mass (1.06 \pm 0.02 and 1.0 \pm 0.03 M$_\odot$
% from left to right) and have asteroseismic ages older than the Sun (6.82 \pm
% 0.28 and 7.28 \pm 0.51 Gyr from left to right), and yet both rotate more
% rapidly than the Sun (23.2 \pm 7.4).
This star is around Solar mass (1.0 $\pm$ 0.03 M$_\odot$) and has an
asteroseismic age older than the Sun (7.28 $\pm$ 0.51 Gyr), yet rotates with a
period of only 19.8 $\pm$ 1.3 days.
This is KIC 9098294, a single-lined spectroscopic binary with an orbital
period of around 20 days (Latham, private communication).
It is the only clear SB1 in the \citet{vansaders2016} sample, although some
others do have binary companions with long orbital periods, for which tidal
interactions are not expected to be strong.
}

\begin{figure}
    \caption{ A comparison of stellar ages inferred using
    asteroseismic modeling with ages inferred using a combination of isochrone
    fitting and gyrochronology.
Colored circles show ages inferred using isochrone fitting and gyrochronology
combined via the \sd\ software package.
Black triangles show the ages of all stars inferred via gyrochronology only
    and white circles show ages inferred via isochrone-fitting only.
% This plot shows that, for the majority of stars in this sample, isochrone
%     fitting dominates the age information and gyrochronology only contributes
%     significantly for two or three stars.
% This is a consequence of many of these stars being hot old, extremely metal
%     rich, extremely metal poor, or evolved.
  }
  \centering
    \includegraphics[width=1.2\textwidth]{asteroseismic_results_nosdss_gaia}
\label{fig:astero}
\end{figure}


\section{Algorithms}

\subsection{Alpha-beta negamax}

\textbf{Minimax, introduction}

\vspace{0.5em}

In original Minimax algorithm, one of the players (white) maximizes, another
(black) minimizes. Score is calculated with respect to both players. The value
could be either +high, 0 or --low. For white player, the higher score is the
better. On the contrary, for black the lower score is the better. Such
implementation is rather verbose, therefore we propose concise Negamax variant.

\vspace{0.5em}

\textbf{Negamax, leaf node}

\vspace{0.5em}

This paragraph describes how board score is determined and passed within
decision tree with respect to Minimax algorithm. Negamax deals with both
players equally, both maximize. To achieve such behavior we should properly
adjust score at the leaf node and pass obtained score in a specific way.

\vspace{0.5em}

Lets proceed with example. Suppose evaluation function returns +1 if white
player wins, --1 if black player wins, otherwise 0. White player checks
winning move and calls Negamax with deep 0. Black player recognize that the
game is over and returns +1. White player goes through all returned values and
takes maximum value +1 if it appears.

\vspace{0.5em}

Next situation, the same evaluation function, but black player checks winning
move and neutral move (neither winning nor loosing) and calls Negamax. For the
first move the board is evaluated to --1, and for the second to 0. Maximum of
two values is 0, so black would skip the best option if maximizes. We should
have inverted obtained score at the leaf node before the value is returned.
Then 0 becomes 0, but --1 becomes +1. Smaller values at the leaf nodes become
larger values when returned.

\vspace{0.5em}

All in all, black active player (or white opponent) at a terminal node returns
+score, white active player (or black opponent) at a terminal node returns
--score.

\vspace{0.5em}

\textbf{Negamax, internal node}

\vspace{0.5em}

Suppose, there is a white player from the previous paragraph. It found winning
move at the leaf node. Let black player be above white player within the
decision tree. If white player returns +1 (maximum possible value) and black
maximizes, then black would choose worst possible value and eventually move.
The best values returned from nodes below should be represented as the worst
values for the opponent. Therefore, before value is returned, it should be
inverted (* --1), so that the player above could maximize.

\vspace{0.5em}

\textbf{Negamax, monkey jumps}

\vspace{0.5em}

Another complication is that monkey jump does change recursion depth, but
does not change the color of the player. Therefore, we should consider
returning +/-- score based on the color of active player in the predecessor
game.

\vspace{0.5em}

\textbf{Alpha-beta pruning}

\vspace{0.5em}

This is a standard fail-hard implementation of the algorithm. $\alpha$ is
the best possible solution found so far, $\beta$ is the best possible solution
that the opponent could ensure.

\vspace{0.5em}

\subsection{Hash table}

Hash table is used for identifying transposed boards. Such board could appear
in case the board returns to the original state after a cycle of moves. Hash
table enables an algorithm traversing game tree to recognize such boards and
use already precalculated best score and move. The hashed cell is recognized
as hash hit in case current board has equal or smaller sub-tree in terms of
recursion depth than the board found in the hash. Otherwise, the current board
traverses all valid moves leading to the recursion call. Internally, the hash
table has $2^{18}$ possible cells. Table is accessed by several threads
concurrently, therefore locks are necessary for any read/write operations.

\vspace{0.5em}

Hash eviction policy is implemented based on the board equality. Once the best
move is found in Negamax, the method \textsf{.SetSolution(...)} is called on
the table instance. The board, move and score are stored in case boards are
not equal or better solution is found. Board equality is an efficient
operation, it compares 2 \textsf{ulong} values and 7 \textsf{uint} values.

\vspace{0.5em}

The hash cell is determined based on a board hash value. The hash calculation
is based on a method proposed by Albert Lindsey Zobrist. First, pseudo-random
values are generated for each (color, piece, square) triple. Consider moving
white pawn from \textsf{A1} to \textsf{A3}. First we remove hash of triple
(white, pawn, A1) from the current board hash. Then we add no-piece to the
board hash (?, no-piece, A1). Then we remove no-piece from the board hash (?,
no-piece, A3). Finally, we add (white, pawn, A3) th the board hash. Initial
board hash is calculated via calling method \textsf{.InitHash(CongoBoard
board)} on the table instance. Initial hash will contain only hash addition
operations. All no-pieces (ground and river) are considered black. For each
triple (color, piece, square) retrieve random number and let new hash be
\textsf{hash := XOR(hash, number)}. Removing pieces is done via repeated
\textsf{XOR} application. New board hash is adjusted directly in Negamax
recursive procedure.

\subsection{Iterative deepening}

\vspace{0.5em}

After some considerations, this algorithm has not been implemented in favour
of the hash table.


%!tex root=./thesis.tex
\chapter*{Abstract}
\label{cha:abstract}
\addcontentsline{toc}{chapter}{Abstract}

In this thesis we present a new supervised learning approach to the astronomical
problem of radio cross-identification. We focus in particular on training our
methods using crowdsourced data from the Radio Galaxy Zoo, a citizen science
website that allows volunteers to cross-identify radio objects.

Cross-identification is the problem of matching of objects detected at one
wavelength with objects detected at another. This is a particularly difficult
problem for radio, as a large number of radio objects are active galactic
nuclei, supermassive black holes at the centre of galaxies emitting huge radio
jets. These jets can sprawl across the sky in complex ways as they interact with
their environment and in general have no clear relationship with their host
galaxy.

In this thesis we have cast the radio cross-identification problem into a
machine learning context, framing it as a object localisation problem that can
be solved using binary classification. Using this framework, we then classified
galaxies in the Chandra Deep Field South according to whether they contain an
active galactic nucleus. We used a combination of image features and
astronomical features to represent each galaxy. Image features were extracted
from images of the radio sky using a convolutional neural network.

We trained the classifier using label data from Radio Galaxy Zoo. We found that
a classifier trained on these non-expert labels performs similarly to a
classifier trained on expert labels, attaining balanced accuracies of $(87.17 \pm 0.90)\%$ and $(88.74 \pm 0.77)\%$ respectively.

We investigated multiple ways of handling noise and redundancy in the
crowdsourced labels, and applied the classification model from the Raykar et al.
(2010) paper \emph{Learning From Crowds} to the Radio Galaxy Zoo labels.
Comparing this to a simple classification model trained on the majority vote, we
found that the majority vote approach obtained the highest classification
accuracy.

Finally, we investigated applications of active learning to the radio
cross-identification problem, with a focus on the Radio Galaxy Zoo project. We
applied query-by-committee to the radio cross-identification problem, finding
that query-by-committee outperforms random selection (but not if the random
selection is class-balanced). We then suggested an experiment to investigate
applications of active learning to Radio Galaxy Zoo, and highlighted problems
with current active learning literature when applied to citizen science.


%!TEX root = ../PhD_thesis__Lilian_Besson

% ----------------------------------------------------------------------
\chapter{Improving Spectrum Usage of IoT Networks with Selfish MAB Learning}
\label{chapter:4}

\graphicspath{{2-Chapters/4-Chapter/Images/}}

\abstractStartChapter{}%
%
After detailing in Part~\ref{part:Introduction} the MAB model, we now come back to our main question of interest.
In this chapter, we focus on wireless networks following the hypotheses common to present and future Internet of Things (IoT) networks exposed in Chapter~\ref{chapter:1}.
Our goal is to show that the IoT devices can automatically learn to increase their battery life and their successful transmission rates, without changing anything on the IoT standard side.
%
We propose two models of IoT networks, composed of many independent IoT end-devices, that can use low-cost Reinforcement Learning (RL) algorithms in order to learn how to improve their spectrum access.
%
Decentralized RL for IoT lets the devices use acknowledgements sent back by their Base Station as a reward, instead of sensing feedback like for OSA.
We consider many independent ``dynamic'' devices, each communicating with a small probability at every instant.
Simulations show that dynamic devices can greatly improve their spectrum efficiency,
by using MAB algorithms like \UCB.
% in the case of a non-uniform distribution of the static devices in the channels
We also developed a proof-of-concept using USRP platforms, for a real-world validation of this approach.
% Numerical simulations justify the interest of the approach,
%  of the advantage of using MAB algorithms for the first model.
%
% On a simple example, we present in Section~\ref{sec:4:gnuradio} a demonstration that implements and validates the proposed approach, using real wireless radio hardware.
% The first model
% % without retransmission of packets,
% is interesting for its simplicity,
% and because considering no retransmission can improve the battery life of IoT devices, at the cost of a lower Quality of Service (QoS).
%
In a second step, we consider a second model where a dynamic device has to (try to) retransmit a message, in case of a failed first transmission, up-to a fixed number of retransmissions.
We compare heuristics based on \UCB, and simulations confirm that the non-naive heuristics also significantly improve the network efficiency.
% Interestingly, the simplest heuristic is found to be the most efficient one, as its simplicity implies a faster learning and convergence time.
%
% The second model is more interesting if the main target is an improvement of the QoS, rather than a longer battery life.

\minitocStartChapter{}

% ----------------------------------------------------------------------------
% \section{Motivations for Selfish MAB Learning for IoT Networks}
\section{Introduction and motivations for MAB learning for IoT Networks}
\label{sec:4:motivations}
% ----------------------------------------------------------------------

After the first chapters that presented the model of MAB, we go back to the initial problems studied in this thesis, and thus we focus on Internet of Things (IoT) networks.
%
As explained in the introduction in Chapter~\ref{chapter:1},
unlicensed bands are more and more used and considered for mobile and LAN (Local Area Network) and for Internet of Things communication standards.
%  (Wi-Fi, LTE-U), (IoT) standards for short-range (ZigBee, Z-Wave, Bluetooth) and long-range (LoRaWAN, SIGFOX, Ingenu, Weightless) communications \cite{Centenaro16}.
This heavy use of unlicensed bands, in particular with the expected exponential growth of the number of IoT devices, will cause performance drop, due to radio collisions that could even compromise IoT promises.

Efficient Medium Access Control (MAC) policies allow devices to avoid interfering traffic and can significantly reduce the spectrum contention problem in unlicensed bands.
As end-devices battery life is a key constraint of IoT networks,
and as IoT networks are decentralized, because the devices initiate transmissions,
this leads to IoT protocols using as low signaling overhead as possible and simple ALOHA-based mechanisms.
%
In this chapter, we analyze the performance of Multi-Armed Bandits (MAB) algorithms, that could be used in combination with a time-frequency slotted ALOHA-based protocol.
We highlight that even without changing anything on the level of IoT standards, our proposal is just an add-on capability that can be used on a unit-per-unit basis.
We consider the \UCB \cite{Auer02}, and the Thompson-Sampling (TS) algorithms \cite{Thompson33,AgrawalGoyal11,
Kaufmann12Thompson}, for the first model. For the demonstration as well as for the second model, without loss of generality, we preferred to focus on heuristics based on the simplest algorithm (\ie, \UCB), to give a clear presentation of the different ideas explored to solve the problem of learning in order to retransmit efficiently.

% As detailed in Chapter~\ref{chapter:1},
% MAB learning has already been proposed in Cognitive Radio (CR) \cite{Mitola99,Haykin05}, and in particular, for sensing-based Dynamic Spectrum Access (DSA) in licensed bands \cite{Jouini10}.
% For example,
% proof-of-concepts like \cite{kumar2016two} have proven the capability of such approaches on real radio signals for OSA,
% and \cite{Maghsudi16} shows how MAB learning can be applied for small cell management in licensed 5G networks.
% Some analysis on real radio measurements made for HF ionospheric channels have also proven that solutions based on MAB learning is appropriate and solves efficiently this kind of decision-making problems on real-world wireless signals \cite{Melian15}.
% Recent works show that stationary MAB algorithms work well to solve reinforcement learning models that represent accurately real-world radio problems.
% Recently, TS and \UCB{} algorithms have been used for improving the spectrum access in (unlicensed) Wi-Fi networks, for instance by \cite{Toldov16} or \cite{Wilhelmi19collaborative,Wilhelmi19potential}.
% % However, even with only one dynamic user using the learning algorithm, the background traffic or the traffic of the other devices is never really stationary or \iid{}.

We present in Section~\ref{sec:4:firstModel} how the MAB algorithms can be used in a unlicensed but frequency- and time-slotted IoT network.
Several devices are using bandit algorithms, and the assumptions made by the stochastic bandit algorithms are not satisfied: as several agents learn simultaneously and their activation processes are random, their behavior is not stationary.
As far as we know, we provide the first practical study to confirm robustness of the use of stochastic bandit algorithms for decision making in IoT networks with a large number of intelligent devices in the network, which makes the environment ``strongly not stationary''.
This specific context makes it very hard to give mathematical proofs of convergence and of efficiency of bandit algorithms (that is why we relax the hypothesis and only consider up-to $M \leq K$ players in Chapter~\ref{chapter:5}).
We then validate the model with a hardware implementation on real radio signals, detailed in Section~\ref{sec:4:gnuradio}.
%
We conclude this chapter by presenting in Section~\ref{sec:4:retransmissions} an extension of this model to take into account another aspect of the ALOHA protocol, that is the possibility for dynamic devices to retransmit their packets if the \emph{Ack} was not received.

\paragraph{Publications.}
%
This chapter is mainly based on our articles \cite{Bonnefoi17,Besson2018ICT,Besson2019WCNC,Bonnefoi2019WCNC,MoyBesson2019,MoyBesson2019Annales}.

% ----------------------------------------------------------------------------
\section[Selfish learning for many dynamic devices in an IoT network]{Selfish learning for many dynamic devices with low activation probabilities in an IoT network}
\label{sec:4:firstModel}
% ----------------------------------------------------------------------

% - ``Multi-Armed Bandit Learning in IoT Networks and non-stationary settings'', see https://hal.inria.fr/hal-01575419

\input{2-Chapters/4-Chapter/CrownCom_17/IoT_slotted.tex}

\newpage  % WARNING ?
% ----------------------------------------------------------------------------
% \section{Test-bed implementation of the first model for real-world validation}
\section{Proof-of-concept of our model for real-world validation}
\label{sec:4:gnuradio}
% ----------------------------------------------------------------------

% - ``MALIN: Multi-Arm bandit Learning for Iot Networks with GRC: A TestBed Implementation and Demonstration that Learning Helps'', demo at ICT, and the companion paper ``GNU Radio Implementation of MALIN: "Multi-Armed bandits Learning for Internet-of-things Network"'', see https://hal.inria.fr/hal-02006825

\input{2-Chapters/4-Chapter/IEEE_WCNC_2019__DemoICT/IEEE_WCNC_2019__DemoICT.tex}

\newpage  % WARNING ?
% ----------------------------------------------------------------------------
\section{Extending the model to account for retransmissions}
\label{sec:4:retransmissions}
% ----------------------------------------------------------------------

% - ``Upper-Confidence Bound for Channel Selection in LPWA Networks with Retransmissions'', see \texttt{https://perso.crans.org/besson/articles/BMBM\_\_IEEE\_WCNC\_2019.pdf}

\input{2-Chapters/4-Chapter/IEEE_WCNC__2019__Paper__BMBM/IEEE_WCNC__2019__Paper__BMBM.tex}

\newpage  % WARNING ?
% ----------------------------------------------------------------------------
\section{Conclusion -- Towards theoretical guarantees}
% \section{Conclusion}
\label{sec:4:conclusion}
% ----------------------------------------------------------------------

% % We give in this section conclusions about the two models presented above, one for long-life IoT devices (without retransmission) and for improved QoS devices (\ie, with retransmissions), and about our demonstration, as well as directions of future works, some of which are currently being studied by researchers of our team.

% % \subsection*{Conclusions}

% % ----------------------------------------------------------------------
% % \paragraph{Summary of the first model.}
% \label{sub:41:conclusion}

% In Section~\ref{sec:4:firstModel}, we proposed an evaluation of the performance of MAB learning algorithms in IoT networks,
% with a focus on the convergence of algorithms, in terms of successful transmission rates, when the proportion of intelligent dynamic devices changes.
% Concretely, increasing this probability allows to insert more devices in the same network, while maintaining a good Quality of Service.
% Similarly, if the number of devices remain constant, increasing the successful transmission rate directly extends the IoT devices battery life, as they suffer less from failed transmissions.
% We show that \UCB{} and TS have near-optimal performance, even when their underlying \iid{} assumption (see Chapter~\ref{chapter:2}) is violated by the presence of many ``intelligent'' end-devices which follow a random activation process.
% %
% This is both surprising and encouraging, it shows that applying bandit algorithms tailored for a stochastic model is still useful in broader settings.
% The fully \emph{decentralized} application of classic stochastic MAB algorithms are almost as efficient as the best possible centralized policy in this setting, after a short learning period, even though the dynamic devices \emph{cannot} communicate with each other, and \emph{do not} know the system parameters.

% % ----------------------------------------------------------------------
% % \paragraph{Conclusions taken from the demonstration.}
% \label{sub:42:conclusionFromDemonstration}

% We presented in Section~\ref{sec:4:gnuradio} a demonstration, showed in $2018$ at the ICT conference \cite{Besson2018ICT}, and further detailed in the companion paper \cite{Besson2019WCNC}.
% We gave all the necessary details on both the PHY and the MAC layer, as well as details on the User Interface developed for the demo.
% Results obtained in practice were discussed, to highlight the interest of using learning algorithms for radio online optimization problem, and especially multi-armed bandit learning algorithms.
% %
% By using such low-cost algorithms, we demonstrated empirically that a dynamically re-configurable device can learn on its own to favor a certain channel, if the environment traffic is not uniform between the $K$ different channels, by using the acknowledgement (\Ack) feedback sent from the base station.

% % ----------------------------------------------------------------------
% % \paragraph{Conclusion about this second model.}
% \label{sub:43:conclusion}

% In Section~\ref{sec:4:retransmissions}, we presented an extension of our model of LPWA networks based on an ALOHA protocol, slotted both in time and frequency.
% % , in which dynamic IoT devices can again use machine learning algorithms, to improve their Packet Loss Ratio (PLR) when accessing the network.
% If the priority is the Quality of Service, for instance with renewable energy capabilities, this second model is more appropriate.
% The main novelty of this model is to address the packet retransmissions upon radio collision, by using a Multi-Armed Bandit framework.
% We presented and evaluated several heuristics that try to learn how to transmit and retransmit in a smarter way, by using the \UCB{} algorithm for channel selection for first transmission, and different proposals based on \UCB{} for the retransmissions upon collisions.
% %
% We showed that incorporating learning for the transmission is needed to achieve optimal performance, with significant gain in terms of successful transmission rate in networks with a large number of devices (up-to $30\%$ in the example network).
% Our simulations show that each of our proposed heuristic outperforms a naive random access scheme.
% Surprisingly, the main take-away message is that a simple \UCB{} learning approach, that retransmit in the same channel, turns out to perform as well as more complicated heuristics.

% % ----------------------------------------------------------------------
% \subsection*{Future works}
% \label{sub:4:futureWorks}

% % Future works related to this chapter include the following directions.

% % \paragraph{Possible extensions of the first model.}

% The first model presented in Section~\ref{sec:4:firstModel} could easily be generalized with two probabilities $p_S$ and $p_D$, if static and dynamic devices have different transmission pattern, and less easily with one probability per device. Also, other emission pattern could be considered, instead of a Bernoulli process for each user.
% In this whole Chapter~\ref{chapter:4}, we prefer to consider that all devices have the same activation probability, to keep the notation as simple as possible.
% %
% Moreover, for sake of simplicity we supposed that all devices use the same standard.
% Future works could consider more realistic interference scenarios and IoT networks, with, \eg, non-slotted time, more than one base station etc.

% Another extension could be to not have a Bernoulli process (or any random process), but a fixed rate of transmission, \eg, one transmission a day.
% So additionally to deciding the channel for communication (\ie, \emph{where} to communicate), each device has to also decide \emph{when} to communicate.
% % This is another direction of research, that we will investigate in the future.
% However, this clearly leads to a much larger action space, as there are many time slots in one day (for example), and thus we believe that as soon as the action space becomes too large in this extension, the simple MAB-based learning approach could be no longer appropriate.
% It is well-known in the MAB literature that the larger the action space, the slower is the convergence speed of any stationary MAB algorithms.
% It could be interesting to study the possible application of \emph{contextual} MAB \cite{Li10,Luo18} or structured MAB \cite{Combes17} models and algorithms for this extension.

% % We will investigate this behavior in order to understand it better theoretically.
% % We will also experiment more with adversarial algorithms, to confirm that they work less efficiently than stochastic bandit algorithms in our non-stochastic setting.

% % \paragraph{Extensions of the demonstration.}

% Possible future extensions of our demonstration include the following points.
% We could consider more dynamic devices (\eg, $100$) but it would either cost more in terms of equipment, or in terms of software engineering to simulation more devices with the same card.
% We could also implement a real-world IoT communication protocol (like the LoRaWAN standard), which we prefer not to do as it would cost a significant effort of development.
% Finally, we could also study the interference in case of other gateways located nearby, and this could be done without needing a lot of new hardware (using one extra USRP card to simulate another gateway).
% %
% % Pub pour notre autre article
% % We are also interested in studying the possible gain of using a learning step when the transmission model follows ALOHA-like retransmissions, and this is presented in the next Section~\ref{sec:4:retransmissions}.

% % \paragraph{Possible extensions for the second model.}

% Finally, the utility and impact of the proposed approaches for LPWA networks motivates us to address several subjects as future works. Among them, the non-stationarity of the channel occupancy caused by the learning policy employed by the IoT devices.
% %
% For that end, modifications of MAB algorithms have been proposed, such as Sliding-Window-\UCB{} or Discounted-\UCB{} \cite{Garivier11UCBDiscount}
% or more recently M-\UCB{} \cite{CaoZhenKvetonXie18},
% or more recently GLR-\UCB{} \cite{Besson2019GLRT} which is presented in Chapter~\ref{chapter:6},
% that nevertheless have not been explored for the targeted problem.
% Chapter~\ref{chapter:6} is focusing on this direction, but we did not have enough time to explore the possible applications of MAB policies designed for non-stationary problems to the model with retransmissions presented in this Chapter.

% In order to validate our results in a realistic experimental setting and not only with simulations, future works include a hardware implementation of the analyzed models to complete our demonstration \cite{Besson2019WCNC}.
% Note that Julio César Mango-Vasquez is currently working on this direction, for the EPHYL project, in collaboration with Carlos Faouzi Bader at IETR and CentraleSupélec campus of Rennes.
% % and Christophe Moy at IETR and University Rennes 1.
% %
% A hardware demonstrator could be also benefit to study other settings by removing some hypotheses, for instance by studying a similar model in non-slotted time.

% \subsection*{Summary of this chapter}

We focused in this chapter on models of IoT networks, and we proposed to use classical stationary multi-armed bandit learning algorithms implemented in a selfish and decentralized manner by each of the dynamic devices in the IoT network.
We presented two models of wireless IoT networks, without relying on the feedback provided by spectrum sensing, and inspired by the ALOHA protocol. We proposed two versions, with or without retransmissions of up-link packages in case of collisions.
% In both cases, we try to model the existing standards, like the LoRa standard, and we demonstrated the efficiency of the proposed MAB-based approach in both numerical simulations and empirical measurements on real wireless radio signals.
%
We conclude that this learning-based approach is efficient, as it allows the IoT devices to automatically and independently increase their successful transmission rates.

It is also quite surprising that stochastic MAB algorithms can be of any use in such non-stationary applications.
Unfortunately, it turned out to be of extreme difficulty to analyze analytically the considered model with thousands of independent devices, all communicating and learning in their own (random) time scales.
That is why we focus on two different simplifications of this model in the next chapters, for which we are able to provide a rigorous theoretical analysis.
% %
% We can also highlight that the two models we studied are complementary:
% the first model (without retransmission of packets), is interesting for its simplicity, and focused on improving the battery life of IoT devices,
% % , at the cost of a lower Quality of Service (QoS),
% %
% while the second model is more interesting if the main target is an improvement of the Quality of Service (QoS).
% % , rather than a longer battery life.

\paragraph{Multi-Players MAB.}
%
On the one hand, we are actually able to analyze a simpler model, if we assume to have at most $M \leq K$ devices with a transmission probability of $p=1$.
Instead of the experiment-driven direction pursued in this chapter, another possibility is to consider a \emph{multi-players MAB} model to describe our problem.
%
The main difference between the two models is the fact that in Chapter~\ref{chapter:4}, $M \gg K$ devices transmit their messages at every time step, by following a random activation process (with a fixed transmission probability $p < 1$).
If static and dynamic devices that have to transmit at a fixed time are denoted \emph{active devices},
then their random activation patter makes the number of active devices an (unpredictable) random variable.
Analyzing multi-players MAB models under this hypothesis is much harder, and is left as a future work.
%
We study the case of $M \leq K$ devices learning independently to play a $K$-armed bandit in the next Chapter~\ref{chapter:5}.

% In that case, the static and dynamic devices effect is decoupled, and arms only model the availability of the channels in the absence of dynamic devices: they are \iid{} with mean $\mu_i = 1 - p S_i$.
% Moreover, dynamic devices are usually assumed to be able to \emph{sense} a channel before sending \cite{Zhao10}, and so communicate only if no static device is detected on the channel.
% The smart devices try to learn the arms with highest means, while coordinating to choose different arms, \ie, avoid collisions in their choice, in a decentralized manner.
% However, in this model it is assumed that the multiple agents can know that they experienced a collision with another agent, which is non-realistic for our problem at stake, as our model of smart device cannot do sensing nor differentiate collisions between smart and non-smart devices.

\paragraph{Non-stationary MAB.}
%
On the other hand, while it is hard to analyze the models of this chapter because of the unpredictable behaviors of the IoT devices' activation patterns and the evolving number of active devices, we are also able to analyze a simpler model, if we focus on a single player accessing a network which is assumed to be \emph{piece-wise stationary},
that are bandit problems which are stationary on ``long enough'' intervals, of unknown locations and lengths.
%
We then study this second direction in Chapter~\ref{chapter:6}.

\newpage
% ----------------------------------------------------------------------------
\section{Appendix}
\label{sec:4:appendix}

\subsection{Proof of Proposition~\ref{prop:41:Lagrangian}}
\label{sec:4:proofLagrangian}
% ----------------------------------------------------------------------------

% We include here a missing proof from Section~\ref{sec:4:firstModel}.

\input{2-Chapters/4-Chapter/CrownCom_17/IoT_slotted_appendix.tex}

\newpage
\subsection{Illustration of the GNU Radio Companion Flowcharts}
\label{sec:4:IllustrationFlowcharts}
% ----------------------------------------------------------------------------

\input{2-Chapters/4-Chapter/IEEE_WCNC_2019__DemoICT/IEEE_WCNC_2019__DemoICT_appendix.tex}


\documentclass[11pt]{article}

\usepackage{common}
\usepackage{booktabs}
\usepackage{wrapfig}
\usepackage{titling}
\usepackage{titlesec}
\usepackage{float}
\usepackage[margin=0.99in]{geometry}

\titlespacing\section{0pt}{4pt}{2pt}
\setlength{\parskip}{0.35em}
\setlength{\droptitle}{-5em}
\pagestyle{plain}

\title{Practical 2: Classification\\ Classifying Malicious Software}
\author{Antonio Copete, acopete@cfa.harvard.edu, Camelot.ai Copete \\
	Fangli Geng, fgeng@g.harvard.edu, Camelot.ai fangligeng\\
	Junran Yang, jyang3@gsd.harvard.edu, Camelot.ai junran}

\begin{document}
\maketitle{}

\section{Technical Approach}

The purpose of this practical was to classify a test set of 3,724 files in XML format into 15 classes of malware, using a training set of 3,086 labeled files. Our general approach consisted on the following steps:

\begin{enumerate} %[leftmargin=1cm]

\item \textbf{Feature Engineering}\\
Initial inspection showed the XML data files to be structured hierarchically into a series of \emph{processes}, in turn composed of a series of \emph{threads}, in turn composed of a series of \emph{system calls}. Each of these were identified by a \emph{tag} as well as a number of \emph{attributes} indicated as sets of key-value pairs. Our simplest approach began by drawing $n$-grams (ordered sets) of up to 4 consecutive system call tags, initially without attributes, and our initial exploratory analysis tested the hypothesis that the presence and order of such system call tags was correlated with the distinct anomalous behavior found in different types of malware.

More advanced versions of this approach found in the literature \cite{canali} suggested also drawing features based on $n$-bags (unordered non-consecutive sets) as well as $n$-tuples (ordered non-consecutive sets) of system calls and processes, as well as including attributes such as file names, file sizes, URLs, execution times, etc. We found some of these attributes to be oddly predictive of some specific classes, such as the file \verb|maxtrox.txt| being always related to malware of class VB, and \verb|GrayPigeon_Hacker.com.cn| always indicating malware of class Hupigon. In addition, we also took the number of total processes and thread numbers into account.

To analyze the system call tags (including 4-grams, 29,813 features in total) and predictive attributes (110,465 features in total), we tried 2 methods: using either the number of counts, or the TF-IDF (term frequency---inverse document frequency) \cite{canzanese}. The TF-IDF value measures the relative frequency of a token across the data set; it is proportional to the number of times a token appears in a given file, offset by the frequency of the token across all files, which helps adjust for the absolute frequency of a given token in the whole dataset. The scikit-learn module \verb|sklearn.feature_extraction.text| includes the objects \verb|CountVectorizer| and \verb|TfidfVectorizer|, which tokenize large strings of tags (with or without arguments) into $n$-grams, and then transform them into counts and TF-IDF values in the resulting feature matrix, respectively.

\item \textbf{Feature Selection}\\
Since we derived roughly 140,000 features from a training set of just over 3,000 files, it was easy to cause our model to overfit the data. To avoid overfitting, we used the sklearn module \verb|SelectFromModel| and the classifier \verb|sklearn.ensemble.RandomForestClassifier| to reduce features and use cross-validation to determine the features we wanted to keep. More specifically, we ran a loop to continuously drop features with importance of less than 10\% the mean importance among all features, monitoring the mean score from 5-fold cross-validation in each iteration. The highest score happened with around 5,000 features, which was 1/20 of the original feature size.
% Antonio: I also dropped features with low variance across both the training and test set (unsupervised feature selection), and used a similar strategy to drop features according to RF importance. I went more conservatively and dropped only features that had importance = 0 in all 5 cross-validation sets. But Fangli's strategy is good, so we can leave it at that.

\item\textbf{Model Selection}\\
We began by exploring a number of classifiers for model selection from scikit-learn, including \verb|RandomForestClassifier| (Random Forest), \verb|svm.SVC| and \verb|svm.LinearSVC| (Support Vector), and \verb|linear_model.SGDClassifier| (Stochastic Gradient Descent), with default hyper-parameters. The Random Forest classifier yielded the best score of 0.78105 on the test data, compared to 0.57211 for SVC, 0.76789 for Linear SVC, and 0.77158 for SGD. We thus selected Random Forest as our main model and proceeded to tune it in the subsequent steps.

In addition, we found the class distribution in the training set to be heavily imbalanced. We therefore took a sequential approach to model training, dividing the data into 4 main categories: None (52.14\%), Swizzor (17.56\%), VB (12.18\%) and Others (18.12\%). We first trained the model on each of first 3 categories, and then separately on the minor categories. This approach resulted in an improvement of 0.02 in the test score, compared to training on all categories jointly (0.81211 vs 0.79263). We tried the same approach on a Neural Network for comparison, which yielded a score of 0.79105 for training on separate categories, and confirming again Random Forest as the better model to choose.

Late in the process, we also found a bug in the original code we were provided for extracting system call tags, which was causing the tags for every other thread to be skipped altogether. Our final results are based on the corrected version of feature extraction, but due to time constraints we limited ourselves to 4-gram tokens of system call tags without attributes. With the corrected feature set and a Random Forest classifier trained jointly across all categories, we obtained an improvement of 0.015 in the test data score (0.80789 vs 0.79263).

In our last attempt at model experimentation, we were inspired by a Microsoft Malware Classification Challenge (BIG2015)\footnote{https://www.kaggle.com/c/malware-classification/discussion/13897} to utilize xgboost and a semi-supervised method to detect malware from assembly code. Gradient Boosting is different from Random Forest in that it focuses more on reducing the bias rather than variance. After grid searching for hyperparameters, the highest accuracy we got by xgboost was 0.82526. Furthermore, we tried semi-supervised Gradient Boosting, aiming to further reduce the bias of the model and incorporating information from test dataset into training dataset. We first generate pseudo labels for the test set by selecting the most likely classification based on our best modeling of the training set alone. Then we predict on the test set again in a cross-validation fashion using both the training data and test data.

As an example, consider a test set that is split into 4 folds: A, B, C and D. We draw a new training set that is composed of test folds A, B and C with their pseudo labels, in addition to the original training set, and we then predict on test set fold D. The same method is used to predict on A, B and C. However, this approach didn't provide us with decent results. We suspect the relatively low accuracy in the prediction set (0.82526) provided more inaccurate information than that it could offer to reduce bias.

\end{enumerate}

\section{Results}

\begin{table}[htbp]
\centering
\begin{tabular}{cclccccc}
 \toprule
 Rank & \multicolumn{1}{p{1cm}}{\centering Max Depth} & \multicolumn{1}{p{0.8cm}}{\centering $\eta$} &
 \multicolumn{1}{p{2cm}}{\centering Min. Child Weight} & colsample & \multicolumn{1}{p{2cm}}{\centering CV Mean Error} &
\multicolumn{1}{p{2cm}}{\centering Best Num. Round} & Final Score\\
 \midrule
  1 & 4 & 0.15 & 1 & 0.5 & 0.093332 & 49 & 0.82158\\
  2 & 6 & 0.2 & 1 & 1 & 0.093334 & 25 & 0.811054\\
  3 & 4 & 0.1 & 1 & 0.5 & 0.093983 & 68 & 0.82526\\
  4 & 4 & 0.15 & 2 & 0.5 & 0.094306 & 54 & - - -\\
  5 & 6 & 0.15 & 1 & 1 & 0.094953 & 31 & - - -\\
 \bottomrule
\end{tabular}
\caption{\label{tab:results} Result tables for the best 5 cross-validation scores by Gradient Boosting with grid search}
\end{table}

Table \ref{tab:results}  shows the best 5 cross-validation scores by Gradient Boosting with grid search. The hyperparameters we tuned were \verb|max_depth| (the maximum depth of a tree, same definition as GBM, set as [2, 4, 6]), $\eta$ (analogous to learning rate in GBM, set as [0.05, 0.1, 0.15, 0.2]),  \verb|min_child_weight| (minimum sum of weights of all observations required in a child, set as [1, 2]), \verb|colsample_bytree| (denotes the subsample ratio of columns for each split, in each level, set as [0.5, 1]). Those 48 possible configurations were drawn from the ones people use most commonly. We also tested models with higher and lower round numbers, optimizing the result by cross-validation.

Figures \ref{fig:results_1}--\ref{fig:results_3} show the results of the tuning process for the Random Forest model for 2 hyperparameters: the maximum tree depth \verb|max_depth|, as well as the number of estimators (trees) \verb|n_estimators|. Results are shown for a validation set (blue) drawn from 20\% of the training data in a stratified fashion (i.e. preserving class proportions), as well as for the remaining training data (red).

\begin{figure}[H]
\minipage{0.48\textwidth}
  \includegraphics[width=\linewidth]{hyperparameter_tuning_1}
  \caption{Max depth range from 10 to 235, and number of estimators range from 10 to 460.}
  \label{fig:results_1}
\endminipage\hfill
\minipage{0.48\textwidth}
  \includegraphics[width=\linewidth]{hyperparameter_tuning_2}
  \caption{Max depth range from 10 to 45, and number of estimators range from 10 to 460.}
  \label{fig:results_2}
\endminipage\hfill
\minipage{1\textwidth}
  \begin{center}
  \includegraphics[width=0.48\linewidth]{hyperparameter_tuning_3}
  \caption{Max depth range from 40 to 49, and number of estimators range from 400 to 490.}
  \label{fig:results_3}
  \end{center}
\endminipage
\end{figure}

We first ran the experiment in a rough region (Figure~\ref{fig:results_1}): \verb|max_depth| range from 10 to 235 with a step of 25, while \verb|n_estimators| range from 10 to 460 with a step of 50. From this experiment we found out that when depth is larger than 60, the increasing depth of the decision tree no longer contributes to train score or test score. That is because we only have a training set of $\sim3,000$ samples, and making the tree deeper than the square root of number of samples does not contribute significantly to the accuracy of the decision trees. Then we ran 2 experiments with finer sampling (Figures~\ref{fig:results_2} and \ref{fig:results_3}). We found out from the results that we should make the depth as close to the square root of number of samples as possible, while choosing the number of estimators to be around 400, since larger numbers no longer improve the performance of the model significantly.

\section{Discussion}

Our general strategy focused first on feature generation, first tokenizing the data files into tags of system calls and processes ---both with and without attributes---, and then drawing features based on $n$-grams (ordered consecutive sets) of tokens, and later generalizing to $n$-bags (unordered non-consecutive sets) and $n$-tuples (ordered non-consecutive sets). Different values of $n$ (1, 2, 3, 4, and 10) were considered in the process. The features themselves were either the absolute number of occurrences of the given subsets of tokens, or their TF-IDF values to express them as a proportion of their overall occurrence. While some attributes such as certain file names proved to be of some predictive value, in the end we obtained our best results from 4-grams of system call tags.

Our model selection process considered several classifiers, including Random Forest, C-Support Vector, Linear-Support Vector and Stochastic Gradient Descent, which we tried on different subsets of the training data, and evaluating them using 5-fold cross-validation. The experimentation process showed Random Forest to be the most robust model, and we proceeded to tune its hyperparameters (number of trees in the model and their maximum depth), in order to achieve the best possible results. We also performed feature reduction by iteratively discarding features of low importance and monitoring the cross-validated scores, resulting in an optimal set of $\sim5,000$ features from an initial set of $\sim140,000$ features. One further refinement to our approach consisted in using a 2-step process in training our model, first training on 4 broad categories to predict on the 3 most frequent classes first, and then predicting separately on the 12 remaining minor classes.

Our best results on the Camelot public leaderboard were a score of 0.81632 obtained from the tuned Random Forest classifier, and then lately we improved on it by experimenting with a semi-supervised Gradient Boosting model which gave a score of 0.82263. For the final leaderboard scored on the private dataset, our best score was 0.81798, which placed us among the top 15 scores. Our placement turned out to be much better for the private dataset than for the public one, which is evidence of the robustness of our model that did not overfit the data as much as it did for other teams.

As for further improvements to our model with more time for this practical, first we would have done further experimentation with features beyond $n$-grams of system calls ---such as $n$-bags, $n$-tuples, and combinations thereof---, to result in more complex sets of features. Also, given the promise of the semi-supervised Gradient Boosting approach we discovered late in the game, we would have done further tuning of model parameters, as well as training it separately on different categories of data in a 2-step process, as we did with Random Forest.

\newpage
\nocite{rozenberg}
\bibliographystyle{apalike}
\bibliography{P2_dualspace}

\end{document}


\chapter{Mass Relations and Stoichiometry}
\section{Atomic Mass}
To determine the weight of an atom, scientists developed a small unit at which
such quantities may be measured.  This unit is know as the AMU, or
\textit{atomic mass unit}.

12 AMUs are defined as the weight of one atom of the Carbon-12 isotope.  That is
to say that $1amu$ is exactly the weight of 1/12th of one atom of Carbon-12.

\subsection{Masses of Elements}
However, the numbers on the periodic table are not so round?  Why might this be?
Well, since we are accounting for isotopes, we must apply a weighted average to
all naturally occuring isotopes.  For example:

\begin{table}[h]
\begin{tabular}{lll}
\hline
          & Abundance & \textit{Mass (amu)} \\ \hline
oxygen-16 & 99.762\%  & 15.994              \\
oxygen-17 & 0.038\%   & 16.999              \\
oxygen-18 & 0.200\%   & 17.999
\end{tabular}
\end{table}

The calculation for the relative \textit{amu} of Oxygen is as follows:
$(0.99762*15.994) + (0.00038*16.999) + (0.00200*17.999) = 15.9994amu$

\section{The Mole}
Avagadro's Number (or $N_A$) is thought of a quantity, usually of atoms.  The
number of that quantity is $6.022e23$.  Therefore, since this is a quantity, the
unit is $anything/mole$.

\subsection{Molar Mass}
\subsubsection{Of Elements}
To calculate the molar mass of an element: simply look to the periodic table!
In the upper-left of the element box, you will find the relative molar mass of
the element.  That is to say, the \textit{amu} of an element is the weight in
grams of one mole of that element.  For example, $1.008mol$ of \ce{H} weighs
\ce{1g}.

\subsubsection{Of Compounds}
Simply put, the molar mass of a compound is the sum of all its parts, multiplied
individiually by each of the relative quantities.

\section{Percent Composition}
\textbf{Percent composition} of an element is the percent by mass of itself
within a compound.  For example, the molar mass of \ce{Cu2S} is 159.2.  The mass
amount of \ce{Cu}

\subsection{Empirical Formulae}
An \textit{emperical formula} consists of the sumbols for the elements combined
in a compound, with subscripts showing the smallest whole-number mole ratio of
the different atoms in the compound.

For example, the emperical formula \ce{P2O5} might lead to a chemical formula of
\ce{P4O10}, or similarily, the emp. formula \ce{C3H6O} may lead to the formula
\ce{C6H18O2}.

\subsubsection{Calculating}
To calculate an emperical formula, you must first start with the percentages of
an atom or the mass of the atom present.  Once these are determined, you may
then convert to moles, and simplify the ratio by dividing by the smallest amount
and rounding.  If you cannot round cleanly, multiple the whole ratio by a
factor.

The steps for calculating this emperical formula follow below:

\begin{enumerate}
  \item Assume $100g$s was sampled, unless given other measurements.
  \item Convert the grams to moles, unless given moles.
  \item Simplify the ratio of moles by dividing by the smallest term.
  \item Multiply by factors to arrive at cleaner ratios, if not already clean.
\end{enumerate}

\section{Stoichiometry}
Stoichiometry involves the mass relationships between reactants and products in
a chemical reaction.  IT allows you to determine the amount of additional
reactants or products given the chemical equation and just one amount.

Stoichiometry relies on the ratios of moles between different compounds in a
chemical reaction.  Consider the following reaction: \ce{Al2(SO4)3 + 3Ca(OH)2 ->
2Al(OH)3 + 3CaSO4}.  It has the following mass relations:

\begin{frame}
  \centering
  {\huge $\frac{1mol \ce{Al2(SO4)3}}{3mol \ce{Ca(OH)2}}$, $\frac{1mol
  \ce{Al2(SO4)3}}{2mol \ce{Al(OH)3}}$, $\frac{1mol \ce{Al2(SO4)3}}{3mol
  \ce{CaSO4}}$}
\end{frame}

To work with these mass relations, you can convert from mass $\rightarrow$ molar
mass $\rightarrow$ moles $\rightarrow$ mole-to-mole ratio $\rightarrow$ moles
$\rightarrow$ molar mass $\rightarrow$ mass.

\subsection{Limiting Reagent}
Sometimes when doing stoichometry, one reagent appears in a limiting amount such
that you cannot fully complete the reaction.  These substances are known as
\textit{limiting reagents}.  The substances that is not completely used up in a
reaction is referred to as an \textit{excess reagent (reactant)}.

To determine the limiting reagent, just do the reaction with both against one of
the products and figure out which makes \textit{less}.  The excess reagent will
be what is left over.  To determine the quantity of the excess reagent, use the
quantity of the limiting reagent that you originally plugged in and wire that
through.

\subsubsection{Percent Yield Calculations}
Sometimes, you are given an amount of which is supposed to occur (via
stoichiometry) and an amount that you actually produced.  To calculate this, you
use the following:

{\huge $100 * \frac{actual}{theoretical}$}


\documentclass[a4paper]{article}
%\usepackage[ngerman]{babel}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{textcomp}
\usepackage{geometry}
\geometry{ left=2cm, right=2cm, top=2cm, bottom=3cm, bindingoffset=5mm}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{longtable}
\usepackage{amstext}
\usepackage{array}
\usepackage{amsmath}
\newcolumntype{L}{>{$}l<{$}}
\usepackage{tabularx, ragged2e}
\usepackage{helvet}
\renewcommand{\familydefault}{\sfdefault}
\usepackage{lastpage}
\usepackage{todonotes}
\usepackage{titlesec}
\titleformat*{\section}{\large\bfseries}
\usepackage{listings}
\usepackage{color}

\usepackage{tikz}
%\newcommand{\tikzmark}[2]{\tikz[overlay, remember picture] \node[inner sep=0pt, outer sep=0pt, anchor=base] (#1) {#2};}
\usetikzlibrary{tikzmark}

\definecolor{mygreen}{rgb}{0.18, 0.545, 0.341}
\definecolor{mygray}{rgb}{0.5,0.5,0.5}
\definecolor{myblue}{rgb}{0.53,0.61,0.85}

\lstset{
 keywordstyle=\color{mygreen},
 commentstyle=\color{mygray},
 numbers=left,
 numbersep=5pt,
 numberstyle=\scriptsize\color{mygray}
 }

\usepackage{amsmath,amssymb}

\DeclareRobustCommand{\bbone}{\text{\usefont{U}{bbold}{m}{n}1}}

\DeclareMathOperator{\EX}{\mathbb{E}}% expected value

\date{}
\author{}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\fancyhead[R]{Felix Burk\\ Pascal Huszár}
\fancyhead[L]{Reinforcment Learning \\ Summer Term 2021 }
\fancyfoot[R]{page \thepage \text{ }/ \pageref*{LastPage}}
%\fancyfoot[LE]{Seite \thepage \text{ }von \pageref{LastPage}}
\renewcommand{\headrulewidth}{0.5pt}

\usepackage{amsmath}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\title{\textbf{Exercise 09}}

\begin{document}
	\maketitle
	\thispagestyle{fancy}

    \section*{Task 1 - REINFORCE on the Cart-Pole}
    \subsection*{a)}
    $\frac{\delta}{\delta \theta_a}\frac{se^{s*\theta_a^T+s*\theta_b^T)}}{(e^{\theta_a^T*s}+e^{\theta_b^T*s})^2}$
 	\subsection*{b)}
 	$\pi(a|s,\theta) = \frac{e^{\theta_a^T*s}}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$ where $b,c$ are all possible actions and $a \in \{b,c\}$ \\
 	$\nabla_\theta log(\pi_\theta(s,a)) = \nabla_\theta log(\frac{e^{\theta_a^T*s}}{e^{\theta_b^T*s}+e^{\theta_c^T*s}})$ \\
 	$= \nabla_\theta log(e^{\theta_a^T*s})-\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})$ \\
 	$= \nabla_\theta \theta_a^T*s*log(e)-\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})$ \\
 	$= s -\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})$
 	$= s -\frac{\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$ \\
 	$= s - \frac{e^{\theta_b^T*s}}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$
 	$= \frac{e^{\theta_c^T*s}*s}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$
 	$= \frac{1}{1+e^{\theta_c^T*s}}*s$ \\
 	For the case $a=c$, $a=b$ is equivalent.

\end{document}


\section{Encryption}\label{s:encryption}
The fields of cryptography, privacy, computer and information security, design and utilize software, hardware, and human resources to address the issue of sensitive information and private communication.
One of the most commonly used ways of preserving data privacy is the use of encryption. Encryption is the process of encoding a message (\textit{plaintext}), such that it is accessible only to authorized parties, and no one else.
This is accomplished using an encryption algorithm (\textit{cipher}) that operates on the plaintext and produces a \textit{ciphertext}.
The ciphertext needs to be decrypted in order to be read and a decryption algorithm is responsible for this operation.
Both algorithms use a \textit{key} in order to operate on the plaintext.
The distribution of the key(s) ensures the authorization of the parties involved.

Ultimately, the basic building blocks of an encryption scheme come down to three discrete components.
These are the \textit{Key Generation} mechanism and the \textit{Encryption} and \textit{Decryption} algorithms.