\section{Results}
\label{section:results}

In order to demonstrate the performance of our method, we conducted three sets
of tests.
In the first we simulated observables from a set of stellar parameters for a
few hundred stars using the MIST stellar evolution models and the
gyrochronology model of equation \ref{eqn:gyro}.
The ages predicted with our model were compared to the true parameters used to
generate the data.
In the second we tested our model by measuring the ages of individual stars in
the NGC 6819 open cluster, and in the third we tested our model on \kepler\
asteroseismic stars.

\subsection{Test 1: simulated stars}
For the first test we drew masses, ages, bulk metallicities, distances and
extinctions at random for 1000 stars from the following uniform distributions:
\begin{eqnarray}
& \mathrm{EEP} \sim U(198, 480) \\
% & M \sim U(0.5, 1.5)~[M_\odot] \\
& t \sim U(0.5, 14)\mathrm{~[Gyr]} \\
& [Fe/H] \sim U(-0.2, 0.2) \\
& D \sim U(10, 1000)~\mathrm{[pc]} \\
& A_V \sim U(0, 0.1).
\end{eqnarray}
\teff, \logg, \fhat, parallax, and apparent magnitudes $B$, $V$, $J$, $H$, $K$,
\gaia\ $G$, $G_{BP}$ and $G_{RP}$ were generated from these
stellar parameters using the MIST stellar evolution models.
We added a small amount of noise to the `observed' stellar properties in order
to reflect optimistic observational uncertainties for isochrone-dating.
We added Gaussian noise with a standard deviation of 25 K to \teff, 0.01 dex
to \feh\ and \logg, and 10 mmags to $B$, $V$, $J$, $H$, and $K$ magnitudes.
These are just one choice of uncertainties that we could have adopted and are
extremely optimistic.
The uncertainties on predicted ages {\it will} depend strongly on all
observational uncertainties, however, since this analysis is designed to show
the relative improvement in stellar age precision when {\it rotation periods}
are included, we chose to use best-case spectroscopic parameters.
The noise added to Gaia $G$-band photometry ranged from
0.3 mmag for stars brighter than 13th magnitude, to 10 mmag for stars
around 20th magnitude \citep{evans2017, brown2018}.
Noise added to \gaia\ $G_{BP}$ and $G_{RP}$ bands ranged from 2 mmag for stars
brighter than 13th magnitude to 200 mmag for stars fainter than 17th.
Unphysical combinations of stellar parameters were discarded, resulting in a
final sample size of 841 simulated stars.
Figure \ref{fig:CMD_age} shows the position of these stars on an HRD
(with \logg\ on the y-axis instead of luminosity to improve the visibility of
the MS), colored by their age.
Rotation periods for these stars were generated using the gyrochronology
relation described in equation \ref{eqn:gyro}.
We added 5\% Gaussian noise to all stellar rotation periods to represent
realistic measurement uncertainties of 5\%.
The median uncertainty on rotation periods calculated from \kepler\ light
curves, provided in the \citet{mcquillan2014} catalog is 1\%.
However, the \citet{aigrain2015} injection and recovery study showed that true
rotation period uncertainties are often slightly larger than this, and noise
distribution of rotation periods can be highly non-Gaussian
\citep[\eg][]{aigrain2015, angus2018}.
\begin{figure}
  \caption{
      The simulated star sample plotted on an HRD, colored by age
    (top panel) and rotation period (bottom panel).
    HRD positions were calculated using MIST isochrones via the {\tt
    isochrones.py} {\it Python} package and rotation periods were generated
    using equation \ref{eqn:gyro}.
    This figure was generated in a Jupyter notebook available at
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Simulate_data.ipynb}
}
  \centering
    \includegraphics[width=1\textwidth]{simulated_CMD}
\label{fig:CMD_age}
\end{figure}
Figure \ref{fig:rotation_model} shows the rotation periods of 841
stars generated from the gyrochronology model.
\begin{figure}
  \caption{
Data simulated from the rotation period model.
    Late F, GK and early M dwarfs (stars with 0.56 $<$ \gcolor\ $<$ 2.7 follow
    the Praesepe-calibrated gyrochronology relation (dashed gray lines), with
    the exception of old, slowly
    rotating stars with large Rossby numbers whose rotation periods are fixed
    at 2$\times$ their convective overturn time.
    The rotation periods of early F (\gcolor $<$ 0.56), late M dwarfs (\gcolor
    $<$ 2.7) and subgiants (EEP $\gtrsim$ 420) were generated
    from a log-normal distribution with standard deviation given by equation
    \ref{eqn:gyro}.
% The top panel shows the rotation periods vs. B-V colors of simulated stars,
The top panel shows the rotation periods vs. \gcolor\ colors of simulated stars,
    colored by their age and the bottom panel shows the same stars colored
    by their equivalent evolutionary phase (EEP).
    % 9, 11, and 13 (rotation periods rise with age).
    The gray lines describe the mean gyrochronology model at ages 1,
    3, 5, 7, 9, 11, and 13 (rotation periods rise with age).
    This figure was generated in a Jupyter Notebook, available at
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Simulate_data.ipynb}
}
  \centering
    \includegraphics[width=1.\textwidth]{rotation_model_praesepe}
\label{fig:rotation_model}
\end{figure}

We took two approaches to inferring the ages of these simulated stars:
firstly using isochrone fitting {\it only}, and secondly using isochrone
fitting {\it combined with} a gyrochronology model (\sd).
Since the posterior PDFs of stars are often multimodal, we found that the
choice of initial positions of the {\tt emcee} walkers influenced the final
outcome because walkers occasionally got stuck in local minima.
We found that the following set of initial parameters worked well, though not
perfectly: EEP = 330, $t = 9.56$ Gyr, $[M/H] = -0.05$, $D = 269$ pc and $A_V =
0.0$\footnote{These are the default initial parameters provided in \sd.}.
Figure \ref{fig:simulation_results} shows the results of combining
gyrochronology with isochrone fitting for the simulated sample.
The stars' true ages are plotted against their predicted ages, with ages
inferred with gyrochronology and isochrone fitting
in color, and ages inferred using isochrone fitting only plotted in light
grey.
The different panels show the results for different types of stars: FGK
dwarfs that are still undergoing magnetic braking, FGK dwarfs that have ceased
magnetic braking (their Rossby number is around 2), M dwarfs, and evolved
stars.
The selection criteria for these groups are in the panel headings.
The FGK dwarfs with low Rossby numbers showed the largest improvement: the
median age precision for this group (defined as the standard deviation of the
posterior as a percentage of the median age) was 8\% when using a combination
of isochrones and gyrochronology, and 22\% using isochrone fitting alone.
This equates to an almost 3$\times$ improvement in age precision.
The age RMS of this group was 0.8 Gyr using isochrones and gyrochronology, and
2 Gyr using isochrones only.
Despite the fact that stars with Rossby numbers of 2 have stopped spinning
down and their rotation periods no longer evolve with age, the ages of these
stars can still be relatively precisely constrained with isochrone fitting.
The median age precision for stars with large Rossby numbers was 11\% with
gyrochronology and isochrone fitting, and 13\% with isochrone fitting only.
% This is because their rotation periods are still directly related to their
% age, but also {\it indirectly} related to their age via EEP and metallicity,
% as shown in equation \ref{eqn:gyro}.
% When including rotation periods to infer the ages of stars with large Rossby
% numbers, age precision improves from 21\% to 11\% and the age RMS improves
% from 2.5 to 0.9 Gyr.
% Age precision also increases for F dwarfs when incorporating their rotation
% periods, even though they are not directly related to their ages.
% This is because rotation period relates to color via the variance model in
% equation \ref{eqn:gyro}.
The precision of M dwarf ages did improve overall when their rotation periods
were included, but this improvement was entirely driven by the early M dwarfs.
The precision of this group improved overall from 33\% to 22\% and RMS from
5.4 Gyr to 4.6 Gyr.
The precision of ages inferred for evolved stars changed very little when
rotation periods were included in the inference process.
This is because the variance of the rotation-age relation was inflated by a
large amount in the gyrochronology model (equation \ref{eqn:gyro}), making
rotation periods almost entirely uninformative for this group.
The median age precision of subgiants from both gyrochronology and isochrone
fitting and isochrone fitting alone was 7\%.

\begin{figure}
  \caption{
The true vs. predicted ages of simulated stars.
    Ages calculated by combining gyrochronology
    and isochrone fitting with \sd\ are shown in color and ages calculated with
    isochrone fitting only are shown in gray.
The different panels show the results for stars
    with \gcolor\ $<$ 2.2 (FGK dwarfs) that are still braking magnetically
    ($Ro < 2$), stars with \gcolor\ $<$ 2.2 that have stopped braking
    magnetically ($Ro \geq 2$),
    stars with 2.2 $<$ \gcolor\ (M dwarfs), and evolved stars (EEP $>$ 420).
Gyrochronology is highly effective for FGK stars and ages inferred with both
    gyrochronology and isochrone fitting are more accurate and precise than
    ages inferred via isochrone fitting only for this group.
Neither gyrochronology nor isochrone fitting can provide precise ages for
    M dwarfs, so the ages of these stars are imprecise regardless of
    age-dating method.
    This figure was generated in a Jupyter Notebook available at
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Results_plots.ipynb}.
}
  \centering
    \includegraphics[width=1\textwidth]{simulation_results}
\label{fig:simulation_results}
\end{figure}

An important caveat associated with these results is that they strongly depend
on the uncertainties adopted for all observables used in the analysis: \teff,
\feh, \logg, $G$, $G_BP$, $G_RP$, $J$, $H$, $K$, $B$, $V$, and rotation
period.
Changing the uncertainties on these observables will affect the uncertainties
on inferred ages in different ways.
This demonstration is not intended to reflect the typical age uncertainties
that will result for all stars in reality, it merely exemplifies the increase
in stellar age precision that results from one specific choice of
uncertainties.
Needless to say, estimating the uncertainties on observables accurately can be
as important as accurately measuring the observables themselves.

% Figure \ref{fig:all_iso_gyr} shows the inferred vs. true ages of all simulated
% stars, colored by their \gaia\ \gcolor\ age.
% The ages of FGK stars (\gcolor\ $<$ 2.2) were more precisely recovered with
% gyrochronology and isochrone fitting than with isochrone fitting alone.
% In both cases M dwarfs ages were not precisely recovered.
% \begin{figure}
%   \caption{
%       Inferred vs. true ages for all simulated stars using isochrones plus
%       gyrochronology via the \sd\ code (left) and isochrones only (right).
%       Stellar ages are colored by \gaia\ \gcolor\ color.
%       The ages of FGK stars (\gcolor\ $<$ 2.2)
%       were more precisely recovered with \sd\ than with isochrones only.
%       In both cases M dwarfs ages were not precisely recovered.
%       Points in the lower right of the left panel are either M dwarfs or
%       evolved (EEP $>$ 400).
%       In both panels, the ages of FGK stars were slightly underestimated.
% }
%   \centering
%     \includegraphics[width=1\textwidth]{all_iso_gyro}
% \label{fig:all_iso_gyro}
% \end{figure}

% Figure \ref{fig:eep} shows the inferred equivalent evolutionary phase (EEP)
% for the simulated star sample.
% EEP is a dimensionless number that describes evolutionary stage and, combined
% with age and metallicity, determines stellar mass.
% An interesting result of inferring stellar properties with both isochrones
% {\it and} rotation periods is an increase in EEP precision, .
% The reason for this is partly that EEP and age are correlated, so shrinking
% the age posterior also shrinks the EEP posterior, and partly because rotation
% period is a function of HRD position.
% As shown in equation \ref{eqn:gyro}, rotation period does not just depend on
% color and age, but also on EEP and [Fe/H].
% \begin{figure}
%   \caption{
%     Inferred EEP for the simulated star sample with isochrones plus
%     gyrochronology (left panel) and isochrone fitting only (right panel).
%     Rotation periods provide information about position on the HRD via
% equation \ref{eqn:gyro} and therefore improve all stellar parameters, not
% just age.
%     This figure was generated in a Jupyter Notebook available at
%     \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Results_plots.ipynb}.
% }
%   \centering
%     \includegraphics[width=1\textwidth]{eep}
% \label{fig:eep}
% \end{figure}
% In general, the posterior PDFs of stellar properties, inferred using
% isochrones or stellar evolution models, are multimodal.
% The stars with under-predicted EEP in both panels of figure \ref{fig:eep} are
% slightly evolved, with EEPs $>$ 400.
% These slightly evolved stars have two high probability modes: one at their
% true EEP and age values, and another at a lower EEP and corresponding lower
% age.
% In other words, these are subgiants being mistaken for dwarfs.
% Rotation periods do not carry information for subgiants, so including the
% rotation periods in the analysis of these stars doesn't break the degeneracy,
% and this is why stars with under-predicted EEPs are present in both panels.
% Stars with over-predicted EEPs in the right-hand panel of figure \ref{fig:eep}
% are usually dwarfs being confused for subgiants.
% The rotation periods of dwarfs are informative, so when the rotation periods
% are included in the analysis for these stars, the degeneracy is broken, the
% amplitude of the secondary mode is reduced, and their EEP is correctly
% inferred.

% Figure \ref{fig:precision} shows the simulated stars on an HRD, with
% points colored by the precision of their predicted ages
% (defined as the standard deviation of the age
% posterior PDF, as a percentage of the median age).
% The top panel shows the precision of ages calculated using both gyrochronology
% and isochrone fitting via \sd\ and the bottom panel shows the precision of
% ages calculated with isochrone fitting only.
% Although these are empirical uncertainties, calculated via MCMC which only
% approximates the age posterior PDFs, they nevertheless show that combining
% gyrochronology and isochrone fitting improves age precision on the MS,
% particularly at low masses.
% \begin{figure}
%   \caption{
% Simulated stars on an HRD, colored by their relative age precision
%     using gyrochronology and isochrone fitting via \sd\ (top panel) and
%     isochrone fitting only (bottom panel).
% Combining gyrochronology with isochrone fitting significantly improves stellar
%     age precision on the MS.
% Isochrone fitting provides precise ages for hot stars and subgiants and the
%     rotation periods of these stars are relatively uninformative, so
%     gyrochronology does not significantly improve their age precision.
% The ages of late M dwarfs are highly imprecise because their ages are not well
%     determined by either their rotation periods or their position on the HRD
%     or CMD.
%     This figure was generated in a Jupyter Notebook available at
%     \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Results_plots.ipynb}.
% }
%   \centering
%     \includegraphics[width=1\textwidth]{precision_plot}
% \label{fig:precision}
% \end{figure}

This simulation experiment was designed to show the theoretical improvement in
age precision when gyrochronology is incorporated into isochrone fitting.
However, it does not demonstrate the accuracy of this method because the test
data were simulated from the same model used to infer ages.
The results of this experiment are therefore extremely accurate by design.
When applying this method to real data, the results will only be accurate if
the model is accurate.
In other words, \sd, like any age-dating method, provides model-dependent
ages.
Stellar ages calculated with \sd\ depend on both the accuracy of the MIST
models {\it and} the accuracy of the gyrochronology model (equation
\ref{eqn:gyro}).
In order to test the accuracy of \sd, we applied it to real data, as described
in the following section.

\subsection{Test 2: Open clusters}
In order to test our model on real stars with known ages, we selected a sample
of stars in the 2.5 Gyr NGC 6819 cluster.
We compiled \kepler-based rotation periods \citep{meibom2015}, \Gaia\
photometry and \gaia\ parallaxes for members of the NGC 6819 cluster.
Figure \ref{fig:NGC6819} shows the period-color relation of this
cluster and figure \ref{fig:NGC6819_results} shows the results of inferring
the ages of individual cluster members using a combination of gyrochronology
and isochrone fitting (via \sd) and isochrone fitting alone.
The ages of F stars in this cluster (\gcolor\ $\sim$ 5.5-6.5) were relatively
precisely constrained by isochrone fitting alone because, at 2.5 Gyr, they are
approaching the MS turnoff.
For these hot stars, ages inferred with gyrochronology and isochrones were
similar to ages inferred with isochrones and similarly precise, showing that
isochrones provide a lot of age information for these stars and rotation
periods do not add significantly more information.
The G and early K stars in this cluster (\gcolor\ $\lesssim$ .65) were not
precisely recovered from isochrone fitting alone -- the isochrone-only age
posteriors tend towards the prior which is a uniform distribution between 0
and 13.8 Gyrs.
The median age of stars in the cluster was 4.27 $\pm$ 0.48 Gyr when only
isochrone fitting was used.
In contrast, including gyrochronology when inferring the ages of G and K stars
in this cluster significantly improved age precision.
The median age of stars in this cluster was 2.63 $\pm$ 0.16 Gyr when ages were
inferred with a combination of isochrone fitting and gyrochronology, using the
newly calibrated Praesepe-based gyrochronology model.
The previously-calibrated \citet{angus2015} model resulted in a median stellar
age of 2.66 $\pm$ 0.21 Gyr, which is still consistent with the established
cluster age of 2.5 Gyr.
The median age of stars in the cluster using uncorrected photometry was
slightly underestimated at 1.86 $\pm$ 0.22 Gyr.
This suggests that, despite the fact that V-band extinction is marginalized
over during the inference process, correcting for extinction {\it before} ages
are estimated will reduce bias introduced by dust.

\begin{figure}
  \caption{
    The \kepler-based rotation periods of members of the 2.5 Gyr NGC 6819 open
    cluster.
    The raw \gcolor\ colors are shown in red and the dust-corrected colors are
    shown in black.
    The dashed line shows a gyrochronology model that was fit to the Praesepe
    cluster and the Sun in this work, interpolated to 2.5 Gyrs.
    The solid blue line shows a previously calibrated gyrochronology model
    \citep{angus2015}.
    This figure was generated in a Jupyter Notebook available at:
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/NGC6819.ipynb}
  }
  \centering
    \includegraphics[width=1\textwidth]{NGC6819}
\label{fig:NGC6819}
\end{figure}

\begin{figure}
  \caption{
    The inferred ages of members of the NGC 6819 open cluster as a function of
    their \gcolor\ color.
    Ages of stars inferred using a combination of isochrone fitting and
    gyrochronology (Praesepe and Sun calibration) with dereddened \gaia\ $G$,
    $G_{BP}$, and $G_{RP}$ photometry (black circles) and uncorrected, raw,
    photometry (red squares).
    % Black circles and red squares show the ages of stars inferred using a
    % combination of isochrone fitting and gyrochronology, with a gyrochronology
    % relation that was calibrated to Praesepe and the Sun.  Black circles show
    % ages inferred with dereddened \gaia\ $G$, $G_{BP}$ and
    % $G_{RP}$ photometry and red squares show ages inferred with uncorrected,
    % raw, photometry.
    Even though V-band extinction is marginalized over in the inference
    process, reddening can still bias ages.
    Blue triangles, pointing up, show ages inferred using isochrone fitting
    and gyrochronology, with the \citet{angus2015} gyrochronology model.
    Orange triangles, pointing down, show ages inferred using isochrone
    fitting only.
    The ages of F stars (stars bluer than 0.7) were precisely constrained by
    isochrones and including gyrochronology makes little difference to their
    inferred ages.
    The age precision of G and K dwarfs (stars redder than 0.7) was improved
    by including gyrochronology.
    The median age of stars inferred using the gyrochronology model
    calibrated to Praesepe and the Sun (black circles) was 2.65 $\pm$ 0.13
    which is consistent with the established cluster age (2.5 Gyr).
    This figure was generated in a Jupyter Notebook available at:
    \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/NGC6819.ipynb}
}
  \centering
    \includegraphics[width=1\textwidth]{NGC6819_results}
\label{fig:NGC6819_results}
\end{figure}

We found that 5\% rotation period uncertainties resulted in the most accurate
ages for NGC 6819.
The uncertainties on the measured rotation periods, provided in
\citet{meibom2015} and shown in figure \ref{fig:NGC6819}, were likely
underestimated for some stars.
Underestimated rotation period uncertainties can result in inaccurate age
estimates.
This raises the question: how should uncertainties on rotation periods be
estimated?
The likelihood is weighted by the inverse variance, so uncertainties on the
rotation period control the relative information provided by gyrochronology,
isochrones, and the prior.
If rotation period uncertainties are either too large or too small, the
resulting age estimate will be imprecise and/or inaccurate.
It is difficult to measure uncertainties on rotation periods directly:
standard techniques such as Lomb-Scargle periodograms and autocorrelation
functions do not provide them.
% In general, formal rotation period uncertainties should probably be calculated
% empirically via simulations as in the \citet{aigrain2015} study.
\racomment{Ideally, rotation period uncertainties should capture {\it both} the
measurement precision, {\it and} the physical uncertainty introduced by the
latitudinal movement of star spots on the surface of a differentially rotating
star.
For example, \citet{donahue1996} demonstrated that the seasonal variation in
measurements of G and K star rotation periods is a function of period, $\Delta
\mathrm{P_{rot}} \propto \mathrm{P_{rot}}^{1.3\pm0.1}$.
This variation is presumably caused by a latitudinal drift in the dominant
active regions, which traces the stellar cycle over several years, in
combination with latitudinal differential rotation.
This suggests that latitudinal spot drifting does not significantly
affect stellar rotation periods when stars are young, for example the scatter
of rotation periods about the mean gyrochronology model in Praesepe is only
around 5\%.
However it is likely that this effect will become more important at older
ages.
% The impact of this phenomenon on gyrochronology remain to be seen, and an
% in-depth study is warranted.
A thorough exploration of how rotation period uncertainties, from both
measurement uncertainty and physical variation, affect stellar ages via
gyrochronology is key to understanding the power of gyrochronology as an
age-dating method.
For now, we leave this exploration for a future study.}

\subsection{Test 3: Kepler asteroseismic stars}
\racomment{In order to test our method in the regime where both isochrone
fitting and gyrochronology become important, we recovered the ages of the 21
asteroseismic stars analysed in \citet{vansaders2016}.
These 21 stars were observed in \kepler's short cadence mode and are a mixture
of dwarfs and subgiants.
Their asteroseismic ages were calculated from the analysis of the frequencies
of individual oscillation modes \citep{mathur2012, metcalfe2014,
silvaaguirre2015, ceillier2016}, and their rotation periods from their
\kepler\ light curves \citep{garcia2014}.
We crossmatched these stars with the \Gaia\ catalog to obtain parallaxes and
apparant magnitudes in the \Gaia\ G, $\mathrm{G_{BP}}$ and $\mathrm{G_{RP}}$
band passes.
We also added J, H and K 2MASS magnitudes from the Kepler input catalog
\citep{brown2011},
% asteroseismic parameters $\Delta_\nu$ and $\nu_{\mathrm{max}}$ from
% \citet{ceillier2017} and \citet{serenelli2017},
and used spectroscopic effective temperatures, spectroscopic metallicities and
rotation periods reported in table 1 of \citet{vansaders2016}.
The \gaia\ photometry is extremely precise, and we found that artificially
inflating the uncertainties on \gaia\ apparent magnitudes by a factor of 10
substantially improved the quality of fit, both in terms of MCMC convergence
and agreement with asteroseismic age measurements.
In figure \ref{fig:astero} we show the ages of these 21 stars inferred using
isochrone fitting and gyrochronology, against their asteroseismic ages,
calculated using the Asteroseismic Modeling Portal (AMP) \citep{metcalfe2009,
metcalfe2012, metcalfe2014}.
In this figure, the colored symbols show ages inferred using a combination of
gyrochronology and isochrone fitting, implemented with the \sd\ {\it Python}
package.
The black and grey triangles show ages inferred from gyrochronology only,
where the mass and color of the stars were not inferred, but fixed to be the
asteroseismic mass and the \gaia\ \gcolor\ color.
These gyrochronal ages were calculated with age as the only free parameter,
without marginalizing over stellar mass, \gcolor\ color or extinction.
As a result, these gyrochronal ages are {\it not the same} as the ages given
by the maximum of the gyrochronal likelihood function used in the combined
age model, they simply represent an approximation to the gyrochronal age.
Grey triangles are shown for stars where gyrochronology is not applicable
because the stars are either too metal poor, too metal rich, or too evolved.
Black triangles are shown for stars where gyrochronology {\it is} applicable.
The white circles show ages inferred from isochrones only.
Dashed lines connect the three different age measurements for the same
stars.
}

\racomment{
Although the ages of all 21 stars shown in figure \ref{fig:astero} were
inferred with a joint isochronal and gyrochronal model, most (all but 8) were
either too evolved, too metal poor, or too metal rich for gyrochronology to
contribute any information to the ages.
These metal poor/rich or evolved stars lie in a regime where the variance on
their rotation period was artificially inflated because the gyrochronology
relations are not thoroughly understood or well calibrated.
The rotation periods of the remaining eight stars {\it did} contribute to
their inferred ages to some degree, however isochrones still dominated the age
information for some of them because these stars are relatively old and/or
relatively hot.
% Isochrones are information-rich for these hot and old stars, and this is
% exactly where gyrochronology is at its most information-poor.
}

\racomment{
In general, there is relatively poor agreement between the asteroseismic ages
and the ages inferred using \sd.
Much of this discrepancy is driven by differences in the isochronal ages,
which is likely attributable to differences between the MIST stellar evolution
models and those used in the AMP analysis: a combination of the Aarhus stellar
evolution code \citep[ASTEC][]{christensen-dalsgaard2008a} and the adiabatic
pulsation code \citep[ADIPLS][]{christensen-dalsgaard2008b}.
We compared non-rotating, Solar-metallicity MIST isochrones for middle-aged
stars with Solar-metallicity BaSTI isochrones \citep{pietrinferni2004,
hidalgo2018} and found that, for stars between 4 and 8 Gyrs, in the same
effective temperature range as the asteroseismic stars, the age discrepancy
between the two sets of models can be as large as 1-2 billion years.
The MIST isochrones lie above the BaSTI isochrones on the HR diagram, leading
to a systematic underprediction of ages.
}

\racomment{
The gyrochronal ages, where gyrochronology is applicable, do not show
excellent agreement with the asteroseismic ages either.
The four hot stars to the left in figure \ref{fig:astero} are rotating more
slowly than predicted by the Praesepe-based gyrochronology models, and as a
result their gyro-ages are older than their asteroseismic ones.
For these hot stars, two out of four have ages that are still consistent, or
close to consistent, with their asteroseismic ages.
The third star from the left is an anomalously slow rotator for its age and
mass and \citet{vansaders2016} also found this star to be surprisingly slowly
rotating.
In contrast, the star with a black triangle symbol (indicating that
gyrochronology is applicable) furthest to the right is {\it rapidly} rotating
for its mass and age, even when weakened braking is taken into account.
% Both stars are around Solar mass (1.06 \pm 0.02 and 1.0 \pm 0.03 M$_\odot$
% from left to right) and have asteroseismic ages older than the Sun (6.82 \pm
% 0.28 and 7.28 \pm 0.51 Gyr from left to right), and yet both rotate more
% rapidly than the Sun (23.2 \pm 7.4).
This star is around Solar mass (1.0 $\pm$ 0.03 M$_\odot$) and has an
asteroseismic age older than the Sun (7.28 $\pm$ 0.51 Gyr), yet rotates with a
period of only 19.8 $\pm$ 1.3 days.
This is KIC 9098294, a single-lined spectroscopic binary with an orbital
period of around 20 days (Latham, private communication).
It is the only clear SB1 in the \citet{vansaders2016} sample, although some
others do have binary companions with long orbital periods, for which tidal
interactions are not expected to be strong.
}

\begin{figure}
    \caption{ A comparison of stellar ages inferred using
    asteroseismic modeling with ages inferred using a combination of isochrone
    fitting and gyrochronology.
Colored circles show ages inferred using isochrone fitting and gyrochronology
combined via the \sd\ software package.
Black triangles show the ages of all stars inferred via gyrochronology only
    and white circles show ages inferred via isochrone-fitting only.
% This plot shows that, for the majority of stars in this sample, isochrone
%     fitting dominates the age information and gyrochronology only contributes
%     significantly for two or three stars.
% This is a consequence of many of these stars being hot old, extremely metal
%     rich, extremely metal poor, or evolved.
  }
  \centering
    \includegraphics[width=1.2\textwidth]{asteroseismic_results_nosdss_gaia}
\label{fig:astero}
\end{figure}


%!TEX root = ../PhD_thesis__Lilian_Besson

% ----------------------------------------------------------------------
\chapter{Improving Spectrum Usage of IoT Networks with Selfish MAB Learning}
\label{chapter:4}

\graphicspath{{2-Chapters/4-Chapter/Images/}}

\abstractStartChapter{}%
%
After detailing in Part~\ref{part:Introduction} the MAB model, we now come back to our main question of interest.
In this chapter, we focus on wireless networks following the hypotheses common to present and future Internet of Things (IoT) networks exposed in Chapter~\ref{chapter:1}.
Our goal is to show that the IoT devices can automatically learn to increase their battery life and their successful transmission rates, without changing anything on the IoT standard side.
%
We propose two models of IoT networks, composed of many independent IoT end-devices, that can use low-cost Reinforcement Learning (RL) algorithms in order to learn how to improve their spectrum access.
%
Decentralized RL for IoT lets the devices use acknowledgements sent back by their Base Station as a reward, instead of sensing feedback like for OSA.
We consider many independent ``dynamic'' devices, each communicating with a small probability at every instant.
Simulations show that dynamic devices can greatly improve their spectrum efficiency,
by using MAB algorithms like \UCB.
% in the case of a non-uniform distribution of the static devices in the channels
We also developed a proof-of-concept using USRP platforms, for a real-world validation of this approach.
% Numerical simulations justify the interest of the approach,
%  of the advantage of using MAB algorithms for the first model.
%
% On a simple example, we present in Section~\ref{sec:4:gnuradio} a demonstration that implements and validates the proposed approach, using real wireless radio hardware.
% The first model
% % without retransmission of packets,
% is interesting for its simplicity,
% and because considering no retransmission can improve the battery life of IoT devices, at the cost of a lower Quality of Service (QoS).
%
In a second step, we consider a second model where a dynamic device has to (try to) retransmit a message, in case of a failed first transmission, up-to a fixed number of retransmissions.
We compare heuristics based on \UCB, and simulations confirm that the non-naive heuristics also significantly improve the network efficiency.
% Interestingly, the simplest heuristic is found to be the most efficient one, as its simplicity implies a faster learning and convergence time.
%
% The second model is more interesting if the main target is an improvement of the QoS, rather than a longer battery life.

\minitocStartChapter{}

% ----------------------------------------------------------------------------
% \section{Motivations for Selfish MAB Learning for IoT Networks}
\section{Introduction and motivations for MAB learning for IoT Networks}
\label{sec:4:motivations}
% ----------------------------------------------------------------------

After the first chapters that presented the model of MAB, we go back to the initial problems studied in this thesis, and thus we focus on Internet of Things (IoT) networks.
%
As explained in the introduction in Chapter~\ref{chapter:1},
unlicensed bands are more and more used and considered for mobile and LAN (Local Area Network) and for Internet of Things communication standards.
%  (Wi-Fi, LTE-U), (IoT) standards for short-range (ZigBee, Z-Wave, Bluetooth) and long-range (LoRaWAN, SIGFOX, Ingenu, Weightless) communications \cite{Centenaro16}.
This heavy use of unlicensed bands, in particular with the expected exponential growth of the number of IoT devices, will cause performance drop, due to radio collisions that could even compromise IoT promises.

Efficient Medium Access Control (MAC) policies allow devices to avoid interfering traffic and can significantly reduce the spectrum contention problem in unlicensed bands.
As end-devices battery life is a key constraint of IoT networks,
and as IoT networks are decentralized, because the devices initiate transmissions,
this leads to IoT protocols using as low signaling overhead as possible and simple ALOHA-based mechanisms.
%
In this chapter, we analyze the performance of Multi-Armed Bandits (MAB) algorithms, that could be used in combination with a time-frequency slotted ALOHA-based protocol.
We highlight that even without changing anything on the level of IoT standards, our proposal is just an add-on capability that can be used on a unit-per-unit basis.
We consider the \UCB \cite{Auer02}, and the Thompson-Sampling (TS) algorithms \cite{Thompson33,AgrawalGoyal11,
Kaufmann12Thompson}, for the first model. For the demonstration as well as for the second model, without loss of generality, we preferred to focus on heuristics based on the simplest algorithm (\ie, \UCB), to give a clear presentation of the different ideas explored to solve the problem of learning in order to retransmit efficiently.

% As detailed in Chapter~\ref{chapter:1},
% MAB learning has already been proposed in Cognitive Radio (CR) \cite{Mitola99,Haykin05}, and in particular, for sensing-based Dynamic Spectrum Access (DSA) in licensed bands \cite{Jouini10}.
% For example,
% proof-of-concepts like \cite{kumar2016two} have proven the capability of such approaches on real radio signals for OSA,
% and \cite{Maghsudi16} shows how MAB learning can be applied for small cell management in licensed 5G networks.
% Some analysis on real radio measurements made for HF ionospheric channels have also proven that solutions based on MAB learning is appropriate and solves efficiently this kind of decision-making problems on real-world wireless signals \cite{Melian15}.
% Recent works show that stationary MAB algorithms work well to solve reinforcement learning models that represent accurately real-world radio problems.
% Recently, TS and \UCB{} algorithms have been used for improving the spectrum access in (unlicensed) Wi-Fi networks, for instance by \cite{Toldov16} or \cite{Wilhelmi19collaborative,Wilhelmi19potential}.
% % However, even with only one dynamic user using the learning algorithm, the background traffic or the traffic of the other devices is never really stationary or \iid{}.

We present in Section~\ref{sec:4:firstModel} how the MAB algorithms can be used in a unlicensed but frequency- and time-slotted IoT network.
Several devices are using bandit algorithms, and the assumptions made by the stochastic bandit algorithms are not satisfied: as several agents learn simultaneously and their activation processes are random, their behavior is not stationary.
As far as we know, we provide the first practical study to confirm robustness of the use of stochastic bandit algorithms for decision making in IoT networks with a large number of intelligent devices in the network, which makes the environment ``strongly not stationary''.
This specific context makes it very hard to give mathematical proofs of convergence and of efficiency of bandit algorithms (that is why we relax the hypothesis and only consider up-to $M \leq K$ players in Chapter~\ref{chapter:5}).
We then validate the model with a hardware implementation on real radio signals, detailed in Section~\ref{sec:4:gnuradio}.
%
We conclude this chapter by presenting in Section~\ref{sec:4:retransmissions} an extension of this model to take into account another aspect of the ALOHA protocol, that is the possibility for dynamic devices to retransmit their packets if the \emph{Ack} was not received.

\paragraph{Publications.}
%
This chapter is mainly based on our articles \cite{Bonnefoi17,Besson2018ICT,Besson2019WCNC,Bonnefoi2019WCNC,MoyBesson2019,MoyBesson2019Annales}.

% ----------------------------------------------------------------------------
\section[Selfish learning for many dynamic devices in an IoT network]{Selfish learning for many dynamic devices with low activation probabilities in an IoT network}
\label{sec:4:firstModel}
% ----------------------------------------------------------------------

% - ``Multi-Armed Bandit Learning in IoT Networks and non-stationary settings'', see https://hal.inria.fr/hal-01575419

\input{2-Chapters/4-Chapter/CrownCom_17/IoT_slotted.tex}

\newpage  % WARNING ?
% ----------------------------------------------------------------------------
% \section{Test-bed implementation of the first model for real-world validation}
\section{Proof-of-concept of our model for real-world validation}
\label{sec:4:gnuradio}
% ----------------------------------------------------------------------

% - ``MALIN: Multi-Arm bandit Learning for Iot Networks with GRC: A TestBed Implementation and Demonstration that Learning Helps'', demo at ICT, and the companion paper ``GNU Radio Implementation of MALIN: "Multi-Armed bandits Learning for Internet-of-things Network"'', see https://hal.inria.fr/hal-02006825

\input{2-Chapters/4-Chapter/IEEE_WCNC_2019__DemoICT/IEEE_WCNC_2019__DemoICT.tex}

\newpage  % WARNING ?
% ----------------------------------------------------------------------------
\section{Extending the model to account for retransmissions}
\label{sec:4:retransmissions}
% ----------------------------------------------------------------------

% - ``Upper-Confidence Bound for Channel Selection in LPWA Networks with Retransmissions'', see \texttt{https://perso.crans.org/besson/articles/BMBM\_\_IEEE\_WCNC\_2019.pdf}

\input{2-Chapters/4-Chapter/IEEE_WCNC__2019__Paper__BMBM/IEEE_WCNC__2019__Paper__BMBM.tex}

\newpage  % WARNING ?
% ----------------------------------------------------------------------------
\section{Conclusion -- Towards theoretical guarantees}
% \section{Conclusion}
\label{sec:4:conclusion}
% ----------------------------------------------------------------------

% % We give in this section conclusions about the two models presented above, one for long-life IoT devices (without retransmission) and for improved QoS devices (\ie, with retransmissions), and about our demonstration, as well as directions of future works, some of which are currently being studied by researchers of our team.

% % \subsection*{Conclusions}

% % ----------------------------------------------------------------------
% % \paragraph{Summary of the first model.}
% \label{sub:41:conclusion}

% In Section~\ref{sec:4:firstModel}, we proposed an evaluation of the performance of MAB learning algorithms in IoT networks,
% with a focus on the convergence of algorithms, in terms of successful transmission rates, when the proportion of intelligent dynamic devices changes.
% Concretely, increasing this probability allows to insert more devices in the same network, while maintaining a good Quality of Service.
% Similarly, if the number of devices remain constant, increasing the successful transmission rate directly extends the IoT devices battery life, as they suffer less from failed transmissions.
% We show that \UCB{} and TS have near-optimal performance, even when their underlying \iid{} assumption (see Chapter~\ref{chapter:2}) is violated by the presence of many ``intelligent'' end-devices which follow a random activation process.
% %
% This is both surprising and encouraging, it shows that applying bandit algorithms tailored for a stochastic model is still useful in broader settings.
% The fully \emph{decentralized} application of classic stochastic MAB algorithms are almost as efficient as the best possible centralized policy in this setting, after a short learning period, even though the dynamic devices \emph{cannot} communicate with each other, and \emph{do not} know the system parameters.

% % ----------------------------------------------------------------------
% % \paragraph{Conclusions taken from the demonstration.}
% \label{sub:42:conclusionFromDemonstration}

% We presented in Section~\ref{sec:4:gnuradio} a demonstration, showed in $2018$ at the ICT conference \cite{Besson2018ICT}, and further detailed in the companion paper \cite{Besson2019WCNC}.
% We gave all the necessary details on both the PHY and the MAC layer, as well as details on the User Interface developed for the demo.
% Results obtained in practice were discussed, to highlight the interest of using learning algorithms for radio online optimization problem, and especially multi-armed bandit learning algorithms.
% %
% By using such low-cost algorithms, we demonstrated empirically that a dynamically re-configurable device can learn on its own to favor a certain channel, if the environment traffic is not uniform between the $K$ different channels, by using the acknowledgement (\Ack) feedback sent from the base station.

% % ----------------------------------------------------------------------
% % \paragraph{Conclusion about this second model.}
% \label{sub:43:conclusion}

% In Section~\ref{sec:4:retransmissions}, we presented an extension of our model of LPWA networks based on an ALOHA protocol, slotted both in time and frequency.
% % , in which dynamic IoT devices can again use machine learning algorithms, to improve their Packet Loss Ratio (PLR) when accessing the network.
% If the priority is the Quality of Service, for instance with renewable energy capabilities, this second model is more appropriate.
% The main novelty of this model is to address the packet retransmissions upon radio collision, by using a Multi-Armed Bandit framework.
% We presented and evaluated several heuristics that try to learn how to transmit and retransmit in a smarter way, by using the \UCB{} algorithm for channel selection for first transmission, and different proposals based on \UCB{} for the retransmissions upon collisions.
% %
% We showed that incorporating learning for the transmission is needed to achieve optimal performance, with significant gain in terms of successful transmission rate in networks with a large number of devices (up-to $30\%$ in the example network).
% Our simulations show that each of our proposed heuristic outperforms a naive random access scheme.
% Surprisingly, the main take-away message is that a simple \UCB{} learning approach, that retransmit in the same channel, turns out to perform as well as more complicated heuristics.

% % ----------------------------------------------------------------------
% \subsection*{Future works}
% \label{sub:4:futureWorks}

% % Future works related to this chapter include the following directions.

% % \paragraph{Possible extensions of the first model.}

% The first model presented in Section~\ref{sec:4:firstModel} could easily be generalized with two probabilities $p_S$ and $p_D$, if static and dynamic devices have different transmission pattern, and less easily with one probability per device. Also, other emission pattern could be considered, instead of a Bernoulli process for each user.
% In this whole Chapter~\ref{chapter:4}, we prefer to consider that all devices have the same activation probability, to keep the notation as simple as possible.
% %
% Moreover, for sake of simplicity we supposed that all devices use the same standard.
% Future works could consider more realistic interference scenarios and IoT networks, with, \eg, non-slotted time, more than one base station etc.

% Another extension could be to not have a Bernoulli process (or any random process), but a fixed rate of transmission, \eg, one transmission a day.
% So additionally to deciding the channel for communication (\ie, \emph{where} to communicate), each device has to also decide \emph{when} to communicate.
% % This is another direction of research, that we will investigate in the future.
% However, this clearly leads to a much larger action space, as there are many time slots in one day (for example), and thus we believe that as soon as the action space becomes too large in this extension, the simple MAB-based learning approach could be no longer appropriate.
% It is well-known in the MAB literature that the larger the action space, the slower is the convergence speed of any stationary MAB algorithms.
% It could be interesting to study the possible application of \emph{contextual} MAB \cite{Li10,Luo18} or structured MAB \cite{Combes17} models and algorithms for this extension.

% % We will investigate this behavior in order to understand it better theoretically.
% % We will also experiment more with adversarial algorithms, to confirm that they work less efficiently than stochastic bandit algorithms in our non-stochastic setting.

% % \paragraph{Extensions of the demonstration.}

% Possible future extensions of our demonstration include the following points.
% We could consider more dynamic devices (\eg, $100$) but it would either cost more in terms of equipment, or in terms of software engineering to simulation more devices with the same card.
% We could also implement a real-world IoT communication protocol (like the LoRaWAN standard), which we prefer not to do as it would cost a significant effort of development.
% Finally, we could also study the interference in case of other gateways located nearby, and this could be done without needing a lot of new hardware (using one extra USRP card to simulate another gateway).
% %
% % Pub pour notre autre article
% % We are also interested in studying the possible gain of using a learning step when the transmission model follows ALOHA-like retransmissions, and this is presented in the next Section~\ref{sec:4:retransmissions}.

% % \paragraph{Possible extensions for the second model.}

% Finally, the utility and impact of the proposed approaches for LPWA networks motivates us to address several subjects as future works. Among them, the non-stationarity of the channel occupancy caused by the learning policy employed by the IoT devices.
% %
% For that end, modifications of MAB algorithms have been proposed, such as Sliding-Window-\UCB{} or Discounted-\UCB{} \cite{Garivier11UCBDiscount}
% or more recently M-\UCB{} \cite{CaoZhenKvetonXie18},
% or more recently GLR-\UCB{} \cite{Besson2019GLRT} which is presented in Chapter~\ref{chapter:6},
% that nevertheless have not been explored for the targeted problem.
% Chapter~\ref{chapter:6} is focusing on this direction, but we did not have enough time to explore the possible applications of MAB policies designed for non-stationary problems to the model with retransmissions presented in this Chapter.

% In order to validate our results in a realistic experimental setting and not only with simulations, future works include a hardware implementation of the analyzed models to complete our demonstration \cite{Besson2019WCNC}.
% Note that Julio César Mango-Vasquez is currently working on this direction, for the EPHYL project, in collaboration with Carlos Faouzi Bader at IETR and CentraleSupélec campus of Rennes.
% % and Christophe Moy at IETR and University Rennes 1.
% %
% A hardware demonstrator could be also benefit to study other settings by removing some hypotheses, for instance by studying a similar model in non-slotted time.

% \subsection*{Summary of this chapter}

We focused in this chapter on models of IoT networks, and we proposed to use classical stationary multi-armed bandit learning algorithms implemented in a selfish and decentralized manner by each of the dynamic devices in the IoT network.
We presented two models of wireless IoT networks, without relying on the feedback provided by spectrum sensing, and inspired by the ALOHA protocol. We proposed two versions, with or without retransmissions of up-link packages in case of collisions.
% In both cases, we try to model the existing standards, like the LoRa standard, and we demonstrated the efficiency of the proposed MAB-based approach in both numerical simulations and empirical measurements on real wireless radio signals.
%
We conclude that this learning-based approach is efficient, as it allows the IoT devices to automatically and independently increase their successful transmission rates.

It is also quite surprising that stochastic MAB algorithms can be of any use in such non-stationary applications.
Unfortunately, it turned out to be of extreme difficulty to analyze analytically the considered model with thousands of independent devices, all communicating and learning in their own (random) time scales.
That is why we focus on two different simplifications of this model in the next chapters, for which we are able to provide a rigorous theoretical analysis.
% %
% We can also highlight that the two models we studied are complementary:
% the first model (without retransmission of packets), is interesting for its simplicity, and focused on improving the battery life of IoT devices,
% % , at the cost of a lower Quality of Service (QoS),
% %
% while the second model is more interesting if the main target is an improvement of the Quality of Service (QoS).
% % , rather than a longer battery life.

\paragraph{Multi-Players MAB.}
%
On the one hand, we are actually able to analyze a simpler model, if we assume to have at most $M \leq K$ devices with a transmission probability of $p=1$.
Instead of the experiment-driven direction pursued in this chapter, another possibility is to consider a \emph{multi-players MAB} model to describe our problem.
%
The main difference between the two models is the fact that in Chapter~\ref{chapter:4}, $M \gg K$ devices transmit their messages at every time step, by following a random activation process (with a fixed transmission probability $p < 1$).
If static and dynamic devices that have to transmit at a fixed time are denoted \emph{active devices},
then their random activation patter makes the number of active devices an (unpredictable) random variable.
Analyzing multi-players MAB models under this hypothesis is much harder, and is left as a future work.
%
We study the case of $M \leq K$ devices learning independently to play a $K$-armed bandit in the next Chapter~\ref{chapter:5}.

% In that case, the static and dynamic devices effect is decoupled, and arms only model the availability of the channels in the absence of dynamic devices: they are \iid{} with mean $\mu_i = 1 - p S_i$.
% Moreover, dynamic devices are usually assumed to be able to \emph{sense} a channel before sending \cite{Zhao10}, and so communicate only if no static device is detected on the channel.
% The smart devices try to learn the arms with highest means, while coordinating to choose different arms, \ie, avoid collisions in their choice, in a decentralized manner.
% However, in this model it is assumed that the multiple agents can know that they experienced a collision with another agent, which is non-realistic for our problem at stake, as our model of smart device cannot do sensing nor differentiate collisions between smart and non-smart devices.

\paragraph{Non-stationary MAB.}
%
On the other hand, while it is hard to analyze the models of this chapter because of the unpredictable behaviors of the IoT devices' activation patterns and the evolving number of active devices, we are also able to analyze a simpler model, if we focus on a single player accessing a network which is assumed to be \emph{piece-wise stationary},
that are bandit problems which are stationary on ``long enough'' intervals, of unknown locations and lengths.
%
We then study this second direction in Chapter~\ref{chapter:6}.

\newpage
% ----------------------------------------------------------------------------
\section{Appendix}
\label{sec:4:appendix}

\subsection{Proof of Proposition~\ref{prop:41:Lagrangian}}
\label{sec:4:proofLagrangian}
% ----------------------------------------------------------------------------

% We include here a missing proof from Section~\ref{sec:4:firstModel}.

\input{2-Chapters/4-Chapter/CrownCom_17/IoT_slotted_appendix.tex}

\newpage
\subsection{Illustration of the GNU Radio Companion Flowcharts}
\label{sec:4:IllustrationFlowcharts}
% ----------------------------------------------------------------------------

\input{2-Chapters/4-Chapter/IEEE_WCNC_2019__DemoICT/IEEE_WCNC_2019__DemoICT_appendix.tex}


\chapter{Mass Relations and Stoichiometry}
\section{Atomic Mass}
To determine the weight of an atom, scientists developed a small unit at which
such quantities may be measured.  This unit is know as the AMU, or
\textit{atomic mass unit}.

12 AMUs are defined as the weight of one atom of the Carbon-12 isotope.  That is
to say that $1amu$ is exactly the weight of 1/12th of one atom of Carbon-12.

\subsection{Masses of Elements}
However, the numbers on the periodic table are not so round?  Why might this be?
Well, since we are accounting for isotopes, we must apply a weighted average to
all naturally occuring isotopes.  For example:

\begin{table}[h]
\begin{tabular}{lll}
\hline
          & Abundance & \textit{Mass (amu)} \\ \hline
oxygen-16 & 99.762\%  & 15.994              \\
oxygen-17 & 0.038\%   & 16.999              \\
oxygen-18 & 0.200\%   & 17.999
\end{tabular}
\end{table}

The calculation for the relative \textit{amu} of Oxygen is as follows:
$(0.99762*15.994) + (0.00038*16.999) + (0.00200*17.999) = 15.9994amu$

\section{The Mole}
Avagadro's Number (or $N_A$) is thought of a quantity, usually of atoms.  The
number of that quantity is $6.022e23$.  Therefore, since this is a quantity, the
unit is $anything/mole$.

\subsection{Molar Mass}
\subsubsection{Of Elements}
To calculate the molar mass of an element: simply look to the periodic table!
In the upper-left of the element box, you will find the relative molar mass of
the element.  That is to say, the \textit{amu} of an element is the weight in
grams of one mole of that element.  For example, $1.008mol$ of \ce{H} weighs
\ce{1g}.

\subsubsection{Of Compounds}
Simply put, the molar mass of a compound is the sum of all its parts, multiplied
individiually by each of the relative quantities.

\section{Percent Composition}
\textbf{Percent composition} of an element is the percent by mass of itself
within a compound.  For example, the molar mass of \ce{Cu2S} is 159.2.  The mass
amount of \ce{Cu}

\subsection{Empirical Formulae}
An \textit{emperical formula} consists of the sumbols for the elements combined
in a compound, with subscripts showing the smallest whole-number mole ratio of
the different atoms in the compound.

For example, the emperical formula \ce{P2O5} might lead to a chemical formula of
\ce{P4O10}, or similarily, the emp. formula \ce{C3H6O} may lead to the formula
\ce{C6H18O2}.

\subsubsection{Calculating}
To calculate an emperical formula, you must first start with the percentages of
an atom or the mass of the atom present.  Once these are determined, you may
then convert to moles, and simplify the ratio by dividing by the smallest amount
and rounding.  If you cannot round cleanly, multiple the whole ratio by a
factor.

The steps for calculating this emperical formula follow below:

\begin{enumerate}
  \item Assume $100g$s was sampled, unless given other measurements.
  \item Convert the grams to moles, unless given moles.
  \item Simplify the ratio of moles by dividing by the smallest term.
  \item Multiply by factors to arrive at cleaner ratios, if not already clean.
\end{enumerate}

\section{Stoichiometry}
Stoichiometry involves the mass relationships between reactants and products in
a chemical reaction.  IT allows you to determine the amount of additional
reactants or products given the chemical equation and just one amount.

Stoichiometry relies on the ratios of moles between different compounds in a
chemical reaction.  Consider the following reaction: \ce{Al2(SO4)3 + 3Ca(OH)2 ->
2Al(OH)3 + 3CaSO4}.  It has the following mass relations:

\begin{frame}
  \centering
  {\huge $\frac{1mol \ce{Al2(SO4)3}}{3mol \ce{Ca(OH)2}}$, $\frac{1mol
  \ce{Al2(SO4)3}}{2mol \ce{Al(OH)3}}$, $\frac{1mol \ce{Al2(SO4)3}}{3mol
  \ce{CaSO4}}$}
\end{frame}

To work with these mass relations, you can convert from mass $\rightarrow$ molar
mass $\rightarrow$ moles $\rightarrow$ mole-to-mole ratio $\rightarrow$ moles
$\rightarrow$ molar mass $\rightarrow$ mass.

\subsection{Limiting Reagent}
Sometimes when doing stoichometry, one reagent appears in a limiting amount such
that you cannot fully complete the reaction.  These substances are known as
\textit{limiting reagents}.  The substances that is not completely used up in a
reaction is referred to as an \textit{excess reagent (reactant)}.

To determine the limiting reagent, just do the reaction with both against one of
the products and figure out which makes \textit{less}.  The excess reagent will
be what is left over.  To determine the quantity of the excess reagent, use the
quantity of the limiting reagent that you originally plugged in and wire that
through.

\subsubsection{Percent Yield Calculations}
Sometimes, you are given an amount of which is supposed to occur (via
stoichiometry) and an amount that you actually produced.  To calculate this, you
use the following:

{\huge $100 * \frac{actual}{theoretical}$}


\documentclass[a4paper]{article}
%\usepackage[ngerman]{babel}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{textcomp}
\usepackage{geometry}
\geometry{ left=2cm, right=2cm, top=2cm, bottom=3cm, bindingoffset=5mm}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{longtable}
\usepackage{amstext}
\usepackage{array}
\usepackage{amsmath}
\newcolumntype{L}{>{$}l<{$}}
\usepackage{tabularx, ragged2e}
\usepackage{helvet}
\renewcommand{\familydefault}{\sfdefault}
\usepackage{lastpage}
\usepackage{todonotes}
\usepackage{titlesec}
\titleformat*{\section}{\large\bfseries}
\usepackage{listings}
\usepackage{color}

\usepackage{tikz}
%\newcommand{\tikzmark}[2]{\tikz[overlay, remember picture] \node[inner sep=0pt, outer sep=0pt, anchor=base] (#1) {#2};}
\usetikzlibrary{tikzmark}

\definecolor{mygreen}{rgb}{0.18, 0.545, 0.341}
\definecolor{mygray}{rgb}{0.5,0.5,0.5}
\definecolor{myblue}{rgb}{0.53,0.61,0.85}

\lstset{
 keywordstyle=\color{mygreen},
 commentstyle=\color{mygray},
 numbers=left,
 numbersep=5pt,
 numberstyle=\scriptsize\color{mygray}
 }

\usepackage{amsmath,amssymb}

\DeclareRobustCommand{\bbone}{\text{\usefont{U}{bbold}{m}{n}1}}

\DeclareMathOperator{\EX}{\mathbb{E}}% expected value

\date{}
\author{}
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\fancyhead[R]{Felix Burk\\ Pascal Huszár}
\fancyhead[L]{Reinforcment Learning \\ Summer Term 2021 }
\fancyfoot[R]{page \thepage \text{ }/ \pageref*{LastPage}}
%\fancyfoot[LE]{Seite \thepage \text{ }von \pageref{LastPage}}
\renewcommand{\headrulewidth}{0.5pt}

\usepackage{amsmath}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\title{\textbf{Exercise 09}}

\begin{document}
	\maketitle
	\thispagestyle{fancy}

    \section*{Task 1 - REINFORCE on the Cart-Pole}
    \subsection*{a)}
    $\frac{\delta}{\delta \theta_a}\frac{se^{s*\theta_a^T+s*\theta_b^T)}}{(e^{\theta_a^T*s}+e^{\theta_b^T*s})^2}$
 	\subsection*{b)}
 	$\pi(a|s,\theta) = \frac{e^{\theta_a^T*s}}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$ where $b,c$ are all possible actions and $a \in \{b,c\}$ \\
 	$\nabla_\theta log(\pi_\theta(s,a)) = \nabla_\theta log(\frac{e^{\theta_a^T*s}}{e^{\theta_b^T*s}+e^{\theta_c^T*s}})$ \\
 	$= \nabla_\theta log(e^{\theta_a^T*s})-\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})$ \\
 	$= \nabla_\theta \theta_a^T*s*log(e)-\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})$ \\
 	$= s -\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})$
 	$= s -\frac{\nabla_\theta log(e^{\theta_b^T*s}+e^{\theta_c^T*s})}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$ \\
 	$= s - \frac{e^{\theta_b^T*s}}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$
 	$= \frac{e^{\theta_c^T*s}*s}{e^{\theta_b^T*s}+e^{\theta_c^T*s}}$
 	$= \frac{1}{1+e^{\theta_c^T*s}}*s$ \\
 	For the case $a=c$, $a=b$ is equivalent.

\end{document}


\chapter{Course Introduction}
\label{cha:intro}

\sloppy
%\lecture{1 --- Wednesday, February 19th}
%{Spring 2020}{Rasmus Kyng}{Course Introduction}

\section{Overview}
 This course will take us quite deep into modern approaches to
 graph algorithms using convex optimization techniques.
%
 By studying convex optimization through the lens of graph algorithms,
 we'll try to develop an understanding of fundamental
 phenomena in optimization.
 %
 Much of our time will be devoted to flow problems on graphs.
 We will not only be studying these problems for their own sake,
 but also because they often provide a useful setting for thinking more broadly about optimization.

 The course will cover some traditional discrete approaches to various graph
 problems, especially flow problems, and then contrast these approaches
 with modern, asymptotically faster methods based on combining convex
 optimization with spectral and combinatorial graph theory.

\section{Electrical Flows and Voltages - a Graph Problem from Middle School?}

We will dive right into graph problems by considering how electrical
current moves through a network of resistors.

First, let us recall some middle school physics.
If some of these things don't make sense two you, don't worry, in less
than paragraph from here, we'll be make to safely doing math.

Recall that a typical battery that buy from Migros has two
endpoints, and produces what is called a \emph{voltage
  difference} between these endpoints.

One end of the battery will have a positive charge (I think that means an excess of
positrons\footnote{I'm joking, of course! Try Wikipedia if you want to know
more. However, you will not need it for this class.}), and the other a
negative charge.
If we connect the two endpoints with a wire, then a current will flow
from one end of the battery to the other in an attempt to even out
this imbalance of charge.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_battery9volts.png}
  \captionof{figure}{A 9 volts battery with a wire attached.}
  \label{fig:battery-volt}
\end{figure}

We can also imagine a kind of battery that tries to send a certain
amount of current the wires between its endpoints, e.g. 1 unit of charge per
unit of time.
This will be a little more convenient to work with, so let us focus on
that case.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_battery1ampere.png}
  \captionof{figure}{A 1 ampere battery with a wire attached.}
  \label{fig:battery-current}
\end{figure}

A \emph{resistor} is a piece of wire that connects two
points $u$ and $v$, and is completely described by a single number $r$
called its \emph{resistance}.

If the voltage difference between the endpoints of the resistor is
$x$, and the resistance is $r$ then this will create a flow of charge per unit of time of $f = x / r$.
This is called Ohm's Law.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_ohmslawx-vs-f.png}
  \captionof{figure}{Ohm's Law for a resistor with resistance $r = 1$.}
  \label{fig:ohmslaw}
\end{figure}

Suppose we set up a bunch of wires that route electricity from our
current source $s$ to our current sink $t$ in some pattern:

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_graphpath.png}
  \captionof{figure}{A path of two resistors.}
  \label{fig:graphpath}
\end{figure}

We have one unit of charge flowing out of $s$ per unit of time, and
one unit coming into $t$.
Because charge is conserved, the current flowing into any other point
$u$ must equal the amount flowing out of it.
This is called Kirchoff's Current Law.

To send one unit of current from $s$ to $t$, we must be sending it
first form $s$ to $u$ and then from $u$ to $t$.
So the current on edge $(s,u)$ is 1 and the current on $(u,t)$ is 1.
By Ohm's Law, the voltage difference must also be 1 across each of the two
wires.
Thus if the voltage is $x$ at $s$, it must be $x+1$ at $u$ and $x+2$
at $t$. What is $x$? It turns out it doesn't matter: We only care
about the differences. So let us set $x = 0$.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_graphpath-labelled.png}
  \captionof{figure}{A path of two resistors.}
  \label{fig:graphpathlabelled}
\end{figure}

Let us try one more example:

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_graphtriangle.png}
  \captionof{figure}{A network with three resistors.}
  \label{fig:graphtriangle}
\end{figure}

How much flow will go directly from $s$ to $t$ and how much via $u$?

Well, we know what the net current flowing into and out of each vertex
must be, and we can use to set up some equations.
Let us say the voltage at $s$ is $x_s$, at $u$ is $x_u$ and at $t$ is $x_t$.
\begin{tight_itemize}
\item Net current at $s$:   $-1 = (x_s-x_t) + (x_s-x_u)$
\item  Net current at $u$:   $\phantom{-}0 = (x_u-x_s) + (x_u-x_t)$
\item Net current at  $t$:\,   $\phantom{-}1 = (x_t-x_s)+(x_t-x_u)$
\end{tight_itemize}
The following is a solution: $x_s = 0$, $x_u = \frac{1}{3}$, $x_t
= \frac{2}{3}$.
And as before, we can shift all the voltages by some constant $x$ and
get another solution $a = x+0$, $x_u = x+\frac{1}{3}$, $x_t
= x+\frac{2}{3}$. You might want to convince yourself that these are the only solutions.

\paragraph{Electrical flows in general graphs.}
Do we know enough to calculate the electrical flow in some other
network of resistors?
To answer this, let us think about the network
as a graph.
Consider a undirected graph $G = (V,E)$ with $\abs{V} = n$ vertices and
$\abs{E} = m$ edges, and let us assume $G$ is connected.
Let's associate a resistance
$\rr(e) > 0$ with every edge $e \in E$.

To keep track of the direction of the flow on each edge, it will be
useful to assign an arbitrary direction to every edge. So let's do
that, but remember that this is just a bookkeeping tool that helps
us track where flow is going.

A \emph{flow} in the graph is a vector $\ff: \R^E$.
%
The \emph{net flow} of $\ff$ at a vertex $u \in V$ is defined as
$\sum_{v \to u} \ff(v,u) - \sum_{u  \to v} \ff(u,v).$
% As a second accounting trick, we can define $\ff(v,u) \defeq
% -\ff(u,v)$,
% and then write the net flow as $\sum_{(u,v)\in E} \ff(u,v)$.

We say a flow routes the demands $\dd \in \R^V$ if the net flow at
every vertex $v$ is $\dd(v)$.

We can assign a voltage to every vertex $\xx \in R^V$.
Ohm's Law says that the electrical flow induced by these voltages will
be $\ff(u,v) = \frac{1}{\rr(u,v)} (\xx(u) - \xx(v))$.

Say we want to route unit of current from vertex $s \in V$ to vertex
$t \in V$.
As before, we can write an equation for every vertex saying that the
voltage differences must produce the desired net current:
\begin{tight_itemize}
\item Net current at $s$:
  $\phantom{----,,,}-1 = \sum_{(s,v)} \frac{1}{\rr(s,v)} (\xx(s) - \xx(v))$
\item Net current at $u \in V\setminus\setof{s,t}$:
  $\phantom{..} 0 = \sum_{(u,v)} \frac{1}{\rr(u,v)} (\xx(u) - \xx(v))$
\item Net current at  $t$:  $\phantom{------,} 1=\sum_{(t,v)} \frac{1}{\rr(t,v)} (\xx(t) - \xx(v))$
\end{tight_itemize}
This gives us $n$ constraints, exactly as many as we have voltage
variables.
However we have to be a little careful when trying to conclude that a
solution exists, yielding voltages $\xx$ that gives induce an
electrical flow routing the desired demand.

You will prove in the exercises (Week 1, Exercise 3) that a solution $\xx$ exists. The proof requires two
important observations: Firstly that the graph is connected, and
secondly that summed over all vertices, the net demand is zero,
i.e. as much flow is coming into the network as is leaving it.

\paragraph{The incidence matrix and the Laplacian matrix.}
To have a more compact notation for net flow constraints, we also
introduce the \emph{edge-vertex incidence matrix} of the graph,
$\BB \in \R^{V \times E}$.
\[
  \BB(v,e) =
  \begin{cases}
    1 & \text{ if } e = (u,v) \\
    -1 &\text{ if } e = (v,u) \\
    0 &\text{ o.w.}
  \end{cases}
\]
%
Now we can express the net flow constraint that $\ff$ routes $\dd$ by
\[
  \BB \ff = \dd
  .
\]
This is also called a conservation constraint. In our examples so far,
we have $\dd(s) = -1$, $\dd(t) = 1$ and $\dd(u) = 0$ for all $u\in V\setminus\setof{s,t}$.

If we let $\RR = \diag_{e \in E} \rr(e)$
then Ohm's law tells us that $\ff = \RR^{-1}\BB^{\trp} \xx$.
Putting these observations together, we have $\BB \RR^{-1}\BB^{\trp}
\xx= \dd$.
The voltages $\xx$ that induce $\ff$ must solve this system of linear
equations, and we can use that to compute both $\xx$ and $\ff$.
It is exactly the same linear
equation as the one we considered earlier.
We can show a that for a connected graph, a solution $\xx$
exists if and only if the
flow into the graph equals the net flow out, which we can express as
$\sum_{v} \dd(v) = 0$ or $\vecone^\trp \dd = 0$.
You will show this as part of Exercise 3.
This also implies that an electrical flow routing
$\dd$ exists if and only if the net
flow into the graph equals the net flow out, which we can express as
$\vecone^\trp \dd = 0$.

The matrix $\BB \RR^{-1}\BB^{\trp}$ is called the \emph{Laplacian} of
the graph and is usually denoted by $\LL$.

\paragraph{An optimization problem in disguise.}
So far, we have looked at electrical voltages and flows as arising
from a set of linear equations -- and it might not be apparent that
this has anything to do with optimization.
But transporting current through a resistor requires energy, which will be
dissipated as heat by the resistor (i.e. it will get hot!).
If we send a current of $f$ across a resistor with a potential drop of
$x$, then the amount of energy spent per unit of time by the resistor will be
$f \cdot x$. This is called Joule's Law.
Applying Ohm's law to a resistor with resistance $r$,
we can also express this energy per unit of time
as $f \cdot x = x^2 / r = r \cdot f ^2$.
Since we aren't bothering with units, we will even forget about time,
and refer to these quantities as ``energy'', even though a physicist
would call them ``power''.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.6\linewidth]{fig/lecture1_flow-vs-energy.png}
  \captionof{figure}{Energy has a function of flow in a resistor with
    resistance $r = 1$.}
  \label{fig:flow-vs-energy}
\end{figure}

Now, another interesting question would seem to be: If we want to find
a flow routing a certain demand $\dd$, how should have flow behave in
order to minimize the the electrical energy spent routing the flow?
We can phrase this as an optimization problem:
\begin{align*}
  \min_{\ff \in \R^E} & \sum_e \rr(e) \ff(e)^2
  \\
  \textrm{s.t. } & \BB \ff = \dd .
\end{align*}
We call this problem \emph{electrical energy-minimizing flow}.
It turns out, that the flow $\ff^*$ that minimizes the electrical
energy among all flows that satisfy $\BB \ff = \dd$ is precisely the
electrical flow.

\paragraph{A pair of problems.}
What about our voltages, can we also get them from some optimization
problem?
%
Well, we can work backwards from the fact that our voltages solve the
equation $\LL \xx = \dd$.
%
Consider the function $c(\xx) = \frac{1}{2} \xx^{\trp} \LL \xx -
\xx^{\trp}\dd$.
We should ask ourselves some questions about this function $c : \R^V
\to \R$. Is it continuous and continuously differentiable? The answer to this is
yes, and that is not hard to see.
Does the function have a minimum?
This is maybe not immediately clear, but the minimum does indeed exist.

When this is minimized, the derivative of $c(\xx)$ with respect to each coordinate of $\xx$
must be zero. This condition yields exactly the system of linear
equations $\LL \xx = \dd$.
You will confirm this in Exercise 4.

Based on our derivative condition for the optimum, we can also express the electrical voltages as the solution to
an optimization problem, namely
\begin{align*}
\min_{\xx \in \R^V} & \frac{1}{2} \xx^{\trp} \LL \xx -
\xx^{\trp}\dd
\end{align*}
As you are probably aware, having the derivative of each coordinate equal zero is not a
sufficient condition for being at the optimum of a
function\footnote{Consider the function in one variable $c(x) =
  x^3$.}.
It is also interesting to know whether \emph{all} solutions to $\LL
\xx = \dd$ are in fact minimizers of $c$. The answer is
yes, and will see some very general tools for proving statements like
this in Chapter~\ref{sec:cvopt}.
% Although, we have not yet proven the electrical flow minimizes
% electrical energy $\sum_e \rr(e) \ff(e)^2$, I have promised you that
% this is true.
% Let us denote the electrical flow and voltages by $\fftil$ and $\xxtil$,
% and recall that $\fftil = \RR^{-1} \BB ^\trp \xxtil$.
% Based on this, we can make an interesting observation:
% \[
%   \sum_e \rr(e) \ff(e)^2
%   =
%   \ff^{\trp} \RR \ff
%   =
  % \]
  % \frac{1}{2} \xxtil^{\trp} \LL \xxtil - \xxtil^{\trp}\dd
  % =

Altogether, we can see that routing electrical current through a
network of resistors leads to a \emph{pair} of optimization problems,
let's call them $\ff^*$ and $\xx^*$,
and that the solutions to the two problems are related, in our case
through the equation $\ff^* = \RR^{-1}\BB^{\trp} \xx^*$.
In Exercise 5, you will explore this relationship more.
\\

This turns out to be an instance of a much broader phenomenon, known
as Lagrangian duality, which allows us to learn a lot about many
optimization problems by studying two related pairs of problems.

\paragraph{Solving $\LL \xx = \dd$.}
Given a graph $G$ with resistances for the edges, and some net flow
vector $\dd$, how quickly can we compute $\xx$?
%
Broadly speaking, there are two very different families of algorithms
we could use to try to solve this problem.

Either, we could solve the linear equation using something like
\emph{Gaussian Elimination} to compute an exact solution.

Alternatively,
we could start with a guess at a solution, e.g. $\xx_0
= \veczero$, and then we could try to make a change to $\xx_0$ to reach
a new point $\xx_1$ with a lower value of $c(\xx) = \frac{1}{2} \xx^{\trp} \LL \xx -
\xx^{\trp}\dd$, i.e. $c(\xx_1) < c(\xx_0)$.
If we repeat a process like that for enough steps, say $t$, hopefully we
eventually reach $\xx_t$ with $c(\xx_t)$ close to $c(\xx^*)$, where
$\xx^*$ is a minimizer of $c(\xx)$ and hence $\LL \xx^* = \dd$.
Now, we also need to make sure that $c(\xx_t) \approx c(\xx^*)$
implies that $\LL \xx_t \approx \dd$ in some useful sense.

One of the most basic algorithms in this framework of ``guess and adjust''
is called \emph{Gradient Descent}, which we will study in two weeks.
The rough idea is the following: if we make a very small step from $\xx$ to
$\xx + \ddelta$, then a multivariate Taylor expansion suggests that
$c(\xx + \ddelta) - c(\xx) \approx \sum_{v \in V} \ddelta(v)
\frac{\partial c(\xx)}{\partial\xx(v)} $.

If we are dealing with smooth convex function,
this quantity is negative if we let $\ddelta(v) = -\epsilon \cdot \frac{\partial
  c(\xx)}{\partial\xx(v)}$ for some small enough $\epsilon$ so the
approximation holds well.
So we should be able to make progress by taking a small step in this
direction.
That's Gradient Descent!
The name comes from the vector of partial
derivatives, which is called the gradient.
% But first we need to become familiar with some basic terminology about
% optimization and some convex geometry.
% As we will see later in this course, understanding electrical problems
% from an optimization perspective is crucial to developing fast
% algorithms.

As we will see later in this course, understanding electrical problems
from an optimization perspective is crucial to developing fast
algorithms for computing electrical flows and voltages, but to do very
well, we also need to borrow some ideas from Gaussian Elimination.

What running times do different approaches get?
\begin{enumerate}
\item Using Gaussian Elimination, we can find $\xx$ s.t. $\LL \xx =
  \dd$ in $O(n^3)$ time and with asymptotically faster algorithms based on matrix
  multiplication, we can bring this down to roughly $O(n^{2.372})$.
\item Meanwhile Gradient Descent will get a running time of $O(n^3m)$
  or so -- at least this is a what a simple analysis suggests.
\item However, we can do much better: By combining ideas from both
  algorithms, and a bit more,
  we can get $\xx$ up to very high
  accuracy in time $O(m \log^c n)$ where $c$ is some small constant.
  % This type of result was orignally shown by Spielman and Teng in a
  % breakthrough result in 2004.
\end{enumerate}

\section{Convex Optimization}

Recall our plot in Figure~\ref{fig:flow-vs-energy} of the energy required to
route a flow $f$ across a resistor with resistance $r$, which was
$\energy(f) = r\cdot f^2$.
%
We see that the function has a special structure: the graph of the function sits
below the line joining any two points $(f, \energy(f))$ and $(g,\energy(g))$.
A function $\energy : \R \to \R$ that has this property is said to be convex.

Figure~\ref{fig:flow-vs-energy-convex} shows the energy as a function of flow, along with two
points $(f, \energy(f))$ and $(g,\energy(g))$. We see the function sits below the
line segment between these points.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.6\linewidth]{fig/lecture1_flow-vs-energy-convex.png}
  \captionof{figure}{Energy has a function of flow in a resistor with
    resistance $r = 1$. The function is convex.}
  \label{fig:flow-vs-energy-convex}
\end{figure}
We can also interpret this condition as saying that for all $\theta \in [0,1]$
\[
  \energy(\theta f + (1-\theta)g) \leq \theta \energy(f) +
  (1-\theta) \energy(g)
  .
  \]
This immediately generalizes to functions $\energy:\R^m \to \R$.

A \emph{convex set} is a subset of $S \subseteq \R^m$ s.t.
if $\ff, \gg \in S$ then for all $\theta \in [0,1]$ we have
$\theta \ff + (1-\theta) \gg \in S$.

Figure~\ref{fig:convex_sets} shows some examples of sets that are and
aren't convex.

\begin{figure}[t]
\begin{centering}
                \includegraphics[trim = 0mm 0mm 0mm 0mm, height=60mm]{fig/lec1_convex_sets.pdf}
                 \caption{A depiction of convex and non-convex sets.  The sets $A$ and $B$ are convex since the straight line between any two points inside them is also in the set.  The set $C$ is not convex.}\label{fig:convex_sets}
                 \end{centering}
\end{figure}

Convex functions and convex sets are central to optimization,
because for most problems of minimization a convex function over a
convex set, we can develop fast algorithms \footnote{
 There are some convex optimization problems that are NP-hard.
  That said, polynomial time algorithms exist for almost any convex
  problem you can come up with.
  The most general polynomial time algorithm for convex optimization
  is probably the
  \href{https://en.wikipedia.org/wiki/Ellipsoid_method}{Ellipsoid Method}.
}.

So why convex functions and convex sets?
One important reason is that
for a convex function defined over a convex feasible set,
any local minimum is also a global minimum,
and this fact makes searching for an optimal solution
computationally easier.
In fact, this is closely related to why Gradient Descent works well
on many convex functions.

Notice that the set $\setof{ \ff: \BB \ff = \dd}$ is convex, i.e. the
set of all flows that route a fixed demand $\dd$ is convex.
It is also easy to verify that $\energy(\ff) = \sum_e \rr(e) \ff(e)^2$
is a convex function, and hence finding an electrical flow is an
instance of convex minmization:

\section{More Graph Optimization Problems}

\paragraph{Maximum flow.}
Again, let $G=(V,E)$ be an undirected, connected graph with $n$
vertices and $m$ edges.
Suppose we want to find a flow
$\ff \in \R^E$ that routes $\dd$, but instead of trying to minimize
electrical energy, we try to pick an $\ff$ that minimizes the largest
amount of flow on any edge, i.e. $\max_e \abs{\ff_e}$  -- which we
also denote by $\norm{\ff}_{\infty}$.
We can write this problem as
\begin{align*}
\min_{\ff \in \R^E } & \norm{\ff}_{\infty} \\
\textrm{s.t. } & \BB \ff = \dd
\end{align*}
This problem is known as the Minimum Congested Flow Problem\footnote{This
  version is called undirected, because the graph is undirected, and
  \emph{uncapacitated} because we are aiming for the same
  bound on the flow on all edges.}.
It is equivalent to the more famous Maximum Flow Problem.

The behavior of this kind of flow is very different than electrical
flow. Consider the question of whether a certain demand can be routed
$\norm{\ff}_{\infty} \leq 1$.
Imagine sending goods from a source $s$ to a destination $t$ using a
network of train lines that all have the
same capacity and asking whether the network is able to route the
goods at the rate you want: This boils down to whether routing exists
with $\norm{\ff}_{\infty} \leq 1$, if we set it up right.

We have a very fast, convex optimization-based algorithm for Minimum Congested Flow:
In $m \epsilon^{-1} \log^{O(1)} n$ time, we can find a flow $\fftil$
s.t. $\BB \fftil = \dd$ and $\norm{\fftil}_{\infty} \leq
(1+\epsilon)\norm{\ff^*}_{\infty}$, where $\ff^*$ is an optimal solution, i.e. an
actual minimum congestion flow routing $\dd$.

But what if we want $\epsilon$ to be very small, e.g. $1/m$? Then this running
time isn't so good anymore.
But, in this case, we can use another algorithm, that finds an optimal
flow $\ff^*$ \emph{exactly}, in time\footnote{And there's even a paper on arXiv.org that brings this further
  down to $m^{11/8} \log^{O(1)} n$.} $m^{10/7} \log^{O(1)} n $.

Just as the electrical flow problem had a dual voltage problem, so
maximum flow has a dual voltage problem, which is know as the
$s$-$t$ minimum cut problem.

\paragraph{Maximum flow, with directions and capacities.}
We can make the maximum flow problem harder by introducing directed
edges: To do so, we allow edges in both directions to exist between a
vertex to exist, and we require that that flow on a directed edge is
always non-negative. So now $G=(V,E)$ is a directed graph.
We can also make the problem harder by introducing capacities.
We define a capacity vector $\cc \in \R^E \geq \veczero$ and try to minimize $\norm{\CC^{-1} \ff}_{\infty}$, where $\CC =
\diag_{e \in E}\cc(e)$.
Then our problem becomes
\begin{align*}
\min_{\ff \in \R^E } & \norm{\CC^{-1} \ff}_{\infty} \\
  \textrm{s.t. } &  \BB \ff = \dd\\
                     &  \ff \geq \veczero.
\end{align*}
For this capacitated, directed maximum flow problem, our best
algorithms run in about $O( m \sqrt{n} )$ time\footnote{Provided the
  capacities are integers satisfying a condition like $\cc \leq n^{100} \vecone$.}, even if we are willing to
accept fairly low accuracy solution.
If the capacities are allowed to be exponentially large, the best
running time we can get is $O(m n)$.
For this problem, we do not yet know how to improve over classical
combinatorial algorithms using convex optimization.

\paragraph{Multi-commodity flow.}
We can make the even harder still, by simultaneously trying to route
to types of flow (imagine pipes with Coke and Pepsi).
Our problem now looks like
\begin{align*}
\min_{\ff_1, \ff_2 \in \R^E } & \norm{\CC^{-1} (\ff_1 + \ff_2)}_{\infty} \\
  \textrm{s.t. } &  \BB \ff_1 = \dd_1 \\
                              & \BB \ff_2 = \dd_2\\
                     &  \ff_1, \ff_2 \geq \veczero.
\end{align*}
Solving this problem to high accuracy is essentially as hard as
solving a general linear program! We should see later in the course
how to make this statement precise.

If we in the above problem additionally require that our flows must be
integer valued, i.e. $\ff_1, \ff_2 \in \N_{0}$, then the problem becomes NP-complete.

\paragraph{Random walks in a graph.}
Google famously uses\footnote{At least they did at some point.} the
PageRank problem to help decide how to rank their search results.
This problem essentially boils down to computing the \emph{stable
distribution} of a random walk on a graph.
Suppose $G=(V,E)$ is a directed graph where each edge outgoing edge
$(v,u)$, which we will define as going from $u$ to $v$, has a
transition probability $p_{(v,u)} > 0$ s.t. $\sum_{z \leftarrow u}
p_{(z,u)} = 1$.
We can take a step of a random walk on the vertex set by starting at some vertex
$u_0 = u$, and then randomly picking one of it the outgoing edges
$(v,u)$ with probability $p_{(v,u)}$  and move to
the chosen vertex $u_1 = v$.  Repeating this procedure, to take a step from
the next vertex $u_1$, gives us a
\emph{random walk} in the graph, a sequence of vertices $u_0, u_1, u_2
\ldots, u_k$.

We let $\PP \in \R^{V \times V}$ be the matrix of transition
probabilities given by
\[
  \PP_{vu} =
  \begin{cases}
    p_{(v,u)} & \text{ for } (u,v) \in E \\
    0 & \text{ o.w.}
  \end{cases}
\]

Any probability distribution over the vertices can be specified by a vector
$\pp \in \R^V$ where $\pp \geq
\veczero$ and $\sum_v \pp(v) = 1$.
We say that probability distribution $\ppi$ on the vertices is a \emph{stable distribution} of the random walk
if $\ppi = \PP \ppi$.
A strongly connected graph always has exactly one stable distribution.

How quickly can we compute the stable distribution of a general random
walk? Under some mild conditions on the stable
distribution\footnote{Roughly something like $\max_{v} 1/\ppi(v) \leq n^{100}$.}, we can find
a high accuracy approximation of $\ppi$ in time $O(m \log^cn )$ for
some constant $c$.

This problem does not easily fit in a framework of convex
optimization, but nonetheless, our fastest algorithms for it use ideas
from convex optimization.

\section*{Topics in this Course}
In this course, we will try to address the following questions.
\begin{enumerate}
\item What are the fundamental tools of fast convex optimization?
\item What are some problems we can solve quickly on graphs using optimization?
\item What can graphs teach us about convex optimization?
\item What algorithm design techniques are good for getting algorithms
  that quickly find a crude approximate solution? And what techniques
  are best when we need to get a highly accurate answer?
\item What is special about flow problems?
\end{enumerate}

% \pagebreak
% \section{Exercises}

% NEW
% \begin{enumerate}
% \item sub-level set of a convex function is a convex set (show).
% \item is it true that a function whose sub-level sets are all convex
%   is necessarily convex?
% \item \todo{exercise A}
%   \begin{itemize}
%   \item kernel of the Laplacian
%   \item show that solutions (voltage and flow) exist when
%     graph is connected and $\dd^\trp \vecone = 0$.
%   \item prove matrix symmetric, consider quadratic form,
%     show positive semi-definite.
%   \end{itemize}
% \item \todo{exercise B}
%     define gradient. do the voltage gradient computation
% \item \todo{exercise C: more duality}
% {\color{olive}
%   \\
% Recall $\ff^* = \RR^{-1}\BB^{\trp} \xx^*$ and $\BB \ff^* = \dd$,
% we can conclude
% $(\ff^* )^\trp R \ff^*  = (\ff^* )^\trp \BB^\trp \xx^* = \dd^\trp \xx^* = (\xx^*)^\trp \LL
% \xx^*$
% (Maybe don't directly point this out to students!)

% now we can see that
% \begin{align*}
% \max_{\xx \in \R^V} \xx^{\trp}\dd - \frac{1}{2} \xx^{\trp} \LL \xx
%   &=
% \min_{\ff \in \R^E} \frac{1}{2}\sum_e \rr(e) \ff(e)^2 \\
%   & \qquad \textrm{ s.t. }
%   \BB \ff = \dd
% .
% \end{align*}

% Maybe also point out to students that the stuff they have proved can
% easily be generalized a larger class of quadratic problems.
% }
% \item GRADED? gaussian elimination for Laplacians is closed ?
% \end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "agao21_script"
%%% TeX-engine: luatex
%%% End:


\documentclass[12pt]{amsart}
\usepackage[margin=0.5in]{geometry}
  % see geometry.pdf on how to lay out the page. There's lots.
\usepackage{bsymb}
\usepackage{../unitb}
\usepackage{calculational}
\usepackage{ulem}
\usepackage{hyperref}

\newcommand{\REQ}{\text{REQ}}

\begin{document}
  % \input{main9/machine_m0.tex}
\begin{machine}{m0}

  \newset{\REQ}

We need a variable to keep track of all the requests to mutate the
data structure.

  \[ \variable{ req,req_0 : \set [\REQ] } \]

... and new events:
% % \hide{
  \newevent{req}{req}
  \newevent{handle}{handle}
  % \newevent{req}{req\_b}
  % \newevent{handle}{handle\_b}

\[\param{req}{r : \REQ }\]
\with{sets}
\begin{align*}
  & \evguard{req}{grd0}{ \neg r \in req }  \\
  & \evbcmeq{req}{act0}{ req }{ req \bunion \{ r \} } \\
  & \evbcmeq{req}{act1}{ req_0 }{ req }
\end{align*}
\[\param{handle}{r : \REQ }\]
\begin{align*}
  \evguard{handle}{grd0}{ r \in req }  \\
  \evbcmeq{handle}{act0}{ req }{ req \setminus \{ r \} } \\
  \evbcmeq{handle}{act1}{ req_0 }{ req }
\end{align*}

% \section{Requirements}
%   \dummy{ R : \set[\REQ] }
\begin{align*}
  \constraint{co0}{ req_0' = req \1\lor (req_0' = req_0 \1\land req' = req) }
\end{align*}
\begin{align*}
  & \progress{prog0}
    { \neg req = \emptyset }
    { \neg req_0 \subseteq req }
 \refine{prog0}{ensure}{handle}{ using \ref{handle} }
  & \progress{prog1}
    { V = req}
    { req \subset V
      \1\lor req = \emptyset \1\lor \neg req \subseteq req_0}
 \refine{prog1}{ensure}{handle}{ using \ref{handle} }
\end{align*}
\begin{align*}
  \dummy{ R,V : \set [\REQ] } \\
  \cschedule{handle}{m0:sch0}{ \neg req = \emptyset } \\
  \initialization{m0:in0}{ req = \emptyset }
\end{align*}
\end{machine}

% \input{main9/machine_m1}
\begin{machine}{m1}
  \refines{m0}
\[ \indices{handle}{ b : \Bool } \]
\[ \dummy{ b : \Bool } \]
\[ \variable{ ch : \Bool } \]
\begin{align*}
  & \cschedule{handle}{m1:sch0}{ b = ch } \\
  & \witness{handle}{b}{b = ch} \\
  & \progress{m1:prog1}{ b = ch }{ b = ch }
\end{align*}
\replace{handle}{m1:sch0}{m1:prog1}
  \begin{liveness}{m1:prog1}
    \progstep{\true}{req = \emptyset \1\lor \neg req \subseteq req_0}
      {induction}{}{ \var{req}{down}{\emptyset} }
  \begin{flatstep}
    \progstep
      {V = req}
      {req \subset V
        \1\lor req = \emptyset \1\lor \neg req \subseteq req_0}
      {discharge}{}{}
      \begin{step}
        \trstep{handle}{ \index{b}{ \true }  }
          { req = V \1\land \neg req = \emptyset  }
        \safstep
          { V = req }
          { req \subset V \1\lor \neg req \subseteq req_0 }
          {}
      \end{step}
  \end{flatstep}
  \end{liveness}
\end{machine}

% % \input{main9/machine_m2}
% \begin{machine}{m2}
%   \refines{m0}

% We now partitial $req$ into requests for operation A ($reqA$) and
% requests for operation B ($reqB$).

% \[ \variable{ reqA, reqB : \set [\REQ] } \]
% \begin{align*}
%   \invariant{m1:inv0}{ reqA \bunion reqB = req } \\
%   \invariant{m1:inv1}{ reqA \binter reqB = req }
% \end{align*}
% And consequ
% \begin{align*}
%   \initialization{m1:in0}{ reqA = \emptyset \land reqB = \emptyset }
% \end{align*}
% \end{machine}

\end{document}