\section{Combining isochrone fitting with Gyrochronology: Motivation}
\label{section:motivation}

In order to demonstrate why a combination of gyrochronology and isochrone
fitting can provide more precise ages than either method used in isolation, we
calculated the information provided by each method for a range of stellar
masses, ages and evolutionary stages.

According to observations of cluster stars and the Sun, the decrease in
rotation period with time is roughly proportional to the inverse square root
of age, $\frac{dP_{\mathrm{rot}}}{dt} \propto \mathrm{Age}^{-n}$, where
n$\sim$0.5.
This corresponds to a large rate of change relative to typical rotation
period measurement uncertainties.
For example, the Sun's rotation period is currently decreasing at a rate of
around 3 days per billion years \citep[unless it has already stopped spinning
down, \eg][]{vansaders2016}, and the 1 billion year-old Sun
spun down at a rate of around 6 days per billion years.
These are relatively large changes compared with the average uncertainties on
rotation period measurements: the median rotation period uncertainty in the
\citet{mcquillan2014} catalog is around 0.1 days.
In contrast, the temperature of a K dwarf changes by about 20 K every billion
years which is small compared to typical observational uncertainties of 20-100
K.
Rotational isochrones, or `gyrochrones' provide much more {\it information}
about age than traditional isochrones.
The difference in information conveyed by rotation vs \teff\ and $L$ can be
quantified by calculating the time derivatives of a star's observables.
The rate of change of \teff\ and $L$ dictates the minimum theoretically
achievable uncertainty on an age inferred via isochrone fitting, given some
observational uncertainties.
Similarly, the rate of change of rotation period dictates the minimum
achievable uncertainty on an age inferred via gyrochronology.
In order to quantify the minimum theoretical uncertainty on ages calculated
via isochrone fitting and gyrochronology, we calculated the Fisher information
for the MIST isochrones \citep{paxton2011, paxton2013, paxton2015, dotter2016,
choi2016, paxton2018} and an empirical polynomial gyrochronology model we fit
to the Praesepe cluster.
The Fisher information quantifies the amount of information that an observable
imparts onto an unknown parameter.
In the case of isochrone fitting on a CMD using \Gaia\ data, the observables
are \Gaia\ absolute magnitude $M_G$ and color, \gcolor\ and the parameter is
age, or time, $t$.
The Fisher information is the variance of the parameter, $t$, given the
covariance of the observables and their derivatives with respect to $t$.
The inverse covariance matrix of the parameters (in this case we have just one
parameter, age or time, $t$), given the covariance matrix of the data,
${\bf y} = [M_G, G_{BP} - G_{RP}]$, is given by the following equation,
\begin{equation}
    C_{\mathrm{Age}}^{-1} = \left[\frac{d{\bf y}}{dt}\right]^T
    C_{\bf y}^{-1} \left[\frac{d{\bf y}}{dt}\right].
\end{equation}
Since we just have one parameter, $C_\mathrm{Age}^{-1}$ is a scalar, the
inverse variance of age, $\sigma_{\mathrm{Age}}^{-2}$.
In order to calculate the age uncertainty from the MIST isochrones, we
calculated numerical derivatives of $\frac{dG}{dt}$,
and $\frac{d(G_{BP} - G_{RP})}{dt}$ at every point on the MIST model grids.
We then calculated the isochronal age uncertainty, $\sigma_{\mathrm{Age}}$ at
every point on the grid.
Figure \ref{fig:iso_fisher} shows Solar-metallicity MIST isochrones, colored
by $\sigma_{\mathrm{Age}}$.

Figure \ref{fig:iso_fisher}\footnote{Figures \ref{fig:iso_fisher} and
\ref{fig:gyro_fisher} were generated in a {\it Jupyter} notebook available at
the following url:
\url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Fisher_information.ipynb}}
shows the minimum theoretical absolute age uncertainty,
$\sigma_{\mathrm{Age}}$ (left panel), calculated using typical uncertainties
on \Gaia\ absolute magnitude, $M_G$, of $\pm$0.05 and a color spread of 0.05
that (conservatively) accounts for scatter induced by metallicities ranging
from -0.25 to +0.25 dex.
These uncertainties are represented as black errorbars in the top right
corner.
The typical \Gaia\ uncertainties are $0.5$ in both $M_G$ and \gcolor.
These estimates are based on a calculation of the median uncertainty on \Gaia\
absolute G-magnitude of cool stars which is dominated by the parallax
uncertainty.
We assumed the same uncertainty on \gcolor.
The minimum uncertainty on isochronal age ranges from around 10 million years
at MS turn off (upper left yellow area) to around the age of the Universe for
K dwarfs (middle to lower-right blue area).
The minimum {\it relative} age uncertainty,
$\sigma_{\mathrm{Age}}/\mathrm{Age} \times 100$, plotted in the right-hand
panel ranges from less than 1\% for old MS turn off stars with ages
around 13 Gyr and age uncertainties less than 0.1 Gyr, up to tens-of-thousands
of percent for the youngest K and M dwarfs with unconstrained ages.

We also calculated the Fisher information for a {\it combined isochronal and
gyrochronology model.}
In this case we effectively had four observables: $M_G$ and \gcolor,
determined by the MIST isochrones; and $P_{\mathrm{rot}}$ (rotation period)
and \gcolor\ {\it again}, this time determined by the gyrochronology model.
We used a simple gyrochronology model, calibrated by fitting a fourth-order
polynomial in rotation period-\Gaia\ color space and a first order polynomical
in rotation period-age space to the 650 Myr Praesepe cluster and the Sun,
only.
This model is described in more detail later in this section.
We calculated analytic derivatives for $\frac{dP_{\mathrm{rot}}}{dt}$ and
$\left(\frac{d(G_{BP} - G_{RP})}{dt}\right)_{\mathrm{gyro}}$ and combined
these with the numerical derivatives of $\frac{dG}{dt}$ and
$\left(\frac{d(G_{BP} - G_{RP})}{dt}\right)_{\mathrm{iso}}$ in order to
calculate the total age uncertainty, $\sigma_{\mathrm{Age~(iso~\&~gyro)}}$.
Figure \ref{fig:gyro_fisher} shows Solar-metallicity MIST isochrones, colored
by $\sigma_{\mathrm{Age~(iso~\&~gyro)}}$.
The results are presented the same way as figure \ref{fig:iso_fisher} however,
figure \ref{fig:gyro_fisher} shows age uncertainties calculated using
isochrones {\it and a polynomial gyrochronology model}.
The age uncertainties were calculated using typical \Gaia\ $M_G$ and \gcolor\
uncertainties, represented as black errorbars in the top right corner, and
rotation period uncertainties of 1 day.
The minimum theoretical absolute age uncertainty inferred using gyrochronology
and isochrone fitting simultaneously, $\sigma_{\mathrm{Age~(iso~\&~gyro)}}$
(left panel of figure \ref{fig:gyro_fisher}), ranges from tens of millions of
years for stars at MS turn off to a few billion years (up to around 3 Gyr for
old G dwarfs).
The very precise ages at MS turn off are still provided by isochrone fitting
-- the incredible precision achievable with isochrone fitting at MS turn off
dominates over the precision provided by gyrochronology.
On the MS however, gyrochronology provides extremely precise ages and its
precision dominates over isochrone fitting.
The gyrochronology model used to calculate the Fisher information is not
appropriate for stars turning off the MS as it does not account for a rapid
decrease in rotation period that may be caused by the stellar radius
increasing \citep[see][]{vansaders2013}.
% However, it provides an upper limit on $\sigma_{\mathrm{Age,~gyro}}$ which is
% is, in any case, dominated by $\sigma_{\mathrm{Age,~iso}}$.
The right-hand panel of figure \ref{fig:gyro_fisher} shows the {\it relative}
age uncertainty achievable with joint isochronal and gyrochronal age
inference.
Relative age uncertainty,
$\mathrm{Age}/\sigma_{\mathrm{Age~(iso~\&~gyro)}}\times 100$ ranges from less
than 1\% at MS turn off, where isochrones provide precise ages because they
are widely spaced, to a maximum of around 30\% for young G dwarfs, where
gyrochrones are at their most tightly spaced.
The dramatic improvement in age precision seen across the MS when
gyrochronology is used provides the motivation for combining isochrone fitting
with gyrochronology.
The minimum relative age uncertainty for GKM stars on the MS is typically
around 20\% -- gyrochronology predicts precise ages for these kinds of stars.
20\% age precision for gyrochronology was also predicted in previous studies.
\citep{epstein2014}.
Gyrochronology and isochrone fitting are extremely complementary:
gyrochronology contributes most of the precision on the MS because rotation
period information dominates over color and luminosity information, however
it is not applicable to hot stars without deep convection zones
and evolved stars.
These are precisely the stars optimally fitted with isochrone models.
\cocomment{This analysis does raise a question however.
Combining two independent dating methods is useful when both are contributing
information, \ie\ near MS turn off, however given that isochrone fitting
provides so little information on the MS, is there any point in including it
there at all?
Why not just abandon isochrone fitting and use gyrochronology exclusively?
Although isochrones and stellar evolution tracks do not provide precise ages
for MS stars, they do provide {\it masses}, and mass is essential for
obtaining a precise gyrochronal age.
Conversely, gyrochronology can actually enhance mass measurements too by
providing a tighter age constraint: since mass and age are weakly correlated,
this results in a tighter mass constraint.
There are other good reasons to model stars with both isochrones and
gyrochronology on the MS.
Firstly, gyrochronology is not applicable to hot stars \citep[\teff $\gtrsim$
6250,][]{kraft1967} or evolved stars but it is not possible to know whether a
star is evolved, or hot and not just reddened, without modelling it.
Modelling stars with isochrones/stellar evolution tracks is required in order
to know whether gyrochronology is applicable or not.
Secondly, most gyrochronology models are calibrated to either $B-V$ color
(most commonly) or mass.
Mass is usually not directly observable, so must be inferred via modelsj and
$B-V$ is {\it also} not directly observed for every star.
For example, the most widely available colors are now \gaia\ \gcolor\ colors,
and most \Gaia\ stars do not have $B-V$ colors.
In order to calculate gyrochronal ages for stars with {\it any} apparent
magnitudes, not just $B-V$, it is necessary to model stars with isochrones and
gyrochronology simultaneously.}

An important caveat of this demonstration is that this is the minimum
theoretical precision given the {\it adopted} gyrochronology model and, since
the model used for this calculation does not include intrinsic scatter (which
is particularly large for young stars), these minimum age uncertainty
calculations are over-optimistic, especially for young stars.
Similarly, our model does not account for weakened magnetic braking at old
ages \citep{vansaders2016} so is also optimistic for old dwarfs.
Still, figures \ref{fig:iso_fisher} and \ref{fig:gyro_fisher} provide an idea
of the improvement provided by gyrochronology over isochrone fitting alone.
% The Praesepe model

In order to calculate analytic derivatives of the gyrochronology model, we
fit a linear model to the Praesepe open cluster and the Sun (see figure
\ref{fig:praesepe}.
We used a three-dimensional polynomial model to predict age as a
function of \gaia\ color and rotation period for Praesepe and the Sun.
This model consists of a 4th order polynomial in logarithmic Gaia color:
\gcolor, which we write as $C_G$ for simplicity, and a 1st order polynomial (a
straight line) in logarithmic age.
For this analysis, rotation periods for Praesepe were obtained from
\citet{douglas2017} and their \gaia\ colors were obtained by crossmatching
their sky-projected positions with the \gaia\ DR2 catalog.
% We used \gcolor\ instead of (B-V) because, due to the $\sim$ billion stars
% observed by \gaia, it is now the most abundant and widely available
% photometric color.
% Our gyrochronology likelihood function is designed to compare observed
% rotation period to predicted rotation period.
% For this reason the gyrochronology model we used must predict rotation period
% as a function of age and color.
When fitting this gyrochronology model, we chose to make {\it age} the
dependent variable because the uncertainties on stellar age are much greater
than the uncertainties on rotation period.
We fit the following model to Praesepe members:
\begin{equation}
    \log_{10}(A) = a + b\log_{10}(C_G) + c\log_{10}^2(C_G) +
    d\log_{10}^3(C_G) + e\log_{10}^4(C_G) + f\log_{10}(P)
\label{eqn:gyro_age_praesepe}
\end{equation}
where $P$ is rotation period in days, $C_G$ is Gaia color, $A$ is stellar age
in years and the lower case letters are free parameters.
We adopted an age for Praesepe of 650 $\pm$ 100 million years
\citep{fossati2008, gossage2018}, a Solar age of 4.56 $\pm$ 0.01 Gyr
\citep{connelly2012}, and a Solar rotation period of 26 days \citep[][Morris
\etal, in prep]{balthasar1986, howe2000}.
The Sun's color in the Gaia color is 0.82 \citep{casagrande2018}.
We found best-fit values: $a = 7.37 \pm 0.03,~b = -1.4 \pm 0.1,~c = 5.0 \pm
0.8,~d = -34 \pm 3,~e = 66 \pm 14$, and $f = 1.49 \pm 0.02$.
This model and the data it was fit to are plotted in figure
\ref{fig:praesepe}.
To be clear, this model was only used to calculate the Fisher information and
produce figures \ref{fig:iso_fisher} and \ref{fig:gyro_fisher}, because it has
simple analytic derivatives.
Since it was only fit to a single cluster and the Sun it is not generally
applicable and was not used in any other aspects of the analysis performed in
this paper.
The gyrochronology model used throughout the rest of the analysis is described
in section \ref{section:method} and equation \ref{eqn:gyro}.
% The $f$ parameter is the inverse of the slope of the rotation period and age
% which was originally measured to be around 0.5.
% Our Praesepe and Sun-only fit results in a slightly steeper age dependence of
% around 0.67, however this value is likely be
% We inverted this relation to predict rotation period as a function of color
% and age,
% \begin{equation}
%     \log_{10}(P) = \frac{\log_{10}(A) - a - b\log_{10}(C_G) - c\log_{10}^2(C_G) -
%     d\log_{10}^3(C_G) - e\log_{10}^4(C_G)}{f}.
% \label{eqn:gyro_age_praesepe}
% \end{equation}
% Both gyrochronology models of equations \ref{eqn:gyro} and
% \ref{eqn:gyro_age_praesepe} are used to predict the ages of individual
% Praesepe stars from their rotation periods and apparent magnitudes in section
% \ref{section:results}.

% \begin{figure}
%   \caption{
%     This figure shows Solar-metallicity MIST isochrones in \Gaia\ absolute
%     G-band magnitude and Gaia $G_{BP} - G_{RP}$ color.
%     In the left panel the isochrones are colored by the minimum absolute age
%     uncertainty at each point on the CMD, calculated using the Fisher
%     information.
%     This calculation is based on the typical uncertainties of
%     \Gaia\ $G$-band photometry, and a color spread that covers metallicities
%     ranging from -0.25 to 0.25 dex (represented by black errorbars in the
%     upper right).
%     The Sun's position \citep{casagrande2018} is indicated with the Solar\
%     symbol.
%     The purple color in the top left corresponds to small age uncertainties,
%     \ie\ good age precision.
%     Age precision increases as stars begin to turn off the MS.
%     On the MS however, particularly at low masses, the age precision is poor.
%     For late K dwarfs, for example, isochrone fitting age uncertainties exceed
%     the age of the Universe.
%     This makes sense when you consider that typical \Gaia\ uncertaintes on $G$
%     and $G_{BP} - G_{RP}$ exceed the entire width of the MS, which spans
%     0.01-14 Gyrs.
%     Isochrone fitting is not an appropriate age-dating method for MS stars,
%     especially at low masses.
%     In the right panel the isochrones are colored by the logarithmic
%     {\it relative} age precision at each point in the CMD.
%     Relative age uncertainties range from 100\% for the oldest MS GKM stars,
%     to several thousand percent for the youngest.
%     These age uncertainties were calculated using the derivatives of $G$ and
%     $G_{BP} - G_{RP}$ with age, \ie\ the rate of change in a star's luminosity
%     and temperature.
%     For example, K dwarf temperatures increase at a rate of only around 20 K
%     per billion years.
%     The precision with which an age can be measured is related to the
%     separation between isochrones, which indicate epochs of rapid change
%     (steep gradients).
%     This figure was created using the following Jupyter notebook:
%     \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Fisher_information.ipynb}
%     \label{fig:fischer_iso}
% }
%   \centering
%     \includegraphics[width=1\textwidth]{iso_fisher.pdf}
% \label{fig:iso_fisher}
% \end{figure}

% \begin{figure}
%   \caption{
%     As figure \ref{fig:iso_fisher}, however in this case the minimum age
%     uncertainties are calculated based on isochrone fitting and gyrochronology
%     {\it combined}.
%     The isochrones are colored by the minimum relative age uncertainty at each
%     point on the CMD, calculated using the Fisher information.
%     This calculation is based on the typical uncertainties of
%     \Gaia\ $G$-band photometry, a color spread that covers metallicities
%     ranging from -0.25 to 0.25 dex (represented by black errorbars in the
%     upper right), and Kepler rotation period uncertainties (conservatively
%     estimated at around 1 day).
%     In contrast to figure \ref{fig:iso_fisher}, here the left panel shows the
%     {\it logarithmic} absolute age and the right panel shows the linear
%     relative age.
%     The isochrones still supply precise ages at the MS turn off (purple, upper
%     left) however, gyrochronology supplies precise ages on the MS (15 - 25\%
%     relative precision).
%     Gyrochronology and isochrone fitting complement each other and when used
%     together, all subgiants and MS stars can have ages more precise than 30\%.
%     This figure was created using the following Jupyter notebook:
%     \url{https://github.com/RuthAngus/stardate/blob/master/paper/code/Fisher_information.ipynb}
%     \label{fig:fischer_gyro}
% }
%   \centering
%     \includegraphics[width=1\textwidth]{gyro_fisher.pdf}
% \label{fig:gyro_fisher}
% \end{figure}


\lab{Profiling}{Profiling}
\objective{Efficiency is essential to algorithmic programming.
Profiling is the process of measuring the complexity and efficiency of a program, allowing the programmer to see what parts of the code need to be optimized.
In this lab we present common techniques for speeding up Python code, including the built-in profiler and the Numba module.
}

\section*{Magic Commands in IPython} % ----------------------------------------

IPython has tools for quickly timing and profiling code.
These ``magic commands'' start with one or two \li{\%} characters---one for testing a single line of code, and two for testing a block of code.
\begin{itemize}
\item \li{<p<\%time>p>}: Execute some code and print out its execution time.
\item \li{<p<\%timeit>p>}: Execute some code several times and print out the average execution time.
\item \li{<p<\%prun>p>}: Run a statement through the Python code profiler,\footnote{{\color{purple}{\texttt{\%prun}}} is a shortcut for \texttt{cProfile.run()}; see \url{https://docs.python.org/3/library/profile.html} for details.} printing the number of function calls and the time each takes. We will demonstrate this tool a little later.
\end{itemize}

\begin{lstlisting}
# Time the construction of a list using list comprehension.
<g<In [1]:>g> <p<%time>p> x = [i**2 for i in range(int(1e5))]
<<CPU times: user 36.3 ms, sys: 3.28 ms, total: 39.6 ms
Wall time: 40.9 ms>>

# Time the same list construction, but with a regular for loop.
<g<In [2]:>g> <p<%%time>p>                      # Use a double %% to time a block of code.
   <g<...:>g> x = []
   <g<...:>g> for i in range(int(1e5)):
   <g<...:>g>     x.append(i**2)
   <g<...:>g>
<<CPU times: user 50 ms, sys: 2.79 ms, total: 52.8 ms
Wall time: 55.2 ms>>                  # The list comprehension is faster!
\end{lstlisting}

% Use \li{<p<\%time>p>} and \li{<p<\%timeit>p>} to select fast code snippets, functions, and algorithms (for example, using a list comprehension where possible instead of a regular loop).
% For the complete list of magic IPython commands, see \url{http://ipython.readthedocs.io/en/stable/interactive/magics.html}.

\subsection*{Choosing Faster Algorithms} % ------------------------------------

The best way to speed up a program is to use an efficient algorithm.
A bad algorithm, even when implemented well, is never an adequate substitute for a good algorithm.

\begin{problem} % Triangle path sums from Project Euler.
This problem comes from \url{https://projecteuler.net} (problems 18 and 67).

By starting at the top of the triangle below and moving to adjacent numbers on the row below, the maximum total from top to bottom is 23.
\begin{center}
\textbf{\color{red}{3}}\\
\textbf{\color{red}{7}} 4\\
2 \textbf{\color{red}{4}} 6\\
8 5 \textbf{\color{red}{9}} 3
\end{center}
That is, $3 + 7 + 4 + 9 = 23$.

The following function finds the maximum path sum of the triangle in \texttt{triangle.txt} by recursively computing the sum of every possible path---the ``brute force'' approach.
\begin{lstlisting}
def max_path(filename="triangle.txt"):
    """Find the maximum vertical path in a triangle of values."""
    with open(filename, 'r') as infile:
        data = [[int(n) for n in line.split()]
                        for line in infile.readlines()]
    def path_sum(r, c, total):
        """Recursively compute the max sum of the path starting in row r
        and column c, given the current total.
        """
        total += data[r][c]
        if r == len(data) - 1:      # Base case.
            return total
        else:                       # Recursive case.
            return max(path_sum(r+1, c,   total), # Next row, same column.
                       path_sum(r+1, c+1, total)) # Next row, next column.

    return path_sum(0, 0, 0)        # Start the recursion from the top.
\end{lstlisting}

The data in \texttt{triangle.txt} contains 15 rows and hence 16384 paths, so it is possible to solve this problem by trying every route.
However, for a triangle with 100 rows, there are $2^{99}$ paths to check, which would take billions of years to compute even for a program that could check one trillion routes per second.
No amount of improvement to \li{max_path()} can make it run in an acceptable amount of time on such a triangle---we need a different algorithm.

Write a function that accepts a filename containing a triangle of integers.
Compute the largest path sum with the following strategy: starting from the next to last row of the triangle, replace each entry with the sum of the current entry and the greater of the two ``child entries.''
Continue this replacement up through the entire triangle.
The top entry in the triangle will be the maximum path sum.
In other words, work from the bottom instead of from the top.

\begin{center}
\begin{tabular}{ccccccc}
\begin{tabular}{c}
3\\
7 4\\
2 4 6\\
\color{red}{8 5 9 3}
\end{tabular}
&$\longrightarrow$&
\begin{tabular}{c}
3\\
7 4\\
\color{red}{10 13 15}\\
\color{black}{8 5 9 3}
\end{tabular}
&$\longrightarrow$&
\begin{tabular}{c}
3\\
\color{red}{20 19}\\
\color{black}{10 13 15}\\
\color{black}{8 5 9 3}
\end{tabular}
&$\longrightarrow$&
\begin{tabular}{c}
\color{red}{\textbf{23}}\\
\color{black}{20 19}\\
\color{black}{10 13 15}\\
\color{black}{8 5 9 3}
\end{tabular}
\end{tabular}
\end{center}

Use your function to find the maximum path sum of the 100-row triangle stored in \texttt{triangle\_large.txt}.
Make sure that your new function still gets the correct answer for the smaller \texttt{triangle.txt}.
Finally, use \li{<p<\%time>p>} or \li{<p<\%timeit>p>} to time both functions on \texttt{triangle.txt}.
Your new function should be about 100 times faster than the original.
\end{problem}

\subsection*{The Profiler} % --------------------------------------------------

The profiling command \li{<p<\%prun>p>} lists the functions that are called during the execution of a piece of code, along with the following information.

\begin{table}[H]
\centering
\begin{tabular}{c|l}
    Heading & Description \\ \hline
    \li{primitive calls} & The number of calls that were not caused by recursion.\\
    \li{ncalls} & The number of calls to the function. If recursion occurs, the output\\ & is \texttt{<total number of calls>/<number of primitive calls>}.\\
    \li{tottime} & The amount of time spent in the function, not including calls to other functions.\\
    \li{percall} & The amount of time spent in each call of the function.\\
    \li{cumtime} & The amount of time spent in the function, including calls to other functions.\\
\end{tabular}
\end{table}

\begin{lstlisting}
# Profile the original function from Problem 1.
<g<In[3]:>g> <p<%prun>p> max_path("triangle.txt")
\end{lstlisting}

{\small
\begin{verbatim}
         81947 function calls (49181 primitive calls) in 0.036 seconds
   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  32767/1    0.025    0.000    0.034    0.034 profiling.py:18(path_sum)
    16383    0.005    0.000    0.005    0.000 {built-in method builtins.max}
    32767    0.003    0.000    0.003    0.000 {built-in method builtins.len}
        1    0.002    0.002    0.002    0.002 {method `readlines' of `_io._IOBase' objects}
        1    0.000    0.000    0.000    0.000 {built-in method io.open}
        1    0.000    0.000    0.036    0.036 profiling.py:12(max_path)
        1    0.000    0.000    0.000    0.000 profiling.py:15(<listcomp>)
        1    0.000    0.000    0.036    0.036 {built-in method builtins.exec}
        2    0.000    0.000    0.000    0.000 codecs.py:318(decode)
        1    0.000    0.000    0.036    0.036 <string>:1(<module>)
       15    0.000    0.000    0.000    0.000 {method `split' of `str' objects}
        1    0.000    0.000    0.000    0.000 _bootlocale.py:23(getpreferredencoding)
        2    0.000    0.000    0.000    0.000 {built-in method _codecs.utf_8_decode}
        1    0.000    0.000    0.000    0.000 {built-in method _locale.nl_langinfo}
        1    0.000    0.000    0.000    0.000 codecs.py:259(__init__)
        1    0.000    0.000    0.000    0.000 codecs.py:308(__init__)
        1    0.000    0.000    0.000    0.000 {method `disable' of `_lsprof.Profiler' objects}
\end{verbatim}
}

\section*{Optimizing Python Code} % ===========================================

A poor implementation of a good algorithm is better than a good implementation of a bad algorithm, but clumsy implementation can still cripple a program's efficiency.
The following are a few important practices for speeding up a Python program.
Remember, however, that such improvements are futile if the algorithm is poorly suited for the problem.

\subsection*{Avoid Repetition} % ----------------------------------------------

% {\small
% \begin{verbatim}
%    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
%   32767/1    0.025    0.000    0.034    0.034 profiling.py:18(path_sum)
%     16383    0.005    0.000    0.005    0.000 {built-in method builtins.max}
%     32767    0.003    0.000    0.003    0.000 {built-in method builtins.len}
%         1    0.002    0.002    0.002    0.002 {method `readlines' of `_io._IOBase' objects}
%        15    0.000    0.000    0.000    0.000 {method `split' of `str' objects}
% \end{verbatim}
% }

A clean program does no more work than is necessary.
The \li{ncalls} column of the profiler output is especially useful for identifying parts of a program that might be repetitive.
For example, the profile of \li{max_path()} indicates that \li{len()} was called 32,767 times---exactly as many times as \li{path_sum()}.
This is an easy fix: save \li{len(data)} as a variable somewhere outside of \li{path_sum()}.
\begin{lstlisting}
<g<In [4]:>g> def max_path_clean(filename="triangle.txt"):
   <g<...:>g>     with open(filename, 'r') as infile:
   <g<...:>g>         data = [[int(n) for n in line.split()]
   <g<...:>g>                         for line in infile.readlines()]
   <g<...:>g>     N = len(data)       # Calculate len(data) outside of path_sum().
   <g<...:>g>     def path_sum(r, c, total):
   <g<...:>g>         total += data[r][c]
   <g<...:>g>         if r == N - 1:  # Use N instead of len(data).
   <g<...:>g>             return total
   <g<...:>g>         else:
   <g<...:>g>             return max(path_sum(r+1, c,   total),
   <g<...:>g>                        path_sum(r+1, c+1, total))
   <g<...:>g>     return path_sum(0, 0, 0)
   <g<...:>g>
<g<In [5]:>g> <p<%prun>p> max_path_clean("triangle.txt")
\end{lstlisting}
{\small
\begin{verbatim}
         49181 function calls (16415 primitive calls) in 0.026 seconds
   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  32767/1    0.020    0.000    0.025    0.025 <ipython-input-5-9e8c48bb1aba>:6(path_sum)
    16383    0.005    0.000    0.005    0.000 {built-in method builtins.max}
        1    0.002    0.002    0.002    0.002 {method `readlines' of `_io._IOBase' objects}
        1    0.000    0.000    0.000    0.000 {built-in method io.open}
        1    0.000    0.000    0.026    0.026 <ipython-input-5-9e8c48bb1aba>:1(max_path_clean)
        1    0.000    0.000    0.000    0.000 <ipython-input-5-9e8c48bb1aba>:3(<listcomp>)
        1    0.000    0.000    0.027    0.027 {built-in method builtins.exec}
       15    0.000    0.000    0.000    0.000 {method `split' of `str' objects}
        1    0.000    0.000    0.027    0.027 <string>:1(<module>)
        2    0.000    0.000    0.000    0.000 codecs.py:318(decode)
        1    0.000    0.000    0.000    0.000 _bootlocale.py:23(getpreferredencoding)
        2    0.000    0.000    0.000    0.000 {built-in method _codecs.utf_8_decode}
        1    0.000    0.000    0.000    0.000 {built-in method _locale.nl_langinfo}
        1    0.000    0.000    0.000    0.000 codecs.py:308(__init__)
        1    0.000    0.000    0.000    0.000 codecs.py:259(__init__)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.len}
        1    0.000    0.000    0.000    0.000 {method `disable' of `_lsprof.Profiler' objects}
\end{verbatim}
}

Note that the total number of primitive function calls decreased from 49,181 to 16,415.
Using \li{<p<\%timeit>p>} also shows that the run time decreased by about 15\%.
Moving code outside of a loop or an often-used function usually results in a similar speedup.

Another important way of reducing repetition is carefully controlling loop conditions to avoid unnecessary iterations.
Consider the problem of identifying Pythagorean triples, sets of three distinct integers $a < b < c$ such that $a^2 + b^2 = c^2$.
The following function identifies all such triples where each term is less than a parameter $N$ by checking all possible triples.

\begin{lstlisting}
>>> def pythagorean_triples_slow(N):
...     """Compute all pythagorean triples with entries less than N."""
...     triples = []
...     for a in range(1, N):               # Try values of a from 1 to N-1.
...         for b in range(1, N):           # Try values of b from 1 to N-1.
...             for c in range(1, N):       # Try values of c from 1 to N-1.
...                 if a**2 + b**2 == c**2 and a < b < c:
...                     triples.append((a,b,c))
...     return triples
...
\end{lstlisting}

Since $a < b < c$ by definition, any computations where $b \le a$ or $c \le b$ are unnecessary.
Additionally, once $a$ and $b$ are chosen, $c$ can be no greater than $\sqrt{a^2 + b^2}$.
The following function changes the loop conditions to avoid these cases and takes care to only compute $a^2 + b^2$ once for each unique pairing $(a,b)$.

\begin{lstlisting}
>>> from math import sqrt
>>> def pythagorean_triples_fast(N):
...     """Compute all pythagorean triples with entries less than N."""
...     triples = []
...     for a in range(1, N):               # Try values of a from 1 to N-1.
...         for b in range(a+1, N):         # Try values of b from a+1 to N-1.
...             _sum = a**2 + b**2
...             for c in range(b+1, min(int(sqrt(_sum))+1, N)):
...                 if _sum == c**2:
...                     triples.append((a,b,c))
...     return triples
...
\end{lstlisting}

These improvements have a drastic impact on run time, even though the main approach---checking by brute force---is the same.

\begin{lstlisting}
<g<In [6]:>g> <p<%time>p> triples = pythagorean_triples_slow(500)
<<CPU times: user 1min 51s, sys: 389 ms, total: 1min 51s
Wall time: 1min 52s>>         # 112 seconds.

<g<In [7]:>g> <p<%time>p> triples = pythagorean_triples_fast(500)
<<CPU times: user 1.56 s, sys: 5.38 ms, total: 1.57 s
Wall time: 1.57 s>>           # 98.6% faster!
\end{lstlisting}

\begin{problem}
The following function computes the first $N$ prime numbers.
\begin{lstlisting}
def primes(N):
    """Compute the first N primes."""
    primes_list = []
    current = 2
    while len(primes_list) < N:
        isprime = True
        for i in range(2, current):     # Check for nontrivial divisors.
            if current % i == 0:
                isprime = False
        if isprime:
            primes_list.append(current)
        current += 1
    return primes_list
\end{lstlisting}
This function takes about 6 minutes to find the first 10,000 primes on a fast computer.

Without significantly modifying the approach, rewrite \li{primes()} so that it can compute 10,000 primes in under 0.1 seconds.
Use the following facts to reduce unnecessary iterations.
\begin{itemize}
\item A number is not prime if it has one or more divisors other than 1 and itself.
\\(Hint: recall the \li{break} statement.)
\item If $p\nmid n$, then $ap\nmid n$ for any integer $a$.
Also, if $p \mid n$ and $0 < p < n$, then $p \le \sqrt{n}$.
\item Except for $2$, primes are always odd.
\end{itemize}
Your new function should be helpful for solving problem 7 on \url{https://projecteuler.net}.
\label{prob:profiling-primes-naive}
\end{problem}

\subsection*{Avoid Loops} % ---------------------------------------------------

% Most repetition occurs in a looping structure.
% \textbf{Avoid loops where possible, especially nested loops} (loops within loops).
% If nested loops are unavoidable, focus optimization efforts on the innermost loop, since that part of the code gets the most repetitions.

NumPy routines and built-in functions are often useful for eliminating loops altogether. %, a process called \emph{vectorization}.
Consider the simple problem of summing the rows of a matrix, implemented in three ways.

\begin{lstlisting}
>>> def row_sum_awful(A):
...     """Sum the rows of A by iterating through rows and columns."""
...     m,n = A.shape
...     row_totals = np.empty(m)        # Allocate space for the output.
...     for i in range(m):              # For each row...
...         total = 0
...         for j in range(n):          # ...iterate through the columns.
...             total += A[i,j]
...         row_totals[i] = total       # Record the total.
...     return row_totals
...
>>> def row_sum_bad(A):
...     """Sum the rows of A by iterating through rows."""
...     return np.array([sum(A[i,:]) for i in range(A.shape[0])])
...
>>> def row_sum_fast(A):
...     """Sum the rows of A with NumPy."""
...     return np.<<sum>>(A, axis=1)    # Or A.sum(axis=1).
...
\end{lstlisting}

None of the functions are fundamentally different, but their run times differ dramatically.

\begin{lstlisting}
<g<In [8]:>g> import numpy as np
<g<In [9]:>g> A = np.random.random((10000, 10000))

<g<In [10]:>g> <p<%time>p> rows = row_sum_awful(A)
<<CPU times: user 22.7 s, sys: 137 ms, total: 22.8 s
Wall time: 23.2 s>>         # SLOW!

<g<In [11]:>g> <p<%time>p> rows = row_sum_bad(A)
<<CPU times: user 8.85 s, sys: 15.6 ms, total: 8.87 s
Wall time: 8.89 s>>         # Slow!

<g<In [12]:>g> <p<%time>p> rows = row_sum_fast(A)
<<CPU times: user 61.2 ms, sys: 1.3 ms, total: 62.5 ms
Wall time: 64 ms>>          # Fast!
\end{lstlisting}

In this experiment, \li{row_sum_fast()} runs several hundred times faster than \li{row_sum_awful()}.
This is primarily because looping is expensive in Python, but NumPy handles loops in C, which is much quicker.
Other NumPy functions like \li{np.<<sum>>()} with an \li{axis} argument can often be used to eliminate loops in a similar way.

\begin{problem} % Naive Nearest Neighbor with vectorization.
Let $A$ be an $m\times n$ matrix with columns $\a_0, \ldots, \a_{n-1}$, and let $\x$ be a vector of length $m$.
The \emph{nearest neighbor problem}\footnote{The nearest neighbor problem is a common problem in many fields of artificial intelligence. The problem can be solved more efficiently with a $k$-d tree, a specialized data structure for storing high-dimensional data.} is to determine which of the columns of $A$ is ``closest'' to $\x$ with respect to some norm.
That is, we compute
\[\underset{j}{\text{argmin }} \|\a_j - \x\|.\]
The following function solves this problem na\"ively for the usual Euclidean norm.
\begin{lstlisting}
def nearest_column(A, x):
    """Find the index of the column of A that is closest to x."""
    distances = []
    for j in range(A.shape[1]):
        distances.append(np.linalg.norm(A[:,j] - x))
    return np.argmin(distances)
\end{lstlisting}

Write a new version of this function without any loops or list comprehensions, using array broadcasting and the \li{axis} keyword in \li{np.linalg.norm()} to eliminate the existing loop.
Try to implement the entire function in a single line.
\\(Hint: See the NumPy Visual Guide in the Appendix for a refresher on array broadcasting.)

Profile the old and new versions with \li{<p<\%prun>p>} and compare the output.
Finally, use \li{<p<\%time>p>} or \li{<p<\%timeit>p>} to verify that your new version runs faster than the original.
\end{problem}

\subsection*{Use Data Structures Correctly} % ---------------------------------

Every data structure has strengths and weaknesses, and choosing the wrong data structure can be costly.
Here we consider three ways to avoid problems and use sets, dictionaries, and lists correctly.

\begin{itemize}
\item \textbf{Membership testing}. The question ``is \li{<value>} a member of \li{<container>}'' is common in numerical algorithms.
Sets and dictionaries are implemented in a way that makes this a trivial problem, but lists are not.
In other words, the \li{in} operator is near instantaneous with sets and dictionaries, but not with lists.

\begin{lstlisting}
<g<In [13]:>g> a_list = list(range(int(1e7)))

<g<In [14]:>g> a_set = set(a_list)

<g<In [15]:>g> <p<%timeit>p> 12.5 in a_list
<<413 ms +- 48.2 ms per loop (mean+-std.dev. of 7 runs, 1 loop each)>>

<g<In [16]:>g> <p<%timeit>p> 12.5 in a_set
<<170 ns +- 3.8 ns per loop (mean+-std.dev. of 7 runs, 10000000 loops each)>>
\end{lstlisting}

Looking up dictionary values is also almost immediate.
Use dictionaries for storing calculations to be reused, such as mappings between letters and numbers or common function outputs.

\item \textbf{Construction with comprehension}.
Lists, sets, and dictionaries can all be constructed with comprehension syntax.
This is slightly faster than building the collection in a loop, and the code is highly readable.
% TODO (?): map().

\begin{lstlisting}
# Map the integers to their squares.
<g<In [17]:>g> <p<%%time>p>
   <g<...:>g> a_dict = {}
   <g<...:>g> for i in range(1000000):
   <g<...:>g>     a_dict[i] = i**2
   <g<...:>g>
<<CPU times: user 432 ms, sys: 54.4 ms, total: 486 ms
Wall time: 491 ms>>

<g<In [18]:>g> <p<%time>p> a_dict = {i:i**2 for i in range(1000000)}
<<CPU times: user 377 ms, sys: 58.9 ms, total: 436 ms
Wall time: 440 ms>>
\end{lstlisting}

\item \textbf{Intelligent iteration}.
Unlike looking up dictionary values, indexing into lists takes time.
Instead of looping over the indices of a list, loop over the entries themselves.
When indices and entries are both needed, use \li{enumerate()} to get the index and the item simultaneously.

\begin{lstlisting}
<g<In [19]:>g> a_list = list(range(1000000))

<g<In [20]:>g> <p<%%time>p>          # Loop over the indices of the list.
    <g<...:>g> for i in range(len(a_list)):
    <g<...:>g>     item = a_list[i]
    <g<...:>g>
<<CPU times: user 103 ms, sys: 1.78 ms, total: 105 ms
Wall time: 107 ms>>

<g<In [21]:>g> <p<%%time>p>          # Loop over the items in the list.
    <g<...:>g> for item in a_list:
    <g<...:>g>     _ = item
    <g<...:>g>
<<CPU times: user 61.2 ms, sys: 1.31 ms, total: 62.5 ms
Wall time: 62.5 ms>>      # Almost twice as fast as indexing!
\end{lstlisting}
% <g<In [X]:>g> <p<%%time>p>          # Use enumerate() to get both indices and items.
%     <g<...:>g> for i, item in enumerate(a_list):
%     <g<...:>g>     _ = item
%     <g<...:>g>
% <<CPU times: user 92.5 ms, sys: 1.58 ms, total: 94.1 ms
% Wall time: 94.4 ms>>      # Still slightly faster than indexing.
% \end{lstlisting}
\end{itemize}

\begin{comment} % USELESS
Second, swap values with a single assignment.

\begin{lstlisting}
>>> a, b = 1, 2
>>> a, b = b, a
>>> a, b
(2, 1)
\end{lstlisting}

Third, many non-Boolean objects in Python have truth values.
For example, numbers are \li{False} when equal to zero and \li{True} otherwise.
Similarly, lists and strings are \li{False} when they are empty and \li{True} otherwise.
The following code gives some examples.

\begin{lstlisting}
# Use the truth values of numbers.
>>> if 10:
...     print("Non-zero")
...
Non-zero

# Use the truth values of a list.
>>> my_list = [i for i in range(5)]
>>> if my_list:
...     print(my_list[0])
...
0
\end{lstlisting}
\end{comment}

\begin{problem} % Name scores.
This is problem 22 from \url{https://projecteuler.net}.

Using the rule $A\mapsto 1, B\mapsto 2, \ldots, Z\mapsto 26$, the \emph{alphabetical value} of a name is the sum of the digits that correspond to the letters in the name.
For example, the alphabetic value of ``COLIN'' is $3 + 15 + 12 + 9 + 14 = 53$.

The following function reads the file \texttt{names.txt}, containing over five-thousand first names, and sorts them in alphabetical order.
The \emph{name score} of each name in the resulting list is the alphabetic value of the name multiplied by the name's position in the list, starting at 1.
``COLIN'' is the 938th name alphabetically, so its name score is $938 \times 53 = 49714$.
The function returns the total of all the name scores in the file.

\begin{lstlisting}
def name_scores(filename="names.txt"):
    """Find the total of the name scores in the given file."""
    with open(filename, 'r') as infile:
        names = sorted(infile.read().replace('"', '').split(','))
    total = 0
    for i in range(len(names)):
        name_value = 0
        for j in range(len(names[i])):
            alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
            for k in range(len(alphabet)):
                if names[i][j] == alphabet[k]:
                    letter_value = k + 1
            name_value += letter_value
        total += (names.index(names[i]) + 1) * name_value
    return total
\end{lstlisting}

Rewrite this function---removing repetition, eliminating loops, and using data structures correctly---so that it runs in less than 10 milliseconds on average.
\end{problem}

\subsection*{Use Generators} % ------------------------------------------------

A \emph{generator} is an iterator that yields multiple values, one at a time, as opposed to returning a single value.
For example, \li{range()} is a generator.
Using generators appropriately can reduce both the run time and the spatial complexity of a routine.
Consider the following function, which constructs a list containing the entries of the sequence $\{x_n\}_{n=1}^N$ where $x_{n} = x_{n-1} + n$ with $x_1 = 1$.

\begin{lstlisting}
>>> def sequence_function(N):
...     """Return the first N entries of the sequence x_n = x_{n-1} + n."""
...     sequence = []
...     x = 0
...     for n in range(1, N+1):
...         x += n
...         sequence.append(x)
...     return sequence
...
>>> sequence_function(10)
[1, 3, 6, 10, 15, 21, 28, 36, 45, 55]
\end{lstlisting}

A potential problem with this function is that all of the values in the list are computed before anything is returned.
This can be a big issue if the parameter $N$ is large.
A generator, on the other hand, \emph{yields} one value at a time, indicated by the keyword \li{yield} (instead of \li{return}).
When the generator is asked for the next entry, the code resumes right where it left off.
% The only visible difference between a generator and a function is the use of \li{yield} in place of \li{return}.
% In the following example, note that \li{sequence_generator()} does not keep track of the entire sequence like \li{sequence_function()} does.

\begin{lstlisting}
>>> def sequence_generator(N):
...     """Yield the first N entries of the sequence x_n = x_{n-1} + n."""
...     x = 0
...     for n in range(1, N+1):
...         x += n
...         yield x         # "return" a single value.
...
# Get the entries of the generator one at a time with next().
>>> generated = sequence_generator(10)
>>> next(generated)
1
>>> next(generated)
3
>>> next(generated)
6

# Put each of the generated items in a list, as in sequence_function().
>>> list(sequence_generator(10))    # Or [i for i in sequence_generator(10)].
[1, 3, 6, 10, 15, 21, 28, 36, 45, 55]

# Use the generator in a for loop, like range().
>>> for entry in sequence_generator(10):
...     print(entry, end=' ')
...
1 3 6 10 15 21 28 36 45 55
\end{lstlisting}

Many generators, like \li{range()} and \li{sequence_generator()}, only yield a finite number of values.
However, generators can also continue yielding indefinitely.
For example, the following generator yields the terms of $\{x_n\}_{n=1}^\infty$ forever.
In this case, using \li{enumerate()} with the generator is helpful for tracking the index $n$ as well as the entry $x_n$.

\begin{lstlisting}
>>> def sequence_generator_forever():
...     """Yield the sequence x_n = x_{n-1} + n forever."""
...     x = 0
...     n = 1
...     while True:
...         x += n
...         n += 1
...         yield x         # "return" a single value.
...

# Sum the entries of the sequence until the sum exceeds 1000.
>>> total = 0
>>> for i, x in enumerate(sequence_generator_forever()):
...     total += x
...     if total > 1000:
...         print(i)        # Print the index where the total exceeds.
...         break           # Break out of the for loop to stop iterating.
...
17

# Check that 18 terms are required (since i starts at 0 but n starts at 1).
>>> print(sum(sequence_generator(17)), sum(sequence_generator(18)))
969 1140
\end{lstlisting}

\begin{warn} % Use xrange() in Python 2.
In Python 2.7 and earlier, \li{range()} is \textbf{not} a generator.
Instead, it constructs an entire list of values, which is often significantly slower than yielding terms individually as needed.
If you are using old versions of Python, use \li{xrange()}, the equivalent of \li{range()} in Python 3.0 and later.
\end{warn}

\begin{problem} % Fibonacci sequence.
This is problem 25 from \url{https://projecteuler.net}.

The \emph{Fibonacci sequence} is defined by the recurrence relation $F_{n} = F_{n-1} + F_{n-2}$, where $ F_1 = F_2 = 1$.
The 12th term, $F_{12} = 144$, is the first term to contain three digits.

Write a generator that yields the terms of the Fibonacci sequence indefinitely.
Next, write a function that accepts an integer $N$.
Use your generator to find the first term in the Fibonacci sequence that contains $N$ digits.
Return the index of this term.
\\(Hint: a generator can have more than one \li{yield} statement.)
\end{problem}

% See \url{https://docs.python.org/3/tutorial/classes.html#generators} for more about generators.
% and \url{https://wiki.python.org/moin/Generators}

\begin{problem} % Sieve of Eratosthenes.
The function in Problem \ref{prob:profiling-primes-naive} could be turned into a prime number generator that yields primes indefinitely, but it is not the only strategy for yielding primes.
The \emph{Sieve of Eratosthenes}\footnote{See \url{https://en.wikipedia.org/wiki/Sieve_of_Eratosthenes}.} is a faster technique for finding all of the primes below a certain number.
\begin{enumerate}
\item Given a cap $N$, start with all of the integers from $2$ to $N$.
\item Remove all integers that are divisible by the first entry in the list. \label{step:profiling-sieve-of-eratos}
\item Yield the first entry in the list and remove it from the list.
\item Return to step \ref{step:profiling-sieve-of-eratos} until the list is empty.
\end{enumerate}

Write a generator that accepts an integer $N$ and that yields all primes (in order, one at a time) that are less than $N$ using the Sieve of Eratosthenes.
Your generator should be able to find all primes less than 100,000 in under $5$ seconds.

Your generator and your fast function from Problem \ref{prob:profiling-primes-naive} may be helpful in solving problems 10, 35, 37, 41, 49, and 50 (for starters) of \url{https://projecteuler.net}.
\end{problem}

\section*{Numba} % ============================================================

Python code is simpler and more readable than many languages, but Python is also generally much slower than compiled languages like C.
The \li{numba} module
%\footnote{Numba is \textbf{not} part of the standard library, but it is included in the Anaconda distribution. For installation details, see \url{https://numba.pydata.org/}.}
bridges the gap by using \emph{just-in-time} (JIT) compilation to optimize code, meaning that the code is actually compiled right before execution.

\begin{lstlisting}
>>> from numba import jit

>>> @jit                # Decorate a function with @jit to use Numba.
... def row_sum_numba(A):
...     """Sum the rows of A by iterating through rows and columns,
...     optimized by Numba.
...     """
...     m,n = A.shape
...     row_totals = np.empty(m)
...     for i in range(m):
...         total = 0
...         for j in range(n):
...             total += A[i,j]
...         row_totals[i] = total
...     return row_totals
\end{lstlisting}

Python is a \emph{dynamically typed} language, meaning variables are not defined explicitly with a datatype (\li{x = 6} as opposed to \li{int x = 6}).
This particular aspect of Python makes it flexible, easy to use, and slow.
% One of the reasons compiled languages like C are so much faster than Python is because they have explicitly defined datatypes.
Numba speeds up Python code primarily by assigning datatypes to all the variables.
Rather than requiring explicit definitions for datatypes, Numba attempts to infer the correct datatypes based on the datatypes of the input.
In \li{row_sum_numba()}, if \li{A} is an array of integers, Numba will infer that \li{total} should also be an integer.
On the other hand, if \li{A} is an array of floats, Numba will infer that \li{total} should be a \emph{double} (a similar datatype to float in C).

Once all datatypes have been inferred and assigned, the original Python code is translated to machine code. % by the LLVM library.
Numba caches this compiled version of code for later use.
The first function call takes the time to compile and then execute the code, but subsequent calls use the already-compiled code.

\begin{lstlisting}
<g<In [22]:>g> A = np.random.random((10000, 10000))

# The first function call takes a little extra time to compile first.
<g<In [23]:>g> <p<%time>p> rows = row_sum_numba(A)
<<CPU times: user 408 ms, sys: 11.5 ms, total: 420 ms>>
Wall time: 425 ms

# Subsequent calls are consistently faster that the first call.
<g<In [24]:>g> <p<%timeit>p> row_sum_numba(A)
<<138 ms +- 1.96 ms per loop (mean +- std. dev. of 7 runs, 10 loops each)>>
\end{lstlisting}

Note that the only difference between \li{row_sum_numba()} and \li{row_sum_awful()} from a few pages ago is the \li{@jit} decorator, and yet the Numba version is about 99\% faster than the original!

The inference engine within Numba does a good job, but it's not always perfect.
Adding the keyword argument \li{nopython=True} to the \li{@jit} decorator raises an error if Numba is unable to convert each variable to explicit datatypes.
The \li{inspect_types()} method can also be used to check if Numba is using the desired types.

\begin{lstlisting}
# Run the function once first so that it compiles.
>>> rows = row_sum_numba(np.random.random((10,10)))
>>> row_sum_numba.inspect_types()
# The output is very long and detailed.
\end{lstlisting}

Alternatively, datatypes can be specified explicitly in the \li{@jit} decorator as a dictionary via the \li{<<locals>>} keyword argument.
Each of the desired datatypes must also be imported from Numba.

\begin{lstlisting}
>>> from numba import int64, double

>>> @jit(nopython=True, <<locals>>=dict(A=double[:,:], m=int64, n=int64,
...                                 row_totals=double[:], total=double))
... def row_sum_numba(A):           # 'A' is a 2-D array of doubles.
...     m,n = A.shape               # 'm' and 'n' are both integers.
...     row_totals = np.empty(m)    # 'row_totals' is a 1-D array of doubles.
...     for i in range(m):
...         total = 0               # 'total' is a double.
...         for j in range(n):
...             total += A[i,j]
...         row_totals[i] = total
...     return row_totals
...
\end{lstlisting}

While it sometimes results in a speed boost, there is a caveat to specifying the datatypes: \li{row_sum_numba()} no longer accepts arrays that contain anything other than floats.
When datatypes are not specified, Numba compiles a new version of the function each time the function is called with a different kind of input.
Each compiled version is saved, so the function can still be used flexibly.

\begin{problem} % Compare times for Numba.
The following function calculates the $n$th power of an $m\times m$ matrix $A$.

\begin{lstlisting}
def matrix_power(A, n):
    """Compute A^n, the n-th power of the matrix A."""
    product = A.copy()
    temporary_array = np.empty_like(A[0])
    m = A.shape[0]
    for power in range(1, n):
        for i in range(m):
            for j in range(m):
                total = 0
                for k in range(m):
                    total += product[i,k] * A[k,j]
                temporary_array[j] = total
            product[i] = temporary_array
    return product
\end{lstlisting}

\begin{enumerate}
\item Write a Numba-enhanced version of \li{matrix_power()} called \li{matrix_power_numba()}.
\item Write a function that accepts an integer $n$.
Run \li{matrix_power_numba()} once with a small random input so it compiles.
Then, for $m=2^2,2^3,\ldots,2^7$,
    \begin{enumerate}
        \item Generate a random $m\times m$ matrix $A$ with \li{np.random.random()}.
        \item Time (separately) \li{matrix_power()}, \li{matrix_power_numba()}, and NumPy's \\ \li{np.linalg.matrix_power()} on $A$ with the specified value of $n$.
        \\(If you are unfamiliar with timing code inside of a function, see the \\ Additional Material section on timing code.)
    \end{enumerate}
Plot the times against the size $m$ on a log-log plot (use \li{plt.loglog()}).
\end{enumerate}
With $n=10$, the plot should show that the Numba and NumPy versions far outperform the pure Python implementation, with NumPy eventually becoming faster than Numba.
% NumPy takes products of matrices by calling BLAS and LAPACK, which are heavily optimized linear algebra libraries written in C, assembly, and Fortran.
\end{problem}

\begin{warn}
Optimizing code is an important skill, but it is also important to know when to refrain from optimization.
The best approach to coding is to write unit tests, implement a solution that works, test and time that solution, \textbf{then} (and only then) optimize the solution with profiling techniques.
As always, the most important part of the process is choosing the correct algorithm to solve the problem.
Don't waste time optimizing a poor algorithm.
\end{warn}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Other Timing Techniques} % ---------------------------------------

Though \li{<p<\%time>p>} and \li{<p<\%timeit>p>} are convenient and work well, some problems require more control for measuring execution time.
The usual way of timing a code snippet by hand is via the \li{time} module (which \li{<p<\%time>p>} uses).
The function \li{time.time()} returns the number of seconds since the Epoch\footnote{See \url{https://en.wikipedia.org/wiki/Epoch_(reference_date)\#Computing}.}; to time code, measure the number of seconds before the code runs, the number of seconds after the code runs, and take the difference.

\begin{lstlisting}
>>> import time

>>> start = time.time()             # Record the current time.
>>> for i in range(int(1e8)):       # Execute some code.
...     pass
... end = time.time()               # Record the time again.
... print(end - start)              # Take the difference.
...
4.20402193069458 # (seconds)
\end{lstlisting}

The \li{timeit} module (which \li{<p<\%timeit>p>} uses) has tools for running code snippets several times.
The code is passed in as a string, as well as any setup code to be run before starting the clock.

\begin{lstlisting}
>>> import timeit

>>> timeit.timeit("for i in range(N): pass", setup="N = int(1e6)", number=200)
4.884839255013503       # Total time in seconds to run the code 200 times.
>>> _ / 200
0.024424196275067516    # Average time in seconds.
\end{lstlisting}

The primary advantages of these techniques are the ability automate timing code and being able save the results.
For more documentation, see \url{https://docs.python.org/3.6/library/time.html} and \url{https://docs.python.org/3.6/library/timeit.html}.

\subsection*{Customizing the Profiler} % --------------------------------------

The output from \li{<p<\%prun>p>} is generally long, but it can be customized with the following options.

\begin{table}[H]
\centering
\begin{tabular}{l|l}
    Option & Description \\ \hline
    \li{-l <limit>} & Include a limited number of lines in the output.\\
    \li{-s <key>} & Sort the output by call count, cumulative time, function name, etc. \\
    \li{-T <filename>} & Save profile results to a file (results are still printed).\\
\end{tabular}
\end{table}

For example, \li{<p<\%prun>p> -l 3 -s ncalls -T path_profile.txt max_path()} generates a profile of \li{max_path()} that lists the 3 functions with the most calls, then write the results to \texttt{path\_profile.txt}.
See  \url{http://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-prun} for more details.


\documentclass{beamer}

\usepackage{verbatim}
\usepackage{fancyvrb}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{booktabs}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{calc}
\usepackage{color}
\usepackage{multicol}
\usepackage{wrapfig}
\usepackage{natbib}
\usepackage[ruled,vlined]{algorithm2e}
\usepackage{animate}
\usepackage{mathtools}
\usepackage{listings}

\usepackage{cmbright}
\fontencoding{OT1}\fontfamily{cmbr}\selectfont %to load ot1cmbr.fd
\DeclareFontShape{OT1}{cmbr}{bx}{n}{% change bx definition
<->cmbrbx10%
}{}
\normalfont % back to normalfont

% two col: two columns
\newenvironment{twocol}[4]{
\begin{columns}[c]
\column{#1\textwidth}
#3
\column{#2\textwidth}
#4
\end{columns}
}

\makeatletter
\setbeamertemplate{theorem begin}
{%
\begin{\inserttheoremblockenv}
  {}{\usebeamerfont*{block title}\usebeamercolor[fg]{block title}%
  \inserttheoremname
  %\inserttheoremnumber
  \ifx \inserttheoremaddition \empty \else\ (\inserttheoremaddition)\fi
  \inserttheorempunctuation}
  \normalfont
  }
  \setbeamertemplate{theorem end}{\end{\inserttheoremblockenv}}
\makeatother

\newcommand{\E}{\mathrm{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\sd}{\mathrm{sd}}
\newcommand{\s}{\mathrm{s}}
\newcommand{\Corr}{\mathrm{Corr}}
\newcommand{\rank}{\mathrm{rank}}
\newcommand{\trace}{\mathrm{trace}}
\newcommand{\nullspace}{\mathrm{null}}
\newcommand{\myspan}{\mathrm{span}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\softmax}{softmax}

\definecolor{darkgreen}{rgb}{0,0.5,0}

\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{exe}{Exercise}
\newtheorem{notation}{Notation}
\newtheorem{remark}{Remark}

\definecolor{darkgreen}{rgb}{0,0.5,0}

\title{Model Diagnostics}
\author{Zhenisbek Assylbekov}
\institute{Department of Mathematics}
\date{Regression Analysis}

\AtBeginSection[]
{
  \begin{frame}<beamer>
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\begin{frame}{Diagnostics we have already discussed}
\begin{itemize}
\item Residuals $e_i$ vs.
\begin{itemize}
    \item $i$ (independence)
    \item $x_1$, \ldots , $x_k$ (linearity)
    \item $\hat{Y}_i$ (linearity and homogeneity of variance)
\end{itemize}
\item\pause Q-Q plot of $e_1$, \ldots , $e_n$.
\item\pause VIF$_j$ for $j = 1, \ldots, k$.
\item\pause Significance tests (Runs, Levene's, Shapiro-Wilk)
\item\pause Now we'll discuss
\begin{itemize}
    \item added variable plots,
    \item leverages,
    \item DFFITS,
    \item Cook's distance.
\end{itemize}
\end{itemize}
\end{frame}

\begin{frame}[fragile]{The problem with marginal plots}
\begin{center}
\includegraphics[height=.4\textheight]{plots/marginal.pdf}
\end{center}
\pause No dependence b/w $Y$ and $x_j$ marginally, but significant association jointly:
\begin{footnotesize}
\pause\begin{verbatim}
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.05083    0.07726   0.658    0.514
x1           0.95536    0.07651  12.487   <2e-16 ***
x2           0.96312    0.07658  12.576   <2e-16 ***
\end{verbatim}
\end{footnotesize}
\end{frame}

\section{Added variable plots}

\begin{frame}{Problems with marginal plots}
\begin{small}
\begin{itemize}
\item Residuals $e_i$ versus a predictor values $x_i,j$ can show whether $x_j$ may need to be transformed or whether we should add a quadratic term $x_j^2$.
\item\pause We can omit the predictor  from the model and plot the residuals versus the predictor to see if the predictor explains residual variability.
\item\pause However these plots can also be misleading: \pause e.g., we can have
\vspace{-5pt}
\begin{center}
    \includegraphics[height=.3\textheight]{plots/res-xj.pdf}
\end{center}
where $e(Y\mid x_j)$ are residuals when we regress $Y$ on $x_j$ only.
\end{itemize}
\end{small}
\end{frame}

\begin{frame}[fragile]{Added variables plots}
\begin{itemize}
    \item The previous plots suggest that $x_2$ is not needed when $x_1$ is in the model, \pause or that $x_1$ is not needed when $x_2$ is in the model.
    \item\pause But we still have significance for both $x_1$ and $x_2$ when they are \textit{simultaneously} in the model!
    \begin{footnotesize}
    \begin{verbatim}
                Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.05083    0.07726   0.658    0.514
x1           0.95536    0.07651  12.487   <2e-16 ***
x2           0.96312    0.07658  12.576   <2e-16 ***
    \end{verbatim}
    \end{footnotesize}
    \item\pause An \textbf{added variable plot} tries to fix this problem.
    \item\pause It answers the question: Does $x_j$ explain any \textit{residual} variability once the rest of the predictors are in the model?
\end{itemize}
\end{frame}

\begin{frame}{10.1 Added variable plots}
\begin{itemize}
\item Consider a pool of predictors $x_1 , \ldots, x_k$. Let’s consider predictor $x_j$ where $j = 1, \ldots, k$.
\item\pause Regress $Y_i$ vs. all predictors except $x_j$, call the residuals $e_i(Y\mid\mathbf{x}_{-j})$.
\item\pause Regress $x_j$ vs. all predictors except $x_j$, call the residuals $e_i(x_j\mid\mathbf{x}_{-j})$.
\item\pause The added variable plot for $x_j$ is $e_i(Y\mid\mathbf{x}_{-j})$ vs. $e_i(x_j\mid\mathbf{x}_{-j})$.
\item\pause If you fit a simple linear regression
$$
e_i(Y\mid\mathbf{x}_{-j}) = \beta_j\cdot e_i(x_j\mid\mathbf{x}_{-j})+\epsilon_i
$$
then the LSE $\hat\beta_j$ \textit{is the same} as one would get from fitting the full model $Y_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_k x_{ik} + \epsilon_i$.
\item\pause Gives an idea of the functional form of $x_j$: a transformation in $x_j$ should mimic the pattern seen in the plot. %; the methods of Section 3.9 apply.
\end{itemize}
\end{frame}

\begin{frame}{10.1 Added variables plots illustration}
Let's go back to our synthetic example:

\vspace{-10pt}

\begin{center}
\includegraphics[height=.4\textheight]{plots/marginal.pdf}
\end{center}

\pause
\begin{center}
\includegraphics[height=.4\textheight]{plots/av_plots.pdf}
\end{center}

\end{frame}

\begin{frame}[fragile]
\frametitle{Salary data, first order terms only}
\begin{scriptsize}
\begin{verbatim}
> head(salary_data)
  salary age educ pol
1     38  25    4   D
2     45  27    4   R
3     28  26    4   O
4     55  39    4   D
5     74  42    4   R
6     43  41    4   O
> m = lm(salary ~ ., data=salary_data)
> summary(m)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  17.0313     7.3459   2.318  0.03735 *
age           0.8983     0.1968   4.565  0.00053 ***
educ          1.5039     1.1841   1.270  0.22632
polO        -16.5404     4.8807  -3.389  0.00484 **
polR          9.1587     4.8482   1.889  0.08139 .
---

Residual standard error: 8.209 on 13 degrees of freedom
Multiple R-squared:  0.8374,	Adjusted R-squared:  0.7873
\end{verbatim}
\end{scriptsize}
\end{frame}

\begin{frame}{Added variable plots}
\centerline{\includegraphics[height=0.7\textheight]{plots/salary_av.pdf}}

\pause Age effect is nonlinear; let's add a quadratic term.
\end{frame}

\begin{frame}[fragile]
\frametitle{Salary data, quadratic effect in age}
\begin{verbatim}
> m1 = lm(salary ~ . + I(age^2), data=salary_data)
> summary(m1)

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept) -39.224169  16.810142  -2.333 0.037839 *
age           3.463723   0.740666   4.676 0.000535 ***
educ          2.166475   0.883369   2.453 0.030453 *
polO        -15.455108   3.571147  -4.328 0.000983 ***
polR         10.118144   3.544586   2.855 0.014500 *
I(age^2)     -0.028831   0.008166  -3.530 0.004143 **
---

Residual standard error: 5.984 on 12 degrees of freedom
Multiple R-squared:  0.9202,	Adjusted R-squared:  0.887
\end{verbatim}
\end{frame}

\begin{frame}{Added variables plots w/ quadratic age}
\centerline{\includegraphics[width=.75\textwidth]{plots/salary_av2.pdf}}
Education is now significant! \pause The incorrect
functional form for age was \textit{masking} the
importance of education.
\end{frame}

\begin{frame}[fragile]
\frametitle{Salary data, quadratic effect in education}
\begin{footnotesize}
\begin{verbatim}
> summary(m2)

Call:
lm(formula = salary ~ . + I(age^2) + I(educ^2), data = salary_data)

              Estimate Std. Error t value Pr(>|t|)
(Intercept) -75.977348  18.262402  -4.160 0.001589 **
age           2.787032   0.626151   4.451 0.000977 ***
educ         18.751324   5.739109   3.267 0.007501 **
polO        -13.976910   2.848879  -4.906 0.000467 ***
polR          9.495127   2.790631   3.403 0.005903 **
I(age^2)     -0.018677   0.007298  -2.559 0.026558 *
I(educ^2)    -1.342341   0.461108  -2.911 0.014161 *
---

Residual standard error: 4.697 on 11 degrees of freedom
Multiple R-squared:  0.9549,	Adjusted R-squared:  0.9304
\end{verbatim}
\end{footnotesize}
\end{frame}

\begin{frame}{Added variable plots w/ quad. age and educ}
\begin{center}
\includegraphics[height=.85\textheight]{plots/salary_av3.pdf}
\end{center}
\end{frame}

\section{Outliers, leverages, DFFITs, Cook's distance}

\begin{frame}{Outliers}
\begin{itemize}
\item Outliers are data points which are ``far away'' from the bulk of data. \pause Observations may be outlying
\begin{itemize}
    \item relative to predictors, i.e. $\mathbf{x}_i$ relative to other $\{\mathbf{x}_j\}_{j\ne i}$
    \item\pause relative to the model, i.e. $Y_i$ relative to $\hat{Y}_i$.
\end{itemize}
\item\pause \textbf{Studentized deleted residuals} are designed to detect outlying $Y_i$ observations; \textbf{leverages} detect outlying $\mathbf{x}_i$ points.
\item\pause Outliers have the potential to influence the fitted regression
function:
\begin{itemize}
\item\pause if the outlying points follow the modeling
assumptions and are representative, they may \textit{strengthen} inference and reduce error in predictions
\item\pause if not, outlying values may skew inference a lot and yield
models with poor predictive properties.
\end{itemize}
\end{itemize}
\end{frame}

\begin{frame}{Outliers \& influential points}
\begin{itemize}
\item Often outliers are flagged and deemed suspect as mistakes
or observations not gathered from the same population as the other observations.
\item\pause Outliers are sometimes of interest in their own right and may illustrate aspects of the dataset that require more careful study.
\item\pause Although an observation may be flagged as an outlier, the
point \textit{may or may not} affect the fitted regression function
more than other points.
\item\pause A \textbf{DFFIT} is a measure of influence that an individual point
$(\mathbf{x}_i, Y_i)$ has on the regression surface at $\mathbf{x}_i$.
\item\pause \textbf{Cook's distance} is a consolidated measure of influence the point $(\mathbf{x}_i, Y_i)$ has on the regression surface at all $n$ points $\mathbf{x}_1, \ldots, \mathbf{x}_n$.
\end{itemize}
\end{frame}

\begin{frame}{Variance of $\hat{Y}_i$}
Recall that
\begin{itemize}
    \item $\hat{\mathbf{Y}}=\pause\mathbf{X}\hat{\boldsymbol\beta}$,
    \item\pause $\hat{\boldsymbol\beta}\pause=(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{Y}$,
    \item\pause $\mathbf{Y}\pause\sim\mathcal{N}_n(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I}_n)$
\end{itemize}
Thus,
\begin{align*}
\onslide<7->{\Cov[\hat{\mathbf{Y}}]}\onslide<8->{&=\Cov[\mathbf{X}\hat{\boldsymbol\beta}]}\onslide<9->{=\Cov[\mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{Y}]\\}
\onslide<10->{&=\Cov[\mathbf{HY}]=\mathbf{H}\Cov[\mathbf{Y}]\mathbf{H}^\top=\mathbf{H}\sigma^2\mathbf{IH}^\top\\}
\onslide<11->{&=\sigma^2\mathbf{H}}
\end{align*}
\onslide<12->{$\Rightarrow$\,\,
$\Var[\hat{Y}_i]=\sigma^2h_{ii}$, and its unbiased estimator is $\text{MSE}\cdot h_{ii}$.}

\end{frame}

\begin{frame}{10.2 Studentized deleted residuals}
\begin{itemize}
\item The \textbf{standardized residuals}
$$
r_i=\frac{Y_i-\hat{Y}_i}{\sqrt{\text{MSE}(1-h_{ii})}}
$$
have a constant variance of 1.
\item\pause Typically, $|r_i|>2$ is considered ``large.''
\item\pause $h_{ii}=\mathbf{x}^\top_i(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{x}_i$ is called the $i^\text{th}$ \textbf{leverage value}.
\item\pause A refinement of the standardized residual that has a recognizable distribution is the \textbf{studentized deleted residual}
$$
t_i=\frac{Y_i-\hat{Y}_{i}}{\sqrt{\text{MSE}_{(i)}(1-h_{ii})}}
$$
where $\text{MSE}_{i(i)}$ is obtained from the model when $i$-th example was removed from the data.
\end{itemize}
\end{frame}

\begin{frame}{Studentized deleted residuals}
\begin{itemize}
\item In fact, no need to fit $n$ additional regressions, because there is relationship b/w MSE and MSE$_{(i)}$:
$$
(n-p)\text{MSE}=(n-p-1)\text{MSE}_{(i)}+\frac{e_i^2}{1-h_{ii}}
$$
(prove it)

\item\pause Studentized deleted residuals are distributed as
$$
t_i\sim t_{n-p-1}.
$$
\item\pause Therefore, outlying $Y$-values may be flagged by using Bonferroni's adjustment and taking
$$
|t_i|>t_{1-\alpha/(2n); n-p-1}
$$
as outlying.
\item\pause Typically, in practice, one simply flags observations with $|t_i|>t_{1-\alpha/2; n-p-1}$ as \textit{possibly} outlying in consideration with other diagnostics.
\end{itemize}
\end{frame}

\begin{frame}{10.3 Leverage}
\begin{itemize}
\item The leverages $h_{ii}$ get larger the further the points $\mathbf{x}_i$ are from the mean $\bar{\mathbf{x}} = \frac1n\sum_{i=1}^n\mathbf{x}_i$, adjusted for ``how many'' other
predictors are in the vicinity of $\mathbf{x}_i$.
\item\pause Use the fact that $\mathbf{H} = \mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top = \mathbf{HH}$ to show $\sum_{i=1}^n h_{ii} = p$ and $0 \le h_{ii} \le1$.
\item\pause A large leverage $h_{ii}$ indicates that $\mathbf{x}_i$ is far away from the other predictors $\{\mathbf{x}_j\}_{j\ne i}$ \pause and that $\mathbf{x}_i$ may influence the fitted value $\hat{Y}_i$ more than other $x_j$'s will influence their respective fitted values. \pause This is evident in the variance of the residual $\Var[Y_i - \hat{Y}_i] = \sigma^2 \sqrt{1 - h_{ii}}$. The larger $h_{ii}$ is, the smaller
$\Var[Y_i - \hat{Y}_i]$ will be and hence the closer $\hat{Y}_i$ will be to $Y_i$ on average.
\item\pause The rule of thumb is that any leverage $h_{ii}$ that is larger than
twice the mean leverage $p/n$, i.e. $h_{ii} > 2p/n$, is flagged as
having ``high'' leverage.
\end{itemize}
\end{frame}

\begin{frame}[fragile]{Leverage}
\begin{itemize}
\item Note that the leverages $h_{ii}$ depend only on the $\mathbf{x}_i$ and hence
indicate which points might \textit{potentially} be influential.
\item \pause (p. 400) When making predictions $\mathbf{x}_{n+1}$ at a point not in the
data set, we consider the measure of distance of this point from the points $\mathbf{x}_1 , \ldots, \mathbf{x}_n$ given by $h_{n+1} = \mathbf{x}_{n+1}(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{x}_{n+1}$.
\item \pause If $h_{n+1}$ is much larger than any of the $\{h_{11}, \ldots, h_{nn}\}$ you may
be extrapolating far outside the general region of your data.
\item \pause In \texttt{R}, you can get leverages using \verb|hatvalues| command.
\end{itemize}
\end{frame}

\begin{frame}{10.4 DFFITs}
\begin{itemize}
\item The $i^\text{th}$ DFFIT, denoted DFFIT$_i$, is given by
$$
\text{DFFIT}_i=\frac{\hat{Y}_i-\hat{Y}_{i(i)}}{\sqrt{\text{MSE}_{(i)}h_{ii}}}=t_i\sqrt{\frac{h_{ii}}{1-h_{ii}}},
$$
\pause where $\hat{Y}_i$ is fitted value of regression surface (calculated using all $n$ observations) at $\mathbf{x}_i$ and $\hat{Y}_{i(i)}$ is fitted value of regression surface \textit{omitting the point} $(\mathbf{x}_i, Y_i)$ at the point $\mathbf{x}_i$.
\item\pause $\text{DFFIT}_i$ is standardized distance between \textit{fitted} regression surfaces \textit{with} and \textit{without} the point $(\mathbf{x}_i, Y_i)$.
\item\pause Rule of thumb that $\text{DFFIT}_i$ is ``large'' when $|\text{DFFIT}_i|>1$ for small to medium-sized data sets and $|\text{DFFIT}_i|>2\sqrt{p/n}$ for large data sets. \pause We will often just note those $\text{DFFIT}_i$'s that are considerable larger than the bulk of the $\text{DFFIT}_i$'s.
\end{itemize}
\end{frame}

\begin{frame}{10.4 Cook's distance}
\begin{itemize}
\item The $i^\text{th}$ Cook's distance, denoted $D_i$, is an aggregate measure of the influence of the $i^\text{th}$ observation on all $n$ fitted values:
$$
D_i=\frac{\sum_{j=1}^n(\hat{Y}_j - \hat{Y}_{j(i)})^2}{p\cdot\text{MSE}}
$$
\pause This is the sum of squared distances, at each $\mathbf{x}_j$, between fitted regression surface calculated with all $n$ points and fitted regression surface calculated with the $i^\text{th}$ case removed, standardized by $p\cdot \text{MSE}$.
\item \pause Look for values of Cook's distance significantly larger than other values; these are cases that have disproportionate
influence on the fitted regression surface as a whole.
\end{itemize}
\end{frame}

\section{Review of Diagnostics}

\begin{frame}{Review of diagnostics}
\begin{itemize}
\item \structure{Variance inflation factors} VIF$_j$ tell you which predictors are highly correlated with other predictors. If you have one or more VIF$_j>10$, you \text{may} want to eliminate some of the predictors.
\vspace{10pt}

\pause Multicollinearity affects the interpretability of the model, but does not indicate the model is ``bad'' in any way.
\vspace{10pt}

\pause An alternative approach that allows keeping correlated predictors is ridge regression (Chapter 11).

\item \structure{Deleted residuals} $t_i \sim t_{n-p-1}$, so you can formally define an outlier as being larger than $t_{1 - \alpha/(2n), n-p-1}$.
\end{itemize}
\end{frame}

\begin{frame}{Review of diagnostics}
\begin{itemize}
\item \structure{Residual plots}. Plots of $e_i$ or $t_i$ vs. $\hat{Y}_i$ and versus each $x_1, \ldots, x_k$ help assess (a) correct functional form, (b) constant
variance, and (c) outlying observations. %If an anomaly is apparent in any of these plots I may look at an added variable plot. If the number of predictors is small I may look at every added variable plot.
They may also suggest a transformation for a predictor or two.
\begin{itemize}
\item\pause Heteroscedasticy can be corrected by transforming $Y$, or else
modeling the variance directly (Chapter 11).
\item\pause Constant variance but nonlinear patterns can be accommodated by introducing quadratic terms.
\end{itemize}

\pause\structure{Added variable plots} help figure out functional form of predictors, and whether significance is being driven by one or two points only.
\end{itemize}
\end{frame}

\begin{frame}{Review of diagnostics}
\begin{itemize}
\item \textbf{DFFIT}$_i$ and \textbf{Cook's distance} $D_i$ tell you which observations
\textit{influence} the fitted model the most. Sometimes one or two points can drive the significance of a predictor.
\item\pause \textbf{Leverages} tell you which points \textit{can potentially} influence the fitted model. %Useful for finding ``hidden extrapolations'' via $h_{n+1}$.
%\item (pp.~404–405) DFBETA_{ij} tells you how much observation $i$ affects regression coefficient $j$. Useful to ``zoom in'' on where influential points are affecting the model.
\item\pause A \textbf{normal Q-Q plot} of the residuals will indicate  departures from normality.
\item\pause A list of the studentized deleted residuals, leverages, and Cook's distances helps to determine outlying values that may
be transcription errors or data anomalies and also indicates those observations that affect the fitted regression surface as
a whole.
\end{itemize}
\end{frame}

\begin{frame}{Standard diagnostic plots}
\begin{itemize}
\item $t_i$ vs. $h_i$. Which observations are outlying in $\mathbf{x}$-direction, outlying in $Y$-direction, or both?
\item $D_i$ vs. $i$. Which observations grossly affect fit of regression surface?
\item $e_i$ vs. $\hat{Y}_i$ and $t_i$ vs. $\hat{Y}_i$. Constant variance \& linearity.
\item $Y_i$ vs. $\hat{Y}_i$; how well model predicts its own data. Better models have points close to line $y = x$.
\item Normal probability plot of the $e_1, \ldots, e_n$.
\item Histogram of $e_1, \ldots, e_n$.
\item Plots of $e_i$ vs. each predictor $x_1 , \ldots, x_k$.
\item One more plot that prof. Hanson never looks at.
\end{itemize}
\end{frame}

\section{Example}

\begin{frame}{An example of diagnostics}
\centerline{\includegraphics[height=.7\textheight]{plots/bp-diag}}
\vspace{10pt}
Model is $Y_i=\beta_0+\beta_1 x_{i1} + \beta_2 x_{i2} + \beta_{12} x_{i1} x_{i2} + \epsilon_i$. One highly influential point \& one poorly fit.
\end{frame}

\begin{frame}{Residual plots}
\centerline{\includegraphics[scale=0.30]{plots/bp-res}}
\vspace{10pt}

These look pretty good, except for the one large residual.
\end{frame}

\begin{frame}{Arterial pressure data}
\includegraphics[scale=0.33]{plots/bp-cookd}
\vspace{10pt}

Obs. 7 has largest arterial pressure. Obs. 8 has relatively small arterial pressure.
\end{frame}

\begin{frame}{Dropping obs. 8 and obs. 7}
\includegraphics[scale=0.33]{plots/bp-two}
\vspace{10pt}

How do 7 and 8 affect the significance and/or magnitude of the effects?
\end{frame}

\end{document}


\section{Example}
\label{sec:Example}

In this section, we provide an example of the ETFT interface by depicting the core structure (i.e. finger tree) with internal vertices (i.e. sets) and sequences of pairs on the leaves (i.e. tours). Also, we illustrate the steps for computing \textit{split}s, \textit{insertion}s, \textit{concatenation}s, and lookups or \textit{view}s. All these the supporting functions to \textit{link} and \textit{cut} operations. Later, in Section~\ref{sec:TechDes}, we define these functions precisely describing their performance.

\tcr{The whole section 3 is clumsy. Some of the pictures are useful to explain the idea of the annotations, but in many cases neither the text nor the figure add much information to what is already obviously from the name and type signature.}

Let us define two trees, $t_8$ and $t_2$. In this case we will consider the top vertex as the root, although, this is arbitrary and we can \textit{reroot} the tree at an existent vertex at any time.

Firstly, we have the tree $t_8$ on the left in in Figure~\ref{fig:t1et1ft1}, followed by its (only) Euler tour representation $et_8$ and its finger tree $ft_8$. At the top of $ft_8$ we have the monoidal representation of the set-union operation (shadowed ellipse) over all the leaf-vertices containing the pairs from $et_8$.
\begin{figure}
\begin{center}
\includegraphics[scale=0.35]{./Images/t1et1ft1}
\end{center}
\caption{Tree $t_8$ and its corresponding Euler tour $et_8$ and finger tree $ft_8$}
\label{fig:t1et1ft1}
\end{figure}

As an second example, we have $t_2$, Figure~\ref{fig:t2et2ft2}. The spine of the finger tree now contains a \code{Single} type constructor altogether with a \code{Node3}. Again, shadowed ellipses refer to the monoidal annotations \code{Set} where the set-insertion and set-union take place \textit{automatically}.
\begin{figure}
\begin{center}
\includegraphics[scale=0.35]{./Images/t2et2ft2}
\end{center}
\caption{Tree $t_2$ and its corresponding Euler tour $et_2$ and finger tree $ft_2$}
\label{fig:t2et2ft2}
\end{figure}

Now, the forest $f$ in Figure~\ref{fig:forest} is formed by inserting $ft_8$ and $ft_2$ from left and right respectively, in $O(1)$. The forest's \code{Deep} type constructor has the set-union of all the finger trees ($ft_8$ and $ft_2$), depicted in back colour with white font. The thick lines coming out from the top ellipse are the components of the top finger tree: two \code{Digit} and an \code{Empty} subtree (or subforest).
\begin{figure}
\begin{center}
\includegraphics[scale=0.35]{./Images/forest}
\end{center}
\caption{The forest $f$ comprising finger trees (Euler tours) as elements}
\label{fig:forest}
\end{figure}

Following, we have the basic operations for dynamic trees. Left side of Figure~\ref{fig:rootReroot} shows the \code{root} of a tree $t$ returning a vertex $v$, which is the first element of its finger tree $ft$, implemented in \code{viewl}. Rerooting a tree, by calling \code{reroot}, ask for a vertex $v$ and returns a \textit{new} tree. This is depicted in right side of Figure~\ref{fig:rootReroot}. This operation involves split, concatenation and insertions detailed in Section~\ref{sec:TechDes}.
\begin{figure}
\begin{center}
\includegraphics[scale=0.3]{./Images/rootReroot}
\end{center}
\caption{\textit{root} and \textit{reroot} of a tree}
\label{fig:rootReroot}
\end{figure}

Splitting a finger tree, \code{split}, returns two subtrees (i.e. left and right) and an element, either a pair or an Euler tour, depending on the top container, a tree or a forest. This is depicted in Figure~\ref{fig:split}.
\begin{figure}
\begin{center}
\includegraphics[scale=0.25]{./Images/split}
\end{center}
\caption{Splitting a tree returns two smaller trees altogether an element}
\label{fig:split}
\end{figure}

Concatenating (also called appending) takes two finger trees and returns another (bigger) finger tree, as in Figure~\ref{fig:concatenation} and its represented by the $\bowtie$ symbol.
\begin{figure}
\begin{center}
\includegraphics[scale=0.25]{./Images/concatenation}
\end{center}
\caption{Concatenating two trees, returns a new one bigger}
\label{fig:concatenation}
\end{figure}

\tcr{Since the split operation over finger trees is central to the code, the paper would benefit from a more thorough explanation than the current incomplete type signatures (Concretely I'm missing an explanation of split's two first arguments).}

\tcr{The paper heavily builds on finger trees and tries to explain what they are and how they work, but omit aspects that are crucial to the understanding of the paper. In particular the `split` function is not explained, only listed in Table 2, but used in almost every listing later in the paper. The type listed for `split` mentions an unexplained type constructor `Split` and its type does not match the uses of `split`. The explanation around Fig. 5 does not help. What are the two first arguments to `split`?}

Under Figure~\ref{fig:consleftsnocright} we depict the final four help-functions that support \textit{link} and \textit{cut}: \code{viewl}, $\lhd$ (or \textit{cons}), $\rhd$ (or \textit{snoc}) and \textit{viewr}.
\begin{figure}
\begin{center}
\includegraphics[scale=0.25]{./Images/consleftsnocright}
\end{center}
\caption{Inserting and viewing from left (\code{viewl},$\lhd$) and inserting and viewing from right (\code{viewr},$\rhd$) }
\label{fig:consleftsnocright}
\end{figure}