\section{Factorial designs: Round 2}

Designs with more than one independent variable refer to designs where
the experimenter manipulates at least two independent variables.
Consider the light-switch example from the previous chapter. Imagine you
are trying to figure out which of two light switches turns on a light.
The dependent variable is the light (we measure whether it is on or
off). The first independent variable is light switch \#1, and it has two
levels, up or down. The second independent variable is light switch \#2,
and it also has two levels, up or down. When there are two independent
variables, each with two levels, there are four total conditions that
can be tested. We can describe these four conditions in a 2x2 table.

\begin{longtable}[]{@{}ccc@{}}
\toprule
& Switch 1 Up & Switch 1 Down\tabularnewline
\midrule
\endhead
Switch 2 Up & Light ? & Light ?\tabularnewline
Switch 2 Down & Light ? & Light ?\tabularnewline
\bottomrule
\end{longtable}

This kind of design has a special property that makes it a factorial
design. That is, the levels of each independent variable are each
manipulated across the levels of the other indpendent variable. In other
words, we manipulate whether switch \#1 is up or down when switch \#2 is
up, and when switch numebr \#2 is down. Another term for this property
of factorial designs is ``fully-crossed''.

It is possible to conduct experiments with more than independent
variable that are not fully-crossed, or factorial designs. This would
mean that each of the levels of one independent variable are not
necessarilly manipulated for each of the levels of the other independent
variables. These kinds of designs are sometimes called unbalanced
designs, and they are not as common as fully-factorial designs. An
example, of an unbalanced design would be the following design with only
3 conditions:

\begin{longtable}[]{@{}ccc@{}}
\toprule
& Switch 1 Up & Switch 1 Down\tabularnewline
\midrule
\endhead
Switch 2 Up & Light ? & Light ?\tabularnewline
Switch 2 Down & Light ? & NOT MEASURED\tabularnewline
\bottomrule
\end{longtable}

Factorial designs are often described using notation such as AXB, where
A= the number of levels for the first independent variable, and B = the
number of levels for the second independent variable. The fully-crossed
version of the 2-light switch experiment would be called a 2x2 factorial
design. This notation is convenient because by multiplying the numbers
in the equation we can find the number of conditions in the design. For
example 2x2 = 4 conditions.

More complicated factorial designs have more indepdent variables and
more levels. We use the same notation describe these designs. The number
for each variable represents the number of levels for that variable, and
the number of numbers in the equation represents the number of
variables. So, a 2x2x2 design has three independent variables, and each
one has 2 levels, for a total of 2x2x2=6 conditions. A 3x3 design has
two independent variables, each with three levels, for a total of 9
conditions. Designs can get very complicated, such as a 5x3x6x2x7
experiment, with five independent variables, each with differing numbers
of levels, for a total of 1260 conditions. If you are considering a
complicated design like that one, you should consider how to simplify
it.

\subsection{2x2 Factorial designs}\label{x2-factorial-designs}

For simplicity, we will focus mainly on 2x2 factorial designs. As with
simple designs with only one independent variable, factorial designs
have the same basic empirical question. Did the manipulation cause a
change in the measurement? However, 2x2 designs have more than one
manipulation, so there is more than one way that a change in measurement
can be observed. So, we end up asking the basic empirical question more
than once.

More specifically, the analysis of factorial designs are split into two
parts: main effects and interactions. Main effects are occur when the
levels of one independent variable cause a change in the dependent
variable. In a 2x2 design, there are two independent variables, so there
are two possible main effects: the main effect of independent variable
1, and the main effect of independent variable 2. An interaction occurs
when the effect of one independent variable \emph{depends} on the levels
of the other independent variable. My experience in teaching the concept
of main effects and interactions is that they are confusing. So, I
expect that these definitions will not be very helpful, and although
they are clear and precise, they only become helpful as definitions
after you understand the concepts\ldots{}so they are not useful for
explaining the concepts. To explain the concepts we will go through
several different kinds of examples.

To briefly add to the confusion, or perhaps to illustrate why these two
concepts can be confusing, we will look at the eight possible outcomes
that could occur in a 2x2 factorial experiment.

\begin{longtable}[]{@{}cccc@{}}
\toprule
Possible outcome & IV1 main effect & IV2 main effect &
Interaction\tabularnewline
\midrule
\endhead
1 & yes & yes & yes\tabularnewline
2 & yes & no & yes\tabularnewline
3 & no & yes & yes\tabularnewline
4 & no & no & yes\tabularnewline
5 & yes & yes & no\tabularnewline
6 & yes & no & no\tabularnewline
7 & no & yes & no\tabularnewline
8 & no & no & no\tabularnewline
\bottomrule
\end{longtable}

In the table, a yes means that there was statistically significant
difference for one of the main effects or interaction, and a no means
that there was not a statisically significant difference. As you can
see, just by adding one more independent variable, the number of
possible outcomes quickly become more complicated. When you conduct a
2x2 design, the task for analysis is to determine which of the 8
possibilites occured, and then explain the patterns for each of the
effects that occurred. That's a lot of explaining to do.

\subsection{Main effects}\label{main-effects}

Main effects occur when the levels of an independent variable cause
change in the measurement or dependent variable. There is one possible
main effect for each independent variable in the design. When we find
that independent variable did cause change, then we say there was a main
effect. When we find that the independent variable did not cause change,
then we say there was no main effect.

The simplest way to understand a main effect is to pretend that the
other independent variables do not exist. If you do this, then you
simply have a single-factor design, and you are asking whether that
single factor caused change in the measurement. For a 2x2 experiment,
you do this twice, once for each independent variable.

Let's consider a silly example to illustrate an important property of
main effects. In this experiment the dependent variable will be height
in inches. The independent variables will be shoes and hats. The shoes
independent variable will have two levels: wearing shoes vs.~no shoes.
The hats independent variable will have two levels: wearing a hat
vs.~not wearing a hat. The experiment will provide the shoes and hats.
The shoes add 1 inch to a person's height, and the hats add 6 inches to
a person's height. Further imagine that we conduct a within-subjects
design, so we measure each person's height in each of the fours
conditions. Before we look at some example data, the findings from this
experiment should be pretty obvious. People will be 1 inch taller when
they wear shoes, and 6 inches taller when they where a hat. We see this
in the example data from 10 subjects presented below:

\begin{longtable}[]{@{}rrrr@{}}
\toprule
NoShoes\_NoHat & Shoes\_NoHat & NoShoes\_Hat & Shoes\_Hat\tabularnewline
\midrule
\endhead
56 & 57 & 62 & 63\tabularnewline
57 & 58 & 63 & 64\tabularnewline
58 & 59 & 64 & 65\tabularnewline
57 & 58 & 63 & 64\tabularnewline
57 & 58 & 63 & 64\tabularnewline
57 & 58 & 63 & 64\tabularnewline
58 & 59 & 64 & 65\tabularnewline
57 & 58 & 63 & 64\tabularnewline
57 & 58 & 63 & 64\tabularnewline
57 & 58 & 63 & 64\tabularnewline
\bottomrule
\end{longtable}

The mean heights in each condition are:

\begin{longtable}[]{@{}lr@{}}
\toprule
NoShoes\_NoHat & 57.1\tabularnewline
Shoes\_NoHat & 58.1\tabularnewline
NoShoes\_Hat & 63.1\tabularnewline
Shoes\_Hat & 64.1\tabularnewline
\bottomrule
\end{longtable}

To find the main effect of the shoes manipulation we want to find the
mean height in the no shoes condition, and compare it to the mean height
of the shoes condition. To do this, we \emph{collapse}, or average over
the observations in the hat conditions. For example, looking only at the
no shoes vs.~shoes conditions we see the following averages for each
subject.

\begin{longtable}[]{@{}rr@{}}
\toprule
NoShoes & Shoes\tabularnewline
\midrule
\endhead
59 & 60\tabularnewline
60 & 61\tabularnewline
61 & 62\tabularnewline
60 & 61\tabularnewline
60 & 61\tabularnewline
60 & 61\tabularnewline
61 & 62\tabularnewline
60 & 61\tabularnewline
60 & 61\tabularnewline
60 & 61\tabularnewline
\bottomrule
\end{longtable}

The group means are:

\begin{longtable}[]{@{}lr@{}}
\toprule
NoShoes & 60.1\tabularnewline
Shoes & 61.1\tabularnewline
\bottomrule
\end{longtable}

As expected, we see that the average height is 1 inch taller when
subjects wear shoes vs.~do not wear shoes. So, the main effect of
wearing shoes is to add 1 inch to a person's height.

We can do the very same thing to find the main effect of hats. Except in
this case, we find the average heights in the no hat vs.~hat conditions
by averaging over the shoe variable.

\begin{longtable}[]{@{}rr@{}}
\toprule
NoHat & Hat\tabularnewline
\midrule
\endhead
56.5 & 62.5\tabularnewline
57.5 & 63.5\tabularnewline
58.5 & 64.5\tabularnewline
57.5 & 63.5\tabularnewline
57.5 & 63.5\tabularnewline
57.5 & 63.5\tabularnewline
58.5 & 64.5\tabularnewline
57.5 & 63.5\tabularnewline
57.5 & 63.5\tabularnewline
57.5 & 63.5\tabularnewline
\bottomrule
\end{longtable}

The group means are:

\begin{longtable}[]{@{}lr@{}}
\toprule
NoHat & 57.6\tabularnewline
Hat & 63.6\tabularnewline
\bottomrule
\end{longtable}

As expected, we the average height is 6 inches taller when the subjects
wear a hat vs.~do not wear a hat. So, the main effect of wearing hats is
to add 1 inch to a person's height.

Instead of using tables to show the data, let's use some bar graphs.
First, we will plot the average heights in all four conditions.

\includegraphics{Factorial_files/figure-latex/unnamed-chunk-7-1}

Some questions to ask yourself are 1) can you identify the main effect
of wearing shoes in the figure, and 2) can you identify the main effet
of wearing hats in the figure. Both of these main effects can be seen in
the figure, but they aren't fully clear. You have to do some visual
averaging.

Perhaps the most clear is the main effect of wearing a hat. The red bars
show the conditions where people wear hats, and the green bars show the
conditions where people do not wear hats. For both levels of the wearing
shoes variable, the red bars are higher than the green bars. That is
easy enough to see. More specifically, in both cases, wearing a hat adds
exactly 6 inches to the height, no more no less.

Less clear is the main effect of wearing shoes. This is less clear
because the effect is smaller so it is harder to see. How to find it?
You can look at the red bars first and see that the red bar for
no\_shoes is slightly smaller than the red bar for shoes. The same is
true for the green bars. The green bar for no\_shoes is slightly smaller
than the green bar for shoes.

\begin{figure}
\includegraphics[width=.5\linewidth]{Factorial_files/figure-latex/unnamed-chunk-8-1} \end{figure}

\begin{figure}
\includegraphics[width=.5\linewidth]{Factorial_files/figure-latex/unnamed-chunk-8-2} \end{figure}

Data from 2x2 designs is often present in graphs like the one above. An
advantage of these graphs is that they display means in all four
conditions of the design. However, they do not clearly show the two main
effects. Someone looking at this graph alone would have to guesstimate
the main effects. Or, in addition to the main effects, a researcher
could present two more graphs, one for each main effect (however, in
practice this is not commonly done because it takes up space in a
journal article, and with practice it becomes second nature to ``see''
the presence or absence of main effects in graphs showing all of the
conditions). If we made a separate graph for the main effect of shoes we
should see a difference of 1 inch between conditions. Similarly, if we
made a separate graph for the main effect of hats then we should see a
difference of 6 between conditions. Examples of both of those graphs
appear in the margin.

Why have we been talking about shoes and hats? These independent
variables are good examples of variables that are truly independent from
one another. Neither one influences the other. For example, shoes with a
1 inch sole will always add 1 inch to a person's height. This will be
true no matter whether they wear a hat or not, and no matter how tall
the hat is. In other words, the effect of wearing a shoe does not depend
on wearing a hat. More formally, this means that the shoe and hat
independent variables do not interact. It would be very strange if they
did interact. It would mean that the effect of wearing a shoe on height
would depend on wearing a hat. This does not happen in our universe. But
in some other imaginary universe, it could mean, for example, that
wearing a shoe adds 1 to your height when you do not wear a hat, but
adds more than 1 inch (or less than 1 inch) when you do wear a hat. This
thought experiment will be our entry point into discussing interactions.
A take-home message before we begin is that some independent variables
(like shoes and hats) do not interact; however, there are many other
independent variables that do.

\subsection{Interactions}\label{interactions}

Interactions occur when the effect of an independent variable depends on
the levels of the other independent variable. As we discussed above,
some independent variables are independent from one another and will not
produce interactions. However, other combinations of independent
variables are not independent from one another and they produce
interactions. Remember, independent variables are always manipulated
independently from the measured variable (see margin note), but they are
not necessarilly independent from each other.

\begin{marginfigure}
These ideas can be confusing if you think that the word ``independent''
refers to the relationship between independent variables. However, the
term ``independent variable'' refers to the relationship between the
manipulated variable and the measured variable. Remember, ``independent
variables'' are manipulated independently from the measured variable.
Specifically, the levels of any independent variable do not change
because we take measurements. Instead, the experimenter changes the
levels of the independent variable and then observes possible changes in
the measures.
\end{marginfigure}

There are many simple examples of two independent variables being
dependent on one another to produce an outcome. Consider driving a car.
The dependent variable (outcome that is measured) could be how far the
car can drive in 1 minute. Independent variable 1 could be gas (has gas
vs.~no gas). Independent variable 2 could be keys (has keys vs.~no
keys). This is a 2x2 design, with four conditions.

\begin{longtable}[]{@{}ccc@{}}
\toprule
& Gas & No Gas\tabularnewline
\midrule
\endhead
Keys & can drive & x\tabularnewline
No Keys & x & x\tabularnewline
\bottomrule
\end{longtable}

Importantly, the effect of the gas variable on driving depends on the
levels of having a key. Or, to state it in reverse, the effect of the
key variable on driving depends on the levesl of the gas variable.
Finally, in plain english. You need the keys and gas to drive.
Otherwise, there is no driving.

\subsection{What makes a people
hangry?}\label{what-makes-a-people-hangry}

To continue with more examples, let's consider an imaginary experiment
examining what makes people hangry. You may have been hangry before.
It's when you become highly irritated and angry because you are very
hungry\ldots{}hangry. I will propose an experiment to measure conditions
that are required to produce hangriness. The pretend experiment will
measure hangriness (we ask people how hangry they are on a scale from
1-10, with 10 being most hangry, and 0 being not hangry at all). The
first independent variable will be time since last meal (1 hour vs.~5
hours), and the second independent variable will be how tired someone is
(not tired vs very tired). I imagine the data could look something the
following bar graph.

\includegraphics{Factorial_files/figure-latex/unnamed-chunk-10-1}

The graph shows clear evidence of two main effects,
\emph{and an interaction}. There is a main effect of time since last
meal. Both the bars in the 1 hour conditions have smaller hanger ratings
than both of the bars in the 5 hour conditions. There is a main effect
of being tired. Both of the bars in the ``not tired'' conditions are
smaller than than both of the bars in the ``tired'' conditions. What
about the interaction?

Remember, an interaction occurs when the effect of one independent
variable depends on the level of the other independent variable. We can
look at this two ways, and either way shows the presence of the very
same interaction. First, does the effect of being tired depend on the
levels of the time since last meal? Yes. Look first at the effect of
being tired only for the ``1 hour condition''. We see the red bar
(tired) is 1 unit lower than the green bar (not\_tired). So, there is an
effect of 1 unit of being tired in the 1 hour condition. Next, look at
the effect of being tired only for the ``5 hour'' condition. We see the
red bar (tired) is 3 units lower than the green bar (not\_tired). So,
there is an effect of 3 units for being tired in the 5 hour condition.
Clearly, the size of the effect for being tired depends on the levels of
the time since last meal variable. We call this an interaction.

The second way of looking at the interaction is to start by looking at
the other variable. For example, does the effect of time since last meal
depend on the levels of the tired variable? The answer again is yes.
Look first at the effect of time since last meal only for the red bars
in the ``not tired'' condition. The red bar in the 1 hour condition is 1
unit smaller than the red bar in the 5 hour condition. Next, look at the
effect of time since last meal only for the green bars in the ``tired''
condition. The green bar in the 1 hour condition is 3 units smaller than
the green bar in the 5 hour condition. Again, the size of the effect of
time since last meal depends on the levels of the tired variable.No
matter which way you look at the interaction, we get the same numbers
for the size of the interaction effect, which is 2 units (a difference
between 3 and 1 = 2). The interaction suggests that something special
happens when people are tired and haven't eaten in 5 hours. In this
condition, they can become very hangry. Whereas, in the other
conditions, there are only small increases in being hangry.

\subsection{Identifying main effects and
interactions}\label{identifying-main-effects-and-interactions}

Research findings are often presented to readers using graphs or tables.
For example, the very same pattern of data can be displayed in a bar
graph, line graph, or table of means. These different formats can make
the data look different, even though the pattern in the data is the
same. An important skill to develop is the ability to identify the
patterns in the data, regardless of the format they are presented in.
Some examples of bar and line graphs are presented in the margin, and
two example tables are presented below. Each format displays the same
pattern of data.

\begin{marginfigure}
\includegraphics{Factorial_files/figure-latex/unnamed-chunk-11-1} \end{marginfigure}
\begin{marginfigure}
\includegraphics{Factorial_files/figure-latex/unnamed-chunk-11-2} \end{marginfigure}

\begin{longtable}[]{@{}lll@{}}
\toprule
DV I & V1 I & V2\tabularnewline
\midrule
\endhead
10 & Level\_1 & Level\_1\tabularnewline
12 & Level\_2 & Level\_1\tabularnewline
17 & Level\_1 & Level\_2\tabularnewline
13 & Level\_2 & Level\_2\tabularnewline
\bottomrule
\end{longtable}

\begin{verbatim}
##          df$IV2
## df$IV1    Level_1 Level_2
##   Level_1      10      17
##   Level_2      12      13
\end{verbatim}

After you become comfortable with interpreting data in these different
formats, you should be able to quickly identify the pattern of main
effects and interactions. For example, you would be able to notice that
all of these graphs and tables show evidence for two main effects and
one interaction.

As an exercise toward this goal, we will first take a closer look at
extracting main effects and interactions from tables. This exercise will
how the condition means are used to calculate the main effects and
interactions. Consider the table of condition means below.

\begin{longtable}[]{@{}cccc@{}}
\toprule
& & IV1 &\tabularnewline
\midrule
\endhead
& & A & B\tabularnewline
IV2 & 1 & 4 & 5\tabularnewline
& 2 & 3 & 8\tabularnewline
\bottomrule
\end{longtable}

\subsection{Main effects}\label{main-effects-1}

Main effects are the differences between the means of single independent
variable. Notice, this table only shows the condition means for each
level of all independent variables. So, the means for each IV must be
calculated. The main effect for IV1 is the comparison between level A
and level B, which involves calculating the two column means. The mean
for IV1 Level A is (4+3)/2 = 3.5. The mean for IV1 Level B is (5+8)/2 =
6.5. So the main effect is 3 (6.5 - 3.5). The main effect for IV2 is the
comparison between level 1 and level 2, which involves calculating the
two row means. The mean for IV2 Level 1 is (4+5)/2 = 4.5. The mean for
IV2 Level 2 is (3+8)/2 = 5.5. So the main effect is 1 (5.5 - 4.5). The
process of computing the average for each level of a single independent
variable, always involves collapsing, or averaging over, all of the
other conditions from other variables that also occured in that
condition

\subsection{Interactions}\label{interactions-1}

Interactions ask whether the effect of one independent variable depends
on the levels of the other independent variables. This question is
answered by computing difference scores between the condition means. For
example, we look the effect of IV1 (A vs.~B) for both levels of of IV2.
Focus first on the condition means in the first row for IV2 level 1. We
see that A=4 and B=5, so the effect IV1 here was 5-4 = 1. Next, look at
the condition in the second row for IV2 level 2. We see that A=3 and
B=8, so the effect of IV1 here was 8-3 = 5. We have just calculated two
differences (5-4=1, and 8-3=5). These difference scores show that the
size of the IV1 effect was different across the levels of IV2. To
calculate the interaction effect we simply find the difference between
the difference scores, 5-1=4. \emph{In general, if the difference
between the difference scores is different, then there is an interaction
effect.}

\subsection{Example bar graphs}\label{example-bar-graphs}

\includegraphics{Factorial_files/figure-latex/unnamed-chunk-13-1}

The IV1 graph shows a main effect only for IV1 (both red and green bars
are lower for level 1 than level 2). The IV1\&IV2 graphs shows main
effects for both variables. The two bars on the left are both lower than
the two on the right, and the red bars are both lower than the green
bars. The IV1xIV2 graph shows an example of a classic cross-over
interaction. Here, there are no main effects, just an interaction. There
is a difference of 2 between the green and red bar for Level 1 of IV1,
and a difference of -2 for Level 2 of IV1. That makes the differences
between the differences = 4. Why are their no main effects? Well the
average of the red bars would equal the average of the green bars, so
there is no main effect for IV2. And, the average of the red and green
bars for level 1 of IV1 would equal the average of the red and green
bars for level 2 of IV1, so there is no main effect. The bar graph for
IV2 shows only a main effect for IV2, as the red bars are both lower
than the green bars.

\subsection{Example line graphs}\label{example-line-graphs}

You may find that the patterns of main effects and interaction looks
different depending on the visual format of the graph. The exact same
patterns of data plotted up in bar graph format, are plotted as line
graphs for your viewing pleasure. Note that for the IV1 graph, the red
line does not appear because it is hidden behind the green line (the
points for both numbers are identical).

\includegraphics{Factorial_files/figure-latex/unnamed-chunk-14-1}

\subsection{Interpreting main effects and
interactions}\label{interpreting-main-effects-and-interactions}

The presence of an interaction can sometimes change how we interpet main
effects. For example, a really strong interaction can produce the
appearance of a main effect, even though when we look at the data most
people would agree the main effect is not there.

\includegraphics{Factorial_files/figure-latex/unnamed-chunk-15-1}

In the above graph there is clearly an interaction. IV2 has no effect
under level 1 of IV1 (e.g., the red and green bars are the same). IV2
has a large effect under level 2 of IV2 (the red bar is 2 and the green
bar is 9). So, the interaction effect is a total of 7. Are there any
main effects? This is a debatable question. Consider the main effect for
IV1. The mean for level 1 is (2+2)/2 = 2, and the mean for level 2 is
(2+9)/2 = 5.5. There is a difference between the means of 3.5, which is
consistent with a main effect. Consider, the main effect for IV2. The
mean for level 1 is again (2+2)/2 = 2, and the mean for level 2 is again
(2+9)/2 = 5.5. Again, there is a difference between the means of 3.5,
which is consistent with a main effect. What is going on here is that
the process of averagin over conditions that we use to compute main
effects is causing a main effect to appear, even though we don't really
see clear evidence of main effects.

Clear evidence of a main effect typically refers to cases where there is
a consistent additive influence. For example, if there really was a main
effect of IV1, then both red and green bars for level 2 should be
higher, not just one of them. In other words, the effect of IV1 did not
uniformly raise or lower the means across all of the other conditions.
For this reason, the main effects that we observed by performing the
calculation are really just an interaction in disguise.

The next example shows a case where it would be more appropriate to
conclude that the main effects and the interaction were both real.

\includegraphics{Factorial_files/figure-latex/unnamed-chunk-16-1}

Can you spot the interaction right away? The difference between red and
green bars is small for level 1 of IV1, but large for level 2. The
differences between the differences are different, so there is an
interaction. But, we also see clear evidence of two main effects. For
example, both the red and green bars for IV1 level 1 are higher than IV1
Level 2. And, both of the red bars (IV2 level 1) are higher than the
green bars (IV2 level 2).


\documentclass{article}
\usepackage{expsagetex}
\usepackage{amsmath}
\usepackage{parskip}
\usepackage{shortvrb}
\usepackage{csquotes}
\usepackage{expl3}
\usepackage[svgnames]{xcolor}
\usepackage{tikz}
\usepackage{xfp}
\usepackage{siunitx}
\sisetup{group-separator = \text{\,}}
\usepackage{hyperref}
\hypersetup{colorlinks=true, anchorcolor=Lime, linkcolor=RoyalBlue,
            urlcolor=Crimson}

\title{%
  \pkg{expsagetex} -- a fully expandable interface for SageTeX%
  \thanks{This file describes \pkg{expsagetex}~v0.1, last revised 2020-01-11.}%
}
\author{%
  Florent Rougon%
  \thanks{%
    E-mail: \href{mailto:f.rougon@free.fr}{mailto:f.rougon@free.fr}%
  }%
}

\newcommand*{\pkg}{\textsf}
\newcommand*{\meta}[1]{$\langle \textsl{#1} \rangle$}
\newcommand*{\tikzname}{Ti\emph{k}Z}

\setlength{\parindent}{0pt}     % will probably look better here

\begin{document}

\maketitle
\begin{abstract}
  This package is based on \texttt{sagetex.sty}, the style file for SageTeX.
  One deficiency of current \texttt{sagetex.sty} (version~3.3 from 2019/01/09)
  is that it doesn't provide any user-accessible way to retrieve a result
  computed by Sage in places where only expansion happens. This packages uses
  the low-level SageTeX machinery to implement a set of commands that can
  \enquote{record} results computed by Sage and a fully expandable command
  that can yield any such result wherever you need it, even in expansion-only
  contexts.
\end{abstract}

\MakeShortVerb{\|}

\tableofcontents
\clearpage

Besides the \nameref{sec:quick-start} section below, there is introductory
material to \pkg{expsagetex}
\href{https://tex.stackexchange.com/questions/521319/sagetex-1000sep-for-sage-calculated-number-siunitx/521389#521389}{on
  TeX.stackexchange.com}.

\section{Quick start}
\label{sec:quick-start}

Getting a result from Sage using \pkg{expsagetex} involves three steps in the
\LaTeX\ code:
\begin{enumerate}
\item Recording a Python expression to be written to the
  \texttt{.sagetex.sage} file. This step is done with one of the |\est*Record|
  functions.
\item Using the result where you need it. This is done with |\estGet| and
  works in many places, in particular in expansion-only contexts (inside
  |\edef|, |\message|, |\write|, etc.).
\item Declaring that you use the result, so that a warning can be displayed if
  the result isn't available yet (|\estGet| can't print the warning itself
  because it must work in expansion-only contexts). This is done with
  |\estRefUsed|.
\end{enumerate}

Step~3 can be done before or after step~2, but must come after step~1. When
you run |sage| on the \texttt{.sagetex.sage} file written by step~1, the
Python expression is evaluated, converted to a Python string and written to
the |.sagetex.sout| file. This file is in turn read by \texttt{sagetex.sty} at
startup, which makes the result available for |\estGet|.

Thus, a very simple example can be
\estRecordFormatted{\littleFermatExple}{(45**6) \percent 7}%
\estRefUsed{\littleFermatExple}%
$45^6 \equiv \estGet{\littleFermatExple} \pmod 7$. This was obtained with:
\begin{verbatim}
\estRecordFormatted{\littleFermatExple}{(45**6) \percent 7}%
\estRefUsed{\littleFermatExple}%
$45^6 \equiv \estGet{\littleFermatExple} \pmod 7$
\end{verbatim}

For compiling, use the same method as with \pkg{sagetex} (which
\pkg{expsagetex} relies on): you need one \LaTeX\ run, one |sage| run on the
\texttt{.sagetex.sage} file followed by at least one more \LaTeX\ run (if more
are needed, warnings are printed by \LaTeX\ as usual).

\section{Examples}

\subsection{Integers}

\subsubsection{Local assignments}

When using |\estRecordFormatted{|\meta{macro}|}{|\meta{expr}|}| with a Python
expression \meta{expr} that evaluates to an integer,
|\estGet[|\meta{fallback}|]{|\meta{macro}|}| should give (i.e., expand to) the
expected result. This is because calling |str()| on a Python integer doesn't
use any exponent notation, thus |latex()| should not add any formatting. Let's
execute the following Python statement using the |sageblock| environment:

\begin{sageblock}
x = 2**333
\end{sageblock}

Now, record the value of $x$ and associate it with macro |\mymacro|. This
makes |\mymacro| a kind of reference to the saved value, but to retrieve the
tokens associated to this reference, you have to use |\estGet| instead of
|\ref| (this can be done in an expansion-only context like inside |\edef|,
which is impossible with |\ref|). See section~\ref{sec:technical-details}
below for more details on this.
{%
  \estRecordFormatted{\mymacro}{x}%
  Recover the saved value and format it with the |\num| macro of
  \pkg{siunitx}:\\
  \num{\estGet[-1]{\mymacro}}.%
  \estRefUsed{\mymacro}\\
  This was achieved with the following calls:
\begin{verbatim}
\estRecordFormatted{\mymacro}{x}
\num{\estGet[-1]{\mymacro}}
\estRefUsed{\mymacro}
\end{verbatim}
}%
\ifx\mymacro\undefined\else
  \par Bug: |\mymacro| should be undefined here!
\fi

Another way to do the same without having Sage call |latex()| at all is to
use |\estRecordStr| to record a string representation of the integer we are
interested in:
\begin{verbatim}
\estRecordStr{\mymacro}{str(x)}
\num{\estGet[-1]{\mymacro}}
\estRefUsed{\mymacro}
\end{verbatim}

\pagebreak[3]

Notes:
\begin{itemize}
\item |\estGet| wraps its return value within |\unexpanded| (|\exp_not:n|),
  which implies that it won't expand further when used in |\edef| or an
  \pkg{expl3} |x|-type argument.
\item In order to use the result of evaluating a Python expression in the
  \LaTeX\ document, it is not necessary to store it in a Python variable.
  Example:%
  \estRecordFormatted{\twoToTheThenth}{2**10}%
  \estRefUsed{\twoToTheThenth}%
  \[ 2^{10} = \estGet{\twoToTheThenth} \]
\end{itemize}

Another example with variables:

\begin{sageblock}
x = 23 % 8         # That is, x = 7.
y = 2*x
\end{sageblock}

Save the value of $1000000\times (y-10)$ (as computed by Sage) and associate
it with macro |\numberBasedOnY|.
\estRecordFormatted{\numberBasedOnY}{1000000*(y-10)}%
% Use the value -1000000 when Sage hasn't written the result to the .sout file
% yet.
\[ 1000000\times (y - 10) = \num{\estGet[-1000000]{\numberBasedOnY}} \]
\estRefUsed{\numberBasedOnY}

\subsubsection{Global assignments}

While |\estRecordFormatted| assigns locally to the macro given by its
first argument, all |\est*Record| functions have a global variant that
performs a global assignment instead. The global variant of
|\estRecordFormatted| is |\estGRecordFormatted|, that of
|\estRecordStr| is |\estGRecordStr|, that of
|\estARecordFormatted| is |\estGARecordFormatted|, etc. Example:

\begin{sageblock}
x = 3+4
\end{sageblock}

Save the value of $x$ and globally assign macro |\Iamseven| to allow us
to retrieve the saved value.
{%
  \estGRecordFormatted{\Iamseven}{x}%
}%
Get it back from outside the group where this was done, and format it with
|\num|: the result is
\num{\estGet[-2]{\Iamseven}}.%
\estRefUsed{\Iamseven}

\subsection{Floating point numbers}

|\estRecordFormatted| works with floating point numbers, but it uses \LaTeX\
markup if the string representation of the result (in Python) uses exponent
notation. When this is not desirable---i.e., when you just want a numerical
result without any formatting---use |\estRecordFloat| or one of its variants.
Example:
\begin{sageblock}
x = 3.1415927
\end{sageblock}
\estRecordFloat{\PiRoundedByPython}{x}{.2f}%
\estRefUsed{\PiRoundedByPython}%
$x$ rounded to two decimal places is \estGet[0]{\PiRoundedByPython}. This was
rounded by Python.
\estRecordFloat{\PiRoundedBySiunitx}{x}{.10f}%
\estRefUsed{\PiRoundedBySiunitx}%
Thanks to the expandable nature of |\estGet|, one can also do this: $x$
rounded to four decimal places is
\num[round-mode=places, round-precision=4]{\estGet[0]{\PiRoundedBySiunitx}}.
This was rounded by \pkg{siunitx}.

Of course, one can also compute a float with Sage without storing it into a
Python variable. Here is an approximation of $\pi$ using an expression based
on Machin's formula and the power series expansion of $\arctan$:
\estRecordFloat{\PiViaMachinAndPowerSeries}
  {
    N(4*sum([ (-1)**n / (2*n+1) * (4*(1/5)**(2*n+1) - (1/239)**(2*n+1))
              for n in range(10) ]))
  }{.6f}%
\estRefUsed{\PiViaMachinAndPowerSeries}%
\begin{align*}
  \pi &= 4\sum_{n=0}^{+\infty} \frac{(-1)^n}{(2n+1)}
         \Biggl( 4 \, {\Bigl( \frac{1}{5} \Bigr)}^{2n+1} \! -
                 {\Bigl( \frac{1}{239} \Bigr)}^{2n+1} \Biggr)\\
      &\approx \estGet[0]{\PiViaMachinAndPowerSeries}
\end{align*}

\subsection{Strings}

Functions working on Python strings are the most general here. For instance,
|\estRecordFormatted| first evaluates |latex(...)| in Sage (Python), where
|...| is the Python expression you entered; this results in a string with
\LaTeX\ math-mode formatting commands (e.g., if evaluating |str(...)| in Python string yields something containing brackets or
exponents); then this string is written to the |.sout| file inside an
argument of |\newlabel|, which \pkg{sagetex} reads at startup. Similarly,
|\estRecordFloat| causes Python to convert a float to a string using the
user-specified format, then this string is written to the |.sout| file
inside an argument of |\newlabel|, just as with
|\estRecordFormatted|. This applies to global and array variants of the
\pkg{expsagetex} functions as well.

So, |\estRecordStr| and its variants can be very useful in case your
Python code returns a data type that isn't handled in a satisfactory way by
functions belonging to the families of |\estRecordFormatted| or
|\estRecordFloat|. All you have to do is to make your Python code format
your data as a string with a syntax that is convenient for the \LaTeX\
document; you record this string with |\estRecordStr| or one of its
variants, use |\estGet| and |\estRefUsed| as usual and voilà, your new data
type is nicely handled by \pkg{expsagetex}. We'll see an example of this
approach in
section~\ref{sec:plotting-with-tikzname-based-on-Sage-computations}, where
Python code generates a large amount of $(x_i, y_i)$ coordinates from two
lists of floats $[x_1, \dotsc, x_n]$ and $[y_1, \dotsc, y_n]$. The Python code
formats the list of $(x_i, y_i)$ coordinates in a form that is suitable for
the \tikzname\ \texttt{plot coordinates} operation, which allows one to plot
the data right away.

Here is a very simple example with |\estRecordStr| used to record a Sage
(Python) string:
\begin{sageblock}
 some_string = "abc  def"
\end{sageblock}
\estRecordStr{\somestring}{some_string}%
X\estGet{\somestring}Y%
\estRefUsed{\somestring}

The two spaces between |abc| and |def| are present in the
|.sout| file, but get coalesced into a single space token when
|\newlabel| tokenizes its second argument. If this is not desired, the
easiest way is probably to use another character instead of space (e.g.,
\verb|~|).

\subsection{Using array data}
\label{sec:using-array-data}

\pkg{expsagetex} offers variants of |\estRecordFormatted|,
|\estGRecordFormatted|, |\estRecordFloat|, |\estGRecordFloat|, |\estRecordStr|
and |\estGRecordStr| that allow one to easily use an index inside the macro
name given for the recording operation. These variants all have the letter
\texttt{A} (standing for ``array'') right before \texttt{Record}.

Where the non-array versions of the |\est*Record*| commands take one argument
specifying the destination macro name, the array versions accept two arguments
\meta{base} and \meta{index} and determine the destination macro name as the
result of expanding |\csname |\meta{base}@\meta{index}|\endcsname|. As a
consequence, the \meta{base} and \meta{index} arguments are recursively
expanded, which can be used to dynamically compute a numeric index using for
instance |\numexpr|, or whatever you want.

Here is an example that uses Python to evaluate the elements of a Vandermonde
matrix:
\ExplSyntaxOn
\int_new:N \l__my_row_int
\int_new:N \l__my_col_int
\int_new:N \g__my_row_counter_int
\clist_const:Nn \c__my_clist { 2, sqrt(2)/2, 2/7, 4/3, 1/5 }
\clist_const:Nn \c__my_formatted_clist
  { 2, \frac{\sqrt{2}}{2}, \frac{2}{7}, \frac{4}{3}, \frac{1}{5} }
\seq_new:N \l__my_matrix_rows_seq

\int_step_variable:nnNn { 1 } { 5 } \l__my_row_int
  {
    % One l3seq per matrix row
    \seq_clear_new:c { l__my_matrix_row \l__my_row_int _seq }
    % \use:c needs two expansion steps to do its work
    \exp_args:NNo \seq_put_right:No \l__my_matrix_rows_seq
      { \use:c { l__my_matrix_row \l__my_row_int _seq } }

    \int_step_variable:nnNn { 1 } { 5 } \l__my_col_int
      {
        \tl_set:Nx \l_tmpa_tl
          {
            ( \clist_item:Nn \c__my_clist { \l__my_row_int } ) **
                                                          ( \l__my_col_int - 1 )
          }
        \estARecordFormatted { Vandermonde } { \l__my_row_int - \l__my_col_int }
          { \tl_use:N \l_tmpa_tl }
        \seq_put_right:cx { l__my_matrix_row \l__my_row_int _seq }
          { \estAGet [0] { Vandermonde } { \l__my_row_int - \l__my_col_int } }
        \estARefUsed { Vandermonde } { \l__my_row_int - \l__my_col_int }
      }
  }

\[ \renewcommand{\arraystretch}{1.2}
  \mathcal{V} \Bigl( \clist_use:Nn \c__my_formatted_clist { , } \Bigr) =
   \begin{bmatrix}
     \int_gzero:N \g__my_row_counter_int
     \seq_map_inline:Nn \l__my_matrix_rows_seq
       {
         \int_gincr:N \g__my_row_counter_int
         \seq_use:Nn #1 { & }
         % Append \\ unless it is the last line
         \int_compare:nNnT
           { \g__my_row_counter_int } < { \seq_count:N \l__my_matrix_rows_seq }
           { \\ }
       }
   \end{bmatrix} \]
\ExplSyntaxOff

This technique can be used to plot curves with \tikzname\ based on coordinates
computed by Python, but this is rather wasteful. We'll see a better method
in section~\ref{sec:plotting-with-tikzname-based-on-Sage-computations}.

\subsection{Displaying lists}
\label{sec:lists}

|\estRecordFormatted| is fine for a list of integers:%
\estRecordFormatted{\listA}{list(range(2, 21, 3))}%
\begin{quote}
  \estRefUsed{\listA}%
  $\estGet[{[]}]{\listA}$
\end{quote}

It may do what you want for floats too:
\estRecordFormatted{\listB}{map(float, list(range(2, 21, 3)))}%
\begin{quote}
  \estRefUsed{\listB}%
  $\estGet[{[]}]{\listB}$
\end{quote}

But beware: if you use |\estRecordFormatted|, floats are formatted by Sage's
|latex()| function:
\estRecordFormatted{\listC}{[1e-10] + [5.0, 3e-14]}
\begin{quote}
  \small
  \estRefUsed{\listC}%
  $\estGet[{[]}]{\listC}$
\end{quote}

Here is a way to use a Python list of floats formatted with no exponent
and using a chosen number of decimal places:%
% We format each float individually as a string. We could as well omit the
% outer brackets: they are just string characters here (the final assembled
% expression is a Python string; it is *not* evaluated as a list by Python).
\estRecordStr{\listD}
  {
    '[' +
    ', '.join(map(lambda x: "\percent.14f" \percent (x,), [1e-10, 5.0, 3e-14])) +
    ']'
  }
\begin{quote}
  \estRefUsed{\listD}%
  $\estGet[{[]}]{\listD}$
\end{quote}

\subsection{Plotting with \tikzname\ based on Sage computations}
\label{sec:plotting-with-tikzname-based-on-Sage-computations}

Plotting functions based on coordinates computed by Sage can be done using
array-style functions of \pkg{expsagetex} (see
section~\ref{sec:using-array-data}), however such an approach is vastly
inefficient if the plot uses a significant number of points. A better approach
is to prepare one string on the Python side containing all needed coordinates
in a format that the \LaTeX\ plotting code can easily handle. This way, only
one reference---as in |\label| and |\ref|---is used to pass data for the whole
plot. As an example, here is a part of the Euler spiral where all coordinates
are computed by Sage and transferred in one go to the \LaTeX\ side (for the
full Euler Spiral, you need to make $t$ vary from $-\infty$ to $+\infty$).
%
\begin{sagesilent}
  x = var('x')
  start = -5*pi/2
  end = 5*pi/2
  nb_samples = 2000
  delta = float(end-start)/(nb_samples-1)
\end{sagesilent}
%
\estRecordStr{\computedCoordsRef}
  {
    " ".join([ "(\percent.6f, \percent.6f)" \percent
               (0.1*( numerical_integral(cos(x**2), 0, t)[0] ),
                0.1*( numerical_integral(sin(x**2), 0, t)[0] ))
               for t in ( start + _*delta for _ in range(nb_samples) ) ])
  }
\edef\computedCoords{\estGet[(0,0)]{\computedCoordsRef}}
\estRefUsed{\computedCoordsRef}
%
\begin{center}
\begin{tabular}{cc}
\raisebox{-0.5\height}{%
  \begin{tikzpicture}
    \draw[red!60!black, scale=20] plot coordinates { \computedCoords };
  \end{tikzpicture}%
}\hspace*{0.06\linewidth} &
\hspace*{0.06\linewidth}%
$ \newcommand{\diff}{\mathop{}\!\mathrm{d}}
  \left\{
    \begin{array}{>{\displaystyle}l}
      x(t) = \int_{0}^{t} \cos s^2 \diff s\\[10pt]
      y(t) = \int_{0}^{t} \sin s^2 \diff s
    \end{array}
  \right. ,\ t \in [-\frac{5\pi}{2}, \frac{5\pi}{2}]$
\end{tabular}
\end{center}

\section{Expandable commands work everywhere!}

Since |\estGet| is expandable and we have |\usepackage{xfp}| in the preamble,
we can do:
\begin{verbatim}
$\fpeval{3*\estGet[-1]{\Iamseven}}$
\end{verbatim}
This expands to $\fpeval{3*\estGet[-1]{\Iamseven}}$, since |\Iamseven|
contains the last value of $x$ computed by Sage, which is $7$ (if Sage hasn't
been run yet, it expands to $-3$ due to the fallback value \texttt{-1}
specified in the optional argument of |\estGet|).%
\estRefUsed{\Iamseven}%

{%
  Given that \TeX\ expands the right-hand side of integer, dimen and glue
  assignments until this yields the proper syntactic element, one can also use
  |\estGet| like this:
  \begin{verbatim}
  \count2=4
  \multiply \count2 by \estGet[-1]{\Iamseven}
  \the\count2
  \estRefUsed{\Iamseven}% just to be clean
  \end{verbatim}
  \count2=4
  \multiply \count2 by \estGet[-1]{\Iamseven}
  $\rightarrow$ \the\count2 % prints 28 (4*7)
  \estRefUsed{\Iamseven}
}

\bigskip
Same thing with a |\dimen| register:
\begin{verbatim}
\dimen0=1pt
\dimen0=\estGet[-1]{\Iamseven}\dimen0
\the\dimen0
\estRefUsed{\Iamseven}% just to be clean
\end{verbatim}
\dimen0=1pt
\dimen0=\estGet[-1]{\Iamseven}\dimen0
$\rightarrow$ \the\dimen0 % prints 7.0pt
\estRefUsed{\Iamseven}

\newcommand*{\grammarsymbol}[1]{\mbox{\meta{#1}}}%
And since \LaTeX\ lengths are |\skipdef| tokens and |\setlength| is
basically a glue assignment, one can have fun with
|\estGet[-1]{\Iamseven}| expanding to the \grammarsymbol{factor} of
a \grammarsymbol{normal dimen} (cf.~\TeX book p.~270):
\begin{verbatim}
\dimen0=2pt
\dimen2=3pt
\newlength{\mylength}
\setlength{\mylength}{%
       \estGet[-1]{\Iamseven}\dimen0
  plus \estGet[-1]{\Iamseven}\dimen2}%
\the\mylength
\estRefUsed{\Iamseven}% just to be clean
\end{verbatim}
\dimen0=2pt
\dimen2=3pt
\newlength{\mylength}
\setlength{\mylength}{%
       \estGet[-1]{\Iamseven}\dimen0
  plus \estGet[-1]{\Iamseven}\dimen2}%
$\rightarrow$ \the\mylength % prints 14.0pt plus 21.0pt
\estRefUsed{\Iamseven}

\section{Troubleshooting}

In case something goes wrong when using \pkg{sagetex} or \pkg{expsagetex}, you
may get stuck and unable to perform a full \LaTeX\ run. This can for instance
happen if one of your Python expressions is invalid. This kind of error is
similar to problems that may occur when something ``bad'' was written to the
|.aux| file. The cure is similar too: locate the error (inspecting the
\texttt{.sagetex.sage} and \texttt{.sagetex.sout} files can help; if the
problem is an invalid Python expression, the |sage| run will tell you),
regenerate the files containing invalid things after fixing their source, then
rerun the \LaTeX\ and |sage| commands as usual. Typically, you'll fix a Python
expression in your |.tex| file, remove the |.sagetex.sout| file and rerun
\LaTeX, |sage| and again \LaTeX. If you want to be \emph{really} sure to
restart from a clean state, remove the \texttt{.sagetex.sage},
\texttt{.sagetex.sout} and |.aux| files before redoing the \LaTeX\ \& |sage|
dance.

\section{Technical details}
\label{sec:technical-details}

Some comments about what a call such as |\estRecordStr{\mymacro}{|$s$|}| as
seen below really does ($s$ must evaluate in Python to a string). Informally,
one might be tempted to say that this saves the value of $s$ in the specified
macro, but that would be rather inaccurate. After the call to |\estRecordStr|,
the specified macro contains an integer which is used to form a reference
name---a label, if you wish---through which we can get the value of $s$ using
inner gears of the |\label| and |\ref| machinery. The \texttt{.sagetex.sout}
file written when running Sage on the \texttt{.sagetex.sage} file contains
|\newlabel| commands that define the associated text (tokens) for reference
names \texttt{@sageinline0}, \texttt{@sageinline1}, etc. The trailing number
is what is really stored by the above call to |\estRecordStr| as the
replacement text of |\mymacro|. \texttt{sagetex.sty} reads this
\texttt{.sagetex.sout} file at startup when it exists, which defines the
labels from the point of view of the \LaTeX\ kernel and allows us to retrieve
the Sage output for each recorded expression (after the |\newlabel| commands
have been executed, the Sage outputs are available, after some simple data
extraction, in macros |\r@@sageinline0|, |\r@@sageinline1|, etc.).

\end{document}


\nwfilename{_src/2019/day/01.nw}\nwbegindocs{0}\newpage% ===> this file was generated automatically by noweave --- better not edit it
\section{Day 1: The Tyranny of the Rocket Equation}
\todoo{Copy description}
\marginnote{\url{https://adventofcode.com/2019/day/1}}
\nwenddocs{}\nwfilename{_src/2019/gap/01.nw}\nwbegindocs{0}\subsection{GAP Solution}

\begin{marginfigure}
\[
 \text{fuel} := \text{mass} \backslash 3 - 2
\]
\end{marginfigure}
\nwenddocs{}\nwbegincode{1}\sublabel{NW3RAD1b-3gOu99-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-1}}}\moddef{Day01.g~{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-1}}}\endmoddef\nwstartdeflinemarkup\nwprevnextdefs{\relax}{NW3RAD1b-3gOu99-2}\nwenddeflinemarkup
FuelRequiredModule := function( mass )
    return Int( Float( mass / 3 ) ) - 2;
end;;

\nwalsodefined{\\{NW3RAD1b-3gOu99-2}\\{NW3RAD1b-3gOu99-3}\\{NW3RAD1b-3gOu99-4}}\nwnotused{Day01.g}\nwendcode{}\nwbegindocs{2}\nwdocspar

\nwenddocs{}\nwbegincode{3}\sublabel{NW3RAD1b-3gOu99-2}\nwmargintag{{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-2}}}\moddef{Day01.g~{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-1}}}\plusendmoddef\nwstartdeflinemarkup\nwprevnextdefs{NW3RAD1b-3gOu99-1}{NW3RAD1b-3gOu99-3}\nwenddeflinemarkup
PartOne := function( )
    local input, line, mass, sum;;
    sum := 0;
    input := InputTextFile ( "./input/day01.txt" );
    line := ReadLine( input );
    repeat
        mass := Int( Chomp( line ) );
        sum := sum + FuelRequiredModule( mass );
        line := ReadLine( input );
    until line = fail or IsEndOfStream( input );
    return sum;
end;;

\nwendcode{}\nwbegindocs{4}\nwdocspar

\nwenddocs{}\nwbegincode{5}\sublabel{NW3RAD1b-3gOu99-3}\nwmargintag{{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-3}}}\moddef{Day01.g~{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-1}}}\plusendmoddef\nwstartdeflinemarkup\nwprevnextdefs{NW3RAD1b-3gOu99-2}{NW3RAD1b-3gOu99-4}\nwenddeflinemarkup
TotalFuelRequiredModule := function( mass )
    local fuel;;
    fuel := FuelRequiredModule( mass );
    if IsPosInt( fuel ) then
        return fuel + TotalFuelRequiredModule( fuel );
    else
        return 0;
    fi;
end;;

\nwendcode{}\nwbegindocs{6}\nwdocspar

\nwenddocs{}\nwbegincode{7}\sublabel{NW3RAD1b-3gOu99-4}\nwmargintag{{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-4}}}\moddef{Day01.g~{\nwtagstyle{}\subpageref{NW3RAD1b-3gOu99-1}}}\plusendmoddef\nwstartdeflinemarkup\nwprevnextdefs{NW3RAD1b-3gOu99-3}{\relax}\nwenddeflinemarkup
PartTwo := function( )
    local input, line, mass, sum;;
    sum := 0;
    input := InputTextFile ( "./input/day01.txt" );
    line := ReadLine( input );
    repeat
        mass := Int( Chomp( line ) );
        sum := sum + TotalFuelRequiredModule( mass );
        line := ReadLine( input );
    until line = fail or IsEndOfStream( input );
    return sum;
end;;
\nwendcode{}\nwbegindocs{8}\nwdocspar
\nwenddocs{}\nwfilename{_src/2019/day/04.nw}\nwbegindocs{0}\newpage
\section{Day 4: Secure Container}
\todoo{Copy description}
\marginnote{\url{https://adventofcode.com/2019/day/4}}
\nwenddocs{}\nwfilename{_src/2019/haskell/04.nw}\nwbegindocs{0}\subsection{Haskell Solution}

\newthought{My puzzle input} was the range \text{236491-713787}, which I converted into a
list of lists of \hs{digits}.

\nwenddocs{}\nwbegincode{1}\sublabel{NW35miTa-1GvnV-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW35miTa-1GvnV-1}}}\moddef{Input~{\nwtagstyle{}\subpageref{NW35miTa-1GvnV-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}}\nwenddeflinemarkup
getInput :: IO [[Int]]
getInput = pure $ reverse . digits 10 <$> [236491 .. 713787]
\nwused{\\{NW35miTa-15Rjc8-1}}\nwendcode{}\nwbegindocs{2}\nwdocspar

\newthought{Spoiler:} Parts One and Two vary only in the strictness of the definition of a double, so a generic solver can be parameterized by the binary operation to compare the number of adjacent digits that are the same with \hs{2}. In both parts of the puzzle, it must also be the case that the digits never decrease, i.e. the password \hs{isSorted}.

\nwenddocs{}\nwbegincode{3}\sublabel{NW35miTa-2ApJRg-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW35miTa-2ApJRg-1}}}\moddef{Generic solver~{\nwtagstyle{}\subpageref{NW35miTa-2ApJRg-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}}\nwenddeflinemarkup
solve :: (Int -> Int -> Bool) -> [[Int]] -> Int
solve = count . (isSorted <&&>) . hasDouble
  where
    hasDouble cmp = any ((`cmp` 2) . length) . group
\nwused{\\{NW35miTa-15Rjc8-1}}\nwendcode{}\nwbegindocs{4}\nwdocspar

\newthought{For Part One,} there must be a double, i.e. at least two adjacent digits that are the same.

\nwenddocs{}\nwbegincode{5}\sublabel{NW35miTa-2iOjQS-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}}\moddef{Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{\relax}{NW3AzXtx-2iOjQS-1}\nwenddeflinemarkup
\nwlinkedidentc{partOne}{NW3AzXtx-2iOjQS-4} :: [[Int]] -> Int
\nwlinkedidentc{partOne}{NW3AzXtx-2iOjQS-4} = solve (>=)
\nwalsodefined{\\{NW3AzXtx-2iOjQS-1}\\{NW3AzXtx-2iOjQS-2}\\{NW3AzXtx-2iOjQS-3}\\{NW3AzXtx-2iOjQS-4}}\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{partOne}}{partOne}}}\nwindexuse{\nwixident{partOne}}{partOne}{NW35miTa-2iOjQS-1}\nwendcode{}\nwbegindocs{6}\nwdocspar

\newthought{For Part Two,} the password must have a strict double.

\nwenddocs{}\nwbegincode{7}\sublabel{NW35miTa-4P9qKy-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW35miTa-4P9qKy-1}}}\moddef{Part Two~{\nwtagstyle{}\subpageref{NW35miTa-4P9qKy-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{\relax}{NW3AzXtx-4P9qKy-1}\nwenddeflinemarkup
\nwlinkedidentc{partTwo}{NW3AzXtx-4P9qKy-1} :: [[Int]] -> Int
\nwlinkedidentc{partTwo}{NW3AzXtx-4P9qKy-1} = solve (==)
\nwalsodefined{\\{NW3AzXtx-4P9qKy-1}}\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{partTwo}}{partTwo}}}\nwindexuse{\nwixident{partTwo}}{partTwo}{NW35miTa-4P9qKy-1}\nwendcode{}\nwbegindocs{8}\nwdocspar

\newthought{Bring it} all together.

\nwenddocs{}\nwbegincode{9}\sublabel{NW35miTa-15Rjc8-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW35miTa-15Rjc8-1}}}\moddef{Day04.hs~{\nwtagstyle{}\subpageref{NW35miTa-15Rjc8-1}}}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
module AdventOfCode.Year2019.Day04 where

import AdventOfCode.TH (defaultMain)
import AdventOfCode.Util (count, (<&&>))
import Data.FastDigits (digits)
import Data.List (group)
import Data.List.Ordered (isSorted)

\nwlinkedidentc{main}{NW3AzXtx-19Srvv-1} :: IO ()
\nwlinkedidentc{main}{NW3AzXtx-19Srvv-1} = $(defaultMain)

\LA{}Input~{\nwtagstyle{}\subpageref{NW35miTa-1GvnV-1}}\RA{}

\LA{}Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}\RA{}

\LA{}Part Two~{\nwtagstyle{}\subpageref{NW35miTa-4P9qKy-1}}\RA{}

\LA{}Generic solver~{\nwtagstyle{}\subpageref{NW35miTa-2ApJRg-1}}\RA{}
\nwnotused{Day04.hs}\nwidentuses{\\{{\nwixident{main}}{main}}}\nwindexuse{\nwixident{main}}{main}{NW35miTa-15Rjc8-1}\nwendcode{}\nwbegindocs{10}\nwdocspar
\nwenddocs{}\nwfilename{_src/2019/day/08.nw}\nwbegindocs{0}\newpage
\section{Day 8: }\todor{Add missing title}
\todoo{Copy description}
\marginnote{\url{https://adventofcode.com/2019/day/8}}
\nwenddocs{}\nwfilename{_src/2019/haskell/08.nw}\nwbegindocs{0}\subsection{Haskell solution}

\newthought{A pixel} can be black, white, or transparent.

\nwenddocs{}\nwbegincode{1}\sublabel{NW3AzXtx-2M5oYw-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-2M5oYw-1}}}\moddef{Define a Pixel data type~{\nwtagstyle{}\subpageref{NW3AzXtx-2M5oYw-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW3AzXtx-19Srvv-1}}\nwenddeflinemarkup
data \nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1}
  = \nwlinkedidentc{Black}{NW3AzXtx-2M5oYw-1}
  | \nwlinkedidentc{White}{NW3AzXtx-2M5oYw-1}
  | \nwlinkedidentc{Transparent}{NW3AzXtx-2M5oYw-1}
  deriving (Enum, Eq)
\nwindexdefn{\nwixident{Pixel}}{Pixel}{NW3AzXtx-2M5oYw-1}\eatline
\nwindexdefn{\nwixident{Black}}{Black}{NW3AzXtx-2M5oYw-1}\eatline
\nwindexdefn{\nwixident{White}}{White}{NW3AzXtx-2M5oYw-1}\eatline
\nwindexdefn{\nwixident{Transparent}}{Transparent}{NW3AzXtx-2M5oYw-1}\eatline
\nwused{\\{NW3AzXtx-19Srvv-1}}\nwidentdefs{\\{{\nwixident{Black}}{Black}}\\{{\nwixident{Pixel}}{Pixel}}\\{{\nwixident{Transparent}}{Transparent}}\\{{\nwixident{White}}{White}}}\nwendcode{}\nwbegindocs{2}\nwdocspar

Show black pixels as spaces, white ones as hashes, and transparent as dots.

\nwenddocs{}\nwbegincode{3}\sublabel{NW3AzXtx-QyGx2-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-QyGx2-1}}}\moddef{Implement \hs{Show} for \code{}Pixel\edoc{}~{\nwtagstyle{}\subpageref{NW3AzXtx-QyGx2-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW3AzXtx-19Srvv-1}}\nwenddeflinemarkup
instance Show \nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1} where
  show \nwlinkedidentc{Black}{NW3AzXtx-2M5oYw-1} = " "
  show \nwlinkedidentc{White}{NW3AzXtx-2M5oYw-1} = "#"
  show \nwlinkedidentc{Transparent}{NW3AzXtx-2M5oYw-1} = "."
\nwused{\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{Black}}{Black}}\\{{\nwixident{Pixel}}{Pixel}}\\{{\nwixident{Transparent}}{Transparent}}\\{{\nwixident{White}}{White}}}\nwindexuse{\nwixident{Black}}{Black}{NW3AzXtx-QyGx2-1}\nwindexuse{\nwixident{Pixel}}{Pixel}{NW3AzXtx-QyGx2-1}\nwindexuse{\nwixident{Transparent}}{Transparent}{NW3AzXtx-QyGx2-1}\nwindexuse{\nwixident{White}}{White}{NW3AzXtx-QyGx2-1}\nwendcode{}\nwbegindocs{4}\nwdocspar

\newthought{Define a {\Tt{}\nwlinkedidentq{Layer}{NW3AzXtx-LSl4Q-1}\nwendquote}} as a list of {\Tt{}\nwlinkedidentq{Row}{NW3AzXtx-LSl4Q-1}\nwendquote}s, and a {\Tt{}\nwlinkedidentq{Row}{NW3AzXtx-LSl4Q-1}\nwendquote} as a list of {\Tt{}\nwlinkedidentq{Pixel}{NW3AzXtx-2M5oYw-1}\nwendquote}s.

\nwenddocs{}\nwbegincode{5}\sublabel{NW3AzXtx-LSl4Q-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-LSl4Q-1}}}\moddef{Define a few convenient type aliases~{\nwtagstyle{}\subpageref{NW3AzXtx-LSl4Q-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW3AzXtx-19Srvv-1}}\nwenddeflinemarkup
type \nwlinkedidentc{Image}{NW3AzXtx-LSl4Q-1} = [\nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1}]

type \nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1} = [\nwlinkedidentc{Row}{NW3AzXtx-LSl4Q-1}]

type \nwlinkedidentc{Row}{NW3AzXtx-LSl4Q-1} = [\nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1}]
\nwindexdefn{\nwixident{Image}}{Image}{NW3AzXtx-LSl4Q-1}\eatline
\nwindexdefn{\nwixident{Layer}}{Layer}{NW3AzXtx-LSl4Q-1}\eatline
\nwindexdefn{\nwixident{Row}}{Row}{NW3AzXtx-LSl4Q-1}\eatline
\nwused{\\{NW3AzXtx-19Srvv-1}}\nwidentdefs{\\{{\nwixident{Image}}{Image}}\\{{\nwixident{Layer}}{Layer}}\\{{\nwixident{Row}}{Row}}}\nwidentuses{\\{{\nwixident{Pixel}}{Pixel}}}\nwindexuse{\nwixident{Pixel}}{Pixel}{NW3AzXtx-LSl4Q-1}\nwendcode{}\nwbegindocs{6}\nwdocspar

\newthought{Parse an {\Tt{}\nwlinkedidentq{Image}{NW3AzXtx-LSl4Q-1}\nwendquote},} i.e. one or more {\Tt{}\nwlinkedidentq{Layer}{NW3AzXtx-LSl4Q-1}\nwendquote}s comprised of \hs{height}
{\Tt{}\nwlinkedidentq{Row}{NW3AzXtx-LSl4Q-1}\nwendquote}s of \hs{width} {\Tt{}\nwlinkedidentq{Pixel}{NW3AzXtx-2M5oYw-1}\nwendquote}s.

\nwenddocs{}\nwbegincode{7}\sublabel{NW3AzXtx-4aeb4o-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-4aeb4o-1}}}\moddef{Parse an image~{\nwtagstyle{}\subpageref{NW3AzXtx-4aeb4o-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW3AzXtx-19Srvv-1}}\nwenddeflinemarkup
\nwlinkedidentc{image}{NW3AzXtx-4aeb4o-1} :: Int -> Int -> Parser \nwlinkedidentc{Image}{NW3AzXtx-LSl4Q-1}
\nwlinkedidentc{image}{NW3AzXtx-4aeb4o-1} width height = some layer
  where
    layer :: Parser \nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1}
    layer = count height row
    row :: Parser \nwlinkedidentc{Row}{NW3AzXtx-LSl4Q-1}
    row = count width pixel
\nwindexdefn{\nwixident{image}}{image}{NW3AzXtx-4aeb4o-1}\eatline
\nwused{\\{NW3AzXtx-19Srvv-1}}\nwidentdefs{\\{{\nwixident{image}}{image}}}\nwidentuses{\\{{\nwixident{Image}}{Image}}\\{{\nwixident{Layer}}{Layer}}\\{{\nwixident{Row}}{Row}}}\nwindexuse{\nwixident{Image}}{Image}{NW3AzXtx-4aeb4o-1}\nwindexuse{\nwixident{Layer}}{Layer}{NW3AzXtx-4aeb4o-1}\nwindexuse{\nwixident{Row}}{Row}{NW3AzXtx-4aeb4o-1}\nwendcode{}\nwbegindocs{8}\nwdocspar

Parse an encoded black, white, or transparent pixel.

\nwenddocs{}\nwbegincode{9}\sublabel{NW3AzXtx-1aCFXy-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-1aCFXy-1}}}\moddef{Parse a pixel~{\nwtagstyle{}\subpageref{NW3AzXtx-1aCFXy-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW3AzXtx-19Srvv-1}}\nwenddeflinemarkup
pixel :: Parser \nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1}
pixel =
  (char '0' *> pure \nwlinkedidentc{Black}{NW3AzXtx-2M5oYw-1} <?> "A black pixel")
    <|> (char '1' *> pure \nwlinkedidentc{White}{NW3AzXtx-2M5oYw-1} <?> "A white pixel")
    <|> (char '2' *> pure \nwlinkedidentc{Transparent}{NW3AzXtx-2M5oYw-1} <?> "A transparent pixel")
\nwused{\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{Black}}{Black}}\\{{\nwixident{Pixel}}{Pixel}}\\{{\nwixident{Transparent}}{Transparent}}\\{{\nwixident{White}}{White}}}\nwindexuse{\nwixident{Black}}{Black}{NW3AzXtx-1aCFXy-1}\nwindexuse{\nwixident{Pixel}}{Pixel}{NW3AzXtx-1aCFXy-1}\nwindexuse{\nwixident{Transparent}}{Transparent}{NW3AzXtx-1aCFXy-1}\nwindexuse{\nwixident{White}}{White}{NW3AzXtx-1aCFXy-1}\nwendcode{}\nwbegindocs{10}\nwdocspar

\newthought{Solve} Part One.

\nwenddocs{}\nwbegincode{11}\sublabel{NW3AzXtx-2iOjQS-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-2iOjQS-1}}}\moddef{Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{NW35miTa-2iOjQS-1}{NW3AzXtx-2iOjQS-2}\nwenddeflinemarkup
\nwlinkedidentc{partOne}{NW3AzXtx-2iOjQS-4} :: \nwlinkedidentc{Image}{NW3AzXtx-LSl4Q-1} -> Int
\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{Image}}{Image}}\\{{\nwixident{partOne}}{partOne}}}\nwindexuse{\nwixident{Image}}{Image}{NW3AzXtx-2iOjQS-1}\nwindexuse{\nwixident{partOne}}{partOne}{NW3AzXtx-2iOjQS-1}\nwendcode{}\nwbegindocs{12}\nwdocspar

Return the product of the number of ones ({\Tt{}\nwlinkedidentq{White}{NW3AzXtx-2M5oYw-1}\nwendquote} pixels) and the number of
twos ({\Tt{}\nwlinkedidentq{Transparent}{NW3AzXtx-2M5oYw-1}\nwendquote} pixels) in the \hs{layer} with the fewest {\Tt{}\nwlinkedidentq{Black}{NW3AzXtx-2M5oYw-1}\nwendquote} pixels.

\nwenddocs{}\nwbegincode{13}\sublabel{NW3AzXtx-2iOjQS-2}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-2iOjQS-2}}}\moddef{Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{NW3AzXtx-2iOjQS-1}{NW3AzXtx-2iOjQS-3}\nwenddeflinemarkup
\nwlinkedidentc{partOne}{NW3AzXtx-2iOjQS-4} layers = numberOf \nwlinkedidentc{White}{NW3AzXtx-2M5oYw-1} layer * numberOf \nwlinkedidentc{Transparent}{NW3AzXtx-2M5oYw-1} layer
  where
\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{partOne}}{partOne}}\\{{\nwixident{Transparent}}{Transparent}}\\{{\nwixident{White}}{White}}}\nwindexuse{\nwixident{partOne}}{partOne}{NW3AzXtx-2iOjQS-2}\nwindexuse{\nwixident{Transparent}}{Transparent}{NW3AzXtx-2iOjQS-2}\nwindexuse{\nwixident{White}}{White}{NW3AzXtx-2iOjQS-2}\nwendcode{}\nwbegindocs{14}\nwdocspar

Find the \hs{layer} with the fewest zeros\todoo{sp?}, i.e. {\Tt{}\nwlinkedidentq{Black}{NW3AzXtx-2M5oYw-1}\nwendquote} pixels.

\nwenddocs{}\nwbegincode{15}\sublabel{NW3AzXtx-2iOjQS-3}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-2iOjQS-3}}}\moddef{Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{NW3AzXtx-2iOjQS-2}{NW3AzXtx-2iOjQS-4}\nwenddeflinemarkup
    layer = minimumBy (compare `on` numberOf \nwlinkedidentc{Black}{NW3AzXtx-2M5oYw-1}) layers
\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentuses{\\{{\nwixident{Black}}{Black}}}\nwindexuse{\nwixident{Black}}{Black}{NW3AzXtx-2iOjQS-3}\nwendcode{}\nwbegindocs{16}\nwdocspar

Return the number of elements equivalent to a given one, in a given list of
lists of elements of the same type. More specifically, return the number of
{\Tt{}\nwlinkedidentq{Pixel}{NW3AzXtx-2M5oYw-1}\nwendquote}s of a given color in a given {\Tt{}\nwlinkedidentq{Layer}{NW3AzXtx-LSl4Q-1}\nwendquote}.

\todoo{There's gotta be a Data.List function for this..}

\nwenddocs{}\nwbegincode{17}\sublabel{NW3AzXtx-2iOjQS-4}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-2iOjQS-4}}}\moddef{Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{NW3AzXtx-2iOjQS-3}{\relax}\nwenddeflinemarkup
    numberOf :: Eq a => a -> [[a]] -> Int
    numberOf x = sum . fmap (length . filter (== x))
\nwindexdefn{\nwixident{partOne}}{partOne}{NW3AzXtx-2iOjQS-4}\eatline
\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentdefs{\\{{\nwixident{partOne}}{partOne}}}\nwendcode{}\nwbegindocs{18}\nwdocspar

\newthought{Solve} Part Two.

\nwenddocs{}\nwbegincode{19}\sublabel{NW3AzXtx-4P9qKy-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-4P9qKy-1}}}\moddef{Part Two~{\nwtagstyle{}\subpageref{NW35miTa-4P9qKy-1}}}\plusendmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwprevnextdefs{NW35miTa-4P9qKy-1}{\relax}\nwenddeflinemarkup
\nwlinkedidentc{partTwo}{NW3AzXtx-4P9qKy-1} :: \nwlinkedidentc{Image}{NW3AzXtx-LSl4Q-1} -> String
\nwlinkedidentc{partTwo}{NW3AzXtx-4P9qKy-1} layers =
  unlines . map (concatMap show) $
    foldl decodeLayer (\nwlinkedidentc{transparentLayer}{NW3AzXtx-dIQyV-1} 25 6) layers
  where
    decodeLayer :: \nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1} -> \nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1} -> \nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1}
    decodeLayer = zipWith (zipWith decodePixel)
    decodePixel :: \nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1} -> \nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1} -> \nwlinkedidentc{Pixel}{NW3AzXtx-2M5oYw-1}
    decodePixel \nwlinkedidentc{Transparent}{NW3AzXtx-2M5oYw-1} below = below
    decodePixel above _ = above
\nwindexdefn{\nwixident{partTwo}}{partTwo}{NW3AzXtx-4P9qKy-1}\eatline
\nwused{\\{NW35miTa-15Rjc8-1}\\{NW3AzXtx-19Srvv-1}}\nwidentdefs{\\{{\nwixident{partTwo}}{partTwo}}}\nwidentuses{\\{{\nwixident{Image}}{Image}}\\{{\nwixident{Layer}}{Layer}}\\{{\nwixident{Pixel}}{Pixel}}\\{{\nwixident{Transparent}}{Transparent}}\\{{\nwixident{transparentLayer}}{transparentLayer}}}\nwindexuse{\nwixident{Image}}{Image}{NW3AzXtx-4P9qKy-1}\nwindexuse{\nwixident{Layer}}{Layer}{NW3AzXtx-4P9qKy-1}\nwindexuse{\nwixident{Pixel}}{Pixel}{NW3AzXtx-4P9qKy-1}\nwindexuse{\nwixident{Transparent}}{Transparent}{NW3AzXtx-4P9qKy-1}\nwindexuse{\nwixident{transparentLayer}}{transparentLayer}{NW3AzXtx-4P9qKy-1}\nwendcode{}\nwbegindocs{20}\nwdocspar

\newthought{Define a helper function} to create a transparent layer.

\nwenddocs{}\nwbegincode{21}\sublabel{NW3AzXtx-dIQyV-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-dIQyV-1}}}\moddef{A transparent layer~{\nwtagstyle{}\subpageref{NW3AzXtx-dIQyV-1}}}\endmoddef\nwstartdeflinemarkup\nwusesondefline{\\{NW3AzXtx-19Srvv-1}}\nwenddeflinemarkup
\nwlinkedidentc{transparentLayer}{NW3AzXtx-dIQyV-1} :: Int -> Int -> \nwlinkedidentc{Layer}{NW3AzXtx-LSl4Q-1}
\nwlinkedidentc{transparentLayer}{NW3AzXtx-dIQyV-1} width height = replicate height (replicate width \nwlinkedidentc{Transparent}{NW3AzXtx-2M5oYw-1})
\nwindexdefn{\nwixident{transparentLayer}}{transparentLayer}{NW3AzXtx-dIQyV-1}\eatline
\nwused{\\{NW3AzXtx-19Srvv-1}}\nwidentdefs{\\{{\nwixident{transparentLayer}}{transparentLayer}}}\nwidentuses{\\{{\nwixident{Layer}}{Layer}}\\{{\nwixident{Transparent}}{Transparent}}}\nwindexuse{\nwixident{Layer}}{Layer}{NW3AzXtx-dIQyV-1}\nwindexuse{\nwixident{Transparent}}{Transparent}{NW3AzXtx-dIQyV-1}\nwendcode{}\nwbegindocs{22}\nwdocspar

\todo[inline]{Add some prose here.}

\nwenddocs{}\nwbegincode{23}\sublabel{NW3AzXtx-19Srvv-1}\nwmargintag{{\nwtagstyle{}\subpageref{NW3AzXtx-19Srvv-1}}}\moddef{Day08.hs~{\nwtagstyle{}\subpageref{NW3AzXtx-19Srvv-1}}}\endmoddef\nwstartdeflinemarkup\nwenddeflinemarkup
module AdventOfCode.Year2019.Day08 where

import AdventOfCode.Input (parseInput)
import AdventOfCode.TH (defaultMain, inputFilePath)
import Control.Applicative ((<|>))
import Data.Function (on)
import Data.List (minimumBy)
import Text.Trifecta (Parser, char, count, some, (<?>))

\LA{}Define a Pixel data type~{\nwtagstyle{}\subpageref{NW3AzXtx-2M5oYw-1}}\RA{}

\LA{}Implement \hs{Show} for \code{}Pixel\edoc{}~{\nwtagstyle{}\subpageref{NW3AzXtx-QyGx2-1}}\RA{}

\LA{}Define a few convenient type aliases~{\nwtagstyle{}\subpageref{NW3AzXtx-LSl4Q-1}}\RA{}

\nwlinkedidentc{main}{NW3AzXtx-19Srvv-1} :: IO ()
\nwlinkedidentc{main}{NW3AzXtx-19Srvv-1} = $(defaultMain)

getInput :: IO \nwlinkedidentc{Image}{NW3AzXtx-LSl4Q-1}
getInput = parseInput (\nwlinkedidentc{image}{NW3AzXtx-4aeb4o-1} 25 6) $(inputFilePath)

\LA{}Part One~{\nwtagstyle{}\subpageref{NW35miTa-2iOjQS-1}}\RA{}

\LA{}Part Two~{\nwtagstyle{}\subpageref{NW35miTa-4P9qKy-1}}\RA{}

\LA{}Parse an image~{\nwtagstyle{}\subpageref{NW3AzXtx-4aeb4o-1}}\RA{}

\LA{}Parse a pixel~{\nwtagstyle{}\subpageref{NW3AzXtx-1aCFXy-1}}\RA{}

\LA{}A transparent layer~{\nwtagstyle{}\subpageref{NW3AzXtx-dIQyV-1}}\RA{}
\nwindexdefn{\nwixident{main}}{main}{NW3AzXtx-19Srvv-1}\eatline
\nwnotused{Day08.hs}\nwidentdefs{\\{{\nwixident{main}}{main}}}\nwidentuses{\\{{\nwixident{Image}}{Image}}\\{{\nwixident{image}}{image}}}\nwindexuse{\nwixident{Image}}{Image}{NW3AzXtx-19Srvv-1}\nwindexuse{\nwixident{image}}{image}{NW3AzXtx-19Srvv-1}\nwendcode{}

\nwixlogsorted{c}{{A transparent layer}{NW3AzXtx-dIQyV-1}{\nwixd{NW3AzXtx-dIQyV-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Day01.g}{NW3RAD1b-3gOu99-1}{\nwixd{NW3RAD1b-3gOu99-1}\nwixd{NW3RAD1b-3gOu99-2}\nwixd{NW3RAD1b-3gOu99-3}\nwixd{NW3RAD1b-3gOu99-4}}}%
\nwixlogsorted{c}{{Day04.hs}{NW35miTa-15Rjc8-1}{\nwixd{NW35miTa-15Rjc8-1}}}%
\nwixlogsorted{c}{{Day08.hs}{NW3AzXtx-19Srvv-1}{\nwixd{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Define a few convenient type aliases}{NW3AzXtx-LSl4Q-1}{\nwixd{NW3AzXtx-LSl4Q-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Define a Pixel data type}{NW3AzXtx-2M5oYw-1}{\nwixd{NW3AzXtx-2M5oYw-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Generic solver}{NW35miTa-2ApJRg-1}{\nwixd{NW35miTa-2ApJRg-1}\nwixu{NW35miTa-15Rjc8-1}}}%
\nwixlogsorted{c}{{Implement \hs{Show} for \code{}Pixel\edoc{}}{NW3AzXtx-QyGx2-1}{\nwixd{NW3AzXtx-QyGx2-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Input}{NW35miTa-1GvnV-1}{\nwixd{NW35miTa-1GvnV-1}\nwixu{NW35miTa-15Rjc8-1}}}%
\nwixlogsorted{c}{{Parse a pixel}{NW3AzXtx-1aCFXy-1}{\nwixd{NW3AzXtx-1aCFXy-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Parse an image}{NW3AzXtx-4aeb4o-1}{\nwixd{NW3AzXtx-4aeb4o-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Part One}{NW35miTa-2iOjQS-1}{\nwixd{NW35miTa-2iOjQS-1}\nwixu{NW35miTa-15Rjc8-1}\nwixd{NW3AzXtx-2iOjQS-1}\nwixd{NW3AzXtx-2iOjQS-2}\nwixd{NW3AzXtx-2iOjQS-3}\nwixd{NW3AzXtx-2iOjQS-4}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{c}{{Part Two}{NW35miTa-4P9qKy-1}{\nwixd{NW35miTa-4P9qKy-1}\nwixu{NW35miTa-15Rjc8-1}\nwixd{NW3AzXtx-4P9qKy-1}\nwixu{NW3AzXtx-19Srvv-1}}}%
\nwixlogsorted{i}{{\nwixident{Black}}{Black}}%
\nwixlogsorted{i}{{\nwixident{Image}}{Image}}%
\nwixlogsorted{i}{{\nwixident{image}}{image}}%
\nwixlogsorted{i}{{\nwixident{Layer}}{Layer}}%
\nwixlogsorted{i}{{\nwixident{main}}{main}}%
\nwixlogsorted{i}{{\nwixident{partOne}}{partOne}}%
\nwixlogsorted{i}{{\nwixident{partTwo}}{partTwo}}%
\nwixlogsorted{i}{{\nwixident{Pixel}}{Pixel}}%
\nwixlogsorted{i}{{\nwixident{Row}}{Row}}%
\nwixlogsorted{i}{{\nwixident{Transparent}}{Transparent}}%
\nwixlogsorted{i}{{\nwixident{transparentLayer}}{transparentLayer}}%
\nwixlogsorted{i}{{\nwixident{White}}{White}}%


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}

\title{Marlin Diagram}
\date{July 2020}

\usepackage[x11names]{xcolor}
\usepackage[b4paper,margin=1.2in]{geometry}
\usepackage{tikz}
\usepackage{afterpage}

\newenvironment{rcases}
  {\left.\begin{aligned}}
  {\end{aligned}\right\rbrace}

\begin{document}

\newcommand{\cm}[1]{\ensuremath{\mathsf{cm}_{#1}}}
\newcommand{\vcm}[1]{\ensuremath{\mathsf{vcm}_{#1}}}
\newcommand{\s}{\ensuremath{\hat{s}}}
\newcommand{\w}{\ensuremath{\hat{w}}}
\newcommand{\x}{\ensuremath{\hat{x}}}
\newcommand{\z}{\ensuremath{\hat{z}}}
\newcommand{\za}{\ensuremath{\hat{z}_A}}
\newcommand{\zb}{\ensuremath{\hat{z}_B}}
\newcommand{\zc}{\ensuremath{\hat{z}_C
}}
\newcommand{\zm}{\ensuremath{\hat{z}_M}}

\newcommand{\val}{\ensuremath{\mathsf{val}}}
\newcommand{\row}{\ensuremath{\mathsf{row}}}
\newcommand{\col}{\ensuremath{\mathsf{col}}}
\newcommand{\rowcol}{\ensuremath{\mathsf{rowcol}}}

\newcommand{\hval}{\ensuremath{\widehat{\val}}}
\newcommand{\hrow}{\ensuremath{\widehat{\row}}}
\newcommand{\hcol}{\ensuremath{\widehat{\col}}}
\newcommand{\hrowcol}{\ensuremath{\widehat{\rowcol}}}

\newcommand{\bb}{\ensuremath{\mathsf{b}}}
\newcommand{\denom}{\ensuremath{\mathsf{denom}}}

\newcommand{\sumcheckinner}{\mathsf{inner}}
\newcommand{\sumcheckouter}{\mathsf{outer}}

\newcommand{\Prover}{\mathcal{P}}
\newcommand{\Verifier}{\mathcal{V}}

\newcommand{\F}{\mathbb{F}}

\newcommand{\DomainA}{H}
\newcommand{\DomainB}{K}

\newcommand{\vPoly}[1]{\ensuremath{v_{#1}}}

This diagram (on the following page) shows the interaction of the Marlin prover and verifier. It is similar to the diagrams in the paper (Figure 5 in Section 5 and Figure 7 in Appendix E, in the latest ePrint version), but with two changes: it shows not just the AHP but also the use of the polynomial commitments (the cryptography layer); and it aims to be fully up-to-date with the recent optimizations to the codebase. This diagram, together with the diagrams in the paper, can act as a ``bridge" between the codebase and the theory that the paper describes.

\section{Glossary of notation}
\begin{table*}[htbp]
  \centering
  \begin{tabular}{c|c}
    $\F$ & the finite field over which the R1CS instance is defined \\
     \hline
    $x$ & public input \\
     \hline
    $w$ & secret witness \\
     \hline
    $\DomainA$ & variable domain \\
     \hline
    $\DomainB$ & matrix domain \\
     \hline
    $X$ & domain sized for input (not including witness) \\
     \hline
    $v_D(X)$ & vanishing polynomial over domain $D$ \\
    \hline
    $u_D(X, Y)$ & bivariate derivative of vanishing polynomials over domain $D$\\
     \hline
    $A, B, C$ & R1CS instance matrices \\
    \hline
    $A^*, B^*, C^*$ &
    \begin{tabular}{@{}c@{}}shifted transpose of $A,B,C$ matries given by $M^*_{a,b} := M_{b,a} \cdot u_\DomainA(b,b) \; \forall a,b \in \DomainA$ \\ (optimization from Fractal, explained in Claim 6.7 of that paper) \end{tabular} \\
     \hline
    $\hrow, \hcol$ &
    \begin{tabular}{@{}c@{}} LDEs of (respectively) row positions and column positions of non-zero elements of any \\ linear combination of $A^*$, $B^*$, and $C^*$ (the choice of combination is irrelevant).\end{tabular} \\
    \hline
    ${\hrowcol}$ &
    \begin{tabular}{@{}c@{}} LDE of the element-wise product of $\row$ and $\col$, given separately for efficiency  \\ (namely to allow this product to be part of a \textit{linear} combination) \end{tabular} \\
    \hline
    $\hval_{\{A^*, B^*, C^*\}}$ &
    \begin{tabular}{@{}c@{}} preprocessed polynomials containing LDEs of \\ the values of non-zero elements of any linear combination of $A^*$, $B^*$, and $C^*$. \\ That is, if $\kappa$ is the $k$-th element of $\DomainB$, then $(\sum_M \eta_M \hval_{M^*})(\kappa)$ is the \\$k$-th non-zero entry of $\sum_M \eta_M M^*$, for arbitrary $\eta_{\{A, B, C\}} \in \F$.\end{tabular} \\
     \hline
    $\Prover$ & prover \\
     \hline
    $\Verifier$ & verifier \\
     \hline
    $\Verifier^{p}$ &
    	\begin{tabular}{@{}c@{}} $\Verifier$ with ``oracle" access to polynomial $p$ (via commitments provided \\ by the indexer, later opened as necessary by $\Prover$) \end{tabular}\\
    \hline
    $\bb$ & bound on the number of queries \\
    \hline
    $r_M(X, Y)$ & an intermediate polynomial defined by $r_M(X, Y) = M^*(Y,X)$\\
    \hline
  \end{tabular}
\end{table*}

\afterpage{%
\newgeometry{margin=0.2in}

\section{Diagram}

\centering
\begin{tikzpicture}[scale=0.95, every node/.style={scale=0.95}]

\tikzstyle{lalign} = [minimum width=3cm,align=left,anchor=west]
\tikzstyle{ralign} = [minimum width=3cm,align=right,anchor=east]

\node[lalign] (prover) at (-3,27.3) {%
$\Prover(\F, \DomainA, \DomainB, A, B, C, x, w)$
};

\node[ralign] (verifier) at (16.2,27.3) {%
$\Verifier^{\hrow, \hcol, \hrowcol, \hval_{A^*}, \hval_{B^*}, \hval_{C^*}}(\F, \DomainA, \DomainB, x)$
};

\draw [line width=1.0pt] (-3,27.0) -- (16,27.0);

\node[lalign] (prover1) at (-3,26.1) {%
$z := (x, w), z_A := Az, z_B := Bz$ \\
sample $\w(X) \in \F^{<|w|+\bb}[X]$ and $\za(X), \zb(X) \in \F^{<|\DomainA|+\bb}[X]$ \\
sample mask poly $\s(X) \in \F^{<3|\DomainA|+2\bb-2}[X]$ such that $\sum_{\kappa \in \DomainA}\s(\kappa) = 0$
};

\draw [->] (-2,24.8) -- node[midway,fill=white] {commitments $\cm{\w}, \cm{\za}, \cm{\zb}, \cm{\s}$} (15,24.8);

\node[ralign] (verifier1) at (16,24.0) {%
$\eta_A, \eta_B, \eta_C \gets \F$ \\
$\alpha \gets \F \setminus \DomainA$
};

\draw [->] (15,23.3) -- node[midway,fill=white] {$\eta_A, \eta_B, \eta_C, \alpha \in \F$} (-2,23.3);

\node[lalign] (prover2) at (-3,22.5) {%
compute $t(X) := \sum_M \eta_M r_M(\alpha, X)$
};

\draw (-2.9,22.0) rectangle (15.9,3.8);

\node (sc1label) at (6.5,21.7) {%
\textbf{sumcheck for} $\s(X) + u_H(\alpha, X) \left(\sum_M \eta_M \zm(X)\right) - t(X)\z(X)$ \textbf{ over } $\DomainA$
};

\node[lalign] (prover3) at (-2,20.7) {%
let $\zc(X) := \za(X) \cdot \zb(X)$ \\
find $g_1(X) \in \F^{|\DomainA|-1}[X]$ and $h_1(X)$ such that \\
$s(X)+u_H(\alpha, X)(\sum_M \eta_M \zm(X)) - t(X)\z(X) = h_1(X)\vPoly{\DomainA}(X) + Xg_1(X)$ \hspace{0.3cm} $(*)$
};

\draw [->] (-1,19.5) -- node[midway,fill=white] {commitments $\cm{t}, \cm{g_1}, \cm{h_1}$} (14,19.5);

\node[ralign] (verifier2) at (15.4,19.1) {%
$\beta \gets \F \setminus \DomainA$
};

\draw [->] (14,18.7) -- node[midway,fill=white] {$\beta \in \F$} (-1,18.7);

\draw (-0.85,18.2) rectangle (13.85,7.6);

\node (sc2label) at (6.5,17.6) {%
\textbf{sumcheck for } $\sum\limits_{M \in \{A, B, C\}} \eta_M \frac{\vPoly{\DomainA}(\beta) \vPoly{\DomainA}(\alpha)\hval_{M^*}(X)}{\color{purple}(\beta-\hrow(X))(\alpha-\hcol(X))} $ \textbf{ over } $\DomainB$
};

\node[align=center] (mid1) at (6.5, 15) {%
$\begin{aligned}
\text{let } {\color{purple} \denom(X)} &{}:= (\beta - \hrow(X)) (\alpha - \hcol(X)) \\
                                       &{}= {\color{gray}\alpha\beta} - {\color{gray}\alpha}\hrow(X) - {\color{gray}\beta}\hcol(X) + \hrowcol(X) \text{ (over $\DomainB$)}\\\\
    \text{ let } {\color{orange} a(X)} &{}:= {\color{gray} \vPoly{\DomainA}(\beta) \vPoly{\DomainA}(\alpha)} \sum\limits_{M \in \{A, B, C\}} \eta_M \hval_{M^*}(X)
\\\\
    \text{ let } {\color{Green4} b(X)} &{}:= {\color{purple} \denom(X)}\\\\
\end{aligned}$
};

\node[lalign] (prover4) at (-0.75,12.2) {%
find $g_2(X) \in \F^{|\DomainB|-1}[X]$ and $h_2(X)$ s.t. \\
$h_2(X)\vPoly{\DomainB}(X) = {\color{orange} a(X)} - {\color{Green4} b(X)} (Xg_2(X)+t(\beta)/|\DomainB|)$ \hspace{0.3cm} $(**)$
};

\draw [->] (0,11.2) -- node[midway,fill=white] {commitments $\cm{g_2}, \cm{h_2}$} (13,11.2);

\draw [->] (13,10.5) -- node[midway,fill=white] {$\gamma \in \F$} (0,10.5);

\node[ralign] (verifier3) at (14.5, 10.9) {%
$\gamma \gets \F$
};

\draw[dashed] (1.5,10.0) rectangle (11.5,7.8);

\node[align=center] (mid4) at (6.5, 8.9) {%
To verify $(**)$, $\Verifier$ will need to check the following: \\[10pt]
$ \underbrace{{\color{orange} a({\color{black} \gamma})} - {\color{Green4} b({\color{black} \gamma})} {\color{gray} (\gamma g_2(\gamma) + t(\beta) / |\DomainB|) - \vPoly{\DomainB}(\gamma)} h_2(\gamma)}_{\sumcheckinner(\gamma)} \stackrel{?}{=} 0 $
};

\node[ralign] (verifier3) at (15.4, 6.9) {%
Compute $\x(X) \in \F^{<|x|}[X]$ from input $x$
};

\draw[dashed] (-2.7,7.4) rectangle (15.7,4.2);

\node[align=center] (mid5) at (6.5, 5.3) {%
To verify $(*)$, $\Verifier$ will need to check the following: \\[10pt]
$ \underbrace{s(\beta) + {\color{gray} v_H(\alpha, \beta)} ({\color{gray} \eta_A} \za(\beta) + {\color{gray} \eta_C\zb(\beta)} \za(\beta) + {\color{gray} \eta_B\zb(\beta)}) - {\color{gray} t(\beta) \vPoly{X}(\beta)} \w(\beta) - {\color{gray} t(\beta) \x(\beta)} - {\color{gray} \vPoly{\DomainA}(\beta)} h_1(\beta) - {\color{gray} \beta g_1(\beta)}}_{\sumcheckouter(\beta)} \stackrel{?}{=} 0 $
};

\node[lalign] (prover5) at (-3,2.9) {%
$v_{g_2} := g_2(\gamma)$ \\[3pt]
$v_{g_1} := g_1(\beta), v_{\zb} := \zb(\beta), v_{t} := t(\beta)$
};

\draw [->] (-2,1.9) -- node[midway,fill=white] {$v_{g_2}, v_{g_1}, v_{\zb}, v_{t}$} (15,1.9);

\node[align=center] (mid7) at (6.5,0.8) {%
use index commitments $\hrow, \hcol, \hrowcol, \hval_{\{A^*, B^*, C^*\}}$, commitment $\cm{h_2}$, {\color{gray} and evaluations $g_2(\gamma),t(\beta)$} \\
to construct virtual commitment $\vcm{\sumcheckinner}$
};

\node[align=center] (mid8) at (6.5,-0.5) {%
use commitments $\cm{\s}, \cm{\za}, \cm{\w}, \cm{h_1}$ {\color{gray} and evaluations $\zb(\beta), t(\beta), g_1(\beta)$} \\
to construct virtual commitment $\vcm{\sumcheckouter}$
};

\node[ralign] (verifier4) at (16,-1.4) {%
$\xi_1, \dots, \xi_5 \gets F$
};

\draw [->] (15,-2.1) -- node[midway,fill=white] {$\xi_1, \dots, \xi_5$} (-2,-2.1);

\node[lalign] (prover6) at (-3,-3.6) {%
use $\mathsf{PC}.\mathsf{Prove}$ with randomness $\xi_1, \dots, \xi_5$ to \\
construct a batch opening proof $\pi$ of the following: \\
$(\cm{g_2}, {\color{red} \vcm{\sumcheckinner}})$ at $\gamma$ evaluate to $(v_{g_2}, {\color{red} 0})$ \hspace{0.3cm} ${\color{red} (**)}$ \\
$(\cm{g_1}, \cm{\zb}, \cm{t}, {\color{red} \vcm{\sumcheckouter}})$ at $\beta$ evaluate to $(v_{g_1}, v_{\zb}, v_{t}, {\color{red} 0})$ \hspace{0.3cm} ${\color{red} (*)}$ \\
};

\draw [->] (-2,-4.7) -- node[midway,fill=white] {$\pi$} (15,-4.7);

\node[ralign] (verifier5) at (16,-6.0) {%
verify $\pi$ with $\mathsf{PC}.\mathsf{Verify}$, using randomness $\xi_1, \dots, \xi_5$, \\
evaluations $v_{g_2}, v_{g_1}, v_{\zb}, v_{t}$, and \\
commitments $\cm{g_2},\vcm{\sumcheckinner}, \cm{g_1}, \cm{\zb}, \cm{t}, \vcm{\sumcheckinner}$
};

\end{tikzpicture}

\clearpage
\restoregeometry
}

\end{document}


% This document is part of the emcee3 project.
% Copyright 2015 Dan Foreman-Mackey
%
%  RULES OF THE GAME
%
%  * 80 characters
%  * line breaks at the ends of sentences
%  * eqnarrys ONLY
%

\documentclass[12pt,preprint]{aastex}

\pdfoutput=1

\usepackage{color,hyperref}
\definecolor{linkcolor}{rgb}{0,0,0.5}
\hypersetup{colorlinks=true,linkcolor=linkcolor,citecolor=linkcolor,
            filecolor=linkcolor,urlcolor=linkcolor}
\usepackage{url}
\usepackage{amssymb,amsmath}
\usepackage{subfigure}
\usepackage{booktabs}

\usepackage{natbib}
\bibliographystyle{apj}

% Typography
\newcommand{\project}[1]{\textsl{#1}}
\newcommand{\license}{MIT License}
\newcommand{\paper}{\textsl{Article}}
\newcommand{\foreign}[1]{\emph{#1}}
\newcommand{\etal}{\foreign{et\,al.}}
\newcommand{\etc}{\foreign{etc.}}

\newcommand{\figref}[1]{\ref{fig:#1}}
\newcommand{\Fig}[1]{\figurename~\figref{#1}}
\newcommand{\fig}[1]{\Fig{#1}}
\newcommand{\figlabel}[1]{\label{fig:#1}}
\newcommand{\Tab}[1]{Table~\ref{tab:#1}}
\newcommand{\tab}[1]{\Tab{#1}}
\newcommand{\tablabel}[1]{\label{tab:#1}}
\newcommand{\Eq}[1]{Equation~(\ref{eq:#1})}
\newcommand{\eq}[1]{\Eq{#1}}
\newcommand{\eqalt}[1]{Equation~\ref{eq:#1}}
\newcommand{\eqlabel}[1]{\label{eq:#1}}
\newcommand{\sectionname}{Section}
\newcommand{\Sect}[1]{\sectionname~\ref{sect:#1}}
\newcommand{\sect}[1]{\Sect{#1}}
\newcommand{\sectalt}[1]{\ref{sect:#1}}
\newcommand{\App}[1]{Appendix~\ref{sect:#1}}
\newcommand{\app}[1]{\App{#1}}
\newcommand{\sectlabel}[1]{\label{sect:#1}}

% Algorithms
\usepackage{algorithm}
\usepackage{algorithmicx}
\usepackage[]{algpseudocode}
\newcommand*\Let[2]{\State #1 $\gets$ #2}
\newcommand{\Alg}[1]{Algorithm~\ref{alg:#1}}
\newcommand{\alg}[1]{\Alg{#1}}
\newcommand{\alglabel}[1]{\label{alg:#1}}

% To-do
\newcommand{\todo}[3]{{\color{#2}\emph{#1}: #3}}
\newcommand{\dfmtodo}[1]{\todo{DFM}{red}{#1}}

% Response to referee
\definecolor{mygreen}{rgb}{0, 0.50196, 0}
\newcommand{\response}[1]{#1}
% \newcommand{\response}[1]{{\color{mygreen} {\bf #1}}}

% Notation for this paper.
\newcommand{\T}{{\ensuremath{\mathrm{T}}}}
\newcommand{\bvec}[1]{{\ensuremath{\boldsymbol{#1}}}}
\newcommand{\lnprob}{{\ensuremath{\mathcal{L}}}}
\newcommand{\pos}{{\bvec{q}}}
\newcommand{\mom}{{\bvec{p}}}
\newcommand{\mass}{{\bvec{M}}}
\newcommand{\normal}[2]{{\ensuremath{\mathcal{N}(#1,\,#2)}}}

\begin{document}

\title{%
    Affine-invariant Hamiltonian Monte Carlo
}

\newcommand{\uw}{2}
\newcommand{\sagan}{3}
\author{%
    Daniel~Foreman-Mackey\altaffilmark{1,\uw,\sagan}
}
\altaffiltext{1}         {To whom correspondence should be addressed:
                          \url{danfm@uw.edu}}
\altaffiltext{\uw}       {Astronomy Department, University of Washington,
                          Seattle, WA 98195}
\altaffiltext{\sagan}    {Sagan Fellow}

\begin{abstract}

Hamiltonian Monte Carlo (HMC) sampling is an efficient method for drawing
samples from a probability density when the gradient of the probability with
respect to the parameters can be computed.
We present a simple but effective affine-invariant HMC method that uses an
ensemble of samplers to adaptively update the mass matrix.
We demonstrate the performance of this method on some simple test cases and
compare its computational cost on a real data analysis problem in exoplanet
astronomy.
A well-tested and efficient Python implementation is released alongside this
note.

\end{abstract}

\keywords{%
methods: data analysis
---
methods: statistical
}

\section{Introduction}

% Text. \citep{Foreman-Mackey:2013}
% Adaptive: \citet{Girolami:2011, Wang:2013, Hoffman:2014}
% \section{Hamiltonian Monte Carlo}

Pseudocode for the standard implementation of the Hamiltonian Monte Carlo
(HMC) algorithm \citep{Neal:2011} is shown in \alg{basic-hmc}.
In this implementation, there are $(D^2 + D) / 2 + 2$ tuning parameters, where
$D$ is the dimension of the problem.
Of these parameters, $(D^2 + D) / 2$ are the elements of the positive
definite mass matrix \bvec{M} and the other 2 are the step size $\epsilon$ and
the number of steps $L$.
Most practical applications of HMC fix the mass matrix to a constant (often
set to 1) times the identity and reduce the tuning to only the two parameters
$\epsilon$ and $L$.
Methods have been developed to automatically tune these parameters \citep[for
example][]{Hoffman:2014}.

The major problem with fixing the mass matrix to be diagonal is that the
\emph{units} of the input space can change the performance of the algorithm.
For example, sampling from a Gaussian with different variances in the
different dimensions or covariance between the parameters will be less
efficient than sampling from an isotropic Gaussian.
It has been demonstrated that samplers that satisfy affine invariance can be
very useful for real problems in science where the dynamic range of parameters
can vary by orders of magnitude \citep{Goodman:2010, Foreman-Mackey:2013}.
It turns out that HMC can be simply adapted to an affine-invariant algorithm.

The affine-invariant samplers proposed by \citet{Goodman:2010} sample the
target density by evolving an \emph{ensemble} of parallel MCMC chains (called
``walkers'') where the instantaneous proposal for one walker is conditioned on
the current locations of the other walkers, the \emph{complementary ensemble}.
If the move preserves the conditional distribution of the target walker given
the complementary ensemble, it will also preserve the joint distribution of
the ensemble.
The intuition from these proposals can be incorporated into HMC to derive an
affine-invariant algorithm.

\begin{algorithm}
    \caption{Standard implementation of a single HMC step \alglabel{basic-hmc}}
    \begin{algorithmic}
        \Function{HMCStep}{$\lnprob(\pos),\,\pos_t,\,\mass,\,\epsilon,\,L$}
        \State $\mom_t \sim \normal{\bvec{0}}{\mass}$
            \Comment{sample the initial momentum exactly}
        \Let{\pos}{$\pos_t$}
        \State
        \Let{\mom}{$\mom_t + \frac{\epsilon}{2}\,\nabla\lnprob(\pos)$}
            \Comment{run $L$ steps of leapfrog integration}
        \For{$l \gets 1 \textrm{ to } L$}
            \Let{\pos}{$\pos + \epsilon\,\mass^{-1}\,\mom$}
            \If{$l < L$}
                \Let{\mom}{$\mom + \epsilon\,\nabla\lnprob(\pos)$}
            \EndIf
        \EndFor
        \Let{\mom}{$\mom + \frac{\epsilon}{2}\,\nabla\lnprob(\pos)$}
            \Comment{synchronize the momentum and position}
        \State
        \State{$r \sim \mathcal{U}(0, 1)$}
        \If{$r < \exp\left[\lnprob(\pos) - \frac{1}{2}\mom^T\mass^{-1}\mom
            - \lnprob(\pos_t)+\frac{1}{2}{\mom_t}^T\mass^{-1}\mom_t \right]$}
            \State\Return{$\pos$}   \Comment{accept}
        \Else
            \State\Return{$\pos_t$} \Comment{reject}
        \EndIf
        \EndFunction
    \end{algorithmic}
\end{algorithm}

\begin{algorithm}
    \caption{Affine-invariant HMC \alglabel{ai-hmc}}
    \begin{algorithmic}
\Function{AIHMCStep}{$\lnprob(\pos),\,\{\pos_k\}_{k=1}^K,\,\epsilon,\,L$}
\For{$k \gets 1 \textrm{ to } K$}
    \Let{${\mass_k}^{-1}$}{$\mathrm{Cov}(\pos_{[k]})$}
        \Comment{estimate the empirical mass matrix}
    \State $\mom_k \sim \normal{\bvec{0}}{\mass_k}$
        \Comment{sample the initial momentum exactly}
    \State
    \Let{$\mom^\prime$}{$\mom_k$} \Comment{save the initial coordinates}
    \Let{$\pos^\prime$}{$\pos_k$}
    \State
    \Let{$\mom^\prime$}{$\mom^\prime +
                \frac{\epsilon}{2}\,\nabla\lnprob(\pos^\prime)$}
        \Comment{run $L$ steps of leapfrog integration}
    \For{$l \gets 1 \textrm{ to } L$}
        \Let{$\pos^\prime$}{$\pos^\prime +
                \epsilon\,{\mass_k}^{-1}\,\mom^\prime$}
        \If{$l < L$}
            \Let{$\mom^\prime$}{$\mom^\prime +
                \epsilon\,\nabla\lnprob(\pos^\prime)$}
        \EndIf
    \EndFor
    \Let{$\mom^\prime$}{$\mom^\prime +
                \frac{\epsilon}{2}\,\nabla\lnprob(\pos^\prime)$}
        \Comment{synchronize the momentum and position}
    \State
    \State{$r \sim \mathcal{U}(0, 1)$}
    \If{$r < \exp\left[
        \lnprob(\pos^\prime)
        - \frac{1}{2}{\mom^\prime}^T{\mass_k}^{-1}\mom^\prime
        - \lnprob(\pos_k)
        + \frac{1}{2}{\mom_k}^T{\mass_k}^{-1}\mom_k \right]$}
        \Let{$\pos_k$}{$\pos^\prime$}   \Comment{accept}
    \Else
        \Let{$\pos_k$}{$\pos_k$}   \Comment{reject}
    \EndIf
\EndFor
\State\Return{$\{ \pos_k \}_{k=1}^K$}
\EndFunction
    \end{algorithmic}
\end{algorithm}

\clearpage
\bibliography{emcee-hmc}
\clearpage

\end{document}