%\documentclass{book}
%\begin{document}
\chapter{Converting Coordinates \& Time}
\section{Transformations}

Let $\mathbf{i}_{x}$, $\mathbf{i}_{y}$, $\mathbf{i}_{z}$ and $\mathbf{i}_{\varepsilon}$, $\mathbf{i}_{\eta}$, $\mathbf{i}_{\zeta}$ be two sets of orthogonal unit vectors

\[\mathbf{i}_{\xi}=l_{1}\mathbf{i}_{x}+m_{1}\mathbf{i}_{y}+n_{1}\mathbf{i}_{z}\]
\[\mathbf{i}_{\eta}=l_{2}\mathbf{i}_{x}+m_{2}\mathbf{i}_{y}+n_{2}\mathbf{i}_{z}\]
\[\mathbf{i}_{\zeta}=l_{3}\mathbf{i}_{x}+m_{3}\mathbf{i}_{y}+n_{3}\mathbf{i}_{z}\]

\[ \left[ \begin{array}{c} x \\ y \\ z  \end{array} \right] = \mathbf{R}\left[ \begin{array}{c} \varepsilon \\ \eta \\ \zeta  \end{array} \right] \mbox{or}
\left[ \begin{array}{c} \varepsilon \\ \eta \\ \zeta  \end{array} \right] = \mathbf{R^{T}}\left[ \begin{array}{c} x \\ y \\ z  \end{array} \right] \]

\[\mathbf{R}=\left[ \begin{array}{ccc}
\mathbf{i}_{x}\mathbf{\cdot}\mathbf{i}_{\varepsilon} & \mathbf{i}_{x}\mathbf{\cdot}\mathbf{i}_{\eta} & \mathbf{i}_{x}\mathbf{\cdot}\mathbf{i}_{\zeta} \\
\mathbf{i}_{y}\mathbf{\cdot}\mathbf{i}_{\varepsilon} & \mathbf{i}_{y}\mathbf{\cdot}\mathbf{i}_{\eta} & \mathbf{i}_{y}\mathbf{\cdot}\mathbf{i}_{\zeta} \\
\mathbf{i}_{z}\mathbf{\cdot}\mathbf{i}_{\varepsilon} & \mathbf{i}_{z}\mathbf{\cdot}\mathbf{i}_{\eta} & \mathbf{i}_{z}\mathbf{\cdot}\mathbf{i}_{\zeta}
\end{array} \right] = \left[ \begin{array}{ccc}
l_{1} & l_{2} & l_{3} \\
m_{1} & m_{2} & m_{3} \\
n_{1} & n_{2} & n_{3}
\end{array} \right] \]

\[ \mathbf{R^{T}}=\mathbf{R^{-1}} \]

Equations found  here~\cite[pp. 81-82]{battin:imma}

\section{Time Systems}
\subsection{Solar \& Sidereal Time}
Since the  beginning time has been kept by counting the days. An apparent solar day is the minimum time elapsed between the sun crossing a specified meridian and then recrossing the same meridian. This form of time keeping is problematic because no two apparent solar days are of the same duration due to Earth's rotation around the sun as well as around its axis (the Earth does a little more than one rotation per apparent solar day). Also, Earth's rotational speed is not constant and its axis of rotation is tilted $23.5^{\circ}$ to the orbital plane. These imperfections call for correction, and thus mean solar time was created. A day in mean solar time is defined as one revolution of a hypothetical sun that orbits at the equator, and is more commonly known as Greenwich Mean Time. Another solution is to base our day on the crossing of a star much farther away thus minimizing the effect of the Earth's orbital movement, this method of time keeping is known as sidereal time. A sidereal day is about 4 minutes shorter than a solar day, and is used heavily by astronomers. Sidereal time is not truly stable either so mean sidereal day was introduced, and is known as Greenwich Apparent Sidereal Time. Universal Time (UT) refers to any time scale based on the Earth's rotation. UT0 refers to the mean solar time at the prime meridian as obtained from astronomical observation, and UT1 is UT0 corrected for polar motion. Briefly ephemeris time was introduced to standardize the second, which was defined as 1/31556925.9747 of the year 1900. This was soon replaced by atomic time~\cite[pp. 84-86]{me:gsmp}.
\subsection{Atomic Time}
The second is now defined by an atomic standard that is based on the resonance frequency of the cesium atom. To be precise, the second is defined as ``9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the cesium-133 atom,'' whose duration happens to exactly match the ephemeris second discussed in the previous section. The problem with detaching our time keeping method from the Earth is that as the Earth slows its rotation noon will move closer to midnight (over the duration of thousands of years, of course). Coordinated Universal Time (UTC) was introduced to prevent this. UTC is a compromise between the precision of atomic time and the groundedness of Earth based time keeping, it uses the atomic second but introduces leap seconds (positive or negative) when necessary to keep UTC within .9 seconds of UT1~\cite[pp. 86-87]{me:gsmp}.
\subsection{Time Formats}
We are used to dealing with months, days, years, hours, minutes, and seconds, but such a time format makes for difficult epoch calculations over long periods. To solve this problem Julian Date (JD) was introduced. JD consists of a day count (days since noon UT on January $1^{\mbox{st}}$ 4713 B.C.) and a fraction of the current day. This makes for easy time differencing, but the length of the date can become cumbersome and the fact that a new day starts at noon confusing. To make things even easier Modified Julian Date (MJD) was created whose origin is midnight November $17^{\mbox{th}}$, 1858.
\[ \mbox{MJD}=\mbox{JD}-2400000.5\]

In order to make Julian Date useful we need an easy way to go between calendar dates and JD. \application{timeconvert}\index{timeconvert!theory} does this and more with ease. The equations to convert from calendar date to JD are
\[ \mbox{JD}=\mbox{INT}[365.25y]+\mbox{INT}[30.6001(m+1)]+D+\mbox{UT}/24+1720981.5\]
\[ \begin{array}{lll}
y=Y-1 & \mbox{and}~m=M+12 & \mbox{if}~M \leq2 \\
y=Y & \mbox{and}~m=M & \mbox{if}~M > 2
\end{array} \]
where $M$ is the month, $D$ is the day, $Y$ is the year, and INT[$x$] returns just the integer part of the number. To go from JD to calendar date
\[ a=\mbox{INT[JD}+0.5] \]
\[ b=a+1537 \]
\[ c=\mbox{INT}[(b-122.1)/365.25] \]
\[ d=\mbox{INT}[365.25c] \]
\[ e=\mbox{INT}[(b-d)/30.6001] \]
\[ D=b-d-\mbox{INT}[30.6001e]+\mbox{FRAC[JD}+0.5] \]
\[ M=e-1-12\mbox{INT}[e/14] \]
\[ Y=c-4715-\mbox{INT}[(7+M)/10] \]
where FRAC[$x$] returns just the fractional part of a real number. MJD Conversion found here~\cite[p. 88]{me:gsmp}. All other date conversions were found here~\cite[pp. 36-37]{hlc:gtp}
\subsection{GPS Time}
GPS Time (GPST) is a continuously running composite time kept by cesium and rubidium frequency standards aboard the satellites and at monitor stations. While there are no leap seconds in GPST as there are in UTC, it is steered to stay within 1 $\mu$s of UTC, that is the difference between GPST and UTC is an integer number of seconds plus a fraction of a $\mu$s. GPST is formatted in terms of GPS weeks and the number of seconds into the current week. Finding these values is done easily if the Julian Date is known.
\[ \mbox{GPS WEEK}=\mbox{INT[(JD}-2444244.5)/7] \]
\[ \mbox{SOW}=\mbox{FRAC[(JD}-2444244.5)/7]\times 604800 \]
where INT[$x$] returns the integer part of a real number, FRAC[$x$] returns the fractional part, and SOW stands for Second of Week.

Other useful quantities such as Day of Week and  Second of Day can be found using \application{timeconvert}\index{timeconvert!theory} or the following equations.
\[\mbox{DOW}=\mbox{modulo\{INT[JD}+0.5],7\}\]
\[\mbox{SOD}=\mbox{modulo\{FRAC[JD}+0.5],7\}\times 86400\]
where DOW=0 corresponds to Monday, DOW=1 corresponds to Tuesday, and so on.

JD and GPS Week equations were found here~\cite[pp. 36-37]{hlc:gtp}, SOD derived from DOW equation.
\subsection{Z-Count}
Satellites keep internal time with Z-count, whose epoch period is 1.5 seconds (a convenient unit for communications timing). The full Z-count is 29 bits, the 10 bit GPS week folloed by a 19 bit Time of Week (TOW) expressed in Z-counts (or 1.5 second units). The truncated Z-count has a 17 bit TOW that is expressed in units of 6 seconds, or the length of one subframe's transmission time. Simply multiply the truncated TOW by 4 to get the full TOW~\cite[pp. 86-88]{tsui:fgpsr}.
\[ \mbox{TOW}=\mbox{FRAC[(JD}-2444244.5)/7]\times 403200 \]
\[ \mbox{Truncated TOW}=\mbox{FRAC[(JD}-2444244.5)/7]\times 100800 \]
Equations derived from SOW equation above
\section{Earth Fixed Coordinates}
\subsection{ECI to ECF}
\[\left[\begin{array}{c} x \\ y \\ z \end{array}\right]_{ECF}=T_{XYZ}^{xyz}\left[\begin{array}{c} X \\ Y \\ Z \\ \end{array}\right]_{ECI}\]
\[T_{XYZ}^{xyz}=WSNP\]

P - applies precession, from epoch 2000.0 to the current time;
N - applies nutation, from epoch 2000.0 to the current time;
S - applies rotation to account for true sidereal time;
W - applies polar motion;

Equations found on page 85 of Fundamentals of Orbit Determination paper book.

\subsection{WGS-84}
The World Geodetic System 1984 (WGS-84) is a fixed physical model of Earth produced by the Department of Defense to which many different reference frames can be attached. WGS-84 consists of two parts, a model of Earth's gravitational field, and an ellipsoid describing the Earth's general shape. When dealing with locations on the Earth's surface the ellipsoid provides the foundation for the geodetic coordinate system used by GPS. The ellipsoid's cross-sections parallel to the equatorial plane are circular while those orthogonal are elliptical. The ellipses are parameterized by an eccentricity $e$, a flattening $f$, and sometimes a second eccentricity $e'$
\[e=\sqrt{1-\frac{b^{2}}{a^{2}}}\]
\[f=1-\frac{b}{a}\]
\[e'=\sqrt{\frac{a^{2}}{b^{2}}-1}=\frac{a}{b}e\]
where $a$, the semimajor axis, is the value of the mean equatorial radius of Earth (6,378.137 km) and $b$, the semiminor axis, is the value of the polar radius of Earth (6,356.7523142 km) \cite[pp. 25-26]{kaplan:ugpspa}.
\subsection{Coordinate Systems}
Now that WGS-84 is defined it is important to understand what coordinate systems can be attached to the ellipsoid and how to move between these different systems. The GPS Toolkit comes with \application{poscvt}\index{poscvt!theory}, an application that gives users the ability to easily convert coordinates in one reference frame to another. The coordinate
systems that \application{poscvt}\index{poscvt!theory} recognizes are Cartesian (or XYZ), geodetic, geocentric, and spherical coordinates. These systems and the formulas to convert between them are discussed below.
\subsubsection{Cartesian (XYZ) Coordinates}
The Earth Centered Earth Fixed (ECEF) Cartesian coordinate system is fixed to the WGS-84 ellipsoid and is the common ground that makes going between the Earth Centered Inertial (ECI) reference frame used by the satellites and the systems we are used to (such as latitude, longitude, and height) manageable. The equatorial plane makes the $xy$-plane with the $+x$-axis pointing toward $0^{\circ}$ longitude and the $+y$-axis pointing toward $90^{\circ}$ E longitude. The $z$-axis is normal to the equatorial plane and points to the geographical north pole. The conversion formulas presented in the next sections will convert to and from this Cartesian reference frame, and so to convert between two non-Cartesian coordinate systems the XYZ system will be used as an intermediary~\cite[p. 24]{kaplan:ugpspa}.
\subsubsection{Geodetic Coordinates}
The geodetic coordinate parameters are longitude $\lambda$, latitude $\phi$, and height $h$. Longitude is defined as the angle between the position and the $x$-axis in the equatorial plane, and is easily computed given a position in Cartesian coordinates. Let a user's position $\mathbf{U}=(x_{u},y_{u},z_{u})$, then
\[ \lambda = \left \{ \begin{array}{ll}
\arctan \left( \frac{y_{u}}{x_{u}} \right), & \mbox{$x_{u} \geq 0$} \\
180^{\circ} + \arctan \left( \frac{y_{u}}{x_{u}} \right), & \mbox{$x_{u} < 0$ and $y_{u} \geq 0$} \\
-180^{\circ} +\arctan \left( \frac{y_{u}}{x_{u}} \right), & \mbox{$x_{u} < 0$ and $y_{u} < 0$}
\end{array}
\right. \]
where negative angles signal west longitude.

Latitude and height are not so straight forward. Latitude is determined by drawing a vector normal to the ellipsoid, beginning somewhere on the equatorial plane and terminating at the users position, we will call this the user vector. The smallest angle between this vector and the equatorial plane is the user's latitude, it is a North latitude for positive angles and South for negative. Notice that unless the user is at a pole or on the equator the vector does not pass through the center of the Earth. The users height is found by taking the magnitude of the vector originating on and normal to the ellipsoid and terminating at the user's position. Latitude $\phi$ and height $h$ are found using the following equations
\[ \phi = \arctan\left(\frac{z_{u}+e'^{2}z_{0}}{r}\right) \]
\[ h = U \left(1-\frac{b^{2}}{aV}\right) \]

where

\[ r = \sqrt{x_{u}^{2}+y_{u}^{2}} \]
\[ E^{2} = a^{2} - b^{2} \]
\[ F = 54 b^{2} z_{u}^{2} \]
\[ G = r^{2} + (1-e^{2}) z_{u}^{2} - e^{2} E^{2} \]
\[ c = \frac{e^{4} F r^{2}}{G^{3}} \]
\[ s = \sqrt[3]{1+c+\sqrt{c^{2} + 2c}}\]
\[ P = \frac{F}{3 \left( s + \frac{1}{s} + 1  \right)^{2}G^{2} } \]
\[ Q = \sqrt{1+2e^{4}P} \]
\[ r_{0} = -\frac{Pe^{2}r}{1+Q}+\sqrt{\frac{1}{2}a^{2} \left(1+\frac{1}{Q}\right)-\frac{P(1-e^{2})z_{u}^{2}}{Q(1+Q)}-\frac{1}{2}Pr^{2}}\]
\[ U = \sqrt{(r-e^{2}r_{0})^{2}+z_{u}^{2}}\]
\[ V = \sqrt{(r-e^{2}r_{0})^{2}+(1-e^{2})z_{u}^{2}} \]
\[ z_{0} = \frac{b^{2}z_{u}}{aV} \]

%\[x=(N_{h}+h)\cos\phi'\cos\lambda\]
%\[y=(N_{h}+h)\cos\phi'\sin\lambda\]
%\[z=(N_{h}+h-\tilde{e}^{2}N_{h})\sin\phi'\]
%\[\tilde{e}^{2}=\tilde{f}(2-\tilde{f})\]
%\[N_{h}=\frac{R_{e}}{(1-\tilde{e}^{2}\sin^{2}\phi')^{1/2}}\]
%\[\tilde{f}=\frac{R_{e}-R_{p}}{R_{e}}\]
%\[x^{2}+y^{2}+\left(\frac{R_{e}}{R_{p}}\right)^{2}z^{2}=R_{e}^{2}\]

Going back to Cartesian coordinates from the geodetic system ($\lambda$ $\phi$ $h$) can be done more compactly
\[ \mathbf{u} = \left[  \begin{array}{c}
\frac{a\cos\lambda}{\sqrt{1+(1-e^2)\tan^{2}\phi}}+h\cos\lambda\cos\phi \\
\frac{a\sin\lambda}{\sqrt{1+(1-e^2)\tan^{2}\phi}}+h\sin\lambda\cos\phi \\
\frac{a(1-e^{2})\sin\phi}{\sqrt{1-e^{2}\sin^{2}\phi}}+h\sin\phi
\end{array}
\right] \]
where $\mathbf{u}$ is the user's position vector~\cite[pp. 26-28, p. 76]{kaplan:ugpspa,me:gsmp}.

\subsubsection{Geocentric Coordinates}
\[x=r\cos\phi\cos\lambda\]
\[y=r\cos\phi\sin\lambda\]
\[z=r\sin\phi\]

where $\lambda$ and $\phi$ are geocentric longitude and latitude

found on page 82 in the Fundamentals of Orbital Determination paper book

%\subsubsection{Spherical Coordinates}

\subsubsection{Topocentric Coordinates}
\[\mathbf{r}_{t}=T_{t}(\mathbf{r}-\mathbf{r}_{s})=T_{t}\rho\]

$\mathbf{r}$ and $\mathbf{r}_{s}$ are the position vectors of the observer and satellite respectively in the Earth-fixed system

\[T_{t}=\left[ \begin{array}{ccc}
-\sin\lambda & \cos\lambda & 0 \\
-\sin\phi\cos\lambda & -\sin\phi\sin\lambda & \cos\phi \\
\cos\phi\cos\lambda & \cos\phi\sin\lambda & \sin\phi \end{array} \right] \]

where $\lambda$ and $\phi$ are geocentric longitude and latitude

found on page 84 in the Fundamentals of Orbital Determination paper book

to find \emph{azimuth} (Az) and \emph{elivation} (El)

\[ \begin{array}{ll}
\sin\mbox{El}=\frac{z_{t}}{r_{t}} & -90^{\circ} \leq \mbox{El} \leq 90^{\circ} \\
\sin\mbox{Az}=\frac{x_{t}}{r_{xy}} & \\
\cos\mbox{Az}=\frac{y_{t}}{r_{xy}} & 0^{\circ} \leq \mbox{Az} \leq 360^{\circ}
\end{array} \]

Equations found on pages 84-85 in  Fundamentals of Orbit Determination paper book

\putbib[gpstk]
%\end{document}


\chapter{The Theory of \\ Automatic Differentiation}

Automatic differentiation is a powerful computation tool for the exact, at least up to
arithmetic precision, evaluation of linear differential operators on the elaborate,
composite functions common to practical applications.  In particular, automatic
differentiation techniques factor a linear differential operator into analytic calculations,
which can be explicitly implemented, from the algebraic manipulations that can be
programmatically automated.

In this chapter we review the theory behind linear differential operators, dual numbers,
and, ultimately, automatic differentiation.  We begin by considering differential operators
in a formal setting, in particular the spaces on which they act, before introducing dual
numbers and showing how they compliment these operators.  Last we integrate both
perspectives into automatic differentiation and its implementation.

\section{Linear Differential Operators}

When considering a one-dimensional function, $f : \mathbb{R} \rightarrow \mathbb{R}$,
a common notion is that the derivative defines a ``best'' linear approximation
to the function in a neighborhood around any point, $x_{0} \in \mathbb{R}$,
%
\begin{equation*}
f \! \left( x \right) - f \! \left( x_{0} \right)
\approx \frac{ \mathrm{d} f }{ \mathrm{d} x } \! \left( x_{0} \right) \left( x - x_{0} \right).
\end{equation*}
%
A more revealing interpretation, however, arises when we consider
$\delta x = x - x_{0}$ and $\delta f = f \! \left( x \right) - f \! \left( x_{0} \right)$ as
perturbations, in which case the derivative is just a linear map that propagates
an input perturbation into an output perturbation,
%
\begin{equation*}
\delta f = \frac{ \mathrm{d} f }{ \mathrm{d} x } \! \left( x_{0} \right) \delta x.
\end{equation*}

Linear differential operators formalize this intuition of propagating perturbations
through functions.  Any formal treatment, however, first requires a much more
careful definition of perturbations, especially for multivariate functions.  In this
section we review the necessary theory of first-order linear differential operators
and then briefly discuss the generalization to higher-order operations.

\subsection{First-Order Linear Differential Operators}

First-order linear differential operators generalize the one-dimensional construction
above.  The most significant complication is that, in general, there are actually two
kinds of first-order perturbations: \textit{tangents} and \textit{sensitivities}.

\subsubsection{Tangents}

A defining property of perturbations is that they can be added together and scaled,
in other words they form a vector space.  Given the real space $\mathbb{R}^{N}$,
the $N$-dimensional vector space of perturbations at every point, $x \in \mathbb{R}^{N}$,
is called the \textit{tangent space} and denoted $T_{x} \mathbb{R}^{N}$.  The notation
comes from interpreting first-order perturbations as tangent vectors of curves
passing through $x$.

Consequently, given a smooth function $f : \mathbb{R}^{N} \rightarrow \mathbb{R}^{M}$
a first-order linear differential operator is a linear map from the tangent space of
the input to the tangent space of the output,
%
\begin{equation*}
D : T_{x} \mathbb{R}^{N} \rightarrow T_{ f ( x ) } \mathbb{R}^{M}.
\end{equation*}
%
Taking the standard bases in both tangent spaces, we can identify any tangent,
$\delta x \in T_{x} \mathbb{R}^{N}$, by its components, $\delta x_{n}$.  In this case
the operator $D$ is equivalent to an $m \times n$ matrix,
%
\begin{equation*}
\delta f_{m} = \sum_{n = 1}^{N} D_{mn} \, \delta x_{n} \in T_{f(x)} \mathbb{R}^{M}.
\end{equation*}

One particularly important first-order linear differential operator is the \textit{pushforward}
or \textit{tangent map} canonically defined for any function,
%
\begin{equation*}
f_{*} : T_{x} \mathbb{R}^{N} \rightarrow T_{ f ( x ) } \mathbb{R}^{M}.
\end{equation*}
%
In components the pushforward is given exactly by the Jacobian matrix of partial derivatives,
%
\begin{equation*}
\left( f_{*} \right)_{mn} = \frac{ \partial f_{m} }{ \partial x_{n} }.
\end{equation*}

Pushforwards behave well under function composition.  Given two functions,
$f : \mathbb{R}^{N} \rightarrow \mathbb{R}^{L}$ and
$g : \mathbb{R}^{L} \rightarrow \mathbb{R}^{M}$, the pushforward of their compositions
is simply
%
\begin{equation*}
\left( g \circ f \right)_{*} = g_{*} \cdot f_{*},
\end{equation*}
%
or, in components,
%
\begin{align*}
\left( \left( g \circ f \right)_{*} \right)_{mn} &=
\sum_{l = 1}^{L} \left( g_{*} \right)_{ml} \left( f_{*} \right)_{ln}
\\
\frac{ \partial g_{m} }{ \partial x_{n} } &=
\sum_{l = 1}^{L} \frac{ \partial g_{m} }{ \partial f_{l} } \cdot \frac{ \partial f_{l} }{ \partial x_{n} },
\end{align*}
%
which is just the chain rule.  Because of this behavior, the pushforward of a
composite function is straightforward computationally, provided that the pushforwards
of each component function are known.  In particular, using the standard basis the
composition pushforward is given simply by a succession of vector-matrix products.

\subsubsection{Sensitivities}

An immediate consequence of this more formal perspective is the introduction of a second
kind of first-order perturbation.  The space of linear operators acting on tangents,
%
\begin{equation*}
\alpha : T_{x} \mathbb{R}^{N} \rightarrow \mathbb{R},
\end{equation*}
%
is itself a vector space, known as the \textit{cotangent space} and denoted
$T^{*}_{x} \mathbb{R}^{N}$.  Composing these maps with a first-order linear differential
operator assigns a value to each tangent,
%
\begin{equation*}
\alpha \circ D : T_{x} \mathbb{R}^{N} \rightarrow \mathbb{R},
\end{equation*}
%
in some sense quantifying how sensitive the underlying function is to a given
perturbation.  Consequently elements of the cotangent space are often known as
\textit{sensitivities} and provide dual perturbation to the tangents.

Any first-order linear differential operator,
$D : T_{x} \mathbb{R}^{N} \rightarrow T_{ f ( x ) } \mathbb{R}^{M}$, acting on
the tangent spaces canonically defines an \textit{adjoint operator} that acts on the
cotangent spaces,
%
\begin{equation*}
D^{*} : T^{*}_{f (x) } \mathbb{R}^{M} \rightarrow T^{*}_{x} \mathbb{R}^{N}.
\end{equation*}
%
The matrix representation of the adjoint operator is, perhaps unsurprisingly,
simply the adjoint of the matrix representation of the original operator,
%
\begin{equation*}
\mathrm{d} x_{n} = \sum_{m = 1}^{M} D_{nm} \mathrm{d} f_{m},
\end{equation*}
%
where $\mathrm{d} x \in T^{*}_{x} \mathbb{R}^{N}$ and
$\mathrm{d} f \in T^{*}_{x} \mathbb{R}^{M}$.

The adjoint of the pushforward is known as the \textit{pullback},
%
\begin{equation*}
f^{*} : T^{*}_{f (x) } \mathbb{R}^{M} \rightarrow T^{*}_{x} \mathbb{R}^{N}.
\end{equation*}
%
Under the composition,
%
\begin{equation*}
\left( g \circ f \right)^{*} = f^{*} \cdot g^{*}
\end{equation*}
%
or, in components,
%
\begin{equation*}
\left( \left( g \circ f \right)^{*} \right)_{mn} =
\sum_{l = 1}^{L} \left( g^{*} \right)_{ml} \left( f^{*} \right)_{ln}.
\end{equation*}
%
Computing the input sensitivities corresponding to the output sensitivities then reduces
to vector-matrix products moving backwards, against the flow of the function evaluation.

\subsection{Higher-Order Linear Differential Operators}

Unfortunately, a theory of higher-order perturbations, and the corresponding
linear differential operators, is substantially more complicated, requiring an elegant but
abstract construction known as \textit{jet spaces} to generalize the tangent space.
For example, a second-order linear differential operator does simply map second-order
tangents to second-order tangents but also requires \textit{two} first-order tangents,
%
 \begin{equation*}
\delta^{2} f_{m}
= \sum_{n = 1}^{N} \frac{ \partial f_{m} }{ \partial x_{n} } \delta^{2} x_{n}
+ \sum_{n = 1}^{N} \sum_{n' = 1}^{N}
\frac{ \partial^{2} f_{m} }{ \partial x_{n} \partial x_{n'} }
\delta x_{n} \delta x_{n'}.
\end{equation*}
%
Moreover, adjoint operators do not immediately generalize -- at higher-orders adjoint
maps require the interaction of tangents and sensitivities.

Fortunately we do not have to consider these higher-order jet spaces directly
because the introduction of \textit{dual numbers} provides an alternative means
of computing how higher-order perturbations propagate across a function.

\section{Dual Numbers}

Dual numbers are a generalization of real numbers that naturally unify the
action of a function, its pushforward, and, with some care, its pullback;
consequently they are the ideal basis for implementing automatic differentiation.
In this section we derive first-order dual numbers and then consider the construction
of higher-order dual numbers and higher-order linear differential operators.

\subsection{First-Order Dual Numbers}

Dual numbers extend the real numbers, $\mathbb{R}$, with the
addition of a new element, $\mathfrak{a}$, which is nilpotent, $\mathfrak{a}^{2} = 0$,
and denoted a \textit{dual unit}.

This extended space of dual numbers forms a two-dimensional associative algebra
over $\mathbb{R}$.  In other words, the addition of the dual unit generates a two-dimensional
space, $\mathbb{D}$, comprised of elements,
%
\begin{equation*}
\xi = x + \mathfrak{a} \, \delta x; \, \xi \in \mathbb{D}, \, x, \delta x \in \mathbb{R},
\end{equation*}
%
that are naturally equipped with associative addition and multiplication operations,
%
\begin{align*}
\xi_{1} + \xi_{2}
&=
 \left( x_{1} + \mathfrak{a} \, \delta x_{1} \right)
 + \left( x_{2} + \mathfrak{a} \,  \delta x_{2} \right)
\\
&=
\left( x_{1} + x_{2} \right)
+ \mathfrak{a} \left( \delta x_{1} + \delta x_{2} \right)
\\
\xi_{1} \cdot \xi_{2}
&=
\left( x_{1} + \mathfrak{a} \, \delta x_{1} \right)
\cdot \left( x_{2} + \mathfrak{a} \, \delta x_{2} \right)
\\
&=
x_{1} x_{2} + \left( x_{1} \delta x_{2}
+ \delta x_{1} x_{2} \right) \mathfrak{a}
+ \delta x_{1} \delta x_{2} \, \mathfrak{a}^{2}
\\
&=
x_{1} x_{2} + \left( x_{1} \delta x_{2} + \delta x_{1} x_{2} \right) \mathfrak{a}.
\end{align*}

Any smooth function $f : \mathbb{R} \rightarrow \mathbb{R}$ induces a function on
dual numbers, $f : \mathbb{D} \rightarrow \mathbb{D}$, via a Taylor series around any
purely real point, $\xi_{0} = x_{0} + \mathfrak{a} \, 0$,
%
\begin{align*}
f \! \left( \xi \right)
&=
\sum_{n = 0}^{\infty} \frac{ \left( \xi - \xi_{0} \right)^{n} }{n!}
\frac{ \partial^{n} f }{ \partial \xi^{n} } \! \left( \xi_{0} \right)
\\
&=
\sum_{n = 0}^{\infty} \frac{ \left( \left( x - x_{0} \right) + \mathfrak{a} \, \delta x \right)^{n} }{n!}
\frac{ \partial^{n} f }{ \partial x^{n} } \! \left( x_{0} \right)
\\
&=
\sum_{n = 0}^{\infty} \frac{ \left( x - x_{0} \right)^{n} }{n!}
\frac{ \partial^{n} f }{ \partial x^{n} } \! \left( x_{0} \right)
+ \mathfrak{a} \, \delta x \sum_{n = 0}^{\infty} \frac{ \left( x - x_{0} \right)^{n - 1} }{\left( n - 1 \right)!}
\frac{ \partial^{n} f }{ \partial x^{n} } \! \left( x_{0} \right)
\\
&=
f \! \left( x \right)
+ \mathfrak{a} \, \delta x \frac{ \partial f }{ \partial x } \! \left( x \right).
\end{align*}

Generalizing to multivariate functions is straightforward: any function
$f : \mathbb{R}^{N} \rightarrow \mathbb{R}^{M}$ is first decomposed into $M$
functions, $f_{m} : \mathbb{R}^{N} \rightarrow \mathbb{R}$, which extend
to dual numbers as
%
\begin{equation} \label{dual_function}
f_{m} \! \left( \mbox{\boldmath{$\xi$}} \right) = f_{m} \! \left( \mathbf{x} \right)
+ \mathfrak{a} \sum_{n = 1}^{N}
\delta x_{n} \frac{ \partial f_{m} }{ \partial x_{n} } \! \left( \mathbf{x} \right).
\end{equation}
%
If we consider the dual component as identifying an element of the tangent
space, then the dual part of the extended function is exactly the pushforward
of the original function.

Similarly, we can interpret the dual component as a sensitivity.  In this case
the action of a function on the dual number is given by the adjoint action of the
pushforward,
%
\begin{equation} \label{dual_function}
\xi_{n}  = x_{n}  + \mathfrak{a} \sum_{m = 1}^{M}
\delta f_{m} \frac{ \partial f_{m} }{ \partial x_{n} } \! \left( \mathbf{x} \right).
\end{equation}

Because dual numbers inherently integrate a function with its pushforward or
pullback, they are the natural type from which we can implement linear
differential operators and, consequently, automatic differentiation.

\subsection{Higher-Order Dual Numbers}

The naturalness of dual numbers also generalizes to higher-order perturbations
with the introduction of multiple dual units, providing an means of computing
higher-order linear differential operators without having to introduce the mathematical
machinery of jet spaces.

At first-order, the real and dual parts of a first-order dual number are real numbers,
%
\begin{equation*}
\zeta = z + \mathfrak{a} \, \delta z, \, z, \delta z \in \mathbb{R}.
\end{equation*}
%
If we introduce a second dual unit, $\mathfrak{b}$, then we can construct a second-order
dual number by replacing the real-valued parts of $\zeta$ with dual numbers,
%
\begin{alignat*}{3}
\zeta
&=
\;\, \xi
&&+ \mathfrak{a} \;\;\, \eta
\\
&=
\left( x + \mathfrak{b} \, \delta x \right)
&&+ \mathfrak{a} \left( \delta y + \mathfrak{b} \, \delta^{2} y \right),
\, x, \delta x, \delta y, \delta^{2} y \in \mathbb{R}.
\end{alignat*}
%
From the linear differential operator perspective, $x$ is a the value, $\delta x$ and
$\delta y$ are first-order perturbations, and $\delta^{2} y$ is a second-order perturbation.
Higher-order dual numbers follow recursively with the introduction of a new dual unit
and the replacement of real-valued components with new dual numbers.  An $n$-th
order dual number, for example, is constructed from $n$ distinct dual units and
$2^{n}$ real components.

When constructing higher-order linear differential operators, these components may
be considered as both higher-order tangents and higher-order sensitivities. Consequently
it's convenient to define notation agnostic to the specific map being considered.
We will refer to the components of each bottem-level dual number as \textit{values} and
\textit{gradients}; for example, at second-order $x$ would be a first-order value, $\delta x$
a first-order gradient, $\delta y$ a second-order value, and $\delta^{2} y$ as the second-order
gradient.

Extending functions to higher-order dual numbers, and constructing higher-order
linear differential operators in the process, also proceeds recursively.  Here we
explicitly evaluate the extension to second and third-order dual numbers.

\subsubsection{Second-Order}

Given the second-order dual number,
%
\begin{alignat*}{3}
\zeta_{i}
&=
\xi_{i}
&&+ \mathfrak{a} \, \eta_{i}
\\
&=
\left( x_{i} + \mathfrak{b} \, \delta x_{i} \right)
&&+ \mathfrak{a} \left( \delta y_{i} + \mathfrak{b} \, \delta^{2} y_{i} \right),
\end{alignat*}
%
a second-order dual number valued-function, evaluates to
%
\begin{align*}
f_{i} \! \left(  \mbox{\boldmath{$\zeta$}} \right)
=& \,
f_{i} \! \left(  \mbox{\boldmath{$\xi$}} \right)
+ \mathfrak{a} \sum_{j} \eta_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left(  \mbox{\boldmath{$\xi$}} \right)
\\
=& \,
f_{i} \! \left( \mathbf{x} + \mathfrak{b} \, \delta \mathbf{x} \right)
+ \mathfrak{a} \sum_{j} \left( \delta y_{j} + \mathfrak{b} \, \delta^{2} y_{j} \right)
\frac{ \partial f_{i} }{ \partial x_{j} }
\! \left( \mathbf{x} + \mathfrak{b} \, \delta \mathbf{x} \right)
\\
=& \,
f_{i} \! \left( \mathbf{x} \right)
+ \mathfrak{b} \sum_{j} \delta x_{j}  \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)
\\
&+
\mathfrak{a} \sum_{j} \left( \delta y_{j} + \mathfrak{b} \, \delta^{2} y_{j} \right)
\left( \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x}\right) +
\mathfrak{b} \, \sum_{k} \delta x_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{x} \right) \right)
\\
=&
\quad\quad\quad
f_{i} \! \left( \mathbf{x} \right)
\quad\quad\quad\;\;\;
+ \mathfrak{b} \;\;\;
\sum_{j} \delta x_{j}  \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)
\\
&+
\mathfrak{a} \left(
\sum_{j} \delta y_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)
+ \mathfrak{b} \left(
\sum_{j} \delta^{2} y_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)
+ \sum_{jk} \delta x_{k} \, \delta y_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{x} \right)
\right)
\right)
\end{align*}
%
These results are summarized in Table \ref{tab:secondOrder}.

\begin{table*}[t!]
	\centering
	\renewcommand{\arraystretch}{2}
	\begin{tabular}{ccc}
	\rowcolor[gray]{0.9} \textbf{Component} & \textbf{Input} & \textbf{Output} \\
	First Value &
	$\mathbf{x}$ &
	$f_{i} \! \left( \mathbf{x} \right)$
	\\
	\rowcolor[gray]{0.9}
	First Gradient &
	$\delta \mathbf{x}$ &
	$\displaystyle \sum_{j} \delta x_{j}  \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)$
	\\
	Second Value &
	$\delta \mathbf{y} $ &
	$\displaystyle \sum_{j} \delta y_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)$
	\\
	\rowcolor[gray]{0.9}
	Second Gradient &
	$\delta^{2} \mathbf{y} $ &
	$\displaystyle \sum_{j} \delta^{2} y_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{x} \right)
	+ \sum_{jk} \delta x_{k} \, \delta y_{j}
	\frac{ \partial f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{x} \right)$
	\\
	\end{tabular}
	\caption{Recursively expanding a function with respect to a second-order dual number
	gives each component of the function's second-order pushforward.
	\label{tab:secondOrder}}
\end{table*}

Considering the first-order gradient, second-order value, and second-order gradient as tangents
immediately defines two first-order linear differential operators and one second-order linear
differential operator.  Unfortunately the same is not true if we try to consider all three components
as sensitivities, as the second term in the second-order gradient map no longer makes
sense.

The generalization of an adjoint map does not consider all components as sensitivities but rather
considers all $k$th-order values as $(k - 1)$-order tangents (with a $0$-order tangent
taken to be an input value) and all $k$th-order gradients as $k$th-order sensitivities.  Once the partial
derivatives are contracted against the tangents they can be transposed to give the
components of the pullback map.  For example, given our usual test function
$f : \mathbb{R}^{N} \rightarrow \mathbb{R}^{M}$ with the perturbations
$\delta y \rightarrow \mathrm{d} f$ and $\delta^{2} y \rightarrow \mathrm{d}^{2} f$, the
second-order pullback is given by
%
\begin{equation*}
\mathrm{d}^{2} x_{n}
= \sum_{m = 1}^{M} \mathrm{d}^{2} f_{m} \frac{ \partial f_{m} }{ \partial x_{n} }
+ \sum_{m = 1}^{M} \mathrm{d} f_{m}
\left[ \sum_{n'=1}^{N} \delta x_{n'} \frac{ \partial^{2} f_{m} }{ \partial x_{n'} \partial x_{n} } \right].
\end{equation*}

\subsubsection{Third-Order}

Continuing to third-order follows in kind.  Given a third-order dual number,
%
\begin{alignat*}{10}
\zeta_{i}
&=
&& \xi_{i} && && &&
&&+ \mathfrak{a} \,
&& \eta_{i} && && &&
\\
&=
( && \sigma_{i} && + \mathfrak{b} && \tau_{i} &&)
&&+ \mathfrak{a} \,
( && \upsilon_{i} && + \mathfrak{b} && \nu_{i} && )
\\
&=
(( && s_{i} + \mathfrak{c} \, \delta s_{i} )
&& + \mathfrak{b} \;
( && \delta t_{i} + \mathfrak{c} \, \delta^{2} t_{i}  ) &&)
&&+ \mathfrak{a} \;
(( && \delta u_{i} + \mathfrak{c} \, \delta^{2} u_{i} )
&& + \mathfrak{b} \;
( && \delta^{2} v_{i} + \mathfrak{c} \, \delta^{3} v_{i} ) &&)
\end{alignat*}
%
we have
%
\begin{align*}
f_{i} \! \left( \mbox{\boldmath{$\zeta$}} \right)
%
=& \,
f_{i} \! \left( \mbox{\boldmath{$\xi$}}\right)
+ \mathfrak{a} \sum_{j} \eta_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mbox{\boldmath{$\xi$}} \right)
\\
%
=& \,
f_{i} \! \left( \mbox{\boldmath{$\sigma$}} + \mathfrak{b} \, \mbox{\boldmath{$\tau$}} \right)
+ \mathfrak{a} \sum_{j} \left( \upsilon_{j} + \mathfrak{b} \, \nu_{j} \right)
\frac{ \partial f_{i} }{ \partial x_{j} }
\! \left( \mbox{\boldmath{$\sigma$}} + \mathfrak{b} \, \mbox{\boldmath{$\tau$}} \right)
\\
%
=& \,
f_{i} \! \left( \mbox{\boldmath{$\sigma$}} \right) + \mathfrak{b}
\sum_{j} \tau_{j} \frac{ \partial f_{i} }{ \partial x_{j} }
\! \left( \mbox{\boldmath{$\sigma$}} \right)
+ \mathfrak{a} \sum_{j} \left( \upsilon_{j} + \mathfrak{b} \, \nu_{j} \right)
\left( \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mbox{\boldmath{$\sigma$}} \right)
+ \mathfrak{b} \sum_{k} \tau_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mbox{\boldmath{$\sigma$}} \right) \right)
\\
%
=& \,
f_{i} \! \left( \mbox{\boldmath{$\sigma$}} \right) + \mathfrak{b}
\sum_{j} \tau_{j} \frac{ \partial f_{i} }{ \partial x_{j} }
\! \left( \mbox{\boldmath{$\sigma$}} \right)
+ \mathfrak{a} \sum_{j}
\upsilon_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mbox{\boldmath{$\sigma$}} \right)
\\
& + \mathfrak{a} \, \mathfrak{b} \sum_{j}
 \nu_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mbox{\boldmath{$\sigma$}} \right)
+ \mathfrak{a} \, \mathfrak{b} \, \sum_{jk} \upsilon_{j} \tau_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mbox{\boldmath{$\sigma$}} \right)
\\
%
=& \,
f_{i} \! \left( \mathbf{s} + \mathfrak{c} \, \delta \mathbf{s} \right) + \mathfrak{b}
\sum_{j} \left( \delta t_{j} + \mathfrak{c} \, \delta^{2} t_{j} \right)
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} + \mathfrak{c} \, \delta \mathbf{s}\right)
\\
&+
\mathfrak{a} \sum_{j}
\left( \delta u_{j} + \mathfrak{c} \, \delta^{2} u_{j} \right)
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} + \mathfrak{c} \, \delta \mathbf{s}\right)
\\
& + \mathfrak{a} \, \mathfrak{b} \sum_{j}
\left( \delta^{2} v_{j} + \mathfrak{c} \, \delta^{3} v_{j} \right)
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} + \mathfrak{c} \, \delta \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \, \sum_{jk}
\left( \delta u_{j} + \mathfrak{c} \, \delta^{2} u_{j} \right)
\left( \delta t_{k} + \mathfrak{c} \, \delta^{2} t_{k} \right)
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} + \mathfrak{c} \, \delta \mathbf{s} \right)
\\
%
=& \,
f_{i} \! \left( \mathbf{s} \right) + \mathfrak{c} \sum_{j} \delta s_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{b}
\sum_{j} \left( \delta t_{j} + \mathfrak{c} \, \delta^{2} t_{j} \right)
\left( \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+  \mathfrak{c} \sum_{k} \delta s_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right) \right)
\\
&+
\mathfrak{a} \sum_{j}
\left( \delta u_{j} + \mathfrak{c} \, \delta^{2} u_{j} \right)
\left( \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{c} \sum_{k} \delta s_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right) \right)
\\
& + \mathfrak{a} \, \mathfrak{b} \sum_{j}
\left( \delta^{2} v_{j} + \mathfrak{c} \, \delta^{3} v_{j} \right)
\left( \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{c} \sum_{k} \delta s_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right) \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \, \sum_{jk}
\left( \delta u_{j} + \mathfrak{c} \, \delta^{2} u_{j} \right)
\left( \delta t_{k} + \mathfrak{c} \, \delta^{2} t_{k} \right)
\\
& \quad\quad\quad\quad \times \left(
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
+ \mathfrak{c} \sum_{l} \delta s_{l}
\frac{ \partial^{3} f_{i} }{ \partial x_{j} \partial x_{k} \partial_{l} } \! \left( \mathbf{s} \right)
\right)
\\
%
\end{align*}

\begin{align*}
f_{i} \! \left( \mbox{\boldmath{$\zeta$}} \right)
%
=& \,
f_{i} \! \left( s_{j} \right) + \mathfrak{c} \sum_{j} \delta s_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{b} \sum_{j} \delta t_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{b} \, \mathfrak{c} \sum_{j} \delta^{2} t_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{b} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta t_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \sum_{j} \delta u_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{c} \sum_{j} \delta^{2} u_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta u_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \sum_{j} \delta^{2} v_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{j} \delta^{3} v_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta^{2} v_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \, \sum_{jk}
\left( \delta u_{j} \delta t_{k}
+ \mathfrak{c} \left(
\delta u_{j} \, \delta^{2} t_{k} + \delta^{2} u_{j} \, \delta t_{k}
\right) \right)
\\
& \quad\quad\quad\quad \times \left(
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
+ \mathfrak{c} \sum_{l} \delta s_{l}
\frac{ \partial^{3} f_{i} }{ \partial x_{j} \partial x_{k} \partial_{l} } \! \left( \mathbf{s} \right)
\right)
\\
%
=& \,
f_{i} \! \left( s_{j} \right) + \mathfrak{c} \sum_{j} \delta s_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{b} \sum_{j} \delta t_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{b} \, \mathfrak{c} \sum_{j} \delta^{2} t_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{b} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta t_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \sum_{j} \delta u_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{c} \sum_{j} \delta^{2} u_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta u_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \sum_{j} \delta^{2} v_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{j} \delta^{3} v_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta^{2} v_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \, \sum_{jk}
\delta u_{j} \delta t_{k} \frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{jk}
\left( \delta u_{j} \, \delta^{2} t_{k} + \delta^{2} u_{j} \, \delta t_{k}  \right)
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{jkl}
\delta u_{j} \delta t_{k} \, \delta s_{l}
\frac{ \partial^{3} f_{i} }{ \partial x_{j} \partial x_{k} \partial_{l} } \! \left( \mathbf{s} \right)
\\
%
\end{align*}

\begin{align*}
f_{i} \! \left( \mbox{\boldmath{$\zeta$}} \right)
%
=& \,
f_{i} \! \left( s_{j} \right) + \mathfrak{c} \sum_{j} \delta s_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{b} \sum_{j} \delta t_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{b} \, \mathfrak{c} \sum_{j} \delta^{2} t_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{b} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta t_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \sum_{j} \delta u_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{c} \sum_{j} \delta^{2} u_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{a} \, \mathfrak{c} \sum_{jk} \delta s_{k} \, \delta u_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b}
\left(
\sum_{j} \delta^{2} v_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\delta u_{j} \delta t_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\right)
\\
&+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{j} \delta^{3} v_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
&+ \mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{jk}
\left(
\delta s_{k} \, \delta^{2} v_{j} + \delta u_{j} \, \delta^{2} t_{k} + \delta^{2} u_{j} \, \delta t_{k}  \right)
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
&+
\mathfrak{a} \, \mathfrak{b} \, \mathfrak{c} \sum_{jkl}
\delta u_{j} \delta t_{k} \, \delta s_{l}
\frac{ \partial^{3} f_{i} }{ \partial x_{j} \partial x_{k} \partial_{l} } \! \left( \mathbf{s} \right)
\\
%
=& \hspace{26mm}
f_{i} \! \left( \mathbf{s} \right)
\hspace{9mm}
+ \mathfrak{c}
\hspace{3mm}
\sum_{j} \delta s_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
& \hspace{8mm} +
\mathfrak{b} \left( \sum_{j} \delta t_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\hspace{1mm}
+ \mathfrak{c} \left(
\sum_{j} \delta^{2} t_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \sum_{jk} \delta s_{k} \, \delta t_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right) \right) \right)
\\
&+
\mathfrak{a} \Bigg(
\hspace{9mm}
\sum_{j} \delta u_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \mathfrak{c} \left( \sum_{j} \delta^{2} u_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
+ \sum_{jk} \delta s_{k} \, \delta u_{j}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right) \right)
\\
&
\hspace{8mm}
+ \mathfrak{b} \Bigg(
\hspace{0.5mm}
\sum_{j} \delta^{2} v_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\delta u_{j} \delta t_{k}
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
& \hspace{34mm}
+ \mathfrak{c} \Bigg(
\sum_{j} \delta^{3} v_{j}
\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
\\
& \hspace{40mm}
+ \sum_{jk}
\left(
\delta s_{k} \, \delta^{2} v_{j} + \delta u_{j} \, \delta^{2} t_{k} + \delta^{2} u_{j} \, \delta t_{k}  \right)
\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)
\\
& \hspace{40mm} +
\sum_{jkl}
\delta u_{j} \delta t_{k} \, \delta s_{l}
\frac{ \partial^{3} f_{i} }{ \partial x_{j} \partial x_{k} \partial_{l} } \! \left( \mathbf{s} \right)
\Bigg)
\Bigg)
\Bigg)
\\
%
\end{align*}
%
These results are summarized in Table \ref{tab:thirdOrder}.

\begin{table*}[t!]
	\centering
	\renewcommand{\arraystretch}{2}
	\begin{tabular}{ccc}
	\rowcolor[gray]{0.9} \textbf{Component} & \textbf{Input} & \textbf{Output} \\
	First Value &
	$\mathbf{s}$ &
	$f_{i} \! \left( \mathbf{s} \right)$
	\\
	\rowcolor[gray]{0.9}
	First Gradient &
	$\delta \mathbf{s} $ &
	$\displaystyle \sum_{j} \delta s_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right) $
	\\
	Second Value &
	$\delta \mathbf{t}$ &
	$\displaystyle \sum_{j} \delta t_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)$
	\\
	\rowcolor[gray]{0.9}
	Second Gradient &
	$\delta^{2} \mathbf{t}$ &
	$\displaystyle \sum_{j} \delta^{2} t_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
	+ \sum_{jk} \delta s_{k} \, \delta t_{j}
	\frac{ \partial f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)$
	\\
	Third Value &
	$\delta \mathbf{u}$ &
	$\displaystyle \sum_{j} \delta u_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)$
	\\
	\rowcolor[gray]{0.9}
	Third Gradient &
	$\delta^{2} \mathbf{u}$ &
	$\displaystyle \sum_{j} \delta^{2} u_{j}
	\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
	+ \sum_{jk} \delta s_{k} \, \delta u_{j}
	\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)$
	\\
	Fourth Value &
	$\delta^{2} \mathbf{v}$ &
	$\displaystyle \sum_{j} \delta^{2} v_{j} \frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right)
	\delta u_{j} \delta t_{k}
	\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right)$
	\\
	\rowcolor[gray]{0.9}
   \multirow{3}{*}{ \vspace{-8mm} Fourth Gradient} &
	\multirow{3}{*}{ \vspace{-8mm} $\delta^{3} \mathbf{v}$} &
	$\displaystyle \sum_{j} \delta^{3} v_{j}
	\frac{ \partial f_{i} }{ \partial x_{j} } \! \left( \mathbf{s} \right) $
	\\
	& &
	$\displaystyle + \sum_{jk} \left(
	\delta s_{k} \, \delta^{2} v_{j} + \delta u_{j} \, \delta^{2} t_{k} + \delta^{2} u_{j} \, \delta t_{k}  \right)
	\frac{ \partial^{2} f_{i} }{ \partial x_{j} \partial x_{k} } \! \left( \mathbf{s} \right) $
	\\
	\rowcolor[gray]{0.9}
	& &
	$ \displaystyle+ \sum_{jkl} \delta u_{j} \delta t_{k} \, \delta s_{l}
	\frac{ \partial^{3} f_{i} }{ \partial x_{j} \partial x_{k} \partial_{l} } \! \left( \mathbf{s} \right) $
	\\
	\end{tabular}
	\caption{Recursively expanding a function with respect to a third-order dual number
	gives each component of the function's third-order pushforward.
	\label{tab:thirdOrder}}
\end{table*}

As with second-order pullbacks, third-order pullbacks require considering each values as
a tangent and each gradient as a sensitivity.  For example, with the notation
%
\begin{align*}
s &\rightarrow x \\
\delta s &\rightarrow \mathrm{d} f \\
\delta t &\rightarrow \delta x \\
\delta^{2} t &\rightarrow \mathrm{d}^{2} f \\
\delta u &\rightarrow \delta x^{2} \\
\delta^{2} u &\rightarrow \mathrm{d}^{3} f \\
\delta^{2} v &\rightarrow \delta x^{3} \\
\delta^{3} v &\rightarrow \mathrm{d}^{4} f
\end{align*}
%
the third-order adjoint map is given by
%
\begin{align*}
\mathrm{d}^{4} x_{n}
&= \quad
\sum_{m = 1}^{M} \mathrm{d}^{4} f_{m} \frac{ \partial f_{m} }{ \partial x_{n} }
\\
& \quad +
\sum_{m = 1}^{M} \mathrm{d} f_{m}
\left[ \sum_{n'=1}^{N} \delta^{3} x_{n'} \frac{ \partial^{2} f_{m} }{ \partial x_{n'} \partial x_{n} } \right]
+
\sum_{m = 1}^{M} \mathrm{d}^{2} f_{m}
\left[ \sum_{n'=1}^{N} \delta^{2} x_{n'} \frac{ \partial^{2} f_{m} }{ \partial x_{n'} \partial x_{n} } \right]
+
\sum_{m = 1}^{M} \mathrm{d}^{3} f_{m}
\left[ \sum_{n'=1}^{N} \delta x_{n'} \frac{ \partial^{2} f_{m} }{ \partial x_{n'} \partial x_{n} } \right]
\\
& \quad +
\sum_{m = 1}^{M} \mathrm{d} f_{m}
\left[ \sum_{n'=1}^{N} \sum_{n''=1}^{N} \delta x_{n'} \delta^{2} x_{n''}
\frac{ \partial^{3} f_{m} }{ \partial x_{n''} \partial x_{n'} \partial x_{n} } \right]
\end{align*}

\section{Automatic Differentiation}

Automatic differentiation is the automated computation of linear differential operators applied
to a user-defined function.  Because dual numbers integrate a function evaluation along with
its pushforward or pullback, they provide a natural type -- ultimately, automatic differentiation
is just the extension of a user-defined, real-valued function to accept dual numbers.

When these functions are composite, as is the case in almost any practical application, they
can be represented by an \textit{expression graph}, with each node carrying the value of
the intermediate function evaluation and each edge denoting a functional dependency
(Figure \ref{fig:exprGraph}).  The dual number-valued functions necessary for automatic
differentiation simply replace the single value at each node with the components of a
$k$th-order dual number.

\begin{figure}
\setlength{\unitlength}{0.1in}
\centering
\begin{picture}(50, 20)
%
%\put(0, 0) { \framebox(50, 20){} }
%\put(25, 0) { \framebox(25, 30){} }
%\put(25, 0) { \framebox(6.25, 30){} }
%\put(25, 0) { \framebox(12.5, 30){} }
%\put(25, 0) { \framebox(18.75, 30){} }
%
%\put(25, 0) { \framebox(3.125, 30){} }
%\put(25, 0) { \framebox(9.375, 30){} }
%\put(25, 0) { \framebox(15.625, 30){} }
%
\put(12.5, 10) { \makebox(0, 0)
{$g \! \left( f_{1} \! \left( x_{1}, x_{2} \right), f_{2} \! \left( x_{2}, x_{3} \right) \right)$} }
%
\put(21.875, 10) { \vector(1, 0){6.25} }
%
\put(31.25, 2.5) { \circle{4} }
\put(31.25, 2.5) { \makebox(0, 0) {$ x_{1} $} }
%
\put(37.5, 2.5) { \circle{4} }
\put(37.5, 2.5) { \makebox(0, 0) { $ x_{2} $ } }
%
\put(43.75, 2.5) { \circle{4} }
\put(43.75, 2.5) { \makebox(0, 0) { $ x_{3} $ } }
%
\put(31.25, 4.5) { \vector(3, 4){2.75} }
\put(37.5, 4.5) { \vector(-3, 4){2.75} }
\put(37.5, 4.5) { \vector(3, 4){2.75} }
\put(43.75, 4.5) { \vector(-3, 4){2.75} }
%
\put(35, 10) {\circle{4} } % Tweaked to the right
\put(34.375, 10) { \makebox(0, 0) { $f_{1}$ } }
%
\put(41.25, 10) {\circle{4} } % Tweaked to the right
\put(40.625, 10) { \makebox(0, 0) { $f_{2}$ } }
%
\put(34.375, 12) { \vector(3, 4){2.75} }
\put(40.625, 12) { \vector(-3, 4){2.75} }
%
\put(38, 17.5) {\circle{4} } % Tweaked to the right
\put(37.5, 17.5) { \makebox(0, 0) { $ g $ } }
%
\end{picture}
\caption{
Composite functions are isomorphic to directed acyclic graphs known as
\textit{expression graphs}; for example, the function
$g \! \left( f_{1} \! \left( x_{1}, x_{2} \right), f_{2} \! \left( x_{2}, x_{3} \right) \right)$
generates a three level graph, with leaves $\left\{ x_{1}, x_{2}, x_{3} \right\}$ and
root $g$.  The expression graph corresponding to a real-valued function
stores single values at each node, and automatic differentiation requires that
each node store all components of a $k$th-order dual number.
}
\label{fig:exprGraph}
\end{figure}

The computation of any linear differential operator comprised entirely of pushforwards is
known as \textit{forward mode} or \textit{forward accumulation} automatic differentiation
because all dual numbers are propagated forward along the expression graph.  Any
adjoint operator, however, requires propagating sensitivities backwards along the expression
graph and consequently the implementation of adjoint operators is known as
\textit{reverse mode} or \textit{reverse accumulation} automatic differentiation.  In this
section we consider both forward and reverse mode automatic differentiation, discuss
their relative performance, and demonstrate a selection of linear differential operator
implementations.

\subsection{Forward Mode Automatic Differentiation}

Forward mode automatic differentiation computes the $k$th-order pushforward
of a given function by first initializing all values and gradients at the function
inputs and propagating the perturbations along the expression graph in a single
forward sweep.

Because only one sweep is necessary for forward mode, the expression graph does
not need to be explicitly stored in memory, reducing overhead and often admitting
more optimized code.  On the other hand, the linear differential operators comprised
entirely of pushforwards are often not of immediate practical interest.

\subsection{Reverse Mode Automatic Differentiation}

Reverse mode automatic differentiation compliments forward mode by computing
$k$th-order adjoint operators which require more elaborate traversals of the
expression graph.  A general reverse mode calculation interleaves the forward
propagation of tangents and the reverse propagation of sensitivities, starting at
first order and working up to the desired order.

Consequently, reverse mode requires the persistent storage of the expression
graph and induces a corresponding performance overhead.  In practice, however,
the adjoint operators admitted by reverse mode automatic differentiation are
more appropriate to applications and worth the additional cost.

\subsection{Relative Performance}

In general neither forward mode nor reverse mode automatic differentiation can be
claimed as optimal, as the relative performance of the two depends not only on
the given function but also on the desired linear differential operation.

Consider, for example, many-to-one functions $f : \mathbb{R}^{N} \rightarrow \mathbb{R}$,
common to many applications.  Forward mode automatic differentiation, computes the
directional derivative of the function along a given tangent in a single forward sweep.
Recovering the full gradient then requires $N$ forward sweeps, each initialized with
only one of the input tangents non-zero.  On the other hand, reverse mode automatic
differentiation computes the gradient with only one forward and one reverse sweep.

If the directional derivative is sufficient or $N$ is small then forward mode automatic
differentiation is the better performing approach, but when the gradient is desired and
$N$ is large, reverse mode automatic differentiation becomes far more computationally
efficient despite the memory overhead.  Because this latter case is ubiquitous is
practice, \nomad implements reverse mode automatic differentiation.

\subsection{Example Implementations}

Here we review the exact computations needed to implement a selection of
linear differential operators natural to many-to-one functions,
$f : \mathbb{R}^{N} \rightarrow 1$.

The gradient and Hessian are denoted by $\mathbf{g}$ and $\mathbf{H}$, respectively,
while adjuncts refers to lower-order linear differential operators that are generated
as a side effect of the desired operation.

\begin{tcolorbox}[colback=white,colframe=gray90, coltitle=black,boxrule=3pt,
fonttitle=\bfseries,title=Directional Derivative]

	\begin{description}
		\item[Form:] $\displaystyle \mathbf{v}^{T} \mathbf{g} = \sum_{i} v_{i} \frac{ \partial f }{ \partial x_{i} } $
		\item[Algorithm:] \hfill \\
		Initialize first gradient of the inputs, $\delta x_{i} = v_{i}$. \\
		Compute first-order forward sweep. \\
		Return first gradient of output.
		\item[Cost:] $\mathcal{O} \! \left( 1 \right)$
		\item[Adjuncts:] $ f $
	\end{description}

\end{tcolorbox}

\begin{tcolorbox}[colback=white,colframe=gray90, coltitle=black,boxrule=3pt,
fonttitle=\bfseries,title=Gradient]

	\begin{description}
		\item[Form:] $\displaystyle g_{i} = \frac{ \partial f }{ \partial x_{i} } $
		\item[Algorithm:] \hfill \\
		Initialize first gradient of the output, $\mathrm{d} f = 1$. \\
		Compute first-order reverse sweep. \\
		Return first gradient of inputs.
		\item[Cost:] $\mathcal{O} \! \left( 1 \right)$
		\item[Adjuncts:] $ f $
	\end{description}

\end{tcolorbox}

\begin{tcolorbox}[colback=white,colframe=gray90, coltitle=black,boxrule=3pt,
fonttitle=\bfseries,title=Hessian Quadratic Form]

	\begin{description}
		\item[Form:]
		$\displaystyle \mathbf{v}^{T} \mathbf{H} \, \mathbf{u} =
		\sum_{ij} v_{i} u_{j} \frac{ \partial^{2} f }{ \partial x_{i} \partial x_{j} } $
		\item[Algorithm:] \hfill \\
		Initialize first gradient, $\delta x_{i} = v_{i}$, second value, $\delta y_{i} = u_{i}$,
		and second gradient, $\delta^{2} y_{i} = 0$, of the inputs. \\
		Compute second-order forward sweep. \\
		Return second gradient of output.
		\item[Cost:] $\mathcal{O} \! \left( 1 \right)$
		\item[Adjuncts:] $ f, \, \mathbf{v}^{T} \mathbf{g}, \, \mathbf{u}^{T} \mathbf{g}$
	\end{description}

\end{tcolorbox}

\begin{tcolorbox}[colback=white,colframe=gray90, coltitle=black,boxrule=3pt,
fonttitle=\bfseries,title=Hessian-Vector Product]

	\begin{description}
		\item[Form:] $\displaystyle \mathbf{H} \, \mathbf{v} =
		\sum_{j} v_{j} \frac{ \partial^{2} f }{ \partial x_{i} \partial x_{j} } $
		\item[Algorithm:] \hfill \\
		Initialize first gradient of the output, $\mathrm{d} f = 1$. \\
		Compute first-order reverse sweep for first gradients at the inputs. \\
		Initialize second values of the inputs, $\delta y_{i} = v_{i}$. \\
		Propagate second values in a forward sweep. \\
		Initialize second gradient of the output, $\mathrm{d}^{2} f = 0.$ \\
		Compute second-order reverse sweep. \\
		Return second gradient of inputs.
		\item[Cost:] $\mathcal{O} \! \left( 1 \right)$
		\item[Adjuncts:] $ f, \, \mathbf{v}^{T} \mathbf{g}, \, \mathbf{g}$
	\end{description}

\end{tcolorbox}

\begin{tcolorbox}[colback=white,colframe=gray90, coltitle=black,boxrule=3pt,
fonttitle=\bfseries,title=Hessian]

	\begin{description}
		\item[Form:] $\displaystyle H_{ij} = \frac{ \partial^{2} f }{ \partial x_{i} \partial x_{j} } $
		\item[Algorithm:] \hfill \\
		Initialize $\mathrm{d} f = 1$. \\
		Compute first-order reverse sweep for first gradients. \\
		For each $j$ compute the $j$th row of the Hessian as:
		\begin{itemize}
			\setlength{\itemsep}{0cm}
			\setlength{\parskip}{0cm}
			\item[] Initialize second values of the inputs, $\delta y_{i} = \delta^{j}_{i}$.
			\item[] Propagate second values in a forward sweep.
			\item[] Initialize second gradient of the output, $\mathrm{d}^{2} f = 0$.
			\item[] Compute second-order reverse sweep.
			\item[] Return second gradient of inputs
		\end{itemize}
		\item[Cost:] $\mathcal{O} \! \left( n \right)$
		\item[Adjuncts:] $ f, \, \mathbf{g}$
	\end{description}

\end{tcolorbox}

\begin{tcolorbox}[colback=white,colframe=gray90, coltitle=black,boxrule=3pt,
fonttitle=\bfseries,title=Gradient of the Trace of a Matrix Hessian Product]

	\begin{description}
		\item[Form:]
		$\displaystyle \frac{\partial}{\partial x_{i} } \mathrm{Tr} \! \left[ \mathbf{M} \, \mathbf{H} \right]
		= \sum_{jk} M_{jk} \frac{ \partial^{3} f }{ \partial x_{i}  \partial x_{j}  \partial x_{k} } $
		\item[Algorithm:] \hfill \\
		Initialize $\mathrm{d} f = 1$. \\
		Compute first-order reverse sweep for first gradients. \\
		Initialize the trace gradient to zero.
		For each $j$ increment the trace gradient with:
		\begin{itemize}
			\setlength{\itemsep}{0cm}
			\setlength{\parskip}{0cm}
			\item[] Initialize second values of the inputs, $\delta t_{i} = \delta^{j}_{i}$.
			\item[] Propagate second values in a forward sweep.
			\item[] Initialize second gradient of the output, $\mathrm{d}^{2} f = 0$.
			\item[] Compute second-order reverse sweep.
			\item[] Initialize third and fourth values of the inputs, $\delta u_{i} = M_{ji}, \delta^{2} v_{i} = 0$.
			\item[] Propagate third and fourth values in a forward sweep.
			\item[] Initialize third gradient of the output, $\mathrm{d}^{3} f = 0$.
			\item[] Compute third-order reverse sweep.
			\item[] Return the fourth gradient of the inputs.
		\end{itemize}
		\item[Cost:] $\mathcal{O} \! \left( n \right)$
		\item[Adjuncts:] $ f, \, \mathbf{g}, \, \mathbf{H}$
	\end{description}

\end{tcolorbox}


\Lecture{Jayalal Sarma}{Nov 2, 2020}{24}{Algebraic Methods in Combinatorics}{Sampriti Roy}{$\alpha$}{JS}
\section{Introduction}
In this chapter the plan is to switch to algebraic methods in combinatorics and the first structure we are going to explore is the concepts related to group theory. It's essentially going to be presented from the combinatorial side. As, a motivation we can think of the following problems:
\begin{itemize}
\item \emph{How many distinct squares can be there with yellow and blue coloured corners?}
\end{itemize}
Our task is to develop a method to count distinct colourings of such squares out of total $2^4=16$ possible colourings. If we consider the set of all coloured squares to be $\Omega$, there will be some squares that are equivalent to each other under some rotation. As, we are interested in counting distinct colouring, the equivalent squares should not be counted twice.
\paragraph{}
One more question, we can think about is :
\begin{itemize}
\item \emph{How many necklaces can be formed with solid beads and transparent beads? }
\end{itemize}
\paragraph{}
Based on the colouring of square, we can say that there are total $2^4=16$ possible colourings but there are few operations under which two colourings can be viewed as same. Let's get familiar with such operations:
\begin{itemize}
\item $R_0$ : Rotation by $0^{\circ}$
\item $R_{90}$ : Rotation by $90^{\circ}$
\item $R_{180}$ : Rotation by $180^{\circ}$
\item $R_{270}$ : Rotation by $270^{\circ}$
\item $H$ : Horizontal flip
\item $V$ : Vertical flip
\item $D$ : Diagonal flip (bottom left top right)
\item $D'$ : Diagonal flip (bottom right top left)
\end{itemize}
Now, let's consider all the above set of operations together to be :
\begin{align*}
G&=\{R_0,R_{90},R_{180},R_{270},H,V,D,D'\}
\end{align*}
We will see that the elements of set $G$ are inter related with each other. If we apply two operations from $G$, one after another, we will get the result of applying an operation from $G$ itself. Let's verify it:
\begin{itemize}
\item Rotate the square two times horizontally, it will result rotating it by $0^{\circ}$. That is $H.H=R_0$
\item Rotate the square two times diagonally, it will result rotating it by $0^{\circ}$. That is $D.D=R_0$
\item Rotate the square two times vertically, it will result rotating it by $0^{\circ}$. That is $V.V=R_0$
\item Rotate the square two times by $180^{\circ}$, it will result rotating it by $0^{\circ}$. That is $R_{180}.R_{180}=R_0$
\item Rotate the square by $90^{\circ}$ first then rotate it by $180^{\circ}$, it will result rotating by $270^{\circ}$. That is $R_{90}R_{180}=R_{270}$
\item  Rotate the square by $90^{\circ}$ first then rotate it by $270^{\circ}$, it will result rotating by $0^{\circ}$. That is $R_{90}R_{270}=R_{0}$
\item Rotate the square by $90^{\circ}$ first then rotate it vertically, it will result rotating diagonally. That is $R_{90}.V=D $
\end{itemize}
\section{Incremental definition of Group}
Consider a set $G$ with binary operation $\circ$. Now, take any two elements of $G$ and apply the operation, a natural question to ask is whether we will get any element from the set $G$ or not. We will answer this question along with other related questions and finally move on to the definition of Group incrementally.
\begin{itemize}
\item \emph{Groupoid} : Let $a,b\in G$. If $a\circ b\in G$, then $G$ is \emph{Closed} under the operation $\circ$. And $G$ is called \emph{Groupoid}. We can easily verify that our example set $G$ is a \emph{Groupiod}.
\item \emph{Semigroup} : $a,b,c\in G$; if $a\circ(b\circ c)=(a\circ b)\circ c $, then $G$ is \emph{associative} under $\circ$ or, the order of $\circ$ does not matter. If a groupoid is associative, it is called \emph{Semigroup}. We can try out and check that our example set $G$ is associative and so it is a \emph{Semigroup}.
\item \emph{Monoid} : $\forall a\in G $, if there is an $e\in G$ such that $a\circ e=e\circ a\forall $, $e$ is called identity element. A Semigoup with identity is called \emph{Monoid}. In our example set $G$, $R_0$ is the identity element, so it is a \emph{Monoid}.
\item \emph{Group} : $\forall a\in G$ if $ \exists b\in G $ such that $a\circ b=b\circ a=e $; where $e$ is identity element, $b$ is called the \emph{inverse} of $a$. A Monoid with inverse element is called a \emph{Group}. In our example set $G$, $R_{180}$ is it's own inverse, $H$ is it's own inverse and we can try out that every element has it's inverse and so it is a \emph{Group}.
\end{itemize}
Hence, we can conclude that $G=\{R_0,R_{90},R_{180},R_{270},H,V,D,D'\}  $ is not just a set, rather it has more structural properties. $(G,\circ) $ is a \emph{Group}. Now, we are ready to define \emph{Group} formally.
\begin{definition}
$(G,\circ)$ is a group if it satisfies closure, associativity and it has identity and inverse element.
\end{definition}
\textbf{Example1} : Let's consider $\mathbb{Z}_p=\{0,1,...,p-1\} $ where $p$ is a prime number. Define the operation to be $+ \  mod \ p $. Let's verify whether $(\mathbb{Z}_p,+ \ mod \ p) $ is a group or not:
\begin{itemize}
\item \emph{Closure} : $\forall a, b\in \mathbb{Z}_p; a+ \ mod \ p\in \mathbb{Z}_p$. So, $\mathbb{Z}_p$ is closed under the operation $+ \ mod \ p$
\item \emph{Associativity} : As modulo arithmetic is associative, $\mathbb{Z}_p$ is associative under $+ \ mod \ p$
\item \emph{Identity} : $0$ is the identity element for all elements in $\mathbb{Z}_p$.
\item \emph{Inverse} : For any element $a\in \mathbb{Z}_p$, $(p-a)$ will be the inverse of $a$.
\end{itemize}
Hence, $(\mathbb{Z}_p,+ \ mod \ p) $ is a group.
\paragraph{}
Let's check whether $(\mathbb{Z}_p,\times \ mod  \ p) $ is group or not:
\begin{itemize}
\item \emph{Closure} : $\forall a, b\in \mathbb{Z}_p; a\times \ mod \ p\in \mathbb{Z}_p$. So, $\mathbb{Z}_p$ is closed under the operation $\times \ mod \ p$
\item \emph{Associativity} : As modulo arithmetic is associative, $\mathbb{Z}_p$ is associative under $\times \ mod \ p$
\item \emph{Identity} : $1$ is the identity element for all elements in $\mathbb{Z}_p$
\item \emph{Inverse} : The element $0$ does not have any inverse.
\end{itemize}
Hence, $(\mathbb{Z}_p,\times \ mod  \ p) $ is not a group but it is a \emph{Monoid}.
\paragraph{}
Consider the group $\mathbb{Z}_p^* $ under the operation $\times \ mod  \ p $; for example we can verify that $(\mathbb{Z}_p^*,\times \ mod  \ p) $ satisfies closure, associativity, identity and inverse properties. Hence, $(\mathbb{Z}_p^*,\times \ mod  \ p) $ is a group.\\
\textbf{Exercise} : Use \emph{pigeon hole principle} to prove that every element has it's inverse in  $(\mathbb{Z}_p^*,\times \ mod  \ p) $.
\paragraph{}
\textbf{Example2} : Set of bijection from $\{1,2,...,n\} $ to $\{1,2,...,n\}$ forms a group under composition ($\circ $).
\begin{itemize}
\item \emph{Closure} : Composition of two bijections is bijection
\item \emph{Associative} : Using function composition property, if $f,g,h$ are functions, $f\circ(g\circ h)=(f\circ g)\circ h$
\item \emph{Identity} : Identity function will be there
\item \emph{Inverse} : Every element will have inverse
\end{itemize}
Hence, set of bijections $(S_n,\circ) $ forms a group.
\section{Group (abstractly)}
So far, we have seen the definition of a group $G$, under some operation $\circ$. Now, we will move on the definition of \emph{Subgroup}.
\begin{definition}
Let $(G,\circ) $ be a group. $H\subseteq G$ is said to be a subgroup if $H$ forms a group by itself with respect to the same operation $\circ$. A subgroup of $G$ is denoted by $H\leq G$.
\end{definition}
\textbf{Example} : $(\mathbb{Z}_{15},+ \ mod \ 15) $ is a group. Consider $H=\{0,3,6,9,12\}$. $H\subseteq G$ and $H$ satisfies closure, associativity, identity, inverse and so it forms a group under $+ \ mod \ 15 $, so $H\leq G$.
\subsection{Subgroup}
Consider a group $G$ and $H\leq G$. Let's take an element $g$ from $G\setminus H$ and multiply it with $H$, define:
\begin{align}
Hg=\{hg|h\in H\}
\end{align}
\begin{observation}
If $g\in H$; then $Hg\subseteq H$ as $H$ is closed by itself.
\end{observation}
It's not just $Hg\subseteq H$; something more is true. For that, we will have the following claim
\begin{claim}
$|Hg|=|H|$ when $g\in H$
\end{claim}
\begin{proof}
$\forall h_1\neq h_2\in H$, we need to argue that $h_1g\neq h_2g$. Or multiplication by $g$ is actually a bijection.\\
Let's prove it by contradiction and let $\forall h_1\neq h_2\in H$
\begin{align*}
h_1g&=h_2g\\
h_1&=h_2gg^{-1}\\
h_1&=h_2
\end{align*}
Hence, we get a contradiction. So, $h_1g\neq h_2g$
\end{proof}
Now, let's move on to the following claim
\begin{claim}
$\forall g_1,g_2$; if $Hg_1\cap Hg_2\neq \emptyset $; then $Hg_1=Hg_2$. In other words, if $Hg_1$ and $Hg_2$ are overlapping, then they are same.
\end{claim}
\begin{proof}
Let $g_1,g_2\in G$. Suppose $g\in Hg_1\cap Hg_2$
\begin{align*}
\exists h\in H; g&=hg_1\\
\exists h'\in H; g&=h'g_2
\end{align*}
Now, combining the above two,
\begin{align*}
hg_1&=h'g_2\\
g_1&=h^{-1}h'g_2
\end{align*}
Now, we are ready to prove $Hg_1\subseteq Hg_2$\\
Consider, any element in LHS $h''\in H $, substituting $g_1$, we get:
\begin{align*}
h''g_1&=h''h^{-1}h'g_2\\
&=h'''g_2
\end{align*}
Hence, any element in LHS is in RHS. Similarly, we can prove $Hg_2\subseteq Hg_1$. So, $Hg_1=Hg_2$.
\end{proof}
\paragraph{}
So, we can argue that the multiplication of $H$ by other elements will result in translation of $H$ which are kind of tiling of group $G$. It does not mean that every $g_i$ will give different tiles, but if they have common element, they are same. One interesting feature is that all such tiles will have equal size. One natural question is can there be any element that is not in any tiling? Yes, there can be. For example, $\forall g\in G, g\in Hg$ because $H$ contains identity as it is subgroup. So,$g$ is an element of $Hg$ always. So, every element $g$ will be there in some tile for sure. If there are total $k$ tiles, we argue that:
\begin{align*}
|G|&=k.|H|
\end{align*}
\begin{theorem}
Lagrange's Theorem : The size of a subgroup $|H$ must divide the size of a group $|G|$.
\end{theorem}
 For example, A group of size $100$ can not have a subgroup of size $99$. In fact it can only have subgroup of size at most $50$. Any group with prime number of elements, can not have any non-trivial subgroup. So, \emph{Lagrange's Theorem} is the example of algebraic structure implying combinatorial bounds.

\Lecture{Jayalal Sarma}{Nov 4, 2020}{25}{A step towards Polya's Theory }{Sampriti Roy}{$\alpha$}{JS}
\section{A quick recap}
 In last lecture, we started with a problem of counting number of distinct colourings when the corner of a square are coloured with $2$ colours. We defined a set $G=\{R_0,R_{90},R_{180},R_{270},H,V,D,D'\} $ that acts on set of all possible $2$-coloured squares and some of them are equivalent under these operations. We have also talked about the definition of subgroup $(H)$ of a group $(G)$ and defined that for $g\in G\setminus H$, $Hg=\{hg: h\in H \}$. This $Hg$ is called \emph{coset} of $H$ in $G$.
We have also seen that if any two cosets overlap, they have to be same. We have also talked about \emph{Lagrange's Theorem}. In this chapter the plan is to understand \emph{Polya's Theory}. We will complete it only in next lecture but we will do a step towards it. The step is known to be \emph{Burnside's Lemma}.
\section{The abstract problem of counting distinct 2-coloured squares}
 $\Omega$ be the set of all $2$-coloured squares. There $|\Omega|=2^4=16$ possibilities. Our task is to count to number of distinct colourings among $\Omega$. Let's define the equivalence between two coloured squares formally:
 \begin{definition}
 Let $\alpha,\beta\in \Omega$, if $\exists g\in G$ such that the action on $g$ on $\alpha$ returns $\beta$, or $\alpha^g=\beta$; then we say $\alpha$ and $\beta$ are equivalent, $\alpha\sim \beta$.
\end{definition}
The relation $\sim$ between $\alpha$ and $\beta$ satisfies the following properties:
\begin{itemize}
\item \emph{Reflexive} : $\alpha\sim \alpha$
\item \emph{Symmetric} : $\alpha\sim \beta\rightarrow \beta\sim \alpha$; if the action of $g$ makes the transformation from $\alpha$ to $\beta$, then $g^{-1}$ will make the transformation from $\beta$ to $\alpha$. For example, $R_{90}$ transform $\alpha$ to $\beta$, $R_{270}$ will transform $\beta$ to $\alpha$.
\item \emph{Transitive} : If $\alpha\sim \beta$ by the action $g_1$, $\beta\sim \gamma$ by the action $g_2$; then the action of $g_1g_2$ (composition of $g_1,g_2$) will transform $\alpha\sim \gamma$.
\end{itemize}
Hence, $\sim$ is an equivalence relation and it splits the set $\Omega$ into set of equivalence classes.
\subsection{Orbit and Stabilizer}
The action of $g$ on $\Omega$, partitions $\Omega$ into different equivalence classes but their sizes need not be equal. This equivalence classes are called \emph{Orbit}. In our context, among set of all possible $2$-colourings of a square, we don't want to count equivalent squares twice. In other words, the abstract problem that we are interested to study is to count the number of orbits of $\Omega$. First, we are going to study the sizes of the orbits and using them we will have a mechanism to count them. Let's define the following
\begin{definition}
$Orbit_G(\alpha)=$ orbit of $\alpha$ under the action of $G$ on $\Omega$
\begin{align*}
Orbit_G(\alpha)&=\{\beta:\exists g\in G \ s.t \ \alpha^g=\beta \}
\end{align*}
\end{definition}
Now, we are ready to define \emph{Stabilizer} of $\alpha$ on an action of $G$. Let, $g\in G$ is acting on $\Omega$. Let $\alpha\in G$, which are  the elements in $G$ that fixes $\alpha$? i.e, the elements in $G$ that takes $\alpha$ to itself.
\begin{definition}
Stabilizer of an element $\alpha$ is a subset of $G$, which acts on $\alpha$ and takes it back to itself.
\begin{align*}
Stab_G(\alpha)&=\{g\in G: \alpha^g=\alpha \}
\end{align*}
\end{definition}
We will see few examples of stabilizer to make it more clear.
\begin{itemize}
\item \emph{Example 1} : A simple example we can think of in our square setting is that, let $\alpha$ be the square where all the corners are coloured yellow. Now, we can easily conclude that any operation from the group $G$ can fix $\alpha$, i.e, no matter which action we are performing, $\alpha$ will be $\alpha$ itself. So, the stabilizer of $\alpha$ is the entire group $G$.
\item \emph{Example 2} : Let's think of another $\alpha$ where bottom left and top right corners are coloured blue and the other two are coloured yellow. Now, we can verify that the operations that fixes $\alpha$ are : $\{R_0,R_{180},D,D' \} $.
\end{itemize}
An interesting observation is that stabilizers inside $G$ are not only subset of $G$, they are actually subgroup of $G$. Let's prove the argument formally,
\begin{claim}
$$Stab_G(\alpha)\leq G$$
\end{claim}
\begin{proof}
Fix an $\alpha$ and verify the following :
\begin{itemize}
\item \emph{Closure} : If $g_1,g_2\in Stab_G(\alpha)$, then $g_1g_2\in Stab_G(\alpha)$. As, $g_1,g_2$ are fixing $\alpha$, then their composition will also fix $\alpha$.
\item \emph{Associativity} : As the operations performed are subset of $G$, associativity is inherited in $Stab_G(\alpha)$.
\item \emph{Identity} : Identity fixes every element, in particular it fixes $\alpha$. So, identity element is present in $Stab_G(\alpha)$.
\item \emph{Inverse} : If an element $g$ fixes $\alpha$, then $g^{-1}$ will also fix $\alpha$. So, inverse element is always present in $Stab_G(\alpha)$. For example,   if $R_{90}$ fixes a coloured square, then $R_{270}$ will also fix the same square.
\end{itemize}
Hence, $Stab_G(\alpha)\leq G$
\end{proof}
There is a combinatorially useful relation between size of the stabilizer and the size of the of $\alpha$. Let's define the following lemma;
\begin{lemma}
\textbf{Orbit-Stabilizer Lemma} : $$\forall\in \Omega;|Orbit_G(\alpha)|.|Stab_G(\alpha)|=|G|$$
\end{lemma}
We will prove this lemma formally in next lecture, but here we are going to draw a outline of the thought process to prove this lemma.
\paragraph{}
\emph{Proof Idea} : Let $Stab_G(\alpha)$ be denoted by $H$. We have already proved that $Stab_G(\alpha)$ is a subgroup of $G$. Using \emph{Lagrange's theorem} we can say that $$|G|=k.|Stab_G(\alpha)|$$
Where $k$ is the number of cosets $H$ in $G$. So, it is sufficient to prove that number of cosets $H$ in $G$ is exactly $|Orbit_G(\alpha)|$. We will show a bijection that for every element in the orbit of $\alpha$, there is a way to associate a corresponding coset with it. Hence, the number of coset in $Stab_G(\alpha) =|Orbit_G(\alpha)|$. Now, we will quickly define the bijection:
\paragraph{}
Consider any $\beta\in Orbit(\alpha)$; it means $\exists g\in G $ s.t. $\alpha^g=\beta $. Now, coset corresponding to $g$ is $Hg$. We required to show the following:
\begin{itemize}
\item \emph{Well definedness} : There could be many $g\in G$ which makes $\alpha$ to $\beta$. We need to argue that no matter which $g\in G$ we choose, we will end up by getting the coset $Hg$.
\item \emph{Injection} : For $\beta\neq \beta'\in Orbit(\alpha)$; there is $g,g'\in G$ where $\alpha^g=\beta,\alpha^{g'}=\beta'$; we will end up by getting two different cosets, i.e., $Hg\neq Hg' $.
\item \emph{Surjection} : To show that for any coset we have corresponding element in the orbit of $\alpha$.
\end{itemize}
An interesting observation we can make from the above lemma is that if the size of a group is any prime number then either $|Stab_G(\alpha)|$ is same as $|G|$ and $|Orbit_G(\alpha)|=1$ or $|Stab_G(\alpha)|=1$ and $|Orbit_G(\alpha)|$ is same as $|G|$. But, we are not interested in individual sizes of orbits but we want to count the number of orbits. We will use the \emph{Orbit-Stabilizer lemma} to define the following lemma to show that if we have a handle on the size of orbits then we can have a handle on the number on number of orbits as well. First, we will define the following:
\begin{definition}
\textbf{Fix Points} : If $G$ is acting on $\Omega$, fix points of a group element are those elements on $\Omega$ that are fixed by $g$.
\begin{align*}
fix(g)&=\{\alpha\in \Omega: \alpha^g=\alpha \}
\end{align*}
\end{definition}
Now, we are ready to state the following lemma:
\begin{lemma}
\textbf{Burnsides Lemma} :
\begin{align*}
\#of \ orbits \ &=\frac{1}{|G|}\sum_{g\in G}|fix(g)|
\end{align*}
\end{lemma}
\begin{proof}
The proof is a classic proof by double counting method. Let's define the set
\begin{align*}
S&=\{(g,\alpha): g\in G, \alpha\in \Omega; \alpha^g=\alpha \}
\end{align*}
We will estimate $|S|$ in two different ways:
\begin{itemize}
\item Answer $1$ : For each $g$ count number of different $\alpha$ that are fixed by $g$.
\begin{align*}
&\sum_{g\in G} (\#of \ \alpha \ s.t \ \alpha^g=\alpha)\\
&=\sum_{g\in G}|fix(g)|
\end{align*}
\item Answer $2$ : For each $\alpha\in \Omega$, count number of different $g$ that fixes $\alpha$.
\begin{align*}
&\sum_{\alpha\in \Omega} (\#of \ g \ s.t \ \alpha^g=\alpha)\\
&=\sum_{\alpha\in \Omega}|Stab_G(\alpha)|\\
&=\sum_{\alpha\in \Omega}\frac{|G|}{|Orbit_G(\alpha)|}\\
&=|G|.\sum_{\alpha\in \Omega}\frac{1}{|Orbit_G(\alpha)|}\\
\end{align*}
Let us consider the orbits of $\Omega$ are denoted by $\{\Omega_1,\Omega_2,...\}$. Then the above expression becomes,
\begin{align*}
&=|G|.(\sum_{\alpha\in \Omega_1}\frac{1}{|\Omega_1|}+\sum_{\alpha\in \Omega_2}\frac{1}{|\Omega_2|} )\\
&=|G|.(\#of \ orbits)
\end{align*}
\end{itemize}
Equating Answer $1$ and $2$, we get:
\begin{align*}
|G|.(\#of \ orbits)&=\sum_{g\in G}|fix(g)|\\
\#of \ orbits&=\frac{1}{|G|}\sum_{g\in G}|fix(g)|
\end{align*}
\end{proof}
\section{Counting number of distinct coloured square using Bernsides Lemma}
We we apply Bernsides lemma to our example problem to count number of distinct coloured squares in $\Omega$. For that, first we have to compute $|fix(g)|$ for each $g\in G$. Where $G=\{R_0,R_{90},R_{180},R_{270},H,V,D,D'\}  $.
\begin{itemize}
\item $g=R_0$; all the coloured squares in $\Omega$ are fixed by $g$. So, $|fix(R_0)|=|\Omega|=16$
\item $g=R_{90}$; the squares whose all corners are either blue or yellow will be fixed by $g$. So, $|fix(R_{90})|=2 $
\item $g=R_{180}$; total $4$ squares will be fixed  by $g$. So, $|fix(R_{180})|=4 $
\item $g=R_{270}$; the number of squares that will be fixed are same as $R_{90}$. Because, if a square is fixed by $R_{90}$, it will be fixed by it's inverse as well. So, $|fix(R_{270})|=2 $
\item $g=H$; it will fix the squares where the upper half and lower half corresponding to the horizontal plane (line towards the middle of the square) will be coloured with same colour. so, $|fix(H)|=4 $
\item $g=V$; the number of squares will be same as that are fixed by $H$. so, $|fix(V)|=4 $
\item $g=D$; consider one diagonal and corresponding to it the upper half and lower half will be coloured with same colour, there are $2$ ways to do it and for each such colour the diagonal can be coloured in $4$ ways. So, $|fix(D)|=8 $
\item $g=D'$; the number of squares will be same as that are fixed by $D$. So, $|fix(D')|=8$
\end{itemize}
Using Bernsides lemma,
\begin{align*}
\#of \ orbits \ in \ \Omega&=\frac{1}{8}(16+2+4+2+4+4+8+8)\\
&=6
\end{align*}
So, out of the $16$ total colourings $6$ of them are in-equivalent to each other. To simulate the thought process, here is an exercise:
\emph{Exercise} : Consider all $2$-coloured squares and the set of operations that are acting on it are defined by $G'=\{R_0,R_{90},R_{180},R_{270}\}$, i.e. we are not worried about the $H,V,D,D'$ rotations. Under this operations count the number of distinct $2$-coloured squares.
\\

\Lecture{Jayalal Sarma}{Nov 11, 2020}{26}{Another step towards Polya's Theory }{Achyuth Prakash}{$\alpha$}{JS}
\section{A quick recap}
 In last lecture, we defined and proved the orbit-stabilizer lemma and Lagrange's theorem. In this chapter the plan is to better understand \emph{Polya's Theory} by going through a further example.

\section{Example 2: Coloring faces of a cube}
Let us consider coloring the faces of a cube with $2$ colors. As before we define, $\Omega$ as the set of all possible colorings. There are $6$ faces, and each can can be colored with $2$ colors, thus we have $|\Omega| = 2^6 = 64$.
\\
Next we define the set of 'operations', $G$ which can act on elements on $\Omega$. What are the different possible operations? We list them and simultaneously compute $|fix(g)|$:
\\
Here is the labelled cube for reference:
\\
\includegraphics[]{./images/cube.png}
\\
The different operations are:
\begin{itemize}
\item $R_0$ : Rotation by $0^{\circ}$, i.e. identity. Since all elements of $\Omega$ when acted upon by $R_0$ give back the same element, we get $|fix(g)| = 64$.

\item $R_1$ : Rotation by $180^{\circ}$ w.r.t axis through the center of the opposite faces. (Observe that there are $3$ pairs of such faces). Now consider $2$ such faces, $A, D$. If we rotate the cube as per $R_1$, how many elements of $\Omega$ remain unchanged?
\\
Consider the faces $A, D$. They can be colored with any color each. Thus, there are $2*2 = 4$ choices. Now, for the other faces, consider $C, F$. These $2$ must be colored with the the same color(because they exchange places) Thus there are $2$ choices for $C, F$ together. Similarly for $B, E$. Thus in total, there are $2^4 = 16$ elements which belong to $fix(g)$.

\item $R_2$ : Rotation by $90^{\circ}$ w.r.t axis through the center of the opposite faces. Observe that there are $3$ pairs of such faces. Again let us compute $|fix(g)|$.
\\
Consider the faces $A, D$. They can be colored with any color each. Thus, there are $2*2 = 4$ choices. Now, for the other faces, they all must be colored with the same color, because, $C \rightarrow B$, $B \rightarrow F$, $F \rightarrow E$, and $E \rightarrow C$. Thus, there are $2$ choices for $B, C, E, F$ together. Hence, $|fix(g)| = 2^3 = 8$.

\item $R_3$ : Rotation by $270^{\circ}$ w.r.t axis through the center of the opposite faces. Observe that there are $3$ pairs of such faces. Let us compute $|fix(g)|$.
\\
Since this is the exact opposite of the previous case, by symmetry, $|fix(g)| = 8$.

\item $R_4$ : Rotation by $180^{\circ}$ w.r.t axis through the midpoints of opposite edges. Observe that there are $6$ pairs of such edges, i.e. $(de, bg),\ (af, ch),\ (ad, hg),\ (cb, ef),\ (ab, eh),\ (cd, fg)$. To compute $|fix(g)|$,
\\
Suppose we do the rotation w.r.t the edges $(de, bg)$. then observe that, now $F$ is now the top face, and $A$ occupies the place where $E$ was initially, similarly, $B$ and $D$ swap places. Thus there are $2$ choices together for $C, F$, $2$ choices for $A, E$ and $2$ for $B, D$, giving $8$ choices overall.
Thus, $|fix(g)| = 8$.

\item $D$ : Diagonal flip by $120^{\circ}$ w.r.t axis through the centers of opposite corners. First, note that there are $4$ pairs of opposite corners. To compute $|fix(g)|$,
\\
Consider the vertex $b$. If we flip the cube along $b$ by $120^{\circ}$, then the face $B \rightarrow C$, i.e. $B$ now occupies the initial position of $C$, $C \rightarrow D$ and $D \rightarrow B$. Thus together, for $B, C, D$ there are $2$ choices. Similarly for $A, E, F$. Thus totally we get $2*2 = 4$ possibilities, and $|fix(g)| = 4$.

\item $D'$ : Diagonal flip by $240^{\circ}$ w.r.t axis through the centers of opposite corners. First, note that there are $4$ pairs of opposite corners. To compute $|fix(g)|$,
\\
By symmetry, this is the opposite/complementary operation of the previous one, i.e. flip by $240^{\circ}$ = flip by $120^{\circ}$ in the other direction. Thus, $|fix(g)| = 4$.
\end{itemize}

Thus totally there are $1 + 3*3 + 6 + 4*2 = 24$ operations in $G$.

Now, if we apply the formula to calculate the number of orbits, we get:
$$\#of \ orbits \ in \ \Omega =\frac{1}{24}(64 + 3*16 + 3*8 + 3*8 + 6*8 + 4*4 + 4*4) = 10$$

Therefore, we observe that out of the $64$ colorings, $10$ of them are in-equivalent.

\section{Cycle Index}
We now introduce and define the notion of cycle indices. Let $G$ be the set of some permutations on $\Omega$. Let us decompose $g \in G$ into a collection of disjoint cycles.
\\
For example, let $n = 5$. Consider the permutation $3, 1, 2, 5, 4$ we write this as $(1, 2, 3)\ (4, 5)$, i.e. this denotes $1 \rightarrow 2$, (i.e. $1$ goes to position $2$), $2 \rightarrow 3, 3 \rightarrow 1$ and so on.
\\
Similarly, the identity permutation, $1, 2, 3, 4, 5$ would be written as $(1)(2)(3)(4)(5)$.
\begin{definition}
\textbf{Type of permutation $\pi$} : A permuatation $\pi$ is said to be of type $(b_1, b_2, \ldots b_m)$ if $b_i$ represents the number of $i$ length cycles in the cycle representation of $\pi$.
\end{definition}

For example if $\pi = (1, 2, 3)\ (4, 5)$ then it is of type $(0, 1, 1, 0, 0)$, because there are no $1$ length cycles, $\implies b_1 = 0$, one $2$ length cycle, i.e. $4 \rightarrow 5 \rightarrow 4 \implies b_2 = 1$, and one $3$ length cycle.

Now, corresponding to type $(b_1, b_2, \ldots b_m)$ consider the monomial $x_1^{b_1}\ x_2^{b_2} \ldots x_m^{b_m}$. Using these terms we define a polynomial as follows:
\begin{definition}
Cycle index polynomial of $G$:
\end{definition}
$$P(x_1, x_2, \ldots x_m) = \frac{1}{|G|} \sum_{g \in G} x_1^{b_1}\ x_2^{b_2} \ldots x_m^{b_m}$$ where $(b_1, b_2, \ldots b_m)$ is the cycle type of $g$.
\\
We observe that the above polynomial generalises the Burnside's lemma because we can now substitute the number of possible colors say $2$ in place of $x_1, x_2, \ldots x_m$. Thus, this theorem is much more versatile.
\\

\Lecture{Jayalal Sarma}{Nov 12, 2020}{27}{Polya's Theory - Part 1}{Achyuth Prakash}{$\alpha$}{JS}
\section{Quick Recap}
In the last lecture, we went through another example(of coloring the faces of a cube) and defined the cycle index polynomial of $G$.
\\
Consider an example:
Let $G$ be a group such that $G = \{e, (1, 2),\ (3, 4),\ (1,2)(3,4)\} \le S_4$
i.e., $G$ is a subgroup of $S_4$.
\\
Thus, let us write down the cycle index polynomial of $G$:
$$P(x_1, x_2, x_3, x_4) = \frac{1}{4} (x_1^4 + x_1^2x_2 + x_1^2x_2 + x_2^2)$$
We get the RHS because $e$ has $4$ length $1$ cycles, thus we get the $x_1^4$ term, $(1, 2)$ has $2$ length $1$ cycles and $1$ length $2$ cycle, thus the term $x_1^2x_2$ and so on.

\section{Polya's Theorem - version 1}
\begin{theorem}
Polya's theorem: Let $G$ be the group of symmetry acting on $\Omega$, then, if the number of distinct colored patterns with $k$ colors = $N$, we have
$$N = P_G(k, k, k, \ldots k)$$,
or $N = P(x_1, x_2, \ldots x_m)$ with $x_1 = x_2 = \ldots x_m = k$.
\end{theorem}

\begin{proof}
Let the domain size be $= m$. (Here by domain we mean the number of different objects we need to color. For example in the squares example, we had $4$ corners and thus $m = 4$, similarly in the cube example, we had six faces to color, hence $m = 6$.) Then, the value of $|\Omega|$ will be $k^m$.
\\
By Burnside's lemma, WKT,
$$Number\ of\ distinct\ colorings\ = Number\ of\ different\ orbits\ of\ G = \frac{1}{|G|} \sum_{g\in G}|fix(g)|$$
Let us thus compute $|fix(g)|$ for $g \in G$.
\\
An example will make this more clear, before we generalize. Consider the previously discussed example of coloring the vertices of a square with $2$ colors. Then, we can think of each operation $g \in G$ as a permutation. Consider the operation of rotating a square by $90^{\circ}$, we can represent this by the cycle index $(1, 2, 3, 4)$, because $1 \rightarrow 2,\ 2 \rightarrow 3, \ldots$. Similarly, the operation of vertical flip corresponds to $(1, 2)\ (3, 4)$. Consider this operation. Since $1 \rightarrow 2$ and $2 \rightarrow 1$, we observe that for $g$ to fix some element of $\Omega$, the corners $1, 2$ must get the same color and corners $3, 4$ must get the same color. Thus there are $k$ choices for $1, 2$ together and $k$ choices for $3, 4$ and thus, totally $k^2$ colorings which are fixed by $g$.
\\
In general, let the cycle structure of $g = (b_1, b_2, \ldots b_m)$. This means, $g$ has $b_1$ cycles of length $1$, $b_2$ cycles of length $2$, and so on. Now, observe that all the elements that are part of a cycle must receive the same color, if $g$ were to fix the particular coloring. Thus for each cycle of length $i$, there are $k$ choices and as there are $b_i$ cycles of length $i$, there are $k^{b_i}$ choices for all of them. Thus,
$$ the\ number\ of\ colorings\ fixed\ by\ g\ = k^{b_1} * k^{b_2} \ldots k^{b_m}$$
i.e. $|fix(g)| = x_1^{b_1} * x_2^{b_2} \ldots x_m^{b_m}$ with $x_1 = x_2 = \ldots x_m = k$.
Thus,
$$P_G(k, k, k, \ldots k) = \frac{1}{|G|} \sum_{g\in G}(k^{b_1} * k^{b_2} \ldots k^{b_m}) = \frac{1}{|G|} \sum_{g\in G}|fix(g)|$$
Hence proved.
\end{proof}

\section{Applying Polya's Theorem}
We now consider some examples and apply Polya's theorem for a more concrete understanding.
\subsection{Example 1 - Coloring Necklaces with circular beads}
Consider a necklace with $3$ circular beads. We label the beads as $1, 2, 3$. First we ask: what are the possible symmetries? We get $G = \{e, R_{120}, R_{240}, F_{12}, F_{23}, F_{31}\}$, i.e. $e$ is the identity, $R_{120}$ represents a $120^{\circ}$ clockwise rotation through the center, $F_{12}$ represents flipping w.r.t the edge $1-2$, etc. Now let us represent each $g \in G$ as a cycle index permutation. Then we get that $G = \{(1)(2)(3), (1, 2, 3), (1, 3, 2), (1, 2)(3), (1)(2, 3), (2)(1, 3)\}$ respectively. Hence if we apply Polya's theorem, the number of distinct colorings = $\frac{1}{6} (x_1^3 + 3x_1x_2 + 2x_3)$, if we consider coloring with $2$ distinct colors, then $k = 2 = x_1 = x_2 = x_3$, thus the number of distinct colorings with 2 colors = $\frac{1}{6} (2^3 + 3*2^2 + 2*2) = 4$.

\subsection{Example 2 - Coloring faces of a cube}
Let us revisit the cube example. We list the different elements of $G$ as before, and simultaneously calculate the term they contribute to in Polya's formula.
\\
Here is the labelled cube for reference:
\\
\includegraphics[]{./images/cube.png}
\\
\begin{itemize}
\item $R_0$ : Rotation by $0^{\circ}$, i.e. identity. The cycle index of this permutation has $6$ cycles of length $1$, thus the number of colorings contributed will be $\boxed{x_1^6}$.

\item $R_1$ : Rotation by $180^{\circ}$ w.r.t axis through the center of the opposite faces. (Observe that there are $3$ pairs of such faces). Consider the faces $A, D$. We have observed before that $A, D$ remain in place, and $B, E$ and $C, F$ exchange their places, thus there are $2$ length $1$ cycles and $2$ length $2$ cycles, hence the contributing term will be $x_1^2x_2^2$. As there are $3$ such pairs of faces, the total number of fixed colorings will be $\boxed{3x_1^2x_2^2}$.

\item $R_2$ : Rotation by $90^{\circ}$ w.r.t axis through the center of the opposite faces. Observe that there are $3$ pairs of such faces. Consider the faces $A, D$. They retain their positions, i.e $A \rightarrow A$, $D \rightarrow D$. Now, for the other faces, we have, $C \rightarrow B$, $B \rightarrow F$, $F \rightarrow E$, and $E \rightarrow C$. Thus, there are $2$ length $1$ cycles and $1$ length $4$ cycle, hence the contributing term will be $x_1^2x_4$.As there are $3$ such pairs of faces, the total number of fixed colorings will be $\boxed{3x_1^2x_4}$.

\item $R_3$ : Rotation by $270^{\circ}$ w.r.t axis through the center of the opposite faces. Observe that there are $3$ pairs of such faces. Since this is the exact opposite of the previous case, by symmetry, the total number of fixed colorings will be $\boxed{3x_1^2x_4}$.

\item $R_4$ : Rotation by $180^{\circ}$ w.r.t axis through the midpoints of opposite edges. Observe that there are $6$ pairs of such edges, i.e. $(de, bg),\ (af, ch),\ (ad, hg),\ (cb, ef),\ (ab, eh),\ (cd, fg)$. Suppose we do the rotation w.r.t the edges $(de, bg)$. Then observe that, now $F$ is now the top face, and $A$ occupies the place where $E$ was initially, similarly, $B$ and $D$ swap places. In other words, $C \iff F$, $A \iff E$ and $B \iff D$, i.e. there are $3$ length $2$ cycles. Thus, the contributing term is $x_2^3$. As there are $6$ pairs of edges, the total contributing term will be $\boxed{6 x_2^3}$.

\item $D$ : Diagonal flip by $120^{\circ}$ w.r.t axis through the centers of opposite corners. First, note that there are $4$ pairs of opposite corners.
Consider the vertex $b$. If we flip the cube along $b$ by $120^{\circ}$, then the face $B \rightarrow C$, i.e. $B$ now occupies the initial position of $C$, $C \rightarrow D$ and $D \rightarrow B$. Thus there is the cycle $B \rightarrow C \rightarrow D \rightarrow B$. Similarly for the other three faces. Thus there are $2$ cycles of length $3$ and hence the contributing term will be $x_3^2$. As there $4$ such pairs of corners, the total term contributed will be $\boxed{4x_3^2}$.

\item $D'$ : Diagonal flip by $240^{\circ}$ w.r.t axis through the centers of opposite corners. First, note that there are $4$ pairs of opposite corners. By symmetry, this is the opposite/complementary operation of the previous one, i.e. flip by $240^{\circ}$ = flip by $120^{\circ}$ in the other direction. Thus, the total term contributed will be $\boxed{4x_3^2}$.
\end{itemize}

Putting all the above together, we get $$P_{G}(x_1, x_2, \ldots x_4) = \frac{1}{24}(x_1^6 + 3x_1^2x_2^2 + 6x_1^2x_4 + 6x_2^3 + 8x_3^2)$$
if we substitute $x_i = 2$ for all $i$, we get
$$P_{G}(x_1, x_2, \ldots x_4) = \frac{1}{24}(2^6 + 3*2^4 + 6*2^3 + 6*2^3 + 8*2^2) = 10$$
\\

\section{Towards a general formula}
Let us go back to the the example of the square coloring, where $G = \{e, R_{90}, R_{180}, R_{270}, H, V, D, D'\}$, and $G$ as a cycle index permutation is $G = \{(1)(2)(3)(4), (1,2,3,4), (1, 3)(2, 4), (1,4,3,2), (1, 4)(2, 3), (1,2)(3,4), (2,4)(1,3), (1)(3)(2, 4), (2)(4)(1,3)\}$ respectively.
\\
Observe the 2nd, 3rd, and 4th elements of $G$. We know that $R_{90} \circ R_{90} = R_{180}$ i.e. $(1,2,3,4) \circ (1,2,3,4) = (1,3)(2,4)$. Therefore these elements are of the form $g, g^2, g^3$.
\\
Thus, if we have a group whose elements are of the form $G = \{e, g, g^2, \ldots g^k\}$ (called cyclic groups).
Then the contribution of each term of $g \in G$ will be:
\begin{itemize}
    \item $e - x_1^n$
    \item $g - x_n$
    \item $g^2 - x_{n/2}^2$ and so on
\end{itemize}
Thus, in general, $$P_G(x_1, x_2, \ldots x_m) = \frac{1}{n} \sum_{d|n} \phi(d) x_{d}^{n/d}$$ where $\phi(m)$ is euler's totient function.
\\
\begin{proof}
Consider the element $g^{i}$. Suppose it has a cycle of length $d$. Observe that under this operation, $1 \rightarrow 1+i$, $1+i \rightarrow 1+2i$, \ldots, $1+d*i \rightarrow 1$. This must mean $d*i$ is a multiple of $n$.  Specifically, we must have $d*i = lcm(n, i)$. Using $lcm(n,i) = \frac{n*i}{gcd(n, i)}$, we get $d*i = \frac{n*i}{gcd(n, i)} \implies gcd(n,i) = \frac{n}{d} \implies \boxed{gcd(d, \frac{i*d}{n}) = 1}$.
\\
However there is a bijection between elements of $\phi(d)$ and $i$, because $i = x \frac{n}{d}$ for $x \in \phi(d)$. This implies the number of such $i = \phi(d)$. Thus, the above result shows that all $i$ such that $gcd(d, i) = 1$ contribute $n/d$ cycles of length $d$. Therefore the contribution to the term is $\phi(d) x_{d}^{\frac{n}{d}}$. Hence the proof.
\end{proof}

Thus, if we have the above formula, the group for the square coloring problem can be represented as $\{e, \sigma, \sigma^2, \ldots, \sigma^{k}, \pi*\sigma, \pi*\sigma^2, \ldots \pi*\sigma^{k-1}\}$. These are called dihedral groups. We discuss more about this in the next lecture.