\section{Spell Power Training}\label{perk:spellpowertraining}
\textbf{Cost:} 50CP\\
\textbf{Requirements:} Any two ranks in spell perks\\
\textbf{Passive, Repeatable, Source(50 Gold)}\\
You have learned to channel your mental power into your spells.
Whenever a Spell allows you to roll damage, you can add \sfrac{1}{4} of your IN to that damage if you want.\\
\\
Level Progression:\\
\\
\rowcolors{2}{lightgray}{white}
\begin{tabular}{l | l | l | l | l}
    Level & CP Cost & Gold Cost & Requires & Effect\\
    II & 500 & 500 & - & You can add \sfrac{1}{3}.\\
    III & 900 & 900 & - & You can add \sfrac{1}{2}.\\
    IV & 1,500 & 1,500 & - & You add your whole Intelligence.\\
\end{tabular}


\section{Tricks in Neural Network}
\label{sec:Solution}
Training neural network to achieve a decent performance in ordinary method
is challenging. In recent years, a growing number of ultra deep neural
networks, like VGG-16 with 16 layers and over 100 million parameters,
can be easily trained. The huge leap attributes to the diligent theorists
who devised various sophisticated tricks boosting the training efficiency
and addressing some of the problems mentioned in \autoref{sec:Problem}.

\subsection{A Good Initialization is All You Need \protect\footnote{The
title of this section is quoted from Mishkin et al. \parencite{mishkin2015all} whose
title had a profound influence on our study of the neural network.}}

From the experience in \autoref{sssec:Saddle}, a initialization with common
Gaussian distribution appears to an obstacle to convergence. Romero et al.
\parencite{romero2014fitnets} states that for deep neural net with too many
layers and especially those with uniform normalization, it is hard to train
using BP. Thus, we hereby conclude a number of effective initialization methods.

\subsubsection{Xavier Initialization}
In \parencite{glorot2010understanding}, Glorot et al. proposed a adoption of standard
Gaussian initialization which was called "Xavier" initialization by Jia et al.
\parencite{jia2014caffe}. Xavier initialization aims to improve the performance of
neural network using sigmoid activation and log-likelihood loss function:
\begin{equation}
    \label{equ:Xavier}
    \begin{split}
        E(W^{[l]}) & = 0, \\
        Var(W^{[l]}) & = \frac{2}{n^{[l-1]} - n^{[l]}}
    \end{split}
\end{equation}

\begin{prf}
    For a activation function $ g $ with $ f'(0) = 1 $, like sigmoid $ \sigma(z) $,
    if we use the notation in \autoref{equ:FP} and \autoref{equ:BP}:
    \begin{align*}
        Z^{[l]} & = W^{[l]}A^{[l-1]} + b^{[l]} \\
        dZ^{[l]} & = (W^{[l]T}dZ^{[l+1]})*g^{[l]'}(Z^{[l]}) \\
        dW^{[l]} & = dZ^{[l]}A^{[l-1]T}
    \end{align*}
    We assume that the weights are independent and the distribution of
    input $ X $ are the same. Then $ g^{[l]'}(Z_k^{l}) \approx 1 $, Using the
    chain rule we have:
    \begin{align}
        Var(A^{[l]}) & = Var(X)\prod\limits_{l'=0}^{l-1}n^{[l'-1]}Var(W^{[l']}) \\
        Var(dZ^{[l]}) & = Var(dZ^{[L]})\prod\limits_{l'=L}^{l}n^{[l']}Var(W^{[l']})
    \end{align}
    To keep information flowing in FP and BP, we would like to have:
    \begin{align}
        & \forall(l,\ l'),\ Var(A^{[l]}) = Var(A^{[l']}) \\
        & \forall(l,\ l'),\ Var(dZ^{[l]}) = Var(dZ^{[l']})
    \end{align}
    So:
    \begin{align}
        & \forall l,\ n^{[l'-1]}Var(W^{[l']}) = 1 \\
        & \forall l,\ n^{[l']}Var(W^{[l']}) = 1
    \end{align}
    To combine the two equations above, we may have:
    \begin{eqnarray}
        Var(W^{[l']}) = \frac{2}{n^{[l-1]} - n^{[l]}}
    \end{eqnarray}
    \textbf{Q.E.D.}
\end{prf}
\par For Gaussian initialization, Xavier suggest to have:
\begin{equation}
    W \sim N(0,\ \frac{2}{n^{[l-1]} - n^{[l]}})
\end{equation}
Though rarely in use, a normalized version of Xavier initialization is
provided in \parencite{glorot2010understanding}:
\begin{equation}
    W \sim U(-\frac{\sqrt{6}}{\sqrt{n^{[l-1]} + n^{[l]}}},\ \frac{\sqrt{6}}{\sqrt{n^{[l-1]} + n^{[l]}}})
\end{equation}

\subsubsection{Kaiming Initialization}
\label{sssec:Kaiming}
He et al. \parencite{he2015delving} extended the \autoref{equ:Xavier} to ReLU
activation and achieved a better performance. In this article, a modification
of ReLU, the Parametric Rectified Linear Unit (PReLU)\footnote{The detail of
PReLU will be discuss in Appendix 2 } was proposed as well.
Due to the lack of $ g'(0) = 1 $ in ReLU, "Xavier" failed to pursue a convergence
in deep neural networks. Kaiming initialization is set as follow:
\begin{equation}
    \label{equ:Kaiming}
    \begin{split}
        E(W^{[l]}) & = 0, \\
        Var(W^{[l]}) & = \frac{2}{n^{[l]}}
    \end{split}
\end{equation}

\begin{prf}
    We use the notation in \autoref{equ:FP} and \autoref{equ:BP}:
    \begin{align*}
        Z^{[l]} & = W^{[l]}A^{[l-1]} + b^{[l]} \\
        dA^{[l]} & = W^{[l]T}dZ^{[l+1]}
    \end{align*}
    And we review the ReLU: $ g(z) = max(0,\ z) $.
    Let the initialization in parameters and input are mutually independent
    and share the same distribution. Then we have:
    \begin{equation}
        \label{equ:Var}
        \begin{split}
            Var(Z^{[l]}) & = n^{[l]}Var(W^{[l]}A^{[l-1]}) \\
            & = n^{[l]}(Var(W^{[l]})Var(A^{[l-1]}) + Var(W^{[l]})E(A^{[l-1]}))
        \end{split}
    \end{equation}
    Note that $ g(z) = max(0,\ z) $, this leads to $ E(g(z)) = \frac{1}{2}E(z),\
    Var(g(z)) = \frac{1}{4}Var(z) $. Therefore, \autoref{equ:Var} can be converted
    to:
    \begin{equation}
        Var(Z^{[l]}) = \frac{1}{2}n^{[l]}Var(W^{[l]})Var(Z^{[l-1]})
    \end{equation}
    Putting all layers together, we have:
    \begin{equation}
        Var(Z^{[L]}) = Var(Z^{1})\prod\limits_{l=2}^L\frac{1}{2}n^{[l]}Var(W^{[l]})
    \end{equation}
    Similar to the proof in Xavier initialization, we hope the maintain the
    variance in the FP to avoid the possible explosion or vanishing.
    So,
    \begin{equation}
        \label{equ:Kaiming_1}
        \forall l,\ \frac{1}{2}n^{[l]}Var(W^{[l]}) = 1
    \end{equation}
    Likewise in the process of BP, due to the property of ReLU function,
    \begin{equation}
        Var(dA^{[1]}) = Var(dA^{L})\prod\limits_{l=1}^L\frac{1}{2}n^{[l]}Var(W^{[l]})
    \end{equation}
    and the corresponding condition is the same as \autoref{equ:Kaiming_1}.
    \textbf{Q.E.D.}
\end{prf}

\subsubsection{Orthogonal Initialization}
Based on deep linear neural networks, Saxe et al. \parencite{saxe2013exact}
exhibits a class of random orthogonal initialization which enjoys depth
independent learning time in linear networks. Orthogonal initialization is
implemented by choosing all weights to be orthogonal matrix:
\begin{equation}
    (W^{[l]})^TW^{[l]} = I
\end{equation}
The implementation can be divided into two steps:
\begin{enumerate}
    \item[1.] Fill the weights with standard Gaussian distribution.
    \item[2.] Decompose the weights matrix using singular value decomposition
    (SVD) and replace weights with one of the components.
\end{enumerate}
Saxe et al. suggest that an approriate condition on weights for generating fast
learning speed would be dynamical isometry\footnote{Dynamical isometry means
that the input-output Jacobian $ J = \frac{\nabla A^{[L]}}{\nabla A^{[0]}} $
has all singular values close to 1.}. Saxe et al. \parencite{saxe2013exact}
illustrate the answer for depth independence by visualize the eigenvalue
spectrum of a random orthogonal matrix in \autoref{fig:orthogonal}, which is
exactly a unit circle. Moreover, its singular values are all exactly 1.
\begin{figure}[H]
    \centering
    \includegraphics[width=10cm]{orthogonal}
    \caption{\label{fig:orthogonal}The eigenvalue spectrum of a
    random orthogonal matrix $ W \in \mathbb{R}^{100\times 100} $}
\end{figure}
Then in the linear neural networks, the input-output Jacobian is:
\begin{equation}
    J = \prod\limits_{l=1}^LW^{[l]}
\end{equation}
Under orthogonal initialization in each layer, $ J $ is a orthogonal matrix and
thus the initialization achieve dynamical isometry. Still the reason for dynamical
isometry explaining the depth independence will be studied in the future work.

\subsubsection{Layer-Sequential Unit-Variance Initialization (LSUV)}
As far as we aware, part from the Kaiming initialization, mentioned in
\autoref{sssec:Kaiming}, there is no extension of Xavier initialization to
other nonlinear function other than ReLU. Mishkin et al. \parencite{mishkin2015all}
provide a general and simple initialization method that works well with different
activation functions:
\begin{enumerate}
    \item[1.] Initialize the weights using orthogonal initialization.
    \item[2.] Scale the variance of output of each layer for each mini-batch
    \footnote{Mini-batch means a small proportion of the data set used in one
    iteration, which will be introduced in \autoref{sssec:MiniBatch}.} to
    be 1.
\end{enumerate}
Though the explanation of why LSUV outperform other initialization methods is
not given in the article, a series of comparison empirically show its advantage
in neural networks initialization problem.

\subsubsection{Comparison between initialization}
We implemented a simple CNN model based on keras to examine
the performance of random, xavier and he initialization.
The model trained the MNIST data set \parencite{mnist} to
classify the handwriting of Arabic numbers. The CNN model
is constructed with two convolution layers, two max pooling
layers and two fully connected layers. Activations use ReLU
and the output layer uses Softmax. Due to the limitation of
the version of keras and time limit, LSUV is not implemented.
The code is attached in the "/code/initialize.py". The size
of data set is $ M = 60000 $, we set $ m = 64 $. The results
are demonstrated in \autoref{fig:init_cost} and \autoref{fig:init_acc}.

\begin{figure}[H]
    \centering
    \includegraphics[width=10cm]{init_cost}
    \caption{\label{fig:init_cost}The loss in training of different initialization
    methods, only random initialization corresponds to the right y axis}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=8cm]{init_acc}
    \caption{\label{fig:init_acc}The loss and accuracy on test set
    of different initialization methods}
\end{figure}

\par Surprisingly, CNN using random initialization did not
converge and did not learn after the second epoch. Others have
a great convergence, he initialization is a bit faster than the
rests since it is tailored for ReLU.

\subsection{Gradient Descend Variants}
There are two variants of GD improving the training efficiency. In brevity,
batch describes the data set used in an iteration\footnote{In an iteration,
all parameters are updated.}, epoch describes a traversal through the whole
data set. GD introduced in \autoref{ssec:GD} is the basic form of GD. If
we unroll the vectorization in \autoref{equ:GD}, we can have:
\begin{equation}
    \begin{split}
        dW^{[l]} & = \frac{1}{m}\sum\limits_{i=1}^mdZ_{\bullet,\ i}^{[l]}A_{\bullet,\ i}^{[l-1]T} \\
        db^{[l]} & = \frac{1}{m}\sum\limits_{i=1}^mdZ_{\bullet,\ i}^{[l]}
    \end{split}
\end{equation}
The variants take on different convergence speed and accuracy by choosing
different $ m $.

\subsubsection{Batch Gradient Descent (BGD)}
Batch gradient descent, computes the gradient for the whole data set, i.e.
$ m $ is the size of the data set $ M $. Despite the high accuracy resulted
from taking full advantage of data set, BGD failed to
manage the online algorithms due to the considerable training time.

\subsubsection{Stochastic Gradient Descent (SGD)}
In comparison to BGD, SGD is competent for computation
new data on the fly. In each iteration, SGD updates the parameters using
only one piece of data $ (x^{(i)},\ y^{(i)}) $, i.e. $ m = 1 $:
\begin{equation}
    \begin{split}
        dW^{[l]} & = \sum\limits_{i=1}^mdZ^{[l]}A^{[l-1]T} \\
        db^{[l]} & = \sum\limits_{i=1}^mdZ^{[l]} \\
        dZ^{[l]},&\ A^{[l]} \in \mathbb{R}^{n^{[l]}\times 1}
    \end{split}
\end{equation}
Since only one sample is applied to the computation of gradient, the speed
of SGD is fast enough for daily application. On the other hand, optimizing
for one sample does not guarantee the convergence to a minimum and a descent
of cost in each iteration. It is therefore SGD suffers a severe fluctuation
and may wander around the minimum.

\subsubsection{Mini-batch Gradient Descent (MGD)}
\label{sssec:MiniBatch}
To balance the merit of both BGD and SGD, mini-batch gradient descent
choose an appropriate $ m = 2^k,\ k\in R^+ $. Usually, $ m $ will be chosen
as 64, 128 and 256. Taking the best of both worlds, MGD
descent has a fast and stable convergence.

\subsubsection{Comparison Between Variants}
To illustrate the different between batch BGD, SGD and MGD, we
implement a multi-classification neural network based on the tensorflow
framework. The architecture
of deep neural network with two hidden layers using ReLU and the output
layer using Softmax, Adam optimization mentioned in \autoref{sssec:Adam}
is applied to pursue better convergence. Due to the limitation of the
old version of tensorflow, only Xavier initialization is available.
The code attached in the " /code/test.py ".
The size of data set is $ M = 1080 $, we set $ m = 1080,\ 1,\
64 $ respectively for three variants. The results are shown in the following
, see \autoref{fig:BGD_cost},\autoref{fig:BGD},\autoref{fig:SGD_cost},\autoref{fig:SGD},\autoref{fig:MGD_cost},\autoref{fig:MGD}.

\begin{figure}[H]
    \centering
    \includegraphics[width=8cm]{BGD_cost}
    \caption{\label{fig:BGD_cost}The loss of BGD during training}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=5cm]{BGD}
    \caption{\label{fig:BGD}The result of training}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=8cm]{SGD_cost}
    \caption{\label{fig:SGD_cost}The loss of SGD during training}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=5cm]{SGD}
    \caption{\label{fig:SGD}The result of training}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=8cm]{MGD_cost}
    \caption{\label{fig:MGD_cost}The loss of MGD during training}
\end{figure}

\begin{figure}[H]
    \centering
    \includegraphics[width=5cm]{MGD}
    \caption{\label{fig:MGD}The result of training}
\end{figure}

\par Unfortunately, under the testing condition, SGD did not converge but stuck
in some kind of local minimum or saddle point. Comparing MGD and BGD, to
achieve the same accuracy of 99\%, MGD is much more faster but some glitches
can be observed in the cost, which means MGD dose not guarantee descent in each
iteration like SGD. In all, the simple comparison roughly demonstrate the advantage
of MGD in the optimization process.

\subsection{Optimization Algorithms}
Mini-batch GD does improve the convergence speed in deep neural
networks, but the aforementioned challenges like saddle points,
learning rate selection are still need to be addressed. We
summarize some prevailing optimization algorithms as follow.

\subsubsection{Momentum}
Momentum method is one of the earliest simple and robust modification to
pursue faster learning by Qian et al. \parencite{qian1999momentum}.
When the cost function is similar to a long and
narrow valley like \autoref{fig:valley}, most gradient is perpendicular
to the long axis, the the back and forth track would drastically slows
down the learning speed.

\begin{figure}[H]
    \centering
    \subfigure[Without momentum]{
        \begin{minipage}[t]{0.5\linewidth}
        \centering
        \includegraphics[width=6cm]{Momentum1}
        \end{minipage}%
    }%
    \subfigure[With momentum]{
        \begin{minipage}[t]{0.5\linewidth}
        \centering
        \includegraphics[width=6cm]{Momentum2}
        \end{minipage}%
    }%
    \centering
    \caption{\label{fig:valley}A long and narrow valley
    situation. From \parencite{momentum}}
\end{figure}

In this situation, we are eager to keep the track
parallel to the long axis by some kind of force. In physics, it is called
the momentum, and thus the terminology comes into being. Momentum updates
the parameters in the following way:
\begin{equation}
    \label{equ:Momentum}
    \begin{split}
        v_{t} & = \beta v_{t-1} - \alpha dW_t^{[l]},\ v_0 = 0 \\
        W_t^{[l]} & = W_{t-1}^{[l]} + v_{t}
    \end{split}
\end{equation}
The recursive definition of $ v_t $ is called the exponentially decaying average.
\par The relation between momentum method and its physics background is readily
comprehensible. When a ball moving in a field with
a fraction w.r.t. the speed, then the Newtonian equation is:
\begin{equation}
    \label{equ:Newton}
    m\frac{d^2x}{dt^2} + \mu\frac{dx}{dt} = -\nabla E(x)
\end{equation}
where $ m $ is the mass, $ \mu $ is the fraction coefficient and $ E(x) $
is the potential energy. Since
$ \frac{dx}{dt} = \frac{x_{t+1} - x_{t}}{\Delta t} $, \autoref{equ:Newton}
can be rewritten as:
\begin{equation}
    \label{equ:physics}
    \begin{split}
        m\frac{(x_{t+1} - x_{t}) - (x_{t} - x_{t-1})}{(\Delta t)^2} + \mu\frac{x_{t+1} - x_{t}}{\Delta t} = -\nabla E(x) \\
        x_{t+1} - x_{t} = -\frac{(\Delta t)^2}{m+\mu\Delta t}\nabla E(x) + \frac{m}{m+\mu\Delta t}(x_{t} - x_{t-1})
    \end{split}
\end{equation}
If we set $ \frac{(\Delta t)^2}{m+\mu\Delta t} = \alpha,\
\frac{m}{m+\mu\Delta t} = \beta $, then we obtain the \autoref{equ:Momentum}.

Here, we briefly demonstrate the convergence analysis of momentum method.
Using the Hessian $ \nabla E(x_t) = Hx_t $ and similarity transformation
$ H = Q^TDQ,\ Q^TQ = I $, we can convert \autoref{equ:physics} into:
\begin{equation}
    \begin{split}
        x_{t+1} & = ((1 + \beta)I -\alpha Q^TDQ)x_t - \beta x_{t-1} \\
        Q^Tx_{t+1} & = (1 + \beta)IQ^Tx_t -\alpha Q^TQ^TDQx_t - \beta Q^Tx_{t-1}
    \end{split}
\end{equation}
We denote $ x_t = x_t' $, then the element-wise equation is:
\begin{equation}
    x_{i,\ t+1}' = ((1+\beta)-\alpha d_i)x_{i,\ t}' - \beta x_{i,\ t-1}'
\end{equation}
Then a matrix representation of this equation is:
\begin{equation}
    \label{equ:linear}
    \begin{split}
        \begin{pmatrix}
            x_{i,\ t}' \\
            x_{i,\ t+1}'
        \end{pmatrix}
        = A_i
        \begin{pmatrix}
            x_{i,\ t-1}'   \\
            x_{i,\ t}'
        \end{pmatrix} \\
        A_i =
        \begin{pmatrix}
            0 & 1   \\
            -\beta & 1 + \beta - \alpha d_i
        \end{pmatrix}
    \end{split}
\end{equation}
The convergence of momentum is determined by the linear system
in \autoref{equ:linear}. Recall the knowledge in numerical
analysis, a theorem w.r.t. the convergent matrix is stated
in Theorem 7.17 in Burden et al. \parencite{burden2010numerical}:
\begin{thm}
    $ A $ is a convergent matrix\footnote{We call a matrix A convergent if
    \begin{align}
        \lim _{k \rightarrow \infty}\left(A^{k}\right)_{i j}=0, \quad \text{for each } i=1,2, \ldots, n \text{and } j=1,2, \ldots, n
    \end{align}.}
    $\iff$ the spectrum radius $ \rho(A) $\footnote{Spectrum radius is
    defined as $ \rho(A) $ = max($|\lambda|$)}
    satisfies
    \begin{equation}
        \rho(A) < 1
    \end{equation}.
\end{thm}
A proposition given in Appendix B in Qian et al.
\parencite{qian1999momentum} provide the theoretical
foundation of the convergence.
\begin{pro}
    Let $ \lambda_{i,\ 1},\ \lambda_{i,\ 2} $ are the
    eigenvalues of $ A_i $. The system in \autoref{equ:linear}
    converges if and only if
    \begin{equation}
        -1 < \beta < 1,\ 0 < \alpha d_i < 2+2\beta
    \end{equation}
\end{pro}

\subsubsection{Nesterov Accelerated Gradient (NAG)}
\label{sssec:NAG}
NAG in Nesterov \parencite{nesterov1983method}
is a optimization method solving convex problems with
convergence rate $ O(1/K^2) $, where $ K $ is the number of iterations.
NAG is virtually similar to momentum method in spite of two differences.
First, learning rate is derived from the Armijo rule with $ \beta = 2,\
\sigma = 0.5 $ in \autoref{sssec:Armijo}. The momentum constant $ \beta $
is formulated in a recursive way that $ \beta_t \sim \frac{t+4}{2} $, where
$ t $ is the index of iteration. Second, a subtle update rule is applied
in NAG. Considering the fact that the former difference just an additional
condition to guarantee the convexity, such formulation seems trivial in
the common non-convex optimization. Instead, we manually choose the parameters
$ \beta $ and $ \alpha $ in advanced and focus more on the later aspect:
\begin{equation}
    \begin{split}
        v_t & = \beta v_{t-1} - \alpha\nabla J(W_{t-1} + \beta W_{t-1}),\ v_0 = 0 \\
        W_t & = W_{t-1} +  v_t
    \end{split}
\end{equation}
\par To the best of our knowledge, there have been no attempts to generalize the
convergence analysis of NAG to non-convex problem. But a empirical experiment
in Sutskever's PhD thesis \parencite{sutskever2013training}.
\par Intuitively, to comprehend the improvement of NAG, imagine SGD as a ball
rolling downhill (like in the physics background of momentum). In a rough terrain,
blindly rolling with gradient seems to be silly, because there is no forecast
when the ball suddenly face a upward slope. As shown in \autoref{fig:Nesterov},
NAG is more stable than momentum due to the backward correction.

\begin{figure}[H]
    \centering
    \includegraphics[width=8cm]{Nesterov}
    \caption{\label{fig:Nesterov}NAG illustration: blue vector represents the
    momentum method, green vector represents the NAG with the brown component
    is the gradient and the red component is the correction.}
\end{figure}

\subsubsection{RMSprop}
\label{sssec:RMSprop}
It is interesting that RMSprop is not published in paper but propose in a
Coursera course by Hinton et al. \parencite{rmsprop}. RMSprop adopts the idea
of only using the sign of the gradient by dividing the gradient with the size
of it:
\begin{equation}
    \begin{split}
        u_t & = \gamma u_{t-1} + (1-\gamma)(dW_t^{[l]})^2,\ u_0 = 0 \\
        W_{t+1}^{[l]} & = W_{t}^{[l]} - \frac{\eta}{\sqrt{u_t+\epsilon}}dW_t^{[l]}
    \end{split}
\end{equation}
Exponentially decaying average $ u_t $ is applied to the second moment of gradient.
In brevity, we denote such average as $ RMS(dW_t^{[l]}) $.
$ \epsilon $ is a smoothing term that prevents division by zero and usually
set to $ 10^{-8} $. In the lecture, Hinton suggests $ \gamma = 0.9 $.

\subsubsection{Adagrad}
Duchi et al. \parencite{duchi2011adaptive} make a vivid metaphor that some
infrequent but predictive features are needles in haystacks. Adagrad is a
subgradient method that crafts domain-specific weightings by performing large
updates for infrequent features and small updates for frequent ones.
\begin{equation}
    W_{t}^{[l]} = W_{t-1}^{[l]} - \frac{\eta}{\sqrt{\sum _{t'=1}^t(dW_{t'}^{[l]})^2} +
        \epsilon}dW_{t}^{[l]}
\end{equation}
\par Note that the update is divided by the sum of the squared gradients, the
learning rate will shrink to vanishing with the iteration. To address this flaw,
an adaption called Adadelta is illustrated in the following.

\subsubsection{Adadelta}
Adadelta presented in Zeiler et al. \parencite{zeiler2012adadelta} aims to improve
Adagrad in two aspect: the diminishing learning rate and the manual tunning
learning rate.
\par First, instead of simply summing up the squared gradients, a accumulation
over window method is introduced, which turns out to be exactly the RMSprop
in \autoref{sssec:RMSprop}. Second, the article suggests that as a
update of $ W $, $ \Delta W $ should have the same unit as $ W $. However, in usual
optimization method, this is not the case:
\begin{equation}
    \text{units of}\Delta W \propto \text{units of} dW \propto \frac{\partial J}{\partial W} \propto \frac{1}{\text{units of}W}
\end{equation}
Comparing with the Newton's method in Chapter 10.2 in Burden et al.
\parencite{burden2010numerical}, the Hessian is used to maintain the same unit:
\begin{equation}
    \Delta W \propto H^{-1} dW \propto \frac{\frac{\partial J}{\partial W}}{\frac{\partial^{2} J}{\partial W^{2}}} \propto \text { units of } W
\end{equation}
The term $ \frac{1}{\frac{\partial^{2} J}{\partial W^{2}}} = \frac{\Delta W}{\frac{\nabla J}{\nabla W}} $
can be added to obtain the match of unit. The unknown $ \Delta W $ in the current iteration
will be substituted by the previous one. Since the denominator use the RMSprop,
the numerator should take the identical form:
\begin{equation}
    \begin{split}
        v_t & = -\frac{RMS(v_{t-1})}{RMS(dW_t^{[l]})} \\
        W_{t+1}^{[l]} & = W_{t}^{[l]} + v_t
    \end{split}
\end{equation}

\subsubsection{Adam}
\label{sssec:Adam}
The name of Adam comes from adaptive moment estimation in Kingma et al. \parencite{kingma2014adam}
It is a memory saving and easy to compute optimization method combining
both the advantages of RMSprop and momentum. The pseudo code of Adam is
shown in \autoref{alg:Adam}.

\begin{algorithm}
    \caption{Adam. Note that all operators are element-wise. The recommended
    hyperparameters are learning rate $ \alpha = 0.001 $, momentum
    constant $ \beta = 0.9 $, RMSprop constant $ \gamma = 0.999 $,
    smoothing term $ \epsilon = 10^{-8} $.}
    \label{alg:Adam}
    \begin{algorithmic}
    \REQUIRE $\alpha$: Stepsize
    \REQUIRE $\beta,\ \gamma \in [0,\ 1)]$: Exponential decay rates
                for moment estimation
    \REQUIRE $J(W)$: Cost function with parameters $ W $
    \REQUIRE $W_0$: Initial parameter
    \STATE $m_0 \gets 0$ (Initialize \engordnumber{1} moment vector)
    \STATE $v_0 \gets 0$ (Initialize \engordnumber{2} moment vector)
    \STATE $t \gets 0$ (Initialize time step)
    \WHILE{$ W_t $ not converged}
    \STATE $ t \gets t + 1 $
    \STATE $ g_t \gets \nabla J(W_{t-1}) $(Get gradients w.r.t. cost function)
    \STATE $ m_t \gets \beta m_{t-1} + (1-\beta)g_t $ (Update biased first moment estimate)
    \STATE $ v_t \gets \gamma v_{t-1} + (1-\gamma)g_t^2 $ (Update biased second moment estimate)
    \STATE $ \hat{m_t} \gets \frac{m_{t-1}}{1-\beta^t}$ (Compute bias-corrected first moment estimate)
    \STATE $ \hat{v_t} \gets \frac{v_{t-1}}{1-\gamma^t} $ (Compute bias-corrected second raw moment estimate)
    \STATE $ W_t \gets W_{t-1} - \alpha \frac{\hat{m_t}}{\sqrt{\hat{v_t}}+\epsilon} $ (Update parameters)
    \ENDWHILE
    \RETURN $ W_t $ (Resulting parameters)
    \end{algorithmic}
\end{algorithm}

Using the definition in mathematical statistic, the Exponential decaying average
calculates the mean and uncentered variance of gradients, i.e. the \engordnumber{1}
moment and the \engordnumber{2} raw moment. Due to the initialization of zero which
is not there actual value, the initialization bias correction. Calculate the
recursive definition of $ m_t $, we can obtain:
\begin{align}
    m_t = \beta^tm_0 + (1-\beta)\sum _{i=1}^t\beta^{t-i}g_i
\end{align}
Initialize $ m_0 = 0 $, and we have:
\begin{equation}
    m_t = (1-\beta)\sum _{i=1}^t\beta^{t-i}g_i
\end{equation}
We want the expectation of $ m_t $ and $ g_t $ is equivalent.
\begin{align}
    E(m_t) & = (1-\beta)\sum _{i=1}^t\beta^{t-i}E(g_t) + \zeta \\
    & = (1-\beta^t)E(g_t) + \zeta
\end{align}
If $ E(g_i) $ does not change with time, then $ \zeta=0 $. Even if $ E(g_i) $
varies, an appropriate $ \beta $ can keep $ \zeta $ involved the past gradients
small. Therefore, adding a $ 1-\beta^t $ in the denominator can correct the bias.
\par A analysis of convergence only for convex cost function is given in the
article \parencite{kingma2014adam}. Under the definition of regret:
\begin{equation}
    R(T) = \sum _{t=1}^T(J(W_t) - J(W^*))
\end{equation}
where $ W^* = \mathop{\arg\min}_{W\in (W_1,\cdots,\ W_T)}\sum _{t=1}^TJ(W) $.
The theorem and proof is too obscure, hence we simply illustrate a corollary
showing the bound of regret.
\begin{cor}
    Assume that the function $J$ has bounded gradients,
    $ ||\nabla J(W)||_2 \leq G,\ ||\nabla J(W)||_\infty \leq G_\infty,
    \ \forall W $, and distance between any $ W_t $ is bounded,
    $ ||W_m - W_n||_2 \leq D,\ ||W_m - W_n||_\infty \leq D_\infty,\
    \forall m,\ n \in (1,\cdots,\ T) $. Adam achieve the following
    guarantee:
    \begin{equation}
        \forall T \geq 1,\ \frac{R(T)}{T} = O(\frac{1}{\sqrt{T}})
    \end{equation}
\end{cor}
Therefore, the regret bound $ O(\sqrt{T}) $ is obtained.

\subsubsection{AdaMax}
AdaMax is a simple and stable extension of Adam is attached
in the same article \parencite{kingma2014adam}. While $L_2$ Norm
is numerically complex, AdaMax uses $L_p$ Norm for the
\engordnumber{2} raw moment, and let $ p \rightarrow \infty $:
\begin{equation}
    \label{equ:AdaMax}
    \begin{split}
        v_t & = \gamma^\infty v_{t-1} + (1-\gamma^\infty)g_t^\infty \\
        & = max(\gamma v_{t-1},\ |g_t|)
    \end{split}
\end{equation}
The derivation in \autoref{equ:AdaMax} is similar to $L_\infty$ Norm.
In such adaption, even though the initialization of $ v_0 = 0 $, there
is no need to correct the bias. See \autoref{alg:AdaMax} for a complete
pseudo code.
\begin{algorithm}
    \caption{AdaMax. Note that all operators are element-wise. The recommended
    hyperparameters are learning rate $ \alpha = 0.002 $, momentum
    constant $ \beta = 0.9 $, RMSprop constant $ \gamma = 0.999 $,
    smoothing term $ \epsilon = 10^{-8} $.}
    \label{alg:AdaMax}
    \begin{algorithmic}
    \REQUIRE $\alpha$: Stepsize
    \REQUIRE $\beta,\ \gamma \in [0,\ 1)]$: Exponential decay rates
                for moment estimation
    \REQUIRE $J(W)$: Cost function with parameters $ W $
    \REQUIRE $W_0$: Initial parameter
    \STATE $m_0 \gets 0$ (Initialize \engordnumber{1} moment vector)
    \STATE $v_0 \gets 0$ (Initialize \engordnumber{2} moment vector)
    \STATE $t \gets 0$ (Initialize time step)
    \WHILE{$ W_t $ not converged}
    \STATE $ t \gets t + 1 $
    \STATE $ g_t \gets \nabla J(W_{t-1}) $(Get gradients w.r.t. cost function)
    \STATE $ m_t \gets \beta m_{t-1} + (1-\beta)g_t $ (Update biased first moment estimate)
    \STATE $ v_t \gets max(\gamma v_{t-1},\ |g_t|) $ (Update the exponentially weighted infinity norm)
    \STATE $ \hat{m_t} \gets \frac{m_{t-1}}{1-\beta^t}$ (Compute bias-corrected first moment estimate)
    \STATE $ W_t \gets W_{t-1} - \alpha \frac{\hat{m_t}}{v_t} $ (Update parameters)
    \ENDWHILE
    \RETURN $ W_t $ (Resulting parameters)
    \end{algorithmic}
\end{algorithm}

\subsubsection{NAdam}
NAG in \autoref{sssec:NAG} shows a powerful correction for momentum,  and it is
generally acknowledged that standard momentum is inferior to NAG. Dozat
\parencite{dozat2016incorporating} improves Adam's momentum part to NAG and
achieves faster convergence speed.
\par First, Dozat modifies the NAG by applying a look ahead momentum to the update:
\begin{equation}
    \begin{split}
        v_t & = \beta v_{t-1} - \nabla J(W_{t-1}),\ v_0 = 0 \\
        W_t & = W_{t-1} +  \beta v_t + \alpha \nabla J(W_{t-1})
    \end{split}
\end{equation}
Then we can modify Adam in the following way:
\begin{equation}
    \begin{split}
        W_t & = W_{t-1} - \alpha \frac{m_t}{(\sqrt{\hat{v_t}} + \epsilon)(1-\beta^t)} \\
        & = W_{t-1} - \alpha \frac{\beta m_t + (1-\beta)g_t}{(\sqrt{\hat{v_t}} + \epsilon)(1-\beta^t)} \\
        & = W_{t-1} - \frac{\alpha}{\sqrt{\hat{v_t}} + \epsilon}(\beta \hat{m_t} + \frac{(1-\beta)g_t}{1-\beta^t})
    \end{split}
\end{equation}
A complete algorithm is shown in \autoref{alg:NAdam}.

\begin{algorithm}[H]
    \caption{NAdam. Note that all operators are element-wise. The recommended
    hyperparameters are learning rate $ \alpha = 0.002 $, momentum
    constant $ \beta = 0.975 $, RMSprop constant $ \gamma = 0.999 $,
    smoothing term $ \epsilon = 10^{-8} $.}
    \label{alg:NAdam}
    \begin{algorithmic}
    \REQUIRE $\alpha$: Stepsize
    \REQUIRE $\beta,\ \gamma \in [0,\ 1)]$: Exponential decay rates
                for moment estimation
    \REQUIRE $J(W)$: Cost function with parameters $ W $
    \REQUIRE $W_0$: Initial parameter
    \STATE $m_0 \gets 0$ (Initialize \engordnumber{1} moment vector)
    \STATE $v_0 \gets 0$ (Initialize \engordnumber{2} moment vector)
    \STATE $t \gets 0$ (Initialize time step)
    \WHILE{$ W_t $ not converged}
    \STATE $ t \gets t + 1 $
    \STATE $ g_t \gets \nabla J(W_{t-1}) $(Get gradients w.r.t. cost function)
    \STATE $ m_t \gets \beta m_{t-1} + (1-\beta)g_t $ (Update biased first moment estimate)
    \STATE $ v_t \gets \gamma v_{t-1} + (1-\gamma)g_t^2 $ (Update biased second moment estimate)
    \STATE $ \hat{m_t} \gets \frac{m_{t-1}}{1-\beta^t}$ (Compute bias-corrected first moment estimate)
    \STATE $ \hat{v_t} \gets \frac{v_{t-1}}{1-\gamma^t} $ (Compute bias-corrected second raw moment estimate)
    \STATE $ W_t \gets W_{t-1} - \frac{\alpha}{\sqrt{\hat{v_t}}+\epsilon}(\beta \hat{m_t} + \frac{(1-\beta)g_t}{1-\beta^t}) $ (Update parameters)
    \ENDWHILE
    \RETURN $ W_t $ (Resulting parameters)
    \end{algorithmic}
\end{algorithm}

\begin{comment}
    \subsubsection{Comparison}
\end{comment}


\documentclass[main.tex]{subfiles}
\begin{document}

\marginpar{Monday\\ 2020-10-5, \\ compiled \\ \today}

These next few lectures, we will consider the motivations for inflationary models.
The problems they solved were the \textbf{shortcomings of the Hot Big Bang} model.

\begin{enumerate}
    \item The horizon problem;
    \item the flatness problem;
    \item unwanted relics / magnetic monopole problem.
\end{enumerate}

We start by recalling some basic elements in cosmology.
In order to describe a homogeneous and isotropic universe we use the FLRW metric:
%
\begin{align}
\dd{s}^2  = - c^2 \dd{t}^2 + a^2(t) \qty[ \frac{ \dd{r}^2}{1 - k r^2} + r^2 \dd{\Omega}^2]
\,,
\end{align}
%
where \(\dd{\Omega}^2 = \dd{\theta}^2 + \sin^2 \theta \dd{\varphi }^2
\). The quantity \(a(t)\) is called the \emph{scale factor}.
The coordinates \(r\), \(\theta \) and \(\varphi \) are called \emph{comoving coordinates}.

Physical distances and comoving distances are related by
%
\begin{align}
\lambda _{\text{phys}} = a(t) \lambda _{\text{comoving}}
\,.
\end{align}

The constant  \(k\) is the spatial curvature of the universe, which can always be rescaled so that it is equal to
\begin{enumerate}
    \item \(+1\) for a spatially closed universe;
    \item \(0\) for a spatially flat universe;
    \item \(-1\) for a spatially open universe.
\end{enumerate}

In terms of the scale factor we define the \textbf{Hubble parameter}
%
\begin{align}
H = \frac{\dot{a}}{a}
\,,
\end{align}
%
which describes the rate at which the universe expands.

The dynamics of gravity are described by the Einstein equations:
%
\begin{align}
G_{\mu \nu } = 8 \pi G T_{\mu \nu }
\,,
\end{align}
%
where \(T_{\mu \nu }\) is the energy-momentum tensor of the particle species filling the universe, while \(G_{\mu \nu } = R_{\mu \nu } - R g_{\mu \nu } / 2\) is the Einstein tensor, describing curvature.

These can be derived from an action principle through the action
%
\begin{align}
S = \underbrace{\frac{1}{16 \pi G} \int R \sqrt{-g} \dd[4]{x}}_{S_{EH}} +
S_{\text{matter}}
\,.
\end{align}

Often we use an ideal fluid energy-momentum tensor:
%
\begin{align}
T_{\mu \nu } = \rho u_{\mu } u_{\nu } + P h_{\mu \nu }
\,,
\end{align}
%
where \(h_{\mu \nu } = u_{\mu } u_{\nu } + g_{\mu \nu }\) is a projector onto the space orthogonal to the four-velocity. This does not account for any anisotropy, it is the most symmetric energy-momentum tensor.
% This is a diagonal

In order to solve the Einstein equations we can proceed with some assumptions, without needing to know the action for all the fundamental fields. The perfect fluid S-E tensor has all the FLRW symmetries, as long as \(\rho \) and \(P\) are only functions of time.

% We are \emph{not} saying that this S-E tensor is only allowed if we are in a FLRW universe.
% \todo[inline]{Clarify\dots }

Requiring the FLRW symmetries means that the S-E tensor must be diagonal, however we can have viscosity as long as it is not \emph{shear} but \emph{bulk} viscosity, which adds onto the diagonal terms.

Inserting the FLRW metric into the Einstein equation yields the Friedmann equations:
%
\begin{align}
\frac{\dot{a}^2}{a^2} &= \frac{8\pi G}{3} \rho - \frac{k}{a^2}  \\
\frac{\ddot{a}}{a} &= - \frac{4 \pi G}{3} \qty(\rho + 3 P)  \\
\dot{\rho} &= - 3 \frac{\dot{a}}{a} \qty(\rho + P)
\,.
\end{align}

The first two can be derived from the Einstein equations directly, the third comes from the ``conservation law'' \(\tensor{T}{_{\mu \nu }^{;\nu }} = 0\).

They are not independent, only two are.
We have too many parameters: \(a\), \(\rho \) and \(P\), but only two independent equations, so we ``close'' the system of equations with an equation of state, commonly \(P = P(\rho ) = w \rho \).

These equations of state describe many kinds of fluids (approximately):
dust with \(w = 0\), which means \(\rho \propto a^{-3}\); radiation with \(w = 1/3\), which means \(\rho \propto a^{-4}\); a cosmological constant with \(w = -1\), which means \(\rho = \const\).

In general as long as \(w \neq 1\) we have \(\rho \propto a^{-3(1+w)}\) and \(a \propto t^{2 / 3 (1+w)}\).

\subsection{The horizon problem}

The \textbf{particle horizon}, denoted as \(d_H (t)\), is given by
%
\begin{align}
d_H (t) = a(t) \int_{0}^{t} \frac{c \dd{\tau }}{a(\tau )}
\,,
\end{align}
%
and it sets the radius of a sphere centered at an observer \(O\). The points inside this sphere have been able to have causal interactions with observer \(O\) in the time from the Big Bang to \(t\).

It is the proper distance (as measured today) which could have been travelled by light starting at the beginning and moving in a geodesic.
It can be derived from the FLRW metric by assuming radial light-like motion
%
\begin{align}
\dd{s}^2 = - c^2 \dd{t}^2 + a(t)\frac{ \dd{r}^2}{1 - kr^2} = 0
\,,
\end{align}
%
and setting \(k = 0\):\footnote{This is a good approximation for early times, even if the universe is not flat.}
%
\begin{align}
c \dd{t} = \pm  a(t) \dd{r}
\,,
\end{align}
%
which we can use to calculate the \emph{comoving distance} from the point of emission to today, which we then multiply by the scale factor calculated at a chosen point.

We know that the scale factor goes to 0 as \(t\) goes to 0, so the integral giving us \(r_H\) could diverge. We can show that \(d_H\) is finite as long as \(\alpha = 2 / 3 (1+w)\) is smaller than one, meaning that \(w > - 1/3\), which is equivalent to \(\ddot{a} < 0\).
In a decelerating universe, the particle horizon is finite.

In general, the calculation yields
%
\begin{align}
d_H (t) = \frac{3(1 +w)}{1 + 3w} ct
\,.
\end{align}

With \(w = 0\), a spatially flat matter-dominated universe, \(d_H = 3 ct\). This is called an Einstein-De Sitter universe.
With \(w = 1/3\), a spatially flat radiation-dominated universe, we have \(d_H = 2 ct\).

Another way to characterize causality is the Hubble radius:
%
\begin{align}
r_C(t) = \frac{c}{H(t)}
\,.
\end{align}

The characteristic time of expansion is \(\tau (H ) =H^{-1}\).

\begin{claim}
In a FLRW universe, typically after a Hubble time the scale factor doubles.
\end{claim}

\begin{proof}
By solving the Friedmann equations we see that the scaling is as \(a(t) \propto t^{\alpha }\), \(H = \alpha /t \), so a Hubble time is just \(\tau _H = H^{-1} = t / \alpha \). Then, the change in the scale factor in a Hubble time is
%
\begin{align}
\frac{a (t + \tau _H)}{a(t)} = \frac{\qty(t + t/ \alpha )^{\alpha }}{t^{\alpha }} = \qty(1 + \frac{1}{\alpha })^{\alpha }
\,.
\end{align}

We know that \(\alpha \) is between \(0\) and \(1\): the value of this ratio of scale factors is 1 for \(\alpha = 0\) (which would correspond to \(w \to \infty \)), and goes monotonically up to 2 for \(\alpha = 1\) (which corresponds to \(w = -1/3\)).
With \(w = 1/3\), for example, we get \(\alpha = 1/2\) and the ratio of the scale factors is \(\approx \num{1.7}\), which is reasonably close to \(2\).
\end{proof}

Since
%
\begin{align}
H(t) = \frac{2}{3 (1+w)} \frac{1}{t} = \frac{\alpha}{t}
\,,
\end{align}
%
we can define
%
\begin{align}
R_H = \frac{1 + 3w}{2} d_C (t) \approx d_H (t)
\,.
\end{align}

Typically the quantity \((1+3w) / 2\) is of order 1, so the Hubble radius \(r_C\) and the particle horizon \(d_H\) are similar.
The two are similar in a regular FLRW universe, while they differ a lot if there is inflation.

\begin{claim}
In an inflationary period, the particle horizon is exponentially larger than the Hubble radius.
\end{claim}

\begin{proof}
Let us assume that the Hubble parameter \(H\) is constant for simplicity. In a real scenario there will be deviations from this hypothesis, but they will not compromise the result.

Then, the Hubble radius is simply given by \(r_C = H^{-1}\); while the particle horizon must be calculated by integrating the conformal time.
We need a relation between the scale factor and time: we can integrate \(\dot{a} = H a\) to get \(a(t) = e^{Ht}\). Then, we find
%
\begin{align}
r_C(t) = a(t) \int_{0}^{t} \frac{ \dd{\tau }}{a(\tau )} = e^{Ht} \int_{0}^{t} e^{-H \tau } \dd{\tau  } = \underbrace{H^{-1}}_{r_C} \qty(e^{Ht} - 1)
\,,
\end{align}
%
which becomes exponentially larger than \(r_C\) for \(t \gg H^{-1}\).
\end{proof}

The particle horizon takes into account \textbf{all the past history of an observer}, the Hubble radius does not care about it: it only describes causal connections taking place in a Hubble time.

Let us introduce the \emph{comoving Hubble radius}: \(r_H (t)\), given by
%
\begin{align}
r_H (t) = \frac{r_C (t)}{a(t)} = \frac{c}{a H}
\,.
\end{align}

Let us plot this for a matter or radiation-dominated FLRW universe.

In radiation domination, \(r_H \propto \sqrt{t}\), while in matter domination \(a \sim t^{2/3}\) so \(r_H \propto t^{1/3} \sim a^{1/2}\).

This comoving radius is then always increasing, initially faster and then slower.

Instead, consider the comoving particle horizon: \(d_H (t) / a(t)\), so just the integral in the definition of \(d_H(t)\):
%
\begin{align}
\frac{d_H (t)}{a(t)} = \int \frac{c \dd{t}}{a} = \int \frac{ \dd{a}}{a} \underbrace{\frac{c}{a H}}_{r_H}
\,,
\end{align}
%
so we can see that the comoving \emph{particle horizon} is the logarithmic integral over the scale factor of the comoving \emph{Hubble radius}: as we mentioned before, this takes into account the whole past history.

In a matter dominated universe, \(d_H = 2 r_C \approx 5 h^{-1} \SI{}{Gpc}\).

\subsection{Horizon problem}

Now we discuss the horizon problem, which is best understood in a comoving plot: \(\log r_H \) versus \(\log t\). We neglect dark energy for simplicity.

If we choose a fixed comoving size \(\lambda \), we get in our model that in early times \(\lambda \) is super-horizon, then at a certain point it crosses the horizon, becoming smaller than \(r_H \). The time at which \(r_H = \lambda \) is called the \emph{horizon crossing} time, \(t_H (\lambda )\).

For times earlier than \(t_H (\lambda )\), by definition it is impossible for points at a distance \(\lambda \) to be causally connected.
This happens for every scale, and it means that for many regions we are interested in there cannot have been causal connection in the early universe.
But, today we observe the universe to exhibit the same properties across the whole sky, even though the regions were causally disconnected earlier.

This is most directly expressed in terms of CMB photons. They would have become causally connected at the quadrupole scale (separations of \SI{90}{\degree}) almost \emph{today}.

We can compute the size of the horizon at the last scattering epoch: this subtends an angle in the sky of around \SI{1}{\degree}; however we observe photons with the same temperature on much larger scales, this was already seen by COBE with an angular resolution of \SI{7}{\degree}.

\begin{claim}
Without inflation, the largest angular scale at which we would expect to see correlations on the Last Scattering Surface is on the order of \SI{1}{\degree}.
\end{claim}

\begin{proof}
Let us establish some notation: we define
%
\begin{align}
E(z) = \frac{H(z)}{H_0 } = \sqrt{\Omega_{0, \Lambda } + \Omega_{0, k} (1+ z)^2 + \Omega_{0, m} (1 + z)^3 + \Omega_{0, r} (1 + z)^{4}}
\,,
\end{align}
%
(the last expression follows from the first Friedmann equation), which allows us to compute distances, since the physical distance as measured today from a point at redshift \(z\) would be
%
\begin{align}
d (z) = \frac{1}{H_0 } \int_{0}^{z} \frac{ \dd{z'}}{E(z')}
\,,
\end{align}
%
since \(d(z)\) is the integral over the appropriate range of \(a_0 \dd{\eta }\) (this is the same integral as one in the definition of the particle horizon), and the integration element of conformal time can be written as
%
\begin{align}
\dd{\eta } = \frac{ \dd{t}}{a} = - \frac{ \dd{z}}{a_0 H(z)}
\,,
\end{align}
%
as can be calculated by differentiating the definition of redshift:
%
\begin{align}
1 + z = \frac{a_0 }{a }
\implies
\dv{z}{t} = - \frac{a_0 \dot{a}}{a^2} = - \frac{a_0 H(z)}{a}
\implies
\frac{ \dd{z}}{H(z) a_0 } = -\frac{ \dd{t}}{a}
\,.
\end{align}

The minus sign will be accounted for by the fact that we are integrating backward in conformal time if we do it for increasing \(z\).
Now, the physical distance from here to the last scattering surface as measured now reads
%
\begin{align}
d(z_{LS}) = \int_{\text{emission}}^{\text{absorption}} a_0 \dd{\eta }
= -a_0  \int_{z_{LS}}^{0} \frac{ \dd{z'}}{a_0 H(z')} = \frac{1}{H_0 } \int_{0}^{z_{LS}} \frac{ \dd{z'}}{E(z')}
\,.
\end{align}

We, however, want a new notion of distance \(d_A\) which allows us to write the small-angle approximation:
%
\begin{align}
d_A (z_{LS}) = \frac{ \Delta x}{\Delta \theta }
\,,
\end{align}
%
where \(\Delta \theta \) is a (small) angle on the sky as measured from here, while \(\Delta x\) is a physical distance as measured at the time of last scattering. The small-angle relation holds in comoving coordinates:
%
\begin{align}
r_{LS} = \frac{\Delta r}{\Delta \theta }
\,,
\end{align}
%
where \(r_{LS} = d(z_{LS}) / a_0 \) is the comoving distance to the last scattering surface, and \(\Delta r\) is the comoving size of the angular feature we are considering.
The physical scale \(\Delta x\) is calculated as \(a_{LS} \Delta r\), and \(a_{LS} = a_0 / (1+z_{LS})\), so the new distance measure must read
%
\begin{align}
\underbrace{a_0 r_{LS}}_{d(z_{LS})} = a_0 \frac{\Delta r}{\Delta \theta } &= (1+z_{LS}) \frac{\Delta x}{\Delta \theta }  \\
\frac{\Delta x}{\Delta \theta } = d_A (z_{LS}) &= \frac{d(z_{LS})}{1 + z_{LS}} = \frac{1}{H_0 (1+z_{LS})} \int_{0}^{z_{LS}} \frac{ \dd{z'}}{E(z')}
\,.
\end{align}

This new distance is called the \emph{angular diameter distance}.
Now, we need to find the appropriate \(\Delta x\): what is the physical size of the largest correlations we can expect to see?
This will just be the particle horizon (corresponding to light-speed communication), measured at that time:
%
\begin{align}
\Delta x = a_{LS} \int \dd{\eta } = \frac{a_{LS}}{a_0 H_0 } \int_{z_{LS}}^{\infty } \frac{ \dd{z'}}{E(z')} = \frac{H_0^{-1}}{1 + z_{LS}} \int_{z_{LS}}^{\infty } \frac{ \dd{z'}}{E(z')}
\,.
\end{align}

Now we are almost done: we can invert the small-angle relation to get
%
\begin{align}
\Delta \theta &= \frac{\Delta x}{d_A(z_{LS})}
= \frac{H_0^{-1} (1+z_{LS})^{-1} \int_{z_{LS}}^{\infty } E^{-1} (z') \dd{z'}}{H_0^{-1} (1+z_{LS})^{-1} \int_0^{z_{LS}} E^{-1} (z') \dd{z'}}  \\
&= \frac{\int_{z_{LS}}^{\infty } E^{-1} (z') \dd{z'}}{\int_0^{z_{LS}} E^{-1} (z') \dd{z'}} \approx \num{.02} \approx \SI{1.2}{\degree}
\,.
\end{align}

The result was obtained by numerical integration of the best-fit values from the Planck mission, as provided by the ``Planck18'' class from the Astropy module\footnote{\url{https://docs.astropy.org/en/stable/cosmology/index.html}} \cite[]{astropycollaborationAstropyProjectBuilding2018}.
However, the same qualitative result can be obtained analytically with much simpler assumptions: taking full matter domination (\(\Omega_{m, 0} = 1\) and the others equal to zero) still yields \(\Delta \theta \approx \SI{1.8}{\degree}\), a very reasonable number, and the calculation can then be done analytically, since \(E(z) = (1 + z)^{3/2}\) is integrable with the usual techniques.
\end{proof}

Photons which could not have been in causal contact in the HBB model are observed to have the same temperature.

The inflationary solution to this issue is to think that, before the radiation-dominated epoch, the comoving Hubble radius decreased for a certain period of time.

This allows the parts of the sky to have been in causal contact in the early universe.

This means that \(\ddot{a} > 0\) in the inflationary phase (which is equivalent to \(w < - 1/3\)), since we are imposing \(0 > \dot{r} = (\dv{}{t})(1/ \dot{a}) = - \ddot{a} / (\dot{a})^2\).
These are only the \emph{kinematics} of inflation, we are not yet discussing how it might come about.

\end{document}


\chapter{Special Plot Text Capabilities} \label{appx:plt}

There are several special capabilities available for user-defined text
that appears on a plot (such as the plot caption). For example, the text
can include subscripted text, Greek letters, and special symbols. The
user requests a capability by inserting an ``escape sequence'' in the
text. The escape sequence starts with a backslash (``\verb|\|'') and
(unless otherwise noted) must end with a space which is
deleted from the text string. Escape sequences are translated only when
{\bf software} characters are plotted. If an invalid escape sequence is
entered, an error message appears when the plot is drawn and the
sequence is ignored.
The following escape sequences are available:

\begin{center} \begin{tabbing}
\verb|\|\cmd{XXXXXXX} \= explanation of symbol xx \=
\verb|\|\cmd{XXXXXXX} \= explanation of symbol xx \kill
%
\verb|\|\cmd{SQ} \> square \>
\verb|\|\cmd{CSQ} \> centered square \\
\verb|\|\cmd{DI} \> diamond \>
\verb|\|\cmd{CDI} \> centered diamond \\
\verb|\|\cmd{CS} \> cross \>
\verb|\|\cmd{CCS} \> centered cross \\
\verb|\|\cmd{X} \> X \>
\verb|\|\cmd{CX} \> centered X \\
\verb|\|\cmd{TR} \> triangle \>
\verb|\|\cmd{CTR} \> centered triangle \\
\verb|\|\cmd{CI} \> circle \>
\verb|\|\cmd{CCI} \> centered circle \\
\verb|\|\cmd{DO} \> dot \>
\verb|\|\cmd{CDO} \> centered dot \\
\verb|\|\cmd{LO} \> logo \>
\verb|\|\cmd{CLO} \> centered logo \\
\\
\verb|\|\cmd{SLINE} \> solid line \>
\verb|\|\cmd{DLINE} \> dotted line \\
\verb|\|\cmd{DDLINE} \> dot-dash line \>
\verb|\|\cmd{SDLINE} \> short dash line \\
\verb|\|\cmd{LDLINE} \> long dash line \>
\verb|\|\cmd{MDLINE} \> medium dash line \\
\\
\verb|\|\cmd{NEQ} \> not equal to \\
\verb|\|\cmd{LEQ} \> less than or equal to \>
\verb|\|\cmd{GEQ} \> greater than or equal to \\
\verb|\|\cmd{NLEQ} \> not less than or equal to \>
\verb|\|\cmd{NGEQ} \> not greater than or equal to \\
\verb|\|\cmd{NLT} \> not less than \>
\verb|\|\cmd{NGT} \> not greater than \\
\verb|\|\cmd{LL} \> much less than \>
\verb|\|\cmd{GG} \> much greater than \\
\\
\verb|\|\cmd{PLUSMIN} \> plus or minus \>
\verb|\|\cmd{PRIME} \> prime \\
\verb|\|\cmd{SUM} \> summation \>
\verb|\|\cmd{APPROX} \> approximation sign \\
\verb|\\| \> single backslash (with no ending space) \\
\\
\verb|\|\cmd{GR} \> Greek font \>
\verb|\|\cmd{ENG} \> normal font \\
\\
\verb|\^| \> start superscript (with no ending space) \\
\verb|\_| \> start subscript (with no ending space) \\
\verb|\-| \> end superscript or subscript (with no ending space) \\
\end{tabbing} \end{center}
%\\
%\verb|\|\cmd{CR} \> carriage return \>
%\verb|\|\cmd{LF} \> linefeed \\
%\verb|\|\cmd{CL} \> carriage return and linefeed \\


\documentclass{../lib}
\begin{document}

\begin{ledCmnt}
\title{Tic Tac Toe}
\date{}
\author{Qianji Zheng and J. Nelson Rushton, Texas Tech University}
\maketitle

This is an LED program that defines a simple tic tac toe game to run in the Easel game engine.

Code is enclosed between begin-code and end-code delimiters (slash-dollar and dollar-slash, respectively), and all other text is documentary.

The game begins with an empty grid and it being x's turn to move. When an empty cell is clicked, the player whose turn it is occupies that cell, and it becomes the other player's turn, until the game is over. When the game is over, a message is displayed giving the result of the  game. The player can press the restart button at any time to restart the game.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Data Model}

This section is documentary, since the current implementation is untyped.

A {\em player} is either $x$ or $o$. In this program, the variable $p$ will range over players.

A {\em cell} is an integer in $\{1..9\}$. In this program, the variable $c$ will range over cells.

Cells represent squares on the tic tac toe board as pictured below:
\begin{verbatim}
  1|2|3
  -----
  4|5|6
  -----
  7|8|9
\end{verbatim}

A {\em move} is a pair $(p,c)$ where $p$ is a player and $c$ is a cell. The move $(p,c)$ represents a move by player $p$ in cell $c$. In this program, the variable $m$ will range over moves.

A {\em state} is a set of moves, thought of as the set of moves made so far in the game. In this program, the variable $S$ will range over states. $currentState$ is a constant symbol denoting the current state of the game, which could be any state in any game of tic tac toe. (Readers familiar with Gurevich Machines will find this device familiar; others, maybe not so much.)

Tell the translator to produce a SequenceL file that runs with Easel:
\end{ledCmnt}

\begin{ledDef}
ledGame
\end{ledDef}

\begin{ledCmnt}
The following are recognized as constant symbols in the Easel environment:
\begin{enumerate}
\item $currentState$: same type as user-defined $initialState$
\item $mouseClicked$: true if the left mouse button is clicked in the current frame, false otherwise
\item $mouseX$, $mouseY$: integer coordinates of the mouse
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Game Rules}

This section defines the rules of tic tac toe in LED.

Since the state of the game is the set of moves that have been made, the beginning state $initialState$ is the empty set.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{initialState}{\setEmpty}
\end{ledDef}

\begin{ledCmnt}
Player $p$ $occupies$ cell $c$ if the move $(p,c)$ is a member of $currentState$. Cell $c$ is $occupied$ if it is occupied by $x$ or by $o$.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{occupies \parentheses{p, c}}{\setMem{\tpl{p, c}}{currentState}}
\end{ledDef}

\begin{ledDef}
\relDefNoWhere{occupied \parentheses{c}}{\disj{occupies \parentheses{`x, c}}{occupies \parentheses{`o, c}}}
\end{ledDef}

\begin{ledCmnt}
A {\em row} is a set of cells that form three in a row either horizontally, vertically, or diagonally.
\end{ledCmnt}

\begin{ledDef}
\funDefWhere{rows}{\unn{\unn{hRows}{vRows}}{diagonals}}{\conj{\conj{\eq{hRows}{\setNonempty{\setNonempty{1, 2, 3}, \setNonempty{4, 5, 6}, \setNonempty{7, 8, 9}}}}{\eq{vRows}{\setNonempty{\setNonempty{1, 4, 7}, \setNonempty{2, 5, 8}, \setNonempty{3, 6, 9}}}}}{\eq{diagonals}{\setNonempty{\setNonempty{1, 5, 9}, \setNonempty{3, 5, 7}}}}}
\end{ledDef}

\begin{ledCmnt}
$threeInRow(p)$ if player $p$ occupies all of the cells in some row.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{threeInRow \parentheses{p}}{\exist{\symsInSet{R}{rows}}{\univ{\symsInSet{c}{R}}{occupies \parentheses{p, c}}}}
\end{ledDef}

\begin{ledCmnt}
$boardFull$ if all cells are occupied.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{boardFull}{\eq{\pipesOp{currentState}}{9}}
\end{ledDef}

\begin{ledCmnt}
$gameOver$ if either the board is full, or one of the players three in a row.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{gameOver}{\disj{\disj{boardFull}{threeInRow \parentheses{`x}}}{threeInRow \parentheses{`o}}}
\end{ledDef}

\begin{ledCmnt}
$playerToMove$ is $x$ if an even number of moves have been made, and $o$ otherwise.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{playerToMove}{
\begin{cases}
\termIfBoolTerm{`x}{even \parentheses{\pipesOp{currentState}}}
\termOw{`o}
\end{cases}
}
\end{ledDef}

\begin{ledDef}
\relDefNoWhere{even \parentheses{n}}{\eq{\md{n}{2}}{0}}
\end{ledDef}

\begin{ledCmnt}
$legalToMoveIn(c)$ means that it is legal for the player whose turn it is to move in cell $c$ in the current state of the game -- that is, if the game is not over and the cell is not occupied.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{legalToMoveIn \parentheses{c}}{\conj{\neg{occupied \parentheses{c}}}{\neg{gameOver}}}
\end{ledDef}

\begin{ledCmnt}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Video Output}

The default color used in this game is $BLACK$.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{BLACK}{color \parentheses{0, 0, 0}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{WHITE}{color \parentheses{255, 255, 255}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{BLUE}{color \parentheses{0, 0, 255}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{GREEN}{color \parentheses{0, 255, 0}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{RED}{color \parentheses{255, 0, 0}}
\end{ledDef}

\begin{ledCmnt}
TThis section defines the $gridDisplay$ function, which specifies the images to display on the screen in each game state. $gridDisplay$ consists of a set of four line segments that make up the tic tac toe playing area.
\end{ledCmnt}

\begin{ledDef}
\funDefWhere{gridDisplay}{\setNonempty{L1, L2, L3, L4}}{\conj{\conj{\conj{\eq{L1}{segment \parentheses{point \parentheses{200, 700}, point \parentheses{200, 400}, BLACK}}}{\eq{L2}{segment \parentheses{point \parentheses{300, 700}, point \parentheses{300, 400}, BLACK}}}}{\eq{L3}{segment \parentheses{point \parentheses{100, 600}, point \parentheses{400, 600}, BLACK}}}}{\eq{L4}{segment \parentheses{point \parentheses{100, 500}, point \parentheses{400, 500}, BLACK}}}}
\end{ledDef}

\begin{ledCmnt}
The default font size for displayed text in this program is 36.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{fontSize}{36}
\end{ledDef}

\begin{ledCmnt}
$centerX(c)$ and $centerY(c)$ are the $x$ and $y$ coordinates of the center of cell $c$, respectively.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{centerX \parentheses{c}}{\plusOp{150}{\starOp{100}{\md{\binaryMinus{c}{1}}{3}}}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{centerY \parentheses{c}}{\binaryMinus{650}{\starOp{100}{\flr{\div{\binaryMinus{c}{1}}{3}}}}}
\end{ledDef}

\begin{ledCmnt}
$cellDisplay(c)$ is a display of a text character `x' or an `o' in cell $c$, or the empty display, respectively in case cell $c$ is occupied by $x$, occupied by $o$, or not occupied in the current game state.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{xImage \parentheses{c}}{text \parentheses{\texttt{"x"}, point \parentheses{centerX \parentheses{c}, centerY \parentheses{c}}, fontSize, BLUE}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{oImage \parentheses{c}}{text \parentheses{\texttt{"o"}, point \parentheses{centerX \parentheses{c}, centerY \parentheses{c}}, fontSize, GREEN}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{cellDisplay \parentheses{c}}{
\begin{cases}
\termIfBoolTerm{\setNonempty{xImage \parentheses{c}}}{\setMem{\tpl{`x, c}}{currentState}}
\termIfBoolTerm{\setNonempty{oImage \parentheses{c}}}{\setMem{\tpl{`o, c}}{currentState}}
\termOw{\setEmpty}
\end{cases}
}
\end{ledDef}

\begin{ledCmnt}
$cellDisplays$ is the set of all images of $x$'s and $o$'s on the board in the current state.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{gameBoard}{\iv{1}{9}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{cellDisplays}{\aggrUnn{\setMem{c}{gameBoard}}{cellDisplay \parentheses{c}}}
\end{ledDef}

\begin{ledCmnt}
If the game is not over, $currentPlayerDisplay$ is a text in the upper left hand region of the game window indicating the player to move, either ``play x's turn'' or ``play o's turn''.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{currentPlayerDisplay}{
\begin{cases}
\termIfBoolTerm{\setNonempty{text \parentheses{\texttt{"x's turn"}, point \parentheses{100, 750}, fontSize, BLACK}}}{\eq{playerToMove}{`x}}
\termOw{\setNonempty{text \parentheses{\texttt{"o's turn"}, point \parentheses{100, 750}, fontSize, BLACK}}}
\end{cases}
}
\end{ledDef}

\begin{ledCmnt}
The {\em restart button} consists of a rectangle around a ``restart'' text, displayed in the upper right region of the screen. Formally, it is a set of four line segments and a text.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{restartLeft}{350}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartRight}{550}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartBottom}{725}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartTop}{775}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartBottomLeftPoint}{point \parentheses{restartLeft, restartBottom}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartBottomRightPoint}{point \parentheses{restartRight, restartBottom}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartTopLeftPoint}{point \parentheses{restartLeft, restartTop}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartTopRightPoint}{point \parentheses{restartRight, restartTop}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{mid \parentheses{a, b}}{\div{\plusOp{a}{b}}{2}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartMidX}{mid \parentheses{restartLeft, restartRight}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{restartMidY}{mid \parentheses{restartBottom, restartTop}}
\end{ledDef}

\begin{ledDef}
\funDefWhere{restartButton}{\setNonempty{A1, A2, A3, A4, txt}}{\conj{\conj{\conj{\conj{\eq{A1}{segment \parentheses{restartBottomLeftPoint, restartBottomRightPoint, BLACK}}}{\eq{A2}{segment \parentheses{restartTopLeftPoint, restartTopRightPoint, BLACK}}}}{\eq{A3}{segment \parentheses{restartBottomLeftPoint, restartTopLeftPoint, BLACK}}}}{\eq{A4}{segment \parentheses{restartBottomRightPoint, restartTopRightPoint, BLACK}}}}{\eq{txt}{text \parentheses{\texttt{"restart"}, point \parentheses{restartMidX, restartMidY}, fontSize, BLACK}}}}
\end{ledDef}

\begin{ledCmnt}
If the game is over, $gameResultDisplay$ is a display in the upper left region of the screen telling the outcome of the game, either ``x won'', ``o won'', or ``cat got it''.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{gameResultDisplay}{
\begin{cases}
\termIfBoolTerm{\setNonempty{text \parentheses{\texttt{"x won"}, point \parentheses{200, 750}, fontSize, BLUE}}}{threeInRow \parentheses{`x}}
\termIfBoolTerm{\setNonempty{text \parentheses{\texttt{"o won"}, point \parentheses{200, 750}, fontSize, GREEN}}}{threeInRow \parentheses{`o}}
\termOw{\setNonempty{text \parentheses{\texttt{"cat got it"}, point \parentheses{200, 750}, fontSize, RED}}}
\end{cases}
}
\end{ledDef}

\begin{ledCmnt}
The {\em display} is the screen display for the current game state. The grid, cell displays, and restart button are always displayed. Additionally, the display includes the game results if the game is over, and the player to move if the game is not over.
\end{ledCmnt}

\begin{ledDef}
\funDefWhere{images}{
\begin{cases}
\termIfBoolTerm{gameOverDisplay}{gameOver}
\termOw{inPlayDisplay}
\end{cases}
}{\conj{\conj{\eq{alwaysDisplay}{\unn{\unn{gridDisplay}{cellDisplays}}{restartButton}}}{\eq{inPlayDisplay}{\unn{alwaysDisplay}{currentPlayerDisplay}}}}{\eq{gameOverDisplay}{\unn{alwaysDisplay}{gameResultDisplay}}}}
\end{ledDef}

\begin{ledCmnt}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Mouse Input}

This section defines {\em update}, which specifies the program's response to mouse input. The variable $pt$ will vary over points.

$xMin(c)$, $xMax(c)$, $yMin(c)$, and $yMax(c)$ denote the graphical boundaries of cell $c$, in the obvious manner.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{xMin \parentheses{c}}{\plusOp{100}{\starOp{100}{\md{\binaryMinus{c}{1}}{3}}}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{xMax \parentheses{c}}{\plusOp{200}{\starOp{100}{\md{\binaryMinus{c}{1}}{3}}}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{yMin \parentheses{c}}{\binaryMinus{600}{\starOp{100}{\flr{\div{\binaryMinus{c}{1}}{3}}}}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{yMax \parentheses{c}}{\binaryMinus{700}{\starOp{100}{\flr{\div{\binaryMinus{c}{1}}{3}}}}}
\end{ledDef}

\begin{ledCmnt}
$cellClicked(c)$ means that cell $c$ has been clicked.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{cellClicked \parentheses{c}}{\conj{\conj{\conj{\conj{mouseClicked}{\greater{mouseX}{xMin \parentheses{c}}}}{\less{mouseX}{xMax \parentheses{c}}}}{\greater{mouseY}{yMin \parentheses{c}}}}{\less{mouseY}{yMax \parentheses{c}}}}
\end{ledDef}

\begin{ledCmnt}
$restartClicked$ means that the most recent mouse click is inside the region of the play again button.
\end{ledCmnt}

\begin{ledDef}
\funDefNoWhere{restartClicked}{\conj{\conj{\conj{\conj{mouseClicked}{\greater{mouseX}{restartLeft}}}{\less{mouseX}{restartRight}}}{\greater{mouseY}{restartBottom}}}{\less{mouseY}{restartTop}}}
\end{ledDef}

\begin{ledCmnt}
$moveMade(c)$ means that cell c has been clicked and the player to move may legally move there.
\end{ledCmnt}

\begin{ledDef}
\relDefNoWhere{moveMadeIn \parentheses{c}}{\conj{cellClicked \parentheses{c}}{legalToMoveIn \parentheses{c}}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{movesMade}{\setCompr{\tpl{playerToMove, c}}{\conj{\setMem{c}{gameBoard}}{moveMadeIn \parentheses{c}}}}
\end{ledDef}

\begin{ledDef}
\funDefNoWhere{newState}{
\begin{cases}
\termIfBoolTerm{initialState}{restartClicked}
\termOw{\unn{currentState}{movesMade}}
\end{cases}
}
\end{ledDef}

\begin{ledCmnt}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Test cases}

This section contains test cases for the SequenceL interpreter.

\begin{verbatim}
cmd:>pp(initialState_)
"{}"
cmd:>pp(rows)
"{{1, 2, 3}, {4, 5, 6}, {7, 8, 9}, {1, 4, 7}, {2, 5, 8}, {3, 6, 9}, {1, 5, 9}, {3, 5, 7}}"
cmd:>pp(BLACK)
"an Easel color"
cmd:>pp(WHITE)
"an Easel color"
cmd:>pp(BLUE)
"an Easel color"
cmd:>pp(GREEN)
"an Easel color"
cmd:>pp(RED)
"an Easel color"
cmd:>pp(gridDisplay)
"{an Easel image, an Easel image, an Easel image, an Easel image}"
cmd:>pp(fontSize)
"36"
cmd:>pp(gameBoard)
"{1, 2, 3, 4, 5, 6, 7, 8, 9}"
cmd:>pp(restartLeft)
"350"
cmd:>pp(restartRight)
"550"
cmd:>pp(restartBottom)
"725"
cmd:>pp(restartTop)
"775"
cmd:>pp(restartBottomLeftPoint)
"an Easel point"
cmd:>pp(restartBottomRightPoint)
"an Easel point"
cmd:>pp(restartTopLeftPoint)
"an Easel point"
cmd:>pp(restartTopRightPoint)
"an Easel point"
cmd:>pp(restartMidX)
"450"
cmd:>pp(restartMidY)
"750"
cmd:>pp(restartButton)
"{an Easel image, an Easel image, an Easel image, an Easel image, an Easel image}"
\end{verbatim}

LED engine took: 20 secs.
\end{ledCmnt}

\end{document}


\documentclass[main.tex]{subfiles}
\begin{document}

\section{The Kerr solution}

\marginpar{Wednesday\\ 2020-10-14, \\ compiled \\ \today}

Some history: the Schwarzschild was found in 1915 while S.\ was serving in the army; the second exact solution was found in 1963 by a PhD student in New Zeland \cite[]{kerrGravitationalFieldSpinning1963}.
In Cartesian coordinates the metric is hideous; nowadays we use the coordinates defined by Boyer and Lyndquist: starting from a flat set of Cartesian coordinates \(x, y, z\) we define \(r, \theta , \varphi \) as
%
\begin{align}
x &= \sqrt{r^2 + a^2} \sin \theta \sin \varphi   \\
y &= \sqrt{r^2 + a^2} \sin \theta \cos \varphi  \\
z &= r \cos \theta
\,.
\end{align}

These are then not \emph{spherical} but \emph{spheroidal} coordinates.
Constant-\(r\) spheroids are oblate ellipsoids in the \(z\) direction.

The Kerr solution, for who wants to see the computation, is derived in ``The mathematical theory of Black Holes'' by Chandrasekhar \cite[]{chandrasekharMathematicalTheoryBlack1998}.
The metric reads
%
\begin{align}
\dd{s^2} = - \qty(1 - \frac{2Mr}{\Sigma }) \dd{t^2}
+ \frac{\Sigma }{\Delta } \dd{r^2}
+ \Sigma \dd{\theta ^2}
+ \qty(r^2 + a^2 + \frac{aA}{\Sigma }) \sin^2 \theta \dd{\varphi^2}
- \frac{2A}{\Sigma } \dd{t} \dd{\varphi }
\,,
\end{align}
%
where:
%
\begin{align}
\Delta = r^2 - 2Mr + a^2
&&
\Sigma = r^2 + a^2 \cos^2 \theta
&&
A = 2Mar \sin^2 \theta
\,,
\end{align}
%
while \(a = J / M\) is the \emph{specific angular momentum} of the black hole.

It describes the spacetime outside an axially symmetric, stationary body.
In the Schwarzschild spacetime we have identified an horizon at \(r = 2M\) by the properties that \(g_{00} \to 0\) and \(g_{rr} \to \infty \).

What about Kerr? Are there horizons? In this case, the two conditions do not happen in the same place.
The region \(g_{00} = 0\) is known as the \emph{limit of staticity}: if it is the case, an observer's worldline cannot be both \emph{timelike} and \emph{stationary} (meaning time-directed).

The condition \(g_{00} > 0\) is equivalent to \(\Sigma < 2Mr\).
Now, consider the \(\dd{r^2}\) coefficient: \(\Sigma / \Delta \): this is positive always.

If we are in the region \(\theta = \pi /2\), at \(g_{00} > 0\) we can still have the metric's signature be \(- +++\), since we have the \(\dd{t} \dd{\varphi }\) term.

As long as \(a \dd{\varphi } > 0\), that term in the metric is negative, allowing the signature of the metric to be preserved.

This means that \textbf{the particle must be co-rotating} with the black hole if it is in the region \(g_{00} > 0\).

As long as this is the case, however, a particle can remain at fixed \(r\) and even escape. This is, then, \emph{not} an horizon.

The horizon, instead, is found when \(g_{rr}\) diverges: this is equivalent to \(\Delta \to 0\), which means
%
\begin{align}
r^2 - 2Mr + a^2 = 0 \implies r = M \pm \sqrt{M^2 - a^2}
\,.
\end{align}

Both of these radii correspond to an horizon.
Let us denote \(r_+ = r_H\), since the inner horizon cannot really affect any observations.

In order for these two solutions to be real, we must have \(a < M\).
There is an horizon as long as \(a/ M < 1\).

If \(a > M\), we have a \textbf{naked singularity}, since the singularity at \(\Sigma  = 0\) is still there.
This singularity, in any case, is not shaped like a point: we only reach it along the equatorial plane.

Penrose proposed the cosmic \textbf{censorship hypothesis}: the universe is a ``prude'', it always hides singularities with horizons.
There are good theoretical reasons to believe that this is verified.

Where is the limit of staticity? The equation is
%
\begin{align}
r^2 + a^2 \cos^2 \theta - 2Mr = 0
\,,
\end{align}
%
which  is solved by
%
\begin{align}
r_{\pm} = M \pm \sqrt{M^2- a^2 \cos^2 \theta }
\,.
\end{align}

There are two of these surfaces as well, and we consider the outer one as before: \(r_E = r_+\). If the horizon exists, then this region also exists.

This region is called the \textbf{ergosphere}, and the region between \(r_H < r < r_E\) is called the \textbf{ergoregion}.

See \textcite[sec. 3]{heinickeSchwarzschildKerrSolutions2015} for an in-depth discussion of the shape of these regions; figure 12 there also shows them graphically.

The name comes from the fact that we can extract rotational energy from the BH.
``Ergo'' means energy.

There has been a long debate about whether the Penrose process actually occurs in a realistic astrophysical setting: the consensus is that the trajectory a particle must take in order for this to happen is way too peculiar.

Note that in this case we also have the cyclic coordinates \(\varphi \) and \(t\). We have two constants of motion like in Schwarzschild.

It can be shown that there exists a third constant of motion, beyond \(E\) and \(L_z\): \(Q\), called Carter's constant.

For motion in the equatorial plane we can write the following expression for the energy integral:
%
\begin{align}
E = \frac{r^{3/2} - 2 r^{1/2} \pm a M^{1/2}}{r^{3/2} \qty(r^{3/2}- 3M r^{1/2} \pm 2a M^{1/2})^{1/2}}
\,,
\end{align}
%
where \(\pm\) refers to whether the particle moves along a prograde or retrograde trajectory.

The expression for the last stable (LS, or sometimes MS for ``marginally stable'') circular orbit is \cite[eq.
\ 28]{puglieseEquatorialCircularMotion2011}:
%
\begin{align}
r_{LS} = M \qty[ 3 + z_2 \mp \qty[(3 - z_1 ) (3 + z_1 +2z_2 )]^{1/2}]
\,,
\end{align}
%
where
%
\begin{align}
z_1 &= 1 + \qty(1 - \frac{a^2}{M^2})^{1/3} \qty[\qty(1 + \frac{a}{M})^{1/3} + \qty(1 - \frac{a}{M})^{1/3}]  \\
z_2 &= \qty(3 \frac{a^2}{M^2} + z_1^2)^{1/2}
\,.
\end{align}

This reduces to \(r_{LS} = 6 M\) in the \(a = 0\) case, since then we have \(z_1 = z_2 = 3\).

What happens for an extreme Kerr BH, with \(a = M\)?
Then \(z_1 = 1\), \(z_2 = 2\): so,
%
\begin{align}
r_{LS} = M \qty[3 + 2 \mp \sqrt{2 \times 8}] = M \qty[5 \mp 4]
\,,
\end{align}
%
which yields \(r_{LS} = M\) in the corotating case, and \(r_{LS} = 9M\) in the counter-rotating case.

We expect that the efficiency of a Kerr BH in the extraction of energy from matter will be higher than the Schwarzschild solution.
Let us use the expression we found for the specific energy \(E\), with respect to the parameter \(x = r/M\):
%
\begin{align}
E = \frac{x^{3/2} - 2 x^{1/2} \pm a/M}{x^{3/2} \sqrt{x^{3/2} - 3x^{1/2} \pm 2a /M}}
\,,
\end{align}
%
which in the extreme case becomes
%
\begin{align}
E = \frac{x^{3/2} - 2 x^{1/2} \pm 1}{x^{3/2} \sqrt{x^{3/2 }- 3 x^{1/2} \pm 2}}
\,,
\end{align}
%
so we can take the limit: in the corotating direct case, sending \(x \to 1\) we get \(E = 1 / \sqrt{3} \approx \num{.577}\).

The efficiency is given by
%
\begin{align}
\eta = \frac{E_\infty  - E}{E_{\infty }} = 1 - 1 / \sqrt{3} \approx \SI{42}{\percent}
\,.
\end{align}

This is a huge amount of energy: we do not expect real black holes to be extreme.
There are estimates of the spins of the black holes.
What is found is that they seem to cover the whole range \(0 < a/ M < 1\).

An issue regarding a wide-spread misconception: the parameter \(M\) in the interior Schwarzschild solution is the same as the \(M\) in the corresponding \emph{exterior} solution. Can we apply the same kind of reasoning for Kerr?
Is there an interior Kerr solution?
We don't know, but most probably not.
Many efforts were put into seeking it, and they all failed.
The properties of the matter and radiation inside are weird.

The wide-spread misconception is to claim that the Kerr spacetime describes the spacetime around a rotating star.
It sounds reasonable, but it's wrong.
Numerically we can derive the true form of the spacetime outside something like a neutron star: it is very different from the Kerr spacetime.

A qualitative argument: a star will generally have a quadrupole moment and emit GWs, while Kerr does not.
Kerr is a Petrov-type-D spacetime, which is \emph{nonradiating}.
If we do the calculation, we find that the quadrupole moment of Kerr is \(Q = J / M\).

For an in-depth discussion see \textcite[]{bertiRotatingNeutronStars2005}, which refers to \textcite[]{bakerMakingUseGeometrical2000} for its tools to measure ``non-type-D-ness''.

The Petrov classification refers to the symmetries of the Weyl tensor --- the ``deviatoric'' part of the Riemann tensor, which describes deformation but not expansion/compression.
Kerr spacetime can be analytically shown to be type D, which means its Weyl tensor is ``highly symmetric'', and the consequence of this is that the radiation it can emit decays like \(r^{-3}\), like the tidal tensor of regular Newtonian gravity.
This prevents the emission of long-range \(r^{-1}\) gravitational radiation, which we know NSs to emit.
% (see, for example, the Hulse-Taylor pulsar).

% About the differences between the Kerr spacetime and the spacetime outside a rotating star: the spacetime outside a rotating body is indeed \emph{not} Kerr.

% Kerr is Petrov type D, which means nonradiative: we would need to compute the Weyl invariants of the metric.
% If we compute the invariant telling us whether a spacetime is radiative or not, we find that it is nonradiative iff \(Q = J / M\).
% The Hartle-Thorne approximation allows us to work up to a certain value of \(\Omega / \Omega _k\), close to the mass shedding limit.

% The thing we find is that in general for a star the quadrupole is \(Q \neq Q _{\text{Kerr}}\), however for small values of the rotation the spacetimes converge.

% A reference for this: \textcite{bertiRotatingNeutronStars2005}.

% To leading order, deviations from type-D are driven by deviations of \(Q \) from \(Q _{\text{Kerr}}\).

% Next time, we will discuss Equations of state and degenerate gasses.

\end{document}


\section{Even and Odd Operators}\index{Even operator}\index{Odd operator}
An operator can be declared to be {\em even\/} or {\em odd\/} in its first
argument by the declarations {\tt EVEN}\ttindex{EVEN} and
{\tt ODD}\ttindex{ODD} respectively.  Expressions involving an operator
declared in this manner are transformed if the first argument contains a
minus sign.  Any other arguments are not affected.  In addition, if say
{\tt F} is declared odd, then {\tt f(0)} is replaced by zero unless
{\tt F} is also declared {\em non zero\/} by the declaration
{\tt NONZERO}\ttindex{NONZERO}.  For example, the declarations
\begin{verbatim}
	even f1; odd f2;
\end{verbatim}
mean that
\begin{verbatim}
	f1(-a)    ->    F1(A)
	f2(-a)    ->   -F2(A)
	f1(-a,-b) ->    F1(A,-B)
	f2(0)     ->    0.
\end{verbatim}
To inhibit the last transformation, say {\tt nonzero f2;}.

\section{Linear Operators}\index{Linear operator}
An operator can be declared to be linear in its first argument over powers
of its second argument.  If an operator {\tt F} is so declared, {\tt F} of
any sum is broken up into sums of {\tt F}s, and any factors that are not
powers of the variable are taken outside.  This means that {\tt F} must
have (at least) two arguments.  In addition, the second argument must be
an identifier (or more generally a kernel), not an expression.

{\it Example:}

If {\tt F} were declared linear, then
\begin{verbatim}
                                5
        f(a*x^5+b*x+c,x) ->  F(X ,X)*A + F(X,X)*B + F(1,X)*C
\end{verbatim}
More precisely, not only will the variable and its powers remain within the
scope of the {\tt F} operator, but so will any variable and its powers that
had been declared to {\tt DEPEND} on the prescribed variable; and so would
any expression that contains that variable or a dependent variable on any
level, e.g. {\tt cos(sin(x))}.

To declare operators {\tt F} and {\tt G} to be linear operators,
use:\ttindex{LINEAR}
\begin{verbatim}
        linear f,g;
\end{verbatim}
The analysis is done of the first argument with respect to the second; any
other arguments are ignored. It uses the following rules of evaluation:
\begin{quote}
\begin{tabbing}
{\tt    f(0)      ->   0} \\
{\tt    f(-y,x)   ->  -F(Y,X)} \\
{\tt    f(y+z,x)  ->   F(Y,X)+F(Z,X)} \\
{\tt    f(y*z,x)  ->   Z*F(Y,X)} \hspace{0.5in}\= if Z does not depend on X \\
{\tt    f(y/z,x)  ->   F(Y,X)/Z} \> if Z does not depend on X
\end{tabbing}
\end{quote}
To summarize, {\tt Y} ``depends'' on the indeterminate {\tt X} in the above
if either of the following hold:
\begin{enumerate}
\item {\tt Y} is an expression that contains {\tt X} at any level as a
      variable, e.g.: {\tt cos(sin(x))}

\item Any variable in the expression {\tt Y} has been declared dependent on
      {\tt X} by use of the declaration {\tt DEPEND}.
\end{enumerate}
The use of such linear operators\index{Linear operator} can be seen in the
paper Fox, J.A. and A. C. Hearn, ``Analytic Computation of Some Integrals
in Fourth Order Quantum Electrodynamics'' Journ. Comp. Phys. 14 (1974)
301-317, which contains a complete listing of a program for definite
integration\index{Integration} of some expressions that arise in fourth
order quantum electrodynamics.

\section{Non-Commuting Operators}\index{Non-commuting operator}
An operator can be declared to be non-commutative under multiplication by
the declaration {\tt NONCOM}.\ttindex{NONCOM}

{\it Example:}

After the declaration \\
{\tt noncom u,v;}\\
the expressions {\tt
u(x)*u(y)-u(y)*u(x)} and {\tt u(x)*v(y)-v(y)*u(x)} will remain unchanged
on simplification, and in particular will not simplify to zero.

Note that it is the operator ({\tt U} and {\tt V} in the above example)
and not the variable that has the non-commutative property.

The {\tt LET}\ttindex{LET} statement may be used to introduce rules of
evaluation for such operators.  In particular, the boolean operator
{\tt ORDP}\ttindex{ORDP} is useful for introducing an ordering on such
expressions.

{\it Example:}

The rule
\begin{verbatim}
        for all x,y such that x neq y and ordp(x,y)
           let u(x)*u(y)= u(y)*u(x)+comm(x,y);
\end{verbatim}
would introduce the commutator of {\tt u(x)} and {\tt u(y)} for all
{\tt X} and {\tt Y}.  Note that since {\tt ordp(x,x)} is {\em true}, the
equality check is necessary in the degenerate case to avoid a circular
loop in the rule.

\section{Symmetric and Antisymmetric Operators}

An operator can be declared to be symmetric with respect to its arguments
by the declaration {\tt SYMMETRIC}.\ttindex{SYMMETRIC} For example
\begin{verbatim}
        symmetric u,v;
\end{verbatim}
means that any expression involving the top level operators {\tt U} or
{\tt V} will have its arguments reordered to conform to the internal order
used by {\REDUCE}.  The user can change this order for kernels by the
command {\tt KORDER}.

For example, {\tt u(x,v(1,2))} would become {\tt u(v(2,1),x)}, since
numbers are ordered in decreasing order, and expressions are ordered in
decreasing order of complexity.

Similarly the declaration {\tt ANTISYMMETRIC}\ttindex{ANTISYMMETRIC}
declares an operator antisymmetric.   For example,
\begin{verbatim}
        antisymmetric l,m;
\end{verbatim}
means that any expression involving the top level operators {\tt L} or
{\tt M} will have its arguments reordered to conform to the internal order
of the system, and the sign of the expression changed if there are an odd
number of argument interchanges necessary to bring about the new order.

For example, {\tt l(x,m(1,2))} would become {\tt -l(-m(2,1),x)} since one
interchange occurs with each operator.  An expression like {\tt l(x,x)}
would also be replaced by 0.

\section{Declaring New Prefix Operators}

The user may add new prefix\index{Prefix} operators to the system by
using the declaration {\tt OPERATOR}. For example:
\begin{verbatim}
        operator h,g1,arctan;
\end{verbatim}
adds the prefix operators {\tt H}, {\tt G1} and {\tt ARCTAN} to the system.

This allows symbols like {\tt h(w), h(x,y,z), g1(p+q), arctan(u/v)} to be
used in expressions, but no meaning or properties of the operator are
implied.  The same operator symbol can be used equally well as a 0-, 1-, 2-,
3-, etc.-place operator.

To give a meaning to an operator symbol, or express some of its
properties, {\tt LET}\ttindex{LET} statements can be used, or the operator
can be given a definition as a procedure.

If the user forgets to declare an identifier as an operator, the system
will prompt the user to do so in interactive mode, or do it automatically
in non-interactive mode. A diagnostic message will also be printed if an
identifier is declared {\tt OPERATOR} more than once.

Operators once declared are global in scope, and so can then be referenced
anywhere in the program.  In other words, a declaration within a block (or
a procedure) does not limit the scope of the operator to that block, nor
does the operator go away on exiting the block (use {\tt CLEAR} instead
for this purpose).

\section{Declaring New Infix Operators}

Users can add new infix operators by using the declarations
{\tt INFIX}\ttindex{INFIX} and {\tt PRECEDENCE}.\ttindex{PRECEDENCE}
For example,
\begin{verbatim}
        infix mm;
        precedence mm,-;
\end{verbatim}
The declaration {\tt infix mm;} would allow one to use the symbol
{\tt MM} as an infix operator:
\begin{quote}
\hspace{0.2in} {\tt a mm b} \hspace{0.3in} instead of \hspace{0.3in}
{\tt mm(a,b)}.
\end{quote}

The declaration {\tt precedence mm,-;} says that {\tt MM} should be
inserted into the infix operator precedence list just {\em after\/}
the $-$ operator.  This gives it higher precedence than $-$ and lower
precedence than * .  Thus

\begin{quote}
\hspace{0.2in}{\tt a - b mm c - d}\hspace{.3in} means \hspace{.3in}
{\tt a - (b mm c) - d},
\end{quote}
while
\begin{quote}
\hspace{0.2in}{\tt   a * b mm c * d}\hspace{.3in} means \hspace{.3in}
{\tt (a * b) mm (c * d)}.
\end{quote}

Both infix and prefix\index{Prefix} operators have no transformation
properties unless {\tt LET}\ttindex{LET} statements or procedure
declarations are used to assign a meaning.

We should note here that infix operators so defined are always binary:
\begin{quote}
\hspace{0.2in}{\tt a mm b mm c}\hspace{.3in} means \hspace{.3in}
{\tt (a mm b) mm c}.
\end{quote}

\section{Creating/Removing Variable Dependency}

There are several facilities in {\REDUCE}, such as the differentiation
\index{Differentiation}
operator and the linear operator\index{Linear operator} facility, that
can utilize knowledge of the dependency between various variables, or
kernels.  Such dependency may be expressed by the command {\tt
DEPEND}.\ttindex{DEPEND} This takes an arbitrary number of arguments and
sets up a dependency of the first argument on the remaining arguments.
For example,
\begin{verbatim}
        depend x,y,z;
\end{verbatim}
says that {\tt X} is dependent on both {\tt Y} and {\tt Z}.
\begin{verbatim}
        depend z,cos(x),y;
\end{verbatim}
says that {\tt Z} is dependent on {\tt COS(X)} and {\tt Y}.

Dependencies introduced by {\tt DEPEND} can be removed by {\tt NODEPEND}.
\ttindex{NODEPEND} The arguments of this are the same as for {\tt DEPEND}.
For example, given the above dependencies,
\begin{verbatim}
        nodepend z,cos(x);
\end{verbatim}
says that {\tt Z} is no longer dependent on {\tt COS(X)}, although it remains
dependent on {\tt Y}.


\section{Karatsuba}

\inputminted{cpp}{code/Karatsuba.cpp}


\documentclass[11pt]{article}
\usepackage{fancyvrb,amsmath,amsfonts,amssymb,graphicx,parskip,listings}
\usepackage[usenames,dvipsnames,svgnames,table]{xcolor}
\usepackage{tikz}

% change margins
\addtolength{\oddsidemargin}{-.875in}
\addtolength{\evensidemargin}{-.875in}
\addtolength{\textwidth}{1.75in}
\addtolength{\topmargin}{-.875in}
\addtolength{\textheight}{1.75in}

\hyphenpenalty=10000

\begin{document}

\begin{center}
{\LARGE Eigenmath Manual}

George Weigt

December 19, 2019
\end{center}

\tableofcontents

\newpage

\input{nabokov}

\input{arithmetic}

\input{exponents}

\input{symbols}

\input{scripting}

\input{draw}

\input{complex}

\input{linear-algebra}

\input{derivative}

\input{integrals}

\input{integral-trick}

\input{fund-thm-of-calculus}

\input{line-integral}

\input{surface-area}

\input{surface-integral}

\input{greens-theorem}

\input{stokes-theorem}

%%%%%

\section{Examples}

\input{francois-viete}

\input{curl-in-tensor-form}

\input{qho}

\input{hydrogen-wavefunctions}

\input{space-shuttle-and-corvette}

\input{avogadro}

\input{zerozero}

\subsection{Euler's identity}
It is easy to ``believe'' that $e^{i\pi}=-1$ by looking at Taylor series expansions.

First, consider the Taylor series expansion of $e^y$.
\[
e^y=1+y+\frac{y^2}{2!}+\frac{y^3}{3!}+\frac{y^4}{4!}
+\frac{y^5}{5!}+\frac{y^6}{6!}+\frac{y^7}{7!}+\cdots
\]
Next, substitute $ix$ for $y$.
\begin{align*}
e^{ix}&=1+ix+\frac{(ix)^2}{2!}+\frac{(ix)^3}{3!}+\frac{(ix)^4}{4!}
+\frac{(ix)^5}{5!}+\frac{(ix)^6}{6!}+\frac{(ix)^7}{7!}+\cdots\\
&=1+ix-\frac{x^2}{2!}-i\frac{x^3}{3!}+\frac{x^4}{4!}
+i\frac{x^5}{5!}-\frac{x^6}{6!}-i\frac{x^7}{7!}+\cdots
\end{align*}
Next, collect the real and imaginary terms.
\begin{align*}
e^{ix}&=\left(1-\frac{x^2}{2!}+\frac{x^4}{4!}-\frac{x^6}{6!}+\cdots\right)
+i\left(x-\frac{x^3}{3!}+\frac{x^5}{5!}-\frac{x^7}{7!}+\cdots\right)\\
&=\cos x+i\sin x
\end{align*}
Finally, substitute $\pi$ for $x$.
\[
e^{i\pi}=\cos\pi+i\sin\pi=-1
\]
The following script checks the identity $e^{ix}=\cos x+i\sin x$ for order $n$.
\begin{Verbatim}[formatcom=\color{blue}]
n = 7
E = taylor(e^y,y,n)
E = eval(E,y,i*x)
C = taylor(cos(x),x,n)
S = taylor(sin(x),x,n)
test(E=C+i*S,"true","false")
\end{Verbatim}

\newpage

\input{list-of-functions}

\newpage

\input{syntax}

\newpage

\input{tricks}

\end{document}


\paragraph{Answer 2.}

When answering these questions, it is important to keep in mind that
the language of words made up on the alphabet~\(\Sigma\)
is~\(\Sigma^{*}\) and that there are, in general, several regular
expressions describing one language.

\begin{enumerate}

  \item The constraint on the words is that they must be of the shape
    \(a \ldots a\) where the dots stand for `any combination of \(a\)
    and \(b\).' In other words, one answer is \(a \lparen a \;
    \disjM{} \, b\rparen\kleeneM \, a \, \disjM \, a\).

  \item This question is very simple since the language of all words
    is \(\lparen a \; \disjM{} \, b\rparen\kleeneM\), we have to
    remove \(\epsilon\), \emph{i.e.,} one simple answer is \(\lparen a \;
    \disjM{} \, b\rparen\plusM\).

  \item The question implies that the words we are looking for are of
    the form \(\ldots a \, \_ \, \_\) where the dots stand for `any
    sequence of \(a\) and \(b\)' and each `\_' stands for a regular
    expression denoting any letter. Any letter is described
    by \(\lparen a \, \disjM{} \, b\rparen\); therefore one possible
    answer is \(\lparen a \, \disjM{} \, b\rparen\kleeneM{}
    a \, \lparen a \, \disjM{} \, b\rparen \, \lparen a \, \disjM{} \,
    b\rparen\).

  \item The words we search contain, at any place, exactly three
    \(a\), so are of the form \(\ldots a \ldots a \ldots a \ldots\),
    where the dots stand for `any letter except \(a\)', \emph{i.e.,} `any
    number of \(b\).' In other words: \(b\kleeneM a b\kleeneM a
    b\kleeneM a b\kleeneM\).

  \item Because the alphabet contains only two letters, the question
    is equivalent to: 'All words containing the substring
    ab', \emph{i.e.,} the words are of the form \(\ldots ab \ldots\)
    where the dots stand for `any sequence of \(a\) and \(b\).' It is
    then easy to understand that a short answer is \(\lparen
    a \, \disjM{} \, b\rparen\kleeneM ab \lparen a \, \disjM{} \,
    b\rparen\kleeneM\).

  \item Because the alphabet is made only of two letters, the answer
    is easy: we put first all the \(a\) and then all the \(b\):
    \(a\kleeneM b\kleeneM\).

  \item Since the alphabet contains only two letters, the only way to
    not repeat a letter is to only have substrings \(ab\) or \(ba\) in
    the words we look for. In other words: \(abab\ldots ab\) or
    \(abab\ldots aba\) or \(baba\ldots ba\) or \(baba\ldots bab\). In
    short: \(\lparen ab \rparen\kleeneM a\opt \, \disjM{} \, \lparen
    ba \rparen\kleeneM b\opt\) or, even shorter: \(a\opt \lparen ba
    \rparen\kleeneM b\opt\).

\end{enumerate}


\documentclass[11pt,a4paper]{report}
\usepackage[margin=1in]{geometry}

\begin{document}
\pagestyle{empty}
\begin{center}
\Large{CHAPTER 2 SUMMARY. \textbf{Motion in One Dimension}}

\large{Justin Yang}

October 16, 2012
\end{center}

\textbf{Kinematics} Description of motion

\section*{Displacement, Velocity, and Speed}

An object moving from intial position $x_{i}$ to final position $x_{f}$ has \textbf{displacement} $$\Delta{x}=x_{f}-x_{i}.$$

\noindent
The \textbf{average velocity} of the object is the ratio of the displacement to the time it takes for the displacement $\Delta{t}=t_{f}-t_{i}$,
$$v_{av}=\frac{\Delta{x}}{\Delta{t}}.$$

\noindent
The \textbf{average speed} of the object is the ratio of the distance traveled to the time it takes, $$\bar{v}=\frac{\Delta{s}}{\Delta{t}}.$$

\noindent
The average velocity and the average speed of an object are very different.

\noindent
\textit{Geometric Interpretation}: The average velocity is the slope of the straight line connecting the points $\left(t_{1}, x_{1}\right)$ and $\left(t_{2}, x_{2}\right)$ in the $x$-versus-$t$ plot.

\noindent
\textbf{Instantaneous velocity} is defined as $$v\left(t\right)=\lim_{\Delta{t} \to 0} \frac{\Delta{x}}{\Delta{t}}.$$

\noindent
The \textbf{instantaneous speed} is the magnitude of the instantaneous velocity.

\section*{Acceleration}

The rate of change of the instantaneous velocity with respect to time.

The \textbf{average acceleration} of an object is the ratio of the \textit{change} in velocity to the time it takes for the change $\Delta{t}=t_{f}-t_{i}$, $$a_{av}=\frac{\Delta{v}}{\Delta{t}}.$$

The \textbf{instantaneous acceleration} is the slope of the line tangent to the $v$-versus-$t$ curve. $$a=\lim_{\Delta{t} \to 0} \frac{\Delta{v}}{\Delta{t}}.$$

\subsection*{Motion with Constant Acceleration}

The motion of a particle that has constant acceleration is common in nature. When air resistance is negligeble, the free fall of an object near Earth's surface has acceleration $g=9.81 m/s^{2}$, which Galileo was the first to conclude.

\noindent
We can use the ``Big Five'' under constant acceleration:
$$v=v_{0}+at$$
$$\Delta{x}=v_{0}t+\frac{1}{2}at^{2}$$
$$v^{2}-v^{2}_{0}=2a\Delta{x}$$
$$\Delta{x}=\frac{1}{2}\left(v_{0}+v\right)t$$
$$\Delta{x}=vt-\frac{1}{2}at^{2}$$
\end{document}