\section{Derivation of the cell age distribution}
\label{supp_cell_age_dist}

E. O. Powell first derive in 1956 the distribution of cell age for a cell
population growing steadily in the exponential phase \cite{Powell1956}. This
distribution is of the form
\begin{equation}
  P(a) = \ln(2) \cdot 2^{1 - a},
\end{equation}
where $a \in [0, 1]$ is the fraction of the cell cycle, 0 being the moment right
after the mother cell divides, and 1 being the end of the cell cycle just before
cell division. In this section we will reproduce and expand the details on each
of the steps of the derivation.

For an exponentially growing bacterial culture, the cells satisfy the growth law
\begin{equation}
  {\dt{n}} = \mu n,
  \label{seq_growth_ode}
\end{equation}
where $n$ is the number of cells and $\mu$ is the growth rate in units of
time$^{-1}$. We begin by defining $P(a)$ to be the probability density function
of a cell having age $a$. At time zero of a culture in exponential growth, i.e.
the time when we start considering the growth, not the initial condition of the
culture, there are $NP(a)da$ cells with age range between $[a, a + da]$. In
other words, for $N \gg 1$ and $da \ll a$
\begin{equation}
  N P(a \leq x \leq a + da) \approx N P(a)da.
\end{equation}
We now define
\begin{equation}
  F(\tau) = \int_\tau^\infty f(\xi) d\xi,
\end{equation}
as the fraction of cells whose division time is greater than $\tau$. This is
because in principle not all cells divide exactly after $\tau$ minutes, but
there is a distribution function $f(\tau)$ for the division time after birth.
Empirically it has been observed that a generalize Gamma distribution fits well
to experimental data on cell division time, but we will worry about this
specific point later on.

From the definition of $F(\tau)$ we can see that if a cell reaches an age $a$,
the probability of surviving to an age $a + t$ without dividing is given by
\begin{equation}
  P(\text{age} = (a + t) \mid \text{age} = a) = F(a + t \mid a) =
  {F(a + t) \over F(a)}.
  \label{seq_prob_a+t_give_a}
\end{equation}
This result comes simply from the definition of conditional probability. Since
$F(a)$ is the probability of surviving $a$ or more minutes without dividing, by
the definition of conditional probability we have that
\begin{equation}
  F(a + t \mid a) = {F(a, a + t) \over F(a)},
\end{equation}
where $F(a, a + t)$ is the joint probability of surviving $a$ minutes and $a +
t$ minutes. But the probability of surviving $a + t$ minutes or more implies
that the cell already survived $a$ minutes, therefore the information is
redundant and we have
\begin{equation}
  F(a, a + t) = F(a + t).
\end{equation}
This explains \eref{seq_prob_a+t_give_a}. From this equation we can find that
out of the $N P(a)da$ cells with age $a$ only a fraction
\begin{equation}
  \left[ NP(a)da \right] F(a + t \mid a) = NP(a) {F(a + t)\over F(a)} da
\end{equation}
will survive without dividing until time $a + t$. During that time interval $t$
the culture has passed from $N$ cells to $N e^{\mu t}$ cells given the
assumption that they are growing exponentially. The survivors $NP(a)F(a + t \mid
a)da$ then represent a fraction of the total number of cells
\begin{equation}
  {\text{\# survivors} \over \text{\# total cells}} =
  {\left[ NP(a)da \right] F(a + t \mid a) \over Ne^{\mu t}} =
  P(a){F(a + t)\over F(a)}da {1 \over e^{\mu t}},
\end{equation}
and their ages lie in the range $[a+t, a+t+da]$. Since we assume that the
culture is in steady state then it follows that the fraction of cells that
transitioned from age $a$ to age $a + t$ must be $P(a + t)da$. Therefore we have
a difference equation - the discrete analogous of a differential equation - of
the form
\begin{equation}
  P(a + t) da = P(a) {F(a + t) \over F(a)}e^{-\mu t} da.
  \label{seq_difference_eq}
\end{equation}
What this equation shows is a relationship that connects the probability of
having a life time of $a + t$ with a probability of having a shorter life time
$a$ and the growth of the population. If we take $t$ to be very small,
specifically if we assume $t \ll \mu^{-1}$ we can Taylor expand around $a$ the
following terms:
\begin{equation}
  F(a + t) \approx F(a) + {dF \over da} t,
\end{equation}
\begin{equation}
  P(a + t) \approx P(a) + {dP \over da} t,
\end{equation}
and
\begin{equation}
  e^{-\mu t} \approx 1 - \mu t.
\end{equation}
Substituting these equations into \eref{seq_difference_eq} gives
\begin{equation}
  P(a) + {dP \over da} t = P(a) \left( {F(a) + {dF \over da}t \over
  F(a)} \right) (1 - \mu t).
\end{equation}
This can be rewritten as
\begin{equation}
  {1 \over P(a)} {dP \over da} =
  {1 \over F(a)} {dF \over da} - \mu - {\mu t \over F(a)} {dF \over da}.
\end{equation}
Since we assumed $t \ll \mu^{-1}$ we then approximate the last term to be close
to zero. We can then simplify this result into
\begin{equation}
  {1 \over P(a)} {dP \over da} = {1 \over F(a)} {dF \over da} - \mu.
\end{equation}
Integrating both sides of the equation with respect to $a$ gives
\begin{equation}
  \ln P(a) = \ln F(a) - \mu a + C,
\end{equation}
where $C$ is the integration constant. Exponentiating both sides gives
\begin{equation}
  P(a) = C' F(a)e^{-\mu a}.
\end{equation}
Where $C' \equiv e^C$. To obtain the value of the unknown constant we recall
that $F(0) = 1$ since the probability of having a life equal or longer than zero
must add up to one, therefore we have that $P(0) = C'$. This gives then
\begin{equation}
  P(a) = P(0) e^{-\mu a} F(a).
  \label{seq_Pa_result}
\end{equation}
Substituting the definition of $F(a)$ gives
\begin{equation}
  P(a) = P(0) e^{-\mu a} \int_a^\infty f(\xi) d\xi.
  \label{seq_Pa_result_int}
\end{equation}
The last step of the derivation involves writing $P(0)$ and the growth rate
$\mu$ in terms of the cell cycle length distribution $f(\tau)$.

The growth rate of the population cell number (not the growth of cell mass) is
defined as the number of cell doublings per unit time divided by the number of
cells. This is more clear to see if we write \eref{seq_growth_ode} as a finite
difference
\begin{equation}
  {N(t + \Delta t) - N(t) \over \Delta t} = \mu N(t).
\end{equation}
If the time $\Delta t$ is the time interval it takes to go from $N$ to $2N$
cells we have
\begin{equation}
  {2N - N \over \Delta t} = \mu N.
\end{equation}
Solving for $\mu$ gives
\begin{equation}
  \mu = \overbrace{{2N - N \over \Delta t}}
  ^{\text{\# doubling events per unit time}}
  \overbrace{{1 \over N}}^{{1 \over \text{population size}}}.
\end{equation}
We defined $F(a)$ to be the probability of a cell reaching an age $a$ or
greater. For a cell to reach an age $a + da$ we can then write
\begin{equation}
  F(a + da) = \int_{a + da}^{\infty} f(\xi) d\xi
  = \int_a^{\infty} f(\xi) d\xi - \int_a^{a + da} f(\xi) d\xi.
\end{equation}
We can approximate the second term on the right hand side to be
\begin{equation}
  \int_a^{a + da} f(\xi) d\xi \approx f(a) da,
\end{equation}
for $da \ll a$, obtaining
\begin{equation}
  F(a + da) \approx F(a) - f(a)da.
\end{equation}
What this means is that from the original fraction of cells $F(a)$ with age $a$
or greater a fraction $f(a)da / F(a)$ will not reach age $(a + da)$ because they
will divide. So out of the $NP(a)$ cells that reached exactly age $a$, the
number of doubling events on a time interval $da$ is given by
\begin{equation}
  {\text{\# doublings of cells of age } a {\text{ on interval } da}} =
  \overbrace{NP(a)}^{\text{\# cells of age }a}
  \overbrace{f(a) da \over F(a)}^{\text{fraction of doublings per unit time}}.
\end{equation}
The growth rate then is just the sum (integral) of each age contribution to the
total number of doublings. This is
\begin{equation}
  \mu = {1 \over N} \int_0^\infty NP(a) {f(a)da \over F(a)}.
\end{equation}
Substituting \eref{seq_Pa_result} gives
\begin{equation}
  \mu = \int_0^\infty [P(0) e^{-\mu a} F(a)] {f(a)da \over F(a)}
  = \int_0^\infty P(0) e^{-\mu a} f(a)da.
  \label{seq_mu_integral}
\end{equation}
We now have the growth rate $\mu$ written in terms of the cell cycle length
probability distribution $f(a)$ and the probability $P(0)$. Since $P(a)$ is a
probability distribution it must be normalized, i.e.
\begin{equation}
  \int_0^\infty P(a) da = 1.
\end{equation}
Substituting \eref{seq_Pa_result} into this normalization constraint gives
\begin{equation}
  \int_0^\infty P(0) e^{-\mu a} F(a) da = 1.
\end{equation}
From here we can integrate the left hand side by parts. We note that given the
definition of $F(a)$, the derivative with respect to $a$ is $-f(a)$ rather than
$f(a)$. This is because if we write the derivative of $F(a)$ we have
\begin{equation}
  {dF(a) \over da} \equiv \lim_{da \rightarrow 0}
  {F(a + da) - F(a) \over da}.
\end{equation}
Substituting the definition of $F(a)$ gives
\begin{equation}
  {dF(a) \over  da} = \lim_{da \rightarrow 0} {1 \over da}
  \left[\int_{a + da}^\infty f(\xi) d\xi - \int_a^\infty f(\xi) d\xi \right].
\end{equation}
This difference in the integrals can be simplified to
\begin{equation}
  \lim_{da \rightarrow 0} {1 \over da} \left[ \int_{a + da}^\infty f(\xi) d\xi -
  \int_a^\infty f(\xi) d\xi \right]\approx {-f(a)da \over da} = -f(a).
\end{equation}
Taking this into account we now perform the integration by parts obtaining
\begin{equation}
 P(0) \left[ {e^{-\mu t} \over -\mu} F(a) \right]^\infty_0
 - P(0) \int_0^\infty {e^{-\mu a} \over -\mu} (-f(a)) da = 1.
 \label{seq_inte_byparts}
\end{equation}
On the first term on the left hand side we have that as $a \rightarrow \infty$,
both terms $e^{-\mu a}$ and $F(a)$ go to zero. We also have that $e^{\mu 0} = 1$
and $F(0) = 1$. This results in
\begin{equation}
  {P(0) \over \mu} - P(0) \int_0^\infty {e^{-\mu a} \over \mu} f(a) da = 1.
\end{equation}
The second term on the left hand side is equal to \eref{seq_mu_integral} since
\begin{equation}
  \mu = \int_0^\infty P(0) e^{-\mu a} f(a)da \Rightarrow
  1 = \int_0^\infty P(0) {e^{-\mu a} \over \mu} f(a)da.
\end{equation}
This implies that on \eref{seq_inte_byparts} we have
\begin{equation}
  {P(0) \over \mu} - 1 = 1 \Rightarrow P(0) = 2 \mu.
\end{equation}
With this result in hand we can rewrite \eref{seq_Pa_result_int} as
\begin{equation}
  P(a) = 2\mu e^{-\mu a} \int_a^\infty f(\xi) d\xi.
  \label{seq_Pa_Posubs}
\end{equation}
Also we can rewrite the result for the growth rate $\mu$ on
\eref{seq_mu_integral} as
\begin{equation}
  \mu = 2 \mu \int_0^\infty e^{-\mu a} f(a) da \Rightarrow
  2 \int_0^\infty e^{-\mu a} f(a) da = 1.
  \label{seq_fa_int}
\end{equation}

As mentioned before the distribution $f(a)$ has been empirically fit to a
generalize Gamma distribution. But if we assume that our distribution has almost
negligible dispersion around the mean average doubling time $a = \tau_d$, we can
approximate $f(a)$ as
\begin{equation}
  f(a) = \delta(a - \tau_d),
\end{equation}
a Dirac delta function. Applying this to \eref{seq_fa_int} results in
\begin{equation}
  2 \int_0^\infty e^{-\mu a} \delta(a - \tau_a) da = 1
  \Rightarrow 2 e^{-\mu \tau_d} = 1.
\end{equation}
Solving for $\mu$ gives
\begin{equation}
  \mu = {\ln 2 \over \tau_d}.
\end{equation}
This delta function approximation for $f(a)$ has as a consequence that
\begin{equation}
  F(a) =
  \begin{cases}
    1 \text{ for } a \in [0, \tau_d],\\
    0 \text{ for } a > \tau_d.
  \end{cases}
\end{equation}
Fianlly we can rewrite \eref{seq_Pa_Posubs} as
\begin{equation}
  P(a) = 2 \left( {\ln 2 \over \tau_d} \right)
  e^{- {\ln 2 \over \tau_d} a} \int_a^\infty \delta(\xi - \tau_d) d\xi
  \Rightarrow = 2 \ln 2 \cdot 2^{-a \over \tau_d}.
\end{equation}
Simplifying this we obtain
\begin{equation}
  P(a) =
  \begin{cases}
    \ln 2 \cdot 2^{1 - {a \over \tau_d}} \text{ for } a \in [0, \tau_d],\\
    0 \text{ otherwise}.
  \end{cases}
\end{equation}
This is the equation we aimed to derive. The distribution of cell ages over
the cell cycle.


\paragraph{Quantiles}
are points taken at regular intervals from the cumulative distribution function.

% \noindent
Quantiles have two equivalent notations:
\begin{itemize}[noitemsep,nolistsep]

  \item Not parametrized, $p$th quantile is equivalent to argument value for which c.d.f. has value
  $p$, where $p \in \mathbb{R} \cap [0,1]$.

  \item Parametrized with number of intervals equal $q$, these are called q-quantiles. For each $q$, there
  are $q-1$ q-quantiles: $\qquantile{1}$, $\qquantile{2}$, \ldots , $\qquantile{q}$.

\end{itemize}

\vspace{-10pt}
\begin{gather*}
\quantile{q}{k} \equiv \qqquantile{k/q}
\end{gather*}

Quantiles of Standard Normal distribution ($\distnormal(0,1)$) are denoted by $z$, for example: $z_{0.5}$ is a median of $\distnormal(0,1)$

\paragraph{Quartiles}
are 4-quantiles. They are denoted as follows:

$Q_1 \equiv \quantile{4}{1} \equiv \qqquantile{0.25}$

$Q_2 \equiv \quantile{4}{2} \equiv \qqquantile{0.5} \equiv \med$

$Q_3 \equiv \quantile{4}{3} \equiv \qqquantile{0.75}$

\paragraph{Deciles}
are 10-quantiles, denoted by $D_1, \ldots , D_9$.

\paragraph{Percentiles}
are 100-quantiles, denoted by $P_{1}, \ldots , P_{99}$.

% \begin{align*}
% \textrm{Median: } & Med =
% \left\{ \begin{array}{cc}
% \textrm{odd }n\textrm{: } & x_{\frac{n+1}{2}} \\ %(n+1)/2  n/2  n/2 + 1
% \textrm{even }n\textrm{: } & \frac{1}{2} \left( x_{\frac{n}{2} } + x_{\frac{n}{2}+1} \right)
% \end{array} \right\} \\
% \textrm{Mode: } & \textrm{observation that appears most often} \\
% \textrm{Quantiles: } & \ldots
% \end{align*}

% \subsubsection{Means}
% \begin{align*}
% \textrm{Average: } & \bar{x} = \frac{1}{n}\sum_{i=1}^n x_i \\
% \textrm{Trimmed: } & \sum_{i=1}^n \ldots \\
% \textrm{Windsorized: } & \sum_{i=1}^n \ldots \\
% \textrm{Geometrical: } & \ldots \\
% \textrm{Harmonic: } & \ldots
% \end{align*}


\section{Probability}

\formdesc{Assumptions}

\begin{center}
  \begin{tabular}{c}
    $0 \leq P(E) \leq 1$ \\
    $P(S) = 1$ \\
    $P(S) = \sum_{i=1}^{\infty} P(E_i)$ \\
  \end{tabular}
\end{center}

Sample space $S$ contains each event $E_i$, e.g., $E = \{$all outcomes in $S$ starting with a $3\}$

\hformbar

\formdesc{Unions and Intersections}

\begin{equation}
	E \cup F
\end{equation}

is the union of the two sets $E$ and $F$, i.e., the event where either $E$ or $F$ occurs. The intersection of two events, the outcomes in both $E$ and $F$ is

\begin{equation}
	E \cap F.
\end{equation}

\begin{center}
  \begin{tabular}{lc}
    Commutative  & $E \cup F = F \cup E$ \\
                 & $E \cap F = F \cap E$ \\
    Associative  & $(E \cup F) \cup G = E \cup (F \cup G) $ \\
                 & $(E \cap F) \cap G = E \cap (F \cap G)$ \\
    Distributive & $(E \cup F) \cap G = (E \cap G) \cap (F \cap G)$  \\
                 & $(E \cap F) \cup G = (E \cup G)(F \cup G)$
  \end{tabular}
\end{center}
\hformbar

\formdesc{Independent Events}

Two events are independent if knowing the outcome of one provides no useful information about the outcome of the other.
\hformbar

\formdesc{Mutually Exclusive Events}

\begin{equation}
   P(A \cap B) = 0
\end{equation}

\vspace{.63em}
are disjoint events, when $A$ and $B$ are mutually exclusive and there is no intersection---it is not possible for both to happen.
\hformbar

\formdesc{Union and Addition Rule}

\begin{equation}
    P(A \cup B) \equiv P(A \vee B) \equiv \{x: x \in A \vee x \in B\}
\end{equation}

\begin{center}
  \begin{tabular}{ll}
    Independent         & $P(A \cup B) = P(A) + P(B) - P(A \cap B)$   \\
    Mutually exclusive  & $P(A \cup B) = P(A) + P(B)$                 \\
  \end{tabular}
\end{center}

\hformbar

\formdesc{Intersection and Multiplication Rules}

\begin{equation}
    P(A \cap B) \equiv P(A \wedge B) \equiv \{x: x \in A \wedge x \in B\}
\end{equation}

\begin{center}
  \begin{tabular}{ll}
    Independent         & $P(A \cap B) = P(A) \cdot P(B)$   \\
    Mutually exclusive  & $P(A \cap B) = 0$                 \\
    Dependent           & $P(A \cap B) = P(A) \cdot P(B|A)$ \\
  \end{tabular}
\end{center}

\hformbar

\formdesc{Complement and Subtraction Rule}

\begin{eqnarray}
P(A') \equiv P(A^c) \equiv P(\neg A) &\equiv& \{x: x \notin A\} \equiv 1 - P(A) \\
P(A') &=& 1 - P(A)
\end{eqnarray}

Some implications:

\begin{center}
  \begin{tabular}{c}
    $P(A \cup A^c) = 1$ \\
    $P(A) = 1 - P(A^c)$ \\
    $P(A|B) = 1 - P(A^c | B)$ \\
  \end{tabular}
\end{center}
\hformbar

\formdesc{Conditional Probability Rule}

\begin{equation}
	P(A|B) = \frac{P(A \cap B)}{P(B)}
\end{equation}

is the probability of the outcome of event A given condition B.
\hformbar

\formdesc{Bayes Theorem}

\begin{equation}
  P(A|B) = \frac{P(A) ~P(B|A)}{ P(A) ~ P(B|A) + P(A) ~ P(B|A)}
\end{equation}
\hformbar

\formdesc{The Fundamental Principle of Counting}

If an operation can be performed in $n_1$ ways, and if for each of these a second operation can be performed $n_2$ ways, and for each of the first two a third operation can be performed in $n_3$ ways, $\ldots$, then the sequence of $k$ operations can be performed in $n_1 n_2 \mdots n_k$ ways.

\hformbar

\formdesc{Factorial}

\begin{equation}
    n! = n (n-1) (n-2) (2) (1)
\end{equation}

with $0! = 1$.
\hformbar

\formdesc{Permutation}

A permutation $\sigma$ any finite set $A$ is a one-to-one mapping of $A$ onto itself. An element mapped to itself in the permutation is a \textit{fixed point}.

\subsection*{Example}

One permutation of the set $A = \{a, b, c\}$ is:

\begin{equation}
    \sigma = \begin{pmatrix}
                  a & b & c \\
                  b & c & a
             \end{pmatrix}
\end{equation}

where $a$ is sent to $b$, $b$ to $c$, and $c$ to $a$.
\hformbar

\formdesc{Permutations of $n$ elements}

\begin{equation}
    n!
\end{equation}

is the number of permutations of $n$ objects, that is, the number of arrangements of a set containing $n$ elements.
\hformbar

\formdesc{Permutations: $n$ taken $r$ at a time}

\begin{equation}
    \perm{n}{k}=\frac{n!}{(n-k)!}
\end{equation}

represents the number of permutations of $n$ distinct objects taken $r$ at a time.
\hformbar

\formdesc{Stirling's Approximation of $n!$}

\begin{equation}
    n^n e^{-n} \sqrt{2 \pi n}
\end{equation}

is a sequence asymptotically equal to $n!$.
\hformbar

\formdesc{Combinations}

\begin{equation}
    \comb{n}{j} = \begin{pmatrix} n \\ j \end{pmatrix} = \frac{n!}{j! (n - j)!}
\end{equation}

is the number of subsets of size $j$ that can be assembled given a set of $n$ elements, for integers $n$ and $j$ such that $0 < j < n$.
\hformbar

\formdesc{Bernoulli Trials Process}

A sequence of $n$ experiments such that

\begin{itemize}
    \item Each experiment has two possible outcomes, called \textit{success} and \textit{failure}
    \item The probability of $p$ of success of each experiment is the same for each, and is independent of previous experiments
\end{itemize}

The probability of failure is $q = 1 - p$.
\hformbar

\formdesc{Binomial Probabilities}

\begin{equation}
    b(n, p, j) = \comb{n}{j} p^j (1 - p)^{n - j}
\end{equation}

represents the probability that in $n$ Bernoulli trials there are exactly $j$ successes, given $n$ trials, and each trials' success rate is $p$.
\hformbar

\formdesc{Binomial Distribution}

\begin{equation}
    B \sim b(n, p, k)
\end{equation}

gives the probability of the number of successes $k$ in a sequence of Bernoulli trials with parameters $p$ and $n$.
\hformbar

PDF P. 107

\newpage


\subsection{Using the inverse to solve a system of equations}

One way in which the inverse of a matrix is useful is to find the
solution of a system of linear equations.  Recall from Definition~\ref{def:matrix-form} that we can write a system of equations in
matrix form, which is in the form
\begin{equation*}
  A\vect{x}=\vect{b}.
\end{equation*}
Suppose we find the inverse $A^{-1}$ of the matrix $A$. Then we can
multiply both sides of this equation by $A^{-1}$ on the left and
simplify to obtain
\begin{equation*}
  \vect{x} = A^{-1}\vect{b}.
\end{equation*}
Therefore we can find $\vect{x}$, the solution to the system, by
computing $\vect{x} = A^{-1}\vect{b}$. Note that once we have found
$A^{-1}$, we can easily get the solution for different right-hand
sides (different $\vect{b}$). It is always just $A^{-1}\vect{b}$.

\begin{example}{Using the inverse to solve a system of equations}{inverse-to-solve-system}
  Consider the following system of equations. Use the inverse of a
  suitable matrix to solve this system.
  \begin{equation*}
    \begin{array}{c}
      x+z=1 \\
      x-y+z=3 \\
      x+y-z=2
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  First, we can write the system in matrix form
  \begin{equation*}
    A\vect{x} =
    \begin{mymatrix}{rrr}
      1 & 0 & 1 \\
      1 & -1 & 1 \\
      1 & 1 & -1
    \end{mymatrix} \begin{mymatrix}{r}
      x \\
      y \\
      z
    \end{mymatrix} =\begin{mymatrix}{r}
      1 \\
      3 \\
      2
    \end{mymatrix} = \vect{b}.
  \end{equation*}
  The inverse of $A$ is
  \begin{equation*}
    A^{-1} =
    \def\arraystretch{1.2}
    \begin{mymatrix}{rrr}
      0 & \frac{1}{2} & \frac{1}{2} \\
      1 & -1 & 0 \\
      1 & -\frac{1}{2} & -\frac{1}{2}
    \end{mymatrix}.
  \end{equation*}
  From here, the solution to the system $A\vect{x}=\vect{b}$ is found
  by $\vect{x}=A^{-1}\vect{b}$, i.e.,
  \begin{equation*}
    \def\arraystretch{1.2}
    \begin{mymatrix}{r}
      x \\
      y \\
      z
    \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      0 & \frac{1}{2} & \frac{1}{2} \\
      1 & -1 & 0 \\
      1 & -\frac{1}{2} & -\frac{1}{2}
    \end{mymatrix} \begin{mymatrix}{r}
      1 \\
      3 \\
      2
    \end{mymatrix} =\begin{mymatrix}{r}
      \frac{5}{2} \\
      -2 \\
      -\frac{3}{2}
    \end{mymatrix}.
  \end{equation*}
\end{solution}

What if the right-hand side had been $\vect{b}=\begin{mymatrix}{r}
  0 \\
  1 \\
  3
\end{mymatrix}$? In this case, the solution would be given by
\begin{equation*}
  \def\arraystretch{1.2}
  \begin{mymatrix}{r}
    x \\
    y \\
    z
  \end{mymatrix} = \begin{mymatrix}{rrr}
    0 & \frac{1}{2} & \frac{1}{2} \\
    1 & -1 & 0 \\
    1 & -\frac{1}{2} & -\frac{1}{2}
  \end{mymatrix} \begin{mymatrix}{r}
    0 \\
    1 \\
    3
  \end{mymatrix} =\begin{mymatrix}{r}
    2 \\
    -1 \\
    -2
  \end{mymatrix}.
\end{equation*}
This illustrates that for a system $A\vect{x}=\vect{b}$ where $A^{-1}$ exists,
it is easy to find the solution when the vector $\vect{b}$ is changed.


%---------------------------------------------------------------------------------
% Monte Carlo Simulations
%---------------------------------------------------------------------------------

\section{Monte Carlo Simulations}

\subsection{Approximation of $\pi$}

We will use this toy example as a brief introduction to Monte Carlo Simulations. Lets assume that we are trying to find an approximation for $\pi$. We can still do this using the same randomization-based process as before. We will use the statistical and programming software R for our process. This is a particularly interesting example, as there are very little conditions to actually be satisfied. The only condition that needs to be met as has been said time again is that the sample is representative of the population. In our case, the population is the points $x,y\in\mathscr{R}: x,y\sim U(-0.5,0.5)$.
\begin{center}
    \begin{lstlisting}[language = R]
        ```{r}
        runs <- 100000
        x <- runif(runs,min=-0.5,max=0.5)
        y <- runif(runs,min=-0.5,max=0.5)
        in.circle <- x^2 + y^2 <= 0.5^2
        mc.pi <- (sum(in.circle)/runs)*4
        ```
    \end{lstlisting}
\end{center}
\newline\\
We've made sure that that $x$ and $y$ are representative of each population by selecting a random value from $U(-0.5,0.5)$ using the \textbf{runif()} command which stands for "random uniform". Rather than a least squares estimator, in this case, our test statistic will be the Bernoulli random variable representing whether a point is contained within the circle which inscribes our population distribution. Now that we have our test statistics, we can approximate $\pi$. Empirically, $A_{square}=(2r)^2=4r^2$. Suppose we don't know a value for $\pi$, but instead know that $A_{circle}\propto r^2$. That is, $A_{circle}=cr^2$ where $c$ is a positive constant. We can now approximate $\pi$ where $c$ is our approximation. This is as simple as taking the proportion of $A_{circle}$ and $A_{square}$.

$$
\frac{A_{circle}}{A_{square}} = \frac{cr^2}{4r^2} = \frac{c}{4}\longrightarrow\frac{4\cdot A_{circle}}{A_{square}}
$$
\\
The simulated $A_{circle}$ that we have is just the total number of points inside of the circle while $A_{square}$ will be the total number of points that we sampled. The approximation for $\pi$ in this example is $3.14252$, not too far off from the true value of $3.14159$.

\begin{figure}[htpb!] % Defines figure environment
    \centering % Centers your figure
\includegraphics[scale=0.4]{figure/monteCarloCircle.png} % Includes your figure and defines the size
    \caption{Plot of sampling distribution and inscribed circle} % For your caption
    \label{fig:my_label} % If you want to label your figure for in-text references
\end{figure}

\subsection{Monte Carlo Simulation and RBI}

The Monte Carlo simulation is a do-it-all kind of method which has many useful applications in statistics and mathematics in general. For randomization based inference, Monte Carlo simulations manifest themselves as a streamlined way to get the asymptotically correct p-value we want. The process is applied as follows: First, instead of taking $n!$ permutations of our data, we take a random sample of permutations usually between 5 to 100. We then calculate the proportion of those permutations that yield a test statistics that are as or more extreme than our original test statistic. Now we have our random variable $P$ (the aforementioned proportion) with which we create a 95\% confidence interval using the largest possible binomial standard deviation, $\sqrt{n*(0.5)(1-0.5)}$. Thus, we create a sufficiently wide confidence interval that, if created over and over again, will contain the true p-value for our effect 95\% of the time.


\section{Running-mean/moving average}
Since we have no replicates and we want to force $s(x)$ to be smooth
we can use the motivation that under some stastical model, for any
$x_0$ values of $f(x)=\E[Y|X=x]$ for
$x$ close to $x_0$ are similar.

How do we define close?
A formal definition is the {\it symmetric nearest neighborhood}
\[
N^S(x_i) = \{ \max(i-k,1),\dots,i-1,i,i+1,\min(i+k,n) \}
\]

We may now define running mean as:
\[
s(x_i) = \ave_{j \in N^S(x_i)} \{ y_j \}
\]

We can also forget about the symmetric part and simply define the
nearest $k$ neighbors.

\begin{figure}[htp]
\caption{\label{f2.4} CD4 cell count since seroconversion for HIV infected men.}
\centerline{\epsfig{figure=Plots/plot-02-04.ps,angle=270,width=.8\textwidth}}
\end{figure}

This usually too wiggly to be considered useful.  Why do you think?

Notice we can also fit a line instead of a constant. This procedure is
called running-line.

Can you write out the recipe for $s(x_i)$ for the running-line smoother?


\section{Posterior Probability Criteria}
Objections have been raised that minimizing Akaike's criterion does
not produce asymptotically consistent estimates of the correct model
Notice that if
we consider Model($p^*$) as the correct model then we have
for any $p>p*$
\begin{equation}
\label{prob}
\Pr\left[ AIC(p) < AIC(p^*) \right] = \Pr\left[ 2 \{l(\bbh_{p}) - l(\bbh_{p^*})
\} > 2(p - p^*)\right].
\end{equation}
Notice that, in this case, the random variable $2\{l(\bbh_{p}) -
l(\bbh_{p^*})\}$ is
the logarithm of the likelihood ratio of two competing models which,
under certain regularity
conditions, is known to converge in
distribution to $\chi^2_{p-p^*}$, and thus it follows that the
probability in Equation (\ref{prob}) is not 0
asymptotically.
Some have suggested multiplying
the penalty term in the
AIC by some increasing function of $n$, say $a(n)$, that makes the
probability
\[
 \Pr\left[ 2 \{l(\bbh_{p}) - l(\bbh_{p^*})
\} > 2a(n)(p - p^*)\right]
\]
asymptotically equal to 0. There are many choices of $a(n)$ that would
work in this context. However, some of the choices made in the
literature seem arbitrary.

Schwarz (1978) \nocite{schw:1978} and Kashyap (1982) \nocite{kash:1982} suggest using a
Bayesian approach
to the problem of model selection which, in the IID case,  results in a
criterion that is similar to AIC in that it is based on a penalized
log-likelihood function evaluated at the maximum likelihood estimate
for the model in question. The
penalty term in the Bayesian Information Criteria (BIC) obtained by
Schwarz (1978) \nocite{schw:1978}
is the AIC penalty term $p$ multiplied by the function $a(n)
=\frac{1}{2}\log(N)$.

The Bayesian approach to model selection is based on maximizing the
posterior probabilities of the alternative models, given the
observations. To do this we must define a strictly positive prior
probability
$\pi_p = \Pr[\mbox{Model}(p)] $ for each model and a conditional prior
$d\mu_p(\bb)$ for the parameter given it is in $\Omega_p$, the subspace
defined by Model(p). Let $\bY = (Y_1,\dots,Y_n)$ be the
response variable and define the distribution given $\bb$
following (\ref{parsimodel})
\[
f_{\bY}(\by| \bX,\bb) \equiv \prod_{i=1}^n f(y_i;\bx_i,\bb)
\]
The posterior probability that we look to
maximize is
\[
\Pr\left[\mbox{Model}(p) | \bY=\by \right] =
\frac{\int_{\Omega_p} \pi_p f_{\bY}(\by|\bX,\bb) d\mu_p(\bb)}
{\sum_{q=1}^P \int_{\Omega_q} \pi_q f_{\bY}(\by|\bX,\bb) d\mu_q(\bb)}
\]
Notice that the denominator depends neither on the model nor the data,
so we need only to maximize the numerator when choosing models.

Schwarz (1978) \nocite{schw:1978} and Kashyap (1982)
\nocite{kash:1982} suggest criteria derived
by taking a Taylor expansion of the log posterior probabilities of the
alternative models. Schwarz (1978) \nocite{schw:1978} presents the following
approximation for the IID case
\[
\log \int_{\Omega_p} \pi_p f_{\bY}(\by|\bX,\bb) d\mu_p(\bb)
\approx  l(\bbh_{p}) - \frac{1}{2} p
\log n
\]
with $\bbh_p$ the maximum likelihood estimate obtained under Model(p).

This fact leads to the Bayesian Information Criteria (BIC) which is
\begin{equation}
\label{BIC}
\mbox{BIC}(p) = - 2 l(\bbh_{p}) + p\log n
\end{equation}

\subsection{Kyphosis Example}
The AIC
and BIC obtained for the gam are:

\begin{verbatim}
AIC(Age) =  83              BIC(Age) = 90
AIC(Age,Start) = 64         BIC(Age,Start) = 78
AIC(Age,Number) = 73        BIC(Age,Number) = 86
AIC(Age,Start,Number) = 60  BIC(Age,Start,Number) = 81
\end{verbatim}


\section{The Peano Axioms}
We know what the natural numbers are, or do we?
In school, we were often told a rather vague idea of the notion of natural numbers.
We know that they are $1,2,3,\ldots$, but what actually are they?
As every mathematical discipline, we do not know what is it until we define it using axioms.
In the case of natural numbers, this is done by the Peano axioms:
\begin{definition}
    Natural numbers is a triple $(\mathbb N, 1, +1)$, where $\mathbb N$ is a set, $1\in\mathbb N$ its element, and $+1:\mathbb N\to\mathbb N$ an operation on $\mathbb N$.
    It satisfies the following axioms:\\
    1. For any $n\in\mathbb N$, $n+1\neq 1$.\\
    2. For $n,m\in\mathbb N$, if $n\neq m$, then $n+1\neq m+1$.\\
    3. Let $P$ be a proposition on $\mathbb N$. If $P(1)$ is true and $P(n)\implies P(n+1)$ for all $n\in\mathbb N$, then $P(n)$ is true for all $n\in\mathbb N$.
\end{definition}
\begin{definition}[Addition]
    We define the operation $+k$ inductively by $n+(k+1)=(n+k)+1$.
\end{definition}
\begin{proposition}
    The operation $+k$ is defined for all natural number $k$.
\end{proposition}
\begin{proof}
    Induction.
\end{proof}
Similarly, using induction, we can define multiplication, exponentiation and order ($a<b\iff\exists k\in\mathbb N, a+k=b$) in the obvious way.
\begin{proposition}
    1. $(a+b)+c=a+(b+c)$.\\
    2. $a+b=b+a$.\\
    3. $(ab)c=a(bc)$.\\
    4. $ab=ba$.\\
    5. $a(b+c)=ab+ac$.\\
    6. $a<b\land b<c\implies a<c$.\\
    7. $\lnot(a<a)$.
\end{proposition}
\begin{proof}
    Induction. Induction. Induction. Induction. Induction. Induction. Induction.
\end{proof}
There is a more useful form of induction.
Induction says that if we have some proposition $P$ such that $P(1)$ is true and $P(n)\implies P(n+1)$, then $P(n)$ is true.
But in fact, we can have a ``stronger'' induction hypothesis.
\begin{theorem}[Strong induction]
    Suppose that $P$ is a proposition on $\mathbb N$.
    If $P(1)$ is true and for any $n\in\mathbb N$,
    $$\forall k\le n, P(k)\implies P(n+1)$$
    Then $P(n)$ is true for any $n\in\mathbb N$.
\end{theorem}
\begin{proof}
    Apply our usual induction on the proposition $Q(n)$ meaning
    $\forall k\le n, P(k)$
\end{proof}
Technically, we do not need to check the base case.
But it is often safer to check it.
If we want to use strong induction on $P$, we can prove $P(n)$ assuming the case for smaller numbers, as induction says if it would help to assume $P(m)$ for some $m<n$, feel free to do so.
There are a few equivalent forms of strong induction which can be pretty useful.
\begin{corollary}
    If some $n$ has $P(n)$ false, then there exists an $n$ with $P(n)$ false but $P(m)$ true for every $m<n$.
\end{corollary}
That is, if there is a counterexample, then there is a minimal counterexample.
\begin{corollary}[Well-ordering Principle]
    If some $n$ has $P(n)$ true, then there exists a minimal $n$ with $P(n)$ true.
\end{corollary}


\newpage
\subsection{Supper approximation of $x^2$ via composition}
We consider a set of equidistant girds $\Omega_\ell$ of level $\ell$ on the unit interval $\bar{\Omega}=[0,1]$ and mesh length $h_\ell = 2^{-\ell}$. The grid points $x_{\ell,i}$ are given by
\begin{equation}
x_{\ell,i}:=ih_\ell,\quad 0\le i\le 2^\ell.
\end{equation}

A key observation about the connections between linear finite element functions and ReLU DNNs in \cite{he2020relu} is that
\begin{equation}\label{def_g}
g(x) = 2{\rm ReLU}(x) - 4{\rm ReLU}({x-\frac{1}{2}}) + 2{\rm ReLU}(x-1)\in {{\rm DNN}}_{1}^{3},
\end{equation}
for $x \in \mathbb{R}$. The nodal basis functions on $\Omega_\ell$ are defined as
\begin{equation}
\phi_{\ell,i}(x):=
\begin{cases}
g\left(\frac{x-(i-1)h_\ell}{2h_\ell}\right)\quad &x\in [x_{\ell,i}-h_\ell,x_{\ell,i}+h_\ell],\\
0 \quad &\text{otherwise}.
\end{cases}
\end{equation}
By definition, we have $\phi_{1,1}(x) = g(x)$ for $x \in[0,1]$. Besides, for $\ell=0$, we have these next two basis functions
\begin{equation}\label{key}
\phi_{0,0} = 1- x \quad \text{and} \quad \phi_{0,1} = x, \quad \forall x \in [0,1].
\end{equation}

These basis functions are used to define the piecewise linear function space
\begin{equation}
V_\ell :=\mbox{span}\{\phi_{\ell,i}: 0 \le i\le 2^\ell\} .
%\subset H_0^1(0,1).
\end{equation}
Then, let's define the the piecewise linear interpolation on $\Omega_\ell$ as:
\begin{equation}
\Pi_\ell u = \sum_{i=0}^{2^\ell}u(x_{\ell,i}) \phi_{\ell,i}.
\end{equation}
Thus, we also have the next so-called hierarchical decomposition for $L \ge 1$
\begin{equation}
\begin{aligned}
\Pi_L u(x) &=\Pi_0 u +  \sum_{\ell=1}^L  (\Pi_\ell - \Pi_{\ell-1})u \\
&=\Pi_0 u +  \sum_{\ell}^L \sum_{i\in I_\ell}\mu_{\ell,i}\phi_{\ell,i},
\end{aligned}
\end{equation}
where
\begin{equation}
I_\ell = \{i\in \mathbb{N}~:~1\le i\le 2^\ell-1, ~i ~\text{ is odd} \}.
\end{equation}
Furthermore, we have
\begin{equation}\label{key}
(\Pi_\ell - \Pi_{\ell-1})u = \sum_{i\in I_\ell}\mu_{\ell,i}\phi_{\ell,i} = \sum_{i\in I_\ell} \left( u(x_{\ell,i}) - \frac{1}{2}\left( u(x_{\ell,i-1})  + u(x_{\ell,i+1})  \right) \right)\phi_{\ell,i}.
\end{equation}

The key observation here is that there exist a special decomposition form once we take $u(x) = x^2$.
By the hierarchical decomposition above, we have
\begin{equation}
\begin{aligned}
(\Pi_{\ell} - \Pi_{\ell-1})u &= \sum_{i\in I_\ell}\left( u(x_{\ell,i}) - \frac{1}{2}\left( u(x_{\ell,i-1})  + u(x_{\ell,i+1})  \right) \right)\phi_{\ell,i}(x)\\
&=\sum_{i\in I_\ell}\left(x_{\ell,i}^2 - \frac{1}{2}(x_{\ell,i-1}^2 + x_{\ell,i+1}^2)\right)\phi_{\ell,i}(x)\\
&= \sum_{i\in I_\ell}\left(x_{\ell,i}^2 - \frac{1}{2}\left((x_{\ell,i}-h_\ell)^2 + (x_{\ell,i}+h_\ell)^2\right) \right)\phi_{\ell,i}(x) \\
&=-h_\ell^2\sum_{i\in I_\ell}\phi_{\ell,i}(x).
\end{aligned}
\end{equation}
Note $\Pi_0 u = u(0)\phi_{0,0} + u(1)\phi_{0,1} = x$, we have
\begin{equation}
\Pi_L u = \Pi_0 u + \sum_{\ell=1}^L  (\Pi_\ell - \Pi_{\ell-1})u = x -\sum_{\ell=1}^\ell h^2_\ell \sum_{i\in I_\ell}\phi_{\ell,i}(x) = x -\sum_{\ell=1}^\ell h^2_\ell g_\ell(x),
\end{equation}
where
\begin{equation}\label{key}
g_\ell(x) := \sum_{i\in I_\ell}\phi_{\ell,i}(x).
\end{equation}

By definition, we have the next diagram of $g_\ell(x)$:
\begin{figure}[h!]
	\centering
	\includegraphics[width=0.6\textwidth]{figures/hierachybasis_gl}
	\caption{The figure of $g_\ell(x)$}
	\label{fig:gl}
\end{figure}

The next property plays the crucial role for the connection between ReLU DNN and hierarchical basis functions:
\begin{equation}\label{func:glinduction}
g_\ell(x)=g(g_{\ell-1}(x)),\quad \ell=2,...,L, \quad \forall x \in [0,1],
\end{equation}
with $g_1(x) = g(x)$ as defined in~\ref{def_g}. Thus, we have the next key relation by the  property of composition
\begin{equation}\label{key}
g_\ell \in {\rm DNN}_\ell^{3}.
\end{equation}
Based on the definition of $\widehat{{\rm DNN}}_{\ell}^{3}$, the above relation leads to
\begin{equation}\label{key}
x -\sum_{\ell=1}^L h^2_\ell g_\ell(x) \in \widehat{{\rm DNN}}_{L}^{3},
\end{equation}
where $f \in \widehat{{\rm DNN}}_{L}$ is a modified DNN structure as:
\begin{equation}\label{def:ReLUDNN2}
\begin{cases}
f^{1}(x) &= \sigma \circ \theta^1 (x) \\
f^{\ell}(x) &= \sigma \circ  \theta^{\ell} ([x, f^{\ell-1}(x)]) \quad \ell = 2:L \\
f(x) &= \theta^{L+1}( [x, f^1, \cdots, f^L] ) \\
\end{cases}.
\end{equation}


\subsection{Type-I - Gumbel distribution}

The probability function is:

\(f(x)=\dfrac{1}{\beta }e^{-(\dfrac{x-\mu}{\beta }+e^{-\dfrac{x-\mu }{\beta }})}\)

We can use:

\(z=\dfrac{x-\mu }{\beta }\)

To get:

\(f(x)=\dfrac{1}{\beta }e^{-(z+e^{-z})}\)

\subsubsection{Link to the logistic function}

The difference between two draws from a Gumbel distribution is drawn from the logistic function.


\chapter{Week 3: The matrix}

\section{What is a matrix}
The traditional notion of a matrix: a two dimensional array
\begin{equation*}
  \mA =
  \begin{bmatrix}
    a & b & c \\
    d & e & f
  \end{bmatrix}
\end{equation*}
where $\mA$ is a $2 \times 3$ matrix.

For a matrix $\mA$, the $i, j$ element of $\mA$ is the element in row $i$, column $j$. It is traditionally written as $\mA_{i, j}$

\begin{definition}
  For finite sets $\sR$ and $\sC$, and $\sR \times \sC$ matrix over field $\fF$ is a function from $\sR \times \sC$ to $\fF$.
\end{definition}

\begin{definition}
  $D \times D$ identity matrix is the matrix $\mathds{1}_D$ such that $\mathds{1}_D[k, k] = 1$ for all $k \in \{1, , \ldots, D\}$ and zero elsewhere.
\end{definition}

Usually we omit the subscript when $D$ is clear from the context.
Often letter $\mI$ is used instead of $\mathds{1}$

\subsection{Column space and row space}
One simple role for a matrix: packing together a bunch of columns or rows.
\begin{definition}
  Two vectors associated with a matrix $\mM$:
  \begin{itemize}
  \item Column space of $\mM = \Span\{\text{columns of }\mM\}$. Written $\Col \mM$.
  \item Row space of $\mM = \Span\{\text{rows of }\mM\}$. Written $\Row \mM$.
  \end{itemize}
\end{definition}

\subsection{Transpose}
Transpose swaps rows and columns. It is written as $\mA^{\top}$.
\begin{equation*}
  \mA =
  \begin{bmatrix}
    a & b & c \\
    d & e & f
  \end{bmatrix}
\end{equation*}

\begin{equation*}
  \mA^{\top} =
  \begin{bmatrix}
    a & d \\
    b & e \\
    c & f
  \end{bmatrix}
\end{equation*}

\subsection{Matrices as vectors}
A matrix can be interpreted as a vector:
\begin{itemize}
\item an $\sR \times \sS$ matrix is a function from $\sR \times \sS$ to $\fF$,
\item it can be interpreted as and $\sR \times \sS$--vector:
  \begin{itemize}
  \item scalar-vector multiplication
  \item vector addition
  \end{itemize}
\end{itemize}


\documentclass[11pt,a4paper]{report}
\usepackage{amsmath,amsfonts,amssymb,amsthm,epsfig,epstopdf,titling,url,array}
\usepackage{changepage}
\usepackage{graphicx}
\usepackage[utf8]{inputenc}
\theoremstyle{plain}
\newtheorem{thm}{Theorem}[section]
\newtheorem{lem}[thm]{Lemma}
\newtheorem{prop}[thm]{Proposition}
\newtheorem*{cor}{Corollary}
\theoremstyle{definition}
\newtheorem{defn}{Definition}[section]
\newtheorem{conj}{Conjecture}[section]
\newtheorem{exmp}{Example}[section]
\newtheorem{exercise}{Exercise}[section]
\theoremstyle{remark}
\newtheorem*{rem}{Remark}
\newtheorem*{note}{Note}
\begin{document}

\section*{Problem}
Find a function $f(x_0, ..., x_{n-1})$ that maps finite sequences $<x_1, ..., x_{n-1}>$ of positive integers one-to-one into the integers.  From a programmer's perspective, this means find a function that takes a variable number of positive integer parameters and returns a value that is unique to the sequence.  For example, the function $f(x_0, ..., x_{n-1}) = \sum_{i=0}^{n-1}{x_i}$ does not work, since with $f$ defined this way $f(1,2,3)$ returns the same value as $f(3,3)$.

What I am asking you to define here is essentially a \textit{guaranteed unique} hash function.

\section*{Solution}
Let $p_0, p_1, ..., p_n, p_{n+1}, ...$ be a list of the (infinitely many) prime numbers in ascending order.  So $p_0 = 2, p_1 = 3$ and so on.  Define
$$f(x_0, ..., x_{n-1}) = p_0^{x_0}p_1^{x_1}...p_{n-1}^{x_{n-1}}$$
$f$ maps $<x_1, ..., x_{n-1}>$ to the product of the first $n-1$ primes with exponents equal to the corresponding numbers in the sequence.  For example, $f(3,2,1) = 2^{3} \times 3^{2} \times 5^{1} = 380.$
This function is 1-1 because of the uniqueness of prime factorization (the Fundamental Theorem of Arithmetic).  If $f(x_0, ..., x_{n-1}) = f(y_0, ..., y_{m-1})$ then $p_0^{x_0}p_1^{x_1}...p_{n-1}^{x_{n-1}} = p_0^{y_0}p_1^{y_1}...p_{m-1}^{x_{m-1}}$; but this means that both of these expressions are prime factorizations of the same number.  By the uniqueness of prime factorization, it follows that $n = m$ and all of the exponents are the same, which means for all $i = 0, ..., n-1, x_i = y_i$, i.e., the sequences are the same.

\textit{Remark:} Above would actually be a bad hash function because it would overflow very quickly. Hash functions are not guaranteed unique partly because they in general can't be: their range is a finite subset of the positive integers.

\end{document}


% !TEX root = cs1textbook.tex

\chapter{Number Systems}
\label{appendix:numbersystems}

\minitoc

This appendix explains why programmers seem to choose to represent numbers in a different base, or \textit{radix}, than what you may be used to, and how you can try to convert between different radices.

\setcounter{section}{9}
\section{Decimal Numbers}
The numbers that the average person encounters in his or her daily life are all represented as \textit{decimal} numbers.  The Latin root \textit{dec-} refers to the number 10, as illustrated in the words for a collection of ten years (\textit{decade}) or a collection of ten units of loudness (\textit{decibel}).

Each digit in a decimal number has a place name - the ones place, the tens place, the hundreds place, etc. - that describes the meaning of the digit.  For instance, in the decimal number 3502, we're meant to understand that there are three thousands, five hundreds, no tens, and two ones, or:

\[ 3502 = 3 \cdot 1000 + 5 \cdot 100 + 0 \cdot 10 + 2 \cdot 1 \]

Notice that each of these place names corresponds to a power of 10, so we can also write 3502 as:

\begin{equation}\label{base10}3502 = 3 \cdot 10^3 + 5 \cdot 10^2 + 0 \cdot 10^1 + 2 \cdot 10^0\end{equation}

This leads to the description of decimal numbers as \textit{base ten} numbers, because the base in each of those exponential expressions in Equation \ref{base10} is 10.  Notice also that the base ten system provides ten digits.  Any number composed of a sequence of only the last digit, like 99, or 999,999, will be followed by a number composed of a 1 followed by a sequence of zeroes, like 100 or 1,000,000.

\setcounter{section}{1}
\section{Binary Numbers}
The decimal number system is not, however, the most convenient way to discuss computer systems.  The state of any component of a computer can usually be described in one of two ways -- open or closed, high voltage or low voltage, magnetically north or magnetically south, etc.  For this reason, data storage and transmission are often described instead using \textit{binary} numbers, from the Latin root \textit{bi-} for two (\textit{bicycle}, \textit{bidirectional}, \textit{binaural}).  The \textbf{binary number system}\index{Binary}, also called \textit{base two}, defines numbers in a two-digit system, where each exponent's base is 2.  For instance, the binary number 11101 represents this idea:

\begin{equation}
    \begin{array}{ccccccccccc}
    11101_2 &=& 1 \cdot 2^4 &+& 1 \cdot 2^3 &+& 1 \cdot 2^2 &+& 0 \cdot 2^1 &+& 1 \cdot 2^0\\
    &=& 16 &+& 8 &+& 4 &+& 0 &+& 1\\
    &=& 29
\end{array}
\end{equation}

Since the largest binary digit is 1, any binary number composed of all 1s will be followed by a number consisting of a 1 and a sequence of zeros; for instance, $111_2 + 1 = 1000_2$.

Notice that the most correct term for a \underline{bi}nary digi\underline{t} is \textit{bit}.

\setcounter{section}{15}
\section{Hexadecimal Numbers}

As one counts upward in binary numbers, the length in bits grows exponentially compared to the length of their decimal equivalents.  Even a relatively small decimal number like 351 turns into the nine-bit number $101011111_2$.  It is incredibly difficult for humans to read long sequences of zeros and ones such as this.  However, it is also difficult to convert between base two and base ten, in either direction, in one's head.  There is no pattern to follow, and the amount of memorization required is way out of proportion with the usefulness of such a task.

There is, however, a way to represent these numbers in a shorter format, that supports much easier conversion.  The term \textbf{hexadecimal}\index{Hexadecimal} was composed from the Greek root \textit{hexa-} for 6 and the Latin root \textit{dec-} for ten, to help us talk about numbers in \textit{base sixteen}.  The sixteen digits in hexadecimal are the ten decimal digits 0-9, and then the first six letters of the alphabet, so that $10_{10} =$ A$_{16}$, $11_{10} =$ B$_{16}$, and so on.  Since $16 = 2^4$, each hex digit corresponds to a unique four-bit sequence.  With a little bit of practice, even a novice programmer can become accustomed to interpreting \verb-0011- as \verb-3-, \verb-1010- as \verb-A-, and \verb-1111- as \verb-F-, and so any binary number can be written as a hex number using no more than 25\% of the digits.

It follows, then, that any byte -- which represents a number in the range $2^0 \le n < 2^8$ -- can be written using two hex digits, since $0_{10} = 00000000_2 = 00_{16}$, and $255_{10} = 11111111_2 = $ FF$_{16}$.  For instance, using the examples from the previous paragraph, the number $00111010_2$ is equal to the number 3A$_{16}$.

\setcounter{section}{7}
\section{Octal Numbers}

While less frequently used, it is useful to mention here that \textit{base eight} digits, called \textbf{octal}\index{Octal} (think about an \textit{octagon}) can represent a three-bit sequence.  The commands to alter a file's permissions in Unix take advantage of octal numbers.  Unix file permissions come in three varieties -- read permission, write permission, and execute permission -- and in three groups -- what the file's owner is allowed to do, what members of the owner's group are allowed to do, and what other users on the system are allowed to do.  Each permission can be thought of as a \textit{flag}, or a Boolean setting: a 1 means permission is granted, and a 0 means that permission is not granted.  So, to give the owner of a file called \texttt{foo.txt} permission to read and write a file, but not to execute it (\texttt{110}), and to allow others only to read the file (\texttt{100}), the Unix command would look like:

\begin{center}\texttt{chmod 644 foo.txt}\end{center}

since $6_8 = 110_2$ and $4_8 = 100_2$.

\section*{Number Systems Compared}

The following table shows how to write the decimal numbers 0 through 16 in binary, octal, and hex.

\begin{center}
\begin{tabular}{rrrr}
\textbf{Binary} & \textbf{Octal} & \textbf{Decimal} & \textbf{Hexadecimal}\\
\hline
0 & 0 & 0 & 0\\
1 & 1 & 1 & 1\\
10 & 2 & 2 & 2\\
11 & 3 & 3 & 3\\
100 & 4 & 4 & 4\\
101 & 5 & 5 & 5\\
110 & 6 & 6 & 6\\
111 & 7 & 7 & 7\\
1000 & 10 & 8 & 8\\
1001 & 11 & 9 & 9\\
1010 & 12 & 10 & A\\
1011 & 13 & 11 & B\\
1100 & 14 & 12 & C\\
1101 & 15 & 13 & D\\
1110 & 16 & 14 & E\\
1111 & 17 & 15 & F\\
10000 & 20 & 16 & 10\\
\end{tabular}
\end{center}


\subsection{Multiples and Lowest Common Multiples (LCM)}

\subsubsection{Multiples}

The multiple of a number is it added to itself iteratively.

The multiples of \(18\) for example are:

\([18,36,54,72,90,...]\)

And for \(24\):

\([24,48,72,96,120,...]\)

\subsubsection{Common multiples}

\subsubsection{Lowest common multiple}

The lowest common multiple of \(2\) numbers is again as the name suggests.

So \(LCM(18,24)=72\).


\section{Spectral Theory}
Spectral Theory is the study of spectrum (eigen-stuff) of operators, which is very important in boths maths and physics.
Fix an inner product space $V$.
Recall that an opeator $\alpha\in L(V)$ is self-adjoint if $\alpha=\alpha^\ast$.
\begin{lemma}
    A self-adjoint opeator $\alpha\in L(V)$ has real eigenvalues and eigenvectors with different eigenvalues are orthogonal.
\end{lemma}
\begin{proof}
    If $v\neq 0$ and $\alpha(v)=\lambda v$, then $\lambda\|v\|^2=\langle \lambda v,v\rangle=\langle\alpha(v),v\rangle=\langle v,\alpha(v)\rangle=\langle v,\lambda(v)\rangle=\bar\lambda\|v\|^2$, so $\lambda=\bar\lambda\implies\lambda\in\mathbb R$.\\
    Now if $v,w\neq 0,\lambda\neq\mu$ have $\alpha(v)=\lambda v,\alpha(w)=\mu w$, then $\lambda,\mu\in\mathbb R$ and hence $\lambda\langle v,w\rangle=\langle\alpha(v),w\rangle=\langle v,\alpha(w)\rangle=\langle v,\mu w\rangle=\bar\mu\langle v,w\rangle=\mu\langle v,w\rangle\implies\langle v,w\rangle=0$ since $\lambda\neq\mu$.
\end{proof}
\begin{theorem}[Spectral Theorem for Self-Adjoint Operators in Finite Dimensions]
    Let $V$ be a finite dimensional inner product space over $F=\mathbb R$ or $\mathbb C$ and $\alpha\in L(V)$ is self-adjoint.
    Then $V$ has an orthogonal basis of eigenvectors of $\alpha$.
\end{theorem}
Consequently, $\alpha$ can be diagonalised in an orthonormal basis of $V$.
\begin{proof}
    We proceed by induction on $n=\dim V$.
    $n=1$ is trivial.
    Now assume it is true for $n-1$.
    Let $\lambda$ be a root of $\chi_A$ (exists by FTA).
    Now $\lambda\in\mathbb R$ by the preceding lemma.
    Choose $v\in V\setminus\{0\}$ such that $\alpha(v)=\lambda v$.
    By normalising we can assume $v$ is unit.
    Let $U=\langle \{v\}\rangle^\perp\le V$.
    Then obviously $\alpha(U)\le U$ since $\langle\alpha(u),v\rangle=\langle u,\alpha(v)\rangle=\langle u,\lambda v\rangle=\bar\lambda\langle u,v\rangle=0$ for any $u\in U$.
    Also $\dim U=n-1$.
    Adding $v$ to the basis of $U$ in the induction hypothesis completes the induction process.
\end{proof}
\begin{corollary}
    If $V$ is a finite dimensional inner product space and $\alpha\in L(V)$ is self-adjoint, then $V$ is the orthogonal direct sum of all the eigenspaces of $\alpha$.
\end{corollary}
\begin{proof}
    Follows directly.
\end{proof}
Recall that $\alpha\in L(V)$ is an isometry iff $\alpha^\ast\circ\alpha=\operatorname{id}$.
It is called unitary when $V$ is a vector space over $\mathbb C$.
\begin{lemma}
    Let $V$ be a complex inner product space and $\alpha\in L(V)$ be unitary.
    Then:\\
    (i) All eigenvalues of $\alpha$ are in the unit circle $S^1$.\\
    (ii) Eigenvectors with distinct eigenvalues are orthogonal.
\end{lemma}
\begin{proof}
    If $v\neq 0,\alpha(v)=\lambda v$ we have $\lambda\neq 0$ as $\alpha\neq 0$ and that $\lambda\|v\|^2=\langle\alpha(v),v\rangle=\langle v,\alpha^{-1}(v)\rangle=\langle v, \lambda^{-1}(v)\rangle=\bar{\lambda}^{-1}\|v\|^2$, so $\lambda\bar\lambda=1\implies\lambda\in S^1$.\\
    Now if $v,w\neq 0$ and $\alpha(v)=\lambda v,\alpha(w)=\mu v$ for $\lambda\neq\mu$, then $\lambda\langle v,w\rangle=\langle\alpha(v),w\rangle=\langle v,\alpha^{-1}(w)\rangle=\langle v,\mu w\rangle=\bar{\mu}^{-1}\langle v,w\rangle=\mu\langle v,w\rangle\implies\langle v,w\rangle=0$.
\end{proof}
\begin{theorem}[Spectral Theorem for Unitary Operators in Finite Dimensions]
    Let $V$ be a finite dimensional inner product space over $F=\mathbb C$ and $\alpha\in L(V)$ is unitary.
    Then $V$ has an orthogonal basis of eigenvectors of $\alpha$.
\end{theorem}
Equivalently, $\alpha$ can be diagonalised in an orthonormal basis.
\begin{proof}
    Same idea as in the case for self-adjoint operators.
\end{proof}
Sadly, we cannot tell the same tale for real orthogonal matrices since it can have complex eigenvalues.


\section{Reparameterization \& Arc Length}
\noindent
VVFs can be reparameterized to trace out the same curve at different speeds by replacing $t$ in $\vec{r}(t)$ with any non-decreasing function of $t$.
This fact can come in handy to make the bounds of an integration problem more convenient.\\

\noindent
The integral of the derivative of a VVF gives the displacement vector because
\begin{equation*}
	\int_{a}^{b}{\vec{r^\prime}(t)\mathrm{d}t}=\vec{r}(b)-\vec{r}(a).
\end{equation*}
This is exactly like how $\text{veclvity} \cdot \text{time} = \text{displacment}$.

\noindent
If we integrate the magnitude of $\vec{r^\prime}(t)$, we can use the fact that $\text{distance} = \text{speed} \cdot \text{time}$ to find the arc length of $\vec{r}(t)$ as
\begin{equation*}
	s=\int{\norm{\vec{r^\prime}(t)}\mathrm{d}t}=\int{\sqrt{\left(\frac{\mathrm{d}x}{\mathrm{d}t}\right)^2+\left(\frac{\mathrm{d}y}{\mathrm{d}t}\right)^2+\left(\frac{\mathrm{d}z}{\mathrm{d}t}\right)^2}\mathrm{d}t}.
\end{equation*}
We can also write this as an arclength function,
\begin{equation*}
	s(t)=\int_{0}^{t}{\norm{\vec{r^{\prime}}(\tau)}\mathrm{d}\tau}.
\end{equation*}

\noindent
If we have a function
\begin{equation*}
	f(t)=s(t)=\int_{0}^{t}{\norm{\vec{r^{\prime}}(\tau)}\mathrm{d}\tau},
\end{equation*}
where $s$ is strictly increasing, then $f$ has an inverse by the horizontal line test.
That is, $t(s) = f^{-1}(s)$ exists and is also non-decreasing.
If we reparameterize $\vec{r}(t)$ to $\vec{r}(t(s))$, which is called the arc length parameterization, the parameterization will have a constant speed.


\section{Dispersion Relation Coefficients}\label{sec:coefficients}

In \eqref{laplacian} and \eqref{gamma determination} we give the definition and how to determine the $\gamma_s^{(\nstep)}$ coefficients that give us finite difference formulas.
This is done by matching the expansion of the cosine to the continuum dispersion
\begin{align}
	p^2
	&\mapsto
	- \frac{1}{\epsilon^2}\sum_{s=0}^{n_s} \gamma_s^{(n_s)} \cos(s p \epsilon)
	=
	\frac{1}{\epsilon^2}\sum_{s=0}^{n_s} \sum_{m=0}^\infty \gamma_s^{(n_s)}\frac{(-)^m}{(2m)!} (s p \epsilon)^{2m}
	\overset{!}{=}
	p^2 \left[ 1 + \mathcal O \left( (\epsilon p)^{2 n_s} \right) \right]
	\, .
\end{align}
Matching this expression order by order in $\epsilon p$ or equivalently $m \leq n_s$ effectively results in a matrix equation for the coefficients $\gamma_s^{(n_s)}$
\begin{align}
	A_{ms} &\equiv \frac{(-)^m}{(2m)!} s^{2m}
	\, , &
	\sum_{s=0}^{n_s} A_{m s} \gamma_s^{(n_s)}
	\overset{!}{=}
	v_m
	&=
	\begin{cases}
		1 & m=1 \\ 0 & \text{otherwise}
	\end{cases}
	\, , &
	\vec \gamma^{(n_s)}
	&= A^{-1} \vec v
\end{align}
Results for order $n_s \leq 4$ are displayed in \tabref{dispersion coefficients}.

\begin{table}[ht]
    \caption{Values for $\gamma_s^{(\nstep)}$ for a variety of different $\nstep$s that give the optimal approximation $\omega^{(\nstep)}(p,\epsilon) = (\epsilon p)^2\left[1+ \order{(\epsilon p)^{2 \nstep}}\right]$.}
    \label{tab:dispersion coefficients}
    \begin{tabular}{c | ccccc}
        $\gamma_s^{(\nstep)}$   &   $s=0$   &   $s=1$   &   $s=2$   &   $s=3$       &   $s=4$   \\ \hline
        $\nstep=1$              &   $2$     &   $-2$    &           &               &           \\
        $\nstep=2$              &   $5/2$   &   $-8/3$  &   $1/6$   &               &           \\
        $\nstep=3$              &   $49/18$ &   $-3$    &   $3/10$  &   $-1/45$     &           \\
        $\nstep=4$              &   $205/72$&   $-16/5$ &   $2/5$   &   $-16/315$   &   $1/280$
    \end{tabular}
\end{table}


\subsection{Localization}\label{subsec:localization}

\begin{definition}\label{def:ring_localization}\mcite[428]{Knapp2016BasicAlgebra}
  Let \( S \subseteq R \) be closed under multiplication.

  Define the following equivalence relation on \( R \times S \):
  \begin{equation*}
    (r, s) \cong (r', s') \iff \exists t \in S: t(rs' - sr') = 0.
  \end{equation*}

  Define the ring
  \begin{equation*}
    S^{-1} R \coloneqq R \times S / \cong
  \end{equation*}
  with operations inherited from \( R \) using the injection
  \begin{balign*}
     & \iota: R \to S^{-1} R        \\
     & \iota(r) \coloneqq [(r, 1)].
  \end{balign*}
\end{definition}

\begin{proposition}\label{thm:ring_localization_universal_property}\mcite[431]{Knapp2016BasicAlgebra}
  Let \( S \subseteq R \) be closed under multiplication. The ring \hyperref[def:ring_localization]{localization} \( S^{-1} R \) satisfies the following universal mapping property: if \( T \) is a nontrivial commutative unital ring and \( \varphi: R \to T \) is a unital ring homomorphism such that \( \varphi(S) \) are units in \( T \), there exists a unique ring homomorphism \( \hat \varphi \) such that the following diagram commutes:

  \begin{alignedeq}\label{thm:ring_localization_universal_property/diagram}
    \text{\todo{Add diagram}}\iffalse\begin{mplibcode}
      beginfig(1);
      input metapost/graphs;

      v1 := thelabel("$S^{-1} R$", origin);
      v2 := thelabel("$T$", (2, 0) scaled u);
      v3 := thelabel("$R$", (1, 1) scaled u);

      a1 := straight_arc(v3, v2);
      a2 := straight_arc(v3, v1);

      d1 := straight_arc(v1, v2);

      draw_vertices(v);
      draw_arcs(a);

      drawarrow d1 dotted;

      label.urt("$\varphi$", straight_arc_midpoint of a1);
      label.ulft("$\iota$", straight_arc_midpoint of a2);
      label.top("$\hat\varphi$", straight_arc_midpoint of d1);
      endfig;
    \end{mplibcode}\fi
  \end{alignedeq}
\end{proposition}

\begin{proposition}\label{thm:ring_localization_preserves_ideals}\mcite[432]{Knapp2016BasicAlgebra}
  If \( I \) is an ideal in \( R \), then
  \begin{equation*}
    S^{-1} I \coloneqq \{ s^{-1} x \mid s \in S, x \in I \}
  \end{equation*}
  is an ideal in the \hyperref[def:ring_localization]{localization} \( S^{-1} R \).
\end{proposition}

\begin{definition}\label{def:local_ring}
  If \( R \) has a unique \hyperref[def:maximal_ring_ideal]{maximal ideal}, we say that it is a \term{local ring}.
\end{definition}

\begin{proposition}\label{thm:localization_of_prime_is_local}\mcite[cor. 8.50]{Knapp2016BasicAlgebra}
  Fix a prime ideal \( P \). Its complement
  \begin{equation*}
    S \coloneqq \BbbZ \setminus P
  \end{equation*}
  is closed under multiplication and we can perform \hyperref[def:ring_localization]{localization}.

  In this case, the ring \( S^{-1} R \) is a local \hyperref[def:local_ring]{ring} and \( M \coloneqq S^{-1} P \) is its unique maximal ideal.
\end{proposition}

\begin{example}\label{ex:ring_localization}\mcite[430]{Knapp2016BasicAlgebra}
  Let \( p \) be a prime \hyperref[def:prime_number]{number} and \( P = \braket p \) be the corresponding \hyperref[def:prime_ring_ideal]{prime ideal}. Denote its complement by \( S \).

  The ring \( S^{-1} R \) then consists of all rational \hyperref[def:set_of_rational_numbers]{numbers} whose denominators are not divisible by \( p \).

  In particular, if \( p = 2 \), then \( S^{-1} R \) is the set of all rational numbers with odd denominators.
\end{example}

\begin{definition}\label{def:field_of_fractions}
  The \term{field of fractions} of \( R \) is defined as the \hyperref[def:ring_localization]{localization} of \( R \) by the set
  \begin{equation*}
    S \coloneqq R \setminus \{ 0 \}.
  \end{equation*}
\end{definition}
\begin{proof}
  This is indeed a field since all nonzero elements are invertible.
\end{proof}


The SHA-256 algorithm \cite{SHA_NIST_FIPS} is a secure hash algorithm that
receives input data of any size up to $2^{64}$ bits and computes an output of
256 bits. This output is called a message digest.
The SHA-256 algorithm has two main stages: preprocessing and hash
computation.

\subsection{Preprocessing}
\label{subsec:preprocessing}
The preprocessing stage pads the input data to obtain an input size multiple of
512 bits. Given a message $M$ of size $\lambda$ bits. The padding process
appends the bit "1" to the end of the message, followed by $\delta$ "0" bits
such that $\delta$ is the smallest positive integer that
solves~(\ref{eq:sha_padding}).

\begin{equation}
\lambda + 1 + \delta \equiv 448 \mod 512.
\label{eq:sha_padding}
\end{equation}

After the padded zeroes, the message is appended with the 64-bit representation
of the size of the original message $\lambda$. At the end of this process, the
padded message size is a multiple of 512 bits. The padded input splits into
blocks of 512 bits. Each blocks forms a set of sixteen words of 32 bits.

The preprocessing stage also sets the initial state for the hash value. The hash
state values are a set of eight 32 bit words. For the SHA-256 algorithm, the
initial values are the first 32 bits of the fractional part of the square root
of the first eight prime numbers.

\subsection{Hash Computation}
\label{subsec:hash_computation}

Figure~\ref{fig:hash_flowchart} presents the hash computation stage which
processes one message block at a time. For each iteration $i$, the hash stage
values $H_{0}^{(i+1)}, H_{1}^{(i+1)},..., H_{7}^{(i+1)}$ are updated using a
message schedule of sixty-four 32 bit words $W_0, W_1,..., W_{63}$, the
previous hash state values $H_{0}^{(i)}, H_{1}^{(i)},..., H_{7}^{(i)}$ and 64
constants $K_{0},K_{1},...,K_{63}$ of 32 bit each.

\begin{figure}[!htbp]
    \centerline{\includegraphics[width=14cm]{./figures/crypto_hashblock_flowchart.pdf}}
    \vspace{0cm}\caption{SHA-256 hash function flowchart.}
    \label{fig:hash_flowchart}
\end{figure}

The initial hash state values and the message block words come from the
preprocessing stage as described in section~\ref{subsec:preprocessing}.
The 64 constants $K_{t}$ are the 32 fractional bits of the cubic roots of the
first 64 prime numbers.

The sixty-four message schedule words are the sixteen 32 bit words from the
input message block plus 48 generated words. Each generated word $W_{t}$ is
computed by the operations presented in~(\ref{eq:gen_w}). The addition is
modulo $2^{32}$.

\begin{equation}
    W_{t} = \sigma_{1}(W_{t-2}) + W_{t-7} + \sigma_{0}(W_{t-15}) + W_{t-16}, \ \ \ 16 \leq t \leq 63.
\label{eq:gen_w}
\end{equation}

Where $\sigma_{0}()$ and $\sigma_{1}()$ functions are a set of logic operations
defined in~(\ref{eq:sigma_funcs}). $ROTR^{n}(x)$ is a rotate right $n$ bits
function and $SHR^{n}(x)$ is a right shift $n$ bits operation.

\begin{align}
    \begin{split}
        \sigma_{0}(x) &= ROTR^{7}(x) \oplus ROTR^{18}(x) \oplus SHR^{3}(x), \\
        \sigma_{1}(x) &= ROTR^{17}(x) \oplus ROTR^{19}(x) \oplus SHR^{10}(x).
    \end{split}
\label{eq:sigma_funcs}
\end{align}

The initial hash state values initialize a set of working variables $a, b, c,
d, e, f, g, h$:

\begin{align}
    \begin{split}
        a &= H_{0}^{(i-1)} \\
        b &= H_{1}^{(i-1)} \\
        c &= H_{2}^{(i-1)} \\
        d &= H_{3}^{(i-1)} \\
        e &= H_{4}^{(i-1)} \\
        f &= H_{5}^{(i-1)} \\
        g &= H_{6}^{(i-1)} \\
        h &= H_{7}^{(i-1)}.
    \end{split}
\label{eq:init_state}
\end{align}

The working variables are updated for 64 iterations ($ 0 \leq t \leq 63$),
following the algorithm in~(\ref{eq:hash_iter}). The functions $\Sigma_1(x)$,
$Ch(x,y,z)$, $\Sigma_0(x)$ and $Maj(x,y,z)$ are defined
in~(\ref{eq:Sigma_Ch_Maj}). $Ch(x,y,z)$ is a choice operation: if $x$ is 1, the
output is $z$, otherwise outputs $y$. $Maj(x,y,z)$ outputs the most common
value between the three inputs.

\begin{align}
    \begin{split}
        T_1 &= h + \Sigma_1(e) + Ch(e,f,g) + K_{t} + W_{t} \\
        T_2 &= h + \Sigma_0(a) + Maj(a,b,c) \\
        h &= g \\
        g &= f \\
        f &= e \\
        e &= d + T_1 \\
        d &= c \\
        c &= b \\
        b &= a \\
        a &= T_1 + T_2.
    \end{split}
\label{eq:hash_iter}
\end{align}

\begin{align}
    \begin{split}
        \Sigma_{0}(x) &= ROTR^{2}(x) \oplus ROTR^{13}(x) \oplus ROTR^{22}(x), \\
        \Sigma_{1}(x) &= ROTR^{6}(x) \oplus ROTR^{11}(x) \oplus ROTR^{25}(x), \\
        Ch(x,y,z) &= (x \land y) \oplus ( \neg x \land z), \\
        Maj(x,y,z) &= (x \land y) \oplus (x \land z) \oplus (y \land z).
    \end{split}
\label{eq:Sigma_Ch_Maj}
\end{align}

The results $H_{0}^{(i+1)}, H_{1}^{(i+1)},..., H_{7}^{(i+1)}$ of hashing
iteration $i$ are the final values of the working variables $a,..., h$.
The resulting hash state values of one iteration are the input of the next.
This process repeats for all message blocks. The hash state values at the final
iteration concatenated form the message digest.


\section{Vectors}
\noindent
A vector is a quantity with both direction and magnitude. One can think of it as a directed line segment. In multivariable calculus, we mostly will work with vectors in $\mathbb{R}^2$ and $\mathbb{R}^3$, but vectors can exist in other dimensions.\\

\noindent
Numerical (scalar) quantities have vector analogues, many of which show up in physics. Speed becomes velocity, distance becomes displacement, and mass becomes weight.\\

\noindent
Say we have a 2D vector, $\vec{v} = \langle v_x, v_y \rangle$.

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.5]{Images/backgroundReview/VectorAddition}
\end{figure}

\noindent
Its length, also called magnitude or norm, is notated $\norm{\vec{v}} = \sqrt{v_x^2+v_y^2}$. This pattern of the norm being equal to the square-root of the sum of the squares of the vector's components continues into higher dimensions.\\

\noindent
The angle a 2D vector forms with the horizontal axis is $\theta = \tan^{-1}{\left(\frac{v_y}{v_x}\right)}$. There is not a useful version of this formula in higher dimensions.\\
Using $\theta$ and $\norm{\vec{v}}$, we can see that $v_x = \norm{\vec{v}}\cos{\theta}$ and $v_y = \norm{\vec{v}}\sin{\theta}$.\\

\noindent
Vectors can be added and subtracted from each other in a way that the result is another vector. We do this numerically by adding the corresponding components of each vector.\\
For example, if $\vec{a} = \langle 1,3 \rangle$ and $\vec{b} = \langle 4,7 \rangle$, then $\vec{a}+\vec{b} = \langle 1+4, 3+7 \rangle = \langle 5,10 \rangle$.\\
and $\vec{b}-\vec{a} = \langle 4-1, 7-3, \rangle = \langle 3,4 \rangle$.

\noindent
Visually, you can think of $\vec{v}+\vec{w}$ as the vector connecting the tail of $\vec{v}$ with the tip of $\vec{w}$ where the tail of $\vec{v}$ is on the tip of $\vec{w}$.

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.33]{Images/backgroundReview/Parallelogram}
\end{figure}

\noindent
We can also multiply vectors by scalars and get another vector as a result. We do this by multiplying each component of the vector by the scalar. This has the effect of stretching or shrinking the vector and possibly changing the vector's direction if the scalar is negative.

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.5]{Images/backgroundReview/ScalarMultiples}
\end{figure}

\noindent
A unit vector is any vector with magnitude 1. Rather than using an arrow like for other vectors, unit vectors are notated with a carat (\^) over top, like $\hat{i}$, which is read as "i hat".\\
We an transform any vector with non-zero magnitude into a unit vector by dividing the vector by its norm. This normalized vector will point in the same direction as the original vector.\\

\noindent
It is common in mathematics for $\hat{i} = \langle 1,0,0 \rangle$ to be the unit vector in the x-direction, $\hat{j} = \langle 0,1,0 \rangle$ to be the unit vector in the y-direction, and $\hat{k} = \langle 0,0,1 \rangle$ to be the unit vector in the z-direction.\\
Together, $\hat{i}$, $\hat{j}$, and $\hat{k}$ are called the standard basis vectors because all other vectors can be written as linear combination of these.

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.33]{Images/backgroundReview/UnitVectors}
\end{figure}

\input{./backgroundReview/dotProducts}
\input{./backgroundReview/crossProducts}

\noindent
Now that we have defined the dot product and cross product, we can put the two together as the scalar triple product, which gives the volume of the parallelepiped spanned by $\vec{a}$, $\vec{b}$, and $\vec{c}$.\\
\begin{equation*}
	\vec{a}\cdot\left(\vec{b}\times\vec{c}\right) = \det\begin{bmatrix}
		a_1 & a_2 & a_2 \\
		b_1 & b_2 & b_3 \\
		c_1 & c_2 & c_3
	\end{bmatrix}
\end{equation*}

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.33]{Images/backgroundReview/Parallelipiped}
\end{figure}