The goal of $\beta_0$ is to capture the expansion of a displaced fluid
element due to the stratification of the atmosphere.  \maestro\ computes
$\beta_0$ as:
\begin{equation}\label{eq:beta_0}
\beta_0(r,t) = \rho_0 \exp\left (  \int_0^r  \frac{1}{\gammabar p_0} \frac{\partial p_0}{\partial r^\prime} dr^\prime \right )
\end{equation}

\section{Constant Composition}
Consider an isentropically stratified atmosphere, with a constant
composition as a function of $r$.  If you displace a parcel of fluid
upwards, it will expand adabatically and continue to rise until its
density matches the background density.  Even if $\gammabar$ is not
constant in $r$, following from the definition of $\beta_0$,
\begin{equation}
\frac{1}{\beta_0} \frac{d\beta_0}{dr} = \frac{1}{\gammabar p_0} \frac{dp_0}{dr}
\end{equation}
and the definition of $\Gamma_1$,
\begin{equation}
\Gamma_1 = \left . \frac{d \log p}{d \log \rho} \right |_s
\end{equation}
So, at constant entropy, from the definition of $\Gamma_1$, it must hold
that
\begin{equation}
\frac{1}{\rho} \frac{d \rho}{dr} = \frac{1}{\Gamma_1 p} \frac{d p}{dz} \enskip .
\end{equation}
Comparing to the defintion of $\beta_0$ then
\begin{equation}
\frac{1}{\beta_0} \frac{\beta_0}{dr} =\frac{1}{\gammabar p_0}\frac{dp_0}{dr} = \frac{1}{\rho_0} \frac{d\rho_0}{dr}  \enskip .
\end{equation}
Therefore, $\beta_0 = \rho_0$.

This means that if we have a constant composition and an
isentropically stratified atmosphere, as we displace a fluid element,
it will always remain neutrally buoyant.

\section{Composition Gradient}

If there is a change in composition with $r$, the situation is more
complicated.  Consider again an isentropically stratified atmosphere,
now with a composition gradient.  If you displace a parcel of fluid
upwards, it will rise.  If there are no processes that change the
composition (e.g.\ reactions), then the composition in the fluid
element will remain fixed.  As it rises, it will the ambient medium
will have a different composition that it has.  In this case, what is
the path to equilibrium?

\section{On the Effect of Chemical Potential}
\label{Sec:On the Affect of Chemical Potential}

In \maestro, we do things in an operator split fashion --- the hydro is
de-coupled from the burning.  This means that during the hydro parts
of the algorithm (where $\beta_0$ is used), the system is fixed in
chemical equilibrium. For completeness, however, here we describe the
effects of the species' chemical potentials, which were neglected in
the original derivation of $\beta_0$.  Note that similar terms appear
in the calculation of things such as specific heats,
which \textit{may} be important in the burning step --- there appears to
be very little about this in the literature, but everyone seems to
assume it makes little difference.

\subsection{Derivation of $\alpha$}
\label{Sec:Derivation of alpha}
In paper I, $\alpha$ is defined as
\begin{equation}\label{eq:alpha}
\alpha\equiv -\left(
\frac{(1-\rho h_p)p_T-\rho c_p}{\rho^2c_pp_\rho}\right)
\end{equation}
where
\[
h_p \equiv \left(\frac{\partial h}{\partial p}\right)_{T,X}, \quad
c_p \equiv \left(\frac{\partial h}{\partial T}\right)_{p,X}, \quad
p_T \equiv \left(\frac{\partial p}{\partial T}\right)_{\rho,X}, \quad
p_\rho \equiv \left(\frac{\partial p}{\partial \rho}\right)_{T,X}
\]
where the subscript $X$ means holding all $X_i$ constant.  In the
absence of reactions, the $X$ subscript can be dropped from all
derivatives and with the use of the equation of state $p=p(\rho,T)$,
$\alpha$ can be written as $\alpha=\alpha(\rho,T)$.  Such a system
without reactions and in thermal equilibrium could be either a pure
system of one species, or a system of many species in chemical (and
therefore \emph{thermodynamic}) equilibrium.  Cox and Giuli (hereafter
CG) call the former type of system a ``simple system'' and therefore
the latter a ``non-simple system'' in chemical equilibrium.  The
analysis in paper I that reduced \eqref{eq:alpha} to
\begin{equation}\label{eq:alpha_simp_no_rxn}
\alpha = \frac{1}{\Gamma_1p_0}
\end{equation}
used CG's discussion of the various adiabatic $\Gamma$'s.  However,
their discussion only pertains to ``simple systems'' or ``non-simple
systems'' in chemical equilibrium.  In general, nuclear reactions will
be important and therefore this analysis needs to be reformed.

Even in the presence of reactions, \eqref{eq:alpha} can be rewritten
as was done in paper I:
\begin{equation}\label{eq:alpha2}
\alpha = -\frac{1}{p\chi_\rho c_p}\left[\left(\frac{1}{\rho\chi_\rho}
- \frac{\rho e_\rho}{p\chi_\rho}\right)\frac{p\chi_T}{T} - c_p\right],
\end{equation}
where
\begin{align*}
\chi_{\rho} &\equiv \left(\frac{\partial\ln p}{\partial\ln\rho}
\right)_{T,X} \\
\chi_{T} &\equiv \left(\frac{\partial\ln p}{\partial\ln T}
\right)_{\rho,X}.
\end{align*}
Following the results of paper I, we want to find a relation
between $p\chi_\rho$ and $\Gamma_1$.

For an equation of state $p=p(\rho,T,X)$ we have
\[
d\ln p = \left(\frac{\partial\ln p}{\partial\ln\rho}\right)_{T,X}d\ln\rho +
\left(\frac{\partial\ln p}{\partial\ln T}\right)_{\rho,X}d\ln T +
\sum_i\left(\frac{\partial\ln p}{\partial\ln X_i}\right)_{\rho,T,(X_j,j
\neq i)} d\ln X_i.
\]
We define another logarithmic derivative
\begin{align*}
\chi_{X_{i}} &\equiv \left(\frac{\partial\ln p}{\partial\ln X_i}
\right)_{\rho,T,(X_j,j\neq i)}
\end{align*}
and therefore
\[
d\ln p = \chi_\rho \ d\ln\rho + \chi_T \ d\ln T + \sum_i \chi_{X_i}\
d\ln X_i.
\]
From here we get the general statement
\[
\frac{\partial\ln p}{\partial \ln \rho} = \chi_\rho +
\chi_T\frac{\partial \ln T}{\partial\ln \rho} +
\sum_i\chi_{X_i}\frac{\partial\ln X_i}{\partial\ln \rho}
\]
which must hold for an adiabatic process as well, and therefore we have
\begin{equation}\label{eq:gamma1}
  \Gamma_1 = \chi_\rho + \chi_T\left(\Gamma_3-1\right)
  + \sum_i\chi_{X_i}\Gamma_{4,i}
\end{equation}
where we use CG's definition of $\Gamma_1$ and $\Gamma_3$ and introduce a
fourth gamma function:
\[
\Gamma_1 \equiv \left(
\frac{\partial \ln p}{\partial \ln \rho}\right)_{\text{AD}},\quad
\Gamma_3-1\equiv \left(
\frac{\partial \ln T}{\partial \ln \rho}\right)_{\text{AD}},\quad
\Gamma_{4,i}\equiv \left(
\frac{\partial\ln X_i}{\partial\ln\rho}\right)_{\text{AD}},
\]
where the subscript AD means along an adiabat.  We now derive an expression
for $\Gamma_3$.

The first law of thermodynamics can be written as
\[
dQ = dE + pdV - \sum_i\mu_idN_i
\]
where $\mu_i=\left(
\frac{\partial E}{\partial N_i}\right)_{\text{AD},\rho,(N_j,j\neq i)}$ is
the chemical potential; or per unit mass we have
\begin{align*}
  dq &= de - \frac{p}{\rho^2}d\rho - \sum_i\mu_id
  \left(\frac{n_i}{\rho}\right)\\
  &= de - \frac{p}{\rho^2}d\rho - \sum_i
  \left(
  \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}dX_i
\end{align*}
where we have used $X_i \equiv \rho_i/\rho = A_in_i/\rho N_\text{A}$
and the chemical potential has been replaced with
$\mu_i = \frac{A_i}{N_\text{A}}\left(\frac{\partial e}{\partial X_i}
  \right)_{\rho,\text{AD},(X_j,j\neq i)}$.
Using this and expressing the specific internal energy as $e=e(\rho,T,X)$
we then have
\[
dq = c_vdT +
\left[\left(\frac{\partial e}{\partial \rho}\right)_{T,X} -\frac{p}{\rho^2}
  \right]d\rho +
\sum_i\left[
  \left(\frac{\partial e}{\partial X_i}\right)_{\rho,T,(X_j,j\neq i)} -
  \left(\frac{\partial e}{\partial X_i}
  \right)_{\rho,\text{AD},(X_j,j\neq i)}\right]dX_i
\]
and
\begin{align}\label{eq:gamma3_first}
\left(\frac{d\ln T}{d\ln\rho}\right)_\text{AD} \equiv \Gamma_3-1
&= \frac{1}{c_vT}\left[
\frac{p}{\rho} - \left(\frac{\partial e}{\partial\ln\rho}\right)_{T,X} +
\right.{}\nonumber\\
&\qquad\qquad  \left.\sum_i \left[
    \left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    -
    \left(\frac{\partial e}{\partial X_i}\right)_{\rho,T,(X_j,j\neq i)}
    \right]X_i\Gamma_{4,i}\right]
\end{align}

Now we need to evaluate $\left(\partial e/\partial \ln\rho\right)_{T,X}$.
Again using the first law and the fact that $ds=dq/T$ is an exact
differential (i.e. mixed derivatives are equal) we have
\begin{align}\label{eq:dedlnrho}
  \left(
  \frac{\partial}{\partial\rho}\left[\frac{c_v}{T}\right]\right)_{T,X} &=
  \left(\frac{\partial}{\partial T}\left[\frac{1}{T}
    \left(\frac{\partial e}{\partial\rho}\right)_{T,X} - \frac{p}{T\rho^2}
    \right]\right)_{\rho,X}{}\nonumber\\
  \frac{1}{T}\left(\frac{\partial}{\partial\rho}\left(
  \frac{\partial e}{\partial T}\right)_{\rho,X}\right)_{T,X} &=
  -\frac{1}{T^2}\left(\frac{\partial e}{\partial\rho}\right)_{T,X} +
  \frac{1}{T}\left(\frac{\partial}{\partial T}\left(
  \frac{\partial e}{\partial\rho}\right)_{T,X}\right)_{\rho,X}
  +\frac{p}{T^2\rho^2} -
  \frac{1}{T\rho^2}\left(\frac{\partial p}{\partial T}\right)_{\rho,X}
  {}\nonumber\\
  \therefore\quad \left(\frac{\partial e}{\partial\ln \rho}\right)_{T,X} &=
  \frac{p}{\rho}\left(1-\chi_T\right),
\end{align}
exactly the same result if we were to exclude species information.
Simlarly, we can find an expression for the derivative of energy with
respect to composition
\begin{align*}
  \left(\frac{\partial}{\partial X_i}\left[
    \frac{c_v}{T}\right]\right)_{\rho,T,(X_j,j\neq i)} &=
  \left(\frac{\partial}{\partial T}\left[
    \frac{1}{T}\left(\frac{\partial e}{\partial X_i}
      \right)_{\rho,T,(X_j,j\neq i)} - \frac{1}{T}\left(
      \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right]\right)_{\rho,X}\\
  \frac{1}{T}\left(\frac{\partial }{\partial X_i}\left(
  \frac{\partial e}{\partial T}\right)_{\rho,X}\right)_{\rho,T,(X_j,j\neq i)}
  &= \frac{1}{T^2}\left[\left(\frac{\partial e}{\partial X_i}
    \right)_{\rho,\text{AD},(X_j,j\neq i)} -
    \left(\frac{\partial e}{\partial X_i}\right)_{\rho,T,(X_j,j\neq i)}
    \right] + \\
  &\ \ \ \ \ \frac{1}{T}\left[
    \left(\frac{\partial}{\partial T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,T,(X_j,j\neq i)}
    \right)_{\rho,X} -
    \left(\frac{\partial }{\partial T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right)_{\rho,X}\right]\\
  \therefore\quad
  \left(\frac{\partial e}{\partial X_i}\right)_{\rho,T,(X_j,j\neq i)} &=
  \left(\frac{\partial e}{\partial X_i}
  \right)_{\rho,\text{AD},(X_j,j\neq i)} - \left(
  \frac{\partial}{\partial\ln T}\left(
  \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
  \right)_{\rho,X}.
\end{align*}
Plugging these back into \eqref{eq:gamma3_first} we have
\begin{equation}\label{eq:gamma3_second}
  \Gamma_3-1 = \frac{1}{c_vT}\left[\frac{p}{\rho}\chi_T +\sum_i
    \left(\frac{\partial}{\partial\ln T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right)_{\rho,X}X_i
    \Gamma_{4,i}\right],
\end{equation}
or
\begin{equation}\label{eq:cv}
  c_v = \frac{1}{T(\Gamma_3-1)}\left[\frac{p}{\rho}\chi_T +\sum_i
    \left(\frac{\partial}{\partial\ln T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right)_{\rho,X}X_i
    \Gamma_{4,i}\right].
\end{equation}

We can obtain an expression for the specific heat at constant pressure
from the enthalpy
\begin{align*}
  c_p \equiv \left(\frac{\partial h}{\partial T}\right)_{p,X} &=
  \left(\frac{\partial e}{\partial T}\right)_{p,X} - \frac{p}{\rho^2}
  \left(\frac{\partial \rho}{\partial T}\right)_{p,X}\\
  &= \left(\frac{\partial e}{\partial T}\right)_{p,X} + \frac{p}{\rho^2}
  \left(\frac{\partial p}{\partial T}\right)_{\rho,X}
  \left(\frac{\partial \rho}{\partial p}\right)_{T,X}\\
  &=\left(\frac{\partial e}{\partial T}\right)_{p,X} + \frac{p}{\rho T}
  \frac{\chi_t}{\chi_\rho}.
\end{align*}
The first term on the rhs can be obtained from writing $e=e(p,T,X)$ and
$p=p(\rho,T,X)$:
\begin{align*}
  de &= \left(\frac{\partial e}{\partial p}\right)_{T,X}dp
  + \left(\frac{\partial e}{\partial T}\right)_{p,X}dT +
  \sum_i \left(\frac{\partial e}{\partial X_i}\right)_{p,T,(X_j,j\neq i)}
  dX_i\\
  dp &= \left(\frac{\partial p}{\partial \rho}\right)_{T,X}d\rho +
  \left(\frac{\partial p}{\partial T}\right)_{\rho,X}dT + \sum_i
  \left(\frac{\partial p}{\partial X_i}\right)_{\rho,T,(X_j,j\neq i)}dX_i\\
  \therefore \ \left(\frac{\partial e}{\partial T}\right)_{\rho,X} &=
  \left(\frac{\partial e}{\partial p}\right)_{T,X}
  \left(\frac{\partial p}{\partial T}\right)_{\rho,X} +
  \left(\frac{\partial e}{\partial T}\right)_{p,X}\\
  \Rightarrow \ \left(\frac{\partial e}{\partial T}\right)_{p,X} &= c_v -
  \left(\frac{\partial e}{\partial \rho}\right)_{T,X}
  \left(\frac{\partial \rho}{\partial p}\right)_{T,X}
  \left(\frac{\partial p}{\partial T}\right)_{\rho,X}\\
  &= c_v - \frac{p\chi_T}{\rho T\chi_\rho}\left(1-\chi_T\right)
\end{align*}
and
\[
c_p = \frac{p}{\rho T}\frac{\chi_T^2}{\chi_\rho} + c_v
\]
Dividing this by \eqref{eq:cv} and using the relation between the
$\Gamma$'s, \eqref{eq:gamma1}, we then have
\begin{align}\label{eq:pchirho}
  \gamma \equiv \frac{c_p}{c_v} &= 1 + \frac{p(\Gamma_3-1)}{\rho }
  \frac{\chi_T^2}{\chi_\rho}\left[\frac{p}{\rho}\chi_T +\sum_i
    \left(\frac{\partial}{\partial\ln T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right)_{\rho,X}X_i
    \Gamma_{4,i}\right]^{-1}{}\nonumber\\
  &= 1 + \frac{p\chi_T\left(\Gamma_1 - \chi_\rho -
    \sum_i \chi_{X_i}\Gamma_{4,i}\right)}{p\chi_\rho\chi_T + \rho
    \chi_\rho\sum_i \left(
    \frac{\partial}{\partial \ln T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right)_{\rho,X}X_i\Gamma_{4,i}}{}\nonumber\\
  &= \frac{p\chi_T\Gamma_1 + \sum_i \left[\rho\chi_\rho\left(
    \frac{\partial}{\partial \ln T}\left(\frac{\partial e}{\partial X_i}
    \right)_{\rho,\text{AD},(X_j,j\neq i)}\right)_{\rho,X}X_i - p\chi_T
    \chi_{X_i}\right]\Gamma_{4,i}}{p\chi_\rho\chi_T + \rho
    \chi_\rho\sum_i \left(
    \frac{\partial}{\partial \ln T}\left(
    \frac{\partial e}{\partial X_i}\right)_{\rho,\text{AD},(X_j,j\neq i)}
    \right)_{\rho,X}X_i\Gamma_{4,i}}{}\nonumber\\
  \Rightarrow p\chi_\rho &= \frac{1}{\chi_T\gamma}\left[p\chi_T\Gamma_1 +
    \sum_i \left[\rho\chi_\rho\left(1-\gamma\right)\left(
      \frac{\partial}{\partial \ln T}\left(\frac{\partial e}{\partial X_i}
      \right)_{\rho,\text{AD},(X_j,j\neq i)}\right)_{\rho,X}X_i - p\chi_T
      \chi_{X_i}\right]\Gamma_{4,i}\right].
\end{align}

Plugging \eqref{eq:pchirho} into \eqref{eq:alpha2} and rewriting the
partial derivative of $e$ with the help of \eqref{eq:dedlnrho} we have
\begin{align*}
\alpha &= -\frac{1}{p\chi_\rho c_p}\left[\left(\frac{1}{\rho\chi_\rho}
  - \frac{\rho e_\rho}{p\chi_\rho}\right)\frac{p\chi_T}{T} - c_p\right] \\
&=\frac{\gamma}{c_p}\frac{c_p\chi_T + \left(\rho
  \left(\frac{\partial e}{\partial\ln\rho}\right)_{T,X}-p\right)
  \frac{\chi_T^2}{T\rho\chi_\rho}}
{p\chi_T\Gamma_1 +
    \sum_i \left[\rho\chi_\rho\left(1-\gamma\right)\left(
      \frac{\partial}{\partial \ln T}\left(\frac{\partial e}{\partial X_i}
      \right)_{\rho,\text{AD},(X_j,j\neq i)}\right)_{\rho,X}X_i - p\chi_T
      \chi_{X_i}\right]\Gamma_{4,i}}\\
&=\frac{\gamma}{\Gamma_1 p c_p}\left[\frac{c_p - \frac{p\chi_T^2}
    {T\rho\chi_\rho}}
  {1 + \sum_i \left[\frac{\rho\chi_\rho}{p\chi_T}
      \left(1-\gamma\right)\left(
      \frac{\partial}{\partial \ln T}\left(\frac{\partial e}{\partial X_i}
      \right)_{\rho,\text{AD},(X_j,j\neq i)}\right)_{\rho,X}X_i -
      \chi_{X_i}\right]\frac{\Gamma_{4,i}}{\Gamma_1}}\right]\\
&=\left(\frac{1}{\Gamma_1p}\right)
\left[1 + \sum_i \left[\frac{\rho\chi_\rho}{p\chi_T}
    \left(1-\gamma\right)\left(
    \frac{\partial}{\partial \ln T}\left(\frac{\partial e}{\partial X_i}
    \right)_{\rho,\text{AD},(X_j,j\neq i)}\right)_{\rho,X}X_i -
    \chi_{X_i}\right]\frac{\Gamma_{4,i}}{\Gamma_1}\right]^{-1}\\
\end{align*}
\[\boxed{
  \alpha = \frac{1}{\Gamma_1p}\left[1 + \sum_i\left[\frac{\rho^2p_\rho}
      {pp_T}(1-\gamma)\frac{N_\text{A}}{A_i}
      \left(\frac{\partial\mu_i}{\partial T}\right)_{\rho,X}X_i -
      \chi_{X_i}\right]\frac{\Gamma_{4,i}}{\Gamma_1}\right]^{-1}
}
\]

\subsection{Recalling Derivation of $\beta_0$}
\label{Recalling Derivation of beta0}
Recall from paper I that $\beta_0$ was derived from the equation
\begin{equation}
\nabla\cdot\mathbf{U} + \alpha\mathbf{U}\cdot\nabla p_0 = \tilde{S}
\end{equation}
in such a fashion that we ended up with an equation of the form
\begin{equation}\label{eq:beta constraint}
\nabla\cdot\left(\beta_0(r)\mathbf{U}\right) = \beta_0\tilde{S}.
\end{equation}
The derivation in Appendix B of paper I for a $\beta_0$ that
satisfies \eqref{eq:beta constraint} automatically assumed $\alpha
= \left(\Gamma_{1_0}p_0\right)^{-1}$.  This would have to be modified
with the above derivation of $\alpha$ to be correct in a non-operator
split fashion.


%
% section for MDP
%

\section{Markov Decision Process}

\subsection{Definition}

A \cindex{Markov decision process} (MDP) is a Markov reward process with decisions. It is an environment in which all stated are Markov. It is a tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{R}, \mathcal{P}, \gamma \rangle$:

\begin{itemize}
	\item $\mathcal{S}$ is a finite set of states.
	\item $\mathcal{A}$ is a finite set of actions.
	\item $\mathcal{R}$ is a reward function.
	\item $\mathcal{P}$ is a state transition probability matrix.
	\item $\gamma \in [0,1]$ is a discount factor.
\end{itemize}

\subsection{Goals and Equations}

\subsubsection{Environment}

\cindex{environment} has \cindex{state} $S_t \in \mathcal{S}$ and generate \cindex{reward} $R_{t+1} \in \mathbb{R}$. Anything that cannot be changed arbitrarily by the agent is part of environment. The goal of reinforcement learning is to maximize expected value of cumulative sum of received scalar reward.

The reward signal should be chosen so it will not affect how agent act.

\subsubsection{Agent}

\cindex{agent} has \cindex{action} $A_t \in \mathcal{A}(S_t)$ and \cindex{observation} $O_t$ of state $S_t$. Agent may know everything about how the environment works but still unable to solve problem, such as the Rubik cube puzzle.

\subsubsection{State and Reward}

The probability of next state and reward is :
\begin{equation}
	p(s^\prime,r|s,a) = \mathbb{P}\{S_t=s',R_t=r|S_{t-1}=s,A_{t-1}=a\}
\end{equation}

The \cindex{state-transition probabilities} is :
\begin{equation}
	p(s'|s,a)=P\{S_t=s'|S_{t-1}=s,A_{t-1} = a \}=\sum_{r \in \mathbb{R}} p(s',r|s,a)
\end{equation}

Usually the state and reward probability will be treated as independent, so their formulas are:

\begin{equation}
	\mathcal{P}_{s,s'}^a = \mathbb{P}[S_{t+1}=s' | S_t = s, A_t = a]
\end{equation}

\begin{equation}
	\mathcal{P}_{s,s'}^{\pi} = \sum_{a \in A(s)} \pi(a|s) \mathcal{P}_{s,s'}^a
\end{equation}

\begin{equation}
	\mathcal{R}_s^a = \mathbb{E}[R_{t+1}|S_t =s, A_t=a]
\end{equation}

\begin{equation}
	\mathcal{R}_s^\pi = \sum_{a \in A(s)} \pi(a|s) \mathcal{R}_s^a
\end{equation}

The expected reward of state-action pair is:
\begin{equation}
	r(s,a) = \mathbb{E}[R_t|S_{t-1}=s,A_{t-1}=a]= \sum_{r \in \mathbb{R}} \sum_{s' \in \mathbf{S}} p(s',r|s,a)
\end{equation}

\subsubsection{Episode}
An \cindex{episode} is an order sequence of $S_0, A_0,R_1,S_1,A_1, \dots, R_n, S_n$ in MDP. So an \cindex{episode} will always end.

\cindex{continuing task} is a list of actions that never terminate.

Episode task can be converted to continuing task by appending infinite \cindex{absorbing state}.

\subsubsection{Goal}
The \cindex{expected return} is the sum of rewards in the episode:
\begin{equation}
	G_t=R_{t+1}+R_{t+2} + R_{t+3} +\dots + R_T
\end{equation}

\begin{itemize}
	\item \cindex{terminal state}: $R_T$
	\item $S$: All non-terminal states
	\item $S^+$: all terminal and non-terminal states
\end{itemize}

\cindex{discounted return} for continuing task is defined as:
\begin{equation}
	\begin{aligned}
		G_t&=R_{t+1}+\gamma R_{t+2} + \gamma^2 R_{t+3} + \dots  \\
		&= \sum_{k=0}^\infty \gamma^k R_{t+k_1} \\
		&= R_{t+1} + \gamma G_{t+1}
	\end{aligned}
\end{equation}

where $0 \leq \gamma \leq 1$.

\subsubsection{Policy}

A stochastic policy is a probability of selecting next possible action:

\begin{equation}
	\pi(a|s)=\mathbb{P}[A_t = a | S_t = s]
\end{equation}

\subsubsection{Value Function}

The \cindex{state-value} function $V_\pi$ for policy $\pi$ is:

\begin{equation}
	\begin{aligned}
		V_{\pi}(s) &= \mathbb{E}_{\pi}[G_t|S_t=s] \\
		&=\mathbb{E}[R_{t+1} + \gamma V_{\pi}(S_{t+1})| S_t = s]
	\end{aligned}
\end{equation}

The \cindex{action-value} function $q_\pi$ for policy $\pi$ is:

\begin{equation}
	\begin{aligned}
		q_{\pi}(s,a) &= \mathbb{E}_{\pi}[G_t|S_t=s,A_t=a] \\
					&= \mathbb{E}[R_{t+1} + \gamma q_{\pi}(S_{t+1}, A_{t+1})| S_t = s, A_t = a]
	\end{aligned}
\end{equation}

\subsubsection{Bellman Equation}

Bellman equation (\cindex{backup} process) is an expansion of value function:

\begin{equation}\label{bellman:v}
	\begin{aligned}
		V_{\pi}(s) &= \sum_{a \in \mathcal{A}(s)} \pi(a|s) q_{\pi}(s,a) \\
		&= \sum_{a \in \mathcal{A}(s)} \pi(a|s) \left( \mathcal{R}_s^a + \gamma \sum_{s' \in \mathcal{S}} \mathcal{P}_{s,s'}^a V(S') \right)
	\end{aligned}
\end{equation}

\begin{equation}
	\begin{aligned}
		q_{\pi}(s,a) &= R_s^a + \gamma \sum_{s' \in \mathcal{S}} \mathcal{P}_{s,s'}^a V_{\pi}(s') \\
		&= R_s^a + \gamma \sum_{s' \in \mathcal{S}} \mathcal{P}_{s,s'}^a \left( \sum_{a', s'} \pi(a'|s') q_{\pi}(s',a') \right )
	\end{aligned}
\end{equation}

\subsubsection{Bellman Equation Solution}

Formula (\ref{bellman:v}) is for a single state. Let
\begin{equation}
	V_{\pi} = \left [\begin{matrix}
 	V_{\pi}(s_1) \\
 	\vdots \\
 	V_{\pi}(s_n)
 \end{matrix} \right ]
\end{equation}

Formula (\ref{bellman:v}) now becomes:

\begin{equation}
	\mathbf{V}_{\pi}=\mathbf{R}^{\pi} + \gamma \mathbf{P}^{\pi} \mathbf{V}_{\pi}
\end{equation}

So the solution to \cindex{Bellman Equation} is:

\begin{equation}
	\mathbf{V}_{\pi}=(\mathbf{I} - \gamma \mathbf{P}^{\pi})^{-1}\mathbf{R}^{\pi}
\end{equation}

It is a fixed point solution to formula (\ref{bellman:v}).

\subsection{Optimal Policy}

\subsubsection{Policy Partial Order}

$\pi \geq \pi^{\prime}$ if $\forall s \in \mathcal{S}, V_{\pi} (s) \geq V_{\pi^{\prime}} (s)$ .

For MDP all optimal solution share the same value function $V_*$ and include at least one deterministic policy $\pi_*$.

\subsubsection{Optimal State-value Function}

The optimal state-value function $V_*$ is defined as:

\begin{equation}
	\begin{aligned}
		V_*(s) &= \underset{\pi}{\max} \ V_{\pi}(s) \\
		&= \underset{a}{\max}\ q_* (s, a) \\
		&= \underset{a}{\max}\ \left( \mathcal{R}_s^a + \gamma \sum_{s' \in \mathcal{S}} \mathcal{P}_{s,s'}^a V_*(s') \right )
	\end{aligned}
\end{equation}

\subsubsection{Optimal Action-value Function}

The optimal action-value function $q_*$ is defined as:

\begin{equation}
	\begin{aligned}
		q_*(s,a)&=\underset{\pi}{\max}\ q_{\pi}(s,a)\\
		&=\mathbb{E}[R_{t+1}+\gamma V_*(S_{t+1})|S_t=s, A_t=a]\\
		&=\mathcal{R}_s^a+\gamma \sum_{s' \in \mathcal{S}} \mathcal{P}_{s,s'}^a V_*(s')\\
		&=\mathcal{R}_s^a+\gamma \sum_{s' \in \mathcal{S}} \mathcal{P}_{s,s'}^a \ \underset{a'}{\max}\  q_*(s',a')
	\end{aligned}
\end{equation}

Here $\mathcal{R}_s^a$ is an expectation. In model-free learning and controlling it is the sample return from environment.

\subsubsection{Optimal Policy from Action Value Function} Once the optimal action value function is known, we can calculate the optimal policy by:

\begin{equation}\label{optimal:policy}
	\pi_*(a|s) =
	\begin{cases}
		1& \text{, if $a = \underset{a \in \mathcal{A}(s)}{\text{argmax}}\ q_*(s,a)$}\\
		0& \text{, else}
	\end{cases}
\end{equation}

So the optimal policy is a greedy algorithm.

Compared with $V_*$, $q_*$ is better because it does not need to do one step lookahead.

\subsubsection{Reason for Complex Algorithms}

There is no closed form for optimal Bellman policy equation, so many iterative solution exists:
\begin{itemize}
	\item value iteration
	\item policy iteration
	\item $Q$-learning
	\item Sarsa
\end{itemize}


The previous chapter began an introduction to the problem of robotic perception, which consists of tasks related to sensing and understanding the robot's own movements as well as the environment in which it operates\cite{SiegwartNourbakhshEtAl2011}. This chapter continues that discussion by diving more deeply into one of the most powerful and challenging tools in robotic perception: computer vision. In particular, this chapter will focus on some of the fundamental mathematical tools for calibrating cameras and processing their images to extract some useful information about the scene\cite{ForsythPonce2011}\cite{HartleyZisserman2002}\nocite{Tsai1987}.
\nocite{Bradski2000}

\notessection{Camera Models and Calibration}
As was discussed in the previous chapter, cameras provide a crucial sensing modality in the context of robotics. This is generally due to the fact that images inherently contain an enormous amount of information about the environment. However, while images do contain a lot of information, extracting the information that is relevant to the robot is quite challenging. One of the most basic tasks related to image processing is determining how a particular point in the scene maps to a point in the camera image, which is sometimes referred to as \textit{perspective projection}. Last chapter, the \textit{pinhole camera model} and the \textit{thin lens model} were presented, and in this chapter the pinhole camera model is leveraged to further explore perspective projection\footnote{All results also hold under the thin lens model, assuming the camera is focused at $\infty$.}.

\begin{figure}[ht]
\includegraphics[width=0.8\textwidth]{tex/figs/ch08_figs/pinholecamera2.png}
\centering
\caption{Graphical representation of the pinhole camera model. In this model the point $O_C$ is the camera center, $c$ is the image center, and $f$ is the focal length of the camera. It is assumed that all light rays from point $P$ in the scene pass through point $O_C$ and are captured on the image plane at point $p$.}
\label{fig:pinhole_cam}
\end{figure}

\subsection{Perspective Projection}
The pinhole camera model, shown graphically in Figure \ref{fig:pinhole_cam}, can be used to mathematically define relationships between points $P$ in the scene and points $p$ on the image plane. Notice that any point $P$ in the scene can represented in two ways: in camera frame coordinates (denoted as $P_C$) or in world frame coordinates (denoted as $P_W$). The overall objective of this section is to find derive a mathematical model that can be used to map a point $P_W$ expressed in world frame coordinates to a point $p$ on the image plane. To accomplish this two transformations are combined together, namely a transformation of $P$ from world frame coordinates to camera frame coordinates ($P_W$ to $P_C$) and a transformation from camera coordinates to image coordinates ($P_C$ to $p$).

\subsubsection{Mapping Camera Frame Coordinates to Image Coordinates ($P_C \xrightarrow{} p$)}
The first step considered is the mapping from a point in the scene expressed in camera frame coordinates, $P_C$, to the corresponding point on the image plane, $p$, using the pinhole camera model. Recall from the previous chapter the pinhole camera equations:
\begin{equation} \label{eq:PC2xy}
    x=f\frac{X_C}{Z_C}, \quad  y = f\frac{Y_C}{Z_C},
\end{equation}
where $P_C = (X_C, Y_C, Z_C)$, $p = (x, y)$, and $f$ is the focal length of the pinhole camera\footnote{The $z$ term of $p$ is generally not included simply because $z=f$ is a fixed value.}.

Note that the quantities $x$ and $y$ are coordinates in the \textit{camera frame}, but it is often desirable to express the point $p$ in terms of \textit{pixel coordinates}. However, pixel coordinates are generally defined with respect to a reference frame in the lower corner of the image plane (to avoid negative coordinates). This new reference frame is shown in Figure \ref{fig:camera_coordinates}, where the image center $c$ is defined in this new reference frame with coordinates $(\tilde{x}_0, \tilde{y}_0)$, where $\tilde{(\cdot)}$ is the notation used to denote a coordinate with respect to this new reference frame.
\begin{figure}[ht]
\includegraphics[width=0.55\textwidth]{tex/figs/ch08_figs/imageframe.png}
\centering
\caption{A new reference frame with coordinates denoted by $\tilde{(\cdot)}$  is defined with its origin in the lower corner of the image plane. The image center coordinates in this new frame are denoted $(\tilde{x}_0, \tilde{y}_0)$.}
\label{fig:camera_coordinates}
\end{figure}
In this new reference frame, the point $P_C$ gets mapped to the coordinates $(\tilde{x}, \tilde{y})$ by:
\begin{equation} \label{eq:PC2xtyt}
    \tilde{x} = f\frac{X_C}{Z_C} + \tilde{x}_0, \quad  \tilde{y} = f\frac{Y_C}{Z_C} + \tilde{y}_0.
\end{equation}
Finally, these new coordinates can be mapped to pixel coordinates if the number of pixels per unit distance are known. In particular, the point $P_C$ is mapped to pixel coordinates $(u,v)$ by:
\begin{equation} \label{eq:PC2uv}
   u = \alpha \frac{X_C}{Z_C} + u_0, \quad  v = \beta \frac{Y_C}{Z_C} + v_0,
\end{equation}
where $\alpha = k_xf$, $u_0 = k_x \tilde{x}_0$, $\beta = k_y f$, $v_0 = k_y \tilde{y}_0$, and $k_x$ and $k_y$ are the number of pixels per unit distance in image coordinates.

\paragraph{Homogeneous Coordinates:}
Note that the transformation from the point $P_C$ in camera frame coordinates to $p$ in pixel coordinates given by \eqref{eq:PC2uv} is not linear. However, this transformation can be represented as a linear mapping\footnote{Expressing the perspective projection as a linear map will simplify the mathematics later on.} through an additional change of coordinates. In particular, the points $P_C$ and $p$ will be expressed in \textit{homogeneous coordinates}.

For a 2D point $(x_1,x_2)$ or a 3D point $(x_1,x_2,x_3)$ in Euclidean space, the point can be represented in homogeneous coordinates by the transformation:
\begin{equation}
(x_1,\:x_2) \implies (\alpha x_1, \:\alpha x_2, \:\alpha), \quad \text{and} \quad (x_1,\:x_2,\:x_3) \implies (\alpha x_1,\:\alpha x_2, \:\alpha x_3, \: \alpha),
\end{equation}
for any $\alpha \neq 0$. These new coordinates are called homogeneous coordinates because the scaling factor $\alpha$ can be chosen arbitrarily as long as $\alpha \neq 0$. A set of homogeneous coordinates can then be transformed back by:
\begin{equation}
(y_1, \: y_2, \:y_3) \implies \Big(\frac{y_1}{y_3}, \: \frac{y_2}{y_3}\Big), \quad \text{and} \quad (y_1,\:y_2,\:y_3,\:y_4) \implies \Big(\frac{y_1}{y_4}, \: \frac{y_2}{y_4}, \: \frac{y_3}{y_4}\Big).
\end{equation}
To denote when a point is described in homogeneous coordinates the superscript $h$ will be used. For example, the point $P_C = (X_C, Y_C, Z_C)$ in camera frame coordinates can be expressed by:
\begin{equation*}
    P_C^h = (X_C, Y_C, Z_C, 1),
\end{equation*}
by choosing $\alpha = 1$, and the point $p = (u,v)$ in pixel coordinates can be expressed in homogeneous coordinates by:
\begin{equation*}
    p^h = (Z_C u, \: Z_C v, \:Z_C) = (\alpha X_C + u_0 Z_C, \: \beta Y_C + v_0 Z_C),
\end{equation*}
by choosing $\alpha = Z_C$ and substituting the expressions \eqref{eq:PC2uv}. With the expression of these points in homogeneous coordinates it can be seen that their relationship is transformed from the nonlinear relationship \eqref{eq:PC2uv} to the \textit{linear} relationship:
\begin{equation}
\begin{bmatrix}
\alpha & 0 & u_{0} & 0 \\
0 & \beta & v_{0} & 0  \\
0 & 0 & 1 & 0
\end{bmatrix}
\begin{pmatrix}
X_{c} \\
Y_{c} \\
Z_{c} \\
1
\end{pmatrix}
=
\begin{pmatrix}
\alpha X_{c} + u_{0}Z_{c} \\
\beta Y_{c} + v_{0}Z_{c} \\
Z_{c} \\
\end{pmatrix}.
\end{equation}

Often in practice a skewness parameter $\gamma$ is also added (which generally ends up being close to 0), and this relationship can be written in the more compact form:
\begin{equation} \label{eq:PC2uvhomo}
    \begin{bmatrix}
        K & 0_{3 \times 1}
    \end{bmatrix} P_C^h = p^h, \quad K = \begin{bmatrix}
\alpha & \gamma & u_{0} \\
0 & \beta & v_{0}  \\
0 & 0 & 1
\end{bmatrix}.
\end{equation}
The matrix $K$ defined in \eqref{eq:PC2uvhomo} is sometimes referred to as the \textit{camera matrix} or \textit{matrix of intrinsic parameters}. It is referred to in this way because it contains the five parameters that define the fundamental characteristics of the camera (from the perspective of the pinhole camera model). While these parameters may be specified by the camera manufacturer, they are often extracted by performing a camera calibration.

\subsubsection{Mapping World Coordinates to Camera Coordinates ($P_W \xrightarrow{} P_C$)}
Recall from Figure \ref{fig:pinhole_cam} that a point $P$ in the scene can either be expressed in terms of camera frame coordinates $P_C$ or world frame coordinates $P_W$. While the previous section discussed the use of the pinhole model to map $P_C$ coordinates to pixel coordinates $p$, this section will discuss the mapping between the camera and world frame coordinates of the point $P$ (see Figure \ref{fig:Pc2Pw}).
\begin{figure}[ht]
\centering
\includegraphics[width=0.65\textwidth]{tex/figs/ch08_figs/world2camera.png}
\caption{A depiction of the point $P$ expressed either in camera coordinates, $P_C$, or in world frame coordinates, $P_W$. The world frame origin is denoted by $O_W$ and the camera frame origin is denoted by $O_C$.}
\label{fig:Pc2Pw}
\end{figure}

From Figure \ref{fig:Pc2Pw} it can be seen that $P_C$ can be written as:
\begin{equation}
P_{C} = t + q,
\end{equation}
where $t$ is the vector from $O_C$ to $O_W$ expressed in camera frame coordinates and $q$ is the vector from $O_W$ to $P$ expressed in camera frame coordinates. However, the vector $q$ is in fact the same vector as $P_W$, just expressed in different coordinates (i.e. with respect to a different frame). The coordinates can be related by a rotation:
\begin{equation}
q = RP_W,
\end{equation}
where R is the rotation matrix relating the camera frame to world frame and is defined as:
\begin{equation} \label{eq:rotmatrix}
R =
\begin{bmatrix}
i_w \cdot i & j_w \cdot i & k_w \cdot i \\
i_w \cdot j & j_w \cdot j & k_w \cdot j \\
i_w \cdot k & j_w \cdot k & k_w \cdot k
\end{bmatrix},
\end{equation}
where $i$, $j$, and $k$ are the unit vectors that define the camera frame and $i_w$, $j_w$, and $k_w$ are the unit vectors that define the world frame. To summarize, the point $P_W$ can be mapped to camera frame coordinates $P_C$ as:
\begin{equation}
P_C = t + R P_W,
\end{equation}
where $t$ is the vector in camera frame coordinates from $O_C$ to $O_W$ and $R$ is the rotation matrix defined in \eqref{eq:rotmatrix}.
Similar to the previous section, these expressions can also be equivalently expressed for the case where the points $P_W$ and $P_C$ are expressed in homogeneous coordinates:
\begin{equation} \label{eq:Pw2Pchomo}
\begin{split}
\begin{pmatrix}
    P_C \\ 1
    \end{pmatrix} = \begin{bmatrix}
R & t \\
0_{1\times3} & 1
\end{bmatrix}
\begin{pmatrix}
P_W \\ 1
\end{pmatrix}.
\end{split}
\end{equation}

\subsubsection{Mapping World Frame Coordinates to Image Coordinates ($P_W \xrightarrow{} p$)}
The original objective of perspective projection was to find a way to mathematically relate the position of a point $P$ in world frame coordinates (denoted $P_W$) to the corresponding pixel coordinates $p$ on the image plane. With the relationship \eqref{eq:Pw2Pchomo} developed for mapping $P_W$ to the camera frame coordinates $P_C$, and the relationship \eqref{eq:PC2uvhomo} for mapping $P_C$ to pixel coordinates $p$, the direct mapping from $P_W$ to $p$ can now be defined. In particular, simply combining the two transformation together yields:
\begin{equation*}
p^h = \begin{bmatrix}
K \quad 0_{3\times 1}
\end{bmatrix}
\begin{bmatrix}
R && t \\
0_{1 \times 3} && 1
\end{bmatrix}
P_W^h,
\end{equation*}
which can then be simplified to:
\begin{equation} \label{eq:Pw2uvhomo}
p^h = K\begin{bmatrix}
    R & t
\end{bmatrix} P_W^h.
\end{equation}
In \eqref{eq:Pw2uvhomo}, $P_W^h$ is the homogeneous coordinate representation of $P_W$ and $p^h$ is the homogeneous coordinate representation of $p$. Additionally, recall that the matrix $K \in \mathbb{R}^{3\times3}$ is the matrix of intrinsic camera parameters, and the matrix $[R \:\:\: t] \in \mathbb{R}^{3\times 4}$ contains \textit{extrinsic} parameters (i.e. that describe the camera's position and orientation relative the points in the scene). Note that the total number of degrees of freedom is 11, where 5 are from the intrinsic parameters that define $K$, 3 are from the rotation matrix $R$, and 3 are from the position vector $t$.

\subsection{Camera Calibration: Direct Linear Method}
Before the expression \eqref{eq:Pw2uvhomo} can be used in practice, the camera's intrinsic and extrinsic parameters need to be determined (i.e. $K$, $R$, and $t$). One approach is to use the direct linear transformation method for camera calibration, which requires a set of known correspondences $p_i \xleftrightarrow[]{} P_{W,i}$ for $i = 1,\dots,n$.

\subsubsection{Direct Linear Calibration: Step 1}
First, each corresponding pair of points $p_i = (u_i,v_i)$ and $P_{W,i} = (X_{W,i}, Y_{W,i}, Z_{W,i})$ is written in homogeneous coordinates and the expression \eqref{eq:Pw2uvhomo} is used to write:
\begin{equation} \label{eq:correspondance}
    p_i^h = M P_{W,i}^h, \quad i = 1,....n
\end{equation}
where $M = K[R \:\:\: t]$ is referred to as the \textit{homography}. The first step of the camera calibration process is to use the $n$ correspondences to compute the homography $M$, and then later the intrinsic and extrinsic parameters can be extracted from $M$. To determine $M$, a useful first step is to rewrite $M$ in terms of its rows:
\begin{equation}
    M = \begin{bmatrix}
        m_1 \\ m_2 \\ m_3
    \end{bmatrix},
\end{equation}
where $m_i \in \R^{1 \times 4}$ is the $i$-th row of $M$. By considering the rows of $M$ individually, the relationship \eqref{eq:correspondance} can be written as:
\begin{equation*}
    \begin{bmatrix}
        \alpha u_i \\ \alpha v_i \\ \alpha
    \end{bmatrix} = \begin{bmatrix}
         m_1 \cdot P_{W,i}^h \\ m_2 \cdot P_{W,i}^h \\ m_3 \cdot P_{W,i}^h \\
    \end{bmatrix}, \quad i = 1,....n
\end{equation*}
which by mapping the homogeneous coordinates $p_i^h$ back to the original coordinates $p_i$ yields the $2n$ expressions:
\begin{equation*}
\begin{split}
    u_i &= \frac{m_1 \cdot P_{W,i}^h}{m_3 \cdot P_{W,i}^h}, \quad i = 1,\dots,n \\
    v_i &= \frac{m_2 \cdot P_{W,i}^h}{m_3 \cdot P_{W,i}^h}, \quad i = 1,\dots,n,
\end{split}
\end{equation*}
or equivalently (via algebraic manipulation) the expressions:
\begin{equation}
\begin{split}
    u_i(m_3 \cdot P_{W,i}^h) - (m_1 \cdot P_{W,i}^h) &= 0, \quad i = 1,\dots,n \\
    v_i(m_3 \cdot P_{W,i}^h) - (m_2 \cdot P_{W,i}^h) &= 0, \quad i = 1,\dots,n.
\end{split}
\end{equation}
Now, these $2n$ equations can be combined together in one large matrix equation:
\begin{equation}
    \tilde{P}m = 0, \quad m = \begin{bmatrix}
        m_1^\top  \\ m_2^\top  \\ m_3^\top
    \end{bmatrix},
\end{equation}
where $m \in \R^{12 \times 1}$ is a vector consisting of the stacked rows of $M$ and $\tilde{P} \in \R^{2n \times 12}$ is a matrix of \textit{known} coefficients determined by the quantities $u_i$, $v_i$, and $P^h_{W,i}$. For a more concrete representation of how $\tilde{P}$ is defined, the first couple rows are given by:
\begin{equation} \label{eq:Pmeq}
\tilde{P} =
     \begin{bmatrix}
  -(P_{W,1}^h)^\top  & 0_{1\times 4} & u_{1} (P_{W,1}^h)^\top  \\
  0_{1 \times 4} & -(P_{W,1}^h)^\top  & v_{1} (P_{W,1}^h)^\top   \\
  -(P_{W,2}^h)^\top  & 0_{1\times 4} & u_{2} (P_{W,2}^h)^\top  \\
  \vdots & \vdots & \vdots
 \end{bmatrix}.
\end{equation}
Note that $n \geq 6$ (i.e. at least 6 correspondences have been made) is a requirement to ensure that $m$ can be uniquely defined. Ideally, with this sufficient number of correspondences the equation \eqref{eq:Pmeq} could be directly solved. However, in practice a more robust procedure is to build $\tilde{P}$ with more than 6 points, which would lead to an overdetermined set of equations that may not have a solution\footnote{This is particularly true in real-world applications where noise corrupts the data.}!
Therefore, the determination of $m$ is accomplished by formulation the optimization problem:
\begin{equation} \label{eq:mopt}
\begin{split}
\underset{m}{\text{min.}} \:\:& \lVert \tilde{P} m \rVert^2, \\
    \text{s.t.}\:\:& \lVert m \rVert^2 = 1,
\end{split}
\end{equation}
where the constraint $\lVert m \rVert^2 = 1$ is required to ensure that the optimization problem does not simply choose $m_i=0$ for each $i = 1,\dots,12$. This optimization problem is called a \textit{constrained least-squares} problem.

\begin{example}[Constrained Least-Squares] \label{ex:constlsq}
\theoremstyle{definition}
The constrained least squares problem
\begin{equation*}
\begin{split}
\underset{x}{\text{min.}} \:\:& \lVert A x \rVert^2, \\
    \text{s.t.}\:\:& \lVert x \rVert^2 = 1,
\end{split}
\end{equation*}
with $x\in \R^n$ and $A \in \R^{m \times n}$ and $m > n$ is a finite-dimensional optimization problem. Consider the corresponding Lagrangian:
\begin{equation*}
L = x^\top  A^\top Ax + \lambda (1 - x^\top x),
\end{equation*}
and the necessary optimality conditions:
\begin{equation*}
\begin{split}
\nabla_x L &= 2(A^\top A - \lambda I)x = 0, \\
\nabla_\lambda L &= 1 - x^\top x = 0. \\
\end{split}
\end{equation*}
The first NOC can be rewritten as $A^\top A x = \lambda x$, and therefore any $x$ that satisfies this condition must be an eigenvector of the matrix $A^\top A$. Additionally, while all the eigenvectors satisfy this condition the minimizer is the eigenvector associated with the smallest eigenvalue.
This eigenvector can efficiently be computed by a singular value decomposition of $A = U\Sigma V^\top $ and then choosing $m$ to be the column of $V$ associated with the smallest singular value (since $A^\top A = V\Sigma^2 V^\top $).
\end{example}

\subsubsection{Direct Linear Calibration: Step 2}
Once the optimization problem \eqref{eq:mopt} has been solved for $m$ the homography $M$ is completely defined. The next step in the camera calibration process is to extract the intrinsic and extrinsic camera parameters from the matrix $M$. For this section the matrix $M$ is expressed in terms of its columns:
\begin{equation*}
M = \begin{bmatrix}
    c_1 & c_2 & c_3 & c_4
\end{bmatrix},
\end{equation*}
where $c_i$ is the $i$-th column of $M$. It is now possible to factorize $M$ as:
\begin{equation}
    M = K\begin{bmatrix}
        R & t
    \end{bmatrix},
\end{equation}
by taking the first three columns of $M$ and performing a \textit{RQ factorization}:
\begin{equation}
\begin{bmatrix}
    c_1 & c_2 & c_3
\end{bmatrix} = KR,
\end{equation}
where $R$ is an orthogonal matrix and $K$ is an upper triangular matrix. Once $K$ is known the vector $t$ can be computed by $t = K^{-1} c_4$.

\subsubsection{A Flexible Camera Calibration Method (Zhang, 2000):} \label{subsubsec:zhang}
The homography $M$ is defined for a \textit{specific} set of extrinsic parameters $R$ and $t$. In practice it might be desirable to estimate the camera's intrinsic parameters from $N$ \textit{different} images from different perspectives (and therefore with $N$ different homographies). In this case the procedure described in \cite{Zhang2000} can be used to extract the intrinsic parameters $K$.

This approach begins by assuming that the known points $P_W$ for each individual image lie on a plane. For example the calibration ``scene'' might consist of a pattern (e.g. a checkerboard pattern) on a planar surface. In this case, it can simply be assumed that the world frame origin also lies on this plane such that $Z_W = 0$ for all points on the plane. Since $Z_W = 0$ the relationship between $p^h$ and $P^h_W$ given by \eqref{eq:Pw2uvhomo} can be simplified to:
\begin{equation}
p^h = \tilde{M} \tilde{P}_W^h,
\end{equation}
with
\begin{equation}
    \tilde{M} = K\begin{bmatrix}
    r_1 & r_2 & t
\end{bmatrix}, \quad \tilde{P}_W^h = \begin{bmatrix}
    X_W & Y_W & 1
\end{bmatrix}^\top ,
\end{equation}
where $\tilde{M}$ is the simplified homography matrix, $\tilde{P}_W^h$ is the simplified position of the point $P$ in world frame written in homogeneous coordinates, and $r_i$ is the $i$-th column of the rotation matrix $R$. Note that the homography matrix $\tilde{M}$ can still be estimated using the same procedure discussed before.

A set of constraints on the intrinsic parameter matrix $K$ are next identified by writing the homography $\tilde{M}$ as:
\begin{equation*}
    \begin{bmatrix}
        \tilde{c}_1 & \tilde{c}_2 & \tilde{c}_3
        \end{bmatrix} = \begin{bmatrix}
        Kr_1 & Kr_2 & Kt
    \end{bmatrix}.
\end{equation*}
This relationship, and the knowledge that $r_1$ and $r_2$ are orthonormal, leads to the following constraints:
\begin{equation} \label{eq:zhangconst}
    \tilde{c}_1^\top  B \tilde{c}_2 = 0, \quad \tilde{c}_1^\top  B \tilde{c}_1 = \tilde{c}_2^\top  B \tilde{c}_2,
\end{equation}
where $B = K^{-\top}K^{-1} \in \R^{3 \times 3}$ is a \textit{symmetric} matrix. Solving for the intrinsic camera parameters $K$ can therefore be accomplished by using the constraints \eqref{eq:zhangconst} to solve for the symmetric matrix $B$, and then to use the definition of $B$ to back out the parameters that define $K$.

Several useful tricks can be employed to compute the matrix $B$ from the constraints \eqref{eq:zhangconst}. The main trick is to notice that even though $B$ consists of nine parameters, since it is symmetric only six parameters are required to fully specify it. Therefore $B \in \R^{3\times 3}$ is reparameterized as a vector $b \in \R^6$ as:
\begin{equation}
    b = \begin{bmatrix}
        B_{11} & B_{12} & B_{22} & B_{13} & B_{23} & B_{33}
    \end{bmatrix}^\top .
\end{equation}
This reparameterization is useful because it allows us to rewrite the expression $\tilde{c}_i^\top B\tilde{c}_j$ as:
\begin{equation}
    \tilde{c}_i^\top B\tilde{c}_j = v_{ij}^\top  b,
\end{equation}
where:
\begin{equation*}
\begin{split}
   v_{ij} = \begin{bmatrix}
        \tilde{c}_{i1}\tilde{c}_{j1}, & \tilde{c}_{i1}\tilde{c}_{j2}+\tilde{c}_{i2}\tilde{c}_{j1}, & \tilde{c}_{i2}\tilde{c}_{j2}, & \tilde{c}_{i3}\tilde{c}_{j1} + \tilde{c}_{i1}\tilde{c}_{j3}, & \tilde{c}_{i3}\tilde{c}_{j2} + \tilde{c}_{i2}\tilde{c}_{j3}, & \tilde{c}_{i3}\tilde{c}_{j3}
    \end{bmatrix}^\top ,
\end{split}
\end{equation*}
where $\tilde{c}_{ik}$ is the $k$-th element of the column vector $\tilde{c}_i$ and $\tilde{c}_{jk}$ is the $k$-th element of the column vector $\tilde{c}_j$. With this reparameterization, the constraints \eqref{eq:zhangconst} can be rewritten as:
\begin{equation*}
\begin{split}
\tilde{c}_1^\top  B \tilde{c}_2 = 0 &\implies v_{12}^\top b = 0 \\
\tilde{c}_1^\top  B \tilde{c}_1 = \tilde{c}_2^\top  B \tilde{c}_2 &\implies (v_{11} - v_{22})^\top  b = 0,
\end{split}
\end{equation*}
or by combining them:
\begin{equation} \label{eq:bconst}
    \begin{bmatrix}
        v_{12}^\top  \\ (v_{11} - v_{22})^\top
    \end{bmatrix}b = 0,
\end{equation}
which is linear in the unknowns $b$. Importantly, while the homographies $M$ are different for each image, the intrinsic camera parameters (i.e. the vector $b$) are the same! Therefore for $N$ images from the same camera (but with potentially different perspectives) these constraints \eqref{eq:bconst} can be stacked to give:
\begin{equation} \label{eq:allbconst}
    Vb = 0,
\end{equation}
where $V \in \R^{2N \times 6}$. In the case where the skewness parameter $\gamma$ is included in $K$ there must be $N \geq 3$ images in order to specify $B$ uniquely. Similar to how the homography for an image $M$ was computed in the previous section, the vector $b$ will be specified by the solution to the constrained least squares problem:
\begin{equation} \label{eq:bopt}
\begin{split}
\underset{b}{\text{min.}} \:\:& \lVert Vb \rVert^2, \\
    \text{s.t.}\:\:& \lVert b \rVert^2 = 1.
\end{split}
\end{equation}
Once $b$ has been determined, the intrinsic camera parameters $K$ can be solved for recalling the definition of $B = K^{-T}K^{-1}$. In particular, the intrinsic parameters are given by:
\begin{equation} \label{eq:B2K}
\begin{split}
    v_0 &= \frac{B_{12}B_{13} - B_{11}B_{23}}{B_{11}B_{22} - B_{12}^2}, \\
    \lambda &= B_{33} - \frac{B_{13}^2 + v_0(B_{12}B_{13} - B_{11}B_{23})}{B_{11}}, \\
    \alpha &= \sqrt{\frac{\lambda}{B_{11}}}, \\
    \beta &= \sqrt{\frac{\lambda B_{11}}{B_{11}B_{22} - B_{12}^2}}, \\
    \gamma &= \frac{-B_{12}\alpha^2\beta}{\lambda}, \\
    u_0 &= \frac{\gamma v_0}{\beta} - \frac{B_{13}\alpha^2}{\lambda},
\end{split}
\end{equation}
where $\lambda$ can be though of as a scaling parameter that accounts for the fact that there are five unknown camera intrinsic parameters but six degrees of freedom in $B$.

Once the camera intrinsic parameters $K$ have been extracted from this procedure, given any new homography $\tilde{M}$ the extrinsic parameters can be computed by:
\begin{equation}
\begin{split}
    r_1 &= \frac{K^{-1}\tilde{c}_1}{\lVert K^{-1}\tilde{c}_1 \rVert}, \\
    r_2 &= \frac{K^{-1}\tilde{c}_2}{\lVert K^{-1}\tilde{c}_2 \rVert}, \\
    r_3 &= r_1 \times r_2, \\
    t &= \frac{K^{-1}\tilde{c}_3}{\lVert K^{-1}\tilde{c}_1 \rVert}.
\end{split}
\end{equation}
As one final step, it is noted that the matrix $R$ defined with columns $r_1$, $r_2$, and $r_3$ will not in generally satisfy the properties of a rotation matrix (i.e. orthonormality). One final step to this overall procedure is to correct this issue by finding the rotation matrix that best corresponds to these column vectors. This is accomplished again by optimization, and in particular by formulating the problem:
\begin{equation} \label{eq:Ropt}
\begin{split}
\underset{R}{\text{min.}} \:\:& \lVert R - Q \rVert^2, \\
    \text{s.t.}\:\:& R^\top R = I,
\end{split}
\end{equation}
where
\begin{equation*}
    Q = \begin{bmatrix}
        r_1 & r_2 & r_3
    \end{bmatrix}.
\end{equation*}
This problem is solved by choosing $R = UV^\top $ where $U$ and $V$ are defined by the singular value decomposition of $Q = U\Sigma V^\top $.

\subsection{Limitations}

\subsubsection{Radial Distortion}
The pinhole camera model provides a nominal camera model for which it is relatively straightforward to develop a mathematical model of the perspective projection. However, in practice this model is not a perfect representation of the imaging process. One such effect that is not captured by the pinhole model is \textit{radial distortion}, which is an effect seen in real lenses where either barrel distortion or pincushion distortion will affect the real pixel coordinates. Images showing both barrel and pincushion distortion are provided in Figure \ref{fig:distortion}.

\begin{figure}[ht]
\includegraphics[width=0.75\textwidth]{tex/figs/ch08_figs/lensdistortion.png}
\centering
\caption{Different kinds of radial distortions that are seen in real lenses, which may affect the accuracy of the pinhole camera model.}
\label{fig:distortion}
\end{figure}

There are methods that can be used to correct for image distortion. A simple and efficient way is to model the relationship between the ideal pixel coordinates $(u,v)$ and the distorted pixel coordinates $(u_d,v_d)$ as:
\begin{equation}
\begin{bmatrix}
    u_d \\
    v_d
\end{bmatrix}
=
\begin{bmatrix}
    u_d \\
    v_d
\end{bmatrix}
(1+kr^2)
\begin{bmatrix}
    u-u_{cd} \\
    v-v_{cd}
\end{bmatrix}
+
\begin{bmatrix}
    u_{cd} \\
    v_{cd}
\end{bmatrix}
\end{equation}
where $k \in \mathbb{R}$ is the radial distortion factor, $(u_{cd}, v_{cd})$ are the pixel coordinates of the image center, and $r^2=(u-u_{cd})^2+(v-v_{cd})^2$ is the square of the distance between the ideal pixel location and the center of distortion.. Note that $k$ differs in different cameras and needs to be pre-determined.

\subsubsection{Measuring Depth}
Once the camera intrinsic and extrinsic parameters $K$, $R$, and $t$ are known it is still not possible to map pixel coordinates to the corresponding point in space. Mathematically this is a result of the matrix $M$ in \eqref{eq:correspondance} not being invertible, but intuitively this is because the distance along the line of sight from $p$ to $P$ in Figure \ref{fig:pinhole_cam} can not be determined!

However, there are some techniques that can enable depth estimates to be made with a single camera. One approach is known as \textit{depth from focus}, where several images are taken until the projection of point $P$ is in focus. Based on the thin lens model, when this occurs:
\begin{equation*}
    \frac{1}{z} + \frac{1}{Z} = \frac{1}{f},
\end{equation*}
where $f$ is the focal length, $Z$ is the depth of the point $P$ in camera frame, and $z$ is the depth of the image plane in the camera frame when the projection of point $P$ is in focus. Since $f$ and $z$ are known, the depth $Z$ can therefore be computed.
If two cameras are used, depth estimation is possible via \textit{binocular reconstruction} or \textit{stereo vision}. This approach requires known corresponding pixel coordinates $p$ and $p'$ of each camera, and then uses \textit{triangulation} to determine the 3D position of the source point $P$ in the scene.

\subsection{Exercises}
\subsubsection{Camera Calibration}
Complete \textit{Problem 1: Camera Calibration} located in the online repository:

\vspace{\baselineskip}

\url{https://github.com/PrinciplesofRobotAutonomy/AA274A_HW3},

\vspace{\baselineskip}

where you will estimate the intrinsic parameters of a camera using the method described in Section \ref{subsubsec:zhang}.


\documentclass[a4paper,14pt]{article}
\title{Tensor Products of Vector Spaces}
\usepackage{amsmath, amssymb, amsthm}
\usepackage{tikz-cd}
\newtheorem*{prop}{Proposition}
\newtheorem*{corollary}{Corollary}
\newtheorem*{remark}{Remark}
\newtheorem*{defn}{Definition}
\newtheorem*{thm}{Theorem}
\begin{document}
\maketitle
\section{Hom spaces, dual spaces}
If $V$ and $W$ are two vector spaces over the same field $\mathbb{F}$, then the set of all linear maps $T: V \to W$ is denoted by $Hom(V, W)$. Under the usual definition of function addition and scalar multiplication, $Hom(V, W)$ is a vector space.

Since any field $\mathbb{F}$ can be considered as a vector space over itself, we can in particular form the space $Hom(V, \mathbb{F})$. This space is called the \textbf{dual space} of $V$, and will be denoted $V^{\ast}$.

\section{Quotient spaces}
If $V$ is a vector space over $\mathbb{F}$ and $W$ is a subspace of $V$, a \textbf{coset} of $W$ is a set $v + W := \{v + w : w \in W\}$.

\begin{prop}
The collection of cosets is a partition of $V$.
\begin{proof}
If $z$ is in both $u + W$ and $v + W$, then $u + w_1 = z = v + w_2$, so $v = u + (w_1 - w_2) \in u + W$. Similarly, $u \in v + W$. If $y \in u + W$, then $y - u \in W$, so $y \in v + W$. Similarly every $z \in v + W$ is in $u + W$, so $u + W = v + W$.
\end{proof}
\end{prop}

\begin{prop}
For any vector space $V$ and subspace $W$, you can define addition between cosets and scalar multiplication on cosets such that collection of cosets becomes a vector space over $\mathbb{F}$.
\end{prop}

\begin{remark}
The vector space of cosets of $W$ in $V$ is called the \textbf{quotient space}, and is denoted $V/W$.
\end{remark}

\begin{prop}
The map $\pi: V \to V/W$ defined by $v \mapsto v + W$ is a linear map.
\end{prop}

\begin{remark}
The function $\pi$ is called the \textbf{canonical map}.
\end{remark}

\begin{prop}
The kernel of any linear map $f: V \to W$ is a subspace of $V$, so $V / ker f$ is a quotient space.
\end{prop}

\begin{thm}
If $f: V \to W$ is linear and $U$ is a subspace of $V$ such that $U \subseteq ker f$, then there is a unique linear map $\phi: V / U \to W$ such that $\pi ; \phi = f$, where $\pi$ is the canonical map.

\begin{proof}
    Define $\phi$ by $v + U \mapsto vf$. Routine verification shows that it's well-defined, linear and satisfies $\pi ; \phi = f$. $\phi$ is unique because any other candidate $\psi$ must have $(v + U)\phi = vf = (v \pi) \psi = (v + U) \psi$, meaning $\phi = \psi$.
\end{proof}
\end{thm}

\section{Vector space of scalar-valued functions}
\begin{prop}
If $\mathbb{F}$ is any field and $X$ is any set, you can form a vector space by considering the collection of all functions $X \to \mathbb{F}$.

\begin{proof}
If $f, g: X \to \mathbb{F}$, we define addition by $x(f + g) := xf + xg$ and scalar multiplication by $x(\alpha f) = \alpha (x f)$. That this is a vector space is a routine verification.
\end{proof}
\end{prop}

\begin{remark}
This vector space is denoted $\mathbb{F}^X$.

You are probably familiar with the vector space $\mathbb{F}^n$. If you think about it, $\mathbb{F}^n$ is actually a special case of a space $\mathbb{F}^X$, where $X = \{1, \ldots, n\}$. The general idea, then, is that a function $X \to \mathbb{F}$ is a bag of scalars labeled by elements of $X$ There is exactly one scalar for each label. We can add two bags $B_1$ and $B_2$ of labeled scalars: the result is another bag of labeled scalars where for each label $x \in X$, the associated scalar is the sum of the scalars labeled by $x$ in $B_1$ and $B_2$. Scalar multiplication of a bag $B$ by $\alpha$ is just the bag where each scalar in $B$ is multiplied by $\alpha$.
\end{remark}

\begin{defn}
A function $f: X \to \mathbb{F}$ is said to have \textbf{finite support} when the set $\{x \in X : xf \neq 0\}$ is finite.
\end{defn}

\begin{prop}
The subset of $\mathbb{F}^X$ of functions with finite support is a subspace of $\mathbb{F}^X$.
\end{prop}

\section{Free vector spaces}
\begin{defn}
For any set $X$, a \textbf{free vector space} on $X$ with respect to some field $\mathbb{F}$, is any pair $(V, f)$ where $V$ is a vector space on $\mathbb{F}$ and $f: X \to V$ is a function such that for any vector space $W$ over $\mathbb{F}$ and any function $g: X \to W$, there is a unique linear map $T: V \to W$ such that $f;T = g$.

In other words, for every $W$ and every function $g: X \to W$ there is a unique linear $T: V \to W$ such that this diagram commutes:

\begin{center}
\begin{tikzcd}[row sep=large]
    X \arrow[rd, "g"] \arrow[r, "f"]
    & V \arrow[d, dashed, "\exists! \> T"] \\
                                   & W
\end{tikzcd}
\end{center}

This property is called the "universal property" for free vector spaces.
\end{defn}

\begin{prop}
    If a free vector space of a set $X$ over a field $\mathbb{F}$ exists, it is ``unique up to unique isomorphism'': if $(V, f: X \to V)$ and $(W, g: X \to W)$ are free vector spaces for $X$ over $\mathbb{F}$, there is a unique linear isomorphism $T: V \to W$ such that $f;T = g$.

\begin{proof}
    Since $(V, f)$ is a free vector space, by the universal property there is a linear map $r: V \to W$ such that $f;r = g$. Similarly there is a linear map $s: W \to V$ such that $g;s = f$. This implies $g;s;r = g$ and $f;r;s = f$.

\begin{center}
\begin{tikzcd}[row sep=large]
    & X \arrow[ld, "f"swap] \arrow[rd, "g"] \\
    V \arrow[rr, yshift=0.7ex, "r"] & & W \arrow[ll, yshift=-0.7ex, "s"]
\end{tikzcd}
\end{center}

 But we can apply the universal properties to $f$ and $g$ themselves: there is a unique linear map $p: V \to V$ such that $f = f;p$, and similarly a unique linear map $q: W \to W$ such that $g = g;q$. Since the identity maps $id_V$ and $id_W$ on $V$ and $W$ work for $p$ and $q$, respectively, and since $s;r$ and $r;s$ also fit the criteria, we must have $r;s = id_V$ and $s;r = id_W$. So $r$ and $s$ are inverses of each other, hence bijective, hence linear isomorphisms.

To prove that $r$ is the unique linear isomorphism $L: V \to W$ such that $f;L = g$, note that $f = g;s$, so $g;s;L = g$. $s;L$, being the composition of linear maps, is linear, so we must have $s;L = id_W$. So $L$ is a post-inverse for $s$, implying $L = r$.
\end{proof}
\end{prop}

\begin{prop}
There is a free vector space for every set $X$ and every field $\mathbb{F}$.
\begin{proof}
    The pair $(V, f)$ works, where $V$ is the subspace of $\mathbb{F}^X$ of functions of finite support and $f: X \to V$ is defined by $xf$ being the function that maps $x$ to $1 \in \mathbb{F}$ and $y \neq x$ to $0 \in \mathbb{F}$.
\end{proof}
\end{prop}

\begin{prop}
If $(V, f)$ is a free vector space for $X$, then $f$ is injective.
\begin{proof}
If $xf = yf$ for some $x \neq y \in X$, then any function $g: X \to W$ which has $xg \neq yg$ will not have a factorization through $f$, contradicting the universal property.
\end{proof}
\end{prop}

\section{Multilinear maps}
If $V_1, \ldots, V_n$ are some vector spaces over a field $\mathbb{F}$, then for any integer $i$, $1 \leq i \leq n$, and for any $v = (v_1, \ldots, v_{i-1}, v_{i+1}, \ldots, v_n) \in \prod_{j=1, j \neq i}^n V_j$, we can define the function $\lambda_i[v]: V_i \to \prod_1^n V_j$ by $z \lambda_i[v] := (v_1, \ldots, v_{i-1}, z, v_{i+1}, \ldots, v_n)$. It is routine to prove that each such function is a linear map $V_i \to \prod_1^n V_j$.

For any vector space $W$ over $\mathbb{F}$, a function $\phi: \prod_1^n V_j$ is said to be \textbf{multilinear} when for every $i$ and every $v \in \prod_{j=1, j \neq i}^n V_j$, the composite function $\lambda_i[v];\phi$ is a linear map $V_i \to W$. When $W = \mathbb{F}$ (that is, the vector space over $\mathbb{F}$, then $\phi$ is said to be a \textbf{multilinear form}.

\begin{prop}
Denote the collection of all multilinear maps $\prod_1^n V_i \to W$ by $ML(V_1, \ldots, V_n; W)$. By the usual definition of function addition and function scalar multiplication, this is a vector space over $\mathbb{F}$.
\end{prop}

\section{Tensor products}
A tensor product of vector spaces $(V_1, \ldots, V_n)$ is a vector space $Z$ and a multilinear map $\otimes$ such that $\otimes$ is, in some sense, the most general multilinear map on $\prod_1^n V_i$. We give a universal property that defines the tensor product (similar to what was done for free vector spaces), prove some general properties about it, and then explicitly construct one of the tensor products.

\begin{defn}
    For any vector spaces $V_1, \ldots, V_n$ over a common field $\mathbb{F}$, a \textbf{tensor product} of $(V_1, \ldots, V_n)$ is a pair $(Z, \otimes)$ where $Z$ is a vector space and $\otimes$ is a multilinear map $\prod_1^n V_i \to Z$ such that for every vector space $W$ over $\mathbb{F}$ and every multilinear map $f: \prod_1^n V_i \to W$, there is a unique linear map $T: Z \to W$ such that $f = \otimes;T$.

In other words, for every vector space $W$ and every multilinear map $f: \prod_1^n V_i \to W$ there is a unique linear $T: Z \to W$ such that this diagram commutes:

\begin{center}
\begin{tikzcd}[row sep=large]
    \prod_1^n V_i \arrow[rd, "f"] \arrow[r, "\otimes"]
    & Z \arrow[d, dashed, "\exists! \> T"] \\
    & W
\end{tikzcd}
\end{center}
\end{defn}

\begin{prop}
If $(Z, \otimes)$ is a tensor product of $(V_1, \ldots, V_n)$, then it is unique up to unique isomorphism.
\begin{proof}
This has the same form as the "unique up to unique isomorphism" proof for free vector spaces, so it is omitted here.
\end{proof}
\end{prop}

\begin{prop}
If $(Z, \otimes)$ is a tensor product of $(V_1, \ldots, V_n)$ and $W$ is any vector space over the same field as the $V_i$'s, then $Hom(Z, W)$ is isomorphic to $ML(V_1, \ldots, V_n; W)$.
\begin{proof}
    The universal property induces a map $\phi: ML(V_1, \ldots, V_n; W) \to Z$. If $T: Z \to W$ is any linear map, then $\otimes;T$ is multilinear since $\otimes$ is, so $(\otimes;T) \phi = T$, hence $\phi$ is surjective. It must also be injective: if $T = f \phi = g \phi$, then $f = \otimes;T = g$.
\end{proof}
\end{prop}
\begin{corollary}
For any tensor product $(Z, \otimes)$ of $(V_1, \ldots, V_n)$, the space of all multilinear forms on $\prod_1^n V_i$ is isomorphic to $Z^{\ast}$.
\end{corollary}

\begin{prop}
If $(Z, \otimes)$ is a tensor product of $(V_1, \ldots, V_n)$, then $img \otimes$ generates $Z$.
\begin{proof}
Let $Y$ be the subspace of $Z$ generated by $img \otimes$. Consider $\otimes': \prod_1^n V_i \to Y$, defined to be $\otimes$ but with its codomain restricted to $Y$. $\otimes'$ is clearly multilinear since $\otimes$ is, so we have some $T: Z \to Y$, linear, such that $\otimes; T = \otimes'$. But also, if we let $j: Y \to Z$ be the inclusion of $Y$ into $Z$, it is linear and $\otimes'; j = \otimes$.

\begin{center}
\begin{tikzcd}
    \prod_1^n V_i \arrow[rd, "\otimes'"] \arrow[r, "\otimes"] \arrow[rdd, "\otimes"]
    & Z \arrow[d, "T"] \arrow[dd, bend left=45, "id_Z"] \\
    & Y \arrow[d, hook, "j"] \\
    & Z
\end{tikzcd}
\end{center}

By combining these two, we get: $\otimes;T;j = \otimes$. But $T;j$ is linear, and by the universal property $id_Z$ is the unique linear map such that $\otimes; id_Z = \otimes$. So $T;j = id_Z$, implying $j$ is surjective. So $Y = Z$.

\end{proof}
\end{prop}

\begin{prop}
For any vector spaces $(V_1, \ldots, V_n)$ over a common field $\mathbb{F}$, there is a tensor product. The vector space is denoted $\bigotimes_1^n V_i$ (or $V_1 \otimes V_2$ in the case of $n = 2$) and the multilinear map is denoted $\otimes: \prod_1^n V_i \to \bigotimes_1^n V_i$, where usually infix notation is used: for all $(v_1, \ldots, v_n) \in \prod_1^n V_i$, $(v_1, \ldots, v_n) \mapsto v_1 \otimes \cdots \otimes v_n$.
\begin{proof}
    Let $(F(\prod_1^n V_i), \iota)$ be any free vector space on $\prod_1^n V_i$. If $W$ is any vector space and $\phi: \prod_1^n V_i \to W$ is any multilinear map, by the universal property of the free vector space we have a unique linear map $T: F(\prod_1^n V_i) \to W$ such that $\iota;T = \phi$.

\begin{center}
\begin{tikzcd}
    \prod_1^n V_i \arrow[d, "\phi"] \arrow[r, "\iota"]
    & F(\prod_1^n V_i) \arrow[dl, "T"] \\
    W
\end{tikzcd}
\end{center}

For all $(v_1, \ldots, v_n) \in \prod_1^n V_i$, let us denote $(v_1, \ldots, v_n) \iota$ by $[v_1, \ldots, v_n]$. $\iota$ is not multilinear since the zero of $F(\prod_1^n V_i)$ is not in the image of $\iota$, but we can make it multilinear by composing it with another map.

Consider the subspace $Z$ generated by all vectors of the form:

$$[v_1, \dots, v_{i-1}, \alpha v, v_{i+1}, \dots, v_n] - [v_1, \dots, v_{i-1}, v, v_{i+1}, \dots, v_n]$$

and

$$[v_1, \dots, v_{i-1}, v + w, v_{i+1}, \dots, v_n] - [v_1, \dots, v_{i-1}, v, v_{i+1}, \dots, v_n] - [v_1, \dots, v_{i-1}, w, v_{i+1}, \dots, v_n]$$

for some $i$, $1 \leq i \leq n$ and some $\alpha \in \mathbb{F}$, $v, w \in V_i$. If we let $\pi: F(\prod_1^n V_i) \to F(\prod_1^n V_i) / Z$ be the canonical map from $F(\prod_1^n V_i)$ to the quotient space induced by $Z$, the claim is that $(F(\prod_1^n V_i), \iota; \pi)$ is a tensor product. If we denote $(v_1, \dots, v_n) \iota \pi$ by $\langle v_1, \dots, v_n \rangle$, then, for example, $\langle v + w, v_2, \dots, v_n \rangle = \langle v, v_2, \dots, v_n \rangle + \langle w, v_2, \dots, v_n \rangle$ since the difference of the two sides is an element of $Z$. So $\iota ; \pi$ is multilinear.

Now, $Z$ is a subset of $ker T$ because $\phi$ is multilinear. Also, by the property of the quotient space, there is a unique linear $f: F(\prod_1^n V_i) / Z \to W$ such that $\pi; f = T$.

\begin{center}
\begin{tikzcd}
    \prod_1^n V_i \arrow[d, "\phi"] \arrow[r, "\iota"]
    & F(\prod_1^n V_i) \arrow[dl, "T"] \arrow[d, "\pi"] \\
    W & F(\prod_1^n V_i) / Z \arrow[l, "f"]
\end{tikzcd}
\end{center}

So $(F(\prod_1^n V_i)/Z, \iota;\pi)$ is a tensor product for $(V_1, \ldots, V_n)$.
\end{proof}
\end{prop}

\begin{prop}
For any vector spaces $X, Y, Z$ over a common field, we have $(X \otimes Y) \otimes Z$ is isomorphic to $X \otimes Y \otimes Z$ is isomorphic to $X \otimes (Y \otimes Z)$.
\begin{proof}
    We will construct the isomorphism between $(X \otimes Y) \otimes Z$ and $X \otimes Y \otimes Z$ and wave our hands about the other isomorphism. First, for any $z \in Z$, we can construct a map $\phi_z: X \times Y \to X \otimes Y \otimes Z$ by $(x, y) \phi_z := x \otimes y \otimes z$. This function is multilinear, so by the universal property of $X \otimes Y$, there's a unique linear map $T_z: X \otimes Y \to X \otimes Y \otimes Z$ such that $(x \otimes y ) T_z = x \otimes y \otimes z$ for all $(x, y) \in X \times Y$.

\begin{center}
\begin{tikzcd}[row sep=large]
    X \times Y \arrow[rd, "\phi_z"] \arrow[r, "\otimes"]
    & X \otimes Y \arrow[d, "T_z"] \\
    & X \otimes Y \otimes Z
\end{tikzcd}
\end{center}

Using $T_z$, we can define a function $\phi: (X \otimes Y) \times Z \to X \otimes Y \otimes Z$ by $(a, z) \phi := a T_z$. This is linear in the first parameter because each $T_z$ is linear. Also,

\begin{align*}
(x \otimes y) T_{z_1 + z_2} & = x \otimes y \otimes (z_1 + z_2) \\
                            & = x \otimes y \otimes z_1 + x \otimes y \otimes z_2 \\
                            & = (x \otimes y) T_{z_1} + (x \otimes y) T_{z_2}
\end{align*}

for all $x \in X$, $y \in Y$. Since elements $x \otimes y$ generate $X \otimes Y$, we have $T_{z_1 + z_2} = T_{z_1} + T_{z_2}$. For a similar reason, $T_{\alpha z} = \alpha T_z$ for all $\alpha \in \mathbb{F}$. So $\phi$ is bilinear, which gives, by the universal property, a linear map $T: (X \otimes Y) \otimes Z \to X \otimes Y \otimes Z$.

\begin{center}
\begin{tikzcd}[row sep=large]
    (X \otimes Y) \times Z \arrow[rd, "\phi"] \arrow[r, "\chi"]
    & (X \otimes Y) \otimes Z \arrow[d, "T"] \\
                                   & X \otimes Y \otimes Z
\end{tikzcd}
\end{center}

Now, if we define a map $\psi: X \times Y \times Z \to (X \otimes Y) \otimes Z$ by $(x, y, z) \psi := (x \otimes y) \otimes z$, this is a multilinear map, so there's an induced linear map $S: X \otimes Y \otimes Z \to (X \otimes Y) \otimes Z$. By the universal property, this map $S$ has that $(x \otimes y \otimes z) ST = x \otimes y \otimes z$ for all $(x, y, z) \in X \times Y \times Z$. So $ST$ is a linear map $X \otimes Y \otimes Z \to X \otimes Y \otimes Z$ such that, when it is restricted to the generating set of $\{x \otimes y \otimes z : (x, y, z) \in X \times Y \times Z\}$, becomes the identity map. So in fact $ST$ is the identity map.

Similarly, it can be proved that the elements of the form $(x \otimes y) \otimes z$ are a generating set for $(X \otimes Y) \otimes Z$ (elements $a \otimes z$ for $a \in X \otimes Y$ definitely are, but $x \otimes y$ generate $X \otimes Y$ and $\otimes$ is bilinear), so for a similar reason $TS$ is an identity map as well. So $S$ and $T$ are linear and inverses of one another, meaning that $X \otimes Y \otimes Z$ and $(X \otimes Y) \otimes Z$ are isomorphic.
\end{proof}
\end{prop}

\begin{prop}
For any spaces $V$ and $W$ over a field $\mathbb{F}$, $V \otimes W$ is isomorphic to $W \otimes V$.
\begin{proof}
    The multilinear maps $\phi: V \times W \to W \otimes V$ and $\psi: W \times V \to V \otimes W$ defined by $(v, w) \phi := w \otimes v$ and $(w, v) \psi := v \otimes w$ induce, respectively, linear maps $S: V \otimes W \to W \otimes V$ and $T: W \otimes V \to V \otimes W$. We have that $(v \otimes w) ST = v \otimes w$ and $(w \otimes v) TS = w \otimes v$ for all $v \in V$, $w \in W$. Since these generate $V \otimes W$ and $W \otimes V$, $S$ and $T$ are isomorphisms.
\end{proof}
\end{prop}

\section{Finite dimensional facts}

\begin{prop}
    If $V_1, \ldots, V_n$ are all finite dimensional vector spaces with $V_i$ having dimension $d_i$, and, for all $i$, $\beta_i = \{v_{i,1}, \dots, v_{i,d_i}\}$ is a basis for $V_i$, then if $f: \prod_1^n \beta_i \to \mathbb{F}$ is any function, $f$ can be extended to exactly one multilinear form $g: \prod_1^n V_i \to \mathbb{F}$.

Furthermore, $dim ML(V_1, \ldots, V_n; \mathbb{F}) = \prod_1^n dim V_i$.

\begin{proof}
    If $\phi: \prod_1^n V_i \to \mathbb{F}$ is any multilinear form, then for all $z \in \prod_1^n V_i$, $z = (z_1, \dots, z_n) = (\sum_j \alpha_{1,j} v_{1,j}, \dots, \sum_j \alpha_{n,j} v_{n,j})$, so

$$z \phi = \sum_{j \in \prod_1^n [d_i]} \alpha_{1,j_1} \cdots \alpha_{n,j_n} (v_{1,j_1}, \ldots, v_{n,j_n}) \phi$$

where $[n] := \{1, \ldots, n\}$ for any $n$. This shows that any multilinear map is completely determined by how the "basis tuples" $(v_{1,j_1}, \ldots, v_{n,j_n})$ get mapped, which proves the first statement.

For any $I \in \prod_1^n [d_i]$ we define $f_I: \prod_1^n V_i \to \mathbb{F}$ to be the multilinear form that maps the basis tuple $(v_{I_1}, \ldots, v_{I_n})$ to $1$ and all other basis tuples to $0$. Then the collection of these functions is a basis for $ML(V_1, \ldots, V_n; \mathbb{F})$.
\end{proof}
\end{prop}

\begin{prop}
If $V_1, \ldots, V_n$ are all finite dimensional vector spaces, then $dim(\otimes_1^n V_i) = \prod_1^n dim V_i$.
\begin{proof}
The previous proposition establishes that $dim ML(V_1, \ldots, V_n; \mathbb{F}) = \prod_1^n dim V_i$. But we know that $ML(V_1, \ldots, V_n; \mathbb{F})$ and $(\otimes_1^n V_i)^{\ast} = Hom(\otimes_1^n V_i, \mathbb{F}$ are isomorphic. If we could establish that $\otimes_1^n V_i$ is finite-dimensional, then the result would be proved since every finite dimensional vector space is isomorphic with its dual.

But we know the set of elements $v_1 \otimes \dots \otimes v_n$ generates $\otimes_1^n V_i$, and we know that $v_1 \otimes \dots \otimes v_n$ is a linear combination of "basis tensors" $e_1 \otimes \dots \otimes e_n$, where each $e_i$ is an element of a basis $\beta_i$ for $V_i$. So the collection of "basis tensors" generates $\otimes_1^n V_i$, hence it is finitely-generated, so finite dimensional.
\end{proof}
\end{prop}

\begin{prop}
If $V$ and $W$ are finite dimensional, then $Hom(V, W) \cong V^{\ast} \otimes W$.
\begin{proof}
    This can be seen from the two spaces having the same dimension. We know $dim(V^{\ast} \otimes W) = dim V^{\ast} dim W = dim V dim W$. To complete the proof, if $\{v_1, \ldots, v_m\}$ and $\{w_1, \ldots, w_n\}$ are bases for $V$ and $W$, respesctively, then the collection of maps $g_{ij}: V \to W$ defined by $v_k g_{ij} = \delta_{ik} w_j$ (where $\delta$ is the Kronecker delta) is a basis for $Hom(V, W)$. So $dim Hom(V, W) = dim V dim W$.
\end{proof}
\end{prop}

\section{Tensor spaces}
\begin{defn}
    If we $V$ is any vector space, and if we define $V_1 := V$ and $V_{-1} := V^{\ast}$, then any tensor product $V_{\epsilon_1} \otimes \cdots \otimes V_{\epsilon_n}$ with each $\epsilon_i \in \{1, -1\}$, is isomorphic to $V^{\otimes p} \otimes (V^{\ast})^{\otimes q}$ for some $p, q \in \mathbb{N}$, by commutativity and associativity of the tensor product. We call such a product the \textbf{tensor space of type $(p, q)$} and denote it by $T_q^p(V)$. Elements of $T_q^p(V)$ are called \textbf{tensors of type $(p, q)$}. Such tensors are said to be \textbf{contravariant of degree $p$} and \textbf{covariant of degree q}. When $q = 0$, the tensors are said to be \textbf{contravariant}, and similary when $p = 0$ they are said to be \textbf{covariant}.
\end{defn}
\end{document}


\documentclass[12pt]{article}
 \usepackage[margin=0.8in]{geometry}
\usepackage{amsmath,amsthm,amssymb,amsfonts}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}

\newenvironment{problem}[2][Problem]{\begin{trivlist}
\item[\hskip \labelsep {\bfseries #1}\hskip \labelsep {\bfseries #2.}]}{\end{trivlist}}
%If you want to title your bold things something different just make another thing exactly like this but replace "problem" with the name of the thing you want, like theorem or lemma or whatever

\begin{document}

%\renewcommand{\qedsymbol}{\filledbox}
%Good resources for looking up how to do stuff:
%Binary operators: http://www.access2science.com/latex/Binary.html
%General help: http://en.wikibooks.org/wiki/LaTeX/Mathematics
%Or just google stuff

\title{Citadel Boston Regional Datathon 2021}
\author{Shinjini Ghosh, Lay Jain, Pawan Goyal}
\date{\today}
\maketitle

\section*{SEIVR Model Equations}
\begin{align*}
    % \def\arraystretch{1.5}
    % \renewcommand\arraystretch{1.5}
    \Dot{S} & = \alpha R_S - \frac{S}{N}\beta I- \frac{S}{N}\chi E-\rho S \\[4pt]
    \Dot{V_1} &= \rho S + \rho R_S - \frac{V_1}{N}\beta I - \frac{V_1}{N}\chi E - \phi V_1 \\[4pt]
    \Dot{V_2} &= \phi V_1 + \phi ' R_1 + (1-\delta_2)I_2 - \frac{V_2}{N} \beta I - \frac{V_2}{N} \chi E \\[4pt]
    \Dot{E_1} &= \frac{V_1}{N} \beta I + \frac{V_1}{N} \chi E - \theta E_1\\[4pt]
    \Dot{E_2} &= \frac{V_2}{N} \beta I + \frac{V_2}{N} \chi E - \theta E_2 \\[4pt]
    \Dot{E_S} &= \frac{S}{N} \beta I + \frac{S}{N} \chi E - \theta E_S \\[4pt]
    \Dot{I_1} &= \theta E_1 - \delta_1 I_1 - (1-\delta_1) I_1 \\[4pt]
    \Dot{I_2} &= \theta E_2 - \delta_2 I_2 - (1-\delta_2) I_2 \\[4pt]
    \Dot{I_S} &= \theta E_S - \delta_S I_S - (1-\delta_S) I_S \\[4pt]
    \Dot{R_1} &= (1-\delta_1) I_1 - \phi ' R_1 \\[4pt]
    \Dot{R_S} &= (1-\delta_S) I_S - \rho R_S - \alpha R_S \\[4pt]
    \Dot{D} &= \delta_1 I_1 + \delta_2 I_2 + \delta_S I_S \\
\end{align*}
\vspace{-4mm}
where
\vspace{-5mm}
\begin{align*}
    I &= I_1 + I_2 + I_3 \\
    E &= E_1 + E_2 + E_3 \\
    N &= S + V_1 + V_2 + E_1 + E_2 + E_S + I_1 + I_2 + I_S + R_1 + R_2 + R_S + D \\
\end{align*}

\end{document}


\subsection{Naive Algorithms}
  Here we show three naive algorithms for calculating the new capacities that make use of the Trust Transfer
  corollary~(\ref{trusttransfer}). To prove the correctness of the algorithms, it suffices to prove that
  \begin{equation}
  \label{naive:req1}
     \forall i \in [n], c'_i \leq x_i \mbox{ and}
  \end{equation}
  \begin{equation}
  \label{naive:req2}
     \sum\limits_{i=1}^{n}c'_i = F - V \enspace.
  \end{equation}
  Due to the nature of these algorithms, the only inputs necessary are the initial flows $x_i$, the number of $A$'s neighbours
  $n$ and the desired flow reduction $V$. Proofs of correctness and complexity can be found in the Appendix.

  \subimport{common/algorithms/}{fcfscode.tex}

  The previous algorithm nullifies all outgoing trust to one player after the other in the order they are given in the input,
  until the desired indirect trust is achieved. The complexity of this algorithm is $O\left(n\right)$.

  \subimport{common/algorithms/}{abscode.tex}

  This algorithm finds a \texttt{reduction} value such that the configuration $\forall i \in \left[n\right], c'_i =
  \max{\left(0, x_i - \mbox{\texttt{reduction}}\right)}$ achieves the desired flow.

  The function \texttt{preprocess(}$x_i$\texttt{)} returns a data structure \texttt{X} containing the set of flows
  $\left(x_i\right)$, such that the corresponding function \texttt{popMin(X)} is able to repeatedly return a tuple
  consisiting of the index of the minimum element and a new data structure missing exactly the minimum element.
  Examples of such pairs of functions are:
  \begin{equation*}
  \begin{gathered}
    \begin{cases}
      \texttt{preprocess = quickSort} \\
      \texttt{popMin = (}x_1\texttt{, X}\setminus \{x_1\}\texttt{)}
    \end{cases}
    \mbox{ and} \\
    \begin{cases}
      \texttt{preprocess = FibonacciHeap} \\
      \texttt{popMin = (find-min(X),delete-min(X))}
    \end{cases} \enspace.
  \end{gathered}
  \end{equation*}
  In the general case, the complexity of the algorithm \texttt{abs} is proven to be
  $O\left(preprocess\right) + O\left(n\right)O\left(popMin\right)$. In both specific cases, the complexity is
  $O\left(n\log{n}\right)$. This algorithm also minimizes the $||\Delta_i||_\infty$ norm for the specific set of old flows
  $x_i$ given as input. A proof of this fact can be found in the Appendix.

  \subimport{common/algorithms/}{propcode.tex}
  This algorithm reduces all outgoing direct trust proportionally in such a way that the desired flow is achieved. The
  complexity of this algorithm is proven to be $O\left(n\right)$.


\paragraph{Answer 1.}

The method to answer these questions is simply to try small words by
constructing them in order to satisfy the constraints.
\begin{enumerate}

  \item The \label{aba} shortest word \(x\) belonging to L(r) is found
    by taking \(\epsilon\) in place of \(\lparen a \disjM{}
    b\rparen\kleeneM\). So \(x = aba\). Let us check if \(x \in L(s)\)
    or not. \(L(s)\) is made of the union of four sub-languages
    (subsets). To make this clear, let us remove the useless
    parentheses on the right side:
    \begin{equation*}
    s = \lparen ab \rparen\kleeneM{} \, \disjM{} \, \lparen ba
    \rparen\kleeneM{} \, \disjM{} \, a\kleeneM{} \, \disjM{} \,
    b\kleeneM.
    \end{equation*}
    Therefore, membership tests on \(L(s)\) have to be split
    into four: one membership test on \(\lparen ab
    \rparen\kleeneM\), one on \(\lparen ba \rparen\kleeneM\),
    one on \(a\kleeneM\) and another one on
    \(b\kleeneM\). In other words, \(x \in L(s)\) is equivalent to
    \begin{equation*}
    x \in L(\lparen ab \rparen\kleeneM) \;
     \text{or} \; x \in L(\lparen ba \rparen\kleeneM) \;
     \text{or} \; x \in L(a\kleeneM) \; \text{or} \; x
     \in L(b\kleeneM).
    \end{equation*}
    Let us test the membership with \(x = aba\):
    \begin{enumerate}

      \item The words in \(L(\lparen ab \rparen\kleeneM)\) are
        \(\epsilon\), \(ab\), \(abab\ldots\) Thus \(aba \not\in
        L(\lparen ab \rparen\kleeneM)\).

      \item The words in \(L(\lparen ba \rparen\kleeneM)\) are
        \(\epsilon\), \(ba\), \(baba\ldots\) Hence \(aba \not\in
        L(\lparen ba \rparen\kleeneM)\).

      \item The words in \(L(a\kleeneM)\) are \(\epsilon\), \(a\),
        \(aa\ldots\) Therefore \(aba \not\in L(a\kleeneM)\).

      \item The words in \(L(b\kleeneM)\) are \(\epsilon\), \(b\),
        \(bb\ldots\) So \(aba \not\in L(b\kleeneM)\).

    \end{enumerate}
    The conclusion is \(aba \not\in L(s)\).

    \item What is the shortest word belonging to \(L(s)\)?  Since the
      four sub-languages composing \(L(s)\) are starred, it means that
      \(\epsilon \in L(s)\). Since we showed at the item~(\ref{aba})
      that \(aba\) is the shortest word of \(L(r)\), it means that
      \(\epsilon \not\in L(r)\) because \(\epsilon\) is of length
      \(0\).

    \item This question is a bit more difficult. After a few tries, we
      cannot find any~\(x\) such that \(x \in L(r)\) and \(x \in
      L(s)\). Then we may try to prove that \(L(r) \cap L(s) =
      \varnothing\), \emph{i.e.,} there is no such~\(x\). How should we
      proceed? The idea is to use the decomposition of \(L(s)\) into
      for sub-languages and try to prove
      \begin{align*}
        L(r) \cap L(\lparen ab \rparen\kleeneM) &= \varnothing,\\
        L(r) \cap L(\lparen ba \rparen\kleeneM) &= \varnothing,\\
        L(r) \cap L(a\kleeneM) &= \varnothing,\\
        L(r) \cap L(b\kleeneM) &= \varnothing.
      \end{align*}
      If all these four equations are true, they imply
        \(L(r) \cap L(s) = \varnothing\).
      \begin{enumerate}

        \item Any word in \(L(r)\) ends with \(a\) whereas any word in
          \(L(\lparen ab \rparen\kleeneM)\) finishes with \(b\) or is
          \(\epsilon\). Thus \(L(r) \cap L(\lparen ab \rparen\kleeneM)
          = \varnothing\).

        \item For the same reason, \(L(r) \cap L(b\kleeneM) =
          \varnothing\).

        \item Any word in \(L(r)\) contains both \(a\) and \(b\)
          whereas any word in \(L(a\kleeneM)\) contains only \(b\) or
          is \(\epsilon\). Therefore \(L(r) \cap L(a\kleeneM) =
          \varnothing\).

        \item Any word in \(L(r)\) starts with \(a\) whereas any word
          in \(L(\lparen ba \rparen\kleeneM)\) starts with \(b\) or is
          \(\epsilon\). Thus \(L(r) \cap L(\lparen ba \rparen\kleeneM)
          = \varnothing\).

      \end{enumerate}
      Finally, since all the four equations are false, they imply
      that
      \begin{equation*}
        L(r) \cap L(s) = \varnothing.
      \end{equation*}

    \item Let us construct letter by letter a word \(x\) which does
      not belong neither to \(L(r)\) not \(L(s)\). First, we note that
      all words in \(L(r)\) start with \(a\), so we can try to start
      \(x\) with \(b\): this way \(x \not\in L(r)\). So we have \(x =
      b\ldots\) and we have to fill the dots with some letters in such
      a way that \(x \not\in L(s)\).

      We use again the decomposition of~\(L(s)\) into four
      sub-languages and make sure that~\(x\) does not belong to any of
      those sub-languages. First, because \(x\) starts with \(b\), we
      have \(x \not\in L(a\kleeneM)\) and \(x \not\in L(\lparen
      ab \rparen\kleeneM)\). Now, we have to add some more letters
      such that \(x \not\in L(b\kleeneM)\) and \(x \not\in L(\lparen
      ba \rparen\kleeneM)\). Since any word in \(L(b\kleeneM)\) has a
      letter \(b\) as second letter or is \(\epsilon\), we can choose
      the second letter of \(x\) to be \(a\). This
      way \(x=ba\ldots \not\in L(b\kleeneM)\). Finally, we have to
      add more letters to make sure that
      \begin{equation*}
      x=ba\ldots \not\in L(\lparen ba\rparen\kleeneM).
      \end{equation*}
      Any word in \(L(\lparen ba\rparen\kleeneM)\) is
      either \(\epsilon\) or \(ba\) or \(baba\ldots\), hence the third
      letter is \(b\). Therefore, let us choose the letter \(a\) as
      the third letter of \(x\) and we thus have \(x=baa \not\in
      L(\lparen ba\rparen\kleeneM)\). In summary, \(baa \not\in L(r),
      baa \not\in L(b\kleeneM), baa \not\in L(\lparen
      ba\rparen\kleeneM), baa \not\in L(a\kleeneM), baa \not\in
      L(\lparen ab\rparen\kleeneM)\), which is equivalent
      to \(baa \not\in L(r)\) and \(baa \not\in L(\lparen
      ab \rparen\kleeneM) \cup L(\lparen ba \rparen\kleeneM) \cup
      L(a\kleeneM) \cup L(b\kleeneM) = L(s)\). Therefore, \(x=baa\) is
      one possible answer.

\end{enumerate}


%Admittedly, structure learning is an overloaded term, so in this chapter, we aim to define it and provide example problems. We will also identify and focus on its Bayesian variant.
In this chapter we define the Bayesian structure learning problem and cover relevant technical background material.

\section{What is structure learning?}
Learning structure is a broad problem; we define it as an unsupervised learning problem wherein a user has data
and a priori assumes some unobserved, or \emph{latent}, organization of the data. Example organizations of the data might include a flat clustering, i.e. the data can be partitioned into some $\numcluster$ distinct groups. A logical extension of a flat clustering organization is a mixed-membership model,
where data can be organized into $\numcluster$ groups but each datum may belong to many groups simultaneously.
In this thesis, we are particularly interested in hierarchical clustering, where data are organized into a tree structure, with data closer on the tree being logically or semantically similar. We are also interested in
linear dynamical systems, wherein time-series data
evolve according to a linear transition function.

A structure learning problem first begins with data, and an assumption of a structure class. It then recovers a plausible structure and returns it to the user. Formally, we assume a structure class $\structures$ and return a candidate structure $\structure \in \structures$.
%In a structure learning problem, corresponding algorithms exist which have various properties and trade-offs.
Perhaps the most famous is the $\numcluster$-means problem, which assumes $\numcluster$
discrete clusters to which the data belong.
The corresponding algorithm recovers a $\numcluster$-partition of the data.
Analogously in mixed-membership modeling, latent Dirichlet allocation \citep[LDA; ][]{Blei2003} recovers a topic model structure for text data.

We formalize structure learning probabilistically. We will later cover the Bayesian variant, but in vanilla structure learning,
there exists some latent structure $\structure$
and a likelihood function $\p(\dataset \given \structure)$.
Recovering a structure amounts to
maximizing likelihood, i.e.
\begin{align*}
    \structure^* = \argmax_{\structure} \p(\dataset \given \structure)
\end{align*}
We now outline two structure learning problems
which are running examples in this thesis. We first introduce the structure classes and present a candidate likelihood function. We then discuss the relevant
algorithms that actually perform
the maximum likelihood optimization.

\subsection{Linear dynamical system}
In a linear dynamical system (LDS),
each observed data point is a sequence,
$\x = \sequence$, and our dataset
is a collection of $\numdata$ such sequences, $\data = \sequencen$.
We focus on stochastic LDSs where
the transition function is probabilistic.
The underlying structure class for the
transition function
is linear-Gaussian $p(\state_{\t + 1} \given \state_t, \dynmat, \dyncovar) = \N(\dynmat \state_\t, \dyncovar)$ with learnable parameters $\dynmat, \dyncovar$.
Although the underlying structure in this situation
is not discrete, as in the case
of flat or hierarchical clusterings, it is still
an assumption about the organization of data. Intuitively,
an LDS structure implies that data that temporally
close to each other are a simple transition away from each other.
Structure learning in an LDS amounts
to solving the maximum
likelihood problem
\begin{align*}
    \dynmat^*, \dyncovar^* = \argmax_{\dynmat, \dyncovar} \prod_{n = 1}^N \prod_{\t = 1}^{\T - 1} \p(\state^{(\n)}_{\t + 1} \given \state^{(\n)}_\t, \dynmat, \dyncovar)
\end{align*}
and lends itself to a closed-form solution, namely
linear regression. This formulation can also be visualized
as a graphical model, shown in \autoref{fig:graphical-model-lds}.

\begin{figure}[htp!]
    \centering
    \includegraphics{tikz/lds}
    \caption{The graphical model for a linear dynamic system}
    \label{fig:graphical-model-lds}
\end{figure}

\subsection{Hierarchical clustering}
Hierarchical clustering is a structure learning problem
where the structure class is trees. Specifically,
given a dataset $\dataset$,
we are interested in rooted trees with $\numdata$ leaves
with each leaf corresponding to a data point.
Such a tree encodes relationships between data:
if data points $a$ and $b$ are close in data space, we'd hope
the leaves corresponding to $a$ and $b$ appear closer
together in a tree that captures the data's structure.

Algorithms for hierarchical clustering can be broadly
divided into two categories: divisive and agglomerative.
Divisive, or top-down, clustering algorithms recover a tree by recursively
partitioning data until just leaves remain. Examples
include spectral clustering \citep{Shi2000} and recursive $\numcluster$-means.
Agglomerative, or bottom-up, clustering algorithms
initialize clusters as leaves, and recursively
merge clusters until a full binary tree is formed.
Pairs of clusters to merge are chosen according to
a heuristic, also called a linkage-criterion,
an example of which is single-linkage,
where clusters are merged according to the minimum
distance between data points in each cluster.

Hierarchical clustering can also be formalized as an optimization problem
by minimizing carefully constructed cost
function,
$C(\structure, \dataset)$,
as shown in \citet{Dasgupta2016}.
Treating this cost as the
energy in a Gibbs distribution,
we can also frame hierarchical clustering
as a probabilistic model
$\p(\dataset \given \tree) = \exp{-C(\tau, \dataset)/T}$ for some temperature $T$.
Recovering a hierarchical clustering of the
data is therefore a maximum likelihood problem:
\begin{align*}
    \tree^* = \argmax_{\tree} \exp{-C(\tau, \dataset)/T}
\end{align*}

\citet{Dasgupta2016} uses a greedy algorithm
to optimize this likelihood.
The graphical model for hierarchical clustering is pictured in \autoref{fig:graphical-model-hc}.

\begin{figure}[htp!]
    \centering
    \includegraphics{tikz/hc}
    \caption{Graphical model for hierarchical clustering}
    \label{fig:graphical-model-hc}
\end{figure}

\section{Bayesian structure learning}

Structure learning is a useful generalization
of several popular problems. However, it has some
drawbacks in its presented formulation.
Consider a dataset that has ambiguity in its
latent structure. For example, in \autoref{fig:ambiguous-structure}, we picture two examples of ambiguous latent structure. In a hierarchical clustering problem, if data are positioned in particular ways, we could organize the data
in several plausible ways. In one example, there are
three equally valid hierarchical clusterings (using binary trees), and in the second example, there are two
plausible clusterings. Structure learning as presented
cannot disambiguate between equally valid clusterings or indicate to a user that there exist alternative candidates.

\begin{figure*}[htp!]
    \centering
    \includegraphics[width=0.8\textwidth]{img/structure/3-cluster-both}
    \caption{Examples of ambiguous hierarchical structure in data. The left set of points could plausibly be hierarchically clustered in three different ways. The right set of points has two plausible hierarchical clusterings.}
    \label{fig:ambiguous-structure}
\end{figure*}

Bayesian structure learning (BSL) puts a prior on the structure class $\p(\structure)$, and rather than
returning a candidate structure $\structure^*$,
it returns a posterior distribution over all
structures $\p(\structure \given \dataset)$. It generalizes
vanilla structure learning, as $\p(\structure \given \dataset)$ could,
in principle,
be a delta distribution for a single candidate structure.
In practice, however, the BSL
formulation enables algorithms that capture
uncertainty and ambiguity in how data is organized.
In \autoref{fig:ambiguous-structure-probability},
we picture the same ambiguous hierarchical clusterings
along with the intuitively correct distribution over possible clusterings.

\begin{figure*}[htp!]
    \centering
    \begin{subfigure}{\textwidth}
        \centering
        \includegraphics[width=0.7\textwidth]{img/structure/3-cluster-distribution}
        \caption{}
    \end{subfigure}
    \begin{subfigure}{\textwidth}
        \centering
        \includegraphics[width=0.7\textwidth]{img/structure/3-cluster-linear-distribution}
        \caption{}
    \end{subfigure}
    \caption{Examples of how Bayesian structure learning could uncover ambiguity in latent structure. In (a), all three possible clusterings are given equal probability and in (b), the two most likely clusterings split the probability.}
    \label{fig:ambiguous-structure-probability}
\end{figure*}

BSL can be formalized
as a latent variable model,
where we assume a prior over structures $\p(\structure)$,
and a likelihood model $p(\dataset \given \structure)$.
The goal is then to perform Bayesian
inference to obtain the posterior distribution $p(\structure \given \dataset)$. The simple graphical model for this generative process is pictured in \autoref{fig:graphical-model-bsl}.

\begin{figure}[htp!]
    \centering
    \includegraphics[]{tikz/lvm}
    \caption{The graphical model for Bayesian structure learning. Note the addition of a prior distribution over the structure.}
    \label{fig:graphical-model-bsl}
\end{figure}

The design space of Bayesian structure learning
amounts to first deciding a structure class (hierarchical clusterings, flat clusterings, linear dynamical systems, etc.) and additionally choosing a
particular prior distribution over the structures.
There is interest, however, in developing strategies for automatically
inferring structure class through search \citep{Kemp2008, Grosse2012}.
Once a structure class and prior are selected, computing the posterior distribution $\p(\structure \given \dataset)$ amounts to Bayesian inference,
and more often than not this posterior distribution is analytically intractable and we must resort to
approximate inference. Thus, in practice, we
also decide on an approximate inference strategy (variational inference, MCMC, etc.).

\subsection{Bayesian linear dynamical system}

A Bayesian linear dynamical system (BLDS)
is an LDS with the addition of a prior
over the transition matrix and noise covariance.
Although there are many choices of prior,
in this thesis, we use the matrix-Normal-inverse-Wishart (MNIW) prior, which is conjugate, details of which
are located in \autoref{sec:stats-mniw}.

The choice of this prior results in a generative model
\begin{align*}
    \dynmat, \dyncovar &\sim MNIW(\Psi, \dynmat_0, V, \nu) \\
    \state_{\t + 1} | \state_t, \dynmat, \dyncovar &\sim \N(\dynmat \state_t, \dyncovar)
\end{align*}
which is also visualized in \autoref{fig:graphical-model-blds}.

\begin{figure}[htp!]
    \centering
    \includegraphics[]{tikz/blds}
    \caption{The graphical model for a Bayesian linear dynamical system. Note the addition of a prior over the dynamics parameters.}
    \label{fig:graphical-model-blds}
\end{figure}

Because the MNIW prior is conjugate
with a linear dynamical system likelihood,
the posterior distribution $\p(\dynmat, \dyncovar \given \sequencen)$ is also MNIW
and can be computed in closed form. This structure class
is explored in more detail in \autoref{chap:solar}.

Another structure class of interest is trees,
specifically hierarchical clusterings.
In the next chapter, we detail
how hierarchical clustering can be formalized
in the Bayesian structure learning framework.


% !TeX root = ../main.tex
\section{Type Rule Derivation Tree Example}
\label{sec:typeRuleTree}
This is an example of how to create a type derivation tree for a program written in HCL. It is based on the following code snippet.

\begin{lstlisting}[language=HCL]
num x = 5
bool b = x greaterThan 0
b print
\end{lstlisting}

The program calls two functions, $greaterThan$ and $print$.
They are declared with the following types:

\begin{lstlisting}[language=HCL]
func[num, num, bool] greaterThan # Two number parameters, returns bool
func[bool, none] print # overloaded to take a bool parameter, returns nothing
\end{lstlisting}

Following the type rules specified in section \ref{typerules}, this derivation tree will be produced:
\begin{center}
	\begin{math}
		\cfrac
		{\cfrac
			{E\vdash 5 : num}
			{E\vdash num\ x = 5 : \texttt{ok}}\quad \cfrac
				{\cfrac
					{\cfrac
						{E(x) : num\quad E \vdash 0 : num}
						{E\vdash x\ greaterThan\ 0: bool}}
					{E\vdash bool\ b = x\ greaterThan\ 0 : \texttt{ok}}\quad \cfrac
					{E\vdash b : bool}
					{E\vdash b\ print : (none, \texttt{ok})}}
				{E\vdash bool\ b = x\ greaterThan\ 0;\ b\ print : \texttt{ok}}}
		{E \vdash num\ x = 5;\ bool\ b = x\ greaterThan\ 0;\ b\ print : \texttt{ok}}
	\end{math}
\end{center}