\breakpagebeforenextheadingtrue
\section{Algorithms}
\label{sec:82algorithms}

\minitoc{83mm}{7}

\noindent
This section gives an overview of the algorithms that
we use to implement the solution process after discretization of
the Bellman equation \eqref{eq:gridBellman}.
In the following, we assume that the probability density functions of the
stochastic variables are known.

\subsection{General Structure}
\label{sec:821generalStructure}

The general approach to solve dynamic portfolio choice models is as follows:
\begin{enumerate}
  \item
  Generation of value function interpolants $\valueintp_t$
  %(\texttt{solveValueFunction}, \cref{alg:financeSolveValueFunction})

  \item
  Generation of optimal policy interpolants $\optpolicyintp_t$
  %(\texttt{solvePolicies}, \cref{alg:financeSolvePolicies})

  \item
  Post-processing, e.g., Monte Carlo simulation
  %or computation of Euler equation errors
  %(\cref{sec:828postProecessing})
\end{enumerate}
The separation of the solution processes for
the value function interpolants $\valueintp_t$
and the optimal policy interpolants $\optpolicyintp_t$
enables the generation of different spatially adaptive sparse grids
for the value function and the optimal policies.
This is useful if the shapes of value function and optimal policies
have different characteristics.

In the following \cref{%
  sec:822solveValueFunction,%
  sec:823optimization,%
  sec:824quadrature,%
  sec:825interpolation,%
  sec:826gridGeneration%
}, we describe the algorithmic details of
\texttt{solveValueFunction} (step 1).
The treatment of the other steps \texttt{solvePolicy} (step 2) and
post-processing (step 3) follows with
\cref{sec:827solvePolicies} and \cref{sec:828postProecessing},
respectively.

%\paragraph{Interpolants}

We track two interpolants $\valueintp[1]_t$ and $\valueintp[p]_t$
for each $t = 0, \dotsc, T$.
The former interpolates value function data at the grid points
with the hierarchical piecewise linear basis
(used for the surplus-based grid generation),
while the latter interpolates the same data with hierarchical B-splines
of degree $p > 1$.
Each $\valueintp[\ast]_t$ ($\ast \in \{1, p\}$)
additionally stores the grid points $\state_t^{(k)}$
and the optimal policies $\optpolicyintp_t(\state_t^{(k)})$
at the grid points ($k = 1, \dotsc, \ngp_t$).
For simplicity, we do not pass them as separate data
to the algorithms.

\subsection{Solution for the Value Function}
\label{sec:822solveValueFunction}

\paragraph{\texttt{solveValueFunction} algorithm}

\Cref{alg:financeSolveValueFunction} shows \texttt{solveValueFunction},
which generates the value function interpolants
$\valueintp[1]_t$ and $\valueintp[p]_t$ ($t = 0, \dotsc, T$).
The algorithm follows a simple optimize--refine--interpolate scheme,
which is visualized in \cref{fig:structureSolveValueFunction}:
First, the Bellman equation \eqref{eq:gridBellman} is solved
on an initial sparse grid (\texttt{optimize}).
Then, we \texttt{refine} the grid spatially adaptively.
Finally, the resulting grid point data are \texttt{interpolate}d
with hierarchical higher-order B-splines.

\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{%
      $\text{%
        $(\valueintp[p]_t)_{t=0,\dotsc,T}$%
      } = \texttt{solveValueFunction}$%
    }{%
      \hspace*{0mm}%
    }
      \State{$\valueintp[p]_{T+1} \gets \emptyset$}
      \Comment{dummy variable (is not used)}%
      \For{$t = T,\, T - 1,\, \dotsc,\, 0$}
        \State{%
          $\valueintp[1]_t \gets \text{%
            Initial regular sparse grid with no values%
          }$%
        }
        \State{%
          $\valueintp[1]_t \gets
          \texttt{optimize($t$, $\valueintp[1]_t$, $\valueintp[p]_{t+1}$)}$%
        }
        \State{%
          $\valueintp[1]_t \gets
          \texttt{refine($t$, $\valueintp[1]_t$, $\valueintp[p]_{t+1}$)}$%
        }
        \State{%
          $\valueintp[p]_t \gets
          \texttt{interpolate($\valueintp[1]_t$)}$%
        }
      \EndFor{}
    \EndFunction{}
  \end{algorithmic}
  \caption[%
    Generation of value function interpolants (\texttt{solveValueFunction})%
  ]{%
    Generation of value function interpolants.
    The output is the higher-order B-spline interpolant $\valueintp[p]_t$
    for all $t = 0, \dotsc, T$.%
  }%
  \label{alg:financeSolveValueFunction}%
\end{algorithm}

\begin{figure}
  \newcommand*{\getscale}{
    \pgfgettransformentries{%
      \myxscaletmp%
    }{\@tempa}{\@tempa}{\myyscaletmp}{\@tempa}{\@tempa}
    \gdef\myxscale{\myxscaletmp}
  }%
  \newcommand*{\drawcircle}[2][]{
    \node[
      circle,minimum size=60mm,fill=mittelblau!#2,draw=mittelblau,
    ] at (0mm,0mm) (myCircle#2) {};
    \IfStrEq{#1}{noArrow}{}{
      \getscale
      \pgfmathparse{150+20/\myxscale^0.8}
      \let\myin\pgfmathresult
      \draw[->] (myCircle#2) to[
        out=150,in=\myin,looseness=2.3,
      ] node[below,scale=1/\myxscale] {\mytext} (myCircle#2);
    }
  }%
  \newcommand*{\drawtext}[1]{
    \getscale
    \centerarc[
      draw=none,postaction={decorate},decoration={
        text along path,
        text={#1},
        text align=center
      },
    ](0mm,0mm)(180:0:30mm-1em/\myxscale);
  }%
  \newenvironment*{circlescope}{
    \getscale
    %\begin{scope}[scale=0.8,shift={(5mm,-5mm)},transform shape]
    \begin{scope}[
      scale=1-0.15/\myxscale,
      shift={(3mm/\myxscale,-3mm/\myxscale)},
      transform shape,
    ]
  }{
    \end{scope}
  }%
  \begin{tikzpicture}
    \draw[dashed] (0mm,9mm) -- (84mm,9mm);
    \draw[dashed] (-4.3mm,-8mm) -- (71.2mm,-48.2mm);
    \begin{scope}[rotate=-130]
      \node[
        circle,
        minimum size=18mm,
        fill=mittelblau!30,
        draw=mittelblau,
        inner sep=0mm,
      ] at (0mm,0mm) (optimize) {\texttt{optimize}};
      \begin{scope}[
        shift={(40mm/2,40mm*sin(60))},
        transform shape,
      ]
        \begin{scope}[
          rotate=130,
          scale=18mm/60mm+0.15,
          transform shape,
        ]
          \getscale
          \drawcircle[noArrow]{30}
          \drawtext{|\ttfamily|refine}
          \begin{scope}[
            scale=1-0.15/\myxscale,
            shift={(3mm/\myxscale,-3mm/\myxscale)},
            transform shape,
          ]
            \def\mytext{}
            \drawcircle{50}
            \getscale
            \node[scale=1/\myxscale] at (0mm,0mm) {\texttt{optimize}};
          \end{scope}
        \end{scope}
      \end{scope}
      \node[
        circle,
        minimum size=18mm,
        fill=mittelblau!30,
        draw=mittelblau,
        inner sep=0mm,
        text width=15mm,
        align=center,
      ] at (40mm,0mm) (interpolate) {\ttfamily{}inter-\\polate};
      \draw[->] (optimize) to[
        bend left=30,
      ] (myCircle30);
      \draw[->] (myCircle30) to[
        bend left=30,
      ] (interpolate);
      \draw[->] (interpolate) to[
        bend left=30,
      ] node[above,rotate=50] {$t \gets (t - 1)$} (optimize);
    \end{scope}
    \begin{scope}[shift={(84mm,-21mm)}]
      \drawcircle[noArrow]{20}
      \drawtext{|\ttfamily|optimize}
      \begin{circlescope}
        \def\mytext{\hspace*{5mm}$\state_t$}
        \drawcircle{30}
        \drawtext{|\ttfamily|optimizeSinglePoint}
        \begin{circlescope}
          \def\mytext{\hspace*{5mm}$\policy_t$}
          \drawcircle{40}
          \drawtext{|\ttfamily|evalObjFcnGrad}
          \begin{circlescope}
          \def\mytext{\hspace*{5mm}$\stochastic_t$}
            \drawcircle{50}
            \drawtext{|\ttfamily|evalQuadPoint}
            \begin{circlescope}
              \def\mytext{\hspace*{4mm}$o$}
              \drawcircle{60}
              \getscale
              \node[
                text width=30mm,align=center,scale=1/\myxscale,
              ] at (0mm,0mm) {%
                \ttfamily{}evalInterp-\\PartDeriv%
              };
            \end{circlescope}
          \end{circlescope}
        \end{circlescope}
      \end{circlescope}
    \end{scope}
  \end{tikzpicture}%
  \caption[Scheme of the generation of value function interpolants]{%
    Scheme of the generation of value function interpolants
    with \texttt{solveValueFunction}
    (\cref{alg:financeSolveValueFunction}, \emph{left}),
    which repeatedly calls the \texttt{optimize} routine
    (\cref{alg:financeOptimize}, \emph{right}),
    which in turn consists of various sub-functions.
    The function \texttt{optimize} iterates over all state grid points
    $\state_t = \state_t^{(k)}$ ($k = 1, \dotsc, \ngp_t$)
    and calls \texttt{optimizeSinglePoint} for each point.
    The optimization method evaluates the objective function and
    its gradient at a sequence of different policy points $\policy_t$
    to find $\optpolicyintp_t(\state_t^{(k)})$.
    This evaluation (denoted by \texttt{evalObjFcnGrad})
    has to compute the expectation in \cref{eq:gridBellman},
    which is done using a quadrature rule.
    For every quadrature point $\stochastic_t = \stochastic_t^{(j)}$
    ($j = 1, \dotsc, m_{\quadweight}$),
    \texttt{evalQuadPoint} computes the corresponding value of
    the expression in the expectation.
    Finally, \texttt{evalInterpPartDeriv} evaluates the interpolant
    $\valueintp[p]_{t+1}$ and its partial derivatives,
    for which we have to loop over the state dimensions $o = 1, \dotsc, d$.%
  }%
  \label{fig:structureSolveValueFunction}%
\end{figure}

At the beginning of every iteration $t$,
the grid of the piecewise linear interpolant is reset
to an initial, possibly regular sparse grid.
It would also be possible to reuse the grid from the
previous iteration $t + 1$.
Nevertheless, the results we then obtain become worse,
likely due to the different characteristics of $\valueintp[1]_t$
for different $t$ (e.g., kinks).

The higher-order B-spline interpolant
$\valueintp[p]_{t+1}$ of the previous iteration $t+1$ is used
for the \rhs of the Bellman equation \eqref{eq:gridBellman},
if $t < T$.
In the first iteration $t = T$,
there is no such interpolant.
However,
%it is not needed anyway, as
the terminal solution $\valuefcn_T$ is usually a known function.

\subsection{Optimization}
\label{sec:823optimization}

\paragraph{\texttt{optimize} algorithm}

The \texttt{optimize} step is given as \cref{alg:financeOptimize}.
The grid of the argument $\valueintp[1]_t$
is some spatially adaptive sparse grid
$\gridset{t}{\sparse}
= \{\state_t^{(k)} \mid k = 1, \dotsc, \ngp_t\}$,
where the function values $\valueintp[1]_t(\state_t^{(k)})$
may already be known for some grid points $\state_t^{(k)}$,
if \texttt{optimize} is called from within \texttt{refine}.
The function \texttt{optimize} computes the missing value function values.
For $t = T$, we assume that the terminal solution
$\valuefcn_T$ can be computed by some function
\texttt{computeKnownTerminalSolution}.%
\footnote{%
  In any case, the terminal solution may be computed as the
  solution of the corresponding single-time optimization problem,
  e.g., $\valuefcn_T(\state_T^{(k)})
  = \max_{\policy_T} \utilityfcn(\consume_T(\state_T^{(k)}, \policy_T))$.%
}
Otherwise, for $t < T$, we solve the Bellman equation
\eqref{eq:gridBellman} by using the higher-order B-spline interpolant
$\valueintp[p]_{t+1}$ of the previous iteration $t + 1$
(\texttt{optimizeSinglePoint}).
The computations for the different $\state_t^{(k)}$ are
independent of each other,
which means that they can be computed in parallel \cite{Horneff16Efficient}.%
\footnote{%
  Such a problem is usually referred to as \term{embarrassingly parallel.}%
}
After generating all missing data,
we update the hierarchical surpluses of the
piecewise linear interpolant $\valueintp[1]_t$
to interpolate the new data at all grid points of $\gridset{t}{\sparse}$.

\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{$\valueintp[1]_t = \texttt{optimize}$}{%
      $t$, $\valueintp[1]_t$, $\valueintp[p]_{t+1}$%
    }
      \State{%
        $(\state_t^{(k)})_{k=1,\dotsc,\ngp_t}
        \gets \text{grid of $\valueintp[1]_t$}$%
      }
      \For{$k = 1, \dotsc, \ngp_t$}
        \If{$\valueintp[1]_t(\state_t^{(k)})$ not previously computed}
          \IfOneLine{$t = T$}{%
            $\valueintp[1]_T(\state_T^{(k)})
            \gets \texttt{computeKnownTerminalSolution($\state_T^{(k)}$)}$%
          }
          \ElseOneLine{%
            $\valueintp[1]_t(\state_t^{(k)})
            \gets \texttt{%
              optimizeSinglePoint(%
                $t$, $\state_t^{(k)}$, $\valueintp[p]_{t+1}$%
              )%
            }$%
          }
        \EndIf{}
      \EndFor{}
      \State{%
        Re-interpolate
        $(\valueintp[1]_t(\state_t^{(k)}))_{k=1,\dotsc,\ngp_t}$
        with piecewise linear functions%
      }
    \EndFunction{}
  \end{algorithmic}
  \caption[Evaluation of the value function (\texttt{optimize})]{%
    Evaluation of the value function at all
    grid points $\state_t^{(k)}$ of $\valueintp[1]_t$
    at which the value function has not been evaluated yet.
    Inputs are
    the time $t$,
    the piecewise linear interpolant $\valueintp[1]_t$
    of the current iteration $t$ (with the underlying sparse grid and
    corresponding function values, possibly unset), and
    the higher-order B-spline interpolant $\valueintp[p]_{t+1}$
    of the previous iteration $t + 1$
    (not used if $t = T$).
    The output is the updated piecewise linear interpolant $\valueintp[1]_t$,
    where all missing function values at grid points have been computed.%
  }%
  \label{alg:financeOptimize}%
\end{algorithm}

\paragraph{Certainty-equivalent transformation}

For utility functions of \crra-type, i.e., of the form
$\utilityfcn(\consume_t) = \consume_t^{1-\riskav}/(1-\riskav)$,
the curvature of the objective function in the Bellman equation
\eqref{eq:gridBellman} can be very high
(depending on the risk aversion parameter $\riskav$),
which may impede convergence of the optimizer.
As a remedy, we transform the value function $\valueintp_t$ with the
\term{certainty-equivalent transformation}
$\valueintp_t \mapsto \cetvalueintp_t
\ceq ((1 - \riskav) \valueintp_t)^{1/(1-\riskav)}$ if $\riskav > 1$.
\Cref{eq:gridBellman} then becomes
$\cetvalueintp_T(\state_T^{(k)}) = \max_{\policy_T}
\consume_T(\state_T^{(k)}, \policy_T)$ for $t = T$ and
\begin{equation}
  \label{eq:gridBellmanCET}
  \cetvalueintp_t(\state_t^{(k)})
  = \max_{\policy_t} \left(
    \left(
      \consume_t(\state_t^{(k)}, \policy_t)^{1-\riskav} +
      \patience \expectation[t]{
        \bigl(
          \cetvalueintp_{t+1}(
            \statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t)
          )
        \bigr)^{1-\riskav}
      }
    \right)^{1/(1-\riskav)}
  \right)
\end{equation}
for $t < T$, since for $\riskav > 1$,
$(\cdot)^{1/(1-\riskav)}$ is strictly monotonously decreasing and
$(1-\riskav) < 0$.
The notation in the remainder of this section
does not distinguish between $\valueintp[\ast]_t$ and $\cetvalueintp[\ast]_t$
and uses $\valueintp[\ast]_t$ for both if it is not relevant
whether the value function is transformed.
%and hence,
%$(\max ((1-\riskav) (\cdot)))^{1/(1-\riskav)}
%= ((1-\riskav) \min (\cdot))^{1/(1-\riskav)}
%= (1-\riskav)^{1/(1-\riskav)} \max ((\cdot)^{1/(1-\riskav)})$.

%\paragraph{State space transformation}
%
%Depending on the dynamic portfolio lifecycle model at hand,
%it might be beneficial to transform the state space as well
%with a \term{state space transformation}
%$\statetrafofcn_t\colon \clint{\*0, \*1} \to \real^d$,
%for instance, to increase the density of grid points for low
%wealth values $\wealth_t$,
%where the value function might have a high curvature.
%The untransformed states $\state_t^{(k)}$ are then still contained in
%the unit cube $\clint{\*0, \*1}$,
%and the occurrences of $\state_t^{(k)}$ in \cref{eq:gridBellmanCET}
%are replaced with $\statetrafofcn_t(\state_t^{(k)})$.

\subsection{Quadrature}
\label{sec:824quadrature}

We need to approximate the expectation in \cref{eq:gridBellmanCET}
by quadrature,
\begin{equation}
  \label{eq:bellmanQuadrature}
  \expectation[t]{
    \bigl(
      \cetvalueintp[p]_{t+1}(
        \statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t)
      )
    \bigr)^{1-\riskav}
  }
  \approx \sum_{j=1}^{m_{\quadweight}} \quadweight_t^{(j)}
  \bigl(
    \cetvalueintp[p]_{t+1}(
      \statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t^{(j)})
    )
  \bigr)^{1-\riskav},
\end{equation}
for some weights $\quadweight_t^{(j)} \in \real$ and
nodes $\stochastic_t^{(j)} \in \stochdomain$
($j = 1, \dotsc, m_{\quadweight}$).
Since the stochastic domain $\stochdomain \subset \real^{m_{\stochastic}}$
might be high-dimensional as well,
full grid quadrature rules suffer from the curse of dimensionality.
Therefore, we use sparse grid quadrature rules based
on Gauss--Hermite quadrature
\multicite{Gerstner98Numerical,Horneff16Efficient}.
Note that this sparse grid in the stochastic space $\stochdomain$
is independent of the sparse grid in the state space $\clint{\*0, \*1}$.
However, it would also be feasible to employ Monte Carlo quadrature,
albeit usually far more expensive.

\subsection{Interpolation and Extrapolation}
\label{sec:825interpolation}

\paragraph{Sparse grid interpolation}

As already mentioned,
$\valueintp[1]_t$ is constructed as the sparse grid interpolant
of the grid data $\state_t^{(k)}$ ($k = 1, \dotsc, \ngp_t$)
using the hierarchical piecewise linear basis.
For $\valueintp[p]_t$,
we use cubic hierarchical weakly fundamental not-a-knot splines
(see \cref{sec:454wfs}).
The not-a-knot boundary conditions help to decrease the
interpolation error (see \cref{sec:541interpolation}),
while the weakly fundamental property eases the hierarchization
complexity by enabling us to use the unidirectional principle
(see \cref{sec:45spatAdaptiveUP,sec:543complexity}).

\paragraph{Extrapolation}

Unfortunately, for many dynamic portfolio choice models,
the state transition is not a function
$\statefcn_t\colon \clint{\*0, \*1} \times \real^{m_{\policy}} \times
\stochdomain \to \clint{\*0, \*1}$,
especially if the state space is actually unbounded.
It may then happen that
$\statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t^{(j)})
\notin \clint{\*0, \*1}$
for some quadrature nodes
$\stochastic_t^{(j)} \in \stochdomain$ in \cref{eq:bellmanQuadrature}.
%i.e., $\statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t^{(j)})$
%is not a feasible state.
Hence, we might not be able to evaluate the value function interpolant
$\valueintp[p]_{t+1}(
  \statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t^{(j)})
)$, as it is only defined on $\clint{\*0, \*1}$.
Scaling of the domain is not an option due to the dynamic nature of
the problem.
%Modifying infeasible states by cropping them to $\clint{\*0, \*1}$
%is not straightforward, as this introduces new errors and
%the modified state should satisfy the model constraints.

Instead, we extend the interpolant $\valueintp[p]_{t+1}$
to $\real^d$ by an extrapolation method based on Taylor approximation.
First, we crop the evaluation point
$\state_{t+1} \in \real^d \setminus \clint{\*0, \*1}$ to a point
$
  \state_{t+1}^\mathrm{in}
  = \statefcn_t(\state_t^{(k)}, \policy_t, \stochastic_t^{(j)})
  \in \clint{\*0, \*1}
$
with $\state_{t+1}^\mathrm{in} \ceq \vecmin(\vecmax(\state_{t+1}, \*0), \*1)$
(component-wise minimum/maximum).
The extrapolation type,
which may be \texttt{constant}, \texttt{linear}, and \texttt{quadratic},
determines the degree of the Taylor approximation:
\begin{equation}
  \begin{split}
    \valueintp[p]_{t+1}(\state_{t+1})
    &\approx \valueintp[p]_{t+1}(\state_{t+1}^\mathrm{in}) +
    \tr{(\gradient{\state_{t+1}}{\valueintp[p]_{t+1}}(\state_{t+1}^\mathrm{in}))}
    (\state_{t+1} - \state_{t+1}^\mathrm{in})\\
    &\qquad + \tr{(\state_{t+1} - \state_{t+1}^\mathrm{in})}
    (\hessian{\state_{t+1}}{\valueintp[p]_{t+1}}(\state_{t+1}^\mathrm{in}))
    (\state_{t+1} - \state_{t+1}^\mathrm{in}),
  \end{split}
\end{equation}
where \texttt{constant} and \texttt{linear}
only use the first summand and first two summands, respectively.
Since hierarchical B-splines enable us to
exactly and efficiently compute
the gradient $\gradient{\state_{t+1}}{\valueintp[p]_{t+1}}$ and
the Hessian $\hessian{\state_{t+1}}{\valueintp[p]_{t+1}}$,
we do not have to approximate the derivatives with finite differences.

\subsection{Grid Generation}
\label{sec:826gridGeneration}

\paragraph{\texttt{refine} algorithm}

\Cref{alg:financeRefine} shows how to generate the spatially adaptive
sparse grid in \texttt{solveValueFunction}
(\cref{alg:financeSolveValueFunction}).
The underlying criterion is the common surplus-based refinement criterion
\cite{Pflueger13Spatially}.
As for the application in topology optimization (see \cref{chap:60topoOpt}),
we use the piecewise linear interpolant for the surplus-based
grid generation,
since the surpluses are easier to compute in the piecewise linear case,
and they are more meaningful
due to the integral representation formula \eqref{eq:surplusIntegral}.
Parameters for \cref{alg:financeRefine} are
the tolerance $\refinetol_t \in \nonnegreal$,
by which the set of grid points to be refined is determined, and
the number $\norefine_t \in \natz$ of refinement iterations.
These parameters may depend on the time $t$,
since it might be beneficial to change the adaptivity of the
grid over time.

\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{$\valueintp[1]_t = \texttt{refine}$}{%
      $t$, $\valueintp[1]_t$, $\valueintp[p]_{t+1}$%
    }
      \For{$j = 1, \dotsc, \norefine_t$}
        \State{%
          $\ngp_t
          \gets \text{number of grid points of $\valueintp[1]_t$}$%
        }
        \ForOneLine{$k = 1, \dotsc, \ngp_t$}{%
          $\surplus[(k)]{t}
          \gets \text{%
            surplus of $\state_t^{(k)}$ in $\valueintp[1]_t$%
          }$%
        }
        \State{%
          $K_\mathrm{refine}
          \gets \{k = 1, \dotsc, \ngp_t \mid
          \abs{\surplus[(k)]{t}} \ge \refinetol_t\}$%
        }
        \IfOneLine{$K_\mathrm{refine} = \emptyset$}{\Break}
        \State{%
          Refine all grid points in
          $\{\state_t^{(k)} \mid k \in K_\mathrm{refine}\}$%
        }
        \State{%
          $\valueintp[1]_t \gets
          \texttt{optimize($t$, $\valueintp[1]_t$, $\valueintp[p]_{t+1}$)}$%
        }
      \EndFor{}
    \EndFunction{}
  \end{algorithmic}
  \caption[Refinement of the value function (\texttt{refine})]{%
    In-place refinement of the value function $\valueintp[1]_t$.
    Inputs are
    the time $t$,
    the piecewise linear interpolant $\valueintp[1]_t$
    of the current iteration $t$, and
    the higher-order B-spline interpolant $\valueintp[p]_{t+1}$
    of the previous iteration $t + 1$
    (not used if $t = T$).
    The output is the updated piecewise linear interpolant $\valueintp[1]_t$
    with the refined sparse grid.%
  }%
  \label{alg:financeRefine}%
\end{algorithm}

\paragraph{Gradient grids}

The classical surplus-refinement criterion focuses on
regions where the mixed second derivative
$\partialderiv[2d]{
  \partialdiff \stateentry_{t,1}^2 \dotsm \partialdiff \stateentry_{t,d}^2
}{
  \valueintp[1]_t
}$
of $\valueintp[1]_t$ has large absolute values, i.e.,
where $\valueintp[1]_t$ has large high-frequency oscillations.
In gradient-based optimization,
it might be advisable to apply this criterion also
to the partial derivatives
$\partialderiv{\partialdiff \stateentry_{t,o}}{\valueintp[1]_t}$
of $\valueintp[1]_t$ ($o = 1, \dotsc, d$),
since the optimizer depends on the accuracy of the gradient.
In this case, we have to track in \cref{alg:financeSolveValueFunction}
additional sparse grid interpolants for every partial derivative
$\partialderiv{\partialdiff \stateentry_{t,o}}{\valueintp[1]_t}$
that is affected by a policy variable.
This possibility is omitted from the algorithms in this section,
as it would unnecessarily complicate their presentation.

\subsection{Solution for Optimal Policies}
\label{sec:827solvePolicies}

\paragraph{\texttt{solvePolicies} algorithm}

After explaining the generation of the value function interpolants
$\valueintp[p]_t$ ($t = 0, \dotsc, T$),
we move on to step 2 of the general structure of our method
(see \cref{sec:821generalStructure}),
which is the generation of optimal policy interpolants.
The corresponding \cref{alg:financeSolvePolicies} is similar to
\texttt{solveValueFunction} (\cref{alg:financeSolveValueFunction}),
except that it operates on the policy instead of
the value function interpolants.
The functions \texttt{optimize}, \texttt{refine}, and \texttt{interpolate}
have been replaced by corresponding policy versions
\texttt{optimizePolicy}, \texttt{refinePolicy}, and \texttt{interpolatePolicy}
that work very much like their value function counterparts.
\texttt{optimizePolicy} only has to generate new values
if the initial regular sparse grid for the policies
is not contained in the grid of $\valueintp[p]_t$.
The policy grid is then refined and interpolated independently
of the value function grid.
The iterations over time are independent of each other,
which means that they can be parallelized.

\begin{algorithm}
  \begin{algorithmic}[1]
    \Function{%
      $\text{%
        $(\optpolicyintp[p]_t)_{t=0,\dotsc,T}$%
      } = \texttt{solvePolicies}$%
    }{%
      $(\valueintp[p]_t)_{t=0,\dotsc,T}$%
    }
      \State{$\valueintp[p]_{T+1} \gets \emptyset$}
      \Comment{dummy variable (is not used)}%
      \For{$t = 0, \dotsc, T$}
        \State{%
          $\optpolicyintp[1]_t \gets \text{%
            Initial regular sparse grid,
            retrieve values from $\valueintp[p]_t$%
          }$%
        }
        \State{%
          $\optpolicyintp[1]_t \gets
          \texttt{%
            optimizePolicy($t$, $\optpolicyintp[1]_t$, $\valueintp[p]_{t+1}$)%
          }$%
        }
        \State{%
          $\optpolicyintp[1]_t \gets
          \texttt{%
            refinePolicy($t$, $\optpolicyintp[1]_t$, $\valueintp[p]_{t+1}$)%
          }$%
        }
        \State{%
          $\optpolicyintp[p]_t \gets
          \texttt{interpolatePolicy($\optpolicyintp[1]_t$)}$%
        }
      \EndFor{}
    \EndFunction{}
  \end{algorithmic}
  \caption[%
    Generation of interpolants for optimal policies (\texttt{solvePolicies})%
  ]{%
    Generation of interpolants for optimal policies.
    The input is the higher-order B-spline interpolant $\valueintp[p]_t$
    of the value function for all $t = 0, \dotsc, T$.
    The output is the higher-order B-spline interpolant $\optpolicyintp[p]_t$
    of the optimal policies for all $t = 0, \dotsc, T$.%
  }%
  \label{alg:financeSolvePolicies}%
\end{algorithm}

\vspace{-0.5em}
\subsection{Post-Processing}
\label{sec:828postProecessing}
\vspace{-0.5em}

\paragraph{Monte Carlo simulation}

There are various ways to assess
whether the resulting optimal policy B-spline interpolants
$(\optpolicyintp[p]_t)_{t=0,\dotsc,T}$
are reasonable.
One possibility is a Monte Carlo simulation,
where we calculate the mean optimal policy
{%
  \setlength{\abovedisplayskip}{9pt}%
  \setlength{\belowdisplayskip}{9pt}%
  \begin{equation}
    \optpolicymean_t
    \ceq \frac{1}{m_\mathrm{MC}} \sum_{j=1}^{m_\mathrm{MC}} \optpolicyfcn_{t,(j)}
  \end{equation}%
}%
for $m_\mathrm{MC} \in \nat$ individuals.
The optimal policies $\optpolicyfcn_{t,(j)}$ of the individuals
($t = 0, \dotsc, T$ and $j = 1, \dotsc, m_\mathrm{MC}$)
are determined by
\begin{subequations}
  \setlength{\abovedisplayskip}{9pt}%
  \setlength{\belowdisplayskip}{9pt}%
  \begin{align}
    \optpolicyfcn_{t,(j)}
    &\ceq \optpolicyintp[p]_t(\state_{t,(j)}),\\
    \state_{t,(j)}
    &\ceq \statefcn_{t-1}(
      \state_{t-1,(j)}, \optpolicyfcn_{t-1,(j)}, \stochastic_{t-1,(j)}
    ),\quad
    t > 0,\qquad
    \state_{0,(j)}
    \sim P_{0,\state},\\
    \stochastic_{t,(j)}
    &\centerhphantom{\sim}{\hspace*{1.6em}} P_{t,\stochastic},
  \end{align}
\end{subequations}
i.e., the initial state $\state_{0,(j)}$ and the
stochastic variables $\stochastic_{t,(j)}$ are samples of random variables.
Monte Carlo simulations enable us to draw macro-economic conclusions,
e.g., the evolution of the amount of consumption of the average individual
over time.

%\paragraph{Euler equation errors}
%
%\dummytext[1]{}


\documentclass[a4paper,10pt]{article}
\usepackage[dvips]{color,graphicx}
\usepackage[dvips, bookmarks, colorlinks=false]{hyperref}

%opening
\title{Math508 Homework 7}
\author{Yu Huang}

\date{2007-03-03}
\begin{document}

\maketitle

\begin{abstract}
Viterbi Decoding Algorith to find optimal path
\end{abstract}

\section{No. 1}
Tossing of coins with initial coin randomly(uniformly) selected. Tossing continues possibly changing the coin with probability $1/3$. Assume we observed $Y_{[0,10]} = HHHHTHTTTT$.

The optimal path (order of coins chosen) found by Viterbi algorithm is
\begin{verbatim}
[1, 1, 1, 1, 3, 1, 3, 3, 3, 3]
\end{verbatim}

\section{No. 2}
Hidden markov model on a simple reflected r.w. $X_n$ on A[0,20] starting at 10. $P(W_1 = \pm L) = \frac{1}{2}$. $Y_n = min\{max\{X_n+W_n, 0\}, 20\}$. Simulate $X_n$, $Y_n$, $0 \le n \le 400$, for $L=10,14,16,17$, find two optimal path estimates $X^{\ast max}_n$ and $X^{\ast min}_n$ of $X_n$ by choosing maximal and minimal solution of the maximization problem in every Viterbi Algorithm step.

For $L=10$, check Figure~\ref{f2}; for $L=14$, check Figure~\ref{f3}; for $L=16$, check Figure~\ref{f4}; for $L=17$, check Figure~\ref{f5}.

\begin{figure}[p]
\includegraphics[width=1\textwidth]{hw7_2_K_20_L_10_T_400.eps}
\caption{}\label{f2}
\end{figure}

\begin{figure}[p]
\includegraphics[width=1\textwidth]{hw7_2_K_20_L_14_T_400.eps}
\caption{}\label{f3}
\end{figure}

\begin{figure}
\includegraphics[width=1\textwidth]{hw7_2_K_20_L_16_T_400.eps}
\caption{}\label{f4}
\end{figure}

\begin{figure}
\includegraphics[width=1\textwidth]{hw7_2_K_20_L_17_T_400.eps}
\caption{}\label{f5}
\end{figure}

\end{document}


\documentclass[
  xhtml,%
  use filename%
]{internet}

\usepackage{tutorial}
\usepackage{hyperref}

\title{Program with Style}
\date{\today}
\begin{document}
\maketitle

\section{Introduction}

With LuaCanvas, one can draw lots of different things on the screen and in different styles.
This tutorial will take you through the various things that can be drawn and how they can be styled.
Make sure you have \href{http://luacanvas.mathforge.org}{http://luacanvas.mathforge.org} open in a tab on your browser.

\section{What You Can Draw}

The default program is:

\begin{verbatim}
function setup()
  print("hello world")
end

function draw()
  background(40,40,50)
  fill(150,200,30)
  stroke(200,30,150)
  strokeWidth(10)
  rect(20,20,100,100)
end
\end{verbatim}

The key drawing line here is the second-to-last one: \verb+rect(20,20,100,100)+.
This draws a rectangle with lower-left corner at \((20,20)\) with width \(100\) and height \(100\).
You can experiment with changing those numbers to see what you get.
Try working out how wide and high your drawing space is by altering the numbers (you will get different answers depending on how big your browser window is).

You can also draw ellipses with the command \verb+ellipse+.
Change the \verb+rect+ to \verb+ellipse+ to see what happens.
Again, try changing the numbers.

As well as the rectangle and ellipse shapes, you can draw lines.
The command for drawing a line is:

\begin{verbatim}
line(sx,sy,tx,ty)
\end{verbatim}

\noindent where \((sx,sy)\) are the coordinates of the start of the line and \((tx,ty)\) are the coordinates of the end.
At the start, \((0,0)\) is the lower left corner of your drawing area.
For example

\begin{verbatim}
line(30,40,200,250)
\end{verbatim}

\noindent draws a line from \((30,40)\) to \((200,250)\).

Using a combination of \verb+rect+, \verb+ellipse+, and \verb+line+ you can draw lots of things on the screen.

There are currently\begin{marginpar}There are plans to implement more complicated shapes.\end{marginpar} two more commands that draw on the canvas: \verb+arc+ and \verb+bezier+.
The \verb+arc+ command draws an arc with a radius, starting angle, and ending angle.
The \verb+bezier+ command draws a special type of curve called a \emph{cubic b\'ezier} curve.
It takes four sets of coordinates: it starts at the first coordinate, leaves in the direction of the second, and it ends at the fourth coordinate arriving from the direction of the third coordinate.
To see an example, try:

\begin{verbatim}
arc(300,50,20,45,135)
bezier(200,200,200,300,300,200,300,300)
\end{verbatim}

So we might have a project currently looking like:

\begin{verbatim}
function setup()
  print("hello world")
end

function draw()
  background(40,40,50)
  fill(150,200,30)
  stroke(200,30,150)
  strokeWidth(10)
  rect(20,20,100,100)
  ellipse(50,200,30,20)
  line(300,50,400,100)
  arc(250,300,20,45,135)
  bezier(500,200,500,100,600,200,600,100)
end
\end{verbatim}

\section{Styling}

Styling commands change how the drawing appears.
Many involve colours, so we'll start with a word about colour.

\subsection{Colours}

Colours in LuaCanvas are specified by mixing red, green, and blue.
The amount of each mixed in is specified by giving a number between \(0\) (for none) to \(255\) (for maximum).
Thus \((0,0,0)\) is black and \((255,255,255)\) is white.
The fourth number is called the \emph{alpha} and determines how transparent the colour will be.
A \(0\) means that the colour is fully transparent, while \(255\) means that it is fully opaque.
In normal blending mode, a transparent colour won't be seen while an opaque one will obscure everything underneath it.
(It is possible to change this behaviour by changing the \verb+blendMode+.)

Sometimes it is useful to consider a colour as a single thing rather than having to specify its ingredients each time.
The command \verb+colour+ defines a ``colour object'' that can be used wherever a colour can be specified.
This means that you can save a colour into a variable for using time and again.
For example:

\begin{verbatim}
col = colour(255,255,0,255)
\end{verbatim}

\noindent stores a yellow colour in the variable \verb+col+.
When specifying a colour, you can use the red, green, blue, and alpha as above; but you can also use a colour name from the list of \href{https://en.wikipedia.org/wiki/Web_colors#X11_color_names}{SVG names} or \href{http://raebear.net/comp/emacscolors.html}{X11 names} (names are case-insensitive and can have spaces, so \verb+colour("Aqua Blue")+ and \verb+colour("aquablue")+ are the same thing).
For example:

\begin{verbatim}
col = colour("Yellow")
\end{verbatim}

Try adding some different colours to your project.
For example:

\begin{verbatim}
function setup()
  print("hello world")
end

function draw()
  background(40,40,50)
  fill(150,200,30)
  stroke(200,30,150)
  strokeWidth(10)
  rect(20,20,100,100)
  fill(colour("Teal"))
  ellipse(50,200,30,20)
  stroke(colour("Yellow"))
  line(300,50,400,100)
  arc(250,300,20,45,135)
  bezier(500,200,500,100,600,200,600,100)
end
\end{verbatim}

\subsection{Styling Commands}

There are a few styling commands which change how things appear on the screen.
The key thing to remember is that the commands affect all commands that appear after them.
Also, whatever is in effect at the end of the \verb+draw+ function will be in place at the start as well.

You can read the full description of the commands on the documentation.
In brief, they are:

\begin{itemize}
\item \verb+stroke(colour)+: sets the colour for drawing lines (and outlines of shapes).
\item \verb+fill(colour)+: sets the colour for filling shapes.
\item \verb+strokeWidth(w)+: sets the width of the lines (and outlines).
\item \verb+noStroke()+: disables drawing lines (and outlines).
\item \verb+noFill()+: disables filling shapes.
\end{itemize}

Two more useful commands are \verb+pushStyle()+ and \verb+popStyle()+.
The first of these saves the current style and the second restores the corresponding saved style.
For example:

\begin{verbatim}
function setup()
  print("hello world")
end

function draw()
  background(40,40,50)
  fill(150,200,30)
  stroke(255,0,0)
  strokeWidth(10)
  rect(20,20,100,100) -- outlined in red
  pushStyle() -- save the current style
  fill(colour("Teal"))
  ellipse(50,200,30,20) -- still in red
  stroke(colour("Yellow"))
  line(300,50,400,100) -- drawn in yellow
  popStyle() -- restore the old style
  arc(250,300,20,45,135) -- drawn in red
  bezier(500,200,500,100,600,200,600,100)
end
\end{verbatim}

There's also \verb+resetStyle()+ which resets the style back to the default.

Now try making different shapes in different styles on the screen.
If you have already worked through the \href{WhackARect.xhtml}{Whack A Rectangle} tutorial, you could try applying different styles to the rectangles that appear.
If you are feeling particularly adventurous, try using some random numbers in your colours.

\end{document}


\section{State of the Art\qsong{to be detailed...}}
In literature, the performance of ALOHA-based LPWAN are usually analyzed with Stochastic geometry. Baccelli et al.~\cite{baccelli2006aloha} use Laplace transform for interference analysis for slotted ALOHA. Haenggi et al.~\cite{haenggi2009interference} extensively study the outage probability in SINR-based capture model for slotted ALOHA. B{\l}aszczyszyn et al.~\cite{blaszczyszyn2010stochastic} open the door to study pure ALOHA with stochastic geometry. However, to our best knowledge, macro reception diversity for ALOHA systems has not been studied in literature.


\documentclass{article}
\usepackage[minionint,mathlf,textlf]{MinionPro} % To gussy up a bit
\usepackage[margin=1in]{geometry}
\usepackage{graphicx} % For .eps inclusion
%\usepackage{indentfirst} % Controls indentation
\usepackage[compact]{titlesec} % For regulating spacing before section titles
\usepackage{adjustbox} % For vertically-aligned side-by-side minipages
\usepackage{array, mathrsfs, mathrsfs, mhchem, amsmath} % For centering of tabulars with text-wrapping columns
\usepackage{hyperref, chemfig}
\usepackage{subfigure}
\newcommand{\Lapl}{\mathscr{L}}

\pagenumbering{gobble}
\setlength\parindent{0 cm}
\begin{document}
\large

\section*{Introduction to the issue of accuracy in translation}

A common problem in biological systems is the identification of a correct binding partner. Many incorrect binding interactions exist, and some of these will have on and off rates similar to the rates for the correct binding interaction.\\

A straightforward biological example of the problem occurs during translation. The ribosome's goal is to match the correct transfer RNA to a three-nucleotide codon on the mRNA. The mRNA interacts with the tRNA only through its codon, so the hydrogen bonds that make up base pairs may be the sole discriminator between correct and incorrect pairing. Steric hindrance may prevent some incorrect interactions as well, but at least in some cases, like G-U vs. G-C base pairing, the difference in interaction energy is expected to be largely attributable to the difference in number of hydrogen bonds formed. Since a G-U base pair forms two hydrogen bonds and a G-C base pair forms three, the primary energetic difference in binding of the correct vs. a certain incorrect tRNA is just the energy of a single hydrogen bond, approximately 10 kJ/mol.\\

Let's focus our attention on a ribosome $R$ and its interactions with two tRNAs: the correct one, $c$, and the incorrect tRNA $w$. We will assume that $c$ and $w$ differ from one another only in that $w$ forms a G-U base pair where $c$ forms a G-C base pair. The on and off rates for each binding event may differ, but it is most likely that the on rates are similar and diffusion-limited, so that only the off rates depend significantly on the free energy of binding. The two binding reactions are:
\[ \ce{c + R <=>[k_1][k_{-1}] cR ->[k_t] \textrm{ Correct AA}} \hspace{3 cm} \ce{w + R <=>[k_2][k_{-2}] wR ->[k_t] \textrm{ Incorrect AA}} \]
We will assume that the binding events are in equilibrium:
\[ \left[ c R \right] = \frac{k_1\left[ c \right]\left[ R \right]}{k_{-1}} \hspace{3 cm} \left[ w R \right] = \frac{k_2\left[ w \right]\left[ R \right]}{k_{-2}} \]
The rates of correct and incorrect product formation are $k_t [cR]$ and $k_t [wR]$, respectively, so the error rate (i.e. the ratio of incorrect to correct amino acid incorporations) will be:
\[ \textrm{ Error rate: } \hspace{0.5 cm} \frac{k_t  \left[ w R \right]  }{k_t  \left[ c R \right] } = \frac{k_2k_{-1}  \left[ w \right]  }{k_1k_{-2}  \left[ c \right] } \approx  \frac{k_{-1}  \left[ w \right]  }{k_{-2}  \left[ c \right] }  \]
The cell's solution cannot be to simply increase the concentration of the ``correct" tRNA, since for any given codon the correct tRNA will be different. The off rate in this case will be due to a difference in binding energy of 10 kJ/mol, so
\[ \frac{k_{-1}}{k_{-2}} = e^{10000 \textrm{ J/mol } / 8.3 \textrm{ J/mol$\cdot$K } \times 298 \textrm{ K}} \approx e^4 = 55 \]
An error rate of 1:55 is likely to be unacceptable: real amino acid misincorporation rates are closer to $10^{-4}$. This large discrepancy was resolved in the mid-seventies with the work of John Hopfield (1974), who introduced the concept of kinetic proofreading\footnote{Later, Hopfield was also instrumental in developing the theory of cyclic (``recurrent") artificial neural nets, a form of which still bears his name. Their stochastic counterpart, the Boltzmann machine, implement probabilistic behaviors in logic circuits. }.\\

Before explaining what kinetic proofreading is, I should clearly state what it isn't: kinetic proofreading in translation does \textit{not} involve incorporating the wrong amino acid into the polypeptide, detecting the error, and running the ribosome in reverse to fix the mistake. (That type of proofreading is used, however, by many nucleic acid polymerases.) Kinetic proofreading is a way to improve accuracy before the incorrect amino acid is added to the growing polypeptide chain.

\section*{Kinetic proofreading by analogy}

Uri Alon uses a wonderful analogy, which I'll modify slightly here to avoid repetition with your textbook, to describe kinetic proofreading.\\

Imagine that you would like to get as many Republicans as possible into one building, making as few mistakes as possible. (What you'll do with them once they're in there is for you to imagine.) You set up a mobile unit on the Boston common on a busy Saturday with the front door wide open, allowing any curious passerby to wander in. Inside, you set up a stereo to play the Rush Limbaugh show on loop. Republicans and others enter at the same rate, but Republicans take longer to leave: let's say, five minutes on average vs. one minute for everyone else. After the door has been open for a long time, you'll have about five-fold enrichment of Republicans compared to their frequency in the Boston tourist population. Obviously you can do better than that!\\

Now suppose that once the population in the room has come to equilibrium, you shut the front door and reveal a rear door with a one-way turnstyle so that it is only possible to exit. After three minutes, most Democrats will be gone and most Republicans will have stayed.

\section*{Kinetic proofreading }
This basic principle -- imposing a delay during which an irreversible unbinding can occur which is likely to eliminate more incorrect than correct interactions -- is also at play in translation. The equivalent of the one-way door in prokaryotic translation is elongation factor thermo-unstable (EF-Tu). EF-Tu binds each tRNA long before it enters the ribosome and is essentially required for the tRNA to begin interacting with the ribosome. Once the tRNA is in place, in order for translation to proceed, EF-Tu must be released in a reaction catalyzed by GTP hydrolysis. Once EF-Tu is gone, there is still an opportunity for the tRNA to fall off before amino acid incorporation occurs: this is a one-way reaction, since it would be impossible for a tRNA to simply bind to the ribosome at this stage. Therefore the complete reaction scheme looks more like:
\begin{center}
\schemestart
 c + R
 \arrow{<=>[$k_1$][$k_{-1}$]}
 cR
 \arrow{->[$k_d$]}
 cR$^*$
 \arrow{->[*0$\ell_1$]}[-90]
 c + R
 \arrow(@c3--){->[$k_t$]}
 Correct incorporation
\schemestop
\end{center}
The scheme for the incorrect tRNA incorporation is similar: $k_d$ is assumed to be the same, but the irreversible dissociation rate after GTP hydrolysis will differ and is therefore labeled $\ell_2$. The codon-anticodon binding is not affected by GTP hydrolysis and EF-Tu association, so $\ell_1/\ell_2 = k_{-1}/k_{-2}$. Proceeding as before, assuming that the initial binding step is at equilibrium:
\[ \frac{d \left[ c R^* \right]}{dt} = k_d \left[ c \right]\left[ R \right] - \left( \ell_1 + k_t \right) \left[ c R^* \right] \hspace{1 cm} \frac{d \left[ w R^* \right]}{dt} = k_d \left[ w \right]\left[ R \right] - \left( \ell_2 + k_t \right) \left[ w R^* \right] \]
Therefore the error rate is:
\[ \textrm{ Error rate: } \hspace{0.5 cm} \frac{k_t  \left[ w R^* \right]  }{k_t  \left[ c R^* \right] } =  \frac{k_{-1} \left( \ell_1 + k_t \right) \left[ w \right]  }{k_{-2} \left( \ell_2 + k_t \right) \left[ c \right] } \approx \frac{k_{-1}^2 \left[ w \right]}{k_{-2}^2 \left[ c \right]} \]
This gets us into the regime where we can explain the ribosome's low error rate with a hundred-fold difference in off rate between the correct and incorrect tRNAs. We would see improvement if we added additional kinetic proofreading steps.

\section*{Time delay}
The improved accuracy gained by kinetic proofreading is purchased in gratuitous GTP hydrolysis and decreased overall translation rates, the latter because of the delay used to allow more incorrect tRNA to dissociate before proceeding to amino acid incorporation. This can be calculated by generating a matrix $\mathbf{P}$ corresponding to the master equation for the system: $P_{ij}$ is the probability of transitioning from state $i$ to state $j$ in one time step. The first step will be considered the state with unbound ribosome, and the ``final" $n$th state will be an imagined absorbing state which the system enters once either type of amino acid has been incorporated. Then the expected time for transition from state 1 to $n$ is:
\[ T = \sum_{k=0}^{\infty} k \left[ \mathbf{P}^k \right]_{1n} = \left[ \mathbf{P} \right]_{1n} + 2 \left[ \mathbf{P}^2 \right]_{1n} + 3 \left[ \mathbf{P}^3 \right]_{1n} + \ldots \]
which it turns out is roughly proportional to all of the reverse reaction rates and inversely proportional to all of the forward reaction rates, viz. $T \sim k_{-1} \ell_1 / k_1k_dk_t$.$T$ is large because $k_{-1} \gg k_d$ (rapid equilibrium assumption for initial tRNA binding) and $\ell_1 \gg k_t$. In your discussion paper this week, you'll learn more about the trade-off in expected time elapsed per reaction vs. error rate motivated by analogy to dynamical instability in microtubules.

\section*{Lateral inhibition through Delta-Notch signaling}

You are probably used to thinking of the ligand in an intercellular signaling system as a free-floating secreted molecule, but in this case, the ligand Delta is a transmembrane protein. By keeping Delta attached to the cell that produced it, this system ensures that it will only signal to immediate neighbors.\\

Delta's receptor is called Notch. Named for the phenotype its mutation produces in the fruit fly wing, this gene has been known to science since the 1910s, when it was identified and mapped. Delta is also named for its phenotype in the wing (where it refers to the shape of the veins near the edge), though Delta and Notch are used in many developmental contexts in many organisms, notably including the pattern of mechanosensory bristles along the fly's back. Binding of Notch to Delta causes the intracellular domain of Notch (NICD) to be cleaved off and enter the nucleus, where it functions as a transcription factor.\\

Activation of Notch by ligand binding typically causes the cell to downregulate its own expression of Delta and activate a positive feedback loop that enhances the sensitivity of its own Notch proteins. If signaling is prolonged, as expected if a neighboring cell continues to express Delta, then the cell assumes a Notch$^+$ Delta$^-$ identity. Failing to receive a signal from Notch, however, does not result in a stable a Notch$^-$ Delta$^+$ identity.\\

This patterning system is used in many contexts throughout development. In the case of bristle patterning, its goal is to establish a certain ratio and spatial pattern of Delta$^+$ vs. Notch$^+$ cells (the descendants of each Delta$^+$ cell will eventually form a bristle). This feat is accomplished by amplifying small initial differences in gene expression between cells (the result of noisy gene expression or protein localization).

\subsection*{A synthetic lateral inhibition system}

Earlier this year, Matsuda et al. reported reconstructing Notch-Delta signaling in q mammalian cells that normally lacks this pathway by expressing four components: Notch, Delta under a promoter containing tetO (repressed by tTS), tTS under a promoter activated by Notch signaling (pTP1), and lunatic fringe (Lfng) -- an enhancer of Notch which acts by glycosylating it -- under pTP1 to mimic the positive feedback loop of Notch activity normally induced by Notch signaling. (Other components that enhance Delta-Notch signaling may also have been natively produced by the cells, though this particular cell line does not express crucial components of the pathway natively.) A fluorescent marker for the Delta$^+$ state is placed under the same promoter as Delta; similarly, a fluorescent marker for the Notch$^+$ state is placed under the same promoter as tTS.\\

The authors used Chinese hamster ovary (CHO) cells\footnote{This peculiar-sounding cell type rose to prominence, particularly in industrial recombinant protein expression, because of its ease of transformation and selectable markers.}, an ``epithelial-like" cell line which grows in a monolayer without extensive contact between cells. By contrast, the epithelium of the fruit fly involved in bristle patterning is columnar, with cells squished against their neighbors in the plane. One result of this difference is that when Matsuda et al.'s synthetic system is cultured under typical conditions, neighboring cells may not signal strongly to one another. Enough signaling occurs, however, to get a robust ratio between cell types (i.e. the ratio is restored after it is perturbed via cell sorting). Furthermore, when the cell line is cultured on special substrates that force two daughter cells to remain in close contact after cell division, one cell always assumes each fate, even though initially their gene expression pattern was identical (modulo segregation at mitosis). Forcing cells to overexpress N-cadherin, an intercellular cell adhesion protein, also reduced the frequency of neighboring Delta$^+$ cells.\\

The problem of modeling this system deterministically becomes apparent when we would like to write down, for example, the rate of change in Notch concentration. Clearly we need to keep track of Notch, etc. expression levels in each cell individually and to have some notion of which cells are adjacent to one another. The authors use, e.g.:
\[ \frac{dN}{dt} = \beta_N - \gamma_N N - \frac{k_L L N}{k_m + N} - k_t N \left< D \right> - k_c N D \]
The first two terms reflect the rate of production and decay in the absence of other proteins, the third term reflects glycosylation by lunatic fringe ($L$), the fourth represents activation of Notch by Delta on other cells, and the fifth represent self-activation. The issue is how to define $\left< D \right>$, the expression of Delta in cells which neighbor the cell of interest. Clearly the cells must be modeled with a physical position and shape in order to determine which are their nearest neighbors.\\

Spatial modeling is the art of accounting for these spatial inhomogeneities without breaking the complexity bank. A simple solution is to assume that no divisions are occurring and place cells on a square or hexagonal lattice. Sometimes this approximation is perfectly justifiable: many columnar epithelia even look like hexagonal lattices from above. However, the geometry of a lattice can impose patterns which would not necessarily be present in the geometry of the real system.\\

A more computationally taxing alternative is to arrange cells on an amorphous lattice. To explore the challenges in constructing this lattice, consider starting with a small initial cluster of circular cells and adding each new cell at a random location on the edge, requiring that it touch at least two other cells. It can be shown (Rubinstein and Nelson, 1982) that even with this form of random placement one arrives at something very closely approximating a hexagonal lattice. A better option is to place cells of different sizes or shapes, or (if physical adjacency is not an expectation) to place cells randomly on the plane at sufficiently low density. The latter option is taken by Matsuda et al.\\

The importance of cell shape becomes clear when we consider that many cells in the epithelium where bristle patterning occurs make long filopodial extensions that may contact targets multiple cell diameters away. To make matters worse, these extensions are dynamic: that is, they are extended and retracted.\\

Even with the geometry ironed out, the deterministic model cannot alone account for the system's behavior, since it would not explain why an isolated pair of daughter cells adopts different cell fates (or, for that matter, in an epithelium where all cells are assumed initially identical). We require at least some initial difference between these cells, which ultimately is provided by noise in gene expression or in segregation of protein products at mitosis. Thus noise is a driving factor in the adoption of different cell fates through lateral inhibition.

\section*{Heterocyst formation}

In the section of this course on oscillators, I mentioned that the apparent value of the cyanobacterial circadian clock is to balance the competing pressures of nitrogen fixation and photosynthesis by relegating each to a different portion of the day. An alternative strategy for segregating nitrogen fixation and photosynthesis has also evolved in some multicellular cyanobacteria which form long (and in some clades, branched) filaments one cell wide. To understand the mechanism I'll soon present, it is worthwhile to note that multicellularity has evolved in these gram-negative bacteria by preventing the septation of the outer cell wall, so that the bacteria are containined in a long tube where signaling molecules can diffuse without convected away by currents or diluted to kingdom come.\\

This alternative approach involves creating a specialized cell type called a heterocyst, which performs nitogen fixation but not photosynthesis. It suffices to intersperse these cells at regular intervals along the filament for every cell to have sufficient access to fixed nitrogen. However, heterocysts are a somatic cell type incapable of further division or de-differentiation, so as their neighbors divide, the average distance between heterocysts increases, and it is necessary for new heterocysts to differentiate in between.\\

Suppose someone na\"{i}ve -- Donny Do-Not, if you will -- were to design a control system for the differentiation of heterocysts based on a simple sensor for nitrogen availability: any cell with sufficiently poor access to fixed nitrogen would differentiate into a heterocyst. At first this option appears attractive: as the filament grows, some cell equidistant between two heterocysts will eventually become nitrogen-starved, resulting in a new heterocyst right where we need it. However, there is nothing to prevent two heterocysts from forming right beside each other in this location by chance, which is a waste to the multicellular organism, since heterocysts are reproductively dead. Furthermore, if the filament were grown in a nitrogen-rich environment and the media suddenly changed, every cell would become a heterocyst.\\

Biology does not do what Donny Do-Not does. Instead, a more direct form of lateral inhibition is used: cells begin to produce a ``signaling" protein, PatS, as soon as they can after committing to the heterocyst fate. PatS then diffuses away from its source in both directions down the filament, where its binding to a heterocyst transcription factor inhibits other cells from differentiating into heterocysts. The number of ``erroneous" neighboring heterocysts formed is still elevated, however, after a transfer back to nitrogen-poor media.

\end{document}

lectures/Lecture 26 - Kinetic Proofreading; Lateral Inhibition/lecture notes/lecture 26 notes.tex