\subsection{Actuators and Sensors}
Here we consider the actions that may be chosen to be performed by actuators and percepts that may be received by sensors. The problem of \textit{target localisation} in the context of this chapter requires the agent to move around a discrete grid and use a calibrated sensor to record noisy readings that indicate whether the target is present or not at the location of the reading. We therefore describe the set of possible actions to be performed by the actuators by the set of all $n$ possible grid locations that the agent can move to and take a sensor reading at, indexed by an arbitrary ordering: $\{move\_x_1, move\_x_2, ..., move\_x_n\}$. We did not restrict the agent to only move between adjacent grid cells since our use case deals with agile aerial vehicles, which can move freely between any two grid points. We add search termination actions to this set, $\{terminate\_search\_x_{i}\}$, for $i \in \{1, 2, ..., n, n+1\}$, which lead to an absorbing terminal state representing the agent's conclusion regarding whether a target is present or not, $terminated\_x_{i}$. To summarise, the agent may either choose a move action or a search termination action, which respectively move the agent to a new grid location at which they record a sensor reading, or conclude the search and return the most likely target location $x_i$. \par

The set of percepts that the agent will receive from its sensors come from the binary set \{1, 0\}, indicating the target has or has not been detected, respectively.\par


\section{Drone learning}
\label{sec:dlearn}

The training of the drone network requires that the original network is
extensively probed in the parameter space in which accuracy is desired.
The principle utilised in the training of the drone is that sufficient
approximation of the original network is achieved with sufficient expansion
of the hyperparameter space of the drone, and that the same global minimum
of the loss function can be found, as reported in Ref.~\cite{losssurfaces}.
The ability of a neural network with a continuous, bounded, non-constant activation
function to approximate functions to an arbitrary degree has been indeed known
since the early 1990s~\cite{HORNIK1991251}.

\subsection{Initial drone structure and corresponding training}

The drone chosen for use in this article is initialised as a
neural network with a single intermediate (hidden) layer of 5
nodes using a standard sigmoid activation function. The network
has the number of inputs determined from the number of desired
characteristics of the decay signature. A single output is taken
from the network and a linear model is used to relate layers.

The model is made to approximate the original classifier through
a supervised learning technique, though not in the traditional sense.
Instead of a label as {\tt signal} or {\tt background} taken from the training data, the
output of the original classifier is used as a label. This means that the
loss function is defined as
\begin{align}
\mathcal{L} = \sum_i \left( F(\vec{x}_i) - G_i(\vec{x}_i) \right)^2,
\end{align}
where $F(\vec{x}_i)$ and $G(\vec{x}_i)$ are the outputs
of the original and drone models on datapoint
$i$ of the mini-batch, respectively. The advantage of such a loss function is per-event
equivalence of the original and drone model, in addition to equivalence
of performance. For the drone training detailed in this article, standard
mini-batch stochastic gradient descent is used. A feature of this method
is that the drone classifier does not see any training data,
but rather learns the same properties from the original classifier,
and thus is a neural network that learns from another neural network in an
empirical manner.

\subsection{Model morphing during the learning phase}

In order to keep the hyperparameter space to the minimum required level,
additional degrees of freedom are added only when required.
This removes the possibility of choosing an incorrect size of the
drone network. During the learning phase, the following conditions are required
to trigger the extension of the hidden layer in the $j^{\rm th}$ epoch:
\begin{align}
\delta_{j} &\equiv |\mathcal{L}_j-\mathcal{L}_{j-1}|/\mathcal{L}_j < \kappa,\label{eq:cond1}\\
\sigma_{j} &\equiv m (1 - e^{-b(\hat{t} + n)})\delta_{j}\mathcal{L}_j \nonumber\\
\mathcal{L}_j &< \hat{\mathcal{L}} - \sigma_{j} \label{eq:cond2},
\end{align}
where $\kappa$ is the required threshold, $\sigma$ is the required minimum improvement
of the loss function and $\hat{\mathcal{L}}$ is the value of the loss function when
the hidden layer was last extended. The required improvement starts from a minimum at $n$,
increases with epoch number after previous extension $\hat{t}$ and steepness $b$
until a maximum at $m$. The precise values of the parameters
$\kappa$, $n$, $m$, $b$ are not of particular importance. Rather, the topology described by
eqs.~\ref{eq:cond1} and \ref{eq:cond2} is crucial. The relative loss function improvement,
$\delta_{j}$, can never realistically be larger than $1$ and the limit, $\kappa$, at which
no significant improvement occurs is acceptably set at 0.02 (smaller than $2\sigma$
standard deviations). The descent in loss space, $\hat{\mathcal{L}} - \mathcal{L}_j$,
is further required to be significantly large, minimizing the chance of getting stuck in
isolated local minima. The function, $\sigma_{j}$, is chosen to increase this requirement
with each epoch for two reasons - it is bounded and can approach its asymptote arbitrarily fast.
It scales $\delta_{j}$ such that the loss descent must be significant
before an update is triggered. Since $\delta_{j}$ is expected to decrease with epoch number,
the minumum and maximum values of $\sigma_{j}$ are chosen as such:
\begin{align}
\sigma_{j}(\hat{t} = 0) &\equiv min(\sigma_{j}) \equiv 2.5\delta_{j}\mathcal{L}_j \implies 5\sigma ~\text{std.dev.} \\
\sigma_{j}(\hat{t} = \infty) &\equiv min(\sigma_{j}) \equiv 25\delta_{j}\mathcal{L}_j \implies 50\sigma ~\text{std.dev.}
\end{align}
The steepness, $b$, is chosen such that the transition from the minimum to maximum takes
on average 50 epochs. This ensures a change cannot be triggered immediately
after a previous one and the learning can still proceed if more freedom is indeed required.
Also, it allows the network to stabilize after a big change.

When the conditions in eqs.~\ref{eq:cond1} and \ref{eq:cond2} are met, the linear model
is updated to extend the weights matrices and bias vectors
to accommodate the layer addition.
The associated neurons are initialised with a zero weight
to ensure continuity of the loss function value.


\input{../../assignment-header}

% Special settings for matlab code on exam
\lstset{morekeywords={runSloppySimulation,runSimBadly,simStepMidpointMethod}}

\newcommand{\scoreMark}[0]{$\rule{1.5cm}{0.15mm}$ / $\rule{1.5cm}{0.15mm}$}

%========================================================================
\title{ME 149:  Midterm Exam}
\date{March 15  ---  Start: 6:00pm  ---  End: 7:15pm}
\author{Optimal Control for Robotics}
%========================================================================
\begin{document}
\maketitle

\begin{abstract*}
No calculators, notes, books, or computers allowed. Total time:  75 minutes.
\end{abstract*}
%=================================================

\vspace{1em}

\section*{Student Name:  $\rule{10cm}{0.3mm}$}

\vspace{1em}

\section*{How to optimize your score?}

\begin{itemize}  \setlength\itemsep{0.1em} \setlength\itemindent{18pt}
  \item Be neat and well organized
  \item For longer problems, show intermediate steps and box your
  \fbox { \parbox{3em}{answer} }
  \item If you need extra space...
        use the back of the page,
        the final (blank) page of the exam,
        or ask for more paper.
        Clearly indicate where the extra work is.
  \item Define all variables that you use and state any assumptions that you make.
\end{itemize}

\vspace{1em}

\section*{Score:  \scoreMark}
\vspace{1em}
\begin{large}
\begin{multicols}{2}
\begin{enumerate}  \setlength\itemsep{1.1em} \setlength\itemindent{36pt}
\item \scoreMark
\item \scoreMark
\item \scoreMark
\item \scoreMark
\item \scoreMark
\item \scoreMark
\item \scoreMark
\item \scoreMark
\item \scoreMark
\end{enumerate}
\end{multicols}
\end{large}

% Force enumerated lists to use letters
\renewcommand{\theenumi}{\Alph{enumi}}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\pagebreak
\section{Newton's Method}

\begin{enumerate}
  \item Suppose that you want to solve the scalar nonlinear equation $f(x) = 0$. \\
        The current estimate of the root is given by $x_k$
        and the next estimate of the root is given by $x_{k+1}$.
        Derive the Newton-Rhapson update that computes $x_{k+1}$ given $x_k$.
  \vspace{25em}
  \item What is the difference between the Newton--Rhapson
        and the secant methods for scalar root finding?
        In what situation would you prefer one to the other?

  \pagebreak
  \item Draw a figure that clearly demonstrates a situation in which
        Newton's method will fail to converge to a root of the function.
        Why does the Newton--Rhapson method fail in this situation?
  \vspace{25em}
\end{enumerate}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\section{Midpoint Method Implementation}

Implement the function \texttt{simStepMidpoint()} on the following page.
This function computes a single simulation step using the midpoint method.
You code should be clear, correct, and follow the best practices that
have been discussed throughout the course.
Use the space below for planning your solution.
Write your Matlab code inside of the function template on the following page.

\pagebreak
\lstset{stepnumber=0}
\lstinputlisting{simStepMidpoint.m}

\pagebreak
\section{Bisection Search}
\begin{NoHyper}
Use a bisection search to iteratively reduce the interval that is known
to bracket the root of the function shown in Figure \ref{fig:RootSolveExampleFigure}.
\textbf{Populate the table below}, showing the bracket for the first five iterations
and the new point that will be evaluated on that iteration.
\end{NoHyper}
\begin{itemize}  \setlength\itemsep{0.3em}
  \item \texttt{Iter 0:   bracket: [-1.000,  \hspace{2.9em} 1.000]  \hspace{1em}  xNew = }
  \item \texttt{Iter 1:   bracket: [ \hspace{10em} ]  \hspace{1em}  xNew = }
  \item \texttt{Iter 2:   bracket: [ \hspace{10em} ]  \hspace{1em}  xNew = }
  \item \texttt{Iter 3:   bracket: [ \hspace{10em} ]  \hspace{1em}  xNew = }
  \item \texttt{Iter 4:   bracket: [ \hspace{10em} ]  \hspace{1em}  xNew = }
\end{itemize}

\begin{figure}[ht]
	\centering
  \includegraphics[width=\textwidth]{RootSolveExampleFigure.pdf}
  \caption{Bisection Search Example Figure}
  \label{fig:RootSolveExampleFigure}
\end{figure}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\pagebreak
\section{Scalar Taylor Series}

Write Taylor series approximation of $f(t)$ to second-order around the point $t_0$.

\vspace{2em}
\begin{equation*}
  f(t) \approx \hspace{50em}
\end{equation*}
\vspace{7em}

\section{Vector Taylor Series}

Write the Taylor series approximation of $f(t, \bm{x}, \bm{u})$ to first-order
about the point: $t_0$, $\bm{x}_0$, and $\bm{u}_0$.

\vspace{2em}
\begin{equation*}
  f(t, \bm{x}, \bm{u}) \approx \hspace{46em}
\end{equation*}
\vspace{4em}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\pagebreak
\section{Function Handle Gymnastics}
\textbf{What will the following script print to the command prompt?} \\
For each part A, B, C, show your work and then put a box around
the text that Matlab will print to the command prompt.

\lstinputlisting{functionHandleGymnastics.m}

\subsection*{Part A:}
\vspace{7em}
\subsection*{Part B:}
\vspace{9em}
\subsection*{Part C: }
\vspace{11em}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\pagebreak
\section{Matlab Programming Style}
The program below simulates a simple pendulum. It works, but is written poorly. \\
\textbf{Clearly identify at least 5 distinct issues with the code.}

\lstset{stepnumber=1}
\lstinputlisting{runSloppySimulation.m}

\vspace{30em}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

\pagebreak
\section{Linearized Dynamical System}

Given the non-linear system dynamics described below,

\begin{equation*}

  \dot{\bm{z}} =

  \begin{bmatrix}
    \dot{\alpha}  \\
    \dot{\gamma}  \\
    \dot{\theta}
  \end{bmatrix}

  =

  \begin{bmatrix}
    \beta * \gamma + \theta^2 \\
    \alpha \, \sigma  - \theta\\
    \sin(\sigma) + \beta \, \gamma
  \end{bmatrix}

  = \bm{f}(\bm{z}, \bm{u})

  \quad \quad \quad \quad \quad \quad

  \bm{z} =
  \begin{bmatrix}
    \alpha \\
    \gamma \\
    \theta
  \end{bmatrix}

\quad \quad \quad \quad \quad \quad

  \bm{u} =
  \begin{bmatrix}
    \beta \\
    \sigma
  \end{bmatrix}

\end{equation*}

\vspace{0.5em}
\begin{enumerate}
  \item \textbf{List the state variables: }
  \vspace{0.6em}
  \item \textbf{List the control variables: }
  \vspace{0.6em}
  \item \textbf{What is the difference between a state and a control variable? }
  \vspace{4em}
  \item \textbf{Compute: } $\dfrac{\delta \bm{f}}{\delta \bm{z}} = $
  \vspace{15em}
  \item \textbf{Compute: } $\dfrac{\delta \bm{f}}{\delta \bm{u}} = $
  \vspace{15em}
\end{enumerate}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\pagebreak
\section{Trajectory Optimization}

Given the following continuous-time trajectory optimization problem.

\begin{align*}
  & \text{minimize: } \qquad \int_0^T \! \bm{g}(t,\, \bm{x},\, \bm{u}) \, dt \\
  & \text{subject to: } \qquad \bm{0} = \bm{h}(\bm{x}(0), \, \bm{x}(T)) \\
  & \text{dynamics: } \qquad \dot{\bm{x}} = \bm{f}(t,\, \bm{x},\, \bm{u}) \\
\end{align*}

Suppose that you plan to solve the optimization using
direct multiple shooting with Euler's method
on a uniform grid of $N$ segments with one integration step per segment.
Write out the decision variables,
objective function, and constraints that will form the resulting non-linear program.
The objective function and constraints should be written in terms of the
decision variables, known parameters ($T$),
and known functions ($\bm{g}$, $\bm{h}$, $\bm{f}$).

\begin{enumerate}
  \item \textbf{Decision Variables}
  \vspace{4em}
  \item \textbf{Objective Function}
  \vspace{6em}
  \item \textbf{Boundary Constraints}
  \vspace{10em}
  \item \textbf{System Dynamics Constraints}
  \vspace{10em}
\end{enumerate}

\pagebreak
\section*{Blank page for additional work space for any problem.}

%=================================================
\end{document}


\subsection{How code \acrshort{twm} actions}%
\label{sub:how_code_twm_actions}

In this section we will discuss of how we codify actions for the \gls{twm} as defined in section~\ref{sub:definition_of_the_twm_problem}.
The \gls{twm} is a multi-agent problem.
Each team of firefighter can be control on their own.
We decided to have them play in sequence and not in parallel.
Meaning that, we identify each team by a number and then, team 1 play, then team 2, etc\dots
This way, the algorithm can place team one by one.
It is important to do so because teams can be stacked on the same cell to extinguish it faster or can be spread across the grid to cover the fire.

Each cell \(x \in X\) can be seen as a pair of coordinates \((i, j)\).
Each cell can then be associated to an index \(k\) such as \(k=(j \times W + i)\).
As said before each firefighter team \(t \in T\) is associated to a number \(o\) which represents its turn order from \(0\) to \(|T| - 1\).
So we define the function \textit{code}  as follows:

\[
    code(x, s_{n}) = k_{x} + (o_{n} + |T|B_{t}(x)) * |X|
\]

where \(x\) is the cell on which we want to send the next team \(t in T\) and \(s_{t}\) is a state of the game at turn \(n\).

With this definition a code holds three information:

\begin{itemize}
    \item which cell \(x \in X\) is targeted
    \item which team is send to the cell
    \item is the cell burning or not
\end{itemize}

Therefore, the \gls{snrpa} can make the difference between action for each team of firefighters and action targeting burning cells or not.


%!TEX root = ../notes.tex
\section{March 17, 2022}
\subsection{Midterm Review}
\subsubsection*{General Advice}
\begin{itemize}
    \item 5-7pm. Location: Barus \& Holley 168.
    \item There are 5 problems:
          \begin{itemize}
              \item Each are weighted equally, some have multiple sections in them.
              \item There is a bonus problem for a \emph{token} number of points.
          \end{itemize}
    \item Think about problems before starting! Don't begin immediately.
\end{itemize}

\subsubsection*{Key Topics}
\begin{enumerate}[1)]
    \item
          Unique factorization in $\ZZ$ (\cref{thm:unique-factorization}). Key points:
          \begin{itemize}
              \item Existence (using well-ordering of $\ZZ_+$)
              \item Uniqueness (using prime elements being irreducible elements in $\ZZ$)
          \end{itemize}
    \item
          $\ZZ$ is a Euclidean domain with Euclidean function $\mathsf{abs}$ (absolute value) (\cref{cor:z-euclidean}).
          \begin{itemize}
              \item Argument uses well-ordering of $\ZZ_+$ applied to the set $S = \{a - bq\mid b\in \ZZ\}$ when trying to divide $a$ by $b$.
              \item Repeated application of this property yields the Euclidean algorithm for finding $\gcd$'s.
          \end{itemize}
    \item
          Bezout's Identity (\emph{not} Bezout's Theorem)

          If $a, b\in\ZZ$ are integers (not both $0$) and $c\in \ZZ$, then there exists $x, y\in\ZZ$ such that
          \[ax + by = c\]
          if and only if $\gcd(a, b)\mid c$.
          \begin{itemize}
              \item We take set $S = \{ax + by\mid x, y\in\ZZ\}$ and use well-ordering to show that the smallest element has to be $c$.
          \end{itemize}
    \item
          From Bezout to solving linear congruences in $1$ variable, the linear congruence
          \[ax\equiv b\pmod{m}\]
          is equivalent to
          \[ax - my = b\]
          for some $y\in \ZZ$. Applying Bezout's tells us that this equation is solvable if and only if $\gcd(a, b)\mid b$. When a solution exists, there are $d$ solutions modulo $m$.
          \begin{itemize}
              \item Showing there are $d$ solutions: you divide $a, b, m$ by $\gcd(a, m)$, then you have a modulus $\frac{m}{\gcd(a, m)}$ where we have a unique solution. We lift up to solutions modulo $m$.
          \end{itemize}
    \item
          Sunzi's theorem (\cref{thm:crt}). For $m, n\in\ZZ_+$ with $(m, n) = 1$. And $a, b\in\ZZ$, then the simultaneous congruences
          \begin{align*}
              x\equiv a\pmod{m} \\
              x\equiv b\pmod{n}
          \end{align*}
          have a \emph{unique} solution modulo $mn$.
          \begin{itemize}
              \item We have $\pi : \ZZ/mn\ZZ\to \ZZ/m\ZZ\times \ZZ/n\ZZ$ be the natural projectsion where $\ker(\pi) = \{0\}$ since $(m, n) = 1$.
          \end{itemize}
    \item
          Structure of group of units (\cref{cor:cyclicity-of-unit-groups}). $U(m)$ is cyclic $\iff$ $m = 1, 2, 4, p^e, 2p^e$.
\end{enumerate}

\subsubsection*{Practice Problems}
\begin{problem}
Find the integer $0\leq a\leq 36$ such that
\[3777^{\left(1144523^{56245501}\right)} \equiv a\pmod{37}\]
\end{problem}
We can reduce the base $3777\equiv 3\pmod{37}$. We reduce $1144523\equiv 11\pmod{\phi(37)}$. We can reduce the upper power $56245501\equiv 1\pmod{\phi(\phi(37))}$. This reduces to
\[3^{11}\equiv a\pmod{37}\]
which gives $a\equiv 28\pmod{37}$.

\begin{problem}
Let $p\in\ZZ$ be a prime and let $g$ be a primitive root mod $p$. Describe the set
\[\{g^k\mid g^k \text{ is a primitive root mod $p$}\}\]
\end{problem}
\begin{proof}
    We claim that $\gcd(k, p-1) = 1$. Then for any element $a = g^\alpha$, we can find power $(g^k)^\beta = g^\alpha$ since we have $g^{p-1}\equiv 1$ so $k\beta - x(p-1) = \alpha$ for some $x$, which only has solutions by Bezout's identity when $\gcd(k, p-1)$.
\end{proof}

\begin{lemma*}
    Prove that for any finite group $G$ of order $n$ and any $g\in G$, the cyclic group $\langle g^k\rangle$ for $k$ such that $\gcd(k, \ord(g)) = 1$ equals $\langle g\rangle$.
\end{lemma*}
\begin{proof}
    Let $d = (k, \ord(g))$. Then there exists $x, y\in\ZZ$ such that
    \[\ord(g)\cdot x + k\cdot y = d\]
    so
    \begin{align*}
        g^d & = g^{\ord(g)\cdot x + ky}        \\
            & = g^{\ord(g)\cdot x}\cdot g^{ky} \\
            & = g^{ky}
    \end{align*}
    so $g^d\in \langle g^k\rangle \implies \langle g^d\rangle \subseteq \langle g^k\rangle$. We have $\langle g^k\rangle\subseteq \langle g^d\rangle$ since $d\mid k$. Thus $\langle g^d\rangle = \langle g^k\rangle$.

    We also have that $(g^k)^{\ord(g)/d} = (g^{\ord{g}})^{k/d} = 1$ so if $d = (k, \ord(g))>1$ then $\ord(g^k)< \ord(g)$.

    So together we have that $\langle g\rangle = \langle g^k\rangle$ if and only if $(g, \ord(g))=1$.
\end{proof}

\begin{problem}
Prove
\begin{proposition*}
    If $f : \ZZ_+\to \CC$ is a nonzero multiplicative function, then $f^{-1}$ (the Dirichlet inverse) exists and is multiplicative.
\end{proposition*}
\end{problem}
\begin{proof}
    Let $h$ be given by
    \begin{align*}
        h(p^k) & = f^{-1}(p^k)\qquad\text{prime powers $p^k$} \\
        h(n)   & = h(p_1^{e_1})\cdots h(p_k^{e_k})
    \end{align*}
    then $(f\star h)(p^k) = I(p^k)$. Both $f\star h$ and $I$ are multiplicative, so
    \[(f\star h)(n) = I(n)\quad\forall n\in\ZZ\]
    and $h = f^{-1}$.

    (Existence, $f(1) = 1$ for any multiplicative function, so in particular our given $f$ satisfies $f(1)\neq 0$.)
\end{proof}

\begin{problem}
Define $\lambda : \ZZ_+\to \CC$ by
\[\lambda(n) = (-1)^{e_1 + e_2 + \cdots}\]
where the $e_i$'s are the exponents on the prime factorization of $n$. Let
\[g(n) = \sum_{d\mid n}\lambda(d)\]
Prove that
\[g(n) = \begin{cases}
        1 & \text{if $n$ is square} \\
        0 & \text{otherwise}
    \end{cases}\]
\end{problem}
\begin{proof}
    We note that $\lambda$ is multiplicative, and $g$ is a summatory function of $\lambda$ which is multiplicative. So we just prove on prime powers. If we have prime power with even exponent, then $p, p^2, \dots, p^{e_1}\mid p^{e_1}$ gives $1 + (-1) + 1 + (-1) + \cdots + 1 = 1$. We have $0$ otherwise.
\end{proof}


\subsubsection{Hyperboloids of Two Sheets}
\noindent
A hyperboloid of two sheets has 2 -'s and 1 + in its equation. It is made up of two disconnected, mirror image, surfaces.

[INSERT IMAGE]