\chapter{Background}\label{cha:background}

The following chapter lays the theoretical groundwork for the project. An understanding of linear algebra, numerical optimization, algorithms, and statistics is assumed. \Cref{sec:back_mathprog,sec:back_bnb,ssec:back_mlp} are adapted from the project report \textit{Multi-Layer Perceptrons for Branching in Mixed-Integer Linear Programming} (2020).

\section{Mathematical Programming}\label{sec:back_mathprog}

This section presents the field of \textit{mathematical programming} at the level relevant for understanding the thesis.
In this work, the terms mathematical programming, numerical optimization, and optimization are used interchangeably. The differences between these stem largely from the different communities who use them. The section will first cover the topic of linear programming (\gls{LP}), then the topic of mixed-integer linear programming (\gls{MILP}), and lastly a section on computational complexity.

\subsection{Linear Programming}

In mathematical programming, the general linear problem can be stated as \cite{gasse2019exact}:
\begin{align} \label{eq:lp}
    \underset{\mathbf{x}}{\arg \min }\left\{\mathbf{c}^{\top} \mathbf{x} \; \mid \mathbf{A} \mathbf{x} \leq \mathbf{b},\; \mathbf{x} \in \mathbb{R}_+^{n}\right\},
\end{align}
where $ \mathbf{x} \in \mathbb{R}_+^n$ is the variable vector
with the objective coefficient vector $\mathbf{c} \in \mathbb{R}^n $,
the constraint coefficient matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$
and the constraint right-hand-side vector $\mathbf{b} \in \mathbb{R}^m $.

The size of the problem will be measured by the dimensions of the constraint coefficient matrix $ \mathbf{A} $, where the number of rows and columns corresponds to the number of variables and constraints, respectively.

These problems are convex \cite{wolsey2020integer}, and can be solved by several efficient algorithms. The simplex algorithm can solve problems on this form efficiently, and the same for interior-point methods \cite{nocedal2006numerical}. These algorithms are good average performance but do not have guaranteed polynomial running time in the worst case. Guaranteed polynomial solution algorithms do exist, for instance \textit{Karamkar's algorithm} \cite{karamkar1984new}.

\subsection{Mixed Integer Linear Programming}

Mixed integer linear programming is a superset of linear programming, where one or more of the variables can be restricted to discrete values. The general problem can in this case be stated as \cite{gasse2019exact}:
\begin{align}\label{eq:milp}
    \underset{\mathbf{x}}{\arg \min }\left\{\mathbf{c}^{\top} \mathbf{x} \mid \mathbf{A} \mathbf{x} \leq \mathbf{b}, \; \mathbf{x} \in \mathbb{Z}_+^{p} \times \mathbb{R}_+^{n-p}\right\},
\end{align}
where $ p $ is the number of integer variables, otherwise the variables are the same as \Cref{eq:lp}.

% https://texample.net/tikz/examples/colored-diagram/
\begin{figure}
    \centering
    \begin{tikzpicture}[
        thick,scale=0.5,
        every node/.style={scale=0.2}
        every path/.style = {},
        every node/.append style = {font=\sffamily}
      ]
      \begin{scope}
        \shade[right color=gray, left color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (0,6.5);
        \node[rotate=90, above] at (0,3) {};
        \shade[top color=gray, bottom color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (8.5,0);
        \shade[left color=gray, bottom color=gray, right color=white, opacity=0.5]
          (-0.5,5.5) -- (8.5,3) -- (8.5,6.5) -- (-0.5,6.5) -- cycle;
        \path (-0.5,5.5) -- node[pos=0.23, sloped, above] {}
          (8.5,3);
        \shade[left color=gray, right color=white, opacity=0.5]
          (2.5,6.5) -- (8.5,6.5) -- (8.5,0) -- (5,0) -- cycle;
        \path (5,0) -- node[pos=0.3, sloped, above] {} (2.5,6.5);
        \node[text width=6em, align=center] at (2,2)
          {};
        \draw[->] (-0.5,0) -- (8.5,0) node[below] {x};
        \draw[->] (0,-0.5) -- (0,6.5) node[above] {y};
        \node[rotate=-45, above, text width=9em, align=center] at (7.25,5.25)
          {};
        \path[clip] (-0.5,-0.5) rectangle (8.5,6.5);
        \foreach \i in {0.5,3,...,13} {
          \draw[help lines] (-0.5,\i) -- +(-45:15);
        }
      \end{scope}
      \draw[very thick, ->] (9,3.25) -- node[above, text width=3cm, align=center]
        {} (11.5,3.25);
      \begin{scope}[shift={(13,0)}]
        \shade[right color=gray, left color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (0,6.5);
        \shade[top color=gray, bottom color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (8.5,0);
        \shade[left color=gray, bottom color=gray, right color=white, opacity=0.5]
          (-0.5,5.5) -- (8.5,3) -- (8.5,6.5) -- (-0.5,6.5) -- cycle;
       \shade[left color=gray, right color=gray, opacity=0.5]
         (2.5,6.5) -- (8.5,6.5) -- (8.5,0) -- (5,0) -- cycle;
        \draw[->] (-0.5,0) -- (8.5,0) node[below] {x};
        \draw[->] (0,-0.5) -- (0,6.5) node[above] {y};
        \foreach \i in {0,1,...,6.5} {
          \draw[help lines] (-0.5,\i) -- (8.5,\i);
        }
        \foreach \i in {2,4,...,8.5} {
          \draw[help lines] (\i,6.5) -- (\i,-0.5);
        }
        \foreach \i in {0,1,...,5} {
          \node[draw,cross out,label={left:\i}] at (0,\i) {};
        }
        \foreach \i in {0,1,...,4} {
          \node[draw,cross out] at (2,\i) {};
        }
        \foreach \i in {0,1,...,2} {
                \node[draw,cross out] at (4,\i) {};
        }
        \foreach \i in {0,2,...,6} {
          \node[below] at (\i,0) {\pgfmathparse{int(\i/2)}\pgfmathresult};
        }
      \end{scope}
    \end{tikzpicture}
    \caption{Illustration of an \Gls{LP} with its corresponding \Gls{ILP}, i.e. the \gls{LP} with integrality constraints.}
    \label{fig:milpfig}
\end{figure}

In \Cref{fig:milpfig}, an \gls{LP} problem and the problem with \textit{integrality constraints} is shown. For the \gls{LP} problem, the shaded areas represent the inequality constraints, where the diagonal lines represent the level curves of the objective function. In the \gls{ILP} problem, the crosses represent the feasible solutions. The \gls{LP} is also called a \textit{relaxation} of the original \gls{ILP}, which is fundamental to efficient solving algorithms of \gls{MILP} problems.

A problem that includes integrality constraints cannot be convex \cite{wolsey2020integer}. The non-convexity of the feasible set of the problem constitutes a significant challenge, and it is considered unlikely that polynomial-time solutions exist \cite{papadimitriou1982combinatorial}. \gls{MILP} problems belong to the category of $\mathcal{NP}$-hard problems \cite{papadimitriou1982combinatorial} (this class of problems will be discussed in \Cref{ssec:complexity}).

A subset of \gls{MILP} problems can be integer linear programming (\gls{ILP}), where all variables are restricted to integer values, or binary linear programming (\gls{BLP}), where all variables are restricted to binary values.

\gls{ILP} and \gls{BLP} problems belong to the category of \textit{combinatorial optimization} (\gls{CO}) problems, which has been the main focus of the efforts to solve entirely or partially with machine learning methods \cite{bengio2020machine}.

\subsection{Computational Complexity}\label{ssec:complexity}

A basic understanding of computational complexity is required to justify the nature of the algorithms used to solve \gls{MILP} problems.

Problems can be divided into \textit{classes} by the nature of the algorithms that can solve these problems. Problems for which there exist algorithms that can solve the problem in a time that is \textit{polynomial} of the problem size belong to class $\mathcal{P}$. Problems to which a correct solution can be verified in polynomial time belong to the class $\mathcal{NP}$ (non-deterministic polynomial time) \cite{cormen2009introduction}.

Two other central complexity classes in this context are the $\mathcal{NP}$-complete and $\mathcal{NP}$-hard classes. The $\mathcal{NP}$-complete class contains problems that can be \textit{reduced} to any other problem in the $\mathcal{NP}$-complete class in polynomial time.
The $\mathcal{NP}$-hard class contains problems that are at least as hard as the problems in the $\mathcal{NP}$-complete group but has not been proved to be reducible to a $\mathcal{NP}$-complete problem. An illustration showing this is given in \Cref{fig:np}. The general \gls{MILP} belongs to the $\mathcal{NP}$-hard class, and some \gls{MILP}s have been shown to belong to the $\mathcal{NP}$-complete class \cite{cormen2009introduction}.

\begin{figure}
    \centering
    \includegraphics[width=0.4\linewidth]{img/npc.png}
    \caption{\label{fig:np}Illustration of the most commonly held view of the P, NP, NPC, and NP-hard relationship. Adapted from Cormen et al. (2009) \cite{cormen2009introduction}.}
\end{figure}

As stated, it is considered unlikely that \gls{MILP} problems can be solved in polynomial time. Therefore, it is more fruitful to improve upon the best existing solution algorithms and evaluating the improvements on practical problems. As the improvements attempted by substituting variable selection algorithms do not affect the running time complexity of the \gls{BnB} algorithm, this will not be discussed.

\section{Branch and Bound}\label{sec:back_bnb}

A \textit{relaxation} of a \gls{MILP} problem is achieved by relaxing the integrality constraint, as shown in \Cref{fig:milpfig}. Obtaining the solution to the relaxed problem gives a lower bound on the optimal solution (for a minimization problem). Naturally, any feasible solution to the integrality-constrained problem gives an upper bound to the solution. Furthermore, if the solution to the relaxed problem adheres to the integrality constraints, it is also the solution to the \gls{MILP} problem \cite{wolsey2020integer}.

The most prevalent solution algorithm for \gls{MILP} problems exploits these results, by sequentially dividing the solution space until the optimum with the integrality constraint is found. This is done by branching in a binary tree structure according to \cite{gasse2019exact}:
\begin{align} \label{eq:branch}
    x_{i} \leq\left\lfloor x_{i}^{\star}\right\rfloor \vee x_{i} \geq\left\lceil x_{i}^{\star}\right\rceil, \quad \exists \; i \leq p \mid x_{i}^{\star} \notin \mathbb{Z}
\end{align}
Further creating sub-problems with this binary decomposition. A general algorithm for this process is presented in \Cref{alg:bnb}, and an illustration of this process is shown in \Cref{fig:bandb1}.

\begin{algorithm}[H]
    \SetAlgoLined
    \KwResult{Optimal point and solution value of given problem.}
    Set $L = \{X\}$ and initialize $\hat{x}$\;
    \While{$L \neq \emptyset $}{
        Select a subproblem $S$ from $L$ to explore\;
        \If{a solution $\hat{x}_* \in \{x \in S \;|\; f(x) < f(\hat{x})\}$ can be found}{
            Set $\hat{x} = \hat{x}_*$\;
        }
        \If{$S$ cannot be pruned}{
            Partition $S$ into $\{S_1, S_2 ..., S_r\}$\;
            Insert $\{S_1, S_2 ..., S_r\}$ into $L$\;
        }
        Remove  $S$ from $L$\;
    }
    Return $\hat{x}$\;
    \caption{\label{alg:bnb} A generic branch-and-bound algorithm \cite{morrison2016branch}.}
\end{algorithm}

%https://tex.stackexchange.com/questions/416359/branch-and-bound-tree-in-tikz
\comments{
\begin{figure}
\centering
\begin{forest}
  branch and bound,
  where level=1{
    set branch labels={x\leq}{}{x\geq}{},
  }{
    if level=2{
      set branch labels={}{\geq y}{}{\leq y},
    }{},
  }
  [1055.56:S:950
    [1000:S_1:950:5
    ]
    [1033:S_2:950:6
      [1033:{S_2,1}:950:1]
      [950:{S_2,2}:1033:2]
    ]
  ]
\end{forest}
\caption{\label{fig:bandb1}Illustration of the branch and bound algorithm}
\end{figure}}

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{img/bnb.png}
    \caption{\label{fig:bandb1}Illustration of the branch-and-bound algorithm adapted from a maximization problem in \textit{Integer Programming }(2020) \cite{wolsey2020integer}.}
\end{figure}

For each generated solution, represented by nodes in \Cref{fig:bandb1}, a relaxation of the problem is solved in order to obtain an upper and lower bound on the solution of the sub-problem.
These values are shown on the top and bottom right, respectively.
Generating upper and lower bounds for solutions allows for discarding a large number of solutions \cite{wolsey2020integer}. Branches can be \textit{pruned} (meaning no further partitioning from that branch) if they meet at least one of the following three criteria \cite{wolsey2020integer}:
\newpage
\begin{enumerate}[label=(\roman*)]
    \item Pruning by optimality: $Z^t = \{\max \bm{c}^{\top} \bm{x} : \bm{x} \in S_t\}$ has been solved.
    \item Pruning by bound: $\overline{Z}^t \leq \underline{Z}^t$.
    \item Pruning by infeasiblity: $S_t = \emptyset $.
\end{enumerate}
For \Cref{fig:bandb1}, the graph on the right represents the tree after solving the relaxation, resulting in $S_2$ being pruned by infeasibility, $S_3$ pruned by bound, and $S_4$ pruned by optimality.

The choice of node and variable to branch on to find the optimum in the fewest number of branching processes is central to an efficient implementation of \gls{BnB}. Partitioning the feasible set such that the node with the optimal value is found in the fewest possible branching iterations is the optimal policy.

\subsection{Valid Inequalities}\label{ssec:inequalities}

Another important method used in \gls{BnB} algorithms is the concept of valid inequalities. A valid inequality is an inequality that does not remove feasible solutions of the non-convex solution set but can remove potential solutions to the relaxed problems. A valid inequality can be expressed as:
\begin{equation}\label{eq:cut}
    \bm{\pi}^{\top} \mathbf{x} \leq \pi_0 \quad \forall \; \mathbf{x} \in  \bm{X}
\end{equation}
where $\bm{X}$ is the feasible set as described in \Cref{eq:milp}. These inequalities reduce the size of the feasible set for the relaxations of the problem without removing feasible solutions of the original problem. An illustration of an \gls{ILP} with an added valid inequality is shown in \Cref{fig:cut}. Here the feasible set of the relaxation is reduced in size by the added constraint, while the feasible points of the \gls{ILP} remain feasible after the application of the inequality, as is given in \Cref{eq:cut}.

\begin{figure}
    \centering
    \begin{tikzpicture}[
        thick,scale=0.5,
        every node/.style={scale=0.2}
        every path/.style = {},
        every node/.append style = {font=\sffamily}
      ]
      \begin{scope}
        \shade[right color=gray, left color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (0,6.5);
        \shade[top color=gray, bottom color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (8.5,0);
        \shade[left color=gray, bottom color=gray, right color=white, opacity=0.5]
          (-0.5,5.5) -- (8.5,3) -- (8.5,6.5) -- (-0.5,6.5) -- cycle;
       \shade[left color=gray, right color=gray, opacity=0.5]
         (2.5,6.5) -- (8.5,6.5) -- (8.5,0) -- (5,0) -- cycle;
        \draw[->] (-0.5,0) -- (8.5,0) node[below] {x};
        \draw[->] (0,-0.5) -- (0,6.5) node[above] {y};
        \foreach \i in {0,1,...,6.5} {
          \draw[help lines] (-0.5,\i) -- (8.5,\i);
        }
        \foreach \i in {2,4,...,8.5} {
          \draw[help lines] (\i,6.5) -- (\i,-0.5);
        }
        \foreach \i in {0,1,...,5} {
          \node[draw,cross out] at (0,\i) {};
        }
        \foreach \i in {0,1,...,4} {
          \node[draw,cross out] at (2,\i) {};
        }
        \foreach \i in {0,1,...,2} {
                \node[draw,cross out] at (4,\i) {};
        }
      \end{scope}
      \draw[very thick, ->] (9,3.25) -- node[above, text width=3cm, align=center]
        {} (11.5,3.25);
      \begin{scope}[shift={(13,0)}]
        \shade[right color=gray, left color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (0,6.5);
        \shade[top color=gray, bottom color=white, opacity=0.7]
          (-0.5,-0.5) rectangle (8.5,0);
        \shade[left color=gray, bottom color=gray, right color=white, opacity=0.5]
          (-0.5,5.5) -- (8.5,3) -- (8.5,6.5) -- (-0.5,6.5) -- cycle;
        \shade[left color=gray, right color=gray, opacity=0.5]
          (2.5,6.5) -- (8.5,6.5) -- (8.5,0) -- (5,0) -- cycle;
        \shade[left color=gray, right color=gray, opacity=0.5]
          (0.0,6.5) -- (8.5,6.5) -- (8.5,0) -- (6.0,0) -- cycle;
        \draw[->] (-0.5,0) -- (8.5,0) node[below] {x};
        \draw[->] (0,-0.5) -- (0,6.5) node[above] {y};
        \foreach \i in {0,1,...,6.5} {
          \draw[help lines] (-0.5,\i) -- (8.5,\i);
        }
        \foreach \i in {2,4,...,8.5} {
          \draw[help lines] (\i,6.5) -- (\i,-0.5);
        }
        \foreach \i in {0,1,...,5} {
          \node[draw,cross out] at (0,\i) {};
        }
        \foreach \i in {0,1,...,4} {
          \node[draw,cross out] at (2,\i) {};
        }
        \foreach \i in {0,1,...,2} {
                \node[draw,cross out] at (4,\i) {};
        }
      \end{scope}
    \end{tikzpicture}
    \caption{Figure of an \Gls{ILP} before and after an added valid inequality.}
    \label{fig:cut}
\end{figure}

Algorithms that find these inequalities during the \gls{BnB} algorithm are called \textit{branch-and-cut}. The nomenclature comes from calling the application of these inequalities \textit{cuts} or \textit{cutting planes}.
When an application of inequalities are only employed on the root node (before dividing the solution space in the enumeration), the algorithm is sometimes referred to as \textit{cut-and-branch} rather than \textit{branch-and-cut} \cite{wolsey2020integer}.

\subsection{Primal and Dual Heuristics}

The modern implementations of \gls{BnB} solvers base their efficiency on the implementation of \textit{heuristics} \cite{khalil2020towards}, which are divided into the classes \textit{primal} and \textit{dual}. Heuristic is synonymous with "human-designed rule" in this context.

\textit{Primal heuristics} are methods for finding feasible solutions at a given \gls{BnB} node, where the quality, i.e. the distance to the optimal bound, is the determining factor to whether the feasible solution is useful or not \cite{khalil2020towards}. These heuristics are as costly as they are useful, and modern solvers periodically run different heuristics at different times during the solution process \cite{khalil2020towards}.

\textit{Dual heuristics} are the methods that find the lower bound of the optimization problem. This includes solution of relaxations of the problem as well as the addition of valid inequalities.

The relationship is summarized in this quote from Khalil (2020) \cite{khalil2020towards}:
\begin{quote}
    [...] the
primal side refers to the quest for good feasible solutions, whereas the dual side refers to
the search for a proof of optimality.
\end{quote}

\subsection{Branching Variable Selection Policy}\label{ssec:branchingpolicy}

As mentioned, an important decision in the \gls{BnB} algorithm is the choice of the variable that should be branched on.
There exists many heuristics for solving this, who vary in computational complexity and accuracy.
A good branching algorithm should choose to branch on variables that lead to small solution trees (fewer nodes evaluated) and find these variables in a computationally efficient manner.

All popular variable selection policies depend on scoring the candidate branching variables, expressed as $s_i \in \mathbb{R}^1 \; \forall \: i \in \mathcal{C}$, and then selecting the variable with the most optimal score \cite{achterberg2004branching}. The branching operation generates two child nodes, $Q_i^-$ and $Q_i^+$. The branching candidate comparison is done by comparing the two objective function changes of each candidate, denoted as $\Delta_i^- \coloneqq \bar{c}_{Q_i^-}-\bar{c}_{Q}$ and $\Delta_i^+ \coloneqq \bar{c}_{Q_i^+}-\bar{c}_{Q}$ \cite{achterberg2004branching}. The final score is then typically calculated by a function similar to \cite{achterberg2004branching}:
\begin{equation}
    score(q^- , q^+ ) = (1 - \mu) \cdot \min \{q^- , q^+ \} + \mu \cdot \max \{q^- , q^+ \}\:,\quad \mu \in \left[0, 1\right]
\end{equation}

The current branching policy resulting in the smallest solution trees is known as \textit{strong branching} (\gls{SB}) \cite{applegate1995finding}, and the application of this branching policy at every node is known as \textit{full strong branching} (\gls{FSB}) \cite{achterberg2004branching}. This branching policy is based on determining the best variable to branch on by solving the relaxation for every candidate variable, and is therefore very computationally expensive compared to other methods \cite{achterberg2004branching}.

Another branching policy is \textit{most infeasible branching} (\gls{MIB}), where the variable with the fractional part of the relaxation optimum closest to $0.5$ is selected. This policy, though computationally inexpensive, has proved to be very poor \cite{achterberg2004branching}.

An effective and popular policy is \textit{pseudo-cost branching} (\gls{PC}), which relies on the expected change in objective value based on previous branching on the variable in question
\cite{achterberg2004branching}. In short, the objective gain per unit change in a variable is averaged over all nodes where it has been branched upon. This value is termed the \textit{pseudo-cost} of the variable. As is evident, these values depend on a history of branching, and will therefore be inaccurate for the first decisions \cite{achterberg2004branching}.

The policy known as \textit{reliability pseudo-cost branching} (\gls{RPC}) aims to mitigate the inaccuracy of the \gls{PC} algorithm by combining \gls{SB} and \gls{PC} \cite{anand2017comparative}. In \gls{RPC}, \gls{SB} is employed for variables that are either uninitialized (never branched on before) or have \textit{unreliable} pseudo-costs (pseudo-costs that stem from little data) \cite{achterberg2004branching}. This policy is the standard of the \gls{SCIP} optimization suite \cite{achterberg2009scip}.

In the literature, the branching policy is referred to as a policy, strategy, or rule. In this thesis \textit{policy} is used.

\subsection{Learned Branching Policy}

Recently, attempts have been made to find a branching policy based on statistical learning.

Using machine learning, specifically imitation learning, to find good candidate variables for branching in a less computationally demanding manner was proposed by Elias Khalil \cite{khalil2016learning}. Various methods for learning in branching include \textit{ support vector machine ranking} (\Gls{SVM}) \cite{khalil2016learning}, \textit{graph convolutional neural networks} (\gls{GCNN}) \cite{gasse2019exact} and \textit{feature-wise linear modulation} (\gls{FiLM}) \cite{gupta2020hybrid}.

The fundamental assumption to this approach is that a computationally efficient approximation to the most computationally demanding but most accurate branching policy can be learned. The algorithm will use imitation learning on the branching expert to find a computationally less expensive non-linear function approximation to the expert algorithm's variable scoring. Then, the algorithm branches on the variable with the highest score.
%This can be expressed as:
%\begin{align}
%    f(i) &=  \pi_{SB} (i) + \epsilon \quad \forall \; i \in \mathcal{C}\\
%    f(i) &= s_i\\
%    i^*_f &= \underset{i \in \mathcal{C}}{\mathrm{argmin}} \; \bm{s}_i
%\end{align}
%where $f$ is the learned function, $\mathcal{C}$ is the set of possible branching variables, $\pi_{SB}$ is the Strong Branching strategy and $\epsilon$ is the deviation in the scoring function.

\section{Markov Decision Processes}\label{sec:back_mdp}

This section presents the \textit{Markov decision process} formulation of the variable selection problem.

\subsection{Markov Decision Processes Formulation}\label{ssec:mdp}

Central to the advancement of learned policies in \gls{BnB} is the interpretation of the solution algorithm as an \textit{agent} in a \textit{Markov decision process} (\gls{MDP}) \cite{gasse2019exact}. This interpretation relates the problem to a large collection of literature on the topic \cite{howard1960dynamic}.

In an \gls{MDP}, the agent is at time $t$ in a state $\mathcal{S}_t$, from which it performs an action $\mathcal{A}_t$ that transforms the agent to the state $\mathcal{S}_{t+1}$ and receives the \textit{reward} $\mathcal{R}_{t+1}$ \cite{prouvost2021ecole}. The probability of an agent performing action $a$ in state $s$ is given as $\pi (a | s)$. The probability distribution for the agent to transition to a new state $s'$ is given as $\mathbb{P}(s', r | a, s)$ \cite{prouvost2021ecole}.

A sequence of actions generates a sequence of trajectories $\tau$, and is described as an \textit{episode}. The probability of a trajectory is given in Prouvost et al. (2021) \cite{prouvost2020ecole} as:
\begin{equation}
    \mathbb{P}(\tau) \sim \underbrace{\mathbb{P}(\mathcal{S}_0)}_{\text{initial state}}
\prod_{t=0}^\infty \underbrace{\pi(\mathcal{A}_t | \mathcal{S}_t)}_{\text{next action}}
\underbrace{\mathbb{P}(\mathcal{S}_{t+1}, \mathcal{R}_{t+1} | \mathcal{A}_t, \mathcal{S}_t)}_{\text{next state}}
\end{equation}

These definitions now allow a formulation of the \gls{MDP} control problem, which is the problem of interest in this thesis. The control problem consists of finding the action policy that maximizes the reward. %, and can be stated as:
%\cite{prouvost2020ecole}
%\begin{equation}\label{eq:mdprcontrol}
%    \pi^\star = \underset{\pi}{\operatorname{arg\,max}}
%\lim_{T \to \infty} \mathbb{E}_\tau\left[\sum_{t=0}^{T} %\mathcal{R}(\mathcal{S}_t)\right]
%\end{equation}

\subsection{Partially-observable Markov Decision Processes}

A subset or generalization of an \gls{MDP} is the \textit{partially-observable Markov decision process }(\gls{PO-MDP}) \cite{monahan1982state}. Processes of this class allow for uncertainty of the states as well as additional acquisition of state information \cite{monahan1982state}. The agent will therefore decide actions based on the observation of the state, given as $\mathcal{O}$ \cite{prouvost2021ecole}. All past observations of the observations, rewards and actions are given in the history $\mathcal{H}_t$, given as \cite{prouvost2021ecole}:
\begin{equation}
    \mathcal{H}_t = \{\mathcal{O}(\mathcal{S}_0), \mathcal{R}(S_0), \mathcal{A}_0, ..., \mathcal{O}(\mathcal{S}_{t-1}), \mathcal{R}(S_{t-1}), \mathcal{A}_{t-1}, \mathcal{O}(\mathcal{S}_t)\}
\end{equation}
The generalization from \gls{MDP} to \gls{PO-MDP} concedes the Markovian nature of the trajectories \cite{prouvost2020ecole}.

In addition, the initial state is given by the distribution of the problem instance $I$, giving the relation $\mathbb{P}(\mathcal{S}_0) = \mathbb{P}(I) \mathbb{P}(\mathcal{S}_0 | I)  $ \cite{prouvost2021ecole}.

This results in the final formulation \cite{prouvost2021ecole}:
\begin{equation}
    \mathbb{P}(\tau) \sim \underbrace{ \mathbb{P}(I) \mathbb{P}(\mathcal{S}_0 | I)  }_{\text{initial state}}
\prod_{t=0}^\infty \underbrace{\pi(\mathcal{A}_t | \mathcal{H}_t)}_{\text{next action}}
\underbrace{\mathbb{P}(\mathcal{S}_{t+1}, \mathcal{R}_{t+1} | \mathcal{A}_t, \mathcal{S}_t)}_{\text{next state}}
\end{equation}

An illustration of the Markov decision process control loop from the documentation of \gls{Ecole} is shown in \Cref{fig:mdp}.

\begin{figure}
    \centering
    \includegraphics[width=0.55\linewidth]{img/mdp.png}
    \caption{Illustration of the Markov decision process control loop. Figure from Prouvost et al. (2021) \cite{prouvost2021ecole}.}
    \label{fig:mdp}
\end{figure}

\subsection{Branch \& Bound as a PO-MDP}\label{ssec:pomdp}

Interpreted in the language of \gls{MDP}s, the \gls{BnB} algorithm is the \textit{environment} and a concrete \gls{MILP} problem instance is an \textit{episode} in this environment. The \textit{agent} is the brancher, where in this thesis the variable selection policy is the component of interest, ignoring the node selection policy. The state of the solver consists of the \gls{BnB} tree at that instance, as well as the observations at each node (the \textit{history} of the \gls{PO-MDP}).

This formulation is the basis for the \textit{Ecole} framework, which is discussed in \Cref{ssec:ecole}
The \gls{PO-MDP} formulation allows for the agent in the \gls{BnB} environment to be learned through reinforcement learning, discussed in \Cref{ssec:back_rl}.

\subsection{Branch \& Bound Observation}\label{ssec:obs}

A prerequisite for learning in \gls{BnB} is the observation of the state of the episode, i.e. the state of the solver of an instance at a specific node in the solution tree.

%Little attention towards these features are found in the major publications in this field (\cite{gasse2019exact,gupta2020hybrid}), however the observation is the foundation of the learning process.
%\textit{Learning to Branch} by Khalil et al. (2016) \cite{khalil2016learning} contains multiple additional features, however these will not be discussed in this thesis.

The features of a \gls{BnB} node are divided into three classes: variable features, constraint features and edge features.

\textbf{Variable Features}

For a candidate branching variable, relevant features include the type of the variable (binary, integer, etc.), whether the variable has a defined lower and/or upper bound, and whether the solution is at at either of these bounds.
If not, the variable has a fractionality that represents the solution of the relaxed problem.
At the solution node, the incumbent has a value that can be compared to incumbents at other nodes, as well as the relative impact of the variable on the objective value in the incumbent.
The variable also has a state with respect to the solution of the relaxation with a simplex solution algorithm --- if the variable is a basic or non-basic variable or other information relating to this solution.
Presented in Khalil et al. (2016) \cite{khalil2016learning} are also a number of other features that will not be utilized in this work.

\textbf{Constraint Features}

The cosine similarity represents a coefficient of the angle between the variable and the constraint.
The bias of the constraint is also included.
An additional feature is whether the variable is at the constraint in the relaxation
Each constraint also has a value from the solution of the dual problem.

\textbf{Edge Features}

The edge features consist of the constraint coefficient, meaning the coefficient that is multiplied with the candidate variable. This will also give the relations between constraints and variables.

\subsection{Bipartite Graph Representation}

The application of \gls{GCNN}s on \gls{MILP} and sub-\gls{MILP} problems rely on the bipartite representation of constraints and variables as presented in Gasse et al. (2019) \cite{gasse2019exact}.
This concept will be introduced with an example \gls{MILP} given as:
\begin{align}\label{eq:bipex}
    \min \quad &\texttt{v}_1 + \texttt{v}_2 + \texttt{v}_3\\
    s.t. \quad &\texttt{v}_1 + \texttt{v}_2 - \texttt{v}_3 \geq 1 \qquad (c_1)\nonumber\\
    &\texttt{v}_3 \geq \frac{1}{2}\qquad\qquad\quad\;\,\; (c_2)\nonumber\\
    &\mathbf{v} \in \mathbb{B}^3 \nonumber
\end{align}

\begin{figure}
    \centering
    \includegraphics[width=0.40\linewidth]{img/bipartite_zoom.png}
    \caption{Example of a bipartite constraint-variable graph.}
    \label{fig:bipartite_cv}
\end{figure}

For \Cref{eq:bipex}, the corresponding constraint-variable graph representation can be illustrated as in
\Cref{fig:bipartite_cv}.

Constraints and variables are the numbered nodes of the graphs, while the edges represent the relation between the nodes.

\section{Machine Learning Models}\label{sec:back_models}

The \gls{ML}-models used in the thesis are presented in this section. First the multi-layer perceptron and graph convolutional neural network models, then the concepts of \textit{ablation studies} and \textit{reinforcement learning}.

\subsection{Multi-layer Perceptrons}\label{ssec:back_mlp}

Multi-layer perceptrons (\gls{MLP}s), more commonly known as deep feed-forward neural networks, are recommended by Gupta et al. (2020) \cite{gupta2020hybrid} as a less computationally expensive alternative to the approaches by Khalil et al. (2016) \cite{khalil2016learning} and Gasse et al. (2019) \cite{gasse2019exact}.

\gls{MLP}s are networks that generate a nonlinear function $y = f(\mathbf{x}; \bm{\theta})$, where $x$ is the input, $y$ is the output, and $\bm{\theta}$ represents the parameters of the function. The parameters are learned during repeated optimization, and will under ideal circumstances converge to approach the optimal function $y = f^*(\mathbf{x})$. The function is realized as a series of compositions of functions. The composed functions are represented as an acyclical, directed graph \cite{nielsen2018neural}, and can be expressed as:
\begin{align}
    y = f_L \circ f_{L-1} \circ \ldots \circ f_{1} \circ f_{0} (\mathbf{x})
\end{align}
The functions are denoted as \textit{layers} of the perceptron, and are implemented as affine functions of every input parameter at every node, $\mathbf{z}_l = \mathbf{x}_{l-1}^T \mathbf{w}_l + \mathbf{b}_l$. Applying non-linear function, known as an \textit{activation function}, allows the \gls{MLP} to represent arbitrary nonlinear functions \cite{goodfellow2016deep}. This is expressed as $\mathbf{x}_l = \mathbf{a}(\mathbf{z}_l)$.

The computation of the output of the function given its input is known as a \textit{forward pass} through the network. The required computations for a single input vector into a network with $ n $ hidden layers will include $ n + 1 $ matrix multiplications and $ n + 1 $ applications of the non-linear activation function, given that there is an activation function on the output.

%Depending on the training configuration, the problem can be interpreted as a classification problem or regression problem (or a ranking problem, as in \cite{khalil2016learning}). In the following experiments, the former approach is selected, as has become popular after Gasse et al. (2019) \cite{gasse2019exact}.

\subsection{Graph Convolutional Neural Networks }

Graph convolutional neural network (\gls{GCNN}) is a term for neural networks that have input data represented in a graph-structure that is processed by a convolution operation \cite{kipf2016semisupervised}. In this thesis, the terms \gls{GCNN} and \gls{GNN} will be used interchangeably.

The fundamental property of the graph convolution is its ability to create representations of irregular data without altering the structure of the data. This means that, for instance, nodes that share vertices can pass information to each other, so the feature representation of a node can utilize the features of neighboring nodes.
An illustration of this using the bipartite graph from \Cref{fig:bipartite_cv} is given in \Cref{fig:conv_ex}. The features of the nodes are transformed while maintaining the structure of the graph.

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{img/conv_example.png}
    \caption{Example of a graph convolution on a bipartite constraint-variable graph.}
    \label{fig:conv_ex}
\end{figure}

A graph convolution can be expressed as a matrix/tensor multiplication followed by a nonlinear activation function, as in \Cref{sec:back_mdp}. This will be explained in the case of a undirected graph for the sake of simplicity. A prerequisite for this is the representation of the graph with the adjacency matrix, denoted as $\Tilde{\mathbf{A}}$. The adjacency matrix is a square $|\mathbf{V}|\times|\mathbf{V}|$ matrix containing 0 or 1 depending on whether the pair of vertices are connected or not \cite{kipf2016semisupervised}. In addition the adjacency matrix, the degree matrix $\Tilde{\mathbf{D}} = \sum_j \Tilde{\mathbf{A}}_{i j}$ is necessary in order to normalize the operation \cite{kipf2016semisupervised}.

With the given definitions, the graph convolution operation can be expressed by the propagation rule \cite{kipf2016semisupervised}:
\begin{equation}
    \mathbf{H}^{(l+1)} = a \left( \Tilde{\mathbf{D}}^{-\frac{1}{2}} \Tilde{\mathbf{A}} \Tilde{\mathbf{D}}^{-\frac{1}{2}} \mathbf{H}^{(l)}\mathbf{W}^{(l)}\right)
\end{equation}
where $\mathbf{H}^{(l)}$ is the matrix of activations in layer $l$, with the first layer $\mathbf{H}^{0}=\mathbf{X}$.  $\mathbf{W}^{(l)}$ is a layer of learned weights. $a( \cdot) $ is a nonlinear activation function.
This operation is inspired by the first-order approximations to spectral filters on graphs \cite{kipf2016semisupervised}.

Models that leverage the graph nature of combinatorial optimization problems have been shown to have satisfactory performance, see e.g. Dai et al. (2018) \cite{dai2018learning}.
\gls{GCNN}s are proposed by Gasse et al. (2019) \cite{gasse2019exact} as an alternative to the feature-rich approaches by Khalil et al. (2016) \cite{khalil2016learning}.
The application of \gls{GCNN}s on \gls{BnB} algorithms rely on the bipartite constraint-variable representation at each node of the \gls{BnB} solution tree.

The term \textit{embeddings} will be used in this thesis for continuous-variable representations derived from the input features, as it is used in Gasse et al. (2019) \cite{gasse2019exact}.

The state of the \gls{BnB} graph at a node can be represented as $s_t = (\mathcal{G}, \mathbf{C}, \mathbf{E}, \mathbf{V})$, where $\mathcal{G}$ represents the bipartite \gls{BnB} solution graph at that time instance, $\mathbf{C}$ represents the constraints, $\mathbf{E}$ represents the \textit{edges} (connections) between the variables and constraints, and $\mathbf{V}$ represents candidate variables.

Gasse et al. (2019) \cite{gasse2019exact} presents three motivating points for why graph convolutions would be a good architecture for learning to branch:
\begin{enumerate}[label=(\roman*)]
    \item They are well-defined no matter the input graph size.
    \item Their computational complexity is directly related
to the density of the graph, which makes it an ideal choice for processing typically sparse \gls{MILP}
problems.
    \item They are permutation-invariant, that is they will always produce the same output no
matter the order in which the nodes are presented.
\end{enumerate}

\subsection{Ablation Studies}

The concept of ablation studies in machine learning, as presented in
Meyes et al. (2019) \cite{meyes2019ablation} is presented in this section.
Ablation studies hail from the field of neuroscience, in which a complex system, e.g. the brain, is examined after removing different sections. The function of the removed sections can then be inferred by the change in the observed reaction to external stimuli \cite{meyes2019ablation}.

In the context of \gls{ML}, ablation studies are a formalization of observing changes in performance after the removal of components of artificial neural networks \cite{meyes2019ablation}.
The concept, or at least the formalization, is not yet considered a standard method in \gls{ML} research \cite{sheikholeslami2019ablation}.
In this thesis, the concept of an ablation study will be interpreted more broadly than in Meyes et al. (2019) \cite{meyes2019ablation}, as the networks in this thesis are retrained after each section is removed. This form of ablation study is coined as \textit{model ablation} in Sheikholeslami (2019) \cite{sheikholeslami2019ablation}.

\subsection{Reinforcement Learning}\label{ssec:back_rl}

The subset of machine learning described as \textit{reinforcement learning} (\gls{RL}) is highly relevant in the context of \gls{ML} in \gls{CO}. No results will be discussed in this thesis, however, a background in the topic is necessary to understand both related work and the long-term goals of the field.

\gls{RL} encompasses the problem of an \textit{agent} learning a \textit{policy} for behaving in an \textit{environment} so as to achieve a global objective. Any sequential decision-making problem with a measure of optimality that relies on past experience can be formulated as a \gls{RL} problem \cite{francois2018introduction}. The approach has seen success in a number of fields in the past years with the integration of deep learning models, often termed deep \gls{RL} \cite{francois2018introduction}. Most notable of the advancements might be AlphaZero, Google's successful chess-AI \cite{silver2017mastering}. \gls{RL} has the important property of being independent of data, meaning a number of core \gls{ML} challenges (quantity, quality, and bias of data) are rendered irrelevant \cite{goodfellow2016deep}.

%\gls{RL} is particularly interesting in \gls{BnB} because of the \gls{MDP} nature of the algorithm, and the assumption that the handcrafted heuristics and sub-algorithms prevalent in modern solvers are either inefficient or inaccurate compared to the theoretical capabilities of, i.e. neural networks.

Many attempts have been made at implementing \gls{RL} in \gls{CO}, see for example Etheve et al. (2020) \cite{etheve2020reinforcement} or Tang et al. (2020) \cite{tang2020reinforcement}. Approaches for learning variable selection, such as reported in Scavuzzo (2020) \cite{scavuzzo2020learning}, rely on efficient and accurate pre-trained models based on imitation learning, like the models presented in this thesis. More knowledge is likely needed for the pure \gls{RL} approach to take over the mantle. Recently,
Cappart et al. (2021) \cite{cappart2021combinatorial} also concluded that useful \gls{RL} policies are not mature yet.


% !TeX root = ../Thermostats.tex
\subsection{Andersen}

The Andersen thermostat was first introduced by Hans Andersen in 1980 \cite{Andersen1}.
It yields the correct velocity distribution by connecting the system to an external heat bath with the corresponding temperature T.
At every time step each particle has a probability $P=\nu\cdot\Delta t$ --- with $\nu$ the stochastic collision frequency --- to undergo a collision with the heat bath and thus change its momentum. These collisions are instantaneous and affect only the particle involved.
The new momentum of the particle after a collision is drawn at random from a Boltzmann distribution of the given temperature.
Because only a small, random number of particles are affected at each time step, most of the particles move freely according to the Hamiltonian. But the encounters with the heat bath are enough to relax the system to the given temperature and to let the kinetic energy fluctuate around its equilibrium according to the canonical ensemble.

The Andersen thermostat should only be applied to time-independent properties, dynamic problems should not be thermostated by an Andersen algorithm \cite{Andersen2}.

\subsection{Lowe--Andersen}
The Lowe-Andersen thermostat is a Galilean invariant and momentum conserving analogue of the Andersen thermostat \cite{LoweAndersen}.
The collisions with the heat bath now affect a pair of particles, and only the relative velocity along the centre of mass changes. The relative velocity $v_{ij}$ from the Maxwell-Boltzmann distribution is calculated with the equation:
\begin{equation}
v_{ij}'=\zeta\sqrt{2k_BT},
\end{equation}
with $\zeta$ being a random number drawn from a normal distribution with unit variance and zero mean.
The relative velocity of the particles after the collision is then adjusted according to
\begin{align}
2\Delta_{ij}&=r_{ij}\left[(v_{ij}'-v_{ij})\bullet r_{ij}\right]\\
v_i'&=v_i+\Delta_{ij}\\
v_j'&=v_j-\Delta{ij}.\\
\end{align}


\subsection{Chinese anti-leverage}

Figure~\ref{fig:rhos} summarises the results by plotting the posterior densities of $\rho$ obtained from all the model fits.
Even though not all companies behave the same way, there is a consistent picture in general about the differences between the two countries.
The chart shows that, based on the chosen sample, $\rho$ is really estimated to be larger in China than in Germany, except for the periods containing the crisis, when it was similar.

The claims about China behaving differently are underpinned by Figure~\ref{fig:negative-rhos}, which shows the number of companies with a significant leverage or anti-leverage effect.
We define significance at the 5\% level in this master thesis, i.e.\ significant leverage means that the 95\% quantile of $\rho$'s posterior is negative, while significant anti-leverage means that the 5\% quantile of $\rho$'s posterior is positive.
Interestingly, Germany had the least number of companies with significant leverage effect in the period that just preceded the crisis, which was followed by a steep increase into the crisis.
Apart from the periods containing the boom phase of the Subprime Crisis or touching the European Debt Crisis, the figure for Germany is quite stable around 6 to 7.
None of the German stocks had a significantly positive $\rho$.

The picture is much less consistent about China.
Ignoring periods with only one significant value, the leverage effect is only present under the crisis of 2007, with some delay compared to Germany, and only in 3 stocks at most.
Similarly, anti-leverage is hardly observed, only before the crisis.
Still, the companies show some consistency, since there is no period with both leverage and anti-leverage effect being present.

\comment{
Note that the exact distribution or point estimates of $\rho$ are not of high interest.
On the one hand, it is not the correlation of returns with variance changes but with log variance changes, which makes it more difficult to interpret.
On the other hand, comparison with correlation values obtained from other models in the literature is also not trivial, so across models only the sign and highly extreme values are interesting.
}

\subsection{Time-varying leverage}

Figures~\ref{fig:rhos} and~\ref{fig:negative-rhos} suggest that leverage is not constant, even its sign varies.
The charts show that correlation shifts to the negative direction throughout the crisis, which is consistent with recent literature~\citep{Christensen2015}.
It seems to hold especially for the Chinese stocks since the German companies are more stably on the negative side throughout the whole time.

Figure~\ref{fig:company-rhos} shows how the posterior of $\rho$ changes for each company with the rolling window of periods.
The German stocks are on the bottom half of the chart, they have a significant leverage effect in general since the crisis.
They show a quite consistent picture, the 95\% quantile is negative for the vast majority of estimates.
BMW and SAP show a steadily strengthening leverage effect, while it stagnates or weakens for the others.
If we accept that the leverage effect is stronger in crises~\citep{Christensen2015}, then the increasing $\rho$ in 8 out of the 10 German companies signals the end of the impact of recent crises.

The posterior $\rho$ developed far more hectically for the Chinese companies.
However, they all had the negative extremum of the posteriors around the first half of 2008, at the strongest impact of the crisis.
However, the overall picture is unstable throughout the whole examined time, which questions the reliability of the results from the perspective of~\citet{Christensen2015}.
There might be far more and substantially different factors in play that affect Chinese stocks' leverage effect.

\subsection{Volatility estimations}

The posterior of $\phi$ and of the log variance are presented in Figure~\ref{fig:persistence} on a chosen subset of periods and companies that represent the whole dataset.
The prior of $\phi$ is taken to be informative, it reflects the empirical fact of highly autocorrelating volatility.
The posterior of $\phi$ mainly depends on its prior and the estimate for $\bm h$.
Hence, if the posterior is significantly different from the prior, that means that there is valuable information in $\bm h$ about $\phi$.

An immediately noticeable phenomenon in Figure~\ref{fig:persistence} is that the posterior of $\phi$ varies more for the German firms.
More precisely, throughout the crisis and in its short aftermath, the volatility of the German stocks was substantially more persistent than in the other periods.
At the same time, changes in the persistence of the Chinese companies can not be explained by the crisis periods, and the estimates are more stable and closer to the prior, except for the highly persistent 2010-2012 period of 600016 CH Equity.
One can also spot the differences on the log variance timeline: the time series are close to a constant value plus noise in China, containing little information about persistence.
In Germany, on the other hand, clusters of increasing and decreasing trends can be observed around the crisis.

\begin{figure}[p]
	\vspace*{-3.2cm}
	\centering
	\includegraphics[width=\linewidth]{../calculations/rhos.pdf}
	\caption[Visually summarising the results]{Visually summarising the results. Each blue box corresponds to a country and a period and contains 10 rows. One row corresponds to one stock. Each row is a heatmap of $\rho$'s posterior density estimated for that company and period. The red line is constant 0. The stocks' rows are ordered according to their posterior mean in the first period.}
	\label{fig:rhos}
\end{figure}

\begin{figure}[p]
	\vspace*{-3.2cm}
	\centering
	\includegraphics[width=\linewidth]{../calculations/negative-rhos.pdf}
	\includegraphics[width=\linewidth]{../calculations/positive-rhos.pdf}
	\caption[Significant leverage effect]{Top: number of companies with significant leverage effect per country and period, on a 5\% level. Bottom: number of companies with significant anti-leverage effect per country and period, on a 5\% level. Both: Only every second period is shown for readability.}
	\label{fig:negative-rhos}
\end{figure}

\begin{figure}[p]
	\vspace*{-3.2cm}
	\centering
	\includegraphics[width=\linewidth]{../calculations/rho-timeline.pdf}
	\caption[Timeline of posterior $\rho$]{Posterior distributions of $\rho$ in time. The prior distribution is the uniform distribution on $[-1,1]$ in all cases.}
	\label{fig:company-rhos}
\end{figure}

\begin{figure}[p]
	\vspace*{-3.2cm}
	\centering
	\includegraphics[width=\linewidth]{../calculations/phi-timeline.pdf}
	\includegraphics[width=\linewidth]{../calculations/volatility.pdf}
	\caption[Timeline of persistence and volatility]{Top: posterior persistence of the log variance for a subset of companies and periods. Bottom: posterior log variance for a subset of the companies. The time series are glued together from the independent volatility estimations from several periods.}
	\label{fig:persistence}
\end{figure}


\documentclass{article}
\usepackage{fullpage}
\usepackage{amssymb}
\begin{document}

\newcommand{\PiL}{\textsc{\textbf{Pi}}}
\newcommand{\PiCalc}{\PiL${}^\equiv$}
\newcommand{\PiCat}{\ensuremath{\Pi\mathbb{C}}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Goal}

Our first major milestone is (i) to develop \emph{a sound and complete
  calculus of permutations over finite sets}, (ii) to define, in that
setting, a variant of \emph{univalence} that has a computational
content, and to \emph{prove} this univalence property. In the longer
term, we will want to generalize this framework to sets with negative
and fractional cardinalities which would effectively give us (at
least) higher-order functions. One can in fact go further and consider
sets with more general cardinalities: imaginary numbers, irrational
numbers, all the way to algebraic numbers.

To reach the first major milestone, our detailed steps are:
\begin{itemize}

\item to develop a \emph{syntactic notion of permutations over finite
    sets}. We achieve this by defining a typed programming language
  \PiL\ whose types denote finite sets and whose expressions denote
  permutations over finite sets. This language has been completely
  developed: its syntax, type system, and operational semantics are
  well-understood;

\item to develop a \emph{calculus \PiCalc\ for reasoning about \PiL\
    permutations}. We eventually want to prove that this calculus is
  sound and complete with respect to a semantic notion of permutation
  equivalence;

\item to develop a \emph{semantic notion of equivalence of
    permutations over finite sets}.  To make the connections to HoTT
  more evident, we formalize this equivalence using a category
  (groupoid actually) of finite sets and permutations \PiCat\ which we
  establish is \emph{rig category}, i.e., it has two symmetric
  monoidal structures with the multiplicative one distributing over
  the additive one. The objects of this category are discrete
  groupoids representing finite sets; the morphisms of \PiCat\ are
  permutations over the finite sets; the complete definition of the
  category also needs an equivalence relation that specifies when two
  morphisms should be considered equal. This equivalence relation is
  our notion of semantic equivalence of permutations over finite
  sets. The advantage of using the categorical setting to define the
  semantic notion of equivalence is that the coherence laws for rig
  categories provide us with an axiomatization of semantic equivalence
  of permutations that is rich enough to reason about sequential
  compositions of permutations (given by the plain categorical
  structure), parallel compositions of permutations (given by the
  additive monoidal structure), and tensor compositions of
  permutations (given by the multiplicative monoidal structure) while
  taking care of all the induced equivalences that result from the
  interactions among these structures.

\item to derive the complete definition of \PiCalc\ by reifying each
  of the primitives used in defining semantic equivalence as syntactic
  objects; a proof of soundness and completeness is then immediate as
  the syntactic and semantic structures are isomorphic by construction.

\item to define a variant of univalence in the current setting we
  reason as follows. In the HoTT setting, which is not limited to
  finite sets and permutations, we have a larger category of arbitrary
  sets and with equivalences as morphisms. Univalence in that context
  postulates that each equivalence between sets induces a path between
  these sets. Translated to our setting, univalence postulates that
  each semantic equivalence between sets induces a syntactic
  equivalence between these sets. In the richer setting of HoTT,
  univalence is postulated because there is no obvious way to reason
  about extensional equivalence of the functions used to define
  semantic equivalences. In our setting, these semantic equivalences
  are defined using permutations which are amenable to equational
  reasoning via the coherence laws. Therefore to make univalence
  constructive, all we need is a computational mechanism to calculate
  a syntactic equivalence from a semantic one. This mechanism is
  embedded in the proof of completeness above.

\item Our complete development is formalized in Agda.
\end{itemize}

Which still leaves open one big question: is our main representation
of permutations _fin~_ or CPerm ?  Or do we need both, as one is
'syntactic' (through the use of vectors) while the other is more
abstract (as it uses functions)?

I am quite sure that all the work needed to bring FinEquivCat all the
way to RigCategory is feasible, and not too hard [but will be
enlightening].

To do the same with CPermCat (and SkFinSetCategory) will be a fair bit
harder.  So it is very important for us to decide what the exact path
we want to take.  [Which is indeed what your outline is about!]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Basic Utilities}

\paragraph*{LeqLemmas.} A few lemmas about natural numbers.

\paragraph*{FinNatLemmas.} A few lemmas about $\texttt{Fin}~n$ which
are the numbers used to index into vectors.

\paragraph*{SubstLemmas.} A few lemmas about compositions of
propositional equalities..

\paragraph*{VectorLemmas.} A few lemmas about vectors, lookups,
mapping functions over vectors, etc.

\paragraph*{FiniteFunctions.} Proves extensionality for finite functions.

\paragraph*{Proofs.} Collects all the above and re-exports them along
with a couple of other general utilities for managing Agda proofs.

\paragraph*{DivModUtils.} External library for reasoning about
uniqueness of \texttt{divMod}. This is used extensively to mediate
between $\texttt{Fin}~(m*n)$ and
$\texttt{Fin}~m \times \texttt{Fin}~n$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Structures}

\paragraph*{SetoidUtils.} Any discrete type $A$ can be viewed as a
setoid with propositional equality $\equiv$ as the equivalence
relation.

\paragraph*{Groupoid.} A definition of 1-groupoids and operations on
them.

\paragraph*{Categories.Everything.} Basic and some advanced category theory.

\paragraph*{SymmetricMonoidalCategory.} A definition of symmetric
monoidal categories.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Equivalences}

\paragraph*{Equiv.} Defines extensional equivalence of functions
$\sim$ and shows that it is an equivalence relation. Defines
equivalence between sets $\simeq$ using two functions that go back and
forth and whose compositions are extensionally equivalent to the
identity, and shows that this equivalence is indeed an equivalence
relation. Finally shows that equivalences are injective and form a
congruence that respects $\uplus$ and $\times$.

\paragraph*{TypeEquiv.} Establishes that the Agda types $\bot$,
$\top$, $\uplus$, and $\times$ form a commutative semiring using
$\simeq$ as the underlying equivalence relation.

\paragraph*{FinEquiv.} Establishes that $\texttt{Fin}~n$ also forms a
commutative semiring with $\simeq$ as the underlying equivalence
relation. In particular, we have:
\[\begin{array}{rcll}
\texttt{Fin}~0 &\simeq& \bot \\
\texttt{Fin}~1 &\simeq& \top \\
\texttt{Fin}~(m+n) &\simeq& \texttt{Fin}~m \uplus \texttt{Fin}~n \\
\texttt{Fin}~(m*n) &\simeq& \texttt{Fin}~m \times \texttt{Fin}~n
\end{array}\]
and then we have all the commutative semiring axioms, e.g.,
$\texttt{Fin}~(0+m) \simeq \texttt{Fin}~m$. The actual proof says that
0, 1, $+$, and $*$ for a commutative semiring structure under the
equivalence that equates $m$ and $n$ if
$\texttt{Fin}~m \simeq \texttt{Fin}~n$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Bimonoidal Categories}

\paragraph*{SkFinSetCategory.} Defines the skeletal category of finite
sets and all functions between them. There is one object for each
natural number $n$ (including $n=0$), and a morphism from $m$ to $n$
is an $m$-tuple $(f_0,\ldots,f_{m−1})$ of numbers satisfying
$0 \leq f_i < n$. This structure will be a building block for
permutations (defined in \texttt{ConcretePermutation}).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{To do}

PiLevel0
ConcretePermutation
VecOps
PiPerm

Enumeration ??
SEequivSCPermEquiv ??

\paragraph*{Cauchy representation.} A permutation on $n$ elements is
represented by a vector \texttt{v : Vec (Fin n) n}. If the $n$
elements are indexed by positions, the element at position $i$ is
mapped to position \texttt{v !! i} by the permutation. There is always
a trivial permutation called \texttt{1C} that maps each position to
itself. The Cauchy representation does not enforce that the vector
entries are disjoint. This is enforced by the definition of ``concrete
permutations'' below.

\paragraph*{Concrete permutation.} A concrete permutation consists of
two Cauchy vectors and two proofs that their compositions is the
identity permutation \texttt{1C}. Concrete permutations are an
equivalence relation. Concrete permutations actually have more
structure: a sum, a unit for the sum, etc. We can also build setoids
whose carriers are concrete permutations under the standard $\equiv$
propositional equality.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}


% $Header: /cvsroot/latex-beamer/latex-beamer/solutions/conference-talks/conference-ornate-20min.en.tex,v 1.6 2004/10/07 20:53:08 tantau Exp $

\documentclass{beamer}

\mode<presentation>
{
  \usetheme{Hawke}
  % or ...

  \setbeamercovered{transparent}
  % or whatever (possibly just delete it)
}

\usepackage[english]{babel}
% or whatever

\usepackage[latin1]{inputenc}
% or whatever

\usepackage{times}
\usepackage[T1]{fontenc}

\usepackage{multimedia}

%%%%%%
% My Commands
%%%%%%

\newcommand{\bb}{{\boldsymbol{b}}}
\newcommand{\bx}{{\boldsymbol{x}}}
\newcommand{\by}{{\boldsymbol{y}}}
\newcommand{\bfm}[1]{{\boldsymbol{#1}}}

%%%%

\title[Lecture 17] % (optional, use only with long paper titles)
{Lecture 17 - Predictor Corrector Methods}

\author[I. Hawke] % (optional, use only with lots of authors)
{I.~Hawke}

\institute[University of Southampton] % (optional, but mostly needed)
{
%  \inst{1}%
  School of Mathematics, \\
  University of Southampton, UK
}

\date[Semester 1] % (optional, should be abbreviation of conference name)
{MATH3018/6141, Semester 1}

\subject{Numerical methods}
% This is only inserted into the PDF information catalog. Can be left
% out.

\pgfdeclareimage[height=0.5cm]{university-logo}{mathematics_7469}
\logo{\pgfuseimage{university-logo}}

\AtBeginSection[]
{
  \begin{frame}<beamer>
    \frametitle{Outline}
    \tableofcontents[currentsection]
  \end{frame}
}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Predictor-Corrector methods}

\subsection{Predictor-Corrector methods}

\begin{frame}
  \frametitle{Geometrical interpretation of Euler's method}

  Considering IVPs in the form
  \begin{equation*}
    \by'(x) = \bfm{f}(x, \by(x)).
  \end{equation*}

  The simple (first order accurate, explicit) Euler method is
  \begin{equation*}
    \by_{n+1} = \by_n + h \bfm{f}(x_n, \by_n).
  \end{equation*} \pause

  Euler's method is not sufficiently accurate for practical use. Euler predictor-corector is second order. \pause

  Use \emph{multiple} approximations to the slope for a more accurate result.

\end{frame}

\section{Runge-Kutta methods}

\subsection{Runge-Kutta methods}

\begin{frame}
  \frametitle{Runge-Kutta methods}

  In a Runge-Kutta method, Taylor's theorem is used from the start to
  ensure the desired accuracy. \pause

  \vspace{1ex}

  Consider a single step from known data $y_n(x_n)$. Compute one
  estimate ($k_1$) for $f(x_n, y_n)$ using the known data. \pause Then
  compute $y^{(1)}$ at $x_n + \alpha h$ using $y_n + \beta k_1$. \pause
  From this compute another estimate ($k_2$) for $f(x, y)$ at
  $x_n + \alpha h$. Compute $y^{(2)}$ etc; combine as $y_{n+1} = a k_1 + b k_2 + \dots$. \pause

  \vspace{1ex}

  Such methods are called \emph{multistage}:
  \begin{itemize}
  \item a number of estimates of $f$ are combined to
    improve accuracy;
  \item only the previous value $y_n$ is required to start the
    algorithm.
  \end{itemize}

  \vspace{1ex}

  To derive $a, b, \dots, \alpha, \beta, \dots$ expand the algorithm and match to
  exact solution using Taylor's theorem, chain rule and IVP.

\end{frame}

\begin{frame}
  \frametitle{Example: RK2}

  \begin{overlayarea}{\textwidth}{0.8\textheight}
    \only<1|handout:1>
    {
      For the second order method we have
      \begin{align*}
        y_{n+1} & = y_n + a k_1 + b k_2, \\
        k_1 & = h f(x_n, y_n), \\
        k_2 & = h f(x_n + \alpha h, y_n + \beta k_1).
      \end{align*}
      We have four free parameters $a, b, \alpha, \beta$ to fix.
    }
    \only<2|handout:1>
    {
      Taylor expand the definition of $y_{n+1} = y(x_n + h)$:
      \begin{align*}
        y_{n+1} & = y_n + h y'_n + \tfrac{h^2}{2} y''_n + \dots \\
        & = y_n + h f_n +  \tfrac{h^2}{2} \left( f_n \right)' + \dots \\
        \intertext{using the original IVP, then use the chain rule:}
        & = y_n + h f_n +  \tfrac{h^2}{2} \left( \partial_x f_n +
          (\partial_y f)_n f_n \right) + \dots .
      \end{align*}
    }
    \only<3-|handout:2>
    {
      Algorithm:
      \begin{align*}
        y_{n+1} & = y_n + a k_1 + b k_2 \\
        & = y_n + h f_n +  \tfrac{h^2}{2} \left( \partial_x f_n +
          (\partial_y f)_n f_n \right) + \dots
      \end{align*}

      Compare against the Taylor expansion of the second order
      method
      \begin{align*}
        y_{n+1} & = y_n + a h f_n + b h f(x_n + \alpha h, y_n + \beta h
        f_n) \\
        & = y_n + h (a + b) f_n + h^2 \left[ (\partial_x f)_n \alpha b +
          (\partial_y f)_n f_n \beta b \right].
      \end{align*}
    }
    \only<4|handout:2>
    {
      Matching coefficients
      \begin{equation*}
        \left\{
          \begin{aligned}
            a + b & = 1 \\
            \alpha b & = 1 / 2 \\
            \beta b & = 1 / 2
          \end{aligned}
          \right. .
      \end{equation*}
    }
  \end{overlayarea}

\end{frame}

\begin{frame}
  \frametitle{Example: RK2 (II)}

  The RK2 method
  \begin{align*}
    y_{n+1} & = y_n + a k_1 + b k_2, \\
    k_1 & = h f(x_n, y_n), \\
    k_2 & = h f(x_n + \alpha h, y_n + \beta k_1)
  \end{align*}
  with coefficients
  \begin{equation*}
    \left\{
      \begin{aligned}
        a + b & = 1 \\
        \alpha b & = 1 / 2 \\
        \beta b & = 1 / 2
      \end{aligned}
    \right.
  \end{equation*}
  is not completely specified; there is essentially one free
  parameter. \pause

  \vspace{1ex}

  Not all choices are stable. The classic choice is $a = 1/2 = b$,
  $\alpha = 1 = \beta$: this is Euler predictor-corrector.

\end{frame}

\begin{frame}
  \frametitle{Runge-Kutta 4}

  The most used is the classic fourth order Runge-Kutta method. This
  requires fixing eight free parameters by matching to order
  $h^4$. This gives a family of methods again. \pause

  \vspace{1ex}

  Standard choice is
  \begin{align*}
    y_{n+1} & = y_n + \tfrac{1}{6} \left( k_1 + 2 (k_2 + k_3) + k_4
    \right), \\
    k_1 & = h f(x_n, y_n), \\
    k_2 & = h f(x_n + h / 2, y_n + k_1 / 2), \\
    k_3 & = h f(x_n + h / 2, y_n + k_2 / 2), \\
    k_4 & = h f(x_n + h    , y_n + k_3    ).
  \end{align*} \pause

  \vspace{1ex}

  The local error term is ${\cal O}(h^5)$ leading to a global error
  ${\cal O}(h^4)$.

\end{frame}

\begin{frame}
  \frametitle{Example}

  Apply the RK4 method to
  \begin{equation*}
    y'(x) = - \sin(x), \quad y(0) = 1.
  \end{equation*}
  Integrate to $x = 0.5$. Using $h = 0.1$ gives an error $4.8 \times
  10^{-7}\%$; using $h = 0.01$ gives an error of $4.8 \times
  10^{-11}\%$, showing fourth order convergence. \pause

  \vspace{1ex}

  Compare with an error, for $h=0.01$, of $10^{-3}\%$ for the Euler
  predictor-corrector method, and $0.24\%$ for the simple Euler
  method.

  \vspace{1ex}

  RK4 more efficient \emph{despite} needing four times the function
  evaluations of Euler's method.

\end{frame}

\begin{frame}
  \frametitle{Example: 2}

  Consider the system
  \begin{equation*}
    \left\{
      \begin{aligned}
        \dot{x} & = -y \\ \dot{y} & = x
      \end{aligned} \right., \quad x(0) = 1, \, \, y(0) = 0.
  \end{equation*}
  In polar coordinates this is $\dot{r} = 0$, $\dot{\phi} = 1$.
  \begin{columns}
    \begin{column}{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{0.4\textheight}
        \only<2-3|handout:1>
        {
          Use the RK4 method with $h=0.1$. At $t=500$ the
          result matches the correct answer to the eye.
        }
        \only<3|handout:1>
        {

          \vspace{1ex}
          The growth of the radius makes the errors visible, but they
          are still tiny.

        }
        \only<4-5|handout:2>
        {
          Use the RK4 method with $h=0.01$. At $t=500$
          the result matches the correct answer to the eye.
        }
        \only<5|handout:2>
        {

          \vspace{1ex}
          The growth of the radius remains, but is minute.
        }
      \end{overlayarea}
    \end{column}
    \begin{column}{0.5\textwidth}
      \begin{overlayarea}{\textwidth}{0.6\textheight}
        \only<2|handout:0>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/RK4_1}
          \end{center}
        }
        \only<3|handout:1>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/RK4_rad1}
          \end{center}
        }
        \only<4|handout:0>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/RK4_2}
          \end{center}
        }
        \only<5|handout:2>
        {
          \begin{center}
            \includegraphics[height=0.5\textheight]{figures/RK4_rad2}
          \end{center}
        }
      \end{overlayarea}
    \end{column}
  \end{columns}

\end{frame}

\section{Summary}

\subsection{Summary}

\begin{frame}
  \frametitle{Summary}

  \begin{itemize}
  \item Euler's method has local error $\propto h^2$, hence global
    error $\propto h$.
  \item The Euler predictor-corrector method has local error $\propto
    h^3$, hence global error $\propto h^2$.
  \item \emph{Multistage} methods such as Runge-Kutta methods require
    only one known value $\by_{n}$ to start, and compute (many)
    estimates of the function $\bfm{f}$ for the algorithm to update
    $\by_{n+1}$.
  \item Runge-Kutta methods are the classic multistage methods; the
    predictor-corrector method is a second order RK method.
  \item RK4 is useful in practice.
  \end{itemize}

\end{frame}

\end{document}


\section{Materials and methods}
\label{sec:methods}

To demonstrate the objective of replicating \Canna\ display dives,  I simulated the system in \MATLAB\ and Simulink.  This was to be followed up with a proof-of-concept demonstration, provided I am able to achieve successful trials in simulation first; however, the demonstration was affected by the global COVID-19 pandemic. Simulation was used because it was more general and allowed easy changing of parameters and control methods to more completely explore the space; simulations also allowed examination of behavior at actual hummingbird flight speeds without risking excessive damage to the hardware. Simulations make simplifying assumptions by necessity, so actual testing and demonstration using real quadrotors--namely the Crazyflie 2.1--were also planned. Unfortunately, due to the inaccessibility of hardware brought about by the stay-at-home orders issued by the DoD in March of 2020, a final hardware demonstration of trajectory flight was never executed. Progress made towards achieving a hardware demonstration is described later in this report.
% This part is way too wordy and non-sequitur.
%Before I describe these processes, it is important to mention why these are the methods I chose to implement. A simulation is general in that I can run the simulation for many iterations to see how the controller will react to different input parameters. The input parameters themselves are very specific; however, the ability to change parameter values will allow me to assess the full capabilities of the quadrotor system to determine if I can eliminate the need to do a time-scaled comparison to the hummingbird trajectory. The simulation will be coded using MATLAB and Simulink, both are software with which I have the most experience in creating simulations. My simulation code will be made available after the completion to this project on Git for those wishing to replicate my simulated results. Unfortunately, the realism of this simulation is limited, and as a result I will have to do a proof-of-concept demonstration to truly prove the ability of the quadrotor. Being a much more specific process, it will likely vary slightly from the simulation results, and need to be tweaked based on the level of success seen in the first few trials. Ultimately, the proof-of-concept demonstration is an essential piece to this research since a simulation doesn’t have any real world application except in principle. The experiment will enlist the use of a Crazyflie quadrotor platform, and an OptiTrack motion capture system in order to gather position vs. time data. This will help ensure accuracy in the quadrotor’s trajectory and as such, provide a higher level of confidence in the experiment’s success.

For my concept demonstrations, I modeled the rigid body dynamics of the quadrotor using parameters obtained from the Bitcraze website\footnote{\url{https://wiki.bitcraze.io/misc:investigations:thrust}}, and distance measurements taken by calipers. For initial attempts at control, the hover condition was assumed. This is a control linearization region which assumes the attitude of the quadrotor is completely level, i.e. the pitch and roll angles are assumed to be approximately zero degrees during all stages of flight. A basic waypoint position controller was utilized first in order to ensure controller functionality in simulation and on the hardware itself. A PD controller was used initially for position control, as this is a ubiquitous controller well understood by students and control theorists. Initial gains were taken from the simulation \cite{hartman2014quadcopter}
%cite dch33/Quad-Sim on github (this is the 2014 one from hartman cited here) LINK: https://github.com/dch33/Quad-Sim
 and later adjusted to fine-tune the quadrotor response. Additionally, in order to avoid an excessive attitude command in the event of large position errors, saturation limits were be placed on the attitude command of \ang{\pm 45} in order to ensure the quadrotor doesn't over-rotate and accidentally flip over.

The functional block diagram of the overall Crazyflie control system is shown in \fref{fig:demonstration-2}. The laptop is the central node in the control scheme, obtaining position and orientation information from the OptiTrack system, and then relaying this data to the Crazyflie in addition to the next desired position on the trajectory path. The Crazyflie then conducts onboard error calculation and command processing in order to control its position in 3D space.

% the quaternion model in \cite{greiff2017modeling}. A quaternion is defined as
%\begin{equation}
%Q = a + b i + c j + d k
%\end{equation}
%where $i$, $j$, and $k$ are imaginary unit vectors. This model also assumes a right-handed coordinate system where $i\cdot j = k$ , and the fundamental assumptions hold true -- i.e.  $i^2 = j^2 = k^2 = ijk = -1$.  I have chosen to use this model due to the limitations of the Euler model at high rotation angles. The quaternion model will be robust to these extreme angles, and is therefore better suited for my experiment. \emph{\textbf{Evangelista comment here: this is sort of strange to mention here. It's like saying I plan to use math, or matrices, or eigenvalues. Choosing to represent rotations as quaternions is a useful thing for avoiding gimbal lock associated with singularities in the Euler angle representation, but would not be considered a defining characteristic of what you plan to do, more like a low level design choice comparable to what units you use, or if you like floats or doubles.}}

\subsection{Simulation of bio-inspired dive pullout maneuvers in \Matlab}
%To create a feasible simulation for this research, I will model the Crazyflie quadrotor platform to be used in simulation. This involves determining the various thrust vectors of the motors, and the calculation of many performance metrics. Thankfully, much of this work has already been done, and I will be relying heavily on the work in \cite{cheng2016flight} to create a useful model for the quadrotor in \MATLAB\ and Simulink.
% Due to the extreme maneuvering of the quadrotor, I have chosen to use a quaternion coordinate system to describe its position, for flight control purposes only. %A quaternion is defined as a vector of one real and three imaginary vector directions, $i$, $j$, and $k$.

Using the model of the Crazyflie quadrotor described in the \lstinline{PC_Quadcopter_Simulation.slx} \MATLAB\ Simulink simulation \cite{hartman2014quadcopter},
%cite quadsim_68
and hummingbird trajectory data obtained from \cite{clark2009courtship}, several trajectory comparisons were conducted in the $x$, $y$, $z$ Cartesian coordinate frame between the true hummingbird trajectory (desired trajectory) and the actual flown trajectory by the simulated Crazyflie. Since the dive trajectories have little deviation in the $y$ coordinate axis, they are compared only in the $x$ and $z$ axes for all relevant error calculations.

%I added the following bit to explain the initial conditions, but if it's too wordy or doesn't make sense here it should be moved to a spot where it makes more sense to explain this concept.
Simulations were conducted from two different initial quadcopter states. The first simulation was run with the quadcopter starting from rest on the ground. This requires the quadcopter to ascend to the top of the trajectory first and then begin the trajectory flight, which simulates what I would have to replicate for the hardware demonstration. The second initializes the quadcopter at the beginning of the hummingbird trajectory, with an initial velocity determined by calculating the instantaneous velocity of the second data point $\delta x/\delta t$, and $\delta z/\delta t$ (both $x$ and $z$ directions) of the original hummingbird trajectory, where $\delta x = x(t = t_1) - x(t = t_0)$. These velocities were then multiplied by a scale factor based on the simulation speed (e.g. the initial velocities were reduced by a factor of 20 for a simulation trajectory at 20 times reduced speed) in order to maintain a realistic starting velocity at each of the different trajectory speeds.
%end of initial conditions spiel
%The feedback diagram of this control concept is shown below in \fref{fig:demonstration-2}. In principle, a desired trajectory will be developed from a path-planning algorithm that takes in the time scaled hummingbird trajectory and calculates the idealized pitch and motor torque and speed at each time step in order to fit this trajectory with as little error as possible. This signal will be combined with some sort of inertial measurement true position feedback to produce the error signal to the onboard flight controller. The flight controller will send a signal to the motors on the quadrotor based on this error signal to come as close as possible to zero error for the next time step, and the cycle will repeat until the quadrotor has completed its maneuver. My simulation will replicate this decision-making process using numerical integration to obtain the position data for every iteration.

%this figure needs to be moved but I'm not sure how. It's printing well above the section that it's listed in here. add a [h] (for here) or move the figure around where it is in the code.
\begin{figure}[h]
\begin{center}
\includegraphics[width=0.8\columnwidth]{\myroot/figures/FunctionalBlockDiagram_system.png}
\end{center}
\caption{Functional block diagram of the data communication flow concept for the autonomously controlled Crazyflie, with an included onboard controller feedback loop.}
\label{fig:demonstration-2}
\end{figure}

To determine the accuracy of the trial I compared the actual flight path of the quadrotor with the desired flight path, and determined a time-scaled root mean square error between the two position vs time datasets. This error was calculated using the distance formula \fref{eq:demonstration-1}:
\begin{equation}
e_p(t) = \sqrt{(x_a(t)-x_d(t))^2 + (z_a(t)-z_d(t))^2}
%\begin{bmatrix}
%x(t) \\ y(t) \\ z(t)
%\end{bmatrix}_a^2
%-
%\begin{bmatrix}
%x(t) \\ y(t) \\ z(t)
%\end{bmatrix}_d^2
%}
\label{eq:demonstration-1}
\end{equation}
where $e_p(t)$ is the position error at a specific time step (time $t$) in the trajectory, and the subscripts $a$ and $d$ represent the actual traveled trajectory and the desired trajectory respectively. The error in the $y$ axis is omitted since the trajectory is 2D in the $x$-$z$ plane and errors in the $y$ axis are therefore negligible for a stable controller, as is assumed. The error for every time step was averaged together to determine the root mean square error of the data. Additionally, the standard deviation $\sigma$ of the position error was calculated using the \lstinline{std()} function in \MATLAB. Without any other indications, a successful trial was considered to be one with root mean square error of less than \SI{10}{\centi\meter} over the entire trajectory, and with a $\sigma$ value of less than \SI{5}{\centi\meter}. Since the hummingbird trajectory is time-scaled down to only a fraction of its true speed, the quadrotor onboard controller was responsible for marking every position at the time it is supposed to be located at that position, therefore limiting the effects that motor operation constraints, e.g. thrust saturation, as a possible source of error in the trajectory flight.

\subsection{Tuning controller gain} %TALK HERE ABOUT CONTROLLER GAIN TUNING METHODS
After the initial simulation runs, I anticipated large position errors over the trajectory flight. As such, it was desirable to tune the control gains in order to achieve a lower root mean square error. The simulation consists of two separate levels of control. The first level is a high-level (outer loop) PD controller concerning the $x$ and $y$ position of the quadcopter, and the second level is the lower-level (inner loop) PID controller that actually determines the pitch and roll angle of the quadcopter, as well as its vertical ($z$) position. The first level is a path planning step carried out among the Optitrack (sensor), control computer, and the drone.  The second level is flight stabilization (auto mode) carried out by the flight controller native onboard the drone. The output of the first level controller is factored as input into the second level controller, along with the quadcopter state. Separate control gains are used for each degree of freedom, i.e. the control gains for the pitch angle of the quadcopter are completely independent of those for the roll angle, altitude, yaw angle, etc. In all, there are eight different control gains that characterize the behavior of the quadcopter through the 2D $x$-$z$ hummingbird trajectory: the proportional and derivative control gains for the $x$ position ($K_{px}$, and $K_{dx}$), and the proportional, derivative, and integral control gains for both the pitch $\theta$ and altitude $z$ $($respectively, $K_{p\theta}$, $K_{d\theta}$, $K_{i\theta}$, $K_{pz}$, $K_{dz}$, $K_{iz}$).

In general, the effects of changing the proportional control gain will increase the responsiveness of the system with regard to the parameter effected by the controller. As the proportional gain is increased, overcontrolling is noted as oscillations around the desired state grow with a continued gain increase. An increase in the derivative control gain will help to counterbalance this effect to improve the overall stability of the system and reduce overshoot, while an increase in the integral control gain will reduce/eliminate oscillations around the desired settling state, bringing the steady state error to zero. As such, with regard to the effect of the control gains on the simulation output, I expect that increasing both the proportional and derivative gains should help to quicken the response of the quadrotor, and achieve a smaller root mean square error when flying the desired trajectory while remaining stable.

\subsubsection{Manual gain tuning}
Several methods of gain tuning were attempted, with varying levels of success, to reduce the root mean square error of the quadrotor flight. These methods include manual gain tuning, nonlinear optimization routines, and marginal analysis. Manual gain tuning was attempted first. Initially, the proportional gain was increased separately for each of the three characteristic control loops ($x$, $z$, and $\theta$) until signs of overcontrol/instability were noticed in the response (mostly by large oscillations around the desired control point). Then the derivative control gains were increased for each of these controllers to quell the unstable oscillations. Manual gain tuning was successful at reducing the root mean square error, and the detailed/numerical results are recorded in the Results section of this report.

\subsubsection{Attempted optimization using \lstinline{fmincon}}
After the manual gain tuning was unable to achieve the desired performance for the quadrotor, I began looking into different ways to attempt to optimize the controller gains to achieve a minimum root mean square error. I decided to use the \MATLAB\ function, \lstinline{fmincon()} to optimize the controller gains, which finds the minimum value of a constrained nonlinear multivariable function through an iterative process. Before attempting to use the function for my problem, I completed the example problem listed in the \MATLAB\ documentation for \lstinline{fmincon()} to minimize Rosenbrock's function. This test allowed me to orient myself with using the \lstinline{fmincon()} tool, and helped to build my objective function for use on my simulation.

In defining my optimization problem, the eight characteristic control gains ($K_{px}$, $K_{dx}$, $K_{p\theta}$, $K_{d\theta}$, $K_{i\theta}$, $K_{pz}$, $K_{dz}$, and $K_{iz}$) are the decision variables, using the values I had determined during my manual gain tuning as the initial point $x0$. My output variable (what I'm trying to optimize) is the root mean square error between the desired and actual trajectory flown by the quadcopter in the simulation. My objective function was the piece that was a little less straightforward, as it required multiple lines of \MATLAB\ code to achieve. I first set up the appropriate simulation parameters based on the changes to the decision variables made by the \lstinline{fmincon()} optimization routine. Then, I ran the simulation for a set time of 20 seconds to allow sufficient time for the quadcopter to complete the trajectory flight. Following this, I had to determine the start and end points of the trajectory flight. The start point was simply the first index of the simulation output, since the quadcopter started at the top of the trajectory. The endpoint was determined by the index of the output where the commanded position at that index was the exact same as the commanded position two iterations after that point. (The simulation continues to command the last timeseries position provided by the input until the simulation time runs out, therefore a repeated position command in adjacent time steps means that the dive trajectory has ended.) Once the starting and ending index of the simulation output were determined, I was then able to use the error calculation described by \fref{eq:demonstration-1} to determine the root square error of each time step, and average these values to find the root mean square error, completing my objective function.

In my first attempts at running the optimization routine I left the decision variables only constrained by the basic requirement that they had to be nonzero and positive. The optimizer would complete the first iteration at the initial point with no problems, but as soon as it changed the gains for the second iteration, it caused the system to become unstable and the simulation crashed with multiple zero-crossing errors. In attempts to remedy this, I changed several different simulation settings to ignore zero crossing errors and change the simulation step size, all to no avail. I then tried to add constraints to the upper bound of the decision variables (they were already constrained on the lower bound to be greater than zero), in an attempt to prevent the simulation from becoming unstable while \lstinline{fmincon()} tweaked the control gains. This was successful in that the optimization routine was able to run its course; however, the result only made small changes to the initial control gains (less than a tenth of a percent change) which resulted in no significant effect on the output variable, the root mean square error. The simulation is therefore seemingly unable to support optimization routines as it is currently written.

\subsubsection{Marginal analysis... when optimization failed}
After the attempt at control gain optimization proved unsuccessful, I decided to conduct a marginal analysis in order to determine which variables would contribute most to the reduction of the root mean square error. A marginal analysis test simply makes a small change to a decision variable, and then records the overall benefit gained and cost incurred from making that small change. Running this analysis provides insight into how this current controller scheme could be tuned further to achieve greater success at flying the quadcopter through the desired trajectory. I again used the gain values obtained from manual tuning as my initial point, and wrote a \MATLAB\ script to iterate through each gain value and run the simulation on three separate cases for each: a run with the gain at the initial point, a run with the gain decreased by 5\%, and a run with the gain increased by 5\%. After each simulation, error data was collected and saved for processing.

\subsection{Demonstration of bio-inspired dive pullout maneuver using Crazyflie quadrotor}
Hardware implementation is essential to verifying that the findings of the simulation are accurate. The path to a successful hardware demonstration was a multi-step process. We needed to test the capability of the crazyflie to operate under manual control, open loop control, and then finally introduce closed-loop control of the crazyflie using an optical motion tracking system. As such, the hardware components necessary for this experiment include a fully functional Crazyflie quadrotor (Bitcraze, Malm\"{o}, Sweden), and an OptiTrack system (NaturalPoint Inc., Corvallis, OR). The quadrotor will be the object of the experiment, and the OptiTrack system serves as a highly accurate way to obtain position data for the Crazyflie in flight. It achieves this using visual information from nearly 20 cameras staged around the outside of the testing area. Initial experimentation will be conducted indoors to minimize any aerodynamic noise, although future trials could test controller robustness by introducing environment disturbances. Several batteries for the Crazyflie are needed to ensure sufficient trial and testing periods, and approximately five OptiTrack visual markers must be affixed asymetrically\footnote{This is to clearly distinguish the rigid body axes relative to the markers to ensure proper pose readings.} on each Crazyflie drone to ensure that it is able to be detected by the OptiTrack system and that its state is measured appropriately. These marker additions will be taken into account in simulation first in order to ensure readiness to counteract any effect they may have on the dynamics of the quadrotor in the feedback loop.

As an initial test of hardware, I used the \lstinline{cfclient} software (Bitcraze, Malm\"{o}, Sweden; \url{https://github.com/bitcraze/crazyflie-clients-python}) to link to the crazyflie from a linux machine, and test manual control of the crazyflie using a Logitech USB gamepad (Logitech F310; Lausanne, Switzerland) and a Crazyradio sending velocity and thrust commands to the Crazyflie. This allowed me to gain a basic understanding of how the crazyflie communicated with the linux machine over radio control, and how this might be implemented in a python script. I then tested the motion tracking system capabilities by obtaining automatic 3D position tracking data via the OptiTrack. The OptiTrack system worked independently from this workstation, and recorded $x$, $y$, $z$ position data while logging the time $t$ that each data point was recorded at.

%The Crazyflie was controlled manually, from a computer, using the bitcraze \lstinline{cfclient} software (Bitcraze, Malm\"{o}, Sweden; \url{https://github.com/bitcraze/crazyflie-clients-python}). To establish manual flight control, the computer was equipped with a USB gamepad (Logitech F310; Lausanne, Switzerland) and a Crazyradio sending velocity and thrust commands to the Crazyflie. The OptiTrack system worked independently from this workstation, and recorded $x$, $y$, $z$ position data while logging the time $t$ that each data point was recorded at.

%DISCUSS FURTHER HARDWARE DEMONSTRATION HERE. HURDLES, ETC. FIND INFO IN PROGRESS FILE ON DRIVE
%In simulation, we observed that as the speed of the maneuver increased, it was more difficult for the crazyflie to fly it accurately. We wanted to test the same trajectories on hardware in order to see if the actual crazyflie would behave in a similar manner (show a similar error in trajectory flight flying at the same speeds tested in simulation).

%NOTE: THE TeX compiler doesn't like underscores, and won't compile if you have a naked one in the text. To solve, just put a '\' in front of it and it will render the underscore '_' as just text and will compile just fine.
The next step towards a full hardware demonstration was to test autonomous control of the Crazyflie. To support this, I cloned the \lstinline{whoenig/crazyflie_ros} github repository (\url{https://github.com/whoenig/crazyflie_ros}) which utilizes a ROS framework to run multiple python and C++ scripts that send appropriate commands to the Crazyflie based on the tasking in the files that are run. The demo package of the repository contained a plethora of different example scripts and launch files that would provide the capability needed to complete the hardware testing that I desired, including options for manual joystick control, open-loop hovering control, and closed-loop control using both Vicon (Vicon Motion Capture; Oxford, UK), an optical motion tracking system, and VRPN connection to an alternate motion tracking system such as OptiTrack (Optitrack; Corvallis, OR). I progressively ran through several of the lower level control demo packages, testing first manual control with the USB gamepad using ROS, followed by open-loop control (which ultimately resulted in the Crazyflie crashing into the ceiling due to no closed-loop feedback). Through adaptation of the vrpn control demo scripts, I was able to achieve a successful implementation of a closed-loop hover at a desired 3D location. This worked similar to waypoint control, but only consisting of two waypoints (i.e. the starting and ending point). This meant that waypoint trajectory flight would be possible in the Maury lab with the provided controller.

Unfortunately, due to the global COVID-19 pandemic, this was as much as I was able to achieve in terms of a hardware demonstration of capability. My work up to this point has been saved in a github repository (\url{https://github.com/devangel77b/marcello-2020-code}), and should be easily accessible and run-able by any student wishing to use the combined Crazyflie OptiTrack system to conduct waypoint flight hardware demonstrations. I have also left fairly detailed instructions in a Google document, which is available on the \lstinline{marcello-2020} Google Shared Drive.