\section{Technical Details}

\subsection{Problem Statement}

Minimize the objective function:
\begin{equation*}
\underset{t_0, t_F, \bm{x}(t), \bm{u}(t)} \min \;
J_B\big(t_0,t_F,\bm{x}(t_0),\bm{x}(t_F) \big) +
\int_{t_0}^{t_F} \! J_P \big( \tau, \bm{x}(\tau), \bm{u}(\tau) \big)  \; d\tau
\label{eqn:optimTraj_objectiveFunction}
\end{equation*}

Subject to the constraints:
\begin{align*}
& \quad   \dot{\bm{x}}(t) = \bm{f} \big(t, \bm{x}(t), \bm{u}(t)\big)   & \quad & \eqnTxt{system dynamics}\\

& \quad   \bm{C}_P \big(t, \bm{x}(t), \bm{u}(t)\big) \leq \bm{0}    & \quad &   \eqnTxt{path constraints}\\

& \quad   \bm{C}_B\big(t_0,t_F,\bm{x}(t_0),\bm{x}(t_F) \big) \leq \bm{0}    & \quad & \eqnTxt{boundary constraints} \\

& \quad   \bm{x}^- \leq \bm{x}(t) \leq \bm{x}^+    & \quad & \eqnTxt{constant bounds on state} \\
& \quad   \bm{u}^- \leq \bm{u}(t) \leq \bm{u}^+    & \quad & \eqnTxt{constant bounds on control} \\

& \quad   t^- \leq t_0 < t_F \leq t^+    & \quad & \eqnTxt{bounds on initial and final time} \\
& \quad   \bm{x}_0^- \leq \bm{x}(t_0) \leq  \bm{x}_0^+   & \quad & \eqnTxt{bound on initial state} \\
& \quad   \bm{x}_F^- \leq \bm{x}(t_F) \leq \bm{x}_F^+    & \quad & \eqnTxt{bound on final state}
\end{align*}

Assuming that the user-defined objective, dynamics, and constraint functions ($J_P$, $J_B$, $\bm{f}$, $\bm{C}_P$, $\bm{C}_B$) are smooth. The user must provide an initial guess for the decision variables $t_0$, $t_F$, $\bm{x}(t)$, and $\bm{u}(t)$, and ensure that a feasible solution exists.

\subsection{Trapezoidal Direct Collocation:  \hc{trapezoid}}

Trapezoidal direct collocation works by making the assumption that the optimal trajectory can be approximated using a low-order spline. In this case, the dynamics, objective function, and control trajectories are approximated using a linear spline, and the state trajectory is the quadratic spline, obtained by integration of the (linear) dynamics spline. Integration of a linear spline is computed using the trapezoid rule, hence the name. The implementation of trapezoidal direct collocation in OptimTraj is almost entirely based on the method as it is described in \cite{Betts2010}.

\subsection{Hermite-Simpson Direct Collocation:  \hc{hermiteSimpson}}

Hermite-Simpson direct collocation works by making the assumption that the optimal trajectory can be approximated using a medium-order spline. In this case, the dynamics, objective function, and control trajectories are approximated using a quadratic spline, and the state trajectory is a cubic hermite spline, obtained by integration of the (quadratic) dynamics spline. Integration of a quadratic spline is computed using Simpson's rule. There are two versions of this method: separated and compressed. In the separated form, the state at the mid-point of each trajectory segment is included as a decision variable, and the Hermite-interpolation is enforced with a constraint. In the compressed form, the state at the mid-point is computed from the definition of the Hermite-interpolant during optimization. OptimTraj implements the separated Hermite-Simpson method, almost entirely based on the method as it is described in \cite{Betts2010}.

\subsection{Chebyshev--Lobatto Orthogonal Collocation:  \hc{chebyshev}}

Chebyshev--Lobatto orthogonal collocation works by representing the entire trajectory using a high-order Chebyshev orthogonal polynomial. The implementation here might also be called pseudospectral or global collocation, since the entire trajectory is represented using a single segment, rather than several segments. Orthogonal collocation methods can be divided into three categories: Gauss, Radau, and Lobatto. In a Gauss method, neither end-point of a segment is a collocation point, in a Radau method, a single end-point of the segment is a collocation point, and in a Lobatto method, both end-points are collocation points \cite{Garg2010}.

\par The implementation here uses the ChebFun toolbox \cite{Driscoll2014} for computing the Chebyshev--Lobatto collocation points, and also for interpolation of the solution. More details regarding orthogonal polynomials and calculations with them can be found in \cite{Berrut2004a} and \cite{Trefethen2012}. The details of the collocation method itself are largely drawn from \cite{Vlassenbroeck1988}.

\subsection{Runge--Kutta $4^\text{th}$-order Multiple Shooting:  \hc{rungeKutta}}

A multiple shooting method works by breaking the trajectory into segments, and approximating each segment using an explicit simulation. In this case, we use a $4^\text{th}$-order Runge--Kutta method for the simulation. A defect constraint is used to ensure that the end of each trajectory segment correctly lines up with the next. The interpolation of the solution trajectory, for both control and state, is approximated using a cubic Hermite spline. The implementation here, except for interpolation, is as described in \cite{Betts2010}.

\subsection{GPOPS-II:    \hc{gpops}}

OptimTraj includes a wrapper to the software GPOPS-II \cite{Patterson2013}, a professionally developed trajectory optimization library for Matlab. It implements a nicely optimized version of Radau orthogonal collocation with adaptive meshing \cite{Darby2011a}. There are a few choice in GPOPS-II for both collocation metho and the adaptive meshing. It also supports analytic gradients using automatic differentiation. It is included in OptimTraj for two reasons: 1) GPOPS-II can be used to benchmark and verify the methods in OptimTraj, and 2) GPOPS-II provides a collection of methods that are not otherwise available in OptimTraj.

\subsection{Resources for Learning Trajectory Optimization}

The single best resource for learning about trajectory optimization is the textbook by John T. Betts: "Practical Methods for Optimal Control and Estimation Using Nonlinear Programming" \cite{Betts2010}. The textbook by Bryson and Ho: "Applied Optimal Control" \cite{Bryson1975}. Both of these books are also excellent for learning about nonlinear programming and optimization.\\

\par There are two good review papers about trajectory optimization. The paper by Betts \cite{Betts1998} is more focused on direct collocation and shooting methods, while the paper by Rao \cite{Rao2009} is more focused on orthogonal collocation methods.\\

\par Understanding orthogonal collocation is rather challenging, and I found that it was first necessary to get a solid understanding of orthogonal polynomials and function approximation. I started by reading "Approximation Theory and Approximation Practice" \cite{Trefethen2012} by Trefethen, and then his paper on barycentric interpolation \cite{Berrut2004a}. Some intuition can also be obtained by reading the source code of ChebFun \cite{Driscoll2014}: \\
\url{http://www.chebfun.org/}\\

\par Russ Tedrake at MIT has also provided some excellent resources for learning about trajectory optimization, and robotics in general. The first is his online course on underactuated robotics. You can download the pdf of the course notes \cite{Tedrake2009}, or access the full course and video lectures at: \\
\href{http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-832-underactuated-robotics-spring-2009/index.htm}{http://ocw.mit.edu/courses/} \\

Second, his group at MIT are producing a planning, control, and analysis toolbox called Drake, which includes trajectory optimization. It is open source, so it is another good place to read the source code and figure out how it works: \\
\url{https://github.com/RobotLocomotion/drake}\\

\par I wrote a tutorial paper for learning trajectory optimization. It goes into depth covering all technical content for the two direct collocation methods in this library, as well as a variety of related topics:
\url{https://epubs.siam.org/doi/pdf/10.1137/16M1062569}

\par There is also a tutorial page on my website. It contains a high-level summary and links to a variety of other resources including a video, slides, documents, and code samples: \\
\url{http://www.matthewpeterkelly.com/tutorials/trajectoryOptimization/index.html}


\lab{CVXPY}{CVXPY}
\objective{CVXPY is a package of Python functions and classes designed for the purpose of convex optimization.
In this lab we use these tools for linear and quadratic programming.
We will solve various optimization problems using CVXPY and optimize eating healthily on a budget.}

\section*{Linear Programs} % ==================================================

A \emph{linear program} is a linear constrained optimization problem. Such a problem can be stated in several
different forms, one of which is
\begin{align*}
\text{minimize}\qquad &\c\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
&A\x = \b.
\end{align*}

The symbol $\preceq$ denotes that the components of $G\x$ are less than the components of $\mathbf{h}$.
In other words, if $\x\preceq\y$, then $x_i < y_i$ for all $x_i\in\x$ and $y_i\in\y$.
CVXPY accepts  $\leq, \geq$, and $=$ in its constraints as long as the equations satisfy convexity requirements described later in this chapter, so we can reformulate this problem in yet another form:
\begin{align*}
\text{minimize}\qquad &\c\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
&P\x \succeq \mathbf{q}\\
&A\x = \b.
\end{align*}
CVXPY accepts NumPy arrays and SciPy sparse matrices for the constraints, but the variable $\x$ must be a CVXPY \li{Variable}.

% Students have not yet learned about the dual problem. May be included in a later lab.
\begin{comment}
The corresponding \emph{dual program} for the above linear program has the form
\begin{align*}
\text{maximize}\qquad &-h\trp z - b\trp y \\
\text{subject to}\qquad &G\trp z + A\trp y + c = 0\\
 &z \geq 0.
\end{align*}
CVXOPT provides functions to solve both the original (\emph{primal}) linear program and its dual program.
\end{comment}

Consider the following example:
\begin{align*}
\text{minimize}\qquad &-4x_1-5x_2 \\
\text{subject to}\qquad &x_1+2x_2 \leq 3 \\
	        &2x_1+x_2 = 3 \\
		&x_1, x_2 \geq 0
\end{align*}
We can solve this problem using the following code.
Note that \li{cvxpy.Problem()} accepts the constraints as a single list and that \li{>=} represents both standard and elementwise greater than or equal to.
The symbols \li{<=} and \li{==} are similarly versatile.

\begin{lstlisting}
>>> import cvxpy as cp
>>> import numpy as np

#First we'll initialize the objective
#We can declare x with its size and sign
>>> x = cp.Variable(2, nonneg = True)
>>> c = np.array([-4, -5])
>>> objective = cp.Minimize(c.T @ x)

#Then we'll write the constraints
>>> A = np.array([2, 1])
>>> G = np.array([1, 2])
>>> P = np.eye(2)
>>> constraints = [A @ x == 3, G @ x <= 3, P @ x >= 0] #This must be a list

#Assemble the problem and then solve it
>>> problem = cp.Problem(objective, constraints)
>>> print(problem.solve())
-8.999999999850528
>>> print(x.value)
array([1., 1.])
\end{lstlisting}
%CVXPY can take in == or >= or <= so I deleted an irrelevant section here about switching constraint types

%CVXPY Install Problem
\begin{warn}
	If you are having trouble with \li{pip install cvxpy} or \li{conda install cvxpy} check the following:
	\begin{itemize}
		\item CVXPY requires a C++ compiler, most MacOs ad Linux Systems have them built in.
		If you are running Windows, make sure that you have the "C++ builder tools" from the Visual Studio Build Tools installed.
		\item CVXPY requires specific versions of packages in order to run, check that you have the right version of your packages.
		The most common is NumPy is not up to date.
	\end{itemize}
\end{warn}

%Problem 1
\begin{problem}
Solve the following convex optimization problem:
\begin{align*}
\text{minimize}\qquad &2x_1+x_2+3x_3 \\
\text{subject to}\qquad &x_1+2x_2 \leq 3 \\
		&x_2-4x_3\leq 1\\
	        &2x_1+10x_2+3x_3 \geq 12 \\
		&x_1 \geq 0 \\
		&x_2 \geq 0 \\
		&x_3 \geq 0
\end{align*}
Return the minimizer $\x$ and the primal objective value.
\end{problem}
\pagebreak
\subsection*{$l_1$ Norm}
The $l_1$ norm is defined
\[||\x||_1=\sum_{i=1}^n |x_i|.\]
An $l_1$ minimization problem is minimizing a vector's $l_1$ norm, while fitting certain constraints. It can be written in the following form:
\begin{align*}
\text{minimize}\qquad &\|\x\|_1\\
\text{subject to} \qquad &A\x = \b.
\end{align*}

CVXPY includes the $l_1$ norm and many other useful functions.
To specify a norm in CVXPY, use the syntax \li{cp.norm(x, a)} where $a$ represents your choice of norm (1 in this case).
%Removed a big section here about how to modify CVXOPT to do the L1 norm. It's significantly trickier than with CVXPY.

%Problem 2
\begin{problem}
Write a function called \li{l1Min()} that accepts a matrix $A$ and vector $\mathbf{b}$ as NumPy arrays and solves the $l_1$ minimization problem.
Return the minimizer $\x$ and the primal objective value.

To test your function consider the matrix $A$ and vector $\mathbf{b}$ below.
\[
A = \begin{bmatrix}
1 & 2 & 1 & 1\\
0 & 3 & -2 & -1
\end{bmatrix} \qquad
\mathbf{b} = \begin{bmatrix}
7 \\
4
\end{bmatrix}
\]
The linear system $A\x = \b$ has infinitely many solutions.
Use \li{l1Min()} to verify that the solution which minimizes $||\mathbf{x}||_1$ is approximately $\x = [0., 2.571, 1.857, 0.]^T$ and the minimum objective value is approximately $4.429$.
\label{prob:l1}
\end{problem}

\section*{The Transportation Problem}

Consider the following transportation problem:
A piano company needs to transport thirteen pianos from their three  supply centers (denoted by 1, 2, 3) to two demand centers (4, 5).
Transporting a piano from a supply center to a demand center incurs a cost, listed in Table \ref{tab:cost}.
The company wants to minimize shipping costs for the pianos while meeting the demand.
%How many pianos should each supply center send to each demand center?

\begin{table}[H]
\centering
\begin{tabular}{|c|c|}
Supply Center & Number of pianos available\\
\hline
1 & 7\\
2 & 2\\
3 & 4\\
\end{tabular}

\caption{Number of pianos available at each supply center}
\label{tab:supply}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|c|c|}
Demand Center & Number of pianos needed\\
\hline
4 & 5\\
5 & 8\\
\end{tabular}

\caption{Number of pianos needed at each demand center}
\label{tab:demand}
\end{table}

\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
Supply Center & Demand Center & Cost of transportation & Number of pianos\\
\hline
1 & 4 & 4 & $p_1$\\
1 & 5 & 7 & $p_2$\\
2 & 4 & 6 & $p_3$\\
2 & 5 & 8 & $p_4$\\
3 & 4 & 8 & $p_5$\\
3 & 5 & 9 & $p_6$\\
\end{tabular}
\caption{Cost of transporting one piano from a supply center to a demand center}
\label{tab:cost}
\end{table}

A system of constraints can be defined using the variables $p_1,p_2,p_3,p_4,p_5,$ and $p_6$.
First, there cannot be a negative number of pianos transported along any route.
Next, use tables \ref{tab:supply} and \ref{tab:demand} and the variables $p_1...p_6$ to define a supply or demand constraint for each location.
You may want to format this as a matrix.
\begin{comment} %I think they can come up with these on their own.
the following three supply constraints and two demand constraints:
\begin{align*}
p_1 + p_2  &= 7\\
p_3 + p_4  &= 2\\
p_5 + p_6  &= 4\\
p_1 + p_3 + p_5 &= 5\\
p_2 + p_4 + p_6 &= 8
\end{align*}
\end{comment}
Finally, the objective function is the number of pianos shipped along each route multiplied by the respective costs (Table \ref{tab:cost}).
%\[
%4p_1 + 7p_2 + 6p_3 + 8p_4 + 8p_5 + 9p_6.
%\]

\begin{info}
Since our answers must be integers, in general this problem turns out to be an NP-hard problem.
There is a whole field devoted to dealing with integer constraints, called \emph{integer linear programming}, which is beyond the scope of this lab.
Fortunately, we can treat this particular problem as a standard linear program and still obtain integer solutions.
\end{info}

%Problem 3. The point was originally to change equality constraints to inequality, but that's not necessary with CVXPY, so now it's mostly about interpreting the tables.
\begin{problem}
Solve the piano transportation problem. %by converting the last equality constraint into an inequality constraint.
Return the minimizer $\x$ and the primal objective value.
\end{problem}

\section*{Quadratic Programming}

Quadratic programming is similar to linear programming, but the objective function is quadratic rather than linear.
The constraints, if there are any, are still of the same form.
Thus, $G, \mathbf{h}, A$, and $\b$ are optional.
The formulation that we will use is
\begin{align*}
\text{minimize}\qquad &\frac{1}{2}\x\trp Q\x + \mathbf{r}\trp \x \\
\text{subject to}\qquad &G\x \preceq \mathbf{h}\\
 &A\x = \b,
\end{align*}
where $Q$ is a positive semidefinite symmetric matrix.
%In this formulation, we require again that $A$ has full row rank and that the block matrix
%$[Q \quad G \quad A]\trp $ has full column rank.

As an example, consider the quadratic function
\[
f(x_1,x_2) = 2x_1^2 +2x_1x_2 + x_2^2 +x_1 -x_2.
\]
There are no constraints, so we only need to initialize the matrix $Q$ and the vector $\mathbf{r}$.
To find these, we first rewrite our function to match the formulation given above.
If we let
\[
Q = \begin{bmatrix}
  a & b\\
  b & c
\end{bmatrix}, \qquad
\mathbf{r} = \begin{bmatrix}
  d\\
  e
\end{bmatrix},
\qquad \text{and} \qquad
\x = \begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix},
\]
then
\begin{align*}
\frac{1}{2}\x\trp Q\x + \mathbf{r}\trp \x &=
\frac{1}{2}
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix}\trp
\begin{bmatrix}
  a & b\\
  b & c
\end{bmatrix}
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix} +
\begin{bmatrix}
  d\\
  e
\end{bmatrix}\trp
\begin{bmatrix}
  x_1\\
  x_2
\end{bmatrix} \\
&= \frac{1}{2}ax_1^2 + bx_1x_2 + \frac{1}{2}cx_2^2 + dx_1 + ex_2
\end{align*}
Thus, we see that the proper values to initialize our matrix $Q$ and vector $\mathbf{r}$ are:
\begin{align*}
a &= 4  &d = 1 \\
b &= 2  &e = -1 \\
c &= 2
\end{align*}
Now that we have the matrix $Q$ and vector $\mathbf{r}$, we are ready to use the CVXPY function for quadratic programming, \li{cp.quad_form()}.
\begin{lstlisting}
>>> Q = np.array([[4, 2],[2, 2]])
>>> r = np.array([1, -1])
>>> x = cp.Variable(2)
>>> prob = cp.Problem(cp.Minimize(.5 * cp.quad_form(x, Q) + r.T @ x))
>>> print(prob.solve())
[-1. 1.5]
>>> print(x.value)
-1.25
\end{lstlisting}

%Problem 4
\begin{problem}
Find the minimizer and minimum of
\begin{equation*}
g(x_1,x_2,x_3) = \frac{3}{2}x_1^2 +2x_1x_2 + x_1x_3+ 2x_2^2 +2x_2x_3+\frac{3}{2}x_3^2+3x_1 + x_3
\end{equation*}
\\(Hint: Write the function $g$ to match the formulation given above before coding.)
\begin{comment}
\begin{equation}
f(x) = \frac{1}{2}x\trp Qx - x\trp p
\end{equation}
where

\begin{center}
$Q =
\begin{bmatrix}
3 & 2 & 1\\
2 & 4 & 2\\
1 & 2 & 3\\
\end{bmatrix}
$
and $p =
\begin{bmatrix}
3\\
0\\
1\\
\end{bmatrix}
$
\end{center}
\end{comment}

\end{problem}

So far we have only dealt with affine constraints.
When working with non-affine constraints, be aware that CVXPY comes with some Disciplined Convex Programming (DCP) rules.
A minimization problem requires a convex objective function; similarly, a maximization problem requires a concave objective function.
Equality constraints (\li{==}) must be affine.
Less-than constraints (\li{<=}) must have the left side convex and the right side concave.
Greater-than constraints (\li{>=}) must have the left side concave and the right side convex.
This webpage provides a list of which of CVXPY's functions are concave or convex.
\url{https://www.cvxpy.org/tutorial/functions/index.html}

%Problem 5
\begin{problem}
Write a function that accepts  a matrix $A$ and vector $\b$ and solves the following problem.
\begin{align*}
\text{minimize}\qquad &\|A\x - \b\|_2\\
\text{subject to}\qquad &\|\x\|_1 = 1\\
		&\x \succeq 0\\
\end{align*}
To test your function, use the matrix $A$ and vector $\b$ from Problem \ref{prob:l1}.
The minimizer is approximately $\x=[0, 1, 0, 0]$ with objective value $5.099$.
Hint: \li{norm()} is a convex function, so you will have to think of a different way to take the 1-norm.
\end{problem}
\begin{comment}
\begin{problem} %Given that CVXPY comes with built-in norm functions, this feels too easy or maybe out of place.
The $l_2$ maximization problem is to
\begin{align*}
\text{maximize}\qquad &\|\x\|_2\\
\text{subject to} \qquad &A\x = \b.
\end{align*}

This problem is equivalent to a quadratic program, since $\|\x\|_2 = \x\trp \x$.
Write a function that accepts a matrix $A$ and vector $\b$ and solves the $l_2$ maximization problem.
Return the maximizer $\x$ and the primal objective value.

To test your function, use the matrix $A$ and vector $\b$ from Problem \ref{prob:l1}.
The minimizer is approximately $\x=[0.966, 2.169, 0.809, 0.888]^T$ and the minimum primal objective value is approximately $7.079$.
\end{problem}
\end{comment}
\begin{comment}
\section*{Allocation Models}
Allocation models lead to simple linear programs. An allocation model seeks to allocate a valuable resource among competing needs. Consider the following example taken from ``Optimization in Operations Research" by Ronald L. Rardin. %%pg 132

The U.S. Forest service has used an allocation model to deal with the task of managing national forests.
The model begins by dividing the land into a set of analysis areas. Several land management policies (also
called prescriptions) are then proposed and evaluated for each area.
An \emph{allocation} is how much land (in acreage) in each unique analysis area will be assigned to each of the possible prescriptions.
We seek to find the best possible allocation, subject to forest-wide restrictions on land use.

The file \li{ForestData.npy} contains data for a fictional national forest (you can also find the data
in Table \ref{tab:forest}). There are 7 areas of analysis and 3 prescriptions for each of them.

\begin{align*}
&\text{Column 1: $i$, area of analysis} \\
&\text{Column 2: $s_i$, size of the analysis area (in thousands of acres)} \\
&\text{Column 3: $j$, prescription number} \\
&\text{Column 4: $p_{i,j}$, net present value (NPV) per acre in area $i$ under prescription $j$} \\
&\text{Column 5: $t_{i,j}$, protected timber yield per acre in area $i$ under prescription $j$} \\
&\text{Column 6: $g_{i,j}$, protected animal grazing capability per acre for area $i$ under prescription $j$} \\
&\text{Column 7: $w_{i,j}$, wilderness index rating (0 to 100) for area $i$ under prescription $j$}
\end{align*}

\begin{table}[H]
\centering
    \begin{tabular}{c c c c c c c}
&&&Forest Data&&& \\
\hline
Analysis & Acres &Prescrip-&NPV&Timber&Grazing&Wilderness \\
Area&(1000)'s &tion&(per acre) &(per acre)&(per acre)& Index\\
$i$ &$s_i$&$j$& $p_{i,j}$ & $t_{i,j}$&$g_{i,j}$&$w_{i,j}$ \\\hline
1&	75	&1	&503	&310	&0.01&	40\\
&&		2&	140&	50&	0.04	&80\\
&&		3&	203&	0&	0&	95\\ \hline
2&	90&	1	&675&	198&	0.03&	55\\
&&		2&	100&	46&	0.06&	60\\
&&		3&	45&	0&	0&	65\\ \hline
3&	140&	1	&630&	210	&0.04&	45\\
&&		2&	105&	57&	0.07&	55\\
&&		3&	40	&0&	0&	60\\ \hline
4	&60&	1&	330&	112&	0.01&	30\\
&&		2	&40&	30&	0.02&	35\\
&&		3&	295&	0&	0	&90\\ \hline
5	&212&	1	&105	&40	&0.05&	60\\
&&		2	&460&	32	&0.08&	60\\
&& 3	&120&0&	0	&70\\ \hline
6	&98	&1	&490	&105	&0.02	&35\\
&&		2&	55	&25	&0.03	&50\\
&&		3	&180	&0	&0	&75\\ \hline
7&	113&	1	&705	&213&	0.02	&40\\
&&		2&	60	&40	&0.04&	45\\
&&		3	&400	&0	&0	&95\\
\hline
    \end{tabular}
\caption{}
\label{tab:forest}
\end{table}
Let $x_{i,j}$ be the amount of land in area $i$ allocated to prescription $j$.
Under this notation, an allocation is a one-dimensional vector consisting of the $x_{i,j}$'s.
For this particular
example, there are 7 acres, with 3 prescriptions each.
So the allocation vector is a one-dimensional vector with 21 entries.
Our goal is to find the allocation vector that maximizes net present value, while producing at least 40 million
board-feet of timber, at least 5 thousand units of grazing capability, and keeping the average wilderness index at least 70.
The allocation vector is also constrained to be nonnegative, and all of the land must be allocated precisely.

Since acres are in thousands, divide the constraints of timber and animal grazing by 1000 in the problem setup, and compensate for this after obtaining a solution.

The problem can be written as follows:
\begin{align*}
\text{maximize } &\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 p_{i,j}x_{i,j} \\
\text{subject to } &\sum\limits_{j=1}^3 x_{i,j} = s_i  \text{ for } i=1,..,7 \\
	        &\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 t_{i,j}x_{i,j} \geq 40,000 \\
		&\sum\limits_{i=1}^7 \sum\limits_{j=1}^3 g_{i,j}x_{i,j} \geq 5 \\
		&\frac{1}{788} \sum\limits_{i=1}^7 \sum\limits_{j=1}^3 w_{i,j}x_{i,j} \geq 70 \\
		&x_{i,j} \geq 0 \text{ for } i=1,...,7  \text{ and } j=1,2,3
\end{align*}

\begin{problem}
Solve the allocation problem above.
Return the minimizing allocation vector of $x_{i,j}$'s and the maximum total net present value.
Remember to consider the following:
\begin{enumerate}
\item The allocation vector should be a (21,1) NumPy array.
\item Recall that the constraints of timber and animal grazing were divided by 1000.
To compensate, the maximum total net value will be equal to the primal objective of the appropriately minimized linear function multiplied by -1000.
\end{enumerate}
\end{problem}

\end{comment}

\section*{Eating on a Budget}

In 2009, the inmates of Morgan County jail convinced Judge Clemon of the Federal District Court in Birmingham to put Sheriff Barlett in jail for malnutrition.
Under Alabama law, in order to encourage less spending, "the chief lawman could go light on prisoners' meals and pocket the leftover change."\footnote[1]{Nossiter, Adam, 8 Jan 2009, "As His Inmates Grew Thinner, a Sheriff’s Wallet Grew Fatter", \emph{New York Times},\url{https://www.nytimes.com/2009/01/09/us/09sheriff.html}}.
Sheriffs had to ensure a minimum amount of nutrition for inmates, but minimizing costs meant more money for the sheriffs themselves.
Judge Clemon jailed Sheriff Barlett until a plan was made to use all allotted funds, \$1.75 per inmate, to feed prisoners more nutritious meals.
While this case made national news, the controversy of feeding prisoners in Alabama continues as of 2019\footnote[2]{Sheets, Connor, 31 January 2019, "Alabama sheriffs urge lawmakers to get them out of the jail food business", \url{https://www.al.com/news/2019/01/alabama-sheriffs-urge-lawmakers-to-get-them-out-of-the-jail-food-business.html}}.

The problem of minimizing cost while reaching healthy nutritional requirements can be approached as a convex optimization problem.
Rather than viewing this problem from the sheriff's perspective, we view it from the perspective of a college student trying to minimize food cost in order to pay for higher education, all while meeting standard nutritional guidelines.

The file \li{food.npy} contains a dataset with nutritional facts for 18 foods that have been eaten frequently by college students working on this text.
A subset of this dataset can be found in Table \ref{tab:food-data}, where the "Food" column contains the list of all 18 foods.

The columns of the full dataset are:
\begin{align*}
& \text{Column 1: $p$, price (dollars)} \\
& \text{Column 2: $s$, servings per container} \\
& \text{Column 3: $c$, calories per serving} \\
& \text{Column 4: $f$, fat per serving (grams)} \\
& \text{Column 5: $\hat{s}$, sugar per serving (grams)} \\
& \text{Column 6: $\hat{c}$, calcium per serving (milligrams)} \\
& \text{Column 7: $\hat{f}$, fiber per serving (grams)} \\
& \text{Column 8: $\hat{p}$, protein per serving (grams)}
\end{align*}

 \begin{table}[H]
% \begin{adjustwidth}{-.5in}{-.5in}
\begin{tabular}{|c||c|c|c|c|c|c|c|c|c|c|}
\hline
\textbf{Food} & \textbf{Price} & \textbf{Servings} & \textbf{Calories} & \textbf{Fat} & \textbf{Sugar} & \textbf{Calcium} & \textbf{Fiber} & \textbf{Protein} \\
& $p$ & $s$ & $c$ & $f$ & $\hat{s}$ & $\hat{c}$ & $\hat{f}$ & $\hat{p}$ \\
& dollars & & & g & g & mg & g & g \\ \hline\hline
Ramen & 6.88 & 48 & 190 & 7 & 0 & 0 & 0 & 5 \\ \hline
Potatoes & 0.48 & 1 & 290 & 0.4 & 3.2 & 53.8 & 6.9 & 7.9 \\ \hline
Milk & 1.79 & 16 & 130 & 5 & 12 & 250 & 0 & 8 \\ \hline
Eggs & 1.32 & 12 & 70 & 5 & 0 & 28 & 0 & 6 \\ \hline
Pasta & 3.88 & 8 & 200 & 1 & 2 & 0 & 2 & 7 \\ \hline
Frozen Pizza & 2.78 & 5 & 350 & 11 & 5 & 150 & 2 & 14 \\ \hline
Potato Chips & 2.12 & 14 & 160 & 11 & 1 & 0 & 1 & 1 \\ \hline
Frozen Broccoli & 0.98 & 4 & 25 & 0 & 1 & 25 & 2 & 1 \\ \hline
Carrots & 0.98 & 2 & 52.5 & 0.3 & 6.1 & 42.2 & 3.6 & 1.2 \\ \hline
Bananas & 0.24 & 1 & 105 & 0.4 & 14.4 & 5.9 & 3.1 & 1.3 \\ \hline
Tortillas & 3.48 & 18 & 140 & 4 & 0 & 0 & 0 & 3 \\ \hline
Cheese & 1.88 & 8 & 110 & 8 & 0 & 191 & 0 & 6 \\ \hline
Yogurt & 3.47 & 5 & 90 & 0 & 7 & 190 & 0 & 17 \\ \hline
Bread & 1.28 & 6 & 120 & 2 & 2 & 60 & 0.01 & 4 \\ \hline
Chicken & 9.76 & 20 & 110 & 3 & 0 & 0 & 0 & 20 \\ \hline
Rice & 8.43 & 40 & 205 & 0.4 & 0.1 & 15.8 & 0.6 & 4.2 \\ \hline
Pasta Sauce & 3.57 & 15 & 60 & 1.5 & 7 & 20 & 2 & 2 \\ \hline
Lettuce & 1.78 & 6 & 8 & 0.1 & 0.6 & 15.5 & 1 & 0.6 \\ \hline
\end{tabular}
\caption{Subset of table containing food data}
\label{tab:food-data}
% \end{adjustwidth}
\end{table}

 According to the FDA\footnote[1]{url{https://www.accessdata.fda.gov/scripts/InteractiveNutritionFactsLabel/pdv.html}} and US Department of Health, someone on a $2000$ calorie diet should have no more than 2000 calories, no more than 65 grams of fat, no more than 50 grams of sugar\footnote[2]{https://www.today.com/health/4-rules-added-sugars-how-calculate-your-daily-limit-t34731}, at least 1000 milligrams of calcium\footnote[1]{26 Sept 2018, \url{https://ods.od.nih.gov/factsheets/Calcium-HealthProfessional/}}, at least 25 grams of fiber, and at least 46 grams of protein\footnote[2]{\url{https://www.accessdata.fda.gov/scripts/InteractiveNutritionFactsLabel/protein.html}} per day.

 We can rewrite this as a convex optimization problem below.

 \begin{align*}
\text{minimize } & \sum_{i=1}^{18}p_ix_i, \\
\text{subject to }& \sum_{i=1}^{18} c_ix_i \leq 2000, \\
			& \sum_{i=1}^{18} f_ix_i \leq 65, \\
			& \sum_{i=1}^{18} \hat{s}_ix_i \leq 50, \\
			& \sum_{i=1}^{18} \hat{c}_ix_i \geq 1000, \\
			& \sum_{i=1}^{18} \hat{f}_ix_i \geq 25, \\
			& \sum_{i=1}^{18} \hat{p}_ix_i \geq 46, \\
			& x_i \geq 0.
\end{align*}

%Problem 6
 \begin{problem}
Read in the file \li{food.npy}.
Use CVXPY to identify how much of each food item a college student should each to minimize cost spent each day given these simplified nutrition requirements.
Return the minimizing vector and the total amount of money spent.

According to this problem, what is the food you should eat most each day?
What are the three foods you should eat most each week?

(Hint: Each nutritional value must be multiplied by the number of servings to get the nutrition value of the whole product).
\label{prob:diet}
\end{problem}

You can learn more about CVXPY at
\url{https://www.cvxpy.org/index.html}.


\chapter{Bayesian Monte Carlo}
Consider the integral
\begin{equation}\label{integralbmc}
Z = \int_{\mathbb{R}^D} f(x) p(x) dx.
\end{equation}
In the following discussion, we drop  $\mathbb{R}^D$ from the integral symbol, for simplicity. If $p(x)$ is a distribution whose sampling is easy, the bottleneck for a simple Monte Carlo method for estimating $Z$ would be the evaluation cost of $f$. If evaluating $f$ is costly then, a naive Monte Carlo method often becomes infeasible. In this section, we present a GP-based method that tries to circumvent this bottleneck.

\section{GP approximation for the integrand}
In Bayesian Monte Carlo (BMC), or Bayesian quadrature \cite{Ghahramani_2003,O_Hagan_1991} \footnote{The original name Bayesian quadrature describes more accurately the method, however the name Bayesian Monte Carlo will be used in this text. At the literature, both names can be found in roughly equal proportion}, $f$ itself is treated as a random function, and a GP prior $GP(m,k)$ is put on $f$. Therefore, given a set $\mathcal{D} = \{(x_i,f(x_i))\}_{i=1}^N$ of $N$ evaluations, the posterior random function $f_{\mathcal{D}}$ is also distributed according to a GP. In particular, since linear maps of GPs are themselves GPs \cite{Rasmussen06,Hennig_2012}, this implies that the random variable
\begin{equation}\label{bmcrv}
Z_{\mathcal{D}} = \int f_{\mathcal{D}}(x) p(x) dx
\end{equation}
is a Gaussian random variable. Since we can find the mean by
\begin{equation}\label{evbmc}
\begin{split}
\Ev[Z_{\mathcal{D}}] = \Ev \left[\int f_{\mathcal{D}}(x) p(x) dx \right] = \int \Ev[f_{\mathcal{D}}(x)] p(x) dx,
\end{split}
\end{equation}
and the variance by,
\begin{equation}\label{varbmc}
\begin{split}
\Var(Z_{\mathcal{D}}) & = \Ev[(Z_{\mathcal{D}} - \Ev[Z_{\mathcal{D}}])^2] \\
& = \Ev \left[\left( \int (f_{\mathcal{D}}(x) - \Ev[f_{\mathcal{D}}(x)]) p(x) dx \right)^2 \right] \\
& = \int \int \Ev \left[(f_{\mathcal{D}}(x) - \Ev[f_{\mathcal{D}}(x)]) (f_{\mathcal{D}}(x') - \Ev[f_{\mathcal{D}}(x')]) \right]  p(x) p(x') dx dx' \\
& = \int \int \Cov(f_{\mathcal{D}}(x),f_{\mathcal{D}}(x')) p(x) p(x') dx dx'.
\end{split}
\end{equation}
We have a complete description of the distribution of $Z_\mathcal{D}$. Now, by substituting \eqref{meancovGPR} in \eqref{evbmc} and \eqref{varbmc}, we have
\begin{equation}\label{evvarbmc2}
\begin{split}
& \Ev[Z_{\mathcal{D}}] = \int m(x) p(x) dx - \mathbf{z}^T K^{-1} (\mathbf{f}-m(\mathbf{x})) \\
& \Var[Z_{\mathcal{D}}] = \Gamma - \mathbf{z}^T K^{-1} \mathbf{z},
\end{split}
\end{equation}
where $\mathbf{z} = (z_1,...,z_N)^T$, with
\begin{equation}\label{zbmcdef}
z_i = \int k(x,x_i) p(x) dx,
\end{equation}
and
\begin{equation}\label{varcoef}
\Gamma = \int \int k(x,x') p(x) p(x') dx dx'.
\end{equation}
In general, a good estimate of $Z_\mathcal{D}$ is its mean, although if there is an asymmetric loss function associated with estimating $Z_\mathcal{D}$, its variance should be taken in account.

An illustration of the Bayesian Monte Carlo approach to integration is shown in Figure \ref{bmcfig}. There, the distribution is $p(x) = \mathcal{N}(x|0,0.5)$, and $f(x) = -x^2$. The true value of the integral, and BMC estimation are shown. Notice that, in this example, for $|x|>2$ the GP estimate $m_\mathcal{D}(x)$ of $f(x)$ becomes very inaccurate. However, since low probability mass is assigned outside the interval, the BMC estimation is still close to the target.

\begin{figure}
	\centering
	\includegraphics[width=0.7\linewidth]{figs/exbmc.png}
	\caption[Illustration of Bayesian Monte Carlo integration]{\label{bmcfig} Illustration of Bayesian Monte Carlo integration. Here, dash-dot black is $f(x)$, whose value is on the left-axis, while in blue is it's GP mean (dark), and covariance (light), given 5 evaluations (blue circles). In red is $p(x)$, whose value is on the right axis. The true value of the integral, the BMC mean and variance are shown on the top right.}
\end{figure}

\subsection{Philosophical remark}
At first one may find strange to consider $f$ as a random function, in order to give a prior for it. After all, $f$ is a known, fixed function. However, notice that $f$ is only actually known in so far as one can evaluate it, and if evaluations for $f$ are limited, so is the knowledge of it. And by the discussion in Chapter 1, any object the learner has limited knowledge about should be considered a random variable, independent of the fact that the object is actually random or not.

If this philosophical approach is not convincing, maybe it is better to just think of the BMC method as an artificial way to integrate functions, and follow the famous dictum in quantum mechanics: "Shut up and calculate" \cite{Mermin_2004}.

\section{Kernel-distribution combinations}\label{kerneldistributionbmc}
In general, neither \eqref{zbmcdef} nor \eqref{varcoef} are analytically available, just like the original integral \eqref{integralbmc}. However, evaluating the kernel function is in general cheap, so if evaluation of $f$ is expensive, there may be still computational gains in using BMC. For some particular distributions $p(x)$, combined with some suitable kernel choices, \eqref{zbmcdef} nor \eqref{varcoef} does lend analytical solutions, or can be treated in a relatively cheap manner. In the following, $m(x) = 0$ for simplicity, thus omitting the term.

\subsection{SQE kernel with Gaussian distributions}
Assume $p(x) = \mathcal{N}(x|\mu,\Sigma)$, and $k$ is a anisotropic squared exponential kernel, with vertical scale $\theta$ and length scales $l = (l_1,...,l_D)$. Notice then that, by letting $A := \text{diag}(l_1^2,...,l_D^2)$, we have
\begin{equation} \label{rbfasnormal}
\begin{split}
k(x,x') = \theta \exp \left( -\frac{1}{2} (x - x')^T A^{-1} (x - x') \right) = & \det(2 \pi A) \mathcal{N}(x'|x,A) = \\
& \det(2 \pi A) \mathcal{N}(x|x',A).
\end{split}
\end{equation}
Then, by \eqref{productgaussians}:
\begin{equation} \label{kpexpansion1}
\begin{split}
& k(x,x') \mathcal{N}(x|\mu,\Sigma) = C(x') \mathcal{N}(x|\tilde{\mu}_{x'},\tilde{\Sigma}) \\
& C(x') = \frac{\theta}{\det(I + A^{-1} \Sigma)^{1/2}} \exp \left(-\frac{1}{2}(x' - \mu)^T (A + \Sigma)^{-1}(x' - \mu) \right) \\
& \tilde{\mu}_{x'} = (A^{-1} + \Sigma^{-1})^{-1} (A^{-1} x' + \Sigma^{-1} \mu) \\
& \tilde{\Sigma} = (A^{-1} + \Sigma^{-1})^{-1},
\end{split}
\end{equation}
and
\begin{equation} \label{kpexpansion2}
\begin{split}
& k(x,x') \mathcal{N}(x|\mu,\Sigma) \mathcal{N}(x'|\mu,\Sigma) = \hat{C} \mathcal{N}(x|\tilde{\mu}_{x'},\tilde{\Sigma}) \mathcal{N}(x'|\tilde{\mu}_{x},\tilde{\Sigma}) \\
& \hat{C} = \frac{\theta}{\det(I + 2 A^{-1} \Sigma)^{1/2}}.
\end{split}
\end{equation}
Hence, by substituting \eqref{kpexpansion1} and \eqref{kpexpansion2} into \eqref{zbmcdef} and \eqref{varcoef}, we find
\begin{equation}\label{termsbmcgaussian}
\begin{split}
& z_i = \frac{\theta}{\det(I + A^{-1} \Sigma)^{1/2}} \exp \left(-\frac{1}{2}(x_i - \mu)^T (A + \Sigma)^{-1}(x_i' - \mu) \right) \\
& \Gamma = \frac{\theta}{\det(I + 2 A^{-1} \Sigma)^{1/2}}.
\end{split}
\end{equation}
Substituting in \eqref{evvarbmc2} we find the desired result:
\begin{equation}\label{evvarbmcgaussian}
\begin{split}
& \Ev[Z_{\mathcal{D}}] = \mathbf{z}^T K^{-1} \mathbf{f} \\
& \Var[Z_{\mathcal{D}}] = \frac{\theta}{\det(I + 2 A^{-1} \Sigma)^{1/2}} - \mathbf{z}^T K^{-1} \mathbf{z} \\
& z_i = \frac{\theta}{\det(I + A^{-1} \Sigma)^{1/2}} \exp \left(-\frac{1}{2}(x_i - \mu)^T (A + \Sigma)^{-1}(x_i' - \mu) \right) \\
& \Gamma = \frac{\theta}{\det(I + 2 A^{-1} \Sigma)^{1/2}}.
\end{split}
\end{equation}

\subsection{Mixture distributions}\label{bmcmixtures}
Consider taking expectations in respect to a mixture distribution
\begin{equation}
 p(x) = \sum_{i=1}^M \alpha_i p_i(x).
\end{equation}
From \eqref{zbmcdef} and \eqref{varcoef}, it is straightforward to see that in this case
\begin{equation}
\begin{split}
 z_i & = \sum_{i=1}^M \alpha_i \int k(x,x_i) p_i(x) dx \\
 \Gamma &= \sum_{i=1}^M \sum_{j=1}^M \alpha_i \alpha_j \int \int k(x,x') p_i(x) p_j(x') dx dx'.
\end{split}
\end{equation}
Then, provided it is possible to calculate the integrals above for each component $p_i(x)$, one can easily use mixture distributions from these coefficients.

An important case is considering mixtures of normal distributions $p(x) = \sum_{j=1}^{M} \alpha_j \mathcal{N}(x|\mu_j,\Sigma_j)$, and the squared-exponential kernel. Then, by substituting in \eqref{zbmcdef} and \eqref{varcoef}, and considering the results for normal distributions in \eqref{termsbmcgaussian}, we find
\begin{equation}\label{bmcmixgaussians}
\begin{split}
 & z_i = \sum_{j=1}^M \alpha_j z_{i,j} \\
 & z_{i,j} = \frac{\theta}{\det(I + A^{-1} \Sigma_j)^{1/2}} \exp \left(-\frac{1}{2}(x_i - \mu_j)^T (A + \Sigma_j)^{-1}(x_i' - \mu_j) \right) \\
 & \Gamma = \sum_{j=1}^M \sum_{m=1}^M \alpha_j \alpha_m \Gamma_{j,m}, \\
 & \Gamma_{j,m} = \frac{\theta}{\det(I + A^{-1} (\Sigma_j+\Sigma_m))^{1/2}} \exp \left(-\frac{1}{2}(\mu_j-\mu_m)^T (A + \Sigma_j + \Sigma_m)^{-1}(\mu_j-\mu_m)\right).
\end{split}
\end{equation}

\subsection{Tensor product kernels and diagonal-covariance Gaussian distributions}\label{tensorprodbmc}
Consider tensor product kernels in $\mathbb{R}^D$ of the form
\begin{equation}
 k(x,x') = \prod_{d=1}^D k_d(x_d,x'_d),
\end{equation}
and a Gaussian distribution $p(x) = \mathcal{N}(x|\mu,\Sigma)$ with diagonal covariance $\Sigma = \text{diag}(\sigma_1^2,\ldots,\sigma_D^{d})$, so that $p(x) = \prod_{d=1}^D \mathcal{N}(x_d|\mu_d,\sigma_d^2)$. Then,
\begin{equation}\label{prodkernelz}
\begin{split}
 z_i & = \int \prod_{d=1}^D k_d(x_d,x_{i,d}) \prod_{d=1}^D \mathcal{N}(x_d|\mu_d,\sigma_d^2) dx \\
 & = \prod_{d=1}^D \int k_d(x,x_{i,d}) \mathcal{N}(x|\mu_d,\sigma_d^2) dx,
\end{split}
\end{equation}
and
\begin{equation}\label{prodkernelgamma}
\begin{split}
 \Gamma & = \int \int \prod_{d=1}^D k_d(x_d,x'_d) \prod_{d=1}^D \mathcal{N}(x_d|\mu_d,\sigma_d^2) \prod_{d=1}^D \mathcal{N}(x'_d|\mu_d,\sigma_d^2) dx dx' = \\
 & = \prod_{d=1}^D \int \int k_d(x,x')\mathcal{N}(x|\mu_d,\sigma_d^2)\mathcal{N}(x'|\mu_d,\sigma_d^2) dx dx'.
\end{split}
\end{equation}
Consider each individual component in \eqref{prodkernelz}. Even with none of then being analytically computable, they can easily be approximated using Gauss-Hermite quadrature, that approximates
\begin{equation}\label{gausshermite}
\int f(x) \mathcal{N}(x|\mu,\sigma^2) dx \approx \frac{1}{\sqrt{\pi}} \sum_{k=1}^K w_k f(\sqrt{2} \sigma x_k + \mu),
\end{equation}
where $x_i$ are the roots of the physicists' Hermite polynomial
\begin{equation}
 H_K(x) = (-1)^N e^{x^2} \frac{d^K}{dx^K} e^{-x^2},
\end{equation}
and $w_k$ are the associated weights
\begin{equation}
 w_k = \frac{2^{K-1} K! \sqrt{\pi}}{K^2 (H_{K-1}(x_k))^2}.
\end{equation}
An analysis of this approximation can be found in \cite{Liu_1994}. It is important to notice that for each $K$, $\{(x_k,w_k)\}_{k=1}^K$ are fixed, unlike standard Monte Carlo methods that would require sampling from $\mathcal{N}(x|\mu,\sigma^2)$. Applying \eqref{gausshermite} to \eqref{prodkernelz}, one finds that
\begin{equation}
 z_i \approx \prod_{d=1}^D \frac{1}{\sqrt{\pi}}\sum_{k=1}^K w_k k_d(\sqrt{2} \sigma_d x_k + \mu_d,x_{i,d}),
\end{equation}
and that
\begin{equation}
\Gamma \approx \prod_{d=1}^D \frac{1}{\pi} \sum_{k,k'=1}^K w_k w_{k'} k_d(\sqrt{2} \sigma_d x_k + \mu_d,\sqrt{2} \sigma_d x_{k'} + \mu_d).
\end{equation}

By the discussion in section \ref{bmcmixtures}, one can easily extend this to mixtures of Gaussians with diagonal covariance. Therefore, one is free to use more flexible kernels than the squared exponential one, provided they are tensor product kernels. The trade-off is that $p(x)$ is a more restrictive distribution, but, as discussed next, this is not an insurmountable restriction. Moreover, the techniques presented in chapters 4 and 5 uses exactly those kinds of distributions as approximations.

\subsection{Importance reweighting for Bayesian Monte Carlo}
One can use the above results for doing integral estimations for general distributions using squared exponential kernels, without Monte Carlo integration of kernels, by an importance re-weighting trick
\begin{equation}
\int f(x) p(x) dx = \int \frac{f(x) p(x)}{q(x)} q(x) dx,
\end{equation}
where $q(x)$ may be either a normal distribution or a mixture of normals. This becomes interesting because since mixtures of normals can approximate continuous densities arbitrarily close as the number of mixtures goes to infinity \cite{Epanechnikov_1969}, thus providing good re-weighting distributions.

\section{Bayesian Monte Carlo for positive integrands}\label{positivebmc}
For many cases the integral we are interested is one arising from marginalization:
\begin{equation}
 p(\mathcal{D}) = \int L(x) p(x) dx = \int p(\mathcal{D}|x) p(x) dx.
\end{equation}
In particular, in this case $L(x)$ must be a positive function. However, applying the BMC method naively can result in rather inaccurate evaluations, due to the fact that in general, GP regression can predict negative means even when all function evaluations are positive. This way, the positivity of the GP mean for $L(x)$ is not guaranteed, resulting in possibly pathological predictions \cite{Ghahramani_2003}.

In \cite{Osborne_2012}, it is proposed to make a GP regression by putting an prior over $\log L(x) \sim GP(m,k)$. However, this means that for predictive distribution in the original space, $L(x|x_{D}) \sim \text{Lognormal}(m_\mathcal{D}(x),k_\mathcal{D}(x,x))$, which results in
\begin{equation}\label{lntranswrong}
 \Ev \left[ \int L(x|x_\mathcal{D}) p(x) dx \right] =
 \int e^{m_\mathcal{D}(x)+\frac{1}{2}k_\mathcal{D}(x,x)} p(x) dx,
\end{equation}
which is still non-tractable integral, requiring further approximations. It is proposed in \cite{OHagan_1992} a somehow complicated heuristic to circumvent this problem, relying in a number of approximations whose accuracy is questionable, and an inner application of BMC.

In \cite{Gunter_2014}, the transformation used is the square-root transformation, so the prior for GP regression is over $\tilde{L}(x) = \sqrt{2 L(x) - \alpha}$, with $\alpha$ being a small positive scalar, resulting in $L(x|x_\mathcal{D}) = \alpha + \frac{1}{2} \tilde{L}(x|x_\mathcal{D})^2$. However, this way we have $\Ev[L(x|x_\mathcal{D})] = \alpha + \frac{k_\mathcal{D}(x,x)}{2}(1+m_\mathcal{D}(x)^2)$, hence, just like \eqref{lntranswrong}, we can't arrive at a tractable mean. In order to circunvent this, in \cite{Gunter_2014} two approaches are proposed. The first is a linearization of $\alpha + \frac{1}{2} \tilde{L}(x|x_\mathcal{D})^2$ around $m_\mathcal{D}(x)$, resulting in
\begin{equation}
 L(x|x_\mathcal{D}) \approx L^{\mathcal{L}}(x|x_\mathcal{D}) = \alpha - \frac{1}{2} m_\mathcal{D}(x)^2 + m_\mathcal{D}(x) \tilde{L}(x),
\end{equation}
which, since it is a affine transformation of $\tilde{L}(x)$, results an approximate GP distribution for $L(\cdot|x_\mathcal{D})$:
\begin{equation}\label{sqlinearization}
\begin{split}
 & L^{\mathcal{L}}(\cdot|x_\mathcal{D}) \sim GP(m_\mathcal{D}^\mathcal{L}(x),k_\mathcal{D}^\mathcal{L}(x)) \\
 & m_\mathcal{D}^\mathcal{L}(x) = \alpha + \frac{1}{2} m_\mathcal{D}(x) \\
 & k_\mathcal{D}^\mathcal{L}(x,x') = m_\mathcal{D}(x) k_\mathcal{D}(x,x') m_\mathcal{D}(x).
\end{split}
\end{equation}
The second proposal is to approximate $L(\cdot|x_\mathcal{D})$ by a random GP-distributed function $L^\mathcal{M}(\cdot|x_\mathcal{D}) = GP(m^\mathcal{M}_\mathcal{D},k^\mathcal{M}_\mathcal{D})$, where $m^\mathcal{M}_\mathcal{D}(x)$ and $k^\mathcal{M}_\mathcal{D}(x)$ are chosen so that $L^\mathcal{M}(\cdot|x_\mathcal{D})$ is moment-matched with $L(\cdot|x_\mathcal{D})$, that is,
 $m^\mathcal{M}_\mathcal{D}(x) = \Ev[L(x|x_\mathcal{D})]$ and $k^\mathcal{M}_\mathcal{D}(x,x') = \Cov(L(x|x_\mathcal{D}),L(x'|x_\mathcal{D}))$. This results in the approximation
 \begin{equation}\label{sqmomentmatched}
  \begin{split}
  & L^{\mathcal{M}}(\cdot|x_\mathcal{D}) \sim GP(m_\mathcal{D}^\mathcal{M}(x),k_\mathcal{D}^\mathcal{M}(x)) \\
  & m_\mathcal{D}^\mathcal{M}(x) = \alpha + \frac{1}{2} (m_\mathcal{D}(x) + k_\mathcal{D}(x,x)) \\
  & k_\mathcal{D}^\mathcal{M}(x,x') = \frac{1}{2} k_\mathcal{D}(x,x')^2 + m_\mathcal{D}(x) k_\mathcal{D}(x,x') m_\mathcal{D}(x).
  \end{split}
 \end{equation}
In particular, both approaches results in tractable integrals for gaussian distributions and SQE kernels.

The idea of moment-matching is extended to a general setting in \cite{Chai_2019}, where it is considered the integral \eqref{integralbmc}, where $f$ is now a function from $\mathbb{R}^D$ to a strict subset $\mathcal{Y}$ of $\mathbb{R}$. In the case presented previously, $\mathcal{Y} = (0,\infty)$, while another important case is $\mathcal{Y} = (0,1)$. Considering an bijective map $\epsilon : \mathbb{R} \to \mathcal{Y}$, it is placed a GP prior $GP(m,k)$ over $g = \epsilon^{-1} \circ f$. Then, the posterior distribution for $f(\cdot|x_\mathcal{D}$ is approximated by a moment-matched GP with mean
$m^\mathcal{M}_\mathcal{D}(x) = \Ev[\epsilon(g(x|x_\mathcal{D}))]$ and $k^\mathcal{M}_\mathcal{D}(x,x') =
\Cov(\epsilon(g(x|x_\mathcal{D})),\epsilon(g(x'|x_\mathcal{D})))$. In particular, for $\epsilon^{-1}(x) = \log(x)$, the same warping considered in \cite{Osborne_2012}, the moment matched mean and covariance becomes
\begin{equation}\label{logmomentmatched}
\begin{split}
 & m^\mathcal{M}_\mathcal{D}(x) = e^{m_\mathcal{D}(x) + \frac{1}{2}k_\mathcal{D}(x,x)}\\
 & k^\mathcal{M}_\mathcal{D}(x,x') =e^{m_\mathcal{D}(x) + \frac{1}{2}k_\mathcal{D}(x,x)} e^{m_\mathcal{D}(x') + \frac{1}{2}k_\mathcal{D}(x',x')} \left(e^{k_\mathcal{D}(x,x')}-1\right).
\end{split}
\end{equation}

However, when integrated this GP does not result in a integrable mean or variance. One further proposal of \cite{Chai_2019} is to do a Taylor expansion of $m^\mathcal{M}_\mathcal{D}(x)$,
\begin{equation}\label{taylormomentmatched}
 m^\mathcal{M}_\mathcal{D}(x) \approx 1 + m_\mathcal{D}(x) + \frac{1}{2} k_\mathcal{D}(x,x) + \frac{1}{2}\left(m_\mathcal{D}(x) + \frac{1}{2} k_\mathcal{D}(x,x)\right)^2 + \ldots,
\end{equation}
and of $k^\mathcal{M}_\mathcal{D}(x,x')$,
\begin{equation}
\begin{split}
 k^\mathcal{M}_\mathcal{D}(x,x')  \approx 1 & + k_\mathcal{D}(x,x') + \frac{1}{2} k_\mathcal{D}(x,x')^2 \\ & +
 k_\mathcal{D}(x,x') \left(m_\mathcal{D}(x) + \frac{1}{2} k_\mathcal{D}(x,x) + m_\mathcal{D}(x') + \frac{1}{2} k_\mathcal{D}(x',x')\right)^2 + \ldots\end{split},
\end{equation}
which, depending on the mean function and kernel function (for instance, zero mean and SQE kernel), is integrable when truncated \cite{Chai_2019}.

The use of function warping raises a question on how to choose the GP hyperparameters. In \cite{Chai_2019}, it is argued that for the moment-matched procedure, one should choose the hyperparameters considering the approximated GP distribution for $f$,  $GP(m^\mathcal{M},k^\mathcal{M}(x,x'))$ and the data $(\mathbf{x},f(\mathbf{x}))$, as opposed to the exact GP distribution for $g$, where it is considered $GP(m,k)$ and the data $(\mathbf{x},g(\mathbf{x}))$. The authors of \cite{Chai_2019} call the first approach \textit{$f$-space} optimization, and the second one \textit{$g$-space} optimization, and it is argued that empirically, $f$-space optimization results in better integral estimations for their test cases. \footnote{Tests with $f$-space optimization in this work weren't successful.}

\section{Choosing evaluation points}

The integral estimate variance $\Var[Z_{\mathcal{D}}]$ yields a natural metric for choosing a set of evaluation points $X_\mathcal{D} = \{x_1,...,x_N\}$. Namely, since $\Var[Z_\mathcal{D}]$ does not depend on the evaluation values $\{f(x_1),\ldots,f(x_N)\}$, one can, given a budget of $N$ evaluations, minimize the function $\alpha_{\text{OBQ}}:\mathbb{R}^{dN} \to \mathbb{R}$ given by
\begin{equation}\label{optimal_bmc}
\alpha_{\text{OBQ}}(X_\mathcal{D}) =  \Var[Z_{\mathcal{D}}](X_\mathcal{D}) = \Gamma(X_\mathcal{D}) - \mathbf{z}(X_\mathcal{D})^T K(X_\mathcal{D})^{-1} \mathbf{z}(X_\mathcal{D})
\end{equation}
However, the evaluation of this function has a $\mathcal{O}(N^3)$ cost, due to the need for matrix inversion, is not necessarily convex, and is defined on a very high dimensional space, so its minimization will be feasible only in specific cases. An easier way would be to take a greedy approach, that is, given $X_{\mathcal{D}_{m}} = \{x_1,...,x_m\}$ previously chosen evaluation points, with $m < n$, we choose $x_{m+1}$ such that the variance of $Z_{\mathcal{D}_m \cup \{x_{m+1},f(x_{m+1})\}}$ is minimized, that is, we are looking to minimize the objective function
\begin{equation}
\alpha^m_{\text{SBQ}}(x_{m+1}) = \alpha^m_\text{OBQ}(x_{m+1};x_1,\ldots,x_m)
\end{equation}

\begin{equation}\label{sequential_bmc}
\alpha^m_{SBQ} : \mathbb{R}^d \to \mathbb{R} , \quad x_{m+1} \to \Var[Z_{\mathcal{D}^m \cup \{x_{m+1},f(x_{m+1})\}}](x_{m+1}),
\end{equation}
This algorithm is referred as Sequential Bayesian Quadrature (SBQ) in \cite{Briol_2015}, while in the same work the first objective is referred as Optimal Bayesian Quadrature (OBQ). Notice that the kernel matrix Cholesky decomposition can be updated in $\mathcal{O}(N^2)$ by the discussion in Section \ref{onlinelearningsection}, and that this operation is differentiable, since the Cholesky decomposition is differentiable \cite{Smith_1995,Murray_2016}.

One can simplify this objective function, by substituting it for an heuristic of finding the point of maximum variance of the integrand, that is, substituting
\begin{equation}
\argmin_{x_{m+1}} \int f_{\mathcal{D} \cup \{x_{m+1}\}}(x) p(x)
\end{equation}
for
\begin{equation}
\argmax_{x_{m+1}} \Var[f_\mathcal{D}(x_{m+1}) p(x_{m+1})] = \argmax_{x_{m+1}} k_\mathcal{D}(x_{m+1},x_{m+1}) p(x)^2
\end{equation}
This maximization objective
\begin{equation}\label{us_bmc}
\alpha^m_{\text{US}}(x_{m+1}) = k_\mathcal{D}(x_{m+1},x_{m+1}) p(x)^2 ,
\end{equation}
in referred as \textit{uncertainty sampling} (US).

In the approaches above, since the variance of $f_\mathcal{D}(x)$ and $\mathcal{Z}_\mathcal{D}$ is independent of evaluation values, there is no active selection for evaluation points, and in principle one can choose then beforehand. This is not true anymore if one optimizes the GP hyperparameters as the evaluation points are selected, or if one considers one of the approaches to treat positive integrands, as shown in previous sections. However, the heuristic motivating the maximization of \eqref{us_bmc} can be extended for warped approaches, as proposed in \cite{Gunter_2014}. For the model \eqref{sqlinearization}, this returns the maximization objective \footnote{In these functions, WSABI refers to \textit{warped sequential active Bayesian integration}, with the letter L standing for linearization and M for moment-matched, and MMLT stands for moment-matched log transform}
\begin{equation}
 \alpha^m_{\text{WSABI-L}}(x_{m+1}) = k_{\mathcal{D}^m}^\mathcal{L}(x,x) p(x)^2 = k_{\mathcal{D}^m}(x,x) m_{\mathcal{D}^m}(x)^2 p(x)^2,
\end{equation}
while for \eqref{sqmomentmatched},
\begin{equation}
\alpha^m_{\text{WSABI-M}}(x_{m+1}) = k_{\mathcal{D}^m}^\mathcal{M}(x,x) p(x)^2 = \left(k_{\mathcal{D}^m}(x,x)^2 k_{\mathcal{D}^m}(x,x) m_{\mathcal{D}^m}(x)^2\right) p(x)^2.
\end{equation}
Finally, for the model \eqref{logmomentmatched} proposed in \cite{Chai_2019}, we arrive at:
\begin{equation}\label{mmlt1}
\alpha^m_{\text{MMLT}_1}(x_{m+1}) = e^{2 m_\mathcal{D}(x) + k_\mathcal{D}(x,x)} \left(e^{k_\mathcal{D}(x,x')}-1\right)p(x)^2.
\end{equation}

Recent work \cite{Kanagawa_2019} analyzes jointly those methods under the umbrella term \textit{adaptive Bayesian quadrature}, where their convergence rates are studied. In particular, there it is considered a version of \eqref{mmlt1} without the $p(x)$ term, resulting in the maximization objective
\begin{equation}\label{mmlt2}
\alpha^m_{\text{MMLT}_2}(x_{m+1}) = e^{2 m_\mathcal{D}(x) + k_\mathcal{D}(x,x)} \left(e^{k_\mathcal{D}(x,x')}-1\right).
\end{equation}

In the spirit of Bayesian optimization (discussed below), the maximization objectives presented, and others will be called \textit{acquisition functions}, nomenclature that implies they are criteria for acquiring new information.

\section{Bayesian Monte Carlo and Bayesian Optimization}
Bayesian Monte Carlo belongs to the general family of surrogate model or response surface methods, that are popular in engineering \cite{Booker_1999,Jones_2001,Asher_2015}. Namely, surrogate models tries to approximate a function $f$ of hard evaluation by a function $\hat{f}$ that is easy to evaluate, and try to work with it. Gaussian processes are popular as surrogate models \footnote{In engineering literature they are often called kriging}, since they offer a measure of uncertainty, are flexible, and since the original function evaluation is hard, relatively few data will be available, which mitigates the scaling problem of GPs.

In particular, an important GP-based surrogate method is Bayesian optimization \cite{Shahriari_2016,Brochu_2010,Snoek_2012}, which tries to optimize an expensive function, usually without gradient information. The idea is, from function evaluations $\mathcal{D}_N = \{(x_i,f(x_i)\}_{i=1}^N$, to construct a GP model $\hat{f}_N$, and with it, sequentially use some criteria $\alpha_f$ to choose the $x_{N+1}$ evaluation. Such functions $\alpha_f$ are called acquisition functions in the context. Bayesian optimization in particular is a popular method in machine learning, to tune training parameters of learning algorithms.


\chapter{Circular Motion}

Let's say you tie a 0.16 kg billard ball to a long string and begin to swing
it around in a circle above your head. Let's say the string is 3
meters long, and the ball returns to where it started every 4
seconds. If you start your stopwatch as the the ball crosses the
$x$-axis, the position of the ball at any time $t$ given by:

$$p(t) = [3 \cos{\left( \frac{2 \pi} {4}t\right)}, 3 \sin{ \left( \frac{2 \pi}{4}t\right) }, 2]$$

(This assumes that the ball would be going counter-clockwise if viewed
from above. The spot you are standing on is considered the origin $[0, 0, 0]$.)

Notice that the height is a constant -- 2 meters in this
case. That isn't very interesting, so we will talk just about the the
first two components.  Here is what it would look like from above:

% 3sin = 1.267854785222098
% 3cos = 2.71892336110995
\begin{tikzpicture}[declare function={angle=25;},bullet/.style={inner
    sep=1pt,fill,draw,circle,solid}, scale=1.7]
    % Axis
    \draw[thick,-stealth,black] (-3.2,0)--(3.2,0) node[right] {$x$}; % x axis
    \draw[thick,-stealth,black] (0,-3.2)--(0,3.2) node[left] {$y$}; % y axis
    % Rest
    \draw [dashed, sdkblue] (0,0) circle (3);
    \draw[thick] (0,0) -- (angle:3.0) node [midway, above] {3};
    \draw[sdkblue] (1.05, 0.2) node[right] {$\theta = \frac{2\pi t}{4}\text{ radians}$};
    \draw[-stealth,sdkblue] (1,0) arc (0:angle:1);
    \draw[dashed, black] (2.71892336110995, 1.267854785222098) -- (2.71892336110995, 0)
    node[below] {$3 \cos(\theta)$}; % vertical
    \draw[dashed, black] (2.71892336110995, 1.267854785222098) -- (0, 1.267854785222098)
    node[left] {$3 \sin(\theta)$}; % horizontal
    \filldraw[black] (angle:3.0) circle(4pt);
    \draw[->, thick] (2.71892336110995, 1.267854785222098) --
    (2.71892336110995 - 3 * 0.1267, 1.267854785222098 + 3 * 0.2718);
\end{tikzpicture}

In this case, the radius, $r$, is 3 meters.  The period, $T$ is 4
seconds.  In general, we say that circular motion is given by:

$$p(t) = \left[ r \cos{\frac{2 \pi t}{T}}, r \cos{\frac{2 \pi t}{T}}\right]$$

A common question is ``How fast is it turning right now?''  If you
divide the $2\pi$ radians of a circle by the 4 seconds it takes, you
get the answer ``About 1.57 radians per second.''  This is known as
\newterm{angular velocity} and we typically represent it with the
lowercase Omega: $\omega$. (Yes, it looks a lot like a ``w''.)  To be
precise, in our example, the angular velocity is $\omega = \frac{\pi}{2}$.

Notice that this is different from the question ``How fast is it
going?''  This ball is traveling the circumference of $6\pi \approx
18.85$ meters every 4 seconds.  So the speed of the ball is about
4.71 meters per second.

\section{Velocity}

The velocity of the ball is a vector, and we can find that vector by
differentiating each component of the position vector.

For any constants $a$ and $b$:

\begin{tabular}{c | c }
  Expression & Derivative \\
  \hline
  $a \sin{b t}$ & $ab \cos{b t}$ \\
  $a \cos{b t}$ & $-ab \sin{b t}$  \\
\end{tabular}

Thus, in our example, the velocity of the ball at any time $t$ is given by:

$$v(t) = \left[ -\frac{3 (2\pi)}{4} \sin{\frac{2\pi t}{4}}, \frac{3(2\pi)}{4} \cos{\frac{2\pi t}{4}}, 0 \right]$$

Notice that the velocity vector is perpendicular to the position vector.  It has a constant magnitude.

In general, an object traveling in a circle at a constant speed has the velocity vector:

$$v(t) = \left[ -r\omega \sin{\omega t}, r\omega \cos{\omega t}\right]$$

where $t = 0$ is the time that it crosses the $x$ axis.  If \omega is
negative, that means the motion would be clockwise when viewed from
above.

The magnitude of the velocity vector is $r\omega$.

\begin{tikzpicture}[declare function={angle=25;},bullet/.style={inner
    sep=1pt,fill,draw,circle,solid}, scale=1.7]
    % Axis
    \draw[thick,-stealth,black] (-3.2,0)--(3.2,0) node[right] {$x$}; % x axis
    \draw[thick,-stealth,black] (0,-3.2)--(0,3.2) node[left] {$y$}; % y axis
    % Rest
    \draw [dashed, sdkblue] (0,0) circle (3);
    \filldraw[black] (angle:3.0) circle(4pt);
    \draw[->, thick] (2.71892336110995, 1.267854785222098) --
    (2.71892336110995 - 0.6 * 1.267, 1.267854785222098 + 0.6 * 2.718) node[right]{$v(t)=\left[ -r\omega \sin{\omega t}, r\omega \cos{\omega t}\right]$};
\end{tikzpicture}

\section{Acceleration}

We can get the acceleration by differentiating the components of the velocity vector.

$$a(t) = \left[-r \omega^2 \cos{\omega t}, -r \omega^2 \sin{\omega t} \right]$$

Notice that the acceleration vector points toward the center of the
circle it is traveling on.  That is, when an object is traveling on a
circle at a constant speed, its only acceleration is toward the center
of the circle.

\begin{tikzpicture}[declare function={angle=25;},bullet/.style={inner
    sep=1pt,fill,draw,circle,solid}, scale=1.7]
    % Axis
    \draw[thick,-stealth,black] (-3.2,0)--(3.2,0) node[right] {$x$}; % x axis
    \draw[thick,-stealth,black] (0,-3.2)--(0,3.2) node[left] {$y$}; % y axis
    % Rest
    \draw [dashed, sdkblue] (0,0) circle (3);
    \filldraw[black] (angle:3.0) circle(4pt);
    \draw[->, thick] (2.71892336110995, 1.267854785222098) --
    (2.71892336110995 * 0.2 , 1.267854785222098 * 0.2) node[midway, right]{$a(t) = \left[-r \omega^2 \cos{\omega t}, -r \omega^2 \sin{\omega t} \right]$};
\end{tikzpicture}

The magnitude of the acceleration vector is $r \omega^2$.

\section{Centripetal force}

How hard is the ball pulling against your hand?  That is, if you let
go, the ball would fly in a straight line.  The force you are exerting
on the string is what causes it to accelerate toward the center of the
circle. We call this the \newterm{centripetal force}.

Recall that $F = m a$.  The magnitude of the acceleration is $r
\omega^2 = 3 \left(\frac{2 pi}{4}\right)^2 \approx 7.4$ m/s.  The mass
of the ball is 0.16 kg.  So the force pulling against your hand is
about 1.18 newtons.

The general rule is that when something is traveling in a circle at a
constant speed, the centripetal force needed to keep it traveling in a
circle is:

$$F = m r \omega^2$$

If you know the radius $r$ and the speed $v$ of the the object, here is the rule:

$$F = \frac{m v^2}{r}$$

\begin{Exercise}[title={Circular Motion}, label=circular]
Just as your car rolls onto a circular track with a radius of 200 m,
you realize your 0.4 kg cup of coffee is on the slippery dashboard of your
car.  While driving 120 km/hour, you hold the cup to keep it from sliding.

What is the maximum amount of force you would need to use (The friction of
the dashboard helps you, but the max is when the friction is zero.)

\end{Exercise}
\begin{Answer}[ref=circular]
  $$\frac{120 \text{ km}}{1 hour} = \frac{1000 \text{ m}}{1 \text{ km}}\frac{120 \text{ km}}{1 hour} \frac{1 \text{ hour}}{3600 \text{ seconds}}= 33.3 \text{ m/s}$$

  $$F = \frac{m v^2}{r} = \frac {0.4 (33.3)^2}{200} = 2.2 \text{ newtons}$$
\end{Answer}