\chapter{Methodology}
The methodology used in this research uses a modification of the classical Grey Wolf Optimization algorithm first introduced by Mirjalili, S., Mirjalili, S., and Lewis, A. in 2014 \cite{Mirjalili2014}. As we will be discussing in this chapter, we have determined that using classical GWO as is does not result in usable solutions for the instance of facility layout problem we are solving. Hence, the necessity for the modification.

In this chapter, we will first discuss about the mathematical model of the problem being solved. Later, we will be delving into the inner workings of the solution representation, the algorithm (including the justification for the modification), and then the technologies that were used in implementing the approach.

\section{Mathematical Model}
The goal of any metaheuristic, like what is being proposed in this paper, is to optimize a certain objective function. As mentioned in the first chapter, in facility layout problems, we minimize the following function:

$$
\text{min} F = \sum_{i=1}^{n}\sum_{j=1}^{n}c_{ij}f_{ij}d_{ij}
$$

For the problem we are solving in this paper, we are optimizing the following equation that is not only a slight modification of the basic mathematical model for FLPs, but also adds penalties to solutions that are infeasible, no matter the degree of infeasiblity.

\begin{align*}
	\text{min }F &= \sum_{i=1}^{\left | B \right |}\sum_{j=i + 1}^{\left | B \right |}c_{ij}d_{ij} \\
	& + \sum_{i=1}^{\left | B \right |}\sum_{j=i + 1}^{\left | B \right |} \left ( P_{B}\frac{A_{0}(i, j)}{\text{min}(w_{i}h_{i}, w_{j}h_{j})} + P_{B} \right ) \cdot \alpha_{0}(i, j) \\
	& + \sum_{i=1}^{\left | B \right |}\left( P_{R}\frac{w_{i}h_{i} - A_{1}(i)}{w_{i}h_{i}} + P_{R} \right) \cdot \alpha_{1}(i)
\end{align*}

where:

\begin{table}[h!]
	\centering
	\begin{tabular}{| l | p{10cm} |}
	\hline
	$x_{i}$  & top-left $x$ coordinate of building $i$ \\
	\hline
	$y_{i}$  & top-left $y$ coordinate of building $i$ \\
	\hline
	$w_{i}$  & width of building $i$ \\
	\hline
	$h_{i}$  & height of building $i$ \\
	\hline
	$R_{x}$  & top-left $x$ coordinate of the bounding region \\
	\hline
	$R_{y}$  & top-left $y$ coordinate of the bounding region \\
	\hline
	$R_{w}$  & width of the bounding region \\
	\hline
	$R_{h}$  & height of the bounding region \\
	\hline
	$c_{ij}$ & flow rate from building $i$ to building $j$ \\
	\hline
	$d_{ij}$ & distance from the center of building $i$ to the
	           center of building $j$ \\
	\hline
	$P_{B}$  & penalty value for building intersection \\
	\hline
	$P_{T}$  & penalty value for any building going out of
	           bounds, even with a portion of a building \\
	\hline
	\end{tabular}
\end{table}

We elected to remove the flow rate from the basic formulation of the model that was discussed earlier in Equation \ref{mm-equation-mhc}.

\begin{equation}\label{mm-equation-mhc}
	\sum_{i=1}^{\left | B \right |}\sum_{j=i + 1}^{\left | B \right |}c_{ij}d_{ij}
\end{equation}

This is because we can consider flow rate as simply part of the cost. In the original formulation, we were considering it from a material handling cost perspective, which requires having both a cost and flow rate variable. However, in a general problem, we can consider cost to also include the frequency of movement from one facility to another, which is essentially the flow rate. As such, we can merge cost and flow rate into one variable.

The mathematical model allows for infeasible solutions to allow for better solutions in the long run. To follow this specification, the model includes expressions that penalizes solutions that meet any of the following conditions: (1) at least one building is intersecting with another building, and (2) a building, either in whole or in part, is outside the bounding area.

\begin{equation}\label{mm-equation-intersection}
	\sum_{i=1}^{\left | B \right |}\sum_{j=i + 1}^{\left | B \right |} \left ( P_{B}\frac{A_{0}(i, j)}{\text{min}(w_{i}h_{i}, w_{j}h_{j})} + P_{B} \right ) \cdot \alpha_{0}(i, j)
\end{equation}

Equation \ref{mm-equation-intersection} is the expression that applies a penalty to solutions that meet the first condition. Notice that it has the functions $A_{0}(i, j)$ and $\alpha_{0}(i, j)$. They are defined by the following:

\begin{align}
	A_{0}(i, j) &= I_{L}(x_{i}, x_{j}, w_{i}, w_{j})
	               \cdot I_{L}(y_{i}, y_{j}, h_{i}, h_{j}) \\
	I_{L}(x_{1}, x_{2}, l_{1}, l_{2}) &= \text{max}(0, \text{min}(x_{1} + l_{1}, x_{2} + l_{2}) - \text{max}(x_{1}, x_{2})) \\
    \alpha_{0}(i, j) &=
    \left\{\begin{matrix}
    	1 & \text{if } A_{0}(i, j) > 0 \\
    	0 & \text{otherwise}
    \end{matrix}\right.
\end{align}

$A_{0}(i, j)$ simply gets the area of intersection of buildings $i$ and $j$. This is achieved by the use of $I_{L}(x_{1}, x_{2}, l_{1}, l_{2})$, which computes the length or width of an intersection of buildings.

In the equation, for every pair of buildings that intersect, we apply a penalty that is the percentage of the area of the smallest building by area that is intersecting with the other building multiplied by the penalty value for building intersection. This will allow for rewarding the algorithm for moving the buildings towards non-intersection. The same penalty value is also added to ensure that the algorithm prioritizes removing intersections over reducing the distance between the centers of the buildings. $\alpha_{0}(i, j)$ ensures that the penalty is only applied to pairs of buildings that intersect with one another.

\begin{equation}\label{mm-equation-oob}
	\sum_{i=1}^{\left | B \right |}\left( P_{R}\frac{w_{i}h_{i} - A_{1}(i)}{w_{i}h_{i}} + P_{R} \right) \cdot \alpha_{1}(i)
\end{equation}

The other part of the mathematical model, Equation \ref{mm-equation-oob}, works in a similar principle as Equation \ref{mm-equation-intersection}. This equation applies a penalty value when the second condition of infeasibility is met. Like in \ref{mm-equation-intersection}, it has specific functions to help compute the penalty. They are defined as:

\begin{align}
	A_{1}(i) &= I_{L}(x_{i}, R_{x}, w_{i}, R_{w})
	\cdot I_{L}(y_{i}, R_{y}, h_{i}, R_{h}) \\
	\alpha_{1}(i) &=
	\left\{
	\begin{matrix}
		0 & \text{if} &
		\begin{aligned}
			R_{x} &\leq x_{i} &\leq R_{x} + R_{w} \\
			R_{x} &\leq x_{i} + w_{i} &\leq R_{x} + R_{w} \\
			R_{y} &\leq y_{i} &\leq R_{y} + R_{h} \\
			R_{y} &\leq y_{i} + h_{i} &\leq R_{y} + R_{h} \\
		\end{aligned} \\
		1 & \text{otherwise}
	\end{matrix}\right.
\end{align}

$A_{1}(i)$ simply computes the area of intersection of the building and the bounding area. Now, since this only computes the intersection, we must subtract the intersection with the area of the building to get the area of the building that is outside of the bounding area. This is expressed by the numerator of the fractional expression in Equation \ref{mm-equation-oob}. Similar to Equation \ref{mm-equation-intersection}, the equation applies a penalty value that is the percentage of the area of the total building area that is outside the bounding region multiplied and then added by the penalty value. The addition is also to ensure that the algorithm gives more priority to removing out-of-bounds buildings. $\alpha_{1}(i)$ ensures that the penalty is only applied to buildings that are, in part or in whole, out of bounds.

\section{Solution Representation}
The solution is represented using a one-dimensional array of floating numbers. In the array, every group of three consecutive elements are considered to be the x and y positions, and angle, respectively, of one building. While the x and y positions are allowed to be of any value, the angle value is restricted to only $0^{\circ}$ and $90^{\circ}$. A visualization of the solution representation is shown by Figure \ref{solution-repr-viz}.

\begin{figure}
	\begin{center}
		\begin{tabular}{| c | c | c | c | c | c | c |}
			\hline
			$x_{0}$ & $y_{0}$ & $\angle_{0}$ & $\ddots$ & $x_{n - 1}$ & $y_{n - 1}$ & $\angle_{n - 1}$ \\
			\hline
		\end{tabular}
	\end{center}
	\caption{Visualization of the solution representation.}
	\label{solution-repr-viz}
\end{figure}

\section{The Algorithm}
In this paper, we are adapting the Grey Wolf Optimization algorithm into solving our instance of the facility layout problem. There have been no publicly available research that have previously used the metaheuristic in solving FLP, basing from our survey. This increases the significance of this paper. As mentioned earlier, the proposed algorithm requires modifications in order to produce feasible solutions. We will first be discussing the reasons why we require them, before proceeding to detailing the algorithm we are using for this research.

\subsection{The Problem with Classical GWO}
In classical GWO, the following equations are used:

\begin{align}
	\vec{X_{1}^{'}} &= \vec{X_{\alpha}}(t) - \vec{A_{\alpha}} \cdot \vec{D_{\alpha}} \label{gwo-x1-eqn} \\
	\vec{X_{2}^{'}} &= \vec{X_{\beta}}(t) - \vec{A_{\beta}} \cdot \vec{D_{\beta}} \\
	\vec{X_{3}^{'}} &= \vec{X_{\delta}}(t) - \vec{A_{\delta}} \cdot \vec{D_{\delta}} \label{gwo-x3-eqn} \\
	\vec{X}(t + 1)  &= \frac{\vec{X_{1}^{'}} + \vec{X_{2}^{'}} + \vec{X_{3}^{'}}}{3} \label{gwo-xt1-eqn}
\end{align}

where $\vec{X_{\alpha}}$, $\vec{X_{\beta}}$, and $\vec{X_{\delta}}$ represent the $\alpha$, $\beta$, and $\delta$ solutions \cite{Gupta2018}. $\vec{D}$ and $\vec{A}$ are defined as:

\begin{align*}
	\vec{D}        &= \left | \vec{C} \cdot \vec{X_{l}}(t)
	- \vec{X}(t) \right | \\
	\vec{C}        &= 2 \cdot \vec{r_{2}} \\
	\vec{A}        &= 2 \cdot \vec{a} \cdot \vec{r_{1}}
					  - \vec{a}                \\
\end{align*}

The aforementioned equations may be usable as is for other problems. However, as we have discovered through our prior experiments, using these equations results in solutions that are infeasible and where the buildings tend to move to the axes of an origin and the origin itself. One example solution with these characteristics is shown in Figure \ref{bad-solution-unmodified-gwo}, where we have set the origin of the buildings to the center of the bounding region.

\begin{figure}[h!]
	\centering
	\includegraphics{./images/chap05-methodology/bad-solution-unmodified-gwo.png}
	\caption{Flowchart detailing the algorithm.}
	\label{bad-solution-unmodified-gwo}
\end{figure}

As one may infer, using the equations above will require setting an origin point for the buildings. Not considering the affinity of building towards the axes, having the origin point at the center or in a certain location in the bounding region restricts the possible locations where the buildings can cluster around. This restriction prevents us from exploring the solution subspace where solutions are feasible but where the cluster point is not the origin. This lead us to solutions that are less ideal. Aside from requiring setting the origin point, buildings moving towards the axes also presents another problem. Basing from our experiments, it prevents us from producing feasible solutions.

This behaviour can be attributed primarily to the formula, $\vec{D} = \left | \vec{C} \cdot \vec{X_{p}}(t) - \vec{X}(t) \right |$. To understand why the aforementioned formula contributes to the behaviour we have discussed earlier, we should understand what the formula means. It is helpful to simply consider that only building is being optimized in understanding the problems. Considering only the alpha solution may also provide better understanding as well.

Let us start with $\vec{C} \cdot \vec{X_{l}}(t)$ from $\vec{D} = \left | \vec{C} \cdot \vec{X_{l}}(t) - \vec{X}(t) \right |$. To simplify our explanation, let $K = \vec{C} \cdot \vec{X_{l}}(t)$. The range of each $i$th element in $K$ will be $[0, 2 \cdot \vec{C}_{l,i}]$. Note that the operation is a dot product, but it is actually pairwise multiplication. This means that $K$ simply scales the x and y positions, and angle of the buildings. Figure \ref{gwo-c-effect} shows a visualization of this effect. Despite the figure only showing the effect with a building's position in the first quadrant, the same effect can be observed with other buildings located in other quadrants. Now, considering the entirety of $\vec{D}$, $\vec{D}$ would mean to be the distance between a building $i$ moved to a different point in the region $S$ (see Figure \ref{gwo-c-effect}) in $\vec{X_{l}}$ and a building $i$ in $\vec{X}(t)$. A visualization for this is provided by Figure \ref{gwo-d-effect}.

\begin{figure}[h!]
	\centering
	\includegraphics[scale=0.45]{./images/chap05-methodology/gwo-c-effect.png}
	\caption{In $K$, $C$ simply scales the x and y positions and angles of buildings. Assuming that the point $B$ represents the x and y positions of a building, the region $S$ is where $B$ may be repositioned based on the values of $C$.}
	\label{gwo-c-effect}
\end{figure}

\begin{figure}[h!]
	\centering
	\includegraphics[scale=0.45]{./images/chap05-methodology/gwo-d-effect.png}
	\caption{A visualization of how $D$ is computed and its inherent meaning.}
	\label{gwo-d-effect}
\end{figure}

Let us also take note, $\vec{A} \cdot \vec{D}$. First, we should take note that $\vec{A} = 2 \cdot \vec{a} \cdot \vec{r_{1}} - \vec{a}$. $a$, as mentioned before, linearly decreases over time. Since $a$ decreases linearly over time, $\vec{A}$ will also decrease over time. This behaviour of $\vec{A}$ would mean that in $\vec{A} \cdot \vec{D}$, $\vec{D}$ will eventually decrease as well. Considering equations \ref{gwo-x1-eqn} to \ref{gwo-x3-eqn}, $\vec{A}$ influences the distance of a building from its counterpart in the leading wolves. This would mean that as the number of iterations increase in a run, buildings will eventually follow the placements of the leading wolves.

Let us now return back to $\vec{C} \cdot \vec{X_{l}}(t)$. Over the course of iterations, this equation will make it difficult for a building to change its position. Around 50\% of the time (due to the fact that $\vec{r_{2}}$ is a \textit{uniform} random vector), the value of $\vec{C}$ will be less than $1$. The position of the buildings will be moved towards the axes. Since $\vec{C}$ is a scaling factor, it will be difficult for a building position to move away from an axis. This affects all solutions, and noting that the leading wolves guide the entire population, the movement towards an axis will be propagated towards the entire population, especially with the fact that the $\vec{A}$ reduces the difference between the leading wolves/solutions and the rest of the solutions as the number of iterations increase in a run. Note that the penalty value for intersection prevents them from overlapping with one another. Buildings that are already on a certain axis will find it practically impossible to move in the direction of the perpendicular axis. Buildings that are on the origin itself will practically cease to move at all. Buildings will still be able to change their orientations, however. The reason for this behaviour of being stuck on an axis is due to the nature of axes themselves, where the value in one or both axes is zero, and to the scaling phenomenom caused by $\vec{C}$. Since $K = \vec{C} \cdot \vec{X_{l}}(t)$ and when a building is near or already on an axis, the x, y, or both x and y positions of a building will barely, if at all, move away from the axes it is currently stuck to, when multiplying with $\vec{C}$. Hence, the behaviour we are noticing.

The aforementioned formula makes the classical GWO inadequate for our problem instance. We are unable to produce feasible nor satisfying results. In order for the grey wolf optimization algorithm to be successfully adapted into solving the facility layout problems, we must introduce a few changes into the algorithm. These changes will be discussed in the next subsection.

\subsection{Modified GWO}
% Draft Note: We should maybe give a name to the modified GWO. Maybe call it "Ballais-Romero GWO variant". HAHAHA.

Mirjalili, S., Mirjalili, S., and Lewis, A. \cite{Mirjalili2014} included a figure similar to Figure \ref{gwo-positioning-update}. It visualizes how a wolf $\omega$ will update its position based on the information provided by the leading wolves.

\begin{figure}[h!]
	\centering
	\includegraphics[scale=0.3]{./images/chap05-methodology/gwo-position-updating.png}
	\caption{Visualization of how wolves in GWO update their positions. An $\omega$ wolf will move towards a random point inside the circle of the estimated prey position.}
	\label{gwo-positioning-update}
\end{figure}

Basing from the visualization, notice that the $\vec{C}$ of the leading wolves specify the radius of the circle in which a $\vec{C} \cdot \vec{X_{l}}$ will be located it. The circle does \textbf{not} include an origin point. We have discussed before that performing a pairwise multiplication between $\vec{C}$ and $\vec{X_{l}}$ simply scales the elements $i$ in the vector $\vec{X_{l}}$. This is different from the visualization. To achieve the same effect as the visualization, instead of performing pairwise multiplication, we must utilize vector addition between $\vec{C}$ and $\vec{X_{l}}$. See Figure \ref{vector-addition-visualization} for a visualization of vector addition. This is the first modification we are introducing to classical GWO.

\begin{figure}[h!]
	\centering
	\includegraphics[scale=0.45]{./images/chap05-methodology/vector-addition-visualization.png}
	\caption{Vector addition pushes the point represented by $\vec{A}$ towards the direction of $\vec{B}$ by the magnitude of the same vector.}
	\label{vector-addition-visualization}
\end{figure}

In our modified GWO, $D$ is now defined as:

\begin{align}
	D &= \left | (\vec{C} + \vec{X_{l}}) - \vec{X}(t) \right | \label{modified-gwo-d}
\end{align}

However, this alone is not enough to comply with the aforementioned visualization. Using this will only move the buildings to the right and/or top. In order for us to move the buildings, we must also modify $\vec{C}$ as shown below:

\begin{align}
	C &= c \cdot \vec{r_{3}} \label{modified-gwo-c}
\end{align}

In this equation, $c$ is a real-valued variable, and $r_{3}$ is a random vector in $[-1, 1]$. This modification will now allow us to move a building from any direction and at any magnitude. The magnitude in which the building will be moved is controlled by $c$.

These modifications remove the necessity to specify an origin point in the bounding region, and the behaviour of buildings to move towards the origin or axes. Unfortunately, this alone is not enough to produce feasible results. We have to add two more modifications before we are able to produce good results.

The first additional modification is the building clamping. Each building is restricted to the boundary. If a building is moved towards outside the boundary, it will be pulled back to within the boundary. Building clamping can be mathematically defined as the following. Given a building $B$ in a solution $X(t)$ at iteration $t$ after being updated by Equations \ref{gwo-x1-eqn} to \ref{gwo-xt1-eqn}, \ref{modified-gwo-d}, and \ref{modified-gwo-c}, clamping can be mathematically modeled as:

\begin{align}
	B_{x} &= \text{max}\left (R_{x} + \frac{B_{w}}{2}, \text{min}\left(B_{x}, R_{x} + \left(R_{w} - \frac{B_{w}}{2}\right)\right)\right ) \label{bx-clamp-equation} \\
	B_{y} &= \text{max}\left (R_{y} + \frac{B_{h}}{2}, \text{min}\left(B_{y}, R_{y} + \left(R_{h} - \frac{B_{h}}{2}\right)\right)\right ) \label{by-clamp-equation}
\end{align}

where $B_{x}$ and $B_{y}$ are the $x$ and $y$ positions of the centroid of a building $B$, $B_{w}$ and $B_{h}$ are the width and height from the top left corner of a building $B$, $R_{x}$ and $R_{y}$ are the $x$ and $y$ positions of the top-left corner of the bounding region $R$, and $R_{w}$ and $R_{h}$ are the width and height of the bounding region $R$. Based on our prior experiments, without this clamping, buildings will freely move to points outside the boundary, and, at the end of the run, will produce a bad solution.

This clamping should \textit{almost} solve the positioning of the buildings and allow us to produce results that are feasible. Since GWO is a continuous metaheuristic, building attributes that must only be one of two values will eventually be a value that is between the two aforementioned values. In our problem, this attribute that is affected is the building orientation. The building orientation may only be $0^{\circ}$ or $90^{\circ}$. It must never be a value between two. To solve this problem, we simply use the orientation of a building $B$ from the $\alpha$, $\beta$, or $\delta$ solutions, which are randomly selected. This idea is based off from the nature of GWO, where the best three solutions lead the search for the local optima. The building orientation of a building $B$ is, therefore, obtained using:

\begin{align}
	B_{o} = \left\{\begin{matrix}
		\alpha_{B_{o}} & \text{if } 0 \leq r < \frac{1}{3} \\
		\beta_{B_{o}}  & \text{if } \frac{1}{3} \leq r < \frac{2}{3}  \\
		\delta_{B_{o}} & \text{otherwise}
	\end{matrix}\right.
	\label{modified-gwo-bo}
\end{align}

where $B_{o}$ is the current orientation of a building $B$, $\alpha_{B_{o}}$, $\beta_{B_{o}}$, and $\delta_{B_{o}}$ are the orientations of building $B$ in the $\alpha$, $\beta$, and $\delta$ solutions, respectively, and $r$ is a random variable in $[0, 1]$. In our approach, assigning the building orientations is performed before clamping the buildings.

With all these modifications already discussed, we now need to briefly discuss how population initialization performed. The population is initialized by providing each building $B$ a random x and y position values, and random orientation. The orientation is either $0$ or $90$. The x and y positions are clamped as well to ensure that the buildings are inside the bounding region. The positions are clamped using Equations \ref{bx-clamp-equation} and \ref{by-clamp-equation}, respectively. Algorithm \ref{modified-gwo-algorithm-pop-initialization} shows the pseudocode for the population initialization.

\begin{algorithm}[h!]
\caption{Pseudocode for the population initialization.}
\label{modified-gwo-algorithm-pop-initialization}
\begin{algorithmic}[1]
\State Set $\vec{X}$ to be the solution.
\State Set $R_{x}$ to be the x position of the top-left corner of the bounding region $R$.
\State Set $R_{y}$ to be the y position of the top-left corner of bounding region $R$.
\State Set $R_{w}$ to be the width of the bounding region $R$.
\State Set $R_{h}$ to be the height of the bounding region $R$.
\State Set $N$ to be the number of buildings in a population.
\For{i = 0 until $N$}
	\State $\vec{X}_{(i * 3)}$ = $U(R_{x}, R_{x} + R_{w})$
	\State $\vec{X}_{(i * 3) + 1}$ = $U(R_{y}, R_{y} + R_{h})$
	\State $\vec{X}_{(i * 3) + 2}$ = $U({0, 90})$

	\State Apply Equation \ref{bx-clamp-equation} to $\vec{X}_{(i * 3)}$.
	\State Apply Equation \ref{by-clamp-equation} to $\vec{X}_{(i * 3) + 1}$.
\EndFor
\end{algorithmic}
\end{algorithm}

All these modifications for the classical GWO have allowed us to successfully adapt GWO to the facility layout problem. Equations \ref{summary-modified-gwo-a} to \ref{summary-modified-gwo-xt1}, and Algorithm \ref{modified-gwo-algorithm} summarises the entire modified GWO. Notice that these modifications are relatively simple, and do not significantly change the characteristics of classical GWO. The simplicity of GWO is still preserved. The next chapters will discuss how our modified version of GWO performs against configurations of unequal-area facility layout ptoblems.

\begin{align}
	\vec{A} &= 2\vec{a} \cdot \vec{r_{1}} - \vec{a} \label{summary-modified-gwo-a} \\
	\vec{C} &= c \cdot \vec{r_{2}} \\
	\vec{D}_{\alpha} &= \left | \left ( \vec{C}_{1} + \vec{X_{\alpha}(t)} \right ) - \vec{X(t)} \right | \label{summary-modified-gwo-DAlpha} \\
	\vec{D}_{\beta} &= \left | \left ( \vec{C}_{2} + \vec{X_{\beta}(t)} \right ) - \vec{X(t)} \right | \\
	\vec{D}_{\delta} &= \left | \left ( \vec{C}_{3} + \vec{X_{\delta}(t)} \right ) - \vec{X(t)} \right | \\
	\vec{X}_{1} &= \vec{X}_{\alpha} - \vec{A}_{1} \cdot \vec{D}_{\alpha} \\
	\vec{X}_{2} &= \vec{X}_{\beta} - \vec{A}_{2} \cdot \vec{D}_{\beta} \\
	\vec{X}_{3} &= \vec{X}_{\delta} - \vec{A}_{3} \cdot \vec{D}_{\delta} \\
	\vec{X}(t + 1) &= \frac{\vec{X}_{1} + \vec{X}_{2} + \vec{X}_{3}}{3} \label{summary-modified-gwo-xt1}
\end{align}

\begin{algorithm}[h!]
\caption{Pseudocode for the proposed modified GWO.}
\label{modified-gwo-algorithm}
\begin{algorithmic}[1]
\State Set $T$ to be the maximum number of iterations.
\State Initialize the grey wolf population $\vec{X}_{i} (i = 1, 2, \ldots, n)$
\State Initialize $c$, and $a = 2$.
\State Calculate the fitness of each wolf.
\State Set $\vec{X}_{\alpha}$ to be the fittest wolf.
\State Set $\vec{X}_{\beta}$ to be the second fittest wolf.
\State Set $\vec{X}_{\delta}$ to be the third fittest wolf.
\While{t $<$ T}
	\For{each wolf $\vec{X}_{i}$}
		\State Initialize $\vec{A}_{1}$, $\vec{A}_{2}$, $\vec{A}_{3}$, $\vec{C}_{1}$, $\vec{C}_{2}$, and $\vec{C}_{3}$.
		\State Update the position of the current wolf $\vec{X}_{i}$ using Equations \ref{summary-modified-gwo-DAlpha} to \ref{summary-modified-gwo-xt1}.
		\State Set the orientation of each building using Equation \ref{modified-gwo-bo}.
		\State Clamp the buildings in $\vec{X}_{i}$ using Equations \ref{bx-clamp-equation} and \ref{by-clamp-equation}.
	\EndFor
	\State Calculate the fitness of each wolf.
	\State Update $\vec{X}_{\alpha}$, $\vec{X}_{\beta}$, and $\vec{X}_{\delta}$.
	\State $a = 2 - \frac{2t}{T}$
	\State $t = t + 1$
\EndWhile \\
\Return $\vec{X}_{\alpha}$
\end{algorithmic}
\end{algorithm}

\section{Implementation Technologies}
Our program was developed using C++17 compiled using the Clang 11 compiler in an elementaryOS Hera environment under release mode with the \texttt{-03} optimized compilation flag turned on. Building was handled by CMake, and package management was handled by Conan. Our implementation is built on top of CoreX, a custom-developed 2D game engine. Using a game engine allowed us to visualize the results and configure experiments in a graphical manner. Using a custom engine over an over-the-shelf engine ensures that the implementation remains light and does not carry unnecessary features that are typically used in commercial game engines. The libraries EASTL, ImGUI, SDL 2, SDL 2 TTF. sdl-gpu, nlohmann JSON, and EnTT were used in developing the engine, with EnTT and ImGUI directly used by our implementation itself.


\lab{Conditioning and Stability}{Conditioning and Stability}
\objective{The condition number of a function measures how sensitive that function is to changes in the input.
On the other hand, the stability of an algorithm measures how accurately that algorithm computes the value of a function from exact input.
Both of these concepts are important for answering the crucial question, ``is my computer telling the truth?''
In this lab, we examine the conditioning of common linear algebra problems, including computing polynomial roots and matrix eigenvalues.
We also present an example to demonstrate how two different algorithms for the same problem may not have the same level of stability.}

\section*{Conditioning} % =====================================================

The \emph{absolute condition number} of a function $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$ at a point $\x\in\mathbb{R}^m$ is defined by
\begin{equation}
\hat{\kappa} (\x) = \lim_{\delta \rightarrow 0^+} \sup_{\norm{\mathbf{h}} < \delta} { \frac{\norm{f(\x + \mathbf{h} ) - f(\x)}}{\norm{\mathbf{h}}} }.
\label{eq:absolute-condition-number}
\end{equation}

In other words, the absolute condition number of $f$ is the limit of the change in output over the change of input.
Similarly, the \emph{relative condition number} of $f$ is the limit of the relative change in output over the relative change in input,
\begin{equation}
\kappa(\x)
= \lim_{\delta \rightarrow 0^+} \sup_{\norm{\mathbf{h}} < \delta} \left({ \frac{\norm{f(\x + \mathbf{h} ) - f(\x)}}{\norm{f(\x)}} } \middle/ { \frac{\norm{\mathbf{h}}}{\norm{\x}} }\right)
= \frac{\norm{\x}}{\norm{f(\x)}} \hat{\kappa}(\x).
\label{eq:relative-condition-number}
\end{equation}

A function with a large condition number is called \emph{ill-conditioned}.
Small changes to the input of an ill-conditioned function may produce large changes in output.
It is important to know if a function is ill-conditioned because floating point representation almost always introduces some input error, and therefore the outputs of ill-conditioned functions cannot be trusted.

The \emph{condition number} of a matrix $A$, $\kappa (A)=\|A\| \|A^{-1}\|$, is an upper bound on the condition number for many of the common problems associated with the matrix, such as solving the system $A\x =\b$.
If $A$ is square but not invertible, then $\kappa(A) = \infty$ by convention.
To compute $\kappa(A)$, we often use the matrix 2-norm, which is the largest singular value $\sigma_{\max}$ of $A$.
Recall that if $\sigma$ is a singular value of $A$, $\frac{1}{\sigma}$ is a singular value of $A^{-1}$.
Thus, we have that
\begin{equation}\label{eq:matrix_cond}
\kappa(A) = \frac{\sigma_{\max}}{\sigma_{\min}},
\end{equation}
which is also a valid equation for non-square matrices.

\begin{warn} % Ill-conditioned matrices.
Ill-conditioned matrices can wreak havoc in even simple applications.
% Solving problems can become difficult or even impossible when working with matrices that have large condition numbers because small changes in input can quickly lead to inaccurate output.
For example, the matrix
\[
A =
\left[\begin{array}{cc}
1 & 1 \\
1 & 1.0000000001
\end{array}\right]
\]
is extremely ill-conditioned, with $\kappa(A) \approx 4\times 10^{10}$.
Solving the systems $A\x = \b_1$ and $A\x = \b_2$ can result in wildly different answers, even when $\b_1$ and $\b_2$ are extremely close.

\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la

>>> A = np.array([[1, 1], [1, 1+1e-10]])
>>> np.linalg.cond(A)
39999991794.058899

# Set up and solve a simple system of equations.
>>> b1 = np.array([2, 2])
>>> x1 = la.solve(A, b1)
>>> print(x1)
[ 2.  0.]

# Solve a system with a very slightly different vector b.
>>> b2 = np.array([2, 2+1e-5])
>>> la.norm(b1 - b2)
>>> x2 = la.solve(A, b2)
>>> print(x2)
[-99997.99172662  99999.99172662]   # This solution is hugely different!
\end{lstlisting}

If you find yourself working with matrices that have large condition numbers, check your math carefully or try to reformulate the problem entirely.
\end{warn}

\begin{info} % Orthonormal matrices are the best!
An \emph{orthonormal matrix} $U$ has orthonormal columns and satisfies $U\trp U = I$ and $\|U\|_2 = 1$.
If $U$ is square, then $U^{-1} = U\trp$ and $U\trp$ is also orthonormal.
Therefore $\kappa(U) = \|U\|_2\|U^{-1}\|_2 = 1$.
Even if $U$ is not square, all of its singular values are equal to $1$, and again $\kappa(U) = \sigma_{\max} / \sigma_{\min} = 1$.

The condition number of a matrix cannot be less than $1$ since $\sigma_{\max} \ge \sigma_{\min}$ by definition.
Thus orthonormal matrices are, in a sense, the best kind of matrices for computations.
This is one of the main reasons why numerical algorithms based on the QR decomposition or the SVD are so important.
\end{info}

\begin{problem} % Compute matrix condition number.
Write a function that accepts a matrix $A$ and computes its condition number using (\ref{eq:matrix_cond}).
Use \li{scipy.linalg.svd()}, or \li{scipy.linalg.svdvals()} to compute the singular values of $A$.
Avoid computing $A^{-1}$.
If the smallest singular value is $0$, return $\infty$ (\li{np.inf}).

Validate your function by comparing it to \li{np.linalg.cond()}.
Check that orthonormal matrices have a condition number of $1$ (use \li{scipy.linalg.qr()} to generate an orthonormal matrix) and that singular matrices have a condition number of $\infty$ according to your function.
\end{problem}

% For large matrices where taking the SVD is difficult, the exact condition number of a matrix cannot always be computed and therefore must be estimated.
% Although not covered here, there exist many algorithms that can efficiently and accurately estimate the condition number of a matrix.

\subsection*{The Wilkinson Polynomial} % --------------------------------------

Let $f:\mathbb{C}^{n+1} \rightarrow \mathbb{C}^n$ be the function that maps a collection of $n+1$ coefficients $(c_n, c_{n-1}, \ldots, c_0)$ to the $n$ roots of the polynomial $c_nx^n+c_{n-1}x^{n-1}+\ldots+c_2x^2+c_1x+c_0$.
Finding polynomial roots is an extremely ill-conditioned problem in general, so the condition number of $f$ is likely very large.
To see this, consider the \emph{Wilkinson polynomial}, made famous by James H. Wilkinson in 1963:
\[
w(x) = \prod_{r=1}^{20}(x-r) = x^{20} - 210x^{19} + 20615x^{18} - 1256850x^{17} + \cdots.
\]
Let $\tilde{w}(x)$ be $w(x)$ where the coefficient on $x^{19}$ is very slightly perturbed from $-210$ to $-210.0000001$.
The following code computes and compares the roots of $\tilde{w}(x)$ and $w(x)$ using NumPy and SymPy.

\begin{comment}
\begin{lstlisting}
>>> w_coeffs = np.array([1, -210, 20615, -1256850, 53327946, -1672280820,
                    40171771630, -756111184500, 11310276995381,
                    -135585182899530, 1307535010540395,
                    -10142299865511450, 63030812099294896,
                    -311333643161390640, 1206647803780373360,
                    -3599979517947607200, 8037811822645051776,
                    -12870931245150988800, 13803759753640704000,
                    -8752948036761600000, 2432902008176640000])
>>> w_roots = np.arange(1, 21)
\end{lstlisting}
\end{comment}

\begin{lstlisting}
>>> import sympy as sy
>>> from matplotlib import pyplot as plt

# The roots of w are 1, 2, ..., 20.
>>> w_roots = np.arange(1, 21)

# Get the exact Wilkinson polynomial coefficients using SymPy.
>>> x, i = sy.symbols('x i')
>>> w = sy.poly_from_expr(sy.product(x-i, (i, 1, 20)))[0]
>>> w_coeffs = np.array(w.all_coeffs())
>>> print(w_coeffs[:6])
[1 -210 20615 -1256850 53327946 -1672280820]

# Perturb one of the coefficients very slightly.
>>> h = np.zeros(21)
>>> h[1]=1e-7
>>> new_coeffs = w_coeffs - h
>>> print(new_coeffs[:6])
[1 -210.000000100000 20615 -1256850 53327946 -1672280820]

# Use NumPy to compute the roots of the perturbed polynomial.
>>> new_roots = np.roots(np.poly1d(new_coeffs))
\end{lstlisting}

Figure \ref{fig:wilkinson-perturbed-poly} plots $w(x)$ and $\tilde{w}(x)$ together, and Figure \ref{fig:wilkinon-perturbed-roots} and compares their roots in the complex plane.

\begin{figure}[H] % Original and pertubed Wilkinson polynomial + roots
\captionsetup[subfigure]{justification=justified}
\centering
\begin{subfigure}{.48\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/perturbed_poly.pdf}
    \caption{The original and perturbed Wilkinson polynomials.
    They match for about half of the domain, then differ drastically.}
    \label{fig:wilkinson-perturbed-poly}
\end{subfigure}
\quad
\begin{subfigure}{.48\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/perturbed_roots.pdf}
    \caption{Roots of the original and perturbed Wilkinson polynomials.
    About half of the perturbed roots are complex.}
    \label{fig:wilkinon-perturbed-roots}
\end{subfigure}
\caption{}
\label{fig:wilkinson-polynomial}
\end{figure}

Figure \ref{fig:wilkinson-polynomial} clearly indicates that a very small change in just a single coefficient drastically changes the nature of the polynomial and its roots.
To quantify the difference, estimate the condition numbers (this example uses the $\infty$ norm to compute $\hat{\kappa}$ and $\kappa$).

\begin{lstlisting}
# Sort the roots to ensure that they are in the same order.
>>> w_roots = np.sort(w_roots)
>>> new_roots = np.sort(new_roots)

# Estimate the absolute condition number in the infinity norm.
>>> k = la.norm(new_roots - w_roots, np.inf) / la.norm(h, np.inf)
>>> print(k)
28262391.3304

# Estimate the relative condition number in the infinity norm.
>>> k * la.norm(w_coeffs, np.inf) / la.norm(w_roots, np.inf)
1.95063629993970+25                     # This is huge!!
\end{lstlisting}

There are some caveats to this example.
\begin{enumerate}
\item Computing the quotients in (\ref{eq:absolute-condition-number}) and (\ref{eq:relative-condition-number}) for a fixed perturbation $\mathbf{h}$ only approximates the condition number.
The true condition number is the limit of such quotients.
We hope that when $\|\mathbf{h}\|$ is small, a random quotient is at least the same order of magnitude as the limit, but there is no way to be sure.

\item This example assumes that NumPy's root-finding algorithm, \li{np.roots()}, is \emph{stable}, so that the difference between \li{w_roots} and \li{new_roots} is due to the difference in coefficients, and not to problems with \li{np.roots()}.
We will return to this issue in the next section.
\end{enumerate}
Even with these caveats, it is apparent that root finding is a difficult problem to solve correctly.
Always check your math carefully when dealing with polynomial roots.

\begin{problem}
Write a function that carries out the following experiment 100 times.
\begin{enumerate}
\item Randomly perturb the true coefficients of the Wilkinson polynomial by replacing each coefficient $c_i$ with $c_i*r_i$, where $r_i$ is drawn from a normal distribution centered at 1 with standard deviation $10^{-10}$ (use \li{np.random.normal()}).
\item Plot the perturbed roots as small points in the complex plane.
That is, plot the real part of the coefficients on the $x$-axis and the imaginary part on the $y$-axis.
Plot on the same figure in each experiment.
\\(Hint: use a pixel marker, \li{marker=','}, to avoid overcrowding the figure.)
\item Compute the absolute and relative condition numbers with the $\infty$ norm.
\end{enumerate}
Plot the roots of the unperturbed Wilkinson polynomial with the perturbed roots.
Your final plot should resemble Figure \ref{fig:wilkinsonpolynomial_many}.
Finally, return the average computed absolute and relative condition numbers.

\begin{figure}[H]
\includegraphics[width=.7\linewidth]{figures/wilkinson_prob_solution.pdf}
\caption{
This figure replicates Figure 12.1 on p. 93 of \cite{Trefethen1997}.}
\label{fig:wilkinsonpolynomial_many}
\end{figure}

\label{prob:wilkinson-polynomial-roots}
\end{problem}

\subsection*{Calculating Eigenvalues} % ---------------------------------------

Let $f:M_n(\mathbb{C}) \rightarrow \mathbb{C}^n$ be the function that maps an $n \times n$ matrix with complex entries to its $n$ eigenvalues.
This problem is well-conditioned for symmetric matrices, but it can be extremely ill-conditioned for non-symmetric matrices.
Let $A$ be an $n\times n$ matrix and let $\boldsymbol{\lambda}$ be the vector of the $n$ eigenvalues of $A$.
If $\tilde{A} = A + H$ is a pertubation of $A$ and $\tilde{\boldsymbol{\lambda}}$ are its eigenvalues, then the condition numbers of $f$ can be estimated by
\begin{equation}
\hat{\kappa}(A) = \frac{\|\boldsymbol{\lambda} - \tilde{\boldsymbol{\lambda}}\|}{\|H\|},
\qquad
\kappa(A) = \frac{\|A\|}{\|\boldsymbol{\lambda}\|}\hat{\kappa}(A).
\label{eq:eig-condition-numbers}
\end{equation}

\begin{problem}\label{prob:eigenvalue} % Eigenvalue experiments
Write a function that accepts a matrix $A$ and estimates the condition number of the eigenvalue problem using (\ref{eq:eig-condition-numbers}).
For the perturbation $H$, construct a matrix with complex entries where the real and imaginary parts are drawn from normal distributions centered at $0$ with standard deviation $\sigma = 10^{-10}$.
\begin{lstlisting}
reals = np.random.normal(0, 1e-10, A.shape)
imags = np.random.normal(0, 1e-10, A.shape)
H = reals + 1j*imags
\end{lstlisting}
Use \li{scipy.linalg.eig()} or \li{scipy.linalg.eigvals()} to compute the eigenvalues of $A$ and $A+H$, and use the 2-norm for both the vector and matrix norms.
Return the absolute and relative condition numbers.
\label{prob:eig-condit}
\end{problem}

\begin{problem}
Write a function that accepts bounds $[x_{\min},x_{\max},y_{\min},y_{\max}]$ and an integer \li{res}.
Use your function from Problem \ref{prob:eig-condit} to compute the relative condition number of the eigenvalue problem for the $2\times 2$ matrix
\[
\left[\begin{array}{cc}
1 &  x\\
y & 1\end{array}\right]
\]
at every point of an evenly spaced \li{res}$\times$\li{res} grid over the domain $[x_{\min}, x_{\max}]\times [y_{\min}, y_{\max}]$.
Plot these estimated relative condition numbers using \li{plt.pcolormesh()} and the colormap \li{cmap='gray_r'}.
With \li{res=200}, your plot should look similar to the following figure.

\begin{figure}[H]
    \includegraphics[width=.7\linewidth]{figures/eigenvalue_conditioning.png}
\end{figure}

\label{prob:eigenvalue-conditioning-plot}
\end{problem}

Problem \ref{prob:eigenvalue-conditioning-plot} shows that the conditioning of the eigenvalue problem depends heavily on the matrix, and that it is difficult to know a priori how bad the problem will be.
Luckily, most real-world problems requiring eigenvalues are symmetric.
In their book on Numerical Linear Algebra, L. Trefethen and D. Bau III summed up the issue of conditioning and eigenvalues when they stated, ``\emph{if the answer is highly sensitive to perturbations, you have probably asked the wrong question}.''

\section*{Stability} % ========================================================

The \emph{stability} of an algorithm is measured by the error in its output.
Let $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$ be a problem to be solved, as in the previous section, and let $\tilde{f}$ be an actual algorithm for solving the problem.
The \emph{forward error} of $f$ at $\x$ is $||f(\x)-\tilde f(\x)||$, and the \emph{relative forward error} of $f$ at $\x$ is
\[
\frac{||f(\x)-\tilde f(\x)||}{||f(\x)||}.
\]
An algorithm is called \emph{stable} if its relative forward error is small.%
\footnote{See the Additional Material section for alternative (and more rigorous) definitions of algorithmic stability.}

As an example, consider again NumPy's root-finding algorithm that we used to investigate the Wilkinson polynomial.
The exact roots of $w(x)$ are clearly $1,2,\ldots,20$.
Had we not known this, we could have tried computing the roots from the coefficients using \li{np.roots()} (without perturbing the coefficients at all).

\begin{lstlisting}
# w_coeffs holds the coefficients and w_roots holds the true roots.
>>> computed_roots = np.sort(np.roots(np.poly1d(w_coeffs)))
>>> print(computed_roots[:6])       # The computed roots are close to integers.
[ 1.          2.          3.          3.99999999  5.00000076  5.99998749]

# Compute the forward error.
>>> forward_error = la.norm(w_roots - computed_roots)
>>> print(forward_error)
0.020612653126379665

# Compute the relative forward error.
>>> forward_error / la.norm(w_roots)
0.00038476268486104599              # The error is nice and small.
\end{lstlisting}

This analysis suggests that \li{np.roots()} is a stable algorithm, so large condition numbers of Problem \ref{prob:wilkinson-polynomial-roots} really are due to the poor conditioning of the problem, not the way in which the problem was solved.

\begin{info} % Conditioning versus Stability
Conditioning is a property of a \textbf{problem} to be solved, such as finding the roots of a polynomial or calculating eigenvalues.
Stability is a property of an \textbf{algorithm} to solve a problem, such as \li{np.roots()} or \li{scipy.linalg.eig()}.
If a problem is ill-conditioned, any algorithm used to solve that problem may result in suspicious solutions, even if that algorithm is stable.
\end{info}

\subsection*{Least Squares} % ----------------------------------------
% An example of a problem with stable and unstable algorithms is the least squares problem.

The \emph{ordinary least squares} (OLS) problem is to find the $\x$ that minimizes $\norm{A\x -\b}_2$ for fixed $A$ and $\b$.
It can be shown that an equivalent problem is finding the solution of $A\hrm A\x =A\hrm \b$, called the \emph{normal equations}.
% This is repeated in the problem statement (as it should be).
% \begin{enumerate}
% \item Invert the matrix $A\hrm A$ and then right multiply by $A\hrm \b$ to solve the normal equations.
% Although this approach seems intuitive, it is actually highly unstable and can return an answer with a very large forward error.
%
% \item
% Use the QR-decomposition by factoring the $m\times n$ matrix $A$ of rank $n\geq m$ into $QR$, where $Q$ has orthonormal columns and $R$ is upper triangular.
% It can also be shown that the solution of $R \x = Q\hrm\b$ is equivalent to solving the least squares problem.
% This algorithm has the advantage of being stable.
% \end{enumerate}
A common application of least squares is polynomial approximation.
Given a set of $m$ data points $\{(x_k, y_k)\}_{k=1}^m$, the goal is to find the set of coefficients $\{c_i\}_{i=0}^n$ such that
\[
y_k \approx c_n x_k^n + c_{n-1} x_k^{n-1} + \cdots + c_2 x_k^2 + c_1 x_k + c_0
\]
for all $k$, with the smallest possible error.
These $m$ linear equations yield the linear system
\begin{equation}
A\x =
\left[\begin{array}{cccccc}
x_1^n & x_1^{n-1} & \cdots & x_1^2 & x_1 & 1 \\
x_2^n & x_2^{n-1} & \cdots & x_2^2 & x_2 & 1 \\
x_3^n & x_3^{n-1} & \cdots & x_3^2 & x_3 & 1 \\
\vdots & \vdots & & \vdots & \vdots & \vdots \\
x_m^n & x_m^{n-1} & \cdots & x_m^2 & x_m & 1 \\
\end{array}\right]
\left[\begin{array}{c}
c_n \\ c_{n-1} \\ \vdots \\ c_2 \\ c_1 \\ c_0
\end{array}\right]
=
\left[\begin{array}{c} y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_m \end{array}\right]
= \b.
\label{eq:condstab-poly-leastsq}
\end{equation}

\begin{comment} % Too much detail, they can work this part out.
\begin{lstlisting}
# Use least squares to approximate sin(x) with a five-degree polynomial.
>>> x = np.linspace(0, 6, 10)           # The x-values of the data.
>>> b = np.sin(x) + .2*np.random.randn(10) # The y-values of the data (noisy).
>>> A = np.vander(x, 6)                 # Set up the matrix of data values.
>>> coeffs = la.lstsq(A, b)[0]          # Get the polynomial coefficients.

>>> domain = np.linspace(0, 6, 100)     # Define a finer domain for plotting.
>>> plt.plot(x, b, 'k*')				# Plot the approximated points.
>>> plt.plot(domain, np.sin(domain))

# np.poly1d() converts an array of coefficients into a callable polynomial.
>>> plt.plot(domain, np.poly1d(coeffs)(domain))
\end{lstlisting}
\end{comment}

\begin{problem}
Write a function that accepts an integer $n$.
Solve for the coefficients of the polynomial of degree $n$ that best fits the data found in \texttt{stability\_data.npy}.
Use two approaches to get the least squares solution:

\begin{enumerate}
\item Use \li{la.inv()} to solve the normal equations: $\x = (A\trp A)^{-1}A\trp \b$.
Although this approach seems intuitive, it is actually highly unstable and can return an answer with a very large forward error.

\item Use \li{la.qr()} with \li{mode='economic'} and \li{la.solve_triangular()} to solve the system $R\x = Q\trp\b$, which is equivalent to solving the normal equations.
This algorithm has the advantage of being stable.
\end{enumerate}

Load the data and set up the system (\ref{eq:condstab-poly-leastsq}) with the following code.

\begin{lstlisting}
xk, yk = np.load("stability_data.npy").T
A = np.vander(xk, n+1)
\end{lstlisting}

Plot the resulting polynomials together with the raw data points.
Return the forward error $\norm{A\x-\mathbf{b}}_2$ of both approximations.
\\(Hint: The function \li{np.polyval()} will be helpful for plotting the resulting polynomials.)

Test your function using various values of $n$, taking special note of what happens for values of $n$ near $14$ (pictured below).

\begin{figure}[H]
    \centering
    \includegraphics[width=.7\textwidth]{figures/lstsq_stability.pdf}
\end{figure}
\end{problem}

\subsection*{Catastrophic Cancellation} % -------------------------------------

When a computer takes the difference of two very similar numbers, the result is often stored with a small number of significant digits and the tiniest bit of information is lost.
However, these small errors can propagate into large errors later down the line.
This phenomenon is called \emph{catastrophic cancellation}, and is a common cause for numerical instability.

Catastrophic cancellation is a potential problem whenever floats or large integers that are very close to one another are subtracted.
This problem can be avoided by either rewriting the program to not use subtraction, or by increasing the number of significant digits that the computer tracks.

For example, consider the simple problem of computing $\sqrt{a}-\sqrt{b}$.
The computation can be done directly with subtraction, or by performing the equivalent division
\[
\sqrt{a}-\sqrt{b} = (\sqrt{a}-\sqrt{b})\frac{\sqrt{a}+\sqrt{b}}{\sqrt{a}+\sqrt{b}} = \frac{a-b}{\sqrt{a}+\sqrt{b}}.
\]

\begin{lstlisting}
>>> from math import sqrt           # np.sqrt() fails for very large numbers.

>>> a = 10**20 + 1
>>> b = 10**20
>>> sqrt(a) - sqrt(b)               # Do the subtraction directly.
0.0                                 # a != b, so information has been lost.

>>> (a - b) / (sqrt(a) + sqrt(b))   # Use the alternative formulation.
5e-11                               # Much better!
\end{lstlisting}

In this example, $a$ and $b$ are distinct enough that the computer can still tell that $a - b = 1$, but $\sqrt{a}$ and $\sqrt{b}$ are so close to each other that $\sqrt{a} - \sqrt{b}$ is computed as $0$.

\begin{problem}
Let $I(n) = \int_0^1 x^n e^{x - 1} dx$.
It can be shown that for a positive integer $n$,
\begin{equation}
I(n) = (-1)^n(!n - \frac{n!}{e}),
\label{eq:integral-subfactorial-formula}
\end{equation}
where $!n=n!\sum_{k=0}^{n} \frac{(-1)^k}{k!}$ is the \emph{subfactorial} of $n$.
Write a function to do the following.
\begin{enumerate}
\item Use SymPy's \li{sy.integrate()} to evaluate the integral form of $I(n)$ for $n=5,10,\ldots,50$.
Convert the symbolic results of each integration to a float.
Since this is done symbolically, these values can be accepted as the true values of $I(n)$.
\\(Hint: be careful that the values of $n$ in the integrand are of type \li{int}.)

\item Use (\ref{eq:integral-subfactorial-formula}) to compute $I(n)$ for the same values of $n$.
Use \li{sy.subfactorial()} to compute $!n$ and \li{sy.factorial()} to compute $n!$.
\\(Hint: be careful to only pass Python integers to these functions.)
\label{step:subfactorial-cheat}

\item Plot the relative forward error of the results computed in step \ref{step:subfactorial-cheat} at each of the given values of $n$.
Use a log scale on the $y$-axis.
Is (\ref{eq:integral-subfactorial-formula}) a stable way to compute $I(n)$?
Why?
\end{enumerate}
\end{problem}

The examples presented in this lab are just a few of the ways that a mathematical problem can turn into a computational train wreck.
Always use stable algorithms when possible, and remember to check if problems are well conditioned or not.

\begin{comment}
\begin{center}
\begin{tabular}{|l|l|}
\hline
$n$  & Actual value of $I(n)$ \\
\hline
$1$  & $0.367879441171$ \\
$5$  & $0.145532940573$ \\
$10$ & $0.0838770701034$ \\
$15$ & $0.0590175408793$ \\
$20$ & $0.0455448840758$ \\
$25$ & $0.0370862144237$ \\
$30$ & $0.0312796739322$ \\
$35$ & $0.0270462894091$ \\
$40$ & $0.023822728669$ \\
$45$ & $0.0212860390856$ \\
$50$ & $0.0192377544343$ \\
\hline
\end{tabular}
\end{center}

\end{comment}

%\begin{table}
%\centering
%\begin{tabular}{|l|l|l|}
%\hline
%Integrand & Computed Value & Actual Value \\
%\hline
%$x^{1}e^{x}$: & $0.367879441171$ & $0.367879441171$ \\
%$x^{5}e^{x}$: & $0.145532940573$ & $0.145532940573$ \\
%$x^{10}e^{x}$: & $0.0838770701084$ & $0.0838770701034$ \\
%$x^{15}e^{x}$: & $0.0590209960938$ & $0.0590175408793$ \\
%$x^{20}e^{x}$: & $0.0$ & $0.0455448840758$ \\
%$x^{25}e^{x}$: & $1073741824.0$ & $0.0370862144237$ \\
%$x^{30}e^{x}$: & $-1.80143985095 \cdot 10^{16}$ & $0.0312796739322$ \\
%$x^{35}e^{x}$: & $6.04462909807 \cdot 10^{23}$ & $0.0270462894091$ \\
%$x^{40}e^{x}$: & $0.0$ & $0.023822728669$ \\
%$x^{45}e^{x}$: & $0.0$ & $0.0212860390856$ \\
%$x^{50}e^{x}$: & $1.46150163733 \cdot 10^{48}$ & $0.0192377544343$ \\
%\hline
%\end{tabular}
%\caption{Inaccuracy of values computed using an unstable algorithm.}
%\label{table:unstable_computation}
%\end{table}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Other Notions of Stability} % ------------------------------------

The definition of stability can be made more rigorous in the following way.
Let $f$ be a problem to solve and $\tilde{f}$ an algorithm to solve it.
If for every $\x$ in the domain there exists a $\tilde{\x}$ such that
\[
\frac{\|\tilde{\x} - \x\|}{\|\x\|}
\qquad\text{and}\qquad
\frac{\|\tilde{f}(\x) - f(\tilde{\x})\|}{\|f(\tilde{\x})\|}
\]
are small (close to $\epsilon_{\text{machine}} \approx 10^{-16}$), then $\tilde{f}$ is called stable.
In other words, ``A stable algorithm gives nearly the right answer to nearly the right question'' (Trefethen, Bao, 104).
Note carefully that the quantity on the right is slightly different from the plain forward error introduced earlier.

Stability is desirable, but plain stability isn't the best possible condition.
For example, if for every input $\x$ there exists a $\tilde{\x}$ such that
$\|\tilde{\x} - \x\| / \|\x\|$ is small and $\tilde{f}(\x) = f(\tilde{\x})$ exactly, then $\tilde{f}$ is called \emph{backward stable}.
Thus ``A backward stable algorithm gives exactly the right answer to nearly the right question'' (Trefethen, Bao, 104).
Backward stable algorithms are generally more trustworthy than stable algorithms, but they are also less common.

\begin{comment} % Unfinished garbage table.
\subsection*{Index of Conditioning and Stability of Algorithms} % -------------

\begin{table}[h]
\begin{tabular}{|l|l|l|}
\hline \textbf{Well-Conditioned} & \textbf{Ill-Conditioned} & \textbf{Systems of Equations} \\
{\parbox{0.3\textwidth}{\raggedleft
           \begin{itemize}[leftmargin=*]
               \item finding eigenvalues of a symmetric (or normal) matrix
               \item calculating $e^x$ for relatively small values of $x$
               \item calculating $\ln(x)$ for $x$ not close to $1$
           \end{itemize} }}           &

{\parbox{0.3\textwidth}{\raggedleft
           \begin{itemize}[leftmargin=*]
               \item calculating $x_1 - x_2$ when $x_1 \approx x_2$
               \item computing roots of a polynomial, given the coefficients
               \item computing eigenvalues of a non-symmetric matrix
           \end{itemize} }}              &

{\parbox{0.3\textwidth}{
           \begin{itemize}[leftmargin=*]
                \item relative condition number is \[ \mathcal{K} = ||A|| ||A^{-1}|| \]
           \end{itemize} }}   \\ \hline
\end{tabular}
\end{table}

\begin{table}[h]
\begin{tabular}{|l|l|}
\hline \textbf{Stable} & \textbf{Unstable} \\
{\parbox{0.45\textwidth}{\raggedleft
           \begin{itemize}[leftmargin=*]
               \item finding eigenvalues of a symmetric (or normal) matrix
               \item calculating $e^x$ for relatively small values of $x$
               \item calculating $\ln(x)$ for $x$ not close to $1$
           \end{itemize} }}           &

{\parbox{0.45\textwidth}{\raggedleft
           \begin{itemize}[leftmargin=*]
               \item calculating $x_1 - x_2$ when $x_1 \approx x_2$
               \item computing roots of a polynomial, given the coefficients
               \item computing eigenvalues of a non-symmetric matrix
           \end{itemize} }}    \\ \hline
\end{tabular}
\end{table}
\end{comment}

\subsection*{Stabilty of Linear System Solvers} % -----------------------------

The algorithms presented so far in this manual have different levels of stability.
The LU decomposition (with pivoting) is usually very good, but there are some pathological examples of matrices that can cause it to break down.
Even so, \li{scipy.linalg.solve()} uses the LU decomposition.
The QR decomposition (also with pivoting) is generally considered to be a better option than the LU decomposition and is more stable.
However, solving a linear system using the SVD is even more stable than using the QR decomposition.
For this reason, \li{scipy.linalg.lstsq()} uses the SVD.


\subsection{Positive definite matrices}

Positive definite matrices are often encountered in applications such mechanics and statistics.

We begin with a definition.

\begin{definition}{Positive definite matrix}{positive-definite-matrix}
Let $A$ be an $n \times n$ symmetric matrix. Then $A$ is positive definite if all of its eigenvalues are positive.
\index{positive definite}
\end{definition}

The relationship between a negative definite matrix and positive definite matrix is as follows.

\begin{lemma}{Negative definite matrix}{negative-definite-matrix}
An $n\times n$-matrix $A$ is negative definite if and only if $-A$ is
positive definite.
\end{lemma}

%%\begin{proof}
%%Suppose $A$ is negative definite. For all $\vect{x}$,
%%\begin{equation*}
%%\vect{x}^{T}A\vect{x}\leq -\delta \norm{\vect{x}} ^{2}
%\end{equation*}
%%Then multiply both sides by $-1$ to find that $-A$ is positive definite.
%%Next suppose for all $\vect{x}$,
%%\begin{equation*}
%%\vect{x}^{T}(-A) \vect{x}\geq \delta \norm{\vect{x}}
%%^{2}.
%%\end{equation*}
%%Then multiply by $-1$ to see that $A$ is negative definite.
%%\end{proof}
%%
%%\medskip

%%There is an equivalent characterization of a positive definite matrix in
%%terms of eigenvalues.
%%
%%\medskip
%%
%%\begin{theorem}{Positive definite matrix and eigenvalues}{positive-definite-eigenvalues}  %%\label{3-oct-3f}
%%A symmetric matrix $A$ is positive definite if and only if all
%%of its eigenvalues are positive. (Recall that all of the eigenvalues of a symmetric matrix are real.)
%%\index{positive definite!positive eigenvalues}
%%\end{theorem}
%%
%%\medskip
%%
%%\begin{proof}Suppose first that $A$ is positive definite. Let $\lambda $
%%is an eigenvalue with eigenvector $
%%\vect{u}$. Then, since $\lambda $ is real,
%%\begin{equation*}
%%\vect{u}^{T}A\vect{u}=\lambda \vect{u}^{T}\vect{u}=\lambda \norm{
%%u} ^{2}\geq \delta \norm{\vect{u}} ^{2}
%%\end{equation*}
%%Hence $0<\delta \leq \lambda$, this for any eigenvalue. Hence all
%%eigenvalues are positive.
%%
%%Next suppose that all eigenvalues are positive. Then letting $\delta $ be
%%the smallest eigenvalue and $U$ a real orthogonal matrix such that $
%%U^{T}AU=D$, a diagonal matrix having the eigenvalues of $A$ on the main
%%diagonal,
%%\begin{equation*}
%%\vect{x}^{T}A\vect{x}=\vect{x}^{T}UDU^{T}\vect{x}=(U^{T}\vect{x})
%%^{T}DU^{T}\vect{x}=\sum_{i=1}^{n}(U^{T}\vect{x}) _{i}^{2}\lambda
%%_{i}\geq \delta \sum_{i=1}^{n}(U^{T}\vect{x}) _{i}^{2}=\delta
%%\norm{U^{T}\vect{x}} ^{2}
%%\end{equation*}
%%Now $\norm{U^{T}\vect{x}} ^{2}=(U^{T}\vect{x})
%%^{T}U^{T}\vect{x}=\vect{x}^{T}UU^{T}\vect{x}=\vect{x}^{T}\vect{x}=\norm{\vect{x}} ^{2}$ and so
%%\begin{equation*}
%%\vect{x}^{T}A\vect{x}\geq \delta \norm{\vect{x}} ^{2}
%%\end{equation*}
%%and so $A$ is positive definite.
%%\end{proof}
%%
%%\medskip

Consider the following lemma.

\index{positive definite!invertible}
\begin{lemma}{Positive definite matrix and invertibility}{pos-definite-invertible}
If $A$ is positive definite, then it is invertible.
\end{lemma}

\begin{proof}
If $A\vect{v}=\vect{0}$, then $0$ is an eigenvalue if $\vect{v}$ is non-zero, which
does not happen for a positive definite matrix. Hence $\vect{v}=\vect{0}$ and
so $A$ is one to one. This is sufficient to conclude that it is invertible.
\end{proof}

Notice that this lemma implies that if a matrix $A$ is positive definite, then $\det(A) > 0$.

The following theorem provides another characterization of positive definite matrices. It gives a useful test for verifying if a matrix is positive definite.

\begin{theorem}{Positive definite matrix}{positive-definite}
Let $A$ be a symmetric matrix. Then $A$ is positive definite if and
only if $\vect{x}^T A \vect{x} $ is positive for all non-zero $\vect{x}
\in \R^n$.
\end{theorem}

\begin{proof}
Since $A$ is symmetric, there exists an orthogonal matrix $U$ so that

\[ U^{T}AU=\func{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n)=D,\]

where $\lambda_1,\lambda_2,\ldots,\lambda_n$ are the (not necessarily
distinct) eigenvalues of $A$.
Let $\vect{x}\in\R^n$, $\vect{x}\neq \vect{0}$, and define
$\vect{y}=U^T\vect{x}$.
Then
\[ \vect{x}^TA\vect{x}=\vect{x}^T(UDU^T)\vect{x}
= (\vect{x}^TU)D(U^T\vect{x})
=\vect{y}^TD\vect{y}.\]

Writing $\vect{y}^T=\begin{mymatrix}{cccc}
y_1 & y_2 & \cdots & y_n\end{mymatrix}$,

\begin{eqnarray*}
\vect{x}^TA\vect{x} & = &
\begin{mymatrix}{cccc} y_1 & y_2 & \cdots & y_n\end{mymatrix}
\func{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n)
\begin{mymatrix}{c} y_1 \\ y_2 \\ \vdots \\ y_n\end{mymatrix}\\
& = & \lambda_1 y_1^2 + \lambda_2 y_2^2 + \ldots + \lambda_n y_n^2.
\end{eqnarray*}

$(\Rightarrow)$ First we will assume that $A$ is positive definite and prove that $\vect{x}^T A \vect{x} $ is positive.

Suppose $A$ is positive definite, and $\vect{x}\in\R^n$,
$\vect{x}\neq\vect{0}$.
Since $U^T$ is invertible, $\vect{y}=U^T\vect{x}\neq \vect{0}$,
and thus $y_j\neq 0$ for some $j$, implying $y_j^2>0$
for some $j$.
Furthermore, since all eigenvalues of $A$ are positive,
$\lambda_i y_i^2\geq 0$ for all $i$ and $\lambda_jy_j^2>0$.
Therefore, $\vect{x}^TA\vect{x}>0$.

$(\Leftarrow)$ Now we will assume $\vect{x}^T A \vect{x} $ is positive and show that $A$ is positive definite.

If $\vect{x}^TA\vect{x}>0$ whenever $\vect{x}\neq \vect{0}$,
choose $\vect{x}=U\vect{e}_j$, where $\vect{e}_j$ is the $j\th$
column of $I_n$.
Since $U$ is invertible, $\vect{x}\neq\vect{0}$,
and thus
\[ \vect{y}=U^T\vect{x}=U^T(U\vect{e}_j) =\vect{e}_j.\]
Thus $y_j=1$ and $y_i=0$ when $i\neq j$, so
\[ \lambda_1 y_1^2 + \lambda_2 y_2^2 + \ldots + \lambda_n y_n^2
=\lambda_j,\]
i.e., $\lambda_j=\vect{x}^TA\vect{x}>0$.
Therefore, $A$ is positive definite.
\end{proof}

There are some other very interesting consequences which result from a
matrix being positive definite. First one can note that the property of
being positive definite is transferred to each of the principal submatrices which we will now define.

\begin{definition}{The submatrix $A_k$}{submatrix-Ak}
Let $A$ be an $n\times n$-matrix. Denote by $A_{k}$ the $k\times k$-matrix
obtained by deleting the $k+1,\ldots,n$ columns and the $k+1,\ldots,n$
rows from $A$. Thus $A_{n}=A$ and $A_{k}$ is the $k\times k$ submatrix of $A$
which occupies the upper left corner of $A$.
\index{principal submatrices}
\end{definition}

\begin{lemma}{Positive definite and submatrices}{positive-matrix-submatrix}
Let $A$ be an $n\times n$ positive definite matrix.  Then each submatrix $A_{k}$ is also positive definite.
\end{lemma}

\begin{proof}
This follows right away from the above definition. Let $\vect{x}\in \R^{k}$ be non-zero. Then
\begin{equation*}
\vect{x}^{T}A_{k}\vect{x}=\begin{mymatrix}{cc}
\vect{x}^{T} & 0
\end{mymatrix} A\begin{mymatrix}{c}
\vect{x} \\
0
\end{mymatrix} >  0
\end{equation*}
by the assumption that $A$ is positive definite.
\end{proof}

There is yet another way to recognize whether a matrix is positive definite
which is described in terms of these submatrices. We state the result,  the proof
of which can be found in more advanced texts.

\begin{theorem}{Positive matrix and determinant of $A_k$}{positive-matrix-determinant-Ak}
Let $A$ be a symmetric matrix. Then $A$ is positive definite if
and only if $\det (A_{k})$ is greater than $0$ for every submatrix $A_{k}$,  $k=1,\ldots,n$.
\index{principal submatrices!positive definite}
\index{positive definite!principal submatrices}
\end{theorem}

\begin{proof}
We prove this theorem by induction on $n$. It is clearly
true if $n=1$. Suppose then that it is true for $n-1$ where $n\geq 2$. Since
$\det (A) =\det (A_{n}) >0$, it follows that all the
eigenvalues are non-zero. We need to show that they are all positive. Suppose
not. Then there is some even number of them which are negative, even because
the product of all the eigenvalues is known to be positive, equalling $\det
(A)$. Pick two, $\lambda _{1}$ and $\lambda _{2}$ and let $A
\vect{u}_{i}=\lambda _{i}\vect{u}_{i}$ where $\vect{u}_{i}\neq \vect{0}$ for $
i=1,2$ and $\vect{u}_{1}\dotprod \vect{u}_{2}=0$. Now if $\vect{y}\equiv \alpha
_{1}\vect{u}_{1}+\alpha _{2}\vect{u}_{2}$ is an element of $\sspan\set{
\vect{u}_{1},\vect{u}_{2}}$, then since these are eigenvalues and $\
\vect{u}_{1}\dotprod \vect{u}_{2}=0$, a short computation shows
\begin{equation*}
(\alpha _{1}\vect{u}_{1}+\alpha _{2}\vect{u}_{2}) ^{T}A(
\alpha _{1}\vect{u}_{1}+\alpha _{2}\vect{u}_{2})
\end{equation*}
\begin{equation*}
=\abs{\alpha _{1}}^{2}\lambda _{1}\norm{\vect{u}
_{1}} ^{2}+\abs{\alpha _{2}}^{2}\lambda
_{2}\norm{\vect{u}_{2}} ^{2}<0.
\end{equation*}
Now letting $\vect{x}\in \R^{n-1}$, we can use the induction
hypothesis to write
\begin{equation*}
\begin{mymatrix}{cc}
x^{T} & 0
\end{mymatrix} A\begin{mymatrix}{c}
\vect{x} \\
0
\end{mymatrix} =\vect{x}^{T}A_{n-1}\vect{x}>0.
\end{equation*}
Now the dimension of $\set{\vect{z}\in \R^{n}:z_{n}=0} $ is
$n-1$ and the dimension of $\sspan\set{\vect{u}_{1},\vect{u}
_{2}} =2$ and so there must be some non-zero $\vect{x}\in \R
^{n} $ which is in both of these subspaces of $\R^{n}$. However, the
first computation would require that $\vect{x}^{T}A\vect{x}<0$ while the
second would require that $\vect{x}^{T}A\vect{x}>0$. This contradiction shows
that all the eigenvalues must be positive. This proves the if part of the
theorem. The converse can also be shown to be correct, but it is the
direction which was just shown which is of most interest.
\end{proof}

\begin{corollary}{Symmetric and negative definite matrix}{symmetric-definite-matrix} %\label{pd3}
Let $A$ be symmetric. Then $A$ is negative definite if and only
if
\begin{equation*}
(-1) ^{k} \det (A_{k}) >0
\end{equation*}
for every $k=1,\ldots,n$.
\end{corollary}

\begin{proof}This is immediate from the above theorem when we notice,
that $A$ is negative definite if and only if $-A$ is positive definite.
Therefore, if $\det (-A_{k}) >0$ for all $k=1,\ldots,n$, it
follows that $A$ is negative definite. However, $\det (-A_{k})
=(-1) ^{k}\det (A_{k})$.
\end{proof}


\lab{Single Pixel Camera}{Single Pixel Camera}
\label{lab:SPC}

\objective{You will build a single pixel camera that to take pictures of 2d and 3d tessellations.}

\section*{Single Pixel Camera}
When you look at a picture on a computer screen it is divided into areas called pixels.
Each pixel is a square that has a Red-Green-Blue (RGB) color value.
When a  modern digital camera takes a picture it divides the picture in to pixels and stores the amount red, green, and blue light that comes in at each square.

A single pixel camera treats the whole image as one pixel and thus only stores one RGB value that is the average color value of the whole image.

\section*{2D Images}
In this lab we are going to represent the pictures as tessellated image.
A tessellated image is a tiling with geometric shapes with no overlaps or gaps.

In this lab we assume that $\mathcal{S}$ has a given tessellation $\mathcal{T} = \{F_i\}$, where $F_i$ represents a particular face of our tessellation.
Without loss of generality, we will assume that each $F_i$ is a triangle.
We will represent the vertices of $F_i$ by $F^j_i$, where $F^j_i$ will be a column vector of length two representing the coordinates of $F_i$.

Recall from multivariable calculus that the area of a triangle is half the magnitude of the cross product.
For $F_i$ this is given by
\begin{align}
v_1 &= F_i^1 - F_i^0 \\
v_2 &= F_i^2 - F_i^0 \\
A_i &= \frac{1}{2}\abs{v_1[0] v_2[1] - v_1[1] v_2[0]}
\end{align}

We represent our signal, in this case our RBG color values, as a vector $s$, where the length of $s$ is the number of faces of our tessellation $\mathcal{T}$.
We assume, naturally, that the value of our signal on the face $F_i$ is exactly the value $s[i]$.
We can then, using the areas previously calculated, express the single-pixel measurement of the signal of the camera as an inner product:
\begin{equation}
A \cdot s = b
\end{equation}
In addition, we assume that the total area of the image is equal to the sum of our tessellation.
So
\begin{equation}
\hat{b} = \frac{b}{\sum_i A_i}
\end{equation}
Then $\hat{b}$ will be the RGB values of the singe pixel.

\begin{problem}
Write a function that takes in $S$, a 3-d array of the vertices of 2d tessellation faces, and $s$, the RBG values of the each tessellation.
Output the RGB value given by the single pixel camera.
Below is code to visualize the tessellation and $\hat{b}$.
Also included below is test code.
\begin{lstlisting}
import matplotlib.pyplot as plt
from matplotlib.collections import PolyCollection
# This function plots a 2D tesselation, S, with the colors s
def visualize2d(S, s):
    fig, ax = plt.subplots()
    coll = PolyCollection(S, facecolors=s)
    ax.add_collection(coll)
    ax.autoscale_view()
    plt.show()
\end{lstlisting}
\begin{lstlisting}
#Let b be a 1d array of 3 values representing the RGB values. This plots the color
>>> plt.imshow(np.array([[b]]))
>>> plt.show()
\end{lstlisting}
\begin{lstlisting}
#if your solution to problem 1 was called SPC2, your code should run similarly
>>> test = np.array([[[.5,0],[0,1],[0,0]],[[.5,0],[0,-1],[0,0]],
               [[-1,0],[0,1],[0,0]],[[-1,0],[0,-1],[0,0]]])
>>> C = np.array([[0,0,1],[1,0,0],[0,1,0],[1,0,1]])
>>> SPC2(test, C)
array([ 0.5,  0.3333, 0.5])
\end{lstlisting}
\end{problem}

\section*{Projective Transforms}
Now say that we have a tessellated 3d object and we would like to take a single pixel picture of it.

We begin by selecting a random camera location $c$.
The location is selected randomly on a sphere centered at the origin, with norm large enough that the entire surface lies inside the sphere.
The camera is assumed to point directly at the origin.
In order to keep the distance of the camera from the origin constant, this is best done by inputting the spherical coordinates and then changing to rectangular.
Recall the that the formula to change polar coordinates to rectangular is
\[
S(\theta,\phi,r)=(r*\sin(\phi)*\cos(\theta),r*\sin(\phi)*\sin(\theta),r*\cos(\phi))
\theta \in [0,2\pi),\phi \in [0,\pi), r \in [0,\infty)
\]
\begin{problem}
Write a function that changes spherical coordinates to rectangular.
\end{problem}

We then utilize the projective transform to calculate how the camera ``sees'' the faces of the surface.
Here we state the appropriate equations:
\begin{align}
t &= \text{acos}(\frac{-c_2}{\|c\|}) \\
w^* &= \begin{bmatrix}
	\frac{c_1}{\|c\|} \\
	\frac{-c_0}{\|c\|} \\
	0
\end{bmatrix} \\
w &= \frac{w^*}{\|w^*\|} \\
\hat{w} &= \begin{bmatrix}
0 & 0 & w_1 \\
0 & 0 & -w_0 \\
-w_1 & w_0 & 0
\end{bmatrix} \\
R &= e^{\hat{w}t} = I + sin(t) \hat{w} + (1-cos(t)) \hat{w}^2 \\
P &= \begin{bmatrix}
R^T & -R^T c \\
0 & 1
\end{bmatrix}
\end{align}

We can then use the matrix $P$ to transform any point $x$ in homogenous world, in this case homogenous coordinates are created by simply appending 1 to the bottom of a given column vector, coordinates to homogenous camera coordinates $y$ by the equation:
\begin{equation*}
Px = y
\end{equation*}

\begin{problem}
Write a function that takes in a camera location and outputs the transformation matrix.
\end{problem}

We can then find the projective transform (i.e. how the camera ``sees'' a point) by the equation:
\begin{equation}
P_c(y) = \begin{bmatrix}
\frac{y[0]}{f \cdot y[2]} \\
\frac{y[1]}{f \cdot y[2]}
\end{bmatrix}
\end{equation}

Here $f$ represents the distance from the camera to the image plane, which is essentially an arbitrary constant.
I recommend using $f=.5$.

\begin{problem}
Write a function that takes in a transformation matrix, a list of points in world coordinates, and a constant $f$ and outputs the projective transform of the points.
\end{problem}

Now we essentially we have 2d points and we can calculate the areas as we did in the 2d case. Now we have the areas of each face of our tessellation with respect to our camera view. One thing that we need to worry about is the visibility of the face $F_j$. This constant, for convex surfaces, will either be zero or one. In this case the constant, if the center of the face denoted $Fc_j$ and $r$ is the radius of the tessellation. Then
\[
e_j=||c-Fc_j||_2<\sqrt{||c||_2^2+r^2}-\epsilon
\]
Where $\epsilon$ is small (I recommend .2 if the camera distance is 3).
Then $e_j$ is a constant representing the  the visibility of of the face $F_j$.

\begin{problem}
Write a function that takes in an array of vertices of a tessellation, a camera position, the radius of the tessellation, and an $\epsilon$ value and outputs an array of visibility constants.
\end{problem}
Then:

\begin{equation}
M(j) = A_j \cdot e_j
\end{equation}
Then
\begin{equation}
M \cdot s = b
\end{equation}
\begin{equation}
\frac{b}{\sum_j M_j} = \hat{b}
\end{equation}
And $\hat{b}$ is the RGB value of a picture taken by the single pixel camera at location $c$.

\begin{problem}
Using the previous problems, write a function that takes in an array of vertices of a tessellation, color values for the faces of the tessellation and a camera position in spherical coordinates and outputs single pixel RGB value.
Below is plotting script for 3d tessellations.
Included is data from a satellite that makes a tessellation of the earth, Use it to test your function.
\begin{lstlisting}
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from mpl_toolkits.mplot3d.art3d import Poly3DCollection
import matplotlib.colors as colors

# This function plots a 2D tesselation, F, with the colors C.
def visualize3d(F, C):
    fig = plt.figure()
    ax = Axes3D(fig)
    l = len(F)
    for n in xrange(l):
        verts = [zip(F[n,0]*.5+.5, F[n,1]*.5+.5, F[n,2]*.5+.5)]
        tri = Poly3DCollection(verts)
        tri.set_color(colors.rgb2hex(C[n]))
        tri.set_edgecolor('k')
        ax.add_collection3d(tri)
    plt.show()
\end{lstlisting}
\end{problem}

You can extend this for any tessellated object 2d or 3d.
The only thing that changes is how you calculate the area of the face (For example shoelace formula gives the area of any simple polygon).
The Projective Transform remains the same.

\section*{Application}
The single pixel camera is used in the field of compressed sensing (though the camera works a little differently than the one you made in this lab)


\section{Homotopy invariance of homology}
As a side product, we'll construct something called the cross product. Here's the theorem.
	\begin{theorem}
	If $f_0,f_1:X\to Y$ and $f_0\sim f_1$, then $f_{0,\ast}\sim f_{1,\ast}:S_\ast(X)\to S_\ast(Y)$.
	\end{theorem}
	\begin{corollary}
	With the same hypotheses, then $f_{0,\ast}=f_{1,\ast}: H_\ast(X)\to H_\ast(Y)$, because chain homotopic maps induce the same map on homology.
	\end{corollary}
The proof uses naturality (a lot). We'll produce a chain homotopy from the two inclusions $i_0,i_1:X\to X\times I$, and then homotope $f_0,f_1$ through a map $X\times I\to Y$. Namely, we'll construct a chain homotopy $h_X:S_n(X)\to S_{n+1}(X\times I)$ which gives a homotopy $i_{0,\ast}\sim i_{1,\ast}$, and this'll be natural in $X$.

This gives the result we want. We can get a homotopy $h:=g\circ h_\ast:S_n(X)\to S_{n+1}(X\times I)\to S_{n+1}(Y)$. Well:
		\begin{multline*}
		\partial h + h\partial = \partial(g_\ast h_\ast) + g_\ast h_\ast\partial=g_\ast\partial h_\ast + g_\ast h_\ast \partial = g_\ast(\partial h_\ast + h_\ast\partial)\\
		 = g_\ast(i_{1,\ast} - i_{0,\ast}) = g_\ast i_{1,\ast} - g_\ast i_{0,\ast} = (g\circ i_1)_\ast - (g\circ i_0)_{\ast} = f_{1,\ast} - f_{0,\ast}
		\end{multline*}
(The last equality is because $h_\ast$ is a chain homotopy between $i_{1,\ast}$ and $i_{0,\ast}$.) So $h=g_\ast\circ h_\ast$ is a chain homotopy. But how do we define $h_\ast:S_n(X)\to S_{n+1}(X\times I)$? You want to take $\sigma\mapsto``\sigma\times I''$. More generally, we'll set up a cross product $\times:S_p(X)\times S_q(Y)\to S_{p+q}(X\times Y)$ that is natural, bilinear, satisfy the Leibniz rule, and are normalized.

Naturality is exactly what you'd expect it to be. If $A,B,C$ are abelian groups, then $A\times B\to C$ is a bilinear map if $f(a+a^\prime,b)=f(a,b)+f(a^\prime,b)$ and similarly in the other variable (just substitute the $S_n(X)$ etc here). The Leibniz formula says:
		\begin{equation*}
		\partial(a\times b) = (\partial a)\times b + (-1)^{|a|}a\times\partial b,\quad\text{where }|a|=p\text{ means that }a\in S_p(X).
		\end{equation*}
The word normalized means that the following construction is correct. Suppose $q=0$; then this is a map $S_p(X)\to S_0(Y)\to S_p(X\times Y)$, which (it suffices to define a map $\Sin_p(X)\times \Sin_0(Y)\to S_p(X\times Y)$ because $\left(\sum_i a_i\sigma_i,\sum_j b_j\tau_j\right)\mapsto \sum_{i,j}a_ib_i(\sigma_i\times\tau_j)$ by bilinearity) sending:
		\begin{equation*}
		(\sigma, c^0_y)\mapsto \left(\begin{pmatrix}\sigma \\ c^p_y\end{pmatrix}:\Delta^p\to X\times Y\right)
		\end{equation*}
This latter map is just the composition $\Delta^p\xrightarrow{\sigma} X\xrightarrow{\text{inclusion at }y\in Y}X\times Y$. When $p=0$, we can send:
		\begin{equation*}
		(c^0_x,\tau)\mapsto \left(\begin{pmatrix}c^p_0 \\ \tau\end{pmatrix}:\Delta^p\to X\times Y\right)
		\end{equation*}
This latter map is just the composition $\Delta^p\xrightarrow{\tau} Y\xrightarrow{\text{inclusion at }x\in X}X\times Y$. We have to check that this behaves correctly for the boundary map, namely, that Leibniz holds. We have:
		\begin{equation*}
		\partial(\sigma\times c^0_y) = \partial\sigma\times c^0_y
		\end{equation*}
and similarly. We're going to use induction to define this for $p+q$; we've only done this for $p+q=0,1$. Let's assume it's done for $p+q-1$. First note that there's a universal example of a $p$-simplex, namely the map $\iota_p:\Delta^p\to \Delta^p$, because given any $p$-simplex $\sigma:\Delta^p\to X$, you get $\sigma=\sigma_\ast(\iota_p)$ where $\sigma_\ast:\Sin_p(\Delta^p)\to \Sin_p(X)$. It suffices to define $\iota_p\times\iota_q\in S_{p+q}(\Delta^p\times\Delta^q)$; you're supposed to think of the product of two simplices as a prism, which isn't a simplex itself - but you can triangulate it and look at it as the formal sum of two simplices. Then $\sigma\times \tau = (\sigma\times\tau)_\ast(\iota_p\times\iota_q)$ where $(\sigma\times\tau)_\ast:S_{p+q}(\Delta^p\times\Delta^q)\to S_{p+q}(X\times Y)$. We need this to satisfy the Leibniz rule, so that:
		\begin{equation*}
		S_{p+q-1}(\Delta^p\times\Delta^q)\ni \partial(\iota_p\times\iota_q) = (\partial\iota_p)\times\iota_q + (-1)^p\iota_p\times\partial\iota_q
		\end{equation*}
A necessary condition for $\iota_p\times\iota_q$ to exist is that $\partial((\partial\iota_p)\times\iota_q + (-1)^p\iota_p\times\partial\iota_q) =0$. Let's compute what this is.
		\begin{equation*}
		\partial((\partial\iota_p)\times\iota_q + (-1)^p\iota_p\times\partial\iota_q) = \partial^2(\iota_p)\times\iota_q + (-1)^{p-1}(\partial \iota_p)\times(\partial \iota_q) + (-1)^p(\partial\iota_p)\times(\partial\iota_q) + (-1)^q\iota_p\times\partial^2\iota_q = 0
		\end{equation*}
because $\partial^2=0$.

The subspace $\Delta^p\times\Delta^q\subseteq\mathbf{R}^{p+1}\times\mathbf{R}^{q+1}$ is convex, so by translation, it's homeomorphic to a star-shaped region. But we know that $ H_{p+q-1}(\Delta^p\times\Delta^q)=0$ because $p+q>1$, which means that every cycle is a boundary. In other words, what we checked above is also sufficient! So, choose any element $\iota_p\times\iota_q$ with the right boundary. This means we're done if we check that this choice satisfies naturality, bilinearity, and the Leibniz rule (left to reader). We'll now define $h_X:S_n(X)\to S_{n+1}(X\times I)$ via $h_Xc = c\times\iota$ where $\iota:\Delta^1\to I$ is the obvious map. The cross product is in $S_{p+1}(X\times I)$. Let's compute:
		\begin{equation*}
		\partial h_X = \partial(c\times \iota) = \partial c\times\iota + (-1)^{|c|}c\times\partial\iota
		\end{equation*}
But now, $\partial\iota = c_1^0 - c_0^0\in S_0(I)$, which means that this becomes $\partial c\times\iota + (-1)^{|c|}(\iota_{1,\ast} - \iota_{0,\ast})$. We'll do a little more next time.


\documentclass{homework}

\course{Fluid Mechanics and Operations}{2}

\begin{document}

\section{MSH Problem 2.2}

\textit{\underline{Solution:}}

$dP + g\rho dZ = 0$, \quad $\rho = \displaystyle \frac{PM}{RT}$, \quad $T = 288-0.005Z$
\vspace{1.2ex}

$\Rightarrow dP + \displaystyle \frac{gPM}{R(288-0.005Z)} dZ = 0 \Rightarrow
\int \frac{1}{P} dP = \int -\frac{gM}{R(288-0.005Z)} dZ$
\vspace{1.2ex}

$\Rightarrow \ln \displaystyle \frac{P_b}{P_a} = -\frac{gM}{R} \frac{-1}{0.005} \ln\left\lvert\frac{288-0.005Z}{288}\right\rvert$
\vspace{1.2ex}

$\Rightarrow \ln \displaystyle \frac{1}{2} = \frac{9.8 \cdot 28.8 \cdot 10^{-3}}{8.3145} \frac{1}{0.005} \ln\left\lvert\frac{288-0.005Z}{288}\right\rvert$
\vspace{1.2ex}

$\Rightarrow Z = 5591$m

\section{MSH Problem 2.7}

\textit{\underline{Solution:}}

$r_2 = \displaystyle \frac{0.150}{2} = 0.075$m, \quad $r_B = r_1 = 0.04$m
\vspace{1.2ex}

$V_A = V_B \Rightarrow \pi(r_2^2 - r_i^2) = \pi (r_i^2 - r_B^2)\Rightarrow r_i = 0.060$m
\vspace{1.2ex}

$\rho_B(r_i^2-r_B^2) = \rho_A(r_i^2-r_A^2) \Rightarrow \displaystyle \frac{\rho_B}{\rho_A}(r_i^2 - r_B^2) = r_i^2 - r_A^2$

\begin{displaymath}
	\Rightarrow r_A = \sqrt{r_i^2 - \frac{\rho_B}{\rho_A} (r_i^2 - r_B^2)} = \sqrt{0.060^2 - \frac{1020}{1109} (0.060^2 - 0.04^2)} = 0.042 m
\end{displaymath}

\section{}

\textit{\underline{Solution:}}

$P_g = -1.7psi + 62.42\displaystyle \frac{lb}{ft^3} \cdot 1.6 \cdot 1.5 ft \frac{ft^2}{144in^2} = -0.66psi$
\vspace{1.2ex}

$-0.66psi + 62.42 \cdot SG_B \cdot 1.25 \cdot \displaystyle\frac{1}{144} = 0$
\vspace{1.2ex}

$\Rightarrow SG_B=1.22$

\end{document}


\section*{Ex.29.3-5}
\subsubsection*{Solve using Simplex}

The problem is

\begin{alignat*}{4}
&\max              \quad && 18x_1 +    &&12.5x_2       &&         \\
&\text{subject to} \quad && x_1   +    &&    x_2       && \leq 20 \\
&                        && x_1        &&              && \leq 12 \\
&                        &&            &&    x_2       && \leq 16 \\
&\text{and} \quad        && x_1 \geq 0,&&    x_2 \geq 0&&
\end{alignat*}

We introduce the slack variables and rewrite the problem such that the new slack variables are in the basis solution

\begin{alignat*}{4}
&z     &&=  0  && +18x_1  &&+12.5x_2    \\
&x_3   &&= 20  && -x_1    &&    -x_2       \\
&x_4   &&= 12  && -x_1    &&               \\
&x_5   &&= 16  &&         &&    -x_2
\end{alignat*}

Both coefficients in the objective function are positive, hence increasing their value will increase the maximum. However, increasing them to much, will cause the current basic variables to become negative.
\\
First, we choose $x_1$ to enter as a basis variable. To determine which variable will leave the basis, we determine which of the current basis variables that will decrease to 0 first (making the constraint \textit{tight}), when $x_1$ is increased. This turns out to be $x_4$. When we let $x_1$ enter the basis instead of $x_4$ we get

\begin{alignat*}{4}
&z     &&=  216  && + 12.5 x_2 &&   -18x_4    \\
&x_1   &&= 12    &&            &&     -x_4       \\
&x_3   &&= 8     &&       -x_2 &&     +x_4       \\
&x_5   &&= 16    &&       -x_2 &&
\end{alignat*}

The objective function can potentially still be increased by inserting $x_2$ into the basis. In this case $x_3$ will leave the basis, by repeating the same procedure as above. The new problem maximization problem is then given by

\begin{alignat*}{4}
&z     &&=  316  && - 12.5 x_3 &&   -5.5x_4    \\
&x_1   &&= 12    &&            &&     -x_4       \\
&x_2   &&= 8     &&       -x_3 &&     +x_4       \\
&x_5   &&= 8    &&        +x_3 &&     -x_4
\end{alignat*}

So the optimal solution is when $x=(12,8,0,0,8)$ with optimal value $z=316$.