\lab{Simplex Method}{Simplex Method}
\objective{Implement the Simplex Algorithm to solve linear constrained optimization problems.}
\label{lab:Simplex}

The Simplex Algorithm numbers among the most important algorithms invented within the last 100 years.
It provides a straightforward method for finding optimal solutions to linear constrained optimization problems.
The algorithm obtains the solution by traversing the edges of the feasible region defined by the constraints.
The theory of convex optimization guarantees that the optimal point will be found among the vertices of the feasible
region, and so a carefully implemented Simplex Algorithm will discover the exact solution in a finite number of steps.

\section*{Standard Form}
The Simplex Algorithm accepts a linear constrained optimization problem, also known as a \emph{linear program},
in the form given below:

\begin{align*}
\text{maximize}\qquad &c^Tx \\
\text{subject to}\qquad A&x \leq b \\
 &x \geq 0
\end{align*}
Note that any linear program can be converted to standard form, so there is no loss of
generality in restricting our attention to this particular formulation.

Such an optimization problem defines a region in space called the \emph{feasible region}, the set of points
satisfying the constraints. Because the constraints are all linear, the feasible region forms a geometric object
called a \emph{polytope}, having flat faces and edges (see Figure \ref{fig:polytope}).
The Simplex Algorithm jumps among the vertices of the feasible region searching for an optimal point.
It does this by moving along the edges of the feasible region in such a way that the objective function
is always increased after each move.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{feasiblePolytope.pdf}
\caption{The feasible region for a linear program. The optimal point
is one of the vertices of the polytope.}
\label{fig:polytope}
\end{figure}

Implementing the Simplex Algorithm is straightforward, provided one carefully follows the procedure.
We will break the algorithm into several small steps, and write a function to perform each one.
To become familiar with the execution of the Simplex algorithm, it is helpful to work several examples by hand.

\section*{The Simplex Solver}
Our program will be more lengthy than many other lab exercises and will consist of a collection of functions working
together to produce a final result.
It is important to clearly define the task of each function and how all the functions will work together.
If this program is written haphazardly, it will be much longer and more difficult to read than it needs to be.
We will walk you through the steps of implementing the Simplex Algorithm as a Python class.
%Since the Simplex Algorithm assumes that all the variables are non-negative, we do not need any special logic for it.
%what is this previous statement saying? What 'special logic' would you need if the variables were negative?

For demonstration purposes, we will use the following linear program.
\begin{align*}
\text{maximize}\qquad & 3x_0 + 2x_1 \\
\text{subject to}\qquad
& x_0 - x_1 \leq 2 \\
& 3x_0 + x_1 \leq 5 \\
& 4x_0 + 3x_1 \leq 7 \\
& x_0, x_1 \geq 0.
\end{align*}

\subsection*{Accepting a Linear Program}
Our first task is to determine if we can even use the Simplex algorithm.
Assuming that the problem is presented to us in standard form, we need
to check that the feasible region is nonempty.

% Problem 1: Check Feasibility
\begin{problem}
Write a class that accepts the arrays \li{c}, \li{A}, and \li{b} of a linear optimization problem in standard form.
In the constructor, check that the system is feasible at the origin\footnote{For simplicity, we only check feasibility at the origin. A more robust solver sets up the auxiliary problem and solves it to find a starting point if the origin is infeasible.}.
That is, check that $Ax \leq b$ componentwise when $x = 0$. % TODO: bell <=
Raise a \li{ValueError} if the problem is not feasible at the origin.
\label{prob:initsolver}
\end{problem}

\subsection*{Adding Slack Variables}
Our next step is to convert the inequality constraints $Ax \leq b$ into equality constraints
by introducing one slack variable for each constraint.
If the constraint matrix $A$ is an $m \times n$ matrix, then there are $m$ slack variables,
one for each row of $A$.
Grouping all of the slack variables into a vector of length $m$, denoted $z$, our
constraints now take the form $Ax + z = b$.
In our example, this gives
\[
\z = \begin{bmatrix}
 x_2 \\
 x_3 \\
 x_4
\end{bmatrix},
\]

When adding slack variables, it is useful to represent all of your variables, both the original primal variables and
the additional slack variables, in a convenient manner.
One effective way is to refer to a variable by its subscript.
For example, we can use the integers $0$ through $n-1$ to refer to the original (non-slack) variables $x_0$ through
$x_{n-1}$, and we can use the integers $n$ through $n+m-1$ to track the slack variables (where the slack variable
corresponding to the $i$-th row of the constraint matrix is represented by the index $n+i-1$).

We also need some way to track which variables are basic (non-zero) and which variables are nonbasic (those that have value $0$).
A useful representation for the variables is a Python list (or NumPy array), where the elements of the list are integers.
Since we know how many basic variables we have ($m$, to be precise), we can partition the list so that all the basic
variables are kept in the first $m$ locations, and all the non-basic variables are stored at the end of the list.
The ordering of this list is important. In particular, if $i \leq m$, the $i$-th element of the list represents
the basic variable corresponding to the $i$-th row of $A$. Henceforth we will refer to this list as the \emph{index list}.

Initially, the basic variables are simply the slack variables, and their values correspond to the values of the matrix $b$.
In our example, we have 2 primal variables $x_0$ and $x_1$, and we must add 3 slack variables.
Thus, we instantiate the following index list:
\begin{lstlisting}
>>> L = [2, 3, 4, 0, 1]
\end{lstlisting}
Notice how the first $3$ entries of the index list are $2, 3, 4$, the indices representing the slack variables.
This reflects the fact that the basic variables at this point are exactly the slack variables.

As the Simplex Algorithm progresses, however, the basic variables change, and it will be necessary to swap
elements in our index list. Suppose the variable represented by the index $4$ becomes nonbasic, while
the variable represented by index $0$ becomes basic. We must swap these two entries in the index list.
This can be done in a single, efficient line of Python code:
\begin{lstlisting}
>>> L[2], L[3] = L[3], L[2]
>>> L
[2, 3, 0, 4, 1]
\end{lstlisting}
Now our index list tells us that the current basic variables have indices $2, 3, 0$.

% Problem 2: Slack variables.
\begin{problem}
Design and implement a way to store and track all of the basic and non-basic variables.

Hint: Using integers that represent the index of each variable is useful for Problem \ref{prob:blands}.
\label{prob:slackvars}
\end{problem}

\subsection*{Creating a Tableau}
After we have determined that our program is feasible, we need to create the \emph{tableau} (sometimes called the \emph{dictionary}, a construct that tracks the state of the algorithm.
You may structure the tableau to suit your specific implementation.
Remember that your tableau will need to include in some way the slack variables that you created in Problem \ref{prob:slackvars}.

There are many different ways to build your tableau.
One way is to mimic the tableau that is often used when performing the Simplex Algorithm by hand.
Define
\[
\bar{A} = \begin{bmatrix}
  A & I_m
\end{bmatrix},
\]
where $I_m$ is the $m \times m$ identity matrix,
and define
\[
\bar{c} = \begin{bmatrix}
  c\\
  0\\
  \vdots\\
  0
\end{bmatrix}.
\]
That is, $\bar{c} \in \mathbb{R}^{n+m}$ such that the first $n$ entries are $c$ and the final $m$ entries are zeros.
Then the initial tableau has the form
\begin{equation}
T = \begin{bmatrix}
    0 & -\bar{c}^T & 1  \\
    b & \bar{A} & 0
    \end{bmatrix}.
\label{eqn:hand_tab}
\end{equation}

The columns of the tableau correspond to each of the variables (both primal and slack), and the rows of the tableau
correspond to the basic variables. Using the convention introduced above
of representing the variables by indices in the index list, we have the following correspondence:
\[
\text{column } i \Leftrightarrow \text{index } i-2, \qquad i = 2, 3, \ldots, n+m+1,
\]
and
\[
\text{row } j \Leftrightarrow L_{j-1}, \qquad j = 2, 3, \ldots, m+1,
\]
where $L_{j-1}$ refers to the $(j-1)$-th entry of the index list.

For our example problem, the initial index list is
\[
L = (2, 3, 4, 0, 1),
\]
and the initial tableau is
\begin{equation*}
T = \begin{bmatrix}
    0 & -3 & -2 & 0 & 0 & 0 & 1\\
    2 & 1 & -1 & 1 & 0 & 0 & 0\\
    5 & 3 & 1 & 0 & 1 & 0 & 0\\
    7 & 4 & 3 & 0 & 0 & 1 & 0
    \end{bmatrix}.
\end{equation*}
The third column corresponds to index $1$, and the fourth row corresponds to index $4$, since this is the
third entry of the index list.

The advantage of using this kind of tableau is that it is easy to check the progress of your algorithm by hand.
The disadvantage is that pivot operations require careful bookkeeping to track the variables and constraints.

% I (Jared) propose that we don't introduce the below tableau in the lab. It is less intuitive, and I think could lead to more confusion.
\begin{comment}
We can also use a tableau of the following format:
\begin{equation}
T = \begin{bmatrix}
    0 & c^T  & 0 \\
    0 & I_n & 0\\
    b & -A  & 0
\end{bmatrix}.
\label{eqn:matrix_tab}
\end{equation}
Here, $T$ is a square matrix of size $(n+m+1) \times (n+m+1)$.
The advantage of this form of the tableau is that all the pivot bookkeeping is built into the matrix.
For our example problem, the initial tableau of this form is
\begin{equation}
T = \begin{bmatrix}
        0 & 3 & 2 & 0 & 0 & 0 \\
        0 & 1 & 0 & 0 & 0 & 0 \\
        0 & 0 & 1 & 0 & 0 & 0 \\
        2 &-1 & 1 & 0 & 0 & 0 \\
        5 &-3 &-1 & 0 & 0 & 0 \\
        7 &-4 &-3 & 0 & 0 & 0
\end{bmatrix}.
\label{eqn:matrix_inittab}
\end{equation}
\end{comment}

\begin{problem}
Add a method to your Simplex solver that will create the initial tableau that you will use.
Using a NumPy array to represent your tableau will simplify several parts of the Simplex algorithm.
\label{prob:maketableau}
\end{problem}

\subsection*{Pivoting}
Pivoting is the mechanism that really makes Simplex useful.
Pivoting refers to the act of swapping basic and nonbasic variables, and transforming the tableau appropriately.
This has the effect of moving from one vertex of the feasible polytope to another vertex in a way that increases
the value of the objective function.
Depending on how you store your variables, you may need to modify a few different parts of your solver to reflect this swapping.

When initiating a pivot, you need to determine which variables will be swapped.
In the tableau representation, you first find a specific element on which to pivot, and the row and column that contain the pivot
element correspond to the variables that need to be swapped.
Row operations are then performed on the tableau so that the pivot column becomes an elementary vector.

Let's break it down, starting with the pivot selection. We need to use some care when choosing the pivot element.
To find the pivot column, search from left to right along the top row of the tableau
(ignoring the first column), and stop once you encounter the first negative value. The index corresponding
to this column will be designated the \emph{entering index}, since after the full pivot operation, it will enter
the basis and become a basic variable.

Using our initial tableau $T$ in the example, we stop at the second column:
\[ T = \left[ \:
\begin{array}{*{7}{c}}
\cline{2-2}
0 & \multicolumn{1}{|c}{-3} & \multicolumn{1}{|c}{-2} & 0 & 0 & 0 & 1\\
2 & \multicolumn{1}{|c}{1} & \multicolumn{1}{|c}{-1} & 1 & 0 & 0 & 0\\
5 & \multicolumn{1}{|c}{3} & \multicolumn{1}{|c}{1} & 0 & 1 & 0 & 0\\
7 & \multicolumn{1}{|c}{4} & \multicolumn{1}{|c}{3} & 0 & 0 & 1 & 0\\
\cline{2-2}
\end{array}
\right] \]
We now know that our pivot element will be found in the second column. Our entering index is thus $0$.

Next, we select the pivot element from among the positive entries in the pivot column (ignoring the entry in the first row).
\emph{If all entries in the pivot column are non-positive, the problem is unbounded and has no solution.} In this case, the algorithm
should terminate.
Otherwise, assuming our pivot column is the $j$-th column of the tableau and that the positive entries of this column are
$T_{i_1, j}, T_{i_2, j}, \ldots, T_{i_k, j}$, we calculate the ratios
\[
\frac{T_{i_1,1}}{T_{i_1,j}}, \frac{T_{i_2,1}}{T_{i_2,j}}, \ldots, \frac{T_{i_k,1}}{T_{i_k,j}},
\]
and we choose our pivot element to be one that minimizes this ratio. If multiple entries minimize the ratio, then we utilize
\emph{Bland's Rule}, which instructs us to choose the entry in the row corresponding to the smallest index.
(Obeying this rule is important, as it prevents the possibility of the algorithm cycling back on itself infinitely.)
The index corresponding to the pivot row is designated as the \emph{leaving index}, since after the full pivot operation,
it will leave the basis and become a nonbasic variable.

In our example, we see that all entries in the pivot column (ignoring the entry in the first row, of course) are positive,
and hence they are all potential choices for the pivot element. We then calculate the ratios, and obtain
\[
\frac{2}{1} = 2,\quad \frac{5}{3} = 1.66...,\quad \frac{7}{4} = 1.75.
\]
We see that the entry in the third row minimizes these ratios. Hence, the element in the second column, third row is our designated
pivot element, and our leaving index is $L_2 = 3$:

\[ T = \left[ \:
\begin{array}{*{7}{c}}

0 & -3 & -2 & 0 & 0 & 0 & 1\\
2 & 1 & -1 & 1 & 0 & 0 & 0\\\cline{2-2}
5 & \multicolumn{1}{|c}{3} & \multicolumn{1}{|c}{1} & 0 & 1 & 0 & 0\\\cline{2-2}
7 & 4 & 3 & 0 & 0 & 1 & 0\\
\end{array}
\right] \]

\begin{comment}
If we are using the tableau representation in equation \ref{eqn:matrix_tab}, pivot operations are reduced to a simple matrix equation:
\[T = T + T_m \otimes T_n,\]
where $T_m$ is the column corresponding to the variable entering the basis and $T_n$ is a normalized vector corresponding to the variable leaving the basis.  The result of the equation is the new tableau.

For example, for the initial tableau, \ref{eqn:matrix_inittab}, we will demonstrate the first pivot operation.
We can do the entire pivot with a single outer product.
the first pivot should occur with $x_1$ leaving and $x_4$ entering.
In other words, we want to pivot at row $i = 4$ and column $j = 1$ in the tableau (the indices are offset by one because of the objective function and the row of constraints).

The row corresponding to $x_4$ is
\[
\begin{bmatrix} 5 &-3 &-1 & 0 & 0 & 0\end{bmatrix}.
\]
This represents the equation
\[
x_4 = 5 - 3x_1 - x_2.
\]
Our eventual goal is to solve for $x_1$ and substitute into the remaining rows of the tableau.
A simple method to accomplish this is to rewrite the equation so that we have zero on the left-hand side:
\[
0 = 5 - 3x_1 - x_2 - x_4.
\]
Now, we can normalize this equation so that the coefficient of $x_1$ is $-1$.
This is always accomplished by dividing the equation by the negative of the coefficient of $x_1$:
\begin{equation}
0 = \frac{5}{3} - x_1 - \frac{1}{3}x_2 - \frac{1}{3}x_4.
\label{eq:zero-equation}
\end{equation}
This is represented by the vector
\[
\begin{bmatrix} 5/3 & -1 & -1/3 & 0 & -1/3 & 0\end{bmatrix}.
\]
Since this left-hand side is zero, I can add any scalar multiple of this equation to any of the equations for $x_i$ and still have an equation for $x_i$.
For example, the equation for $x_5$ is
\[
x_5 = 7 - 4x_1 - 3x_2.
\]
Thus, I can add $-4$ times \eqref{eq:zero-equation} to this equation without changing the left-hand side:
\[ x_5 = 7 - 4x_1 - 3x_2 = 7 - 4x_1 - 3x_2 + -4\left(\frac{5}{3} - x_1 - \frac{1}{3}x_2 - \frac{1}{3}x_4\right) = \frac{1}{3} - \frac{5}{3}x_2 + \frac{4}{3} x_4.
\]
Notice that we end up with an equation that does not include $x_1$ and now has $x_4$, just like we wanted.
In fact, this works in all of our equations, including those for the objective function and even for $x_1$!
Since the coefficient of $x_1$ in \eqref{eq:zero-equation} is $-1$, when we scale it by the coefficient of $x_1$ in any particular row, the $x_1$ cancels out.
\[
T = T + \begin{bmatrix}3 \\ 1 \\ 0 \\ -1 \\ -3 \\ -4\end{bmatrix}\begin{bmatrix} 5/3 & -1 & -1/3 & 0 & -1/3 & 0\end{bmatrix}.
\]
The column vector is just the second column of $T$, which is the column containing the coefficients of $x_1$ in each row.
When we compute this sum, we obtain the tableau.
\[
T = \begin{bmatrix}
        5 &  0 & 1 & 0 & -1 & 0 \\
        5/3 & 0 &-1/3 & 0 &-1/3 & 0 \\
        0 & 0 & 1 & 0 & 0 & 0 \\
        1/3 & 0 & 4/3 & 0 & 1/3 & 0 \\
        0 & 0 & 0 & 0 & 1 & 0 \\
        1/3 & 0 & -5/3 & 0 & 4/3 & 0
\end{bmatrix}.
\]
\end{comment}
\begin{problem}
Write a method that will determine the pivot row and pivot column according to Bland's Rule.
\begin{comment}
\begin{definition}[Bland's Rule]
Choose the nonbasic variable with the smallest index that has a positive coefficient in the objective function
as the leaving variable.  Choose the basic variable with the smallest index among all the binding basic variables.
\end{definition}

Bland's Rule is important in avoiding cycles when performing pivots.
This rule guarantees that a feasible Simplex problem will terminate in a finite number of pivots.
\end{comment}
\label{prob:blands}
\end{problem}

The next step is to swap the entering and leaving indices in our index list.
In the example, we determined above that these indices are $0$ and $3$. We swap these two elements in our index list,
and the updated index list is now
\[
L = (2, 0, 4, 3, 1),
\]
so the basic variables are now given by the indices $2, 0, 4$.

Finally, we perform row operations on our tableau in the following way: divide the pivot row by the value of the pivot entry.
Then use the pivot row to zero out all entries in the pivot column above and below the pivot entry. In our example, we first divide
the pivot row by 3, and then zero out the two entries above the pivot element and the single entry below it:
\begin{align*}
\begin{bmatrix}
    0 & -3 & -2 & 0 & 0 & 0 & 1\\
    2 & 1 & -1 & 1 & 0 & 0 & 0\\
    5 & 3 & 1 & 0 & 1 & 0 & 0\\
    7 & 4 & 3 & 0 & 0 & 1 & 0
    \end{bmatrix} &\rightarrow
\begin{bmatrix}
    0 & -3 & -2 & 0 & 0 & 0 & 1\\
    2 & 1 & -1 & 1 & 0 & 0 & 0\\
    5/3 & 1 & 1/3 & 0 & 1/3 & 0 & 0\\
    7 & 4 & 3 & 0 & 0 & 1 & 0
    \end{bmatrix}\rightarrow\\
\begin{bmatrix}
    5 & 0 & -1 & 0 & 1 & 0 & 1\\
    2 & 1 & -1 & 1 & 0 & 0 & 0\\
    5/3 & 1 & 1/3 & 0 & 1/3 & 0 & 0\\
    7 & 4 & 3 & 0 & 0 & 1 & 0
    \end{bmatrix} &\rightarrow
\begin{bmatrix}
    5 & 0 & -1 & 0 & 1 & 0 & 1\\
    1/3 & 0 & -4/3 & 1 & -1/3 & 0 & 0\\
    5/3 & 1 & 1/3 & 0 & 1/3 & 0 & 0\\
    7 & 4 & 3 & 0 & 0 & 1 & 0
    \end{bmatrix}\rightarrow\\
\begin{bmatrix}
    5 & 0 & -1 & 0 & 1 & 0 & 1\\
    1/3 & 0 & -4/3 & 1 & -1/3 & 0 & 0\\
    5/3 & 1 & 1/3 & 0 & 1/3 & 0 & 0\\
    1/3 & 0 & 5/3 & 0 & -4/3 & 1 & 0
    \end{bmatrix}.
\end{align*}
The result of these row operations is our updated Tableau, and the pivot operation is complete.

% Problem 5: Pivoting
\begin{problem}
Add a method to your solver that checks for unboundedness and performs a single pivot operation from start to completion.
If the problem is unbounded, raise a \li{ValueError}.
\end{problem}

\subsection*{Termination and Reading the Tableau}
Up to this point, our algorithm accepts a linear program, adds slack variables, and creates the initial tableau. After
carrying out these initial steps, it then performs the pivoting operation iteratively until the optimal point is found.
But how do we determine when the optimal point is found? The answer is to look at the top row of the tableau. More specifically,
before each pivoting operation, check whether all of the entries in the top row of the tableau (ignoring the entry in the first
column) are nonnegative. If this is the case, then we have found an optimal solution, and so we terminate the algorithm.

The final step is to report the solution. The ending state of the tableau and index list tell us everything we need to know.
The maximum value attained by the objective function is found in the upper leftmost entry of the tableau. The nonbasic variables,
whose indices are located in the last $n$ entries of the index list, all have the value $0$. The basic variables, whose indices
are located in the first $m$ entries of the index list, have values given by the first column of the tableau. Specifically, the basic
variable whose index is located at the $i$-th entry of the index list has the value $T_{i+1, 1}$.

In our example, suppose that our algorithm terminates with the tableau and index list in the following state:
\[
T = \begin{bmatrix}
5.2 & 0 & 0 & 0 & .2 & .6 & 1\\
.6 & 0 & 0 & 1 & -1.4 & .8 & 0\\
1.6 & 1 & 0 & 0 & .6 & -.2 & 0\\
.2 & 0 & 1 & 0 & -.8 & .6 & 0\\
\end{bmatrix}
\]
\[
L = (2, 0, 1, 3, 4).
\]
Then the maximum value of the objective function is $5.2$. The nonbasic variables have indices $3, 4$ and have the value $0$.
The basic variables have indices $2, 0,$ and $1$, and have values $.6, 1.6$, and $.2$, respectively.
In the notation of the original problem statement, the solution is given by
\begin{align*}
x_0 &= 1.6\\
x_1 &= .2.
\end{align*}

% SimplexSolver.solve()
\begin{problem}
Write an additional method in your solver called \li{solve()} that obtains the optimal solution, then returns the maximum value, the basic variables, and the nonbasic variables.
The basic and nonbasic variables should be represented as two dictionaries that map the index of the variable to its corresponding value.

For our example, we would return the tuple \li{(5.2, \{0: 1.6, 1: .2, 2: .6\}, \{3: 0, 4: 0\})}.
%The correct format of this tuple is critical, as this tuple of information will be used judge whether or not your solver works!
\end{problem}

At this point, you should have a Simplex solver that is simple to use. The following code demonstrates how your solver is
expected to behave:

\begin{lstlisting}
>>> import SimplexSolver

# Initialize objective function and constraints.
>>> c = np.array([3., 2])
>>> b = np.array([2., 5, 7])
>>> A = np.array([[1., -1], [3, 1], [4, 3]])

# Instantiate the simplex solver, then solve the problem.
>>> solver = SimplexSolver(c, A, b)
>>> sol = solver.solve()
>>> print(sol)
(5.200,
 {0: 1.600, 1: 0.200, 2: 0.600},
 {3: 0, 4: 0})
\end{lstlisting}

If the linear program were infeasible at the origin or unbounded, we would expect the solver to alert the user by raising an error.

Note that this simplex solver is \emph{not} fully operational.
It can't handle the case of infeasibility at the origin.
This can be fixed by adding methods to your class that solve the \emph{auxiliary problem}, that of finding an initial feasible tableau when the problem is not feasible at the origin.
Solving the auxiliary problem involves pivoting operations identical to those you have already implemented, so adding this functionality
is not overly difficult.

\section*{The Product Mix Problem}
We now use our Simplex implementation to solve the \emph{product mix problem}, which in its basic form can be expressed as a simple linear program.
Suppose that a manufacturer makes $n$ products using $m$ different resources (labor, raw materials, machine time available, etc).
The $i$-th product is sold at a unit price $p_i$, and there are at most $m_j$ units
of the $j$-th resource available. Additionally, each unit of the $i$-th product requires $a_{j,i}$ units of resource $j$.
Given that the demand for product $i$ is $d_i$ units per a certain time period, how do we choose the optimal amount
of each product to manufacture in that time period so as to maximize revenue, while not exceeding the available resources?

Let  $x_1, x_2, \ldots, x_n$ denote the amount of each product to be manufactured.
The sale of product $i$ brings revenue in the amount of $p_ix_i$.
Therefore our objective function, the profit, is given by
\[
\sum_{i=1}^n p_ix_i.
\]
Additionally, the manufacture of product $i$ requires $a_{j,i}x_i$ units of resource $j$.
Thus we have the resource constraints
\[
\sum_{i=1}^n a_{j,i}x_i \leq m_j  \text{ for } j = 1, 2, \ldots, m.
\]

Finally, we have the demand constraints which tell us not to exceed the demand for the products:
\[
x_i \leq d_i \text{ for } i = 1, 2, \ldots, n
\]

The variables $x_i$ are constrained to be nonnegative, of course. We therefore have a linear program in the appropriate form that is feasible at the origin.
It is a simple task to solve the problem using our Simplex solver.

\begin{problem}
Solve the product mix problem for the data contained in the file \li{productMix.npz}. In this problem, there are 4 products and 3 resources.
The archive file, which you can load using the function
\li{np.load}, contains a dictionary of arrays. The array with key \li{'A'} gives the resource coefficients $a_{i,j}$ (i.e. the $(i,j)$-th entry
of the array give $a_{i,j}$). The array with key \li{'p'} gives the unit prices $p_i$. The array with key \li{'m'} gives the available resource
units $m_j$. The array with key \li{'d'} gives the demand constraints $d_i$.

Report the number of units that should be produced for each product.
\end{problem}
%---------------------------------------------------------------------------------------------------
%
%
% \section*{Auxiliary Problems}
%
% When one of the entries of $b$ is negative, we need to run an auxiliary problem to find a feasible point.
% Try adding $x_0$ as the last variable.
% That way you just need to add a single row and column to your current tableau $T$.
% You can check that subtracting $x_0$ from each of the constraints is the same as adding $x_0$ to each of the expressions for the slack variables.
% Since we also need write $x_0$ in terms of itself ($x_0 = x_0$), this is equivalent to setting the last column equal to 1 for all the rows corresponding to slack variables, plus one new row.
% We also need to change the objective function to $-x_0$.
% Let's let $N$ be the matrix for the auxiliary problem.
% Then we can construct it from $T$ using the code
% \begin{lstlisting}
% >>> N = zeros((s+1,s+1))
% >>> N[0:s,0:s] = T
% >>> N[n:s+1,s] = 1
% >>> N[0,s] = -1
% \end{lstlisting}
%
% Before you run simplex on this auxiliary tableau, make sure you do a pivot on the last column and the row corresponding to the smallest entry of $b$.
% Check that the objective function value is 0 when simplex finishes running.
% If not, your initial problem is infeasible.
% If the problem is feasible, get the system of equations from $N$ and put them back into $T$, leaving off the last row and column of $N$ (for $x_0$) and the first row (corresponding to the objective function).
% \begin{lstlisting}
% T[1:s,0:s] = N[1:s,0:s]
% \end{lstlisting}
% The last thing you need to do is insert your previous objective function.
% However, you need to re-write it in terms of the current nonbasic variables.
% Fortunately, $T$ currently contains all of your variables written in terms of the nonbasic variables.
% You can check (mathematically, or however you want to satisfy yourself) that
% \begin{lstlisting}
% T[0,0:s] = T[0,0:s]*T
% \end{lstlisting}
% will put the objective function into the first row, now written in terms of the new nonbasic variables.

\section*{Beyond Simplex}
The \emph{Computing in Science and Engineering} journal listed Simplex as one of the top ten algorithms of the twentieth century \cite{Nash2000}.
Despite its popularity, like any other algorithm, Simplex has its drawbacks.

In 1972, Victor Klee and George Minty Cube published a paper with several examples of worst-case polytopes for the Simplex algorithm \cite{Klee1972}.
In their paper, they give several examples of polytopes that the Simplex algorithm struggles to solve.

Consider the following linear program from Klee and Minty.
\begin{align*}
\text{max } & 2^{n-1}x_1 & + & 2^{n-2}x_2  & + & \cdots & + & 2x_{n-1} & + & x_n\\
\text{subject to } & x_1 &  &  &  &  &  &  &  &\leq 5\\
& 4x_1 & + & x_2 &  &  &  &  &  &\leq 25\\
& 8x_1 & + & 4x_2 & + & x_3 &  &  &  &\leq 125\\
& \vdots & &      &   &     &  &  &  &\vdots\\
& 2^n x_1 & + & 2^{n-1} x_2 & + & \cdots & + & 4x_{n-1} & + x_n &\leq 5\\
\end{align*}

% When $n = 3$, we have the initial tableau
% \begin*{equation*}
% \begin{bmatrix}
% 0 & 4 & 2 & 1 & 0 & 0 & 0\\
% 0 & 1 & 0 & 0 & 0 & 0 & 0\\
% 0 & 0 & 1 & 0 & 0 & 0 & 0\\
% 0 & 0 & 0 & 1 & 0 & 0 & 0\\
% 5 & -1 & 0 & 0 & 0 & 0 & 0\\
% 25 & -4 & -1 & 0 & 0 & 0 & 0\\
% 125 & -8 & -4 & -1 & 0 & 0 & 0\\
% \end{bmatrix}
% \end{equation*}
%
% After the first pivot with $x_1$ leaving and $s_1$ entering, we have the tableau
%
% \begin{equation*}
% \begin{bmatrix}
% 20 & 0 & 2 & 1 & -4 & 0 & 0\\
% 5 & 0 & 0 & 0 & -1 & 0 & 0\\
% 0 & 0 & 1 & 0 & 0 & 0 & 0\\
% 0 & 0 & 0 & 1 & 0 & 0 & 0\\
% 0 & 0 & 0 & 0 & 1 & 0 & 0\\
% 5 & 0 & -1 & 0 & 4 & 0 & 0\\
% 85 & 0 & -4 & -1 & 8 & 0 & 0\\
% \end{bmatrix}
% \end{equation*}
%
% \begin{problem}
% What is the final tableau for this Klee-Minty example with $n=3$?
% How many iterations does it take to arrive at the final tableau?
% \end{problem}
%
% \begin{problem}
% Using problem 1 as a guide, guess the optimum value of the Klee-Minty example with $n=20$.
% Then find the maximum value for the Klee-Minty example with $n=20$.
% How many iterations does it take to arrive at the final tableau?
% How long does the program take to run for only $20$ variables?
% \end{problem}

Klee and Minty show that for this example, the worst case scenario has exponential time complexity.
With only $n$ constraints and $n$ variables, the simplex algorithm goes through $2^n$ iterations.
This is because there are $2^n$ extreme points, and when starting at the point $x=0$, the simplex algorithm goes through all of the extreme points before reaching the optimal point $(0,0,\dots, 0, 5^n)$.
Other algorithms, such as interior point methods, solve this problem much faster because they are not constrained to follow the edges.

\printbibliography


% !TEX root = Main.tex
\section{Word Embeddings}
\textbf{Distributional Model:}\\
$p_\theta(w|w')$ = Pr[$w$ occurs in context of $w'$]\\
\textbf{Log-likelihood:}\\
$L(\theta; \mathbf{w}) = \sum_{t=1}^T\sum_{\Delta \in I}{\log p_\theta(w^{(t+\Delta)}|w^{(t)})}$\\
\textbf{Latent Vector Model:} $w \rightarrow (\mathbf{x}_w, b_w) \in \mathbb{R}^{D+1} \\p_{\theta}(w|w') = \frac{\exp[\langle \mathbf{x}_w,\mathbf{x}_{w'}\rangle + b_w]}{\sum_{v\in V}{\exp[\langle \mathbf{x}_v,\mathbf{x}_{w'}\rangle + b_v ]}}$ (soft-max).\\
\textbf{Modifications:}\\
$\log p_{\theta}(w|w') = \langle  y_{w} , x_{w'} \rangle + b_w$,  word $y_w$, c'txt $x_{w'}$\\
use GloVe objective\\
negative sampling (logistic classification)

\subsection*{GloVe (Weighted Square Loss)}
\textbf{Co-occurence Matrix:}\\
$\mathbf{N} = (n_{ij}) \in \mathbb{R}^{|V|\times|C|} = \# of word w_i$ in context $w_j$\\
\textbf{Objective:} $H(\theta;\mathbf{N})$\\
$= \sum_{n_{ij} > 0} f(n_{ij})(\log n_{ij} - \log \exp[\langle \mathbf{x}_i, \mathbf{y}_j \rangle + b_i + d_j])^2$\\
with $f(n) = \min\{1, (\frac{n}{n_{max}})^\alpha\}$, $\alpha \in (0;1]$.\\
unnormalized distr. $\rightarrow$ 2-sided loss function\\
1. sample $(i,j) u.a.r, s.t. n_{ij}>0$\\
2. $\mathbf{x}_i^{new} \leftarrow \mathbf{x}_i + 2\eta f(n_{ij})(\log n_{ij} - \langle \mathbf{x}_i, \mathbf{y}_j \rangle)\mathbf{y}_j$\\
3. $\mathbf{y}_j^{new} \leftarrow \mathbf{y}_j + 2\eta f(n_{ij})(\log n_{ij} - \langle \mathbf{x}_i, \mathbf{y}_j \rangle)\mathbf{x}_i$

\subsection*{Discussion}
Word embeddings can model analogies and relatedness, but antonyms are usually not well captured.


% !TEX root = main.tex

%-------------------------------------------------
\section{Appendix}\label{appx:expe}

%-----------------------------
%\subsection*{Bernoulli's law of large numbers}\label{subsec:}

In the proof of Theorem~\ref{thm:wlln}, we used Chebyshev's inequality to show that
\[
\prob(|\bar{X}_n - \mu| \geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2} \qquad\forall\ \epsilon>0.
\]
We say that the \emph{rate} at which $\bar{X}_n\to\mu$ is of order $O(1/n)$ as $n\to\infty$.
%
In the proof of the following theorem, we use Bernstein's inequality to show that the sample mean of Bernoulli random variables satisfies
\[
\prob\left(|\bar{X}_n - \mu| > \epsilon\right) \leq e^{-\frac{1}{2}n\epsilon^2}  \qquad\forall\ \epsilon>0.
\]
In this case, the rate at which $\bar{X}_n\to\mu$ as $n\to\infty$ is said to be \emph{exponentially fast}.

% theorem
\begin{theorem}[Bernoulli's Law of Large Numbers]\label{thm:bernoulli_lln}
Let $X_1,X_2,\ldots$ be independent, with each $X_i\sim\text{Bernoulli}(p)$, and let $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i$ be the sample mean of the first $n$ variables in the sequence.
Then for every $\epsilon > 0$,
\[
\prob(|\bar{X}_n - p| > \epsilon) \to 0 \text{\quad as\quad} n\to\infty.
\]
\end{theorem}

% proof
\begin{proof}
Let $\epsilon>0$ and define $S_n=\sum_{i=1}^n X_i$. Then
\[
\prob\left(\bar{X}_n - p > \epsilon\right)
	= \prob\big[S_n > n(p+\epsilon)\big]
\]

%\bit
%\it Recall Bernstein's inequality: $\prob(X>a) \leq e^{-ta}\expe(e^{tX})$ for all $t>0$.
%\eit

Applying Bernstein's inequality (Theorem~\ref{thm:bernstein}) to the random variable $S_n$ with $a = n(p+\epsilon)$,
\begin{align*}
\prob\big[S_n > n(p+\epsilon)\big]
	& \leq e^{-tn(p+\epsilon)}\expe(e^{tS_n}) \\
	& = e^{-tn(p+\epsilon)}\big[1-p+pe^t\big]^n \\
	& = e^{-tn\epsilon}\big[ e^{-tp}(1-p+pe^t) \big]^n \\
	& = e^{-tn\epsilon}\big[ (1-p)e^{-tp} + pe^{t(1-p)} \big]^n
\end{align*}

Using the inequality $e^x \leq x + e^{x^2}$, which holds for all $x\in\R$, % (see exercises),
\begin{align*}
(1-p)e^{-tp} + pe^{t(1-p)}
	& \leq (1-p)\big[-tp + e^{t^2p^2}\big] + p\big[t(1-p) + e^{t^2(1-p)^2}\big] \\
	& = (1-p)e^{t^2p^2} + pe^{t^2(1-p)^2} \\
	& \leq (1-p)e^{t^2} + pe^{t^2} \\
	& = e^{t^2}.
\end{align*}
Hence, for all $t>0$,
\[
\prob\left(\bar{X}_n -p > \epsilon\right) = \prob\big[S_n > n(p+\epsilon)\big] \leq e^{-tn\epsilon}e^{t^2n} = e^{tn(t-\epsilon)}.
\]
\bit
\it This inequality is valid for all $t>0$.
\it We choose $t$ so that the right-hand side is made as small as possible.
\it Because $e^x$ is an increasing function, this corresponds to the minimum value of the exponent.
\it We differentiate the exponent $tn(t-\epsilon)$ with respect to $t$ and set this equal to zero.
\eit

This yields the value $t = \frac{1}{2}\epsilon$, so
\[
\prob\left(\bar{X}_n -p > \epsilon\right) = \prob\big[S_n > n(p+\epsilon)\big] \leq e^{-\frac{1}{4}n\epsilon^2}.
\]
A similar argument shows that
\[
\prob\left(\bar{X}_n -p < -\epsilon\right) = \prob\big[S_n < n(p-\epsilon)\big] \leq e^{-\frac{1}{4}n\epsilon^2},
\]
Thus we have
\[
\prob\left(|\bar{X}_n - p| > \epsilon\right) \leq e^{-\frac{1}{2}n\epsilon^2} \to 0 \text{\quad as $n\to\infty$,}
\]
as required.
\end{proof}


% This LaTeX was auto-generated from an M-file by MATLAB.
% To make changes, update the M-file and republish this document.

\subsection*{error\_ellipse.m}

\begin{par}
ERROR\_ELLIPSE - plot an error ellipse, or ellipsoid, defining confidence region    ERROR\_ELLIPSE(C22) - Given a 2x2 covariance matrix, plot the    associated error ellipse, at the origin. It returns a graphics handle    of the ellipse that was drawn.
\end{par} \vspace{1em}

\begin{verbatim}  ERROR_ELLIPSE(C33) - Given a 3x3 covariance matrix, plot the
  associated error ellipsoid, at the origin, as well as its projections
  onto the three axes. Returns a vector of 4 graphics handles, for the
  three ellipses (in the X-Y, Y-Z, and Z-X planes, respectively) and for
  the ellipsoid.\end{verbatim}

\begin{verbatim}  ERROR_ELLIPSE(C,MU) - Plot the ellipse, or ellipsoid, centered at MU,
  a vector whose length should match that of C (which is 2x2 or 3x3).\end{verbatim}

\begin{verbatim}  ERROR_ELLIPSE(...,'Property1',Value1,'Name2',Value2,...) sets the
  values of specified properties, including:
    'C' - Alternate method of specifying the covariance matrix
    'mu' - Alternate method of specifying the ellipse (-oid) center
    'conf' - A value betwen 0 and 1 specifying the confidence interval.
      the default is 0.5 which is the 50% error ellipse.
    'scale' - Allow the plot the be scaled to difference units.
    'style' - A plotting style used to format ellipses.
    'clip' - specifies a clipping radius. Portions of the ellipse, -oid,
      outside the radius will not be shown.\end{verbatim}

\begin{verbatim}  NOTES: C must be positive definite for this function to work properly.\end{verbatim}
    \begin{par}
by AJ Johnson \begin{verbatim}http://www.mathworks.de/matlabcentral/fileexchange/4705\end{verbatim}
\end{par} \vspace{1em}

\begin{lstlisting}
function h=error_ellipse(varargin)
\end{lstlisting}

\subsection*{Code}

\begin{lstlisting}
default_properties = struct(...
  'C', [], ... % The covaraince matrix (required)
  'mu', [], ... % Center of ellipse (optional)
  'conf', 0.95, ... % Percent confidence/100
  'scale', 1, ... % Scale factor, e.g. 1e-3 to plot m as km
  'style', '', ...  % Plot style
  'clip', inf); % Clipping radius

if length(varargin) >= 1 & isnumeric(varargin{1})
  default_properties.C = varargin{1};
  varargin(1) = [];
end

if length(varargin) >= 1 & isnumeric(varargin{1})
  default_properties.mu = varargin{1};
  varargin(1) = [];
end

if length(varargin) >= 1 & isnumeric(varargin{1})
  default_properties.conf = varargin{1};
  varargin(1) = [];
end

if length(varargin) >= 1 & isnumeric(varargin{1})
  default_properties.scale = varargin{1};
  varargin(1) = [];
end

if length(varargin) >= 1 & ~ischar(varargin{1})
  error('Invalid parameter/value pair arguments.')
end

prop = getopt(default_properties, varargin{:});
C = prop.C;

if isempty(prop.mu)
  mu = zeros(length(C),1);
else
  mu = prop.mu;
end

conf = prop.conf;
scale = prop.scale;
style = prop.style;

if conf <= 0 | conf >= 1
  error('conf parameter must be in range 0 to 1, exclusive')
end

[r,c] = size(C);
if r ~= c | (r ~= 2 & r ~= 3)
  error(['Don''t know what to do with ',num2str(r),'x',num2str(c),' matrix'])
end

x0=mu(1);
y0=mu(2);

% Compute quantile for the desired percentile
k = sqrt(qchisq(conf,r)); % r is the number of dimensions (degrees of freedom)

hold_state = get(gca,'nextplot');

if r==3 & c==3
  z0=mu(3);

  % Make the matrix has positive eigenvalues - else it's not a valid covariance matrix!
  if any(eig(C) <=0)
    error('The covariance matrix must be positive definite (it has non-positive eigenvalues)')
  end

  % C is 3x3; extract the 2x2 matricies, and plot the associated error
  % ellipses. They are drawn in space, around the ellipsoid; it may be
  % preferable to draw them on the axes.
  Cxy = C(1:2,1:2);
  Cyz = C(2:3,2:3);
  Czx = C([3 1],[3 1]);

  [x,y,z] = getpoints(Cxy,prop.clip);
  h1=plot3(x0+k*x,y0+k*y,z0+k*z,prop.style,'linewidth',2);hold on
  [y,z,x] = getpoints(Cyz,prop.clip);
  h2=plot3(x0+k*x,y0+k*y,z0+k*z,prop.style,'linewidth',2);hold on
  [z,x,y] = getpoints(Czx,prop.clip);
  h3=plot3(x0+k*x,y0+k*y,z0+k*z,prop.style,'linewidth',2);hold on

  [eigvec,eigval] = eig(C);

  [X,Y,Z] = ellipsoid(0,0,0,1,1,1);
  XYZ = [X(:),Y(:),Z(:)]*sqrt(eigval)*eigvec';

  X(:) = scale*(k*XYZ(:,1)+x0);
  Y(:) = scale*(k*XYZ(:,2)+y0);
  Z(:) = scale*(k*XYZ(:,3)+z0);
  h4=surf(X,Y,Z);
  colormap gray
  alpha(0.3)
  camlight
  if nargout
    h=[h1 h2 h3 h4];
  end
elseif r==2 & c==2
  % Make the matrix has positive eigenvalues - else it's not a valid covariance matrix!
  if any(eig(C) <=0)
    error('The covariance matrix must be positive definite (it has non-positive eigenvalues)')
  end

  [x,y,z] = getpoints(C,prop.clip);
  h1=plot(scale*(x0+k*x),scale*(y0+k*y),prop.style,'linewidth',2);
  set(h1,'zdata',z+1)
  if nargout
    h=h1;
  end
else
  error('C (covaraince matrix) must be specified as a 2x2 or 3x3 matrix)')
end
%axis equal

set(gca,'nextplot',hold_state);

%---------------------------------------------------------------
% getpoints - Generate x and y points that define an ellipse, given a 2x2
%   covariance matrix, C. z, if requested, is all zeros with same shape as
%   x and y.
\end{lstlisting}

\begin{lstlisting}
function [x,y,z] = getpoints(C,clipping_radius)

n=100; % Number of points around ellipse
p=0:pi/n:2*pi; % angles around a circle

[eigvec,eigval] = eig(C); % Compute eigen-stuff
xy = [cos(p'),sin(p')] * sqrt(eigval) * eigvec'; % Transformation
x = xy(:,1);
y = xy(:,2);
z = zeros(size(x));

% Clip data to a bounding radius
if nargin >= 2
  r = sqrt(sum(xy.^2,2)); % Euclidian distance (distance from center)
  x(r > clipping_radius) = nan;
  y(r > clipping_radius) = nan;
  z(r > clipping_radius) = nan;
end

%---------------------------------------------------------------
function x=qchisq(P,n)
% QCHISQ(P,N) - quantile of the chi-square distribution.
if nargin<2
  n=1;
end

s0 = P==0;
s1 = P==1;
s = P>0 & P<1;
x = 0.5*ones(size(P));
x(s0) = -inf;
x(s1) = inf;
x(~(s0|s1|s))=nan;

for ii=1:14
  dx = -(pchisq(x(s),n)-P(s))./dchisq(x(s),n);
  x(s) = x(s)+dx;
  if all(abs(dx) < 1e-6)
    break;
  end
end

%---------------------------------------------------------------
function F=pchisq(x,n)
% PCHISQ(X,N) - Probability function of the chi-square distribution.
if nargin<2
  n=1;
end
F=zeros(size(x));

if rem(n,2) == 0
  s = x>0;
  k = 0;
  for jj = 0:n/2-1;
    k = k + (x(s)/2).^jj/factorial(jj);
  end
  F(s) = 1-exp(-x(s)/2).*k;
else
  for ii=1:numel(x)
    if x(ii) > 0
      F(ii) = quadl(@dchisq,0,x(ii),1e-6,0,n);
    else
      F(ii) = 0;
    end
  end
end

%---------------------------------------------------------------
function f=dchisq(x,n)
% DCHISQ(X,N) - Density function of the chi-square distribution.
if nargin<2
  n=1;
end
f=zeros(size(x));
s = x>=0;
f(s) = x(s).^(n/2-1).*exp(-x(s)/2)./(2^(n/2)*gamma(n/2));

%---------------------------------------------------------------
function properties = getopt(properties,varargin)
%GETOPT - Process paired optional arguments as 'prop1',val1,'prop2',val2,...
%
%   getopt(properties,varargin) returns a modified properties structure,
%   given an initial properties structure, and a list of paired arguments.
%   Each argumnet pair should be of the form property_name,val where
%   property_name is the name of one of the field in properties, and val is
%   the value to be assigned to that structure field.
%
%   No validation of the values is performed.
%
% EXAMPLE:
%   properties = struct('zoom',1.0,'aspect',1.0,'gamma',1.0,'file',[],'bg',[]);
%   properties = getopt(properties,'aspect',0.76,'file','mydata.dat')
% would return:
%   properties =
%         zoom: 1
%       aspect: 0.7600
%        gamma: 1
%         file: 'mydata.dat'
%           bg: []
%
% Typical usage in a function:
%   properties = getopt(properties,varargin{:})

% Process the properties (optional input arguments)
prop_names = fieldnames(properties);
TargetField = [];
for ii=1:length(varargin)
  arg = varargin{ii};
  if isempty(TargetField)
    if ~ischar(arg)
      error('Propery names must be character strings');
    end
    f = find(strcmp(prop_names, arg));
    if length(f) == 0
      error('%s ',['invalid property ''',arg,'''; must be one of:'],prop_names{:});
    end
    TargetField = arg;
  else
    % properties.(TargetField) = arg; % Ver 6.5 and later only
    properties = setfield(properties, TargetField, arg); % Ver 6.1 friendly
    TargetField = '';
  end
end
if ~isempty(TargetField)
  error('Property names and values must be specified in pairs.');
end
\end{lstlisting}


\chapter{Generalized eigenvectors maximize signal-to-noise}
\label{apx:GEvecs-maximise-SNR}

In this appendix, we show that the weight vector that maximizes the ratio of signal to noise variance (i.e. the solution $\what$ to \cref{eq:argmax_R}) is equivalent to the first generalized eigenvector $\w_1$ of the ordered pair of covariance matrices $(\Rss, \Rnn)$ (as defined in \cref{sec:generalized-eigenproblem}).

This ratio of variances in \cref{eq:argmax_R} is a quotient of quadratic forms, namely the so called ``generalized Rayleigh quotient'' of $(\Rss, \Rnn)$.

Formally, the generalized Rayleigh quotient of a non-zero vector $\w \in \reals^N$ and the ordered, symmetric matrix pair $(\A,\B)$ is the scalar $r(\w)$ defined as:
%
\begin{equation}
\label{eq:Rayleigh}
r(\w) = \frac{\w^T \A \w}
             {\w^T \B \w}
\end{equation}

Recall that a vector $\w_i$ is a generalized eigenvector of $(\A,\B)$, with corresponding generalized eigenvalue $\lambda_i$, if
%
\begin{equation}
\label{eq:generalized-eigenproblem-AB}
\A \w_i = \lambda_i \B \w_i
\end{equation}

We must then prove the following:

\section{Theorem}

The generalized eigenvector $\w_1$ corresponding to the largest generalized eigenvalue $\lambda_1$ of $(\A,\B)$, is also the vector $\what$ that maximises the generalized Rayleigh quotient $r(\w)$ of $(\A,\B)$.

\section{Proof}

As a first step, we will show that if $\what$ is the maximum of $r(\w)$, that it is indeed an eigenvector of $(\A,\B)$. In the second step, we will show that the largest eigenvalue $\lambda_1$ of $(\A,\B)$ corresponds to the maximum of $r(\w)$.

If $\what$ is a maximum of $r(\w)$, then
\begin{equation}
\label{eq:critical}
\grad{r(\what) = \vb{0}}.
\end{equation}

Working out the partial derivatives that comprise the gradient of $r(\w)$, we find:
\begin{align*}
\grad{r(\w)} &= \frac{2 \A \w \qty(\w^T \B \w)
                      - 2 \B \w \qty(\w^T \A \w)}
                     {\qty(\w^T \B \w)^2}
\end{align*}

With \cref{eq:critical}, we then have the following condition for our maximising vector $\what$:
\[
2 \A \what \qty(\what^T \B \what) = 2 \B \what \qty(\what^T \A \what)
\]
or
\begin{align*}
\A \what &= \frac{\what^T \A \what}
                 {\what^T \B \what} \; \B \what  \\[1em]
\A \what &= r(\what) \; \B \what
\end{align*}
This is the generalized eigenvalue/eigenvector definition (\cref{eq:generalized-eigenproblem-AB}) for $\w_i = \what$ and $\lambda_i = r(\what)$.

We have thus shown that if $\what$ is a maximum of $r(\w)$, that it is an eigenvector of $(\A,\B)$, with $r(\what)$ its corresponding eigenvalue.

As the second step, we now show that $r(\what)$ is the \emph{largest} eigenvalue of $(\A,\B)$. We follow the reasoning of \citeauthor{Trefethen1997}, who prove a related result for the ordinary Rayleigh quotient \cite[p. 204]{Trefethen1997}.

We will rewrite the generalized Rayleigh quotient $r(\w)$ by writing the arbitrary vector $\w$ as a linear combination of the generalized eigenvectors $\w_i$ of $(\A,\B)$: $\w = \sum_i c_i \w_i$. Then:
\begin{align*}
r(\w) &= \frac{\qty(\sum_i c_i \w_i)^T \A \qty(\sum_i c_i \w_i)}
              {\qty(\sum_i c_i \w_i)^T \B \qty(\sum_i c_i \w_i)} \\[1em]
              %
      &= \frac{\sum_i c_i^2 \w_i^T \A \w_i}
              {\sum_i c_i^2 \w_i^T \B \w_i} \\[1em]
              %
      &= \frac{\sum_i c_i^2 \lambda_i \w_i^T \B \w_i}
              {\sum_i c_i^2 \w_i^T \B \w_i}.
\end{align*}

Generalized eigenvectors are defined up to a scaling factor. We may therefore define our $\w_i$ to be scaled such that $\w_i^T \B \w_i = 1$. We then have:
\[
r(\w) = \frac{\sum_i c_i^2 \lambda_i}{\sum_i c_i^2}.
\]
%
Each generalized Rayleigh quotient is thus a convex combination of generalized eigenvalues $\lambda_i$. The maximum of a convex combination of one-dimensional points is obtained in the largest of these points. If $\lambda_1$ is thus the largest generalized eigenvalue of $(\A,\B)$, then $\max{r(\w)} = \lambda_1$.

We have thus shown that $\argmax r(\w) = \w_1$, where $\w_1$ is an eigenvector of $(\A,\B)$, and that its corresponding eigenvalue $\lambda_1 = \max r(\w)$ is the largest of the eigenvalues of $(\A,\B)$.

\qed


\section{Duality in Optimization}

\subsection{The Primal Problem}

\begin{frame}
  \frametitle{The Primal Problem}

  \begin{itemize}
    \item Consider the \structure{\emph{primal optimization problem}}: \\[.4cm]
      \begin{center}
        \tikz[baseline]{
          \node[fill=bl1!100,anchor=base,rounded corners=3pt] (d1) {
            \color{bl3}
            $\begin{aligned}
               \displaystyle
               \mbox{minimize~}  & \qquad f_0(\vec{x}) \\[.3cm]
               \mbox{subject to} & \qquad f_i(\vec{x}) \leq 0, \quad i=1,2,\dots, m \\
                                 & \qquad h_i(\vec{x}) = 0,    \quad i=1,2,\dots, p
             \end{aligned}$
          };
        }\\[.4cm]
      \end{center}
      with variable $\vec{x}\in\real^n$. \\[.5cm]
    \item The function $f_0(\vec{x})$ is \structure{not} required to be \structure{convex}.
  \end{itemize}
\end{frame}

\subsection{The Lagrangian}

\begin{frame}
  \frametitle{The Lagrangian}

  \begin{citeblock}{Lagrangian}

    The \structure{\emph{Lagrangian} $L$} of the aforementioned problem is defined as
    \begin{displaymath}
      L(\vec{x},\vec{\lambda}, \vec{\nu}) =
      f_0(\vec{x}) + \sum_{i=1}^m \lambda_i f_i(\vec{x}) + \sum_{i=1}^p \nu_i h_i(\vec{x})
    \end{displaymath}
    \pause

    \begin{itemize}
      \item $\lambda_i$ is the Lagrange multipliers associated with the $i$-th \structure{inequality} \\
        constraint $f_i(\vec{x})\leq 0$. \\[.15cm] \pause
      \item $\nu_i$ is the Lagrange multiplier associated with the $i$-th \structure{equality} constraint $h_i(\vec{x})= 0$. \\[.15cm] \pause
      \item The vectors $\vec{\lambda}$ and $\vec{\nu}$ are called {\em Lagrange multiplier vectors} \\
        or simply \structure{\emph{dual variables}}.
    \end{itemize}
  \end{citeblock}
\end{frame}

\subsection{Lagrange Dual Function}

\begin{frame}
  \frametitle{Langrange Dual Function}

  \begin{citeblock}{Lagrange dual function}

     The \structure{\emph{Lagrange dual function}} is defined as the infimum of the Lagrangian \\
     over $\vec{x}$
     \begin{eqnarray*}
       g(\vec{\lambda,\nu})
         & = & \inf_{\vec{x}}L(\vec{x},\vec{\lambda}, \vec{\nu}) \\ \pause
         & = & \inf_{\vec{x}} \left(
                                f_0(\vec{x}) + \sum_{i=1}^m\lambda_i f_i(\vec{x}) +
                                \sum_{i=1}^p\nu_i h_i(\vec{x})
                              \right)
    \end{eqnarray*}
  \end{citeblock}
  \pspread

  \structure{Note:}

  \begin{itemize}
    \item The Lagrange dual function is a \structure{pointwise affine function} \\
      in the dual variables. \pause
    \item The \structure{Lagrange dual function is concave} \\
      (even if the original problem is not convex).
  \end{itemize}

\end{frame}

\begin{frame}{Optimal Value and Lower Bound}

  \begin{lemma}
    Let $p^*$ be the optimal value of the optimization problem. \\
    For any $\vec\lambda \succeq 0$ and any $\vec \nu$ the following bound is valid:
    \begin{displaymath}
      g(\vec{\lambda,\nu}) \leq p^*
    \end{displaymath}
  \end{lemma}
\end{frame}

\begin{frame}{Optimal Value and Lower Bound \cont}

  \begin{center}
    \resizebox{.8\linewidth}{!}{
      \alt<13->{
        \input{\texfigdir/lagrange13.pstex_t}
      }{\alt<12>{
        \input{\texfigdir/lagrange12.pstex_t}
      }{\alt<11>{
        \input{\texfigdir/lagrange11.pstex_t}
      }{\alt<10>{
        \input{\texfigdir/lagrange10.pstex_t}
      }{\alt<9>{
        \input{\texfigdir/lagrange09.pstex_t}
      }{\alt<8>{
        \input{\texfigdir/lagrange08.pstex_t}
      }{\alt<7>{
        \input{\texfigdir/lagrange07.pstex_t}
      }{\alt<6>{
        \input{\texfigdir/lagrange06.pstex_t}
      }{\alt<5>{
        \input{\texfigdir/lagrange05.pstex_t}
      }{\alt<4>{
        \input{\texfigdir/lagrange04.pstex_t}
      }{\alt<3>{
        \input{\texfigdir/lagrange03.pstex_t}
      }{\alt<2>{
        \input{\texfigdir/lagrange02.pstex_t}
      }{
        \input{\texfigdir/lagrange01.pstex_t}
      }}}}}}}}}}}}
    }
  \end{center}
\end{frame}

\begin{frame}{Optimal Value and Lower Bound \cont}

  \begin{columns}
    \column{.6\linewidth}
      \begin{center}
        \resizebox{\linewidth}{!}{
          \input{\texfigdir/dual.pstex_t}
        }
      \end{center}
    \column{.4\linewidth}
      \vspace{.3cm}
      \begin{itemize}
        \item Neither $f_0(x)$ nor $f_1(x)$ is convex,
        \item but the dual function $g(\lambda)$ is concave!
      \end{itemize}
  \end{columns}
\end{frame}

\begin{frame}
  \frametitle{Optimal Value and Lower Bound \cont}

  Let $\tilde{\vec{x}}$ be a \structure{feasible point} of the optimization problem. \\[.3cm]
  If $\vec{\lambda}\succeq 0$, we have due to the \structure{$m$ inequality} and \structure{$p$ equality} constraints:
  \begin{displaymath}
    \sum_{i=1}^m \lambda_i f_i(\tilde{\vec{x}}) +
    \sum_{i=1}^p\nu_i h_i(\tilde{\vec{x}})
    \leq 0~,
  \end{displaymath}
  \pause

  Thus we have
  \begin{displaymath}
    L(\tilde{\vec{x}},\vec{\lambda},\vec{\nu}) =
    f_0 ({\tilde{\vec{x}}}) +
    \sum_{i=1}^m\lambda_i f_i(\tilde{\vec{x}}) +
    \sum_{i=1}^p\nu_i h_i(\tilde{\vec{x}})
    \leq f_0 ({\tilde{\vec{x}}})~.
  \end{displaymath}
\end{frame}

\begin{frame}
  \frametitle{Optimal Value and Lower Bound \cont}

  Using the \structure{definition of the dual function} we get:

  \begin{displaymath}
    g(\vec{\lambda,\nu}) =
    \inf_{{\vec{x}}} L({\vec{x}},\vec{\lambda}, \vec{\nu}) \leq
    L(\tilde{\vec{x}}, \vec{\lambda}, \vec{\nu}) \leq
%    f_0(\tilde{\vec{x}}) +
%    \sum_{i=1}^m\lambda_i f_i(\tilde{\vec{x}}) +
%    \sum_{i=1}^p\nu_i h_i(\tilde{\vec{x}}) \leq
    f_0(\tilde{\vec{x}})
  \end{displaymath}
  \pause

  \vspace{.25cm}
  The inequality $g(\vec{\lambda,\nu}) \leq f_0(\tilde{\vec{x}})$ holds for \structure{every feasible point $\tilde{\vec{x}}$}.\\[.5cm]
  \pause

  Consequently, the dual function $g(\vec{\lambda,\nu})$ is also smaller or equal to the optimal value $p^*$:

  \begin{displaymath}
    g(\vec{\lambda,\nu}) \leq p^*
  \end{displaymath}
  \hfill \qed %\structure{\ensuremath{\blacksquare}}
\end{frame}

\subsection{The Lagrange Dual Problem}

\begin{frame}
  \frametitle{The Lagrange Dual Problem}

  \structure{Problem:} how to find the best lower bound for the primal problem \\[1cm]

  The Lagrange dual problem is given by the optimization problem:

  \begin{eqnarray*}
    \mbox{maximize}   & & g(\vec{\lambda}, \vec{\nu}) \\[.3cm]
    \mbox{subject to} & & \vec{\lambda}\succeq 0
  \end{eqnarray*}
\end{frame}

\begin{frame}
  \frametitle{The Lagrange Dual Problem \cont}

  \begin{citeblock}{Optimal duality gap}

    Let  ${p}^*$ be the optimal value of the primal problem and \\
    ${d}^*$ the optimal value of the Lagrange dual problem. \\[.3cm] \pause

    \begin{itemize}
      \item The difference $p^*-d^*$ is the \structure{\emph{optimal duality gap}}. \\[.3cm] \pause
      \item If $p^*=d^*$, the duality gap is zero. \\
        In this case we talk about \structure{\emph{strong duality}}. \\[.3cm] \pause
      \item If $p^*> d^*$, we have \structure{\emph{weak duality}}.
     \end{itemize}
  \end{citeblock}
\end{frame}

\subsection{Slater's condition}

\begin{frame}
  \frametitle{Slater's Condition}

  \begin{citeblock}{Theorem}

    Given a \structure{\emph{convex} primal optimization problem}:
    \small
    \begin{eqnarray*}
      \mbox{minimize~}  & & \quad f_0(\vec{x}) \\
      \mbox{subject to} & & \quad f_i(\vec{x}) \leq 0, \quad i=1,2,\dots, m \\
                        & & \quad \mat{A} \vec{x} = \vec{b}
    \end{eqnarray*}
    \normalsize
    with $f_0, f_1, \ldots, f_m$ being convex. \\[.3cm]

    If there exists an $\vec{x} \in \text{relint}~\big\{\mathcal{D} = \cap_{i=0}^m \text{dom}(f_i) \big\}$ with
    \small
    \begin{align*}
      & f_i(\vec{x}) < 0, \quad i=1,\dots,m \\
      & \mat{A}\vec{x} = \vec{b}
    \end{align*}
    then \structure{strong duality} holds.
  \end{citeblock}
\end{frame}

\begin{frame}
  \frametitle{Refinement of Slater's Condition}

  \begin{citeblock}{Theorem}

    Given a \structure{\emph{convex} primal optimization problem}. \\[.2cm]

    If the first $k$ constraint functions $f_1, \ldots, f_k$ are \structure{\emph{affine}}, and \\
    if there exists an $\vec{x} \in \text{relint}~\mathcal{D}$ with
    \small
    \begin{displaymath}
      \renewcommand\arraystretch{1.4}
      \begin{array}{l l}
        f_i(\vec{x}) \le 0, \quad i=1,\dots,k                   & \text{\footnotesize (affine constraints)}\\
        f_i(\vec{x}) <   0, \quad i=k+1, \ldots, m \qquad \quad & \text{\footnotesize (convex constraints)}\\
        \mat{A}\vec{x} = \vec{b} & \\
      \end{array}
    \end{displaymath}
    then \structure{strong duality} holds.
  \end{citeblock}
  \pause

  \vspace{.3cm}
  \structure{Note:} the refined Slater's condition reduces to \structure{feasibility} when the constraints are all linear equalities and inequalities, and $\text{dom}(f_0)$ is open.
\end{frame}

\subsection{KKT Optimality Conditions}

\begin{frame}
  \frametitle{Karush-Kuhn-Tucker Optimality Conditions}

  Let $\vec x^*$ be a primal and $(\vec \lambda^*, \vec \nu^*)$ dual optimal points with zero duality gap. \\[.3cm]

  For the primal optimal point $\vec x^*$, the gradient with respect to $\vec x$ of $L(\vec x, \vec \lambda^*, \vec \nu^*)$ is \vec{0}:

  \begin{displaymath}
    \nabla L(\vec x^*, \vec \lambda^*, \vec \nu^*) =
    \nabla f_0(\vec{x}^*) +
    \sum_{i=1}^m \lambda_i^* \nabla f_i(\vec{x}^*) +
    \sum_{i=1}^p \nu_i^* \nabla h_i(\vec{x}^*) =
    \vec{0}
  \end{displaymath}
\end{frame}

\begin{frame}
  \frametitle{Karush-Kuhn-Tucker Optimality Conditions}

  The following four conditions are called KKT conditions: \\[3mm] \pause

  \begin{enumerate}
    \item \structure{Primal constraints:}
      \begin{itemize}
        \item $f_i(\vec{x}) \leq 0, \quad i = 1, 2, \dots, m$
        \item $h_i(\vec{x}) =    0, \quad i = 1, 2, \dots, p$ \\[3mm] \pause
      \end{itemize}
    \item \structure{Dual constraints:} $\vec \lambda \succeq 0$ \\[3mm] \pause
    \item \structure{Complementary slackness:} $ \lambda_i \, f_i(\vec{x})= 0$ \\[3mm] \pause
    \item \structure{Gradient of the Lagrangian $L$ is zero:}
      \begin{displaymath}
        \nabla L(\vec x, \vec \lambda, \vec \nu) =
        \nabla f_0(\vec{x}) +
        \sum_{i=1}^m\lambda_i \nabla f_i(\vec{x}) +
        \sum_{i=1}^p\nu_i \nabla h_i(\vec{x}) =
        \vec{0}
      \end{displaymath}
  \end{enumerate}
   \pause

   If strong duality holds and if $\vec x^*$ and $(\vec \lambda^*, \vec \nu^*)$ are optimal points, \\
   then the KKT conditions hold.
\end{frame}

\begin{frame}
  \frametitle{Karush-Kuhn-Tucker Optimality Conditions}

  \alt<15->{
    \structure{Complementary slackness:} {\color{gr3} $\lambda_i^* \cdot f_i(\vec{x}^*) = 0$}
  }{
    \structure{Complementary slackness}
  }

  \def\phan{\phantom{\stackrel{\color{red} =}{\cancel{\color{black} \le}}}}

  \begin{eqnarray*}
    \alt<10->{
      \tikz[baseline]{
        \node[fill=bl1,anchor=base,rounded corners=2pt] (start) {
          \color{bl3} $ f_0(\vec{x}^*) $
        };
      }
    }{
      \tikz[baseline]{
        \node[fill=white,anchor=base,rounded corners=2pt] (res) {
          \color{black} $ f_0(\vec{x}^*) $
        };
      }
    }
    \pause
      & = & g(\vec{\lambda}^*, \vec{\nu}^*) \\ \pause
      & = & \inf_{\vec{x}} \Bigg( f_0(\vec{x}) + \sum_{i=1}^m \lambda_i^*  f_i(\vec{x}) + \sum_{i=1}^p \nu_i^* h_i(\vec{x}) \Bigg) \\ \pause
      \alt<12->{
        & \stackrel{\color{red} =}{\cancel{\color{black} \le}} &
      }{
        &\le&
      }
        f_0(\vec{x}^*) +
        \alt<15>{
          \tikz[baseline,remember picture]{
            \node[fill=gr1,anchor=base,rounded corners=2pt] (c1) {
              \color{gr3} $ \displaystyle \sum_{i=1}^m $
              \tikz \node[fill=white,anchor=base, rounded corners=2pt] (cs) {\color{gr3} $ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.35cm}, decorate, color=gr3] (cs.west) -- node[pos=0.5,xshift=0mm,yshift=-5.5mm]  {
              \tiny $= 0$
            } (cs.east);
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=black] (c1.west) -- node[pos=0.5,xshift=.4mm,yshift=-1.1cm]  {
              \scriptsize $\stackrel{\color{red}=}{\cancel{\color{black}\le}} 0$
            } (c1.east);
          }
        }{\alt<14>{
          \tikz[baseline,remember picture]{
            \node[fill=gr1,anchor=base,rounded corners=2pt] (c1) {
              \color{gr3} $ \displaystyle \sum_{i=1}^m $
              \tikz \node[anchor=base, rounded corners=2pt] (cs) {\color{gr3} $ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=black] (c1.west) -- node[pos=0.5,xshift=.4mm,yshift=-1.1cm]  {
              \scriptsize $\stackrel{\color{red}=}{\cancel{\color{black}\le}} 0$
            } (c1.east);
          }
        }{\alt<13>{
          \tikz[baseline,remember picture]{
            \node[fill=white,anchor=base,rounded corners=2pt] (c1) {
              $ \displaystyle \sum_{i=1}^m $
              \tikz \node[anchor=base, rounded corners=2pt] (cs) {$ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=black] (c1.west) -- node[pos=0.5,xshift=.4mm,yshift=-1.1cm]  {
              \scriptsize $\stackrel{\color{red}=}{\cancel{\color{black}\le}} 0$
            } (c1.east);
          }
        }{\alt<9->{
          \tikz[baseline,remember picture]{
            \node[fill=white,anchor=base,rounded corners=2pt] (c1) {
              $ \displaystyle \sum_{i=1}^m$
              \tikz \node[anchor=base, rounded corners=2pt] (cs) {$ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=black] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize $\phan\le 0\phan$} (c1.east);
          }
        }{\alt<8>{
          \tikz[baseline,remember picture]{
            \node[fill=bl1,anchor=base,rounded corners=2pt] (c1) {
              \color{bl3} $ \displaystyle \sum_{i=1}^m$
              \tikz \node[fill=bl1, anchor=base, rounded corners=2pt] (cs) {$ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=bl3] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize $\phan\le 0\phan$} (c1.east);
          }
        }{\alt<7>{
          \tikz[baseline,remember picture]{
            \node[fill=bl1,anchor=base,rounded corners=2pt] (c1) {
              \color{bl3} $ \displaystyle \sum_{i=1}^m$
              \tikz \node[fill=bl1, anchor=base, rounded corners=2pt] (cs) {$ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=white] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize \phantom{$\le 0$}} (c1.east);
          }
        }{
          \tikz[baseline,remember picture]{
            \node[fill=white,anchor=base,rounded corners=2pt] (c1) {
              \color{black} $ \displaystyle \sum_{i=1}^m $
              \tikz \node[anchor=base, rounded corners=2pt] (cs) {$ \lambda_i^*  f_i(\vec{x}^*) $};
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=white] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize \phantom{$\le 0$}} (c1.east);
          }
        }}}}}}
        +
        \alt<7->{
          \tikz[baseline]{
            \node[fill=white,anchor=base,rounded corners=2pt] (c1) {
              $ \displaystyle \sum_{i=1}^p \nu_i^* h_i(\vec{x}^*) $
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=black] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize $= 0$} (c1.east);
          }
        }{\alt<6->{
          \tikz[baseline]{
            \node[fill=bl1!100,anchor=base,rounded corners=2pt] (c1) {
              \color{bl3} $ \displaystyle \sum_{i=1}^p \nu_i^* h_i(\vec{x}^*) $
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=bl3] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize $= 0$} (c1.east);
          }
        }{\alt<5>{
          \tikz[baseline]{
            \node[fill=bl1!100,anchor=base,rounded corners=2pt] (c1) {
              \color{bl3} $ \displaystyle \sum_{i=1}^p \nu_i^* h_i(\vec{x}^*) $
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=white] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize \phantom{$= 0$}} (c1.east);
          }
        }{
          \tikz[baseline]{
            \node[fill=white,anchor=base,rounded corners=2pt] (c1) {
              \color{black} $ \displaystyle \sum_{i=1}^p \nu_i^* h_i(\vec{x}^*) $
            };
            \draw [thick, decoration={brace, mirror, raise=0.75cm}, decorate, color=white] (c1.west) -- node[pos=0.5,yshift=-1.1cm]  {\scriptsize \phantom{$= 0$}} (c1.east);
          }
        }}} \\
      \alt<11->{
        & \stackrel{\color{red} =}{\cancel{\color{black} \le}} &
      }{\alt<9->{
        &\le&
      }{}}
      \alt<10->{
        \tikz[baseline]{
          \node[fill=bl1,anchor=base,rounded corners=2pt] (res) {
            \color{bl3} $ f_0(\vec{x}^*) $
          };
        }
      }{\alt<9>{
        \tikz[baseline]{
          \node[fill=white,anchor=base,rounded corners=2pt] (res) {
            \color{black} $ f_0(\vec{x}^*) $
          };
        }
      }{}}
      \phan
  \end{eqnarray*}

  \onslide<15>
    \hfill \qed
\end{frame}

\begin{frame}
  \frametitle{Karush-Kuhn-Tucker Optimality Conditions \cont}

  \structure{Conclusions \scriptsize ~(Boyd 2004, Sec. 5.5.3)}

  \begin{itemize}
    \item For \structure{\emph{any} optimization problem} with differentiable objective and constraint functions for which strong duality obtains, any pair of primal and dual optimal points must satisfy the KKT conditions. \\[.2cm] \pause
    \item For \structure{any \emph{convex} optimization problem} with differentiable objective and and constraint functions, any points that satisfy the KKT conditions are primal and dual optimal, and have zero duality gap. \\[.2cm] \pause
    \item If a \structure{\emph{convex} optimization problem} with differentiable objective and constraint functions satisfies Slater's condition, then the KKT conditions provide necessary and sufficient conditions for optimality.
  \end{itemize}
\end{frame}

\subsection{Lessons Learned}

\begin{frame}
  \frametitle{Lessons Learned}

  \begin{itemize}
    \item Formalization of the primal problem using the Lagrangian \\[.5cm]
    \item Lagrange dual function \\[.5cm]
    \item Duality gap \\[.5cm]
    \item Karush-Kuhn-Tucker optimality conditions
  \end{itemize}
\end{frame}

\input{nextTime.tex}

\subsection{Further Readings}

\begin{frame}
  \frametitle{Further Readings}

  \begin{itemize}
    \item S.~Boyd, L.~Vandenberghe: \\
      \structure{Convex Optimization}, \\
      Cambridge University Press, 2004. \\
      \point{\small \url{http://www.stanford.edu/~boyd/cvxbook/}} \\[.25cm]
    \item Jorge Nocedal, Stephen Wright: \\
      \structure{Numerical Optimization}, \\
      Springer, New York, 1999.
  \end{itemize}
\end{frame}

\subsection{Comprehensive Questions}

\begin{frame}
  \frametitle{Comprehensive Questions}

  \begin{itemize}
    \item What is the Lagrangian of a constrained objective function? \\[1cm]
    \item What is the Lagrange dual function? \\[1cm]
    \item What is the duality gap? \\[1cm]
    \item What are the Karush-Kuhn-Tucker optimality conditions?
  \end{itemize}
\end{frame}


\section{Co-Synthesis of Encodings} \label{sec:co-synthesis}
%
The functions described in the previous section only makes
sense in the context of a fixed encoding, but in general, a
data type accepts more than one.
%
As encodings are just expressions we would like to
automatically derive them as a result of synthesis.
%
To do so we first need to find a way to give an abstract
specification of the meaning of a data type and we need to
do it in a way that is independent of an underlying
encoding.
%
In the following, we see how we can extend the ideas
presented in \cref{sec:synthesis} to express these abstract
specifications in a natural way.

In general, our approach consist of specifying the behavior
of data types in relation with the functions that operates
on them.
%
We call this a \emph{co-specification} of a data type and
its functions.
%
We also refer to the process of synthesizing an encoding
together with the functions that operate on it as
\emph{co-synthesis}.

Our next key insight is that we can build natural and
concise co-specifications by looking at the introduction and
elimination forms of a data type.
%
Consider booleans for example.
%
Their introduction corresponds to the values \true and \false
and their elimination corresponds to case analysis using the
if-then-else (\f{ite}) function.
%
Using the ideas presented in \cref{sec:io} we can co-specify
\true, \false and \f{ite} as:
%
\begin{align*}
  \f{?ite}\;\f{?true}\;x\;y &\equiv x \\
  \f{?ite}\;\f{?false}\;x\;y &\equiv y
\end{align*}

The co-specification for natural numbers requires a little
bit more work.
%
First, we identify $0$ and \f{succ} as the introduction
forms of natural numbers in its usual inductive definition.
%
Second, we note that the elimination form of natural numbers
correspond to the ability of defining recursive functions on
them.
%
Instead of spelling out a general recursive principle we
specify the recursive behavior by picking a particular
recursive definition.
%
In our case we specify the elimination form in terms of the
recursive definition of the function \f{plus}.
%
We can now co-specify $0$, \f{succ} and \f{plus} as:
\begin{align*}
  \f{?plus}\;\f{?}0\;y &\equiv y\\
  \f{?plus}\;(\f{?succ}\;x)\;y &\equiv \f{?succ}\;(\f{?plus}\;x\;y)\\
\end{align*}

We conclude by showing how these ideas also apply to
non-recursive data types by showing a co-specification for pairs.
%
The introduction form corresponds to the constructor
\f{pair} and the elimination forms to the first and
second projections.
%
\begin{align*}
  \f{?first}\;(\f{?pair}\;x\;y) &\equiv x\\
  \f{?second}\;(\f{?pair}\;x\;y) &\equiv y\\
\end{align*}