\lab{Data Structures II}{Data Structures II}
\label{lab:Python_DataStructures2}

\objective{Implement tree data structures and understand their relative strengths and weaknesses.}

\section*{Recursion}

Recursion is an important problem solving technique in computer programming.
A recursive function is one that calls itself.
When the function is executed, it continues calling itself until it reaches a specified base case.
Then the function exits without calling itself again, and each previous function call is resolved.
As a simple example, suppose we want to recursively sum all positive integers from $1$ to some integer $n$.

\begin{lstlisting}
def recursive_sum(n):
	"""Calculate the sum of all positive integers in [1, n] recursively."""

	# Typically the base case comes first. There are no positive integers less
	# than 1, so if 'n' is 1 we stop the recursion and return 1 (since the sum of
	# all integers in [1, 1] is 1).
	if n == 1:
		return 1

	# If the base case hasn't been reached, the function recurses by calling
	# itself on the next smallest integer and adding 'n'.
	else:
		return n + recursive_sum(n-1)
\end{lstlisting}

The computer calculates \li{recursive_sum(5)} with a sequence of function calls.

\begin{lstlisting}
# To find the recursive_sum(5), we need to calculate recursive_sum(4).
# But to find recursive_sum(4), we need to calculate recursive_sum(3).
# This continues until the base case is reached.

recursive_sum(5)		# return 5 + recursive_sum(4)
	recursive_sum(4)		# return 4 + recursive_sum(3)
		recursive_sum(3)		# return 3 + recursive_sum(2)
			recursive_sum(2)		# return 2 + recursive_sum(1)
				recursive_sum(1)		# Base case: return 1.
\end{lstlisting}

Now that we've reached a base case, we can unwind the recursion.
Reading from bottom to top, we substitute the values that result from each function call.

\begin{lstlisting}
recursive_sum(5)		# 5 + 10 = 15
	recursive_sum(4)		# 4 + 6 = 10
		recursive_sum(3)		# 3 + 3 = 6
			recursive_sum(2)		# 2 + 1 = 3
				recursive_sum(1)		# Base case: return 1.
\end{lstlisting}

So \li{recursive_sum(5)} returns 15 (which is correct, since $1 + 2 + 3 + 4 + 5 = 15$).
Many problems that can be solved by iterative methods can also be solved (often more efficiently) with a recursive approach.
Compare, for example, the following two methods for calculating the $n^{th}$ Fibonacci number.

\begin{lstlisting}
def iterative_fib(n):
	"""Calculate the nth Fibonacci number iteratively."""
	fibonacci = list()		# Initialize an empty list.
	fibonacci.append(0)		# append 0 (the 0th Fibonacci number).
	fibonacci.append(1)		# append 1 (the 1st Fibonacci number).
	for i in range(1, n):
		# Starting at the third entry, calculate the next number
		# by adding the last two entries in the list.
		fibonacci.append(fibonacci[-1] + fibonacci[-2])
	# When the entire list has been loaded, return the nth entry.
	return fibonacci[n]

def recursive_fib(n):
	"""Calculate the nth Fibonacci number recursively."""
	# The base cases are the first two Fibonacci numbers.
	if n == 0:				# Base case 1: the 0th Fibonacci number is 0.
		return 0
	elif n == 1:			# Base case 2: the 1st Fibonacci number is 1.
		return 1
	# If we haven't reached a base case, the function recurses by calling
	# itself on the previous two Fibonacci numbers.
	else:
		return recursive_fib(n-1) + recursive_fib(n-2)
\end{lstlisting}

This time, the sequence of function calls is slightly more complicated because \li{recursive_fib} calls itself twice until a base case is reached.

\begin{lstlisting}
recursive_fib(5)		# The original call makes two additional calls:
	recursive_fib(4)		# this one...
		recursive_fib(3)
			recursive_fib(2)
				recursive_fib(1)		# Base case 2: return 1
				recursive_fib(0)		# Base case 1: return 0
			recursive_fib(1)		# Base case 2: return 1
		recursive_fib(2)
			recursive_fib(1)		# Base case 2: return 1
			recursive_fib(0)		# Base case 1: return 0
	recursive_fib(3)		# ...and this one.
		recursive_fib(2)
			recursive_fib(1)		# Base case 2: return 1
			recursive_fib(0)		# Base case 1: return 0
		recursive_fib(1)		# Base case 2: return 1
\end{lstlisting}

The sum of all of the base case results, from top to bottom, is $1 + 0 + 1 + 1 + 0 + 1 + 0 + 1 = 5$, so \li{recursive_fib(5)} returns 5 (correctly).
The key to recursion is understanding the base cases correctly and making correct recursive calls.

% Problem 1: simple recursion for linked lists traversal
\begin{problem}
Rewrite the following iterative function for finding data in a linked list using recursion.
Use a \li{LinkedList} object from the previous lab to test the function.
\begin{lstlisting}
# solutions.py

def iterative_search(linkedlist, data):
	current = linkedlist.head
	while current is not None:
		if current.data == data:
			return current
		current = current.next
	raise ValueError(str(data) + " is not in the list.")
\end{lstlisting}
(Hint: define an inner function to perform the actual recursion)
\end{problem}

\begin{warn}
It is not always better to rewrite an iterative method recursively.
In Python, a function may only call itself 999 times.
On the $1000^{th}$ call, a \li{RuntimeError} is raised to prevent a stack overflow.
Whether or not recursion is appropriate depends on the problem to be solved and the algorithm to solve it.
\end{warn}

\section*{Trees}

A \emph{tree} data structure is a specialized linked list.
Trees are more difficult to build than standard linked lists, but they are almost always more efficient.
While the computational complexity of finding a node in a linked list is $O(n)$, a well-built, balanced tree will find a node with a complexity of $O(\log{n})$.
Some types of trees can be constructed quickly but take longer to retrieve data, while others take more time to build and less time to retrieve data.

The first node in a tree is called the \emph{root}.
The root node points to other nodes, called children.
Each child node in turn points to its children.
This continues on each branch until its end is reached.
A node with no children is called a \emph{leaf node}.

Mathematically, a tree is a directed graph with no cycles.
Therefore a linked lists qualifies as a tree, albeit a boring one.
The head node is the root node, and it has one child node.
That child node also has one child node, which in turn has one child.
This continues until the end of the list, with the last node as the only leaf node.

Other kinds of trees may be more complicated.
See Figure \ref{fig:tree}.

\begin{figure}
\caption{Examples of trees, some of which are binary search trees.}
\label{fig:trees}
\end{figure}

\section*{Binary Search Trees}

A \emph{binary search tree} (BST) data structure is a tree that allows each node to have up to two children, called \li{left} and \li{right}.
The left child of a node contains data that is less than its parent node's data.
The right child's data is greater.
Conceptually, each level of a BST partitions the dataset into two halves.
For any given node, each node in the left subtree contains data that is less than the data in the top node, and each node in the right subtree contains data that is greater than the original node's data.

% TODO: make sure this statement is consistent with the figure
The far right tree in Figure \ref{fig:trees} is an example of a binary search tree.
In practice, binary search tree nodes have attributes that keep track of their data, children, and (in doubly-linked trees) their parent.

% self.prev instead of self.parent for consistency
\begin{lstlisting}
# BST.py

class BSTNode(object):
    """A Node class for Binary Search Trees. Contains some data, a
    reference to the parent node, and references to two child nodes.
    """
    def __init__(self, data):
        """Construct a new node and set the data attribute. The other
        attributes will be set when the node is added to a tree.
        """
        self.data = data
        self.prev = None        # A reference to this node's parent node.
        self.left = None        # This node's data will be less than self.data
        self.right = None       # This node's data will be greater than self.data

    def __str__(self):
        """String representation: the data contained in the node."""
        return str(self.data)
\end{lstlisting}

The actual binary search tree class has an attribute pointing to its root.

\begin{lstlisting}
# BST.py

class BST(object):
    """Binary Search Tree data structure class.
    The first node is referenced to by 'root'.
    """
    def __init__(self):
        """Initialize the root attribute."""
        self.root = None
\end{lstlisting}

\subsection*{Finding in a Binary Search Tree}

Many tree algorithms are best understood and implemented using recursion.
For instance, finding a node in a binary search tree can be done recursively.
Starting at the root, we check if the data we are looking for matches the current node.
If it does not, then if the data is less than the current node's data we search again on the left child.
If the data is greater, we search on the right child.
This process continues until the data is found or, if the data is not in the tree, an empty child is searched.
See Figure \ref{fig:BST.find} for an example.
Carefully read the following code; similar techniques will be used for subsequent methods.

\begin{lstlisting}
# BST.py

def find(self, data):
    """Return the node containing 'data'. If there is no such node in the
    tree, raise a ValueError with error message "<data> is not in the tree."
    """
    # First, check to see if the tree is empty.
    if self.root is None:
        raise ValueError(str(data) + " is not in the tree.")

    # Define a recursive function to traverse the tree.
    def _step(current, item):
        """Recursively step through the tree until the node containing
        'item' is found. If there is no such node, raise a Value Error.
        """
        if current is None:                     # Base case 1: dead end.
            raise ValueError(str(data) + " is not in the tree.")
        if item == current.data:                # Base case 2: the data matches.
            return current
        if item < current.data:                 # Step to the left
            return _step(current.left,item)
        else:                                   # Step to the right
            return _step(current.right,item)

    # Start the recursion on the root of the tree.
    return _step(self.root, data)
\end{lstlisting}

\begin{figure}
% I want a picture of how descending through a tree partitions a number line.
% This will help when they get to K-D trees in the next lab.
\caption{Examples of finding in a BST, including the conceptual partitioning of the number line.}
\label{fig:BST.find}
\end{figure}

\subsection*{Inserting to a Binary Search Tree}

To insert new data into a binary search tree, a leaf node is added at the correct location.
First, we find the node that should be the parent of the new node.
We find the parent recursively, using a similar approach to the \li{find} method.
Once the correct parent is found, the new node is added as the left or right child of the parent.
See Figure \ref{fig:BST.insertion} for an example.

\begin{figure}
\caption{Examples of inserting to a BST.}
\label{fig:BST.insertion}
\end{figure}

% Some pseudocode to help with insertion, if deemed necessary.
\begin{comment}
\begin{lstlisting}
"""Pseudocode for finding the correct parent of a new node

find the parent of node, data
	if data < node.data:
		if node.left is not None
			find the parent of node.left, data
		else:
			return node
	if data > node.right:
		if data > node.data:
			if node.right is not None:
				find the parent of node.right, data
			else:
				return node
"""
\end{lstlisting}
\end{comment}

% Problem 2: BST.insert
\begin{problem}
Implement the \li{insert} method in the \li{BST} class.
To accomplish this, write a recursive \li{_find_parent} method within \li{insert}.
Find the correct parent, then determine whether the new node will be its left or right child.
Then double-link the parent and the new child.
Be sure to consider the special case of inserting to an empty tree.
To test your tree, use (but do not modify) the provided \li{BST.__str__} method.

Do not allow for duplicates in the tree: if the user executes \li{insert(x)} and there is already a node in the tree containing \li{x}, raise a ValueError.
\end{problem}

\subsection*{Removing from a Binary Search Tree}

Deleting nodes from a binary search tree is more difficult than searching and inserting.
Insertion always creates a new leaf node, but removal may delete any kind of node.
This leads to several different cases to consider.

\subsubsection*{Removing a leaf node}

In Python, an object is automatically deleted if there are no references to it.
Call the node to be removed the \emph{target node}, and suppose it has no children.
To remove the target, find the target's parent, then delete the parent's reference to the target.
Then there are no references to the target, so the target node is deleted.
Since the target is a leaf node, removing it does not affect the rest of the tree structure. See Figure \ref{fig:BST.remove_leaf}.

\begin{figure}
\caption{Examples of removing a leaf node.}
\label{fig:BST.remove_leaf}
\end{figure}

\begin{comment}
\begin{lstlisting}
"""Pseudocode for removing a leaf node
if the target node does not have children:
	if the target's data is smaller than its parent's data:
		set the parent's left child to None
	else:
		set the parent's right child to None
"""
\end{lstlisting}
\end{comment}

\subsubsection*{Removing a node with one child}

If the target node has one or more children, we must be careful not to delete the children when the target is removed.
Simply removing the target as if it were a leaf node would delete the entire subtree originating from the target.

To avoid deleting all of the target's descendents, we point the target's parent to an appropriate successor.
If the target has only one child, then that child is the successor.
Connect the target's parent to the successor, and double-link by setting the successor's parent to be the target node's parent.
Then, the target has no references pointing to it, so it is deleted.
The target's successor, however, is pointed to by the target's parent, and so it remains in the tree.
See Figure \ref{fig:BST.remove_oneChild}.

\begin{figure}
\caption{Examples of removing a leaf node with one child. Include a bad example (losing a subtree) and a good example (proper deletion).}
\label{fig:BST.remove_oneChild}
\end{figure}

\begin{comment}
\begin{lstlisting}
"""Pseudocode for removing a node with one child
if the target node has one child:
	if target.parent.right is the target:
		set parent's right child to the target's child.
	if target.parent.left is the target:
		set parent's left child to the target's child.
"""
\end{lstlisting}
\end{comment}

\subsubsection*{Removing a node with two children}

Removal is more complicated if the target node has two children.
To delete this kind of node, first we find it's immediate in-order predecessor.
The predecessor is the node with the largest data that is smaller than the target's data.
It may be found by moving to the left child of the target (so that it's value is less than the target's value), and then to the right for as long as possible (so that it has the largest such value).
Note that because of how the predecessor is chosen, any predecessor can only have at most one child.

Once the predecessor is found, the target and its predecessor must switch places in the graph, and then the target must be removed.
This can be done by simply switching the data values for the target and its predecessor.
Then the node with the target data has at most one child, and may be deleted accordingly.
If the predecessor was chosen appropriately, then the binary search tree structure and ordering will be maintained once the deletion is finished.

The easiest way to implement this is to use recursion.
First, because the predecessor has at most one child, we may recursivley remove the predecessor node by calling \li{remove} on the predecessor's data.
Then set the data stored in the target node as the predecessor's data.
See Figure \ref{fig:BST.remove_twoChild}.

\begin{figure}
\caption{Examples of removing a leaf node with two children.}
\label{fig:BST.remove_twoChild}
\end{figure}

\begin{comment}
\begin{lstlisting}
"""Pseudocode for removing a node with two children.

find the predecessor:
	set the current node to be target.left
	set current to be current.right until there is no right child
	return current.

Remove the node that has the predecessor's data (it will only have one child).
Change the target's data to the predecessor's data.
"""
\end{lstlisting}
\end{comment}

\subsubsection*{Removing the root node}

In each of the above cases, we must also consider the subcase where the target is the root node.
If the root has no children, resetting the root or calling the constructor will do.
If the root has one child, that child becomes the new root of the tree.
If the root has two children, the successor becomes the new root of the tree.

% Problem 3: BST.remove()
\begin{problem}
Implement the \li{remove} method in the \li{BST} class.
If the tree is empty, or if the target node is not in the tree, raise a \li{ValueError}.

Make sure to cover all possible cases:
\begin{enumerate}
\item The tree is empty (\li{ValueError})
\item The target is not in the tree (\li{ValueError})
\item The target is the root node:
	\begin{enumerate}
	\item{the root is a leaf node, hence the only node in the tree}
	\item{the root has one child}
	\item{the root has two children}
	\end{enumerate}
\item The target is in the tree but is not the root:
	\begin{enumerate}
	\item{the target is a leaf node}
	\item{the target has one child}
	\item{the target has two children}
	\end{enumerate}
\end{enumerate}
Test your solution thoroughly with each case.

(Hint: use \li{find} wherever appropriate.)
\end{problem}

\section*{AVL Trees}

Binary search trees are a good way of organizing data so that it is quickly accessible.
However, pathologies may arise when certain data sets are stored using a basic binary serch tree.
This is best demonstrated by inserting ordered data into a binary search tree.
Since the data is already ordered, each node will only have one child, and we essentially end up with a linked list.

\begin{lstlisting}
# Adding ordered integers sequentially destroys the efficiency of a BST.
>>> unbalanced_tree = BST()
>>> for i in xrange(10):
...     unbalanced_tree.insert(i)
...
# The tree is perfectly flat, so it loses its search efficiency.
>>> print(unbalanced_tree)
[0]
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
\end{lstlisting}

Problems also arise when one branch of the tree becomes much longer than the others, leading to longer search times.

An AVL tree is a tree that prevents any one branch from getting longer than the others.
It accomplishes this by periodically ``balancing'' the branches as nodes are added.
See Figure \ref{fig:avl_balance}
The AVL's balancing algorithm is beyond the scope of this text, but details and exercises on the algorithm can be found in Chapter 2 of the Volume II text.

\begin{lstlisting}
>>> balanced_tree = AVL()
>>> for i in xrange(10):
...     balanced_tree.insert(i)
...
# The AVL tree is balanced, so it retains (and optimizes) its search efficiency.
>>> print(balanced_tree)
[3]
[1, 7]
[0, 2, 5, 8]
[4, 6, 9]
\end{lstlisting}

\begin{figure}
\caption{Example of an AVL rebalancing.}
\label{fig:avl_balance}
\end{figure}

% Problem 4: compare build and search speeds
\begin{problem}
Compare the speed of building and searching the different data structures we have implemented so far.
Visualize the results by creating a plot with two subplots: one for build times, and one for search times.
Repeat the following for $n$ varying from 500 to 5000 at intervals of 500:

Use the \li{create_word_list} function in the provided \li{WordList} module to generate a list of $n$ randomized words from the file \li{English.txt}.
Time (separately) how long it takes to load a \li{LinkedList}, a \li{BST}, and an \li{AVL} with the data set.
Use \li{add} to load the \li{LinkedList} and \li{insert} to load the trees.
Then choose 5 random words from the data set, and time how long it takes to find each word in each object.
Use the \li{iterative_search} function from problem 1 to search the \li{LinkedList} and \li{find} to search the trees.
Calculate the average search time for each object.

In the first subplot, plot the number of words in each data set against the time it took to build each object.
In the second subplot, plot the number of words in each data set against the average time it took to search each object.
Your plot should look similar to Figure \ref{fig:times}.

\begin{comment}

% TODO: merge in this picture (as a pdf? png? jpeg?)
\begin{figure}[H]
\includegraphics[width=\textwidth]{times.pdf}
\caption{Note that the \li{BST} has the fastest build times, but the \li{AVL} has the fastest search times. How would the graph change if the data were sorted to begin with?}
\label{fig:times}
\end{figure}

\end{comment}

\end{problem}

\begin{comment}

% Nixed
\begin{problem}
The \li{lab5_data} module has a method called \li{load_data}.
This method returns two objects: a linked list and a binary search tree of a random permutation of the numbers from 1 to 100000.
Each object has a \li{find} method that returns the node that contains the data that the \li{find} method receives as an argument.
Write a method that generates 50 random numbers between 1 and 100000, finds them in the given data structures, and returns the average search time for each.
\end{problem}

\begin{problem}
The \li{Lab5_data} module has a method called \li{load_words} that returns a list of several thousand english words.
The \li{Lab5_spec} module has an implementation of an AVL tree that inherits from a working \li{BST} class.
Insert each word into an instances of the \li{BST} and \li{AVL} classes.
Write a method called \li{test_words} that searches for 50 random words in each object and returns their average search times.
\end{problem}

\end{comment}


\subsubsection{Power methods}

While the $QR$ algorithm can be used to compute eigenvalues, there is a useful and fairly elementary technique for
finding the eigenvector and associated eigenvalue nearest to a given complex number which
is called the {\em shifted inverse power method}. It tends to work extremely well provided you
start with something which is fairly close to an eigenvalue.

Power methods are based the consideration of powers of a given matrix. Let $
\set{
\vect{x}_{1},\ldots,\vect{x}_{n}} $ be a basis of eigenvectors for $
\C^{n}$ such that $A\vect{x}_{n}=\lambda _{n}\vect{x}_{n}$. Now let $
\vect{u}_{1}$ be some non-zero vector. Since $\set{\vect{x}_{1},\ldots,\vect{x}_{n}} $ is a basis, there exists unique scalars, $c_{i}$ such that
\begin{equation*}
\vect{u}_{1}=\sum_{k=1}^{n}c_{k}\vect{x}_{k}
\end{equation*}
Assume you have not been so unlucky as to pick $\vect{u}_{1}$ in such a way
that $c_{n}=0$. Then let $A\vect{u}_{k}=\vect{u}_{k+1}$ so that
\begin{equation}
\vect{u}_{m}=A^{m}\vect{u}_{1}=\sum_{k=1}^{n-1}c_{k}\lambda _{k}^{m}\vect{x}
_{k}+\lambda _{n}^{m}c_{n}\vect{x}_{n}.  \label{20-may-e1}
\end{equation}
For large $m$ the last term, $\lambda _{n}^{m}c_{n}\vect{x}_{n}$, determines
quite well the direction of the vector on the right. This is because $
\abs{\lambda _{n}}$ is larger than $\abs{\lambda
_{k}}$ for $k<n$ and so for a large $m$, the sum, $\sum_{k=1}^{n-1}c_{k}\lambda _{k}^{m}\vect{x}_{k}$, on the right is fairly
insignificant. Therefore, for large $m$, $\vect{u}_{m}$ is essentially a
multiple of the eigenvector $\vect{x}_{n}$, the one which goes with $\lambda
_{n}$. The only problem is that there is no control of the size of the
vectors $\vect{u}_{m}$. You can fix this by scaling. Let $S_{2}$ denote the
entry of $A\vect{u}_{1}$ which is largest in absolute value. We call this a
\index{scaling factor} \textbf{scaling factor}. Then $\vect{u}_{2}$ will not be just $A\vect{u}_{1}$ but $A\vect{u}_{1}/S_{2}$. Next
let $S_{3}$ denote the entry of $A\vect{u}_{2}$ which has largest absolute
value and define $\vect{u}_{3}\equiv A\vect{u}_{2}/S_{3}$. Continue this way.
The scaling just described does not destroy the relative insignificance of
the term involving a sum in {\eqref{20-may-e1}}. Indeed it amounts to nothing more
than changing the units of length. Also note that from this scaling
procedure, the absolute value of the largest element of $\vect{u}_{k}$ is
always equal to 1. Therefore, for large $m$,
\begin{equation*}
\vect{u}_{m}=\vspace{0.05in}\frac{\lambda _{n}^{m}c_{n}\vect{x}_{n}}{S_{2}S_{3}\cdots S_{m}}+(\text{relatively insignificant term}) .
\end{equation*}
Therefore, the entry of $A\vect{u}_{m}$ which has the largest absolute value
is essentially equal to the entry having largest absolute value of
\begin{equation*}
A\paren{\vspace{0.05in}\frac{\lambda _{n}^{m}c_{n}\vect{x}_{n}}{S_{2}S_{3}\cdots S_{m}}} =\vspace{0.05in}\frac{\lambda _{n}^{m+1}c_{n}
\vect{x}_{n}}{S_{2}S_{3}\cdots S_{m}}\approx \lambda _{n}\vect{u}_{m}
\end{equation*}%
and so for large $m$, it must be the case that $\lambda _{n}\approx S_{m+1}$.
This suggests the following procedure.

\begin{procedure}{Finding the largest eigenvalue with its eigenvector}{}
\begin{enumerate}
\item Start with a vector $\vect{u}_{1}$ which you hope has a component in
the direction of $\vect{x}_{n}$. The vector $(1,\ldots,1) ^{T}$
is usually a pretty good choice.

\item If $\vect{u}_{k}$ is known,
\begin{equation*}
\vect{u}_{k+1}=\frac{A\vect{u}_{k}}{S_{k+1}}
\end{equation*}
where $S_{k+1}$ is the entry of $A\vect{u}_{k}$ which has largest absolute
value.

\item When the scaling factors, $S_{k}$ are not changing much, $S_{k+1}$
will be close to the eigenvalue and $\vect{u}_{k+1}$ will be close to an
eigenvector.

\item Check your answer to see if it worked well.
\end{enumerate}
\end{procedure}

The shifted inverse power method involves finding the eigenvalue closest to
a given complex number along with the associated eigenvalue. If $\mu $ is a
complex number and you want to find $\lambda $ which is closest to $\mu$,
you could consider the eigenvalues and eigenvectors of $(A-\mu
I) ^{-1}$. Then $A\vect{x}=\lambda \vect{x}$ if and only if
\begin{equation*}
(A-\mu I) \vect{x}=(\lambda -\mu) \vect{x}
\end{equation*}
If and only if
\begin{equation*}
\frac{1}{\lambda -\mu }\vect{x}=(A-\mu I) ^{-1}\vect{x}
\end{equation*}
Thus, if $\lambda $ is the closest eigenvalue of $A$ to $\mu $ then out of
all eigenvalues of $(A-\mu I) ^{-1}$, you would have $\frac{1}{
\lambda -\mu }$ would be the largest. Thus all you have to do is apply the
power method to $(A-\mu I) ^{-1}$ and the eigenvector you get
will be the eigenvector which corresponds to $\lambda $ where $\lambda $ is
the closest to $\mu $ of all eigenvalues of $A$. You could use the
eigenvector to determine this directly.

\begin{example}{Finding eigenvalue and eigenvector}{}
Find the eigenvalue and eigenvector for
\begin{equation*}
\begin{mymatrix}{rrr}
3 & 2 & 1 \\
-2 & 0 & -1 \\
-2 & -2 & 0
\end{mymatrix}
\end{equation*}
which is closest to $0.9+0.9i$.
\end{example}

\begin{solution}
Form
\begin{eqnarray*}
&&(\begin{mymatrix}{rrr}
3 & 2 & 1 \\
-2 & 0 & -1 \\
-2 & -2 & 0
\end{mymatrix} -(0.9+0.9i) \begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix}) ^{-1} \\
&=&\begin{mymatrix}{ccc}
-0.619\,19-10.\,\allowbreak 545i & -5.\,\allowbreak 524\,9-4.\,\allowbreak
972\,4i & -0.370\,57-5.\,\allowbreak 821\,3i \\
5.\,\allowbreak 524\,9+4.\,\allowbreak 972\,4i & 5.\,\allowbreak
276\,2+0.248\,62i & 2.\,\allowbreak 762\,4+2.\,\allowbreak 486\,2i \\
0.741\,14+11.\,\allowbreak 643i & 5.\,\allowbreak 524\,9+4.\,\allowbreak
972\,4i & 0.492\,52+6.\,\allowbreak 918\,9i%
\end{array}%
}
\end{eqnarray*}
Then pick an initial guess an multiply by this matrix raised to a large
power.
\begin{equation*}
\begin{mymatrix}{ccc}
-0.619\,19-10.\,\allowbreak 545i & -5.\,\allowbreak 524\,9-4.\,\allowbreak
972\,4i & -0.370\,57-5.\,\allowbreak 821\,3i \\
5.\,\allowbreak 524\,9+4.\,\allowbreak 972\,4i & 5.\,\allowbreak
276\,2+0.248\,62i & 2.\,\allowbreak 762\,4+2.\,\allowbreak 486\,2i \\
0.741\,14+11.\,\allowbreak 643i & 5.\,\allowbreak 524\,9+4.\,\allowbreak
972\,4i & 0.492\,52+6.\,\allowbreak 918\,9i%
\end{mymatrix} ^{15}\begin{mymatrix}{c}
1 \\
1 \\
1
\end{mymatrix}
\end{equation*}
This equals
\begin{equation*}
\begin{mymatrix}{c}
1.\,\allowbreak 562\,9\times 10^{13}-3.\,\allowbreak 899\,3\times 10^{12}i
\\
-5.\,\allowbreak 864\,5\times 10^{12}+9.\,\allowbreak 764\,2\times 10^{12}i
\\
-1.\,\allowbreak 562\,9\times 10^{13}+3.\,\allowbreak 899\,9\times 10^{12}i
\end{mymatrix}
\end{equation*}
Now divide by an entry to make the vector have reasonable size. This yields
\begin{equation*}
\begin{mymatrix}{c}
-0.999\,99-3.\,\allowbreak 614\,0\times 10^{-5}i \\
0.499\,99-0.499\,99i \\
1.0
\end{mymatrix}
\end{equation*}
which is close to
\begin{equation*}
\begin{mymatrix}{c}
-1 \\
0.5-0.5i \\
1.0
\end{mymatrix}
\end{equation*}
Then
\begin{equation*}
\begin{mymatrix}{rrr}
3 & 2 & 1 \\
-2 & 0 & -1 \\
-2 & -2 & 0
\end{mymatrix} \begin{mymatrix}{c}
-1 \\
0.5-0.5i \\
1.0
\end{mymatrix} =\begin{mymatrix}{c}
-1.0-1.0i \\
1.0 \\
1.0+1.0i
\end{mymatrix}
\end{equation*}
Now to determine the eigenvalue, you could just take the ratio of
corresponding entries. Pick the two corresponding entries which have the
largest absolute values. In this case, you would get the eigenvalue is $1+i$
which happens to be the exact eigenvalue. Thus an eigenvector and eigenvalue
are
\begin{equation*}
\begin{mymatrix}{c}
-1 \\
0.5-0.5i \\
1.0
\end{mymatrix}, 1+i
\end{equation*}
\end{solution}

Usually it won't work out so well but you can still find what is desired.
Thus, once you have obtained approximate eigenvalues using the $QR$
algorithm, you can find the eigenvalue more exactly along with an
eigenvector associated with it by using the shifted inverse power method.


\section{Finite State Automata}

\subsection{Alphabets \& Strings}

\begin{itemize}

	\item Let $ A $ be a set; then $ A^n $ is the set of all finite sequences $ a_1 \dots a_n $ with $ a_i \in A $, $ 1 \le i \le m $

	\begin{itemize}
		\item Elements of $ A $ are \textit{letters} or \textit{symbols}

		\item Elements of $ A^n $ are \textit{words} or \textit{strings} over $ A $ of length $ m $
	\end{itemize}

	\item $ \varepsilon $ is the special \textit{empty string}, the only string of length $ 0 $

	\item $ A^+ = \bigcup_{m \ge 1} A^m $ -- the set of non-empty strings over $ A $ of any length

	\item $ A^* = A^+ \cup \varepsilon = \bigcup_{m \ge 0} A^m$ -- the set of (possibly empty) strings over $ A $ of any length

	\item If $ \alpha = a_1 \dots a_m $, $ \beta = b_1 \dots b_m \in A^* $, then define $ \alpha \beta $ to be $ a_1 \dots a_m b_1 \dots b_m \in A^{m + n} $. This gives binary `product' or \textit{concatenation} on $ A^* $

	\item For $ \alpha \in A^+ $, define $ \alpha^n, n \in \Nat $ by $ \alpha^0 = \varepsilon $, and $ \alpha^{n+1} = \alpha^n \alpha $

	\item A \textit{language} with alphabet $ A $ is a subset of $ A^* $

\end{itemize}

\clearpage

\subsection{Definition of an FSA}

\begin{itemize}

	\item A Finite State Automaton (FSA) is a tuple $ M = (Q, F, A, \tau, q_0) $

	\begin{itemize}

		\item $ Q $ is a finite set of states

		\item $ F \subseteq Q $ is the set of final states

		\item $ A $ is the alphabet

		\item $ \tau \subseteq Q \times A \times Q $ is the set of transitions

		\item $ q_0 \in Q $ is the initial state

	\end{itemize}

	\item The transition diagram of an FSA is a directed graph with:

	\begin{itemize}

		\item Vertex set $ Q $

		\item An edge for each transition; $ (q, a, q') \in \tau $ corresponds to an edge from $ q $ to $ q' $ with label $ a $

		\item Initial state $ q_0 $ labelled with $ - $

		\item Final states labelled with $ + $

		\item Example: a \textit{non-deterministic} `haha machine', with $ A = \set{h, a} $

		\begin{tikzpicture}
		\node[circle,thick,draw] (q0) at (0, 0) {$ - $};
		\node[circle,thick,draw] (q1) at (2, 0) {$ \; $};
		\node[circle,thick,draw] (q2) at (4, 0) {$ + $};

		\draw[edge] (q0) to[bend left] node[above] {h} (q1);
		\draw[edge] (q1) to[bend left] node[above] {a} (q2);
		\draw[edge] (q1) to[bend left] node[below] {a} (q0);
		\end{tikzpicture}

	\end{itemize}

	\item A \textit{computation} of $ M $ is a sequence $ q_0, a_1, q_1, a_2, \dots, a_n, q_n $ with $ n \ge 0 $ where $ (q_i, a_{i+1}, q_{i+1}) \in \tau $ for $ 0 \le i \le n - 1 $

	\begin{itemize}

		\item The \textit{label} on the computation is $ a_1 \dots a_m $

		\item The computation is \textit{successful} if $ q_n \in F $

		\item A string $ a_1 \dots a_n $ is \textit{accepted} by $ M $ if there is a successful computation with label $ a_1 \dots a_n $, and it is \textit{rejected} otherwise

	\end{itemize}

	\item The language recognised by $ M $ is $ \Lang(M) = \setcomp{w \in A^*}{w \text{ is accepted by } M} $

	\item There is a one-to-one correspondence between computations of $ M $ and paths in the graph from $ q_0 $

	\item Example: $ A = \set{a, b} $ of an FSA accepting only words with an odd number of 'a's

	\begin{tikzpicture}
	\node[circle,thick,draw] (q0) at (0, 0) {$ - $};
	\node[circle,thick,draw] (q1) at (4, 0) {$ + $};

	\draw[edge] (q0) to[bend left] node[above] {a} (q1);
	\draw[edge] (q0) to[loop left] node[left] {b} (q0);

	\draw[edge] (q1) to[bend left] node[above] {a} (q0);
	\draw[edge] (q1) to[loop right] node[right] {b} (q1);
	\end{tikzpicture}

	\item An FSA is deterministic (a DFA) if for all $ q \in Q, a \in A $ there is exactly one $ q' \in Q $ such that $ (q, a, q') \in \tau $

	\item Example: DFA for the `haha machine'

	\begin{tikzpicture}
	\node[circle,thick,draw] (q0) at (0, 0) {$ - $};
	\node[circle,thick,draw] (q1) at (2, 0) {$ \; $};
	\node[circle,thick,draw] (q2) at (4, 0) {$ + $};

	\draw[edge] (q0) to[bend left] node[above] {h} (q1);
	\draw[edge] (q1) to[bend left] node[above] {a} (q2);
	\draw[edge] (q2) to[bend left] node[below] {h} (q1);
	\end{tikzpicture}

	\item Note this machine lacks a transition for $ a $ when in the initial state -- though technically required for a DFA, it is easily fixed by adding an `error state' to catch what would otherwise be missing transitions

\end{itemize}

\clearpage

\subsection{Deterministic FSAs}

\begin{itemize}

	\item For a DFA $ M $, define the transition function $ \delta: Q \times A \to Q $ by $ q' = \delta(q, a) $, where $ q' $ is the unique element such that $ (q, a, q') \in \tau $

	\item If $ \Lang $ is a language with alphabet $ A $, then the following are equivalent:

	\begin{enumerate}
		\item $ \Lang $ is recognised by an FSA

		\item $ \Lang $ is recognised by a DFA
	\end{enumerate}

	\item Given a non-deterministic FSA $ M = (Q, F, A, \tau, q_0) $, an equivalent DFA $ M' = (Q', F', A, \tau', q_0') $ may be generated by the \textit{powerset method}:

	\begin{itemize}

		\item $ Q' = \powerset{Q} \setminus \emptyset $ (i.e. the set of all subsets of $ Q $ that aren't empty)

		\item $ F' = \setcomp{X \in Q'}{q \in X \text{ for some } q \in F} $

		\item For $ X \in Q', a \in A $, define $ \delta(X, a) := \setcomp{q \in Q}{(x, a, q) \in \tau \text{ for some } x \in X} $

		\item $ \tau' = \setcomp{(X, a, \delta(X, a))}{X \in Q', a \in A} $

		\item $ q_0' = \set{q_0} $

	\end{itemize}

	\item Proof: show that $ \Lang(M) = \Lang(M') $

	\begin{itemize}
		\item $ \Lang(M) \subseteq Lang(M') $:

		\begin{itemize}
			\item Given $ w \in \Lang(M) $, $  q_0 a_1 \dots a_n q_n $ is a successful computation of $ M $

			\item Then define $ q_i' = \delta(q_{i-1}', a_i) $ for $ 1 \le i \le n $

			\item $ q_0', a_1, q_1' \dots a_n, q_n' $ will be a successful computation of $ M' $

			\item Therefore $ w \in \Lang(M') $
		\end{itemize}

	\item $ \Lang(M') \subseteq Lang(M) $:

	\begin{itemize}
		\item Let $ w = a_1 \dots a_n \in L(M') $, and $  q_0', a_1, q_1' \dots a_n, q_n' $ be a successful computation of $ M $

		\item Each $ q_i' $ cannot be the empty set

		\item By definition of $ \tau' $, $ \exists q_1 \in q_1' $ s.t. $ (q_0, a_1, q_1) \in \tau $

		\item Then we can find $ q_i \in q_i' $ s.t. $ (q_{i-1}, a_i, q_i) \in \tau $ for $ 1 \le i \le n $

		\item For $ q_n $ we further require $ q_n \in F $

		\item Therefore, $ q_0, a_1, q_1, a_2, \dots a_n, q_n $ is a successful computation

		\item Therefore $ w \in \Lang(M) $
	\end{itemize}

	\end{itemize}

\end{itemize}

\clearpage

\subsection{The Pumping Lemma}

\begin{itemize}

	\item The Pumping Lemma says that for any $ \Lang $ recognised by an FSA $ M $, there is a certain word length beyond which all words can be split into sections as $ xyz$, where $ x y^n z $ is also in the language

	\item Formally there is an integer $ p > 0 $ s.t. any word $ w \in L $ with $ \abs{w} \ge p $ is of the form $ w = xyz $, where $ \abs{y} > 0 $, $ \abs{xy} \le p $ and $ x y^i z \in \Lang $ for $ i \ge 0 $

	\item Proof:

	\begin{itemize}

		\item Let $ p $ be the number of states in $ M $, and suppose $ w = a_1 \dots a_n \in \Lang $, where $ n \ge p $

		\item A successful computation $ q_0, a_1, \dots, q_n $ has to pass through a certain state at least twice (by the pigeonhole principle)

		\item Therefore, $ \exists r < s $ s.t. $ q_r = q_s $; choose minimal such $ s $

		\item Now put $ x = a_1 \dots a_r $, $ y = a_{r+1} \dots a_s $ (note $ \abs{y} > 0$), and $ z = a_{s+1} \dots a_n $

		\item By minimality of $ s $, $ q_0, \dots q_{s-1} $ are distinct, and $ \abs{xy} = s \le p $

		\item Then, note that $ q_r, a_{r+1}, \dots, q_{s} $ is a loop, which may be validly repeated $ i \ge 0 $ times

		\item Therefore, $ x y^i z \in \Lang $

	\end{itemize}

	\item Corollary: there exist languages which are not computable by an FSA

	\item Example: there is no FSA which can recognise $ \Lang = \setcomp{a^n b^n}{n \in \Nat} $

	\item Proof:

	\begin{itemize}

		\item Assume for a contradiction there exists an FSA $ M $ which can recognise $ \Lang $

		\item Let $ p $ be the number from the pumping lemma, and choose $ n \ge p $ and consider $ w = a^n b^n $

		\item By the pumping lemma, $ \exists x, y, z $ s.t. $ a^n b^n = xyz $, with $ \abs{y} \ge 1 $ and $ \abs{xy} \le p \le n $

		\item Then $ y $ is written entirely in terms of the letter a, and $ \abs{y} \ge 1 $

		\item By the pumping lemma, $ x y^i z \in \Lang $ for all $ i $

		\item So choose $ i = 0 $, then some $ w = a^k b^n \in \Lang $ s.t. $ k < n $, which is a contradiction

	\end{itemize}

\end{itemize}


\section{Linear equations}
\subsection{Linear and affine functions}
Superposition condition:$f(\alpha x + \beta y) = \alpha f(x) + \beta f(y)$\\
Such an f is called Linear\\
Matrix vector product function: A is mXn matrix such that $f(x)=Ax$ and f is linear $f(\alpha x + \beta y)=A(\alpha x + \beta y)=\alpha f(x) + \beta f(y)$\\
Converse is true: If $f:R^{n} \mapsto R^{m}$ is linear, then $f(x)=f(x_{1}e_{1}+x_{2}e_{2}+...x_{n}e_{n})$
\newline $=x_{1}f(e_{1})+x_{2}f(e_{2})+...x_{n}f(e_{n})$
\newline$=Ax$ with $A = [f(e_{1})+f(e_{2})+...f(e_{n})]$
\newline Affine Functions:$f:R^{n} \mapsto R^{m}$ is affine if it is a linear function plus a constant i.e $f(x)=Ax+b$ same as $f(\alpha x + \beta y)=\alpha f(x)+\beta f(y)$ holds for all x, y and $\alpha,\beta$ such that $\alpha+\beta=1$
\newline A and B can be calculated as \\
$A=[f(e_{1})-f(0)\thickspace f(e_{2})-f(0) ... f(e_{n})-f(0)]] ; b= f(0)$\\
Affine functions sometimes incorrectly called linear functions
\subsection{Linear function models}
Price elasticity of demand
$\delta_i^{price}=(p_i^{new}-p_{i})/p_{i}$: fractional changes in prices
$\delta_i^{dem}=(d_i^{new}-d_{i})/d_{i}$: fractional change in demand
Price demand elasticity model: $\delta^{dem}=E\delta^{price}$\\
Taylor series approximation
Regression model
\subsection{Systems of linear equations}


%!TEX root = ../TTK18-Summary.tex
\section{Linear matrix inequalities}
Many problems in systems and control can be reduced to optimization problems involving LMIs, for which there exist efficient numerical solvers.

An LMI has the form
%
\begin{equation}
  F(p) = F_0 + \sum_{i=1}^m p_i F_i > 0
\end{equation}
%
where $p$ is the variable vector, and $F_i = F_i^T$. Multiple LMIs can be rewritten as a single LMI by stacking each matrix on the diagonal:
%
\begin{equation}
  \begin{bmatrix}
    F^{(1)}(p) &&\\
    & \ddots &\\
    && F^{(n)}(p)
  \end{bmatrix} > 0
\end{equation}

\subsection{Some LMI problems}
\paragraph{Feasibility problem} Is $\dot{x} = Ax$ stable? Yes, given
%
\begin{equation}\label{eq:linear-auto-stability}
  A\tp P + PA < 0,\quad P > 0
\end{equation}
%
for the Lyapunov function $V(x) = x\tp P x$. (This is equivalent to the requirement that all real parts of the eigenvalues of $A$ are negative.) Equation \eqref{eq:linear-auto-stability} can be written as a single LMI by stacking the two inequalities in one matrix:
%
\begin{equation}
  \begin{bmatrix}
    A\tp P + PA & 0 \\
    0 & -P
  \end{bmatrix}
  < 0
\end{equation}

\paragraph{Convex function minimization} For the system
%
\begin{equation}
  \begin{split}
    \dot{x} &= Ax + Bw \\
    z &= Cx + Dw
  \end{split}
\end{equation}
%
the $H_\infty$-norm of the transfer function from $w$ to $z$ can be found by
%
\begin{gather}
% \begin{equation}
% \begin{split}
  \min \gamma \\
  \mbox{s.t. }
  \begin{bmatrix}
    A\tp P + PA & PB & C\tp \\
    B\tp P & -\gamma I & D\tp \\
    C & D & -\gamma I
  \end{bmatrix}
   < 0
% \end{split}
% \end{equation}
\end{gather}

\subsection{LMI tricks}
\subsubsection{Preliminaries}
If $W$ is full rank, then pre-multiplication with $W\tp$ and post-multiplication with $W$ does not change sign-definiteness:
%
\begin{equation}\label{eq:pre-post-multiply}
  Q > 0 \Leftrightarrow W\tp Q W > 0 \quad (\mbox{for full rank } W)
\end{equation}

\subsubsection{Change of variables}\label{sssec:change-of-variables}
Sometimes changing the variables can linearize the problem.
\paragraph{Example} State feedback: Find $u = Kx$ to stabilize the system. Must find $P > 0$ and $K$ such that
%
\begin{equation}
  (A + BK)\tp P + P(A + BK) < 0.
\end{equation}
%
If this is fullfilled, we have shown Lyapunov stability, but this is not linear in $P$ and $K$. Like in \eqref{eq:pre-post-multiply}, we can pre- and postmultiply by $Q = P^{-1}$ to get
%
\begin{equation}
  QA\tp + AQ + QK\tp B\tp + BKQ < 0
\end{equation}
%
and define $L = KQ$ to get (remember $Q = P^{-1}$ so $Q = Q\tp$)
%
\begin{equation}
  QA\tp + AQ + L\tp B\tp + BL < 0.
\end{equation}
%
This is an LMI in $Q > 0$ and $L$!

\subsubsection{Congruence transform}
With e.g.
%
\begin{equation}
  Q =
  \begin{bmatrix}
    A\tp P + PA & PBK + C\tp V \\
    * & -2V
  \end{bmatrix}
  < 0
\end{equation}
%
in variables $P>0$, $V>0$, and $K$. Choosing the full-rank $W$
%
\begin{equation}
  W =
  \begin{bmatrix}
    P^{-1} & 0 \\
    0 & V^{-1}
  \end{bmatrix}
\end{equation}
%
gives
%
\begin{equation}
  W\tp Q W =
  \begin{bmatrix}
    XA\tp + AX & BL + XC\tp \\
    * & -2U
  \end{bmatrix}
\end{equation}
%
which is an LMI in $X = P^{-1}$, $U = V^{-1}$, $L = KV^{-1}$.

\subsubsection{Schur complement}
These are equivalent:
%
\begin{equation}
  \begin{bmatrix}
    \Phi_{11} & \Phi_{12} \\
    \Phi_{12}\tp & \Phi_{22}
  \end{bmatrix}
  < 0
  \quad \Leftrightarrow \quad
  \begin{cases}
    \Phi_{22} < 0 \\
    \Phi_{11} - \Phi_{12} \Phi_{22}^{-1} \Phi_{12}\tp < 0
  \end{cases}
\end{equation}

\paragraph{Example} Given $Q \geq 0$, $R > 0$, find $P > 0$ such that the Ricatti inequality
%
\begin{equation}
  A\tp P + PA + PBR^{-1}B\tp P + Q < 0.
\end{equation}
%
We can rearrange to reveal a similarity
%
\begin{equation}
  \underbrace{A\tp P + PA + Q}_{\Phi_{11}}
  -
  \underbrace{PB}_{\Phi_{12}}
  \underbrace{(-R^{-1})}_{\Phi_{22}^{-1}}
  \underbrace{B\tp P}_{\Phi_{12}\tp}
  < 0.
\end{equation}
%
From the Schur complement and that $-R < 0$, this is equivalent to
%
\begin{equation}
  \begin{bmatrix}
    A\tp P + PA + Q & PB \\
    * & -R
  \end{bmatrix}
  < 0.
\end{equation}

\subsubsection{S-procedure}
Used when we need a criterion fulfilled locally, such as $F_0(x) \leq 0$ only when $F_i(x) > 0$, that is, the S-procedure gives a criterion for when one inequality is implied by another.

We want $F_0(x) \leq 0$ when $F_i(x) > 0$. This is true given
%
\begin{equation}
  F\sub{aug}(x) = F_0(x) + \sum_{i=1}^q \tau_i F_i(x) \leq 0, \quad \tau_i \geq 0
\end{equation}

\paragraph{Example} Find $P>0$ such that
%
\begin{equation}
  \bmat{x \\ z}\tp
  \underbrace{\bmat{A\tp P & PB \\ * & 0}}_{F_0}
  \bmat{x \\ z}
  < 0
\end{equation}
%
when
%
\begin{equation}
  z\tp z \leq x\tp C\tp C x
  \quad \Leftrightarrow \quad
  \bmat{x \\ z}\tp
  \underbrace{\bmat{C\tp C & 0 \\ * & -I}}_{F_1}
  \bmat{x \\ z}
  \geq 0.
\end{equation}

This can be transformed into an LMI by the S-procedure:
%
\begin{equation}
  \bmat{x \\ z}\tp
  \underbrace
    {
      \begin{bmatrix}
        A\tp P + PA + \tau C\tp C & PB \\
        * & -\tau I
      \end{bmatrix}
    }_{F\sub{aug}}
  \bmat{x \\ z}
  < 0.
\end{equation}
%
This is an LMI in $P>0$ and $\tau \geq 0$.

\subsubsection{Projection lemma}
Sometimes we get equations like
%
\begin{equation}\label{eq:finsler-projection-formulation}
  \Psi(X) + G(X)\Lambda\tp H(X) + H(X)\Lambda\tp G\tp(X) < 0
\end{equation}
%
in matrix variables $X$ and $\Lambda$. $\Psi$, $G$, and $H$ are affine functions of $X$. Equation \eqref{eq:finsler-projection-formulation} is equivalent to
%
\begin{equation}
  \begin{split}
    W_{G(X)}\tp \Psi(X) W_{G(X)} &< 0 \\
    W_{H(X)}\tp \Psi(X) W_{H(X)} &< 0
  \end{split}
\end{equation}
%
where $W_{G(X)}$ and $W_{H(X)}$ are orthogonal complements to $G$ and $H$:
\begin{gather}
  W_{G(X)} G(X) = 0,\quad W_{H(X)} H(X) = 0 \\
  \rank W_{G(X)} + \rank G = n,\quad \rank W_{H(X)} + \rank H = n.
\end{gather}
%
This is useful as it may linearize the formulation, and it may reduce the number of variables in the computation.

\paragraph{Example} State feedback:
%
\begin{equation}
  (A + BK)\tp P + P(A + BK) < 0,\quad P > 0
\end{equation}
%
can be rewritten by a change of variables (Section \ref{sssec:change-of-variables}) as
%
\begin{equation}
  QA\tp + AQ + L\tp B\tp + BL < 0,\quad Q > 0.
\end{equation}
%
Using the projection lemma this can be written without $L$:
%
\begin{equation}
  \begin{rcases}
    W_B\tp(AQ+QA\tp)W_B < 0 \\
    W_I\tp(AQ+QA\tp)W_I < 0
  \end{rcases} Q > 0
\end{equation}
%
In this case the second inequality is unnecessary, because $I$ is full rank. First inequality can be interpreted as ``the system is stable if it is stable in the null space of the inputs.''

\subsubsection{Finsler's lemma}
Finsler's lemma is another way of writing \eqref{eq:finsler-projection-formulation} more compactly:
%
\begin{equation}
  \begin{split}
    \Psi(X) - \sigma G(X) G(X)\tp &< 0 \\
    \Psi(X) - \sigma H(X) H(X)\tp &< 0
  \end{split}
\end{equation}
%
for some real $\sigma$.

\paragraph{Example} State feedback:
%
\begin{equation}
  (A + BK)\tp P + P(A + BK) < 0,\quad P > 0
\end{equation}
%
can be rewritten
%
\begin{equation}
  QA\tp + AQ + L\tp B\tp + BL < 0,\quad Q > 0.
\end{equation}
%
Using Finsler's lemma this can be written without $L$
%
\begin{equation}
  \begin{rcases}
    &AQ + QA\tp - \sigma BB\tp < 0 \\
    &AQ + QA\tp - \sigma I < 0
  \end{rcases} Q > 0.
\end{equation}
%
As long as $Q$ and $\sigma$ fulfilling the first inequality are found, it is always possible to find $\sigma$ to fulfill the second.

\subsection{Applications}

\subsubsection{Discrete-time state feedback stabilization}
Discrete systems use Lyapunov differences. For the LF
%
\begin{equation}
  V(x_k) = x_k\tp P x_k,\quad P>0
\end{equation}
%
the system is stable if
%
\begin{equation}
  V(x_k) - V(x_{k+1}) = x_k\tp P x_k - x_{k+1}\tp P x_{k+1} > 0,
\end{equation}
%
which is equivalent to
%
\begin{equation}
  P - (A + BK)\tp P (A+BK) > 0.
\end{equation}
%
The Schur complement is
%
\begin{equation}
  \begin{bmatrix}
    P & (A + BK)\tp P \\
    P(A + BK) & P
  \end{bmatrix}
  > 0,
\end{equation}
%
and congruence transform with $W = \bmat{P\inv & 0 \\ 0 & P\inv}$ gives
%
\begin{equation}
  \begin{bmatrix}
    P\inv & P\inv (A + BK)\tp \\
    (A + BK) P\inv & P\inv
  \end{bmatrix}
  > 0.
\end{equation}
%
A change of variables $Q = P\inv$, $L = KP\inv$ gives
%
\begin{equation}
  \begin{bmatrix}
    Q & QA\tp + L\tp B\tp \\
    AQ + BL & Q
  \end{bmatrix}
  > 0
\end{equation}
%
which is an LMI in $Q > 0$ and $L$.

\subsubsection{Robust feedback stabilization}
Plant not exactly known, but known to be in a set
%
\begin{equation}
  \bmat{A \\ B} \in \co \bmat{A_1 \dots A_q \\ B_1 \dots B_q}
\end{equation}
%
where $\co$ denotes convex combination. Equivalent:
%
\begin{equation}
    A = \sum_{i=1}^q a_i A_i,\quad B = \sum_{i=1}^q a_i B_i \\
\end{equation}
%
with
%
\begin{equation}
  \sum_{i=1}^q a_i = 1, a_i \geq 0.
\end{equation}
%
A controller that solves the LMIs for all $q$ extreme plants with the same $Q$ and $L$ is stable for all $A$ and $B$.

\subsubsection{Parameter dependent LFs}
With a model
%
\begin{equation}
  A(\delta) = A_0 + \sum_{i=1}^q \delta_i A_i
\end{equation}
%
with bounds and rate limits
%
\begin{equation}
  \delta_i \in [\underline\delta_i, \overline\delta_i], \quad \dot\delta_i \in [\underline\lambda_i, \overline\lambda_i ]
\end{equation}
%
$\Delta$ is the set of all $\delta$ that are within the bounds, and $\Lambda$ is the set of all $\dot\delta$ within the rate limits. $\Delta_0$ and $\Lambda_0$ are their respective vertex sets.

Some LFs have the form
%
\begin{equation}
  V(\delta,x) = x\tp P(\delta)x,\quad P(\delta) = P_0 + \sum_{i=1}^q \delta_i P_i.
\end{equation}
%
which gives
%
\begin{equation}
  \od{P(\delta)}{t} = \dot\delta_1 P_1 + \dots + \dot\delta_q P_q = P(\dot\delta) - P_0
\end{equation}
%
leading to
%
\begin{equation}
  \dot{V}(\delta,x) = x\tp \left(A\tp(\delta) P(\delta) + P(\delta) A(\delta) + P(\dot{\delta}) - P_0 \right)x.
\end{equation}
%
Which is stable if there exists matrices $P_0 \dots P_q$ such that
%
\begin{equation}
  \begin{split}
    A\tp(\delta)P(\delta) + P(\delta)A(\delta) + P(\dot\delta) &< P_0, \quad \forall \delta \in \Delta_0, \forall \dot\delta \in \Lambda_0 \\
    P(\delta) &> I, \quad \forall \delta \in \Delta_0 \\
    A_i\tp + P_iA_i &\leq 0,\quad i = 1,\dots,q.
  \end{split}
\end{equation}
%
This means that we only have to find valid LFs for the ``extreme points'', i.e. when $\delta$ or $\dot\delta$ is at saturation.

TTK18 Optimaliseringsbasert reguleringsdesign og analyse/tex/sec-lmis.tex


\subsection{Trigonometric polynomials}\label{subsec:trigonometric_polynomials}

\begin{definition}\label{def:trigonometric_polynomial}
  We define the \term{trigonometric polynomials} over \( \BbbC \) as the \hyperref[def:laurent_polynomial/polynomial]{Laurent polynomials} \( \BbbC[e^{iz}] \). A trigonometric polynomial \( p \in \BbbC[e^{iz}] \) can be written as
  \begin{equation}\label{def:trigonometric_polynomial/exponential}
    p(z) = \sum_{k \in \BbbZ} c_k e^{ikz}
  \end{equation}
  or, using \hyperref[thm:exponential_trigonometric_identities/eulers_formula]{Euler's formula}, rewritten in the more conventional notation (see \cite[1]{Боянов2008} or \cite[88]{Rudin1987RealAndComplex}):
  \begin{equation}\label{def:trigonometric_polynomial/trigonometric}
    p(z) = a_0 + \sum_{k=1}^\infty [ a_k \cos(kz) + b_k \sin(kz) ],
  \end{equation}
  where we denote \( a_k \coloneqq c_k \) and \( b_k \coloneqq ic_k \).

  In particular, when using \fullref{def:trigonometric_polynomial/trigonometric}, we may regard the coefficients \( \{ a_k \}_{k=0}^\infty \) and \( \{ b_k \}_{k=1}^\infty \) as either real or complex, which is a downside of \fullref{def:trigonometric_polynomial/exponential}.

  Denote by \( \tau_n(\BbbK) \) the vector space of all trigonometric polynomials of degree at most \( n \) with coefficients in \( \BbbK \). We also introduce the subspaces \( \tau_n^\alpha{\BbbK} \) of those polynomials which \( a_0 = 0 \).
\end{definition}


\subsubsection{Supervised Learning}
\label{learning}

A conceptual difference between classical and ML methods is the format
    for the model inputs.
In ML models, a time series $Y$ is interpreted as labeled data.
Labels are collected into a vector $\vec{y}$ while the corresponding
    predictors are aligned in an $(T - n) \times n$ matrix $\mat{X}$:
$$
\vec{y}
=
\begin{pmatrix}
    y_T \\
    y_{T-1} \\
    \dots \\
    y_{n+1}
\end{pmatrix}
~~~~~~~~~~
\mat{X}
=
\begin{bmatrix}
    y_{T-1} & y_{T-2} & \dots & y_{T-n} \\
    y_{T-2} & y_{T-3} & \dots & y_{T-(n+1)} \\
    \dots   & \dots   & \dots & \dots \\
    y_n     & y_{n-1} & \dots & y_1
\end{bmatrix}
$$
The $m = T - n$ rows are referred to as samples and the $n$ columns as
    features.
Each row in $\mat{X}$ is "labeled" by the corresponding entry in $\vec{y}$,
    and ML models are trained to fit the rows to their labels.
Conceptually, we model a functional relationship $f$ between $\mat{X}$ and
    $\vec{y}$ such that the difference between the predicted
    $\vec{\hat{y}} = f(\mat{X})$ and the true $\vec{y}$ are minimized
    according to some error measure $L(\vec{\hat{y}}, \vec{y})$, where $L$
    summarizes the goodness of the fit into a scalar value (e.g., the
    well-known mean squared error [MSE]; cf., Section \ref{mase}).
$\mat{X}$ and $\vec{y}$ show the ordinal character of time series data:
    Not only overlap the entries of $\mat{X}$ and $\vec{y}$, but the rows of
    $\mat{X}$ are shifted versions of each other.
That does not hold for ML applications in general (e.g., the classical
    example of predicting spam vs. no spam emails, where the features model
    properties of individual emails), and most of the common error measures
    presented in introductory texts on ML, are only applicable in cases
    without such a structure in $\mat{X}$ and $\vec{y}$.
$n$, the number of past time steps required to predict a $y_t$, is an
    exogenous model parameter.
For prediction, the forecaster supplies the trained ML model an input
    vector in the same format as a row $\vec{x}_i$ in $\mat{X}$.
For example, to predict $y_{T+1}$, the model takes the vector
    $(y_T, y_{T-1}, ..., y_{T-n+1})$ as input.
That is in contrast to the classical methods, where we only supply the number
    of time steps to be predicted as a scalar integer.


\subsection{Covariant and contravariant bases}

In element form we write a vector as:

\(v=e_iv^i\)

The indices are raised and lowered to reflect whether the value is covariant or contravariant.

\(v^i\) is contravariant. If the basis moves one way, it moves the other.

\(e_i\) is covariant. If the basis moves, it moves with it.


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Choose Your Method}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises choosing a method for computing volume.
\end{abstract}
\maketitle

\begin{exercise}%%%%%[2016.29]
The region in the plane bounded by $y = e^{-x/2}$ and the $x$-axis for $0 \leq x \leq \ln 2$ is rotated about the $x$-axis. The volume of the resulting solid of revolution is
\[ V = \answer{ \frac{\pi}{2}}. \]
(Hints won't be revealed until after you choose a response.)
\begin{feedback}
If $x$ is used as the slicing variable, then slices are vertical and consequently perpendicular to the axis of rotation. \begin{hint}
Furthermore one side of the region lies along the axis, so the disk method is appropriate in this case.
\begin{hint}
The distance from the axis to the upper edge of the region is $e^{-x/2}$, so
\[ \begin{aligned} V & = \int_0^{\ln 2} \pi \left( e^{-x/2} \right)^2 dx = \pi \int_0^{\ln 2} e^{-x} dx \\ & =  \left. - \pi e^{-x} \right|_{x=0}^{\ln 2} = \pi (-e^{-\ln 2} + e^0) = \pi \left( - \frac{1}{2} + 1 \right) = \frac{\pi}{2}. \end{aligned}\]
\end{hint}
\end{hint}
\end{feedback}
\end{exercise}

\begin{exercise}
 The region in the plane bounded on the right by the curve $x = 2 - y^2$, on the left by the curve $x = y^2$, and on the bottom by $y = 0$ is revolved around the $y$-axis. Compute the volume of the resulting solid.
 \[ V = \answer{\frac{8 \pi}{3}}. \]
 \end{exercise}

 \begin{exercise}%%%%%Based on [2018.S.1]; mirrored axes
Compute the volume of the solid of revolution obtained by rotating the region between $x=0$, $y=0$, and $x=\sqrt{2+3y^2 - 5y^4}$ around the $y$-axis.
\[ V = \answer{2 \pi}. \]
\end{exercise}

\begin{exercise}%%%%%[2015C.14]

The region between the graph of \(y = 1-x^2\) and the \(x\)-axis is rotated around the line \(y=1\). What is the volume of the resulting solid?
\[ V= \answer{ \frac{8 \pi}{5}}. \]
%\begin{multiplechoice}
%\choice{\(\displaystyle \frac{2 \pi}{5}\)}
%\choice{\(\displaystyle \frac{4 \pi}{5}\)}
%\choice{\(\displaystyle \frac{6 \pi}{5}\)}
%\choice[correct]{\(\displaystyle \frac{8 \pi}{5}\)}
%\choice{\(2 \pi\)}
%\choice{\(\displaystyle \frac{12 \pi}{5}\)}
%\end{multiplechoice}
\end{exercise}

\begin{exercise}% Inspired by Fall 2011 final exam question 3.
Find the volume obtained by rotating the region between the graph $x = \frac{1}{2} \sin (y^2)$ and the $y$-axis for $0 \leq y \leq \sqrt{\pi}$ about the $x$-axis.
\[ V = \answer{\pi}. \]
\end{exercise}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2015C.15]

Calculate the volume of the solid obtained by rotating the area between the graphs of \(\displaystyle y = \frac{1}{\sqrt{x^2-1}}\) and the \(x\)-axis for \(1 < x < \sqrt{5}\) around the \(y\)-axis.
\begin{multiplechoice}
\choice{\(\pi\)}
\choice[correct]{\(4 \pi\)}
\choice{\(6 \pi\)}
\choice{\(8 \pi\)}
\choice{\(3 \pi\)}
\choice{\(2 \pi\)}
\end{multiplechoice}

\end{question}

\begin{question}%%%%%[2016C.14]

Let \(f(x)\) be a continuous function that satisfies \(f(0) = 0\) and \(f(x) > 0\) for \(x > 0\). For every \(b > 0\), when the region between the graph of \(y = f(x)\), the \(x\)-axis, and the line \(x=b\) is rotated around the \(x\)-axis, the volume of the resulting solid is \(18 \pi b^2\). What is \(f(x)\)?
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle 9x\)}
\choice{\(\displaystyle 3x^2\)}
\choice[correct]{\(\displaystyle 6 \sqrt{x}\)}
\choice{\(\displaystyle 27 x^{3/2}\)}
\choice{\(\displaystyle 9 x^2\)}
\choice{\(\displaystyle \sqrt{3x}\)}
\end{multiplechoice}
\begin{feedback}
By the disk method, we have that
\[ \int_0^b \pi (f(x))^2 dx = 18 \pi b^2 \]
for each $b > 0$. Solve this equation for $b$.
\begin{hint}
Differentiate both sides with respect to $b$; use the Fundamental Theorem of Calculus to differentiate the left-hand side.
\end{hint}
\end{feedback}
\end{question}

\begin{question}%%%%%[2017C.02]

Find the volume of the solid generated by revolving the region bounded above by \(y = \sec x\) and bounded below by \(y=0\) for \(0 \leq x \leq \pi/3\) about the \(x\)-axis.
\begin{multiplechoice}
\choice{\(\pi\)}
\choice{\(2 \pi\)}
\choice[correct]{\(\pi \sqrt{3}\)}
\choice{\(3 \pi\)}
\choice{\(4 \pi\)}
\choice{none of these}
\end{multiplechoice}

\end{question}

\end{document}