\lab{Principal Component Analysis and Latent Semantic Indexing}{PCA and LSI}
\objective{Understand the basics of principal component analysis and latent semantic indexing.}
\label{lab:pca}

\section*{Principal Component Analysis}
Understanding the variance in complex data is one of the first tasks encountered in exploratory data analysis. For an example, consider the scatter plot  displaying the sepal and petal lengths of 100 different irises shown in Figure \ref{fig:iris_1}.
\begin{figure}
\includegraphics[width=\textwidth]{iris0.pdf}
\caption{Sepal Length vs. Petal Length for 100 iris flowers. Note the strong correlation of these variables.}
\label{fig:iris_1}
\end{figure}
There are three distinct types of iris flowers present: \emph{setosa}, \emph{versicolor}, and \emph{virginica}.
Considering this data, we might ask how to best distinguish the different types of irises based on their given sepal and petal lengths.
We can answer this question by finding the characteristic that causes the greatest variance in the data.
(Greater variance implies a greater ability to distinguish between data points. If the variance is very small, the data are clustered tightly together, and it is difficult to distinguish well.)

Upon examination, we see that the petal length ranges between $3$ and $7$ cm, while the sepal length only ranges between $5$ and $8$ cm. We might be tempted to say that the most distinguishing aspect of irises is their petal length, but this is only considering the features of the data individually, and not collectively. The two features of the data are clearly correlated, and a more careful consideration would lead us to conclude that the most distinguishing aspect of irises is their overall size. Some irises are are much larger than others, while the sepal and petal lengths stay roughly in proportion.
\begin{figure}
\includegraphics[width=\textwidth]{iris2.pdf}
\caption{The vectors indicate the two principal components, which are weighted by their contribution to the variance.}
\label{fig:iris_2}
\end{figure}

Principal Component Analysis (PCA) is a multivariate statistical tool used to orthogonally change the basis of a set of observations from the basis of original features (which may be correlated) into a basis of uncorrelated (in fact, orthonormal) variables called the \emph{principal components}. It is a direct application of the singular value decomposition (SVD) from linear algebra. More specifically, the first principal component will account for the greatest variance in the set of observations, the second principal component will be orthogonal to the first, accounting for the second greatest variance in the set of observations, etc. The first several principal components capture most of the variance in the observation set, and hence provide a great deal of information about the data. By projecting the observations onto the space spanned by the principal components, we can reduce the dimensionality of the data in a manner that preserves most of the variance.

In our iris example, the two principal components are shown in Figure \ref{fig:iris_2}. The first principal component, corresponding intuitively to iris size, accounts for $96\%$ of the variance in the data. The second, which accounts for only $4\%$ of the variance, corresponds to the relative sepal and petal length of irises of the same size.

\subsection*{Computing the Principal Components}
We now explore how to use the SVD to compute the principal components of a dataset.
Throughout this lab we will use the \li{sklearn} iris data set, which can be obtained as follows:
\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la
>>> import sklearn.datasets as datasets
>>> iris = datasets.load_iris()
>>> X = iris.data
\end{lstlisting}
We represent the collection of observations as an $n \times m$ matrix $X$, where each row of $X$ is an observation, and each column is a specific feature.
Let $k = \min(m,n)$.  We will use this later.
In the iris example, $X$ contains 150 observations, each consisting of 4 features (so $k = 4$), as shown below:
\begin{lstlisting}
>>> X.shape
(150L, 4L)
>>> iris.feature_names
['sepal length (cm)',
 'sepal width (cm)',
 'petal length (cm)',
 'petal width (cm)']
\end{lstlisting}

The first step in PCA is to pre-process the data. In particular, we first translate the columns of $X$ to have mean 0.
The data may then be optionally scaled to remove discrepancies arising from different units of measure (i.e. centimeters vs meters), and we call the new matrix containing the centered and scaled data $Y$.
In this lab, we will not have any scaling issues, so we won't address this issue any further.
Thus we can pre-process our iris data simply as follows:
\begin{lstlisting}
>>> Y = X - X.mean(axis=0)
\end{lstlisting}

We next compute the truncated SVD of our centered and scaled data,
\[Y = U\Sigma V^{T}\]
where $U$ is $n \times k$, $\Sigma$ is a $k\times k$ diagonal matrix containing the singular values of $Y$ in decreasing order along the diagonal, and $V$ is $m \times k$. The columns of $V$ are the principal components (which form an orthonormal basis for the space spanned by the observations), and the corresponding singular values provide us information about how much variance is captured in each principal component. More specifically, let $\sigma_{i}$ be the $i$-th non-zero singular value. Then the value
\[\frac{\sigma^2_{i}}{\sum_{j=1}^{k} \sigma^2_{j}}\]
is the percentage of the variance captured by the $i$-th principal component.
We compute the truncated SVD of the iris data and show the variance percentages for each component below:
\begin{lstlisting}
>>> U,S,VT = la.svd(Y, full_matrices=False)
>>> S**2/(S**2).sum() # variance percentages
array([ 0.92461621,  0.05301557,  0.01718514,  0.00518309])
\end{lstlisting}

In general, we are only interested with the first several principal components. But just how many principal components should we keep? There are a number of ways to decide this. One is to only keep the first two principal components, as these enable us to project the data into $2$-dimensional space, which is easy to visualize. Another way is to only keep the set of principal components accounting for a certain percentage (say $80\%$) of the variance. A third method is to examine the \emph{scree plot} of the variance percentages for each principal component, as in Figure \ref{fig:iris_scree}.
\begin{figure}
\includegraphics[width=\textwidth]{iris_scree.pdf}
\caption{Scree plot of the percentage of variance for PCA on the iris dataset.}
\label{fig:iris_scree}
\end{figure}
Upon examination of the iris scree plot, we see that there is a distinct change after the first principal component. This method is referred to as finding the ``elbow" of the scree plot, and we keep all the principal components on the left of the elbow. In the case of the iris data, that is simply the first principal component, which accounts for $92\%$ of the variance.

Once we have decided how many principal components to keep (say the first $l$), we can project the observations from the original feature space onto the principal component space by computing
\begin{equation*}
\widehat{Y} = U_{:,:l}\Sigma_{:l,:l}
\end{equation*}
where $\Sigma_{:l,:l}$ is the first $l$ rows and columns of $\Sigma$ and $U_{:,:l}$ is the first $l$ columns of $U$.
Using the SVD formula, note that
\[
\widehat{Y} = YV_{:,:l},
\]
where $V_{:,:l}$ is the first $l$ columns of $V$.
In this way, we see that the $i$-th row of $\widehat{Y}$ is simply the projection of the $i$-th observation onto the orthonormal set of the first $l$ principal components.
Under this projection, the data is represented in fewer dimensions, and in such a way that accentuates the variance (which can help with finding patterns within the data).

In Figure \ref{fig:iris_pca} we display the transformed iris data set, plotting the first principal component against the second. This reduction helps us to see the distinctions between the three different species, using only two dimensions instead of the full four dimensions of the feature space.
\begin{figure}
\includegraphics[width=\textwidth]{iris_pca.pdf}
\caption{Plot of the transformed iris data, keeping only the first two principal components.}
\label{fig:iris_pca}
\end{figure}

\begin{problem}
Recreate the plot shown in Figure \ref{fig:iris_pca} by performing PCA on the iris dataset, keeping the first two principal components.

\emph{Note:}
If \li{Yhat} is your $150 \times 2$ array of transformed observations, you can access the rows corresponding to the setosa flowers as follows:
\begin{lstlisting}
>>> Yhat[iris.target==0]
\end{lstlisting}
To get the rows corresponding to versicolor and virginica specimens, simply replace the $0$ with $1$ and $2$, respectively.
\end{problem}

\section*{Latent Semantic Indexing}
\emph{Latent Semantic Indexing} (LSI) is an application of PCA which applies the ideas we have discussed to the realm of natural language processing.
In particular, LSI employs the SVD to reduce the dimensionality of a large corpus of text documents in order to enable us to evaluate the similarity between two documents.
Many information-retrieval systems used in government and in industry are based on LSI.

To motivate the problem, suppose we have a large collection of documents dealing with various statistical and mathematical topics.
How can we find an article about PCA?
We might consider simply choosing the article which contains the acronym \emph{PCA} the greatest number of times, but this is a crude method.
A better way is to use a form of PCA on the collection of documents.

In order to do so, we need to represent the documents as numerical vectors.
A standard way of doing this is to define an ordered set of words occurring in the collection of documents (called the \emph{vocabulary}), and then to
represent each document as a vector of word counts from the vocabulary.
More formally, let our vocabulary be $V = \{w_1,w_2,\ldots,w_m\}$.
Then a document is a vector $x  = (x_1,x_2,\ldots,x_m) \in \mathbb{R}^m$ such that $x_i$ is the number of occurrences of word $w_i$ in the document.
In this setup, we represent the entire collection of $m$ documents as an $n \times m$ matrix $X$, where $m$ is the number of vocabulary words and $n$ is the number of documents in our collection, each row being a document vector.
As expected, we let $X_{i,j}$ be the number of times term $j$ occurs in document $i$.
Note that $X$ is often a sparse matrix, as any one document likely doesn't contain most of the vocabulary words.
This mode of representation is called the \emph{bag of words} model for documents.

We calculate the SVD of $X$ without centering or scaling the data so that we may retain the sparsity.
We now have $X = U\Sigma V^T$. Once we have selected the number of principal components to keep, say $l$, we can represent the corpus of documents by the matrix
\[
\widehat{X} = U_{:,:l}\Sigma_{:l,:l} = XV_{:,:l}.
\]
Note that $\widehat{X}$ will no longer be a sparse matrix, but it has dimensions $n \times l$, which is much smaller than $n \times m$ when $l \ll m$.

Now that we have our documents represented in terms of the first $l$ principal components, we can find the similarity between two documents.
Our measure for similarity is just the cosine of the angle between the vectors; a small angle (and hence large cosine) indicates greater similarity, while a large angle (hence small cosine)
indicates greater dissimilarity.
Recall that we can use the inner product to find the cosine of the angle between two vectors.
Under this metric, the similarity between document $i$ and document $j$ (represented by the $i$-th and $j$-th row of $\widehat{X}$, notated $\widehat{X}_i$ and $\widehat{X}_j$, respectively) is just
\[
\frac{\langle \widehat{X}_i, \widehat{X}_j\rangle}{\|\widehat{X}_i\|\|\widehat{X}_j\|}.
\]
To find the document most similar to document $i$, we simply compute
\[
\argmax_{j\neq i} \frac{\langle \widehat{X}_i, \widehat{X}_j\rangle}{\|\widehat{X}_i\|\|\widehat{X}_j\|}.
\]

We now discuss some practical issues involved in creating the bag of words representation $X$ from the raw text.
Our dataset will consist of the US State of the Union addresses from 1945 through 2013, each contained in a separate text file in the folder {\tt Addresses}.
We would like to avoid loading in all of the text into memory at once, and so we will \emph{stream} the documents one at a time.

The first thing we need to establish is the vocabulary set, i.e. the set of unique words that occur throughout the collection of documents.
A Python set object automatically preserves the uniqueness of the elements, so we will create a set, and then iteratively read through the documents, adding the unique words of each
document to the set. As we read in each document, we will remove punctuation and numerical characters and convert everything to lower case.
The following code will accomplish this task:
\begin{lstlisting}
>>> # get list of filepaths to each text file in the folder
>>> import string
>>> from os import listdir
>>> path_to_addresses = "./Addresses/"
>>> paths = [path_to_addresses + p for p in os.listdir(path_to_addresses) if p[-4:]==".txt"]

>>> # helper function to get list of words in a string
>>> def extractWords(text):
>>>     trans = string.maketrans("", "")
>>>     return text.strip().translate(trans, string.punctuation+string.digits).lower().split()

>>> # initialize vocab set, then read each file and add to the vocab set
>>> vocab = set()
>>> for p in paths:
>>>     with open(p, 'r') as f:
>>>         for line in f:
>>>             vocab.update(extractWords(line))
\end{lstlisting}

We now have a set containing all of the unique words in the corpus. However, many of the most common words do not provide important information.
We call these \emph{stop words}. Examples in English include \emph{the, a, an, and, I, we, you, it, there}, etc;
a list of common English stop words is given in {\tt stopwords.txt}.
We remove the stop words from our vocabulary set as follows, and then fix an ordering to the vocabulary by creating a dictionary
whose key-value pairs are of the form (word, index):
\begin{lstlisting}
>>> # load stopwords
>>> with open("stopwords.txt", 'r') as f:
>>>     stopwords = set([w.strip().lower() for w in f.readlines()])

>>> # remove stopwords from vocabulary, create ordering
>>> vocab = {w:i for i, w in enumerate(vocab.difference(stopwords))}
\end{lstlisting}

We are now ready to create the word count vectors for each document, and we store these in a sparse matrix $X$.
It is convenient to use the \li{Counter} object from the \li{collections} module, as this
object automatically counts the occurrences of each distinct element in a list.
\begin{lstlisting}
>>> from scipy import sparse
>>> from collections import Counter
>>> counts = [] # holds the entries of X
>>> doc_index = [] # holds the row index of X
>>> word_index = [] # holds the column index of X

>>> # iterate through the documents
>>> for doc, p in enumerate(paths):
>>>     with open(p, 'r') as f:
>>>         # create the word counter
>>>         ctr = Counter()
>>>         for line in f:
>>>             ctr.update(extractWords(line))
>>>         # iterate through the word counter, store counts
>>>         for word, count in ctr.iteritems():
>>>             try: # only look at words in vocab
>>>                 word_index.append(vocab[word])
>>>                 counts.append(count)
>>>                 doc_index.append(doc)
>>>             except KeyError: # if word isn't in vocab, skip it
>>>                 pass

>>> # create sparse matrix holding these word counts
>>> X = sparse.csr_matrix((counts, [doc_index,word_index]), shape=(len(paths),len(vocab)), dtype=np.float)
\end{lstlisting}

\begin{problem}
Using the techniques of LSI discussed above--applied to the word count matrix $X$, and keeping the first 7 principal components--find the most similar and least similar speeches to both Bill Clinton's 1993 speech and to Richard Nixon's 1974 speech.
Are the results plausible?

\emph{Hint:} Since $X$ is a sparse matrix, you will need to use the SVD method found in \li{scipy.sparse.linalg}.
This method operates slightly differently than the SVD method found in \li{scipy.linalg}, so make sure to read the
documentation.
\label{prob:LSI1}
\end{problem}

The simple bag of words representation is a bit crude, as it fails to consider how some words may be more important than others
in determining the similarity of documents.
Words appearing in few documents tend
to provide more information than words occurring in every document.
For example, while the word \emph{war} might not be considered a stop word,
it is likely to appear in quite a few addresses, whereas \emph{Afghanistan} will not.
Thus two speeches sharing the word \emph{Afghanistan} ought to be considered more
related than two speeches sharing the word \emph{war}.
So while $X_{i,j}$ is a good measure of the importance of term $j$ in document $i$,
we also need to consider some kind of global weight for each term $j$,
indicating how important the term is over the entire collection.
There are a number of different weights we could choose; we choose to employ
the following approach:

Let $t_{j}$ be the total number of times term $j$ appears in the entire
collection of documents.
Define
\begin{equation*}
p_{i,j} = \frac{X_{i,j}}{t_{j}}.
\end{equation*}
We then let
\begin{equation*}
g_{j} = 1 + \sum_{i=1}^{m} \frac{p_{i,j} \log (p_{i,j} + 1)}{\log m},
\end{equation*}
where $m$ is the number of documents in the collection.
We call $g_{j}$ the \emph{global weight} of term $j$.
We replace each term frequency in the matrix $X$ by weighting it globally.
Specifically, we define a matrix $A$ with entries
\begin{equation*}
A_{i,j} = g_{j} \log (X_{i,j} + 1).
\end{equation*}
We can now perform LSI on the matrix $A$, whose entries are both locally and globally weighted.

To calculate the matrix $A$ in a streaming manner, we must alter our code above somewhat:
\begin{lstlisting}
>>> from itertools import izip
>>> from math import log
>>> t = np.zeros(len(vocab))
>>> counts = []
>>> doc_index = []
>>> word_index = []

>>> # get doc-term counts and global term counts
>>> for doc, path in enumerate(paths):
>>>     with open(path, 'r') as f:
>>>         # create the word counter
>>>         ctr = Counter()
>>>         for line in f:
>>>             words = extractWords(line)
>>>             ctr.update(words)
>>>         # iterate through the word counter, store counts
>>>         for word, count in ctr.iteritems():
>>>             try: # only look at words in vocab
>>>                 word_ind = vocab[word]
>>>                 word_index.append(word_ind)
>>>                 counts.append(count)
>>>                 doc_index.append(doc)
>>>                 t[word_ind] += count
>>>             except KeyError:
>>>                 pass

>>> # get global weights
>>> g = np.ones(len(vocab))
>>> logM = log(len(paths))
>>> for count, word in izip(counts, word_index):
>>>     p = count/float(t[word])
>>>     g[word] += p*log(p+1)/logM

>>> # get globally weighted counts
>>> gwcounts = []
>>> for count, word in izip(counts, word_index):
>>>     gwcounts.append(g[word]*log(count+1))

>>> # create sparse matrix holding these globally weighted word counts
>>> A = sparse.csr_matrix((gwcounts, [doc_index,word_index]), shape=(len(paths),len(vocab)), dtype=np.float)
\end{lstlisting}

\begin{problem}
Repeat Problem \ref{prob:LSI1} using the matrix $A$. Do your answers seem more reasonable than before?
\end{problem}


\section{Regression and classification}

Classification:
\begin{align*}
S_n = \{(x^{(t)},y^{(t)}))|t=1,\cdots,n\}\\
x^{(t)} \in \mathbb{R}^d, y^{(t)} \in \{-1,1\}
\end{align*}

Regression:
\begin{align*}
y^{(t)} \in \mathbb{R}\\
f(x,\theta,\theta_0) &= \sum_{i=1}^d (\theta_i x_i + \theta_0) = \\
&=\theta \cdot x + \theta_0
\end{align*}

\subsection{Objective for linear regression}

The empirical risk  $R_n$  is defined as
\begin{align*}
R_ n(\theta ) = \frac{1}{n} \sum _{t=1}^{n} \text {Loss}(y^{(t)} - \theta \cdot x^{(t)})
\end{align*}

where  $(x^{(t)},y^{(t)})$  is the  $t$th training example (and there are $n$ in total), and  Loss  is some loss function, such as hinge loss.

Possible to get \textbf{closed form solution} for gradient because function is concave. Only possible if the $dxd$ matrix $A$ is invertible. Computationally expensive if dimensions are very high like in bag of words approach.

\begin{align*}
\nabla R_ n(\theta ) &= A\theta - b (=0)\\
&= A^{-1}b\\
\text {where } \\
A &= \frac{1}{n} \sum _{t=1}^{n} x^{(t)} ( x^{(t)})^ T\\
b &= \frac{1}{n} \sum _{t=1}^{n} y^{(t)} x^{(t)}
\end{align*}

$b$ is a vector with dimensionality $d$.
\subsection{Gradient based Approach}
Nudge gradient in the opposite direction to find (local) minima.

\begin{align*}
\nabla_{\theta}(y^{(t)} - \theta x^{(t)})^2 / 2 =\\
= (y^{(t)} - \theta x^{(t)}) \nabla_{\theta}(y^{(t)} - \theta x^{(t)}) = \\
= -(y^{(t)} - \theta x^{(t)}) \cdot x^{(t)}
\end{align*}

\begin{itemize}
\item initialize $\theta=0$
\item randomly pick $t=\{1,\cdots,n\}$
\item $\theta = \theta + \eta(y^{(t)} - \theta x^{(t)}) \cdot x^{(t)}$
\end{itemize}

Where $\eta$ is the learning rate (steps) and the learning rate gets smaller the closer you get $\eta_k = \frac{1}{1+k}$

\subsection{Ridge Regression}
Regularization is trying to push away from perfect fit.

\begin{align*}
J_{n, \lambda } (\theta , \theta _0) &= \frac{1}{n} \sum _{t=1}^{n} \frac{(y^{(t)} - \theta \cdot x^{(t)}-\theta _0)^2}{2} + \frac{\lambda }{2} \left\|  \theta  \right\| ^2\\
\nabla_{\theta}(J_{n, \lambda })&= \lambda \theta - (y^{(t)}-\theta x^{(t)})x^{(t)}
\end{align*}

\begin{itemize}
\item initialize $\theta=0$
\item randomly pick $t=\{1,\cdots,n\}$
\item $\theta = \theta + \eta\lambda \theta - (y^{(t)}-\theta x^{(t)})x^{(t)} = (1-\eta \lambda)\theta + \eta(y^{(t)}-\theta \cdot x^{(t)})$
\end{itemize}

\subsection{Kernels}

\begin{align*}
\displaystyle  \phi (x) \displaystyle &= \big [x_1, \, x_2,\,  {x_1}^2,\,  \sqrt{2}x_1x_2,\,  {x_2}^2 \big ]\\
\displaystyle \phi (x') \displaystyle &= \big [x_1^\prime ,\,  x_2^\prime ,\,  {x_1^\prime }^2,\,  \sqrt{2}x_1^\prime x_2^\prime ,\,  {x_2^\prime }^2 \big ]\\
\displaystyle  \phi (x) \cdot \phi (x^\prime )&=\displaystyle {x_1}{x_1^\prime } + {x_2}{x_2^\prime } + {x_1x_1^\prime }^2 + 2{x_1}{x_1^\prime }{x_2}{x_2^\prime } + {x_2x_2^\prime }^2\\
&=\displaystyle \left({x_1}{x_1^\prime } + {x_2}{x_2^\prime }\right)+ \left({x_1}{x_1^\prime } + {x_2}{x_2^\prime }\right)^2\\
&=\displaystyle \left({x_1}{x_1^\prime } + {x_2}{x_2^\prime }\right)+ \left({x_1}{x_1^\prime } + {x_2}{x_2^\prime }\right)^2
\end{align*}

\subsection{Kernel Perceptron}

The parameter vector of a preceptron algorithm can also be written as:
\begin{align*}
\displaystyle  \displaystyle \theta = \sum _{j=1}^{n} \alpha _ j y^{(j)} \phi \left(x^{(j)}\right)
\end{align*}

Where $\alpha_j$ represents the number of classification mistakes the perceptron algo made. Every time a missclassification happens the parameter vector is updated with the product of the label and the feature vector $\theta  = \theta + y^{(i)} x^{(i)}$. The goal of the Kernel Perceptron algo is to find the vector $\alpha$ with the counts of the missclassifications.

\textbf{Kernel Perceptron}$\displaystyle \left(\big \{ (x^{(i)}, y^{(i)}), i=1,...,n, T \big \} \right)$
\begin{enumerate}[\indent {}]
	\item initialize  $\alpha _1, \alpha _2, ..., \alpha _ n;$ to some values
	\begin{enumerate}[\indent {}]
		\item for $t=1,\cdots,T$ do
		\begin{enumerate}[\indent {}]
			\item for $i=1,\cdots,n$ do
			\begin{enumerate}[\indent {}]
				\item if $y^{(i)}\sum _{j=1}^{n} \alpha _ j y^{(j)} K(x^{j},x^{i}) \leq 0$ then
				\item update $\alpha _ i = \alpha _ i +1 y^{(i)}$
			\end{enumerate}
		\end{enumerate}
	\end{enumerate}
\end{enumerate}

\subsection{Radial basis Kernel}
\begin{align*}
K(x,x') = e^{-\frac{1}{2} {||x-x'||}^2}
\end{align*}


\subsection{Riemann sums}

Given a function \(f(x)\) and an interval \([a,b]\), we can divide \([a,b]\) into \(n\) sections and calculate:

\(\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})\)

This is the Riemann sum.

\subsection{Riemann integral}

We take the limit of the Riemann sum as \(n\rightarrow \infty\)

\(\int_a^b f(x)dx:= \lim_{n\rightarrow \infty } \sum_{j=0}^{n(b-a)} f(a+ \dfrac{j}{n} )\)

\subsection{Linearity}

\(\int_a^bf(x)+g(x)dx=\lim_{n\rightarrow \infty }\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})+g(a+\dfrac{j}{n})\)

\(\int_a^bf(x)+g(x)dx=\lim_{n\rightarrow \infty }\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})+\lim_{n\rightarrow \infty }\sum_{j=0}^{n(b-a)}g(a+\dfrac{j}{n})\)

\(\int_a^bf(x)+g(x)dx=\int_a^bf(x)dx +\int_a^bg(x)dx\)

\subsection{Continuation}

\(\int_a^bf(x)dx+\int_b^cf(x)dx=\lim_{n\rightarrow \infty }\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})+\lim_{n\rightarrow \infty }\sum_{j=0}^{n(c-b)}f(b+\dfrac{j}{n})\)

\(\int_a^bf(x)dx+\int_b^cf(x)dx=\lim_{n\rightarrow \infty }[\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})+\sum_{j=0}^{n(c-b)}f(b+\dfrac{j}{n})]\)

\(\int_a^bf(x)dx+\int_b^cf(x)dx=\lim_{n\rightarrow \infty }[\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})+\sum_{j=n(b-a)}^{n(c-b)+n(b-a)}f(b+\dfrac{j-n(b-a)}{n})]\)

\(\int_a^bf(x)dx+\int_b^cf(x)dx=\lim_{n\rightarrow \infty }[\sum_{j=0}^{n(b-a)}f(a+\dfrac{j}{n})+\sum_{j=n(b-a)}^{n(c-a)}f(a+\dfrac{j}{n})]\)

\(\int_a^bf(x)dx+\int_b^cf(x)dx=\lim_{n\rightarrow \infty }[\sum_{j=0}^{n(c-a)}f(a+\dfrac{j}{n})]\)

\(\int_a^bf(x)dx+\int_b^cf(x)dx=\int_a^cf(x)dx\)


\section{A Real Time Spectrum Analyser Using LMS}
\subsection{LS solution and relationship to DFT}
Given the cost function $\mathcal{J}(\mathbf w)$between estimated signal $\mathbf {\hat y}(n)=\mathbf{F}\mathbf{w}$ and true signal $\mathbf y(n)$,
\begin{align}
\min_{\mathbf{w}}\|\mathbf{y}-\mathbf {\hat y}\|^2
&=\min_{\mathbf{w}}\|\mathbf{y}-\mathbf {\hat y}\|^H \|\mathbf{y}-\mathbf {\hat y}\|\notag\\
&=\min_{\mathbf{w}}(\mathbf{y}-\mathbf{F}\mathbf{w})^H (\mathbf{y}-\mathbf{F}\mathbf{w}) \notag\\
&=\min_{\mathbf{w}}(\mathbf{y}^H\mathbf{y}-\mathbf{y}^H\mathbf{F}\mathbf{w}-\mathbf{w}^H\mathbf{F}^H\mathbf{y}+\mathbf{w}^H\mathbf{F}^H\mathbf{F}\mathbf{w})
\label{eq:cost}
\end{align}
Thus, in order to minise the cost function, take the derivation with respect to $\mathbf w$ and equal to zero. The optimal weight can be obtained.
\begin{align}
\frac{\partial \mathcal{J}}{\partial \mathbf{w}}
&=-\mathbf{F}^H\mathbf{y}-\mathbf{F}^H\mathbf{y}+2\mathbf{F}^H\mathbf{F}\mathbf{w}=0\notag\\
\mathbf{w}&=(\mathbf{F}^H\mathbf{F})^{-1}\mathbf{F}^H\mathbf{y}
\label{eq:ls}
\end{align}
The DFT of a sequence $x(n)$ is defined as
\begin{align}
X_k =\sum_{n=0}^{N-1}\hat x_n e^{-j2\pi kn/N}=\sum_{n=0}^{N-1}\hat x_n W^{nk}_N
\label{eq:dft}
\end{align}
where $W_N=e^{-j2\pi/N}$. In vector expression of Eq.\ref{eq:dft}, it is
\begin{align}
\mathbf{X=W\hat x}
\end{align}
where transformation matrix $\mathbf{W} is$
\begin{align}
\mathbf{W}=
\begin{bmatrix}
    1 & 1 & 1 & \dots  & 1 \\
    1 & W_N & W_N^2 & \dots  & W_N^{N-1} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    1 & W_N^{N-1} & W_N^{2(N-1)} & \dots  & W_N^{(N-1)^2}
\end{bmatrix}
\end{align}
Due to the transformation matrix $\mathbf{W}$ is orthogonal and symmetric, the DFT is unitray transform. Therefore, the IDFT is
\begin{align}
\mathbf{\hat x=W^{-1}X}\quad \text{with}\quad \mathbf{W^{-1}}=\frac{1}{N}\mathbf{W^H}
\end{align}
Which is corresponding to the estimated signal $\mathbf {\hat y}=\mathbf{F}\mathbf{w}$. Thus, the Eq.\ref{eq:ls} can be expressed as
\begin{align}
\mathbf{w}
=(\mathbf{F}^H\mathbf{F})^{-1}\mathbf{F}^H\mathbf{y}
=(N\mathbf{F^{-1}F})^{-1}\mathbf{F}^H\mathbf{y}
=N\mathbf{F}^H\mathbf{y}
=\mathbf F^{-1}\mathbf{y}
\end{align}
where is corresponding to the DFT $\mathbf{X=W\hat x}$.
\subsection{Projection and baiss of DFT}
Due to the transformation matrix of DFT is symmetric and orthogonal, the Fourier trainsfrom coefficients $\mathbf{w}$ is formed by projecting the signal $y(n)$ onto the transformation matrix $\mathbf{W}$ which is composed of harmonical sinusoids basises. In contrast, the IDFT is the inversely process to reconstruct the original signal by superposition of sinusoidal projections. However, the DFT and IDFT use finite length $N$ to estimate either coefficients $\mathbf{w}$ or signal $\mathbf{\hat y}$, which makes errors occur comparing with true value.
\subsection{DFT-CLMS}
The DFT-CLMS algorithm is implimented on the non-stationary FM signal. Fig.\ref{fig:3_3_c1} depicts the estimated time-varying frequencies. The trend of frequency is generally captured especially in perfectly estimation for constant segment. However, there exists an issue that the estimated frequency remains till the end of time index once it was obtained. Thus, the weights are not updated, which presents long smears in spectrum. The reason which causes this problem is that the gradient of back-propagation vanishes. Due to the gradient of the LMS algorithm is based on the eigenvalues of autocorrelation matrix. For the harmonically related sinusoids $\mathbf x(n)$, there are zero eigenvalues of $\mathbf R_{xx}$, resulting in hard back-propagation.
\begin{figure}[htb]
     \centering
     \includegraphics[width=0.4\textwidth]{fig/33/33c1.eps}
     \caption{DFT-CLMS: estimated FM frequency}
     \label{fig:3_3_c1}
\end{figure}\\
To solve this problem, the Leaky CLMS is applied on the weight update with leakage coefficient, in expression of $\gamma$ $\mathbf{w}(n+1)=(1-\gamma\mu)\mathbf{w}(n)+\mu e^*(n)\mathbf x(n)$. By changing previous value of weight, the weight will update along time. Fig.\ref{fig:3_3_c2} illustrated the performance affected by different $\gamma$. With small value of $\gamma$, most of smears are removed presented as relatively distinct trends. The optimal value of $\gamma$ is 0.1 with acceptable bias. If the leakage coefficient is too large, the estimation is inaccurate due to the large bias added.
\begin{figure}[htb]
     \centering
     \hspace{-0.4cm}
     \begin{subfigure}[b]{0.33\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/33/33c2.eps}
     \end{subfigure}
    \hspace{-0.4cm}
     \begin{subfigure}[b]{0.33\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/33/33c3.eps}
     \end{subfigure}
    \hspace{-0.4cm}
     \begin{subfigure}[b]{0.33\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/33/33c4.eps}
     \end{subfigure}
     \hspace{0.4cm}
        \caption{Leaky DFT-CLMS: estimated FM frequency with different $\gamma$}
        \label{fig:3_3_c2}
\end{figure}
\subsection{DFT-CLMS estimated EEG signal}
The DFT-CLMS algorithm can also be used to analyse EEG signal. As shown in Fig.\ref{fig:3_3_d}, the estimated spectrum agrees with the analysis in Part 1.2(b). A strong response at 8-10$Hz$ called alpha-rhythm is clear shown. The SSVEP at 13$Hz$ is detected as well, following its harmonic frequency at $26Hz$. However, the harmonic frequency at $39Hz$ is hard to recognize. Moreover, the recording apparatus is strongly detected at $50Hz$.\\
Nevertheless, the Leaky CLMS algorithm is not suitable for EEG data, since the EEG \texttt{POz} is stationary signal. Thus, adding a leakage coefficient results in adding bias on the correct estimations.
\begin{figure}[htb]
     \centering
     \hspace{0.4cm}
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/33/33d1.eps}
     \end{subfigure}
    \hspace{0.4cm}
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/33/33d2.eps}
     \end{subfigure}
     \caption{DFT-CLMS: estimated EEG frequency}
     \label{fig:3_3_d}
\end{figure}


% section7

\chapter{Dimension and Structure}

\section{Basis and Dimension}

\begin{exer} (\textit{Linear Combination and Independence})\\
Are any of the vectors in the set
\vspace{1mm}
\begin{displaymath}
S = \{ (2,6,3,4,2), \hspace{1.5mm}(3,1,5,8,3), \hspace{1.5mm}(5,1,2,6,7), \hspace{1.5mm}(8,4,3,2,6), \hspace{1.5mm}(5,5,6,3,4) \}
\end{displaymath}
\vspace{1mm}
linear combinations of predecessors? Justify your answer.

\end{exer}

\begin{sol}
\verb""
One strategy is to form a matrix $V$ of the column vectors $\mathbf{v}_{k}$ mentioned above and decide whether the system $V \mathbf{x} = \mathbf{0}$ has nontrivial solutions. If so, then at least one column is a linear combination of previous ones. Otherwise, the columns are linearly independent.

\begin{verbatim}

v1 = [2 6 3 4 2]'; v2 = [3 1 5 8 3]'; v3 = [5 1 2 6 7]';
v4 = [8 4 3 2 6]'; v5 = [5 5 6 3 4]';

% Construct V of the column vectors v1,v2,v3,v4 and v5.
V = [v1 v2 v3 v4 v5];

format short;

% Find the reduced row echelon form of V.
rref_V = rref(V);

disp('The reduced row echelon form of A is'); disp(rref_V);
\end{verbatim}

\begin{outputs}

\begin{verbatim}

The reduced row echelon form of A is
     1     0     0     0     0
     0     1     0     0     0
     0     0     1     0     0
     0     0     0     1     0
     0     0     0     0     1
\end{verbatim}

\end{outputs}

\noindent Since the reduced row echelon form of $V$ has 5 pivots, the columns of V are linearly independent. Hence, no column of $V$ can be a linear combination of any other columns.
\end{sol}

\vspace{2mm}

\section{Properties of Bases}

\begin{exer}
In this problem, we make a function file \verb"CheckBasis.m" to check that the vectors $\mathbf{v}_{1}$, $\mathbf{v}_{2}$, $\mathbf{v}_{3}$ and $\mathbf{v}_{4}$ form a basis of $\mathbb{R}^4$ using the equivalent statements (a), (g), (h), and (o) of Theorem 7.2.7 in the textbook.

\begin{enumerate}
\item[(a)]
Complete the shadow part (/////) of the m-file given below referring to the comments and the execution results.

\begin{verbatim}

%--- your function file ---%
function [Result]=CheckBasis(v1, v2, v3, v4, case_num)
  %  if case_num=1, check the statement (a),
  %  if case_num=2, check the statement (g),
  %  if case_num=3, check the statement (h).

  % Construct the matrix V.
  ////////////////////////

  % Use the switch statement to check
  % whether one of the statements (a), (g), and (h) holds.
  switch case_num
    case 1
      fprintf('* You enter %d: statement (a) *\n', case_num);
      ////////////////////////
      if ////////////////////////
        disp('Given vectors form a basis of 4 dimensional space.');
      else
        disp('Given vectors do not form a basis of 4 dimensional space.');
      end
    case 2
      fprintf('* You enter %d: statement (g) *\n', case_num);
      Result=det(V);
      if Result~=0
        disp('Given vectors form a basis of 4 dimensional space.');
      else
        disp('Given vectors do not form a basis of 4 dimensional space.');
      end
    ///////////////////////////  % check statement (h)
        ////////////////////////
        ////////////////////////
        ////////////////////////
        ////////////////////////
        ////////////////////////
        ////////////////////////
        ////////////////////////
  end
end
\end{verbatim}

\vspace{3mm}

\noindent The execution results will be as follows:
\begin{verbatim}
>> v1=[1 0 0 0]'; v2=[0 2 0 0]'; v3=[0 0 4 5]'; v4=[0 0 0 -1]'; v5=[0 0 0 1]';
>> C=CheckBasis(v1, v2, v3, v4,3)
* You enter 3: statement (h) *
  Given vectors are basis of 4 dimensional space.

C =

     1
     2
    -1
     4

>> CheckBasis(v1, v2, v4, v5, 1);
* You enter 1: statement (a) *
  Given vectors do not form a basis of 4 dimensional space.

>> determinant=CheckBasis(v1, v2, v3, v5, 2)
* You enter 2: statement (g) *
  Given vectors form a basis of 4 dimensional space.

determinant =
     8
\end{verbatim}

\vspace{3mm}
\item[(b)] Using \verb"CheckBasis.m" from (a), check whether
\begin{itemize}
\item[i.] $\mathbf{v}_{1}=(-1, 0, 1, 0)^{T}$, $\mathbf{v}_{2}=(2, 3, -2, 6)^{T}$, $\mathbf{v}_{3}=(0, -1, 2, 0)^{T}$ and $\mathbf{v}_{4}=(0, 0, 1, 5)^{T}$ form a basis of $\mathbb{R}^4$.

\item[ii.] $\mathbf{v}_{1}=(a, b, c, d)^{T}$, $\mathbf{v}_{2}=(-b, a, d, -c)^{T}$, $\mathbf{v}_{3}=(-c, -d, a, b)^{T}$ and $\mathbf{v}_{4}=(-d, c, -b, a)^{T}$ form a basis of $\mathbb{R}^4$. (Do not use the statement (h). Guess why not?)
\end{itemize}

\end{enumerate}
\end{exer}

\begin{sol}

\vspace{1mm}
\verb""
\begin{enumerate}
\item[(a)]
\verb" "
\verb"% ----- your function file ----- %"\\
\verb"function [Result]=CheckBasis(v1, v2, v3, v4, case_num)"\\
\verb"  %  if case_num=1, check the statement (a),"\\
\verb"  %  if case_num=2, check the statement (g),"\\
\verb"  %  if case_num=3, check the statement (h)."\\
\\
\verb" % Construct the matrix V."\\
  \textcolor{blue}{\texttt{  V=[v1 v2 v3 v4];}}\\

\verb"  % Use the switch statement to check" \\
\verb"  % whether one of the statements (a), (g), and (h) holds."\\
\verb"  switch case_num"\\
\verb"    case 1"  \\
\verb"      fprintf('* You enter %d: statement (a) *', case_num);"\\
\textcolor{blue}{\texttt{ \hspace{8mm}         Result=rref(V)}}

\hspace{8mm}      \texttt{ if \textcolor{blue}{det(Result)$\sim$=0}}\\
\verb"        disp('Given vectors form a basis of 4 dimensional space.');"\\
\verb"      else"\\
\verb"        disp('Given vectors do not form a basis of 4 dimensional space.');"\\
\verb"      end"\\
\verb"    case 2"\\
\verb"      fprintf('* You enter %d: statement (g) *', case_num);"\\
\verb"      Result=det(V);"\\
\verb"      if Result~=0"\\\
\verb"        disp('Given vectors form a basis of 4 dimensional space.');"\\
\verb"      else"\\
\verb"        disp('Given vectors do not form a basis of 4 dimensional space.');"\\
\verb"      end"\\
\textcolor{blue}{\texttt{  \hspace{4mm}  case 3}}\verb"  % check statement (h)"\\
\textcolor{blue}{\texttt{  \hspace{8mm}      [Q D]=eig(V);}}

\textcolor{blue}{\texttt{  \hspace{8mm}            Result=diag(R);}}

\textcolor{blue}{\texttt{  \hspace{8mm}         if det(R)==0}}

\textcolor{blue}{\texttt{   \hspace{12mm}       disp('Given vectors form a basis of 4 dimensional space.');}}

\textcolor{blue}{\texttt{  \hspace{8mm}              else}}

 \textcolor{blue}{\texttt{  \hspace{12mm}               disp('Given vectors do not form a basis of 4 dimensional space.');}}

\textcolor{blue}{\texttt{  \hspace{8mm}        end}}

\textcolor{blue}{\texttt{  \hspace{4mm}      end}}

\item[(b)-i.]
\begin{verbatim}
>> v1=[-1 0 1 0]'; v2=[2 3 -2 6]';  v3=[0 -1 2 0]'; v4 = [0 0 1 5]';
>> CheckBasis(v1, v2, v3, v4, 1);
>> CheckBasis(v1, v2, v3, v4, 2);
>> CheckBasis(v1, v2, v3, v4, 3);
\end{verbatim}

\begin{outputs}
\begin{verbatim}

* You enter 1: statement (a) *
  Given vectors form a basis of 4 dimensional space.
* You enter 2: statement (g) *
  Given vectors form a basis of 4 dimensional space.
* You enter 3: statement (h) *
  Given vectors form a basis of 4 dimensional space.
\end{verbatim}
\end{outputs}

\item[(b)-ii.]
\begin{verbatim}
>> syms a b c d;
>> v1=[a;b;c;d]; v2=[-b;a;d;-c]; v3=[-c;-d;a;b]; v4 = [-d;c;-b;a];
>> CheckBasis(v1, v2, v3, v4, 1);
>> CheckBasis(v1, v2, v3, v4, 2);
\end{verbatim}

\begin{outputs}
\begin{verbatim}

* You enter 1: statement (a) *
  Given vectors form a basis of 4 dimensional space.
* You enter 2: statement (g) *
  Given vectors form a basis of 4 dimensional space.
\end{verbatim}
\end{outputs}
\end{enumerate}
\end{sol}

\section{The Fundamental Spaces of a Matrix}

\begin{exer}\label{ex1}
In this problem, we make a function file \verb"getFSinfo.m" to get the dimension and basis of the fundamental spaces of a given matrix. For example, we execute the followings:
\begin{verbatim}
>> A=[1 0 0 0 2; -2 1 -3 -2 -4; 0 5 -14 -9 0; 2 10 -28 -18 4];
>> getFSinfo(A);
\end{verbatim}
Then, the Command Window displays the results as follows:
\begin{verbatim}
Given matrix is:
     1     0     0     0     2
    -2     1    -3    -2    -4
     0     5   -14    -9     0
     2    10   -28   -18     4

== Dimension of the fundamental spaces of a given matrix ==
dim(row(A))=dim(col(A)): 3, 	 dim(null(A)): 2, 	 dim(null(A_trans)): 1

== Basis of the fundamental spaces of a given matrix (in row vectors) ==
 row(A)
     1     0     0     0     2
     0     1     0     1     0
     0     0     1     1     0

 col(A)
     1     0     0     2
     0     1     0     0
     0     0     1     2

 null(A)
     0    -1    -1     1     0
    -2     0     0     0     1

 null(A_trans)
    -2     0    -2     1

*****************************************************
\end{verbatim}
\begin{enumerate}
\item[(a)] Complete the missing parts of the m-file \verb"getFSinfo" given as follows:
\begingroup
    \fontsize{10pt}{12pt}\selectfont
\begin{verbatim}

%--- function file 'getFSinfo.m' ---%
function [info]=getFSinfo(A)
  % row(A): basis and dimension
  /////// missing part ///////

  % col(A): basis and dimension
  /////// missing part ///////

  % null(A): basis and dimension
  /////// missing part ///////

  % null(A'): basis and dimension
  /////// missing part ///////

  disp('Given matrix is:'); disp(A);
  fprintf('== Dimension of the fundamental spaces of given matrix == \n');
  fprintf('dim(row(A))=dim(col(A)): %d,', rank_A);
  fprintf('\t dim(null(A)): %d,\t dim(null(A_trans)): %d \n\n', nullity, nullity_T);
  fprintf('== Basis of the fundamental spaces of given matrix (in row vectors) == \n');
  disp(' row(A)'); disp(double(rowA_basis));
  disp(' col(A)'); disp(double(colA_basis));
  disp(' null(A)'); disp(nullA_basis);
  disp(' null(A_trans)'); disp(nullAtrans_basis);
  fprintf('\n*****************************************************\n');
end
\end{verbatim}
\endgroup
You may use the MATLAB commands \textit{rank, colspace, rref, null} and so on.
\vspace{3mm}
\item[(b)] Using your function file \verb"getFSinfo.m", find the dimension and basis of the fundamental spaces of

$A = \begin{bmatrix} 3 & 2& 1& 3& 5 \\ 6& 4& 3& 5& 7 \\ 9&6&5&7&9\\3&2&0&4&8 \end{bmatrix}$, $B = \begin{bmatrix} 3 & -1 & 3& 2& 5 \\ 5& -3&2&3&4\\ 1&-3&-5&0& -7\\ 7&-5&1&4&1\end{bmatrix}$, $C = \begin{bmatrix} 1& 3& 2& 1\\ -2 & -6 & 0& -6\\ 3& 9& 1& 8\\ -1& -3& -3& -6\\ 1& 3& 2& 1\\ 4& 12& 1&11 \end{bmatrix}$.
\end{enumerate}
\end{exer}

\begin{sol}
\verb""
\begin{enumerate}
\item[(a)]

\verb"% ----- function file 'getFSinfo.m' ----- %"

\verb"function [info]=getFSinfo(A)"

\verb"  [m,n]=size(A)"

\verb"  % row(A): basis and dimension"

{\color{blue}\verb"    rank_A=rank(A);    % rank of A;"

    \verb"    rowA=colspace(sym(A')); % find the row basis"

    \verb"    rowA_basis=rowA(:, 1:rank_A)';  % basis of row(A)"
    }

\verb"  % col(A): basis and dimension"

{\color{blue}\verb"    colA=colspace(sym(A)); % find the column basis"

\verb"    colA_basis=colA(:, 1:rank_A)';  % basis of col(A)"
    }

\verb"  % null(A): basis and dimension"

{\color{blue}\verb"    nullA=null(A, 'r');"

\verb"    nullity=n-rank_A;   % using Dimension theorem"

\verb"    nullA_basis=nullA(:, 1:nullity)';"
    }

\verb"  % null(A'): basis and dimension"

{\color{blue}\verb"    nullAtrans=null(A', 'r');"

    \verb"    nullity_T=m-rank_A;    "

    \verb"    nullAtrans_basis=nullAtrans(:,1:nullity_T)';"
    }
\begin{verbatim}
  disp('Given matrix is:'); disp(A);
  fprintf('== Dimension of the fundamental spaces of given matrix == \n');
  fprintf('dim(row(A))=dim(col(A)): %d,', rank_A);
  fprintf('\t dim(null(A)): %d,', nullity);
  fprintf('\t dim(null(A_trans)): %d \n\n', nullity_T);
  fprintf('== Basis of the fundamental spaces ');
  fprintf('of given matrix (in row vectors) == \n');
  disp(' row(A)'); disp(double(rowA_basis));
  disp(' col(A)'); disp(double(colA_basis));
  disp(' null(A)'); disp(nullA_basis);
  disp(' null(A_trans)'); disp(nullAtrans_basis);
  fprintf('\n*****************************************************\n');
end
\end{verbatim}

\item[(b)]
\begin{verbatim}
A=[3 2 1 3 5; 6 4 3 5 7; 9 6 5 7 9; 3 2 0 4 8];
B=[3 -1 3 2 5; 5 -3 2 3 4; 1 -3 -5 0 -7; 7 -5 1 4 1];
C=[1 3 2 1; -2 -6 0 -6 ;3 9 1 8; -1 -3 -3 -6; 1 3 2 1; 4 12 1 11];
getFSinfo(A);
getFSinfo(B);
getFSinfo(C);
\end{verbatim}

\begin{outputs}
\begin{verbatim}

Given matrix is:
     3     2     1     3     5
     6     4     3     5     7
     9     6     5     7     9
     3     2     0     4     8
== Dimension of the fundamental spaces of given matrix ==
dim(row(A))=dim(col(A)): 2,	 dim(null(A)): 3,	 dim(null(A_trans)): 2
== Basis of the fundamental spaces of given matrix (in row vectors) ==
 row(A)
    1.0000    0.6667         0    1.3333    2.6667
         0         0    1.0000   -1.0000   -3.0000
 col(A)
     1     0    -1     3
     0     1     2    -1
 null(A)
   -0.6667    1.0000         0         0         0
   -1.3333         0    1.0000    1.0000         0
   -2.6667         0    3.0000         0    1.0000
 null(A_trans)
     1    -2     1     0
    -3     1     0     1
*****************************************************

Given matrix is:
     3    -1     3     2     5
     5    -3     2     3     4
     1    -3    -5     0    -7
     7    -5     1     4     1
== Dimension of the fundamental spaces of given matrix ==
dim(row(A))=dim(col(A)): 3,	 dim(null(A)): 2,	 dim(null(A_trans)): 1
== Basis of the fundamental spaces of given matrix (in row vectors) ==
 row(A)
    1.0000         0    1.7500    0.7500         0
         0    1.0000    2.2500    0.2500         0
         0         0         0         0    1.0000
 col(A)
     1     0    -3     0
     0     1     2     0
     0     0     0     1
 null(A)
   -1.7500   -2.2500    1.0000         0         0
   -0.7500   -0.2500         0    1.0000         0
 null(A_trans)
     3    -2     1     0
*****************************************************

Given matrix is:
     1     3     2     1
    -2    -6     0    -6
     3     9     1     8
    -1    -3    -3    -6
     1     3     2     1
     4    12     1    11
== Dimension of the fundamental spaces of given matrix ==
dim(row(A))=dim(col(A)): 3,	 dim(null(A)): 1,	 dim(null(A_trans)): 3
== Basis of the fundamental spaces of given matrix (in row vectors) ==
 row(A)
     1     3     0     0
     0     0     1     0
     0     0     0     1
 col(A)
    1.0000         0    0.5000         0    1.0000    0.5000
         0    1.0000   -1.2500         0         0   -1.7500
         0         0         0    1.0000         0         0
 null(A)
    -3     1     0     0
 null(A_trans)
   -0.5000    1.2500    1.0000         0         0         0
   -1.0000         0         0         0    1.0000         0
   -0.5000    1.7500         0         0         0    1.0000
*****************************************************
\end{verbatim}
\end{outputs}
\end{enumerate}
\end{sol}

\vspace{3mm}

\begin{exer}
(\textit{Bases for the Fundamental Spaces})

\begin{enumerate}

\item[(a)] Use the MATLAB commands \textit{sym} and \textit{colspace} to find a basis for the column space of the matrix

\begin{displaymath}
A = \left[\begin{array}{rrrr} 2& \hspace{2mm} -1& \hspace{5mm} 3&\hspace{4mm} 5\\ 4 & -3 & 1 & 3 \\ 3 & -2 & 3 & 4 \\ 4 & -1 & 15 & 17 \\ 7 & -6 & -7 & 0 \end{array} \right].
\end{displaymath}

\vspace{1mm}

\item[(b)] Use the same MATLAB commands in (a) to find a basis for the row space of $A$.

\vspace{1mm}

\item[(c)] Confirm that the basis obtained in (b) is consistent with the basis obtained from the reduced row echelon form of $A$.

\vspace{1mm}

\item[(d)] Tell what happens if you use the MATLAB command \textit{orth}?

\end{enumerate}
\end{exer}

\begin{sol}
\verb""

%\vspace{3mm}

\begin{itemize}
\item[(a)]
\begin{verbatim}
% Set a matrix A whose entries are symbolic objects.
A = sym([2 -1 3 5; 4 -3 1 3; 3 -2 3 4; 4 -1 15 17; 7 -6 -7 0]);

% Find a basis for the column space of A.
col_basis = colspace(A);

disp('A basis for the column space of A is');
disp(col_basis(:,1)'); disp(col_basis(:,2)'); disp(col_basis(:,3)');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

A basis for the column space of A is
[ 1, 0, 0, 2, 1]

[ 0, 1, 0, -3, 5]

[ 0, 0, 1, 4, -5]
\end{verbatim}
\end{outputs}

\item[(b)]
\begin{verbatim}
% Set a matrix A_transpose whose entries are symbolic objects.
A_transpose = sym([2 -1 3 5; 4 -3 1 3; 3 -2 3 4; 4 -1 15 17; 7 -6 -7 0]');

% Finding a basis for the row space of A is equivalent to
% finding a basis for the column space of A_transpose.
rowbasis = colspace(A_transpose);

disp('A basis for the row space of A is');
disp(rowbasis(:,1)'); disp(rowbasis(:,2)'); disp(rowbasis(:,3)');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

A basis for the row space of A is
[ 1, 0, 0, 6]

[ 0, 1, 0, 7]

[ 0, 0, 1, 0]
\end{verbatim}
\end{outputs}

\item[(c)]
\begin{verbatim}
% Set a matrix A.
A = [2 -1 3 5; 4 -3 1 3; 3 -2 3 4; 4 -1 15 17; 7 -6 -7 0];

% Find the reduced row echelon form of A.
rref_A = rref(A);

% The nonzero rows of the reduced row echelon form of A
% form a basis for the row space of A.

disp('A basis for the row space of A is');
disp(rref_A(1,:)); disp(rref_A(2,:)); disp(rref_A(3,:));
\end{verbatim}

\begin{outputs}
\begin{verbatim}

A basis for the row space of A is
     1     0     0     6

     0     1     0     7

     0     0     1     0
\end{verbatim}
\end{outputs}

\item[(d)]

\begin{verbatim}
% Set A.
A = [2 -1 3 5; 4 -3 1 3; 3 -2 3 4; 4 -1 15 17; 7 -6 -7 0];

% The command orth gives an orthonormal basis for the column space of A.
B = orth(A);

disp('An orthonormal basis for the column space of A is');
disp('q1='); disp(B(:,1)');
disp('q2='); disp(B(:,2)');
disp('q3='); disp(B(:,3)');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

An orthonormal basis for the column space of A is
q1=
   -0.2427   -0.1508   -0.2229   -0.9246    0.1177

q2=
   -0.1189   -0.3624   -0.2060    0.0253   -0.9008

q3=
    0.3760   -0.6016   -0.5930    0.1848    0.3331
\end{verbatim}
\end{outputs}
\end{itemize}
\end{sol}

\section{The Dimension Theorem and Its Implications}

\begin{exer}
(\textit{Rank and Nullity})

\begin{enumerate}

\item[(a)] Use the MATLAB command \textit{rank} and the Formula (2) in Section $7.4$ to find the nullity of the matrix

\begin{displaymath}
A = \left[\begin{array}{rrrrr} 3& \hspace{1mm} 2& \hspace{1mm} 1&\hspace{1mm} 3&\hspace{1mm} 5\\ 6 & 4 & 3 & 5 & 7 \\ 9 & 6 & 5 & 7 & 9 \\ 3 & 2 & 0 & 4 & 8 \end{array} \right].
\end{displaymath}

\item[(b)] Confirm that the result obtained in (a) is consistent with the number of basis vectors which are obtained by using the MATLAB command \textit{null}.

\end{enumerate}

\end{exer}

\begin{sol}
\verb""
\begin{itemize}

\item[(a)]

\begin{verbatim}
% Set A.
A = [3 2 1 3 5; 6 4 3 5 7; 9 6 5 7 9; 3 2 0 4 8];

% Find the rank of A by using the command rank.
rank_A = rank(A);

% Size of the matrix A.
[ m n ] = size(A);
% m = the number of rows of A, n = the number of columns of A.

 % By (2) in section 7.4, rankA + nullA = n.
null_A = n - rank_A;

disp('The nullity of A is'); disp(null_A);
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The nullity of A is
     3
\end{verbatim}
\end{outputs}

\item[(b)]
\begin{verbatim}
% Set A.
A = [3 2 1 3 5; 6 4 3 5 7; 9 6 5 7 9; 3 2 0 4 8];

% Find a basis for the null space of A.
nullA = null(A,'r');
% null(A,'r') returns a matrix
% whose columns are a basis for the null space of A.

[m n] = size(nullA);
% Since the number of columns of nullA is n,
% thus, n = the number of basis vectors of the null space of A.

disp('The nullity of A is'); disp(n);
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The nullity of A is
     3
\end{verbatim}
\end{outputs}

\end{itemize}

\end{sol}

\section{The Rank Theorem and Its Implications}

\begin{exer}
Note that the rank of a nonzero matrix $A$ is equal to the order of the largest square submatrix of $A$ (formed by deleting rows and columns of $A$) whose determinant is nonzero. In this problem, we make a function file \verb"CheckRank.m" to find the rank of the given matrix using this fact. We want to obtain the execution results as follows:
\begin{verbatim}
>> A=[1 2 3 4; 5 6 7 8; 9 10 11 12; 13 14 15 16];
>> rankA=CheckRank(A)
rankA =
     2
\end{verbatim}
For this, you may start with the largest square matrices to be found in $A$ and a search is started for the first submatrix with a nonzero determinant. Use the MATLAB command \textit{nchoosek} to select all the combinations of rows and columns needed in the search process and you may use the several MATLAB commands if you need. Complete the m-file below and check the determinant of the matrices $A$, $B$, and $C$ given in the Exercise \ref{ex1} (b). Also, compare the results using the MATLAB command \textit{rank}.
\end{exer}

\begin{sol}
\begin{verbatim}

%--- function file 'CheckRank.m' ---%
function [rank_A]= CheckRank(A)
\end{verbatim}

{\color{blue}
\begin{verbatim}
    [m,n]=size(A);  % size of given matrix
    flg=1;  % flag for while loop
    if m>n  % if (# of row) > (# of col)
        A=A';
    end
    A=sym(A);   % Set A as a symbolic object

    K = min(m,n);   N = max(m,n);   % k : row number, N: col number
    k=K;    % from the largest size of submatrix
    while flg == 1
        comb_row=nchoosek(1:K, k);  % combinations of row
        comb_col=nchoosek(1:N, k);  % combinations of columns
        for ii=1:size(comb_row) %
            selected_A=A(comb_row(ii,:),:); % selected row index
            for jj=1:size(comb_col)
                sub_A=selected_A(:,comb_col(jj, :));   % selected col index
                if det(sub_A)~=0    % if non zeros determinant appears
                    rank_A=k;   % the size at that time <- rank
                    flg=0;  % stop the while loop.
                end
            end
        end
        k=k-1;  % if all submatrices of size k have a zero determinant,
                % reduce the size of submatrix.
    end
\end{verbatim}}
\begin{verbatim}
end
\end{verbatim}

\noindent To check the determinant of the matrices $A$, $B$, and $C$ given in the Exercise \ref{ex1}, you execute the followings:
\begin{verbatim}
A=[3 2 1 3 5; 6 4 3 5 7; 9 6 5 7 9; 3 2 0 4 8];
B=[3 -1 3 2 5; 5 -3 2 3 4; 1 -3 -5 0 -7; 7 -5 1 4 1];
C=[1 3 2 1; -2 -6 0 -6 ;3 9 1 8; -1 -3 -3 -6; 1 3 2 1; 4 12 1 11];

fprintf('my rank(A): %.5f, MATLAB rank(A): %.5f \n', CheckRank(A), rank(A));
fprintf('my rank(B): %.5f, MATLAB rank(B): %.5f \n', CheckRank(B), rank(B));
fprintf('my rank(C): %.5f, MATLAB rank(C): %.5f \n', CheckRank(C), rank(C));
\end{verbatim}

\begin{outputs}
\begin{verbatim}

my rank(A): 2.00000, MATLAB rank(A): 2.00000
my rank(B): 3.00000, MATLAB rank(B): 3.00000
my rank(C): 3.00000, MATLAB rank(C): 3.00000
\end{verbatim}
\end{outputs}

\noindent Those are the same results as given in Exercise\ref{ex1}.

\end{sol}

% 7.6
\section{The Pivot Theorem and Its Implications}

\begin{exer}(\textit{Finding a Basis with the Pivot Theorem})\\
Consider the vectors
\begin{displaymath}
\begin{array}{lll} \vspace{1mm} \hspace{2mm} \mathbf{v}_{1} & \hspace{1mm} = & \hspace{1mm} (1, \hspace{1mm} 2, \hspace{1mm} 4, \hspace{1mm} -6,  \hspace{1mm} 11, \hspace{1mm} 23, \hspace{1mm} -14, \hspace{1mm} 0, \hspace{1mm} 2, \hspace{1mm} 2), \\ \vspace{1mm} \hspace{2mm} \mathbf{v}_{2} & \hspace{1mm} = & \hspace{1mm} (3, \hspace{1mm} 1, \hspace{1mm} -1, \hspace{1mm} 7, \hspace{1mm} 9, \hspace{1mm} 13, \hspace{1mm} -12, \hspace{1mm} 8, \hspace{1mm} 6, \hspace{1mm} -30), \\ \vspace{1mm} \hspace{2mm} \mathbf{v}_{3} & \hspace{1mm} = & \hspace{1mm} (5, \hspace{1mm} 5, \hspace{1mm} 7, \hspace{1mm} -5, \hspace{1mm} 31, \hspace{1mm} 59, \hspace{1mm} -40, \hspace{1mm} 8, \hspace{1mm} 10, \hspace{1mm}-26), \\ \vspace{1mm} \hspace{2mm} \mathbf{v}_{4} & \hspace{1mm} = & \hspace{1mm} (5, \hspace{1mm} 0, \hspace{1mm} -6, \hspace{1mm} 20, \hspace{1mm} 7, \hspace{1mm} 3, \hspace{1mm} -10, \hspace{1mm} 16, \hspace{1mm} 10, \hspace{1mm} -62). \end{array}
\end{displaymath}
Use Algorithm $1$ in Section $7.6$ to find a subset of these vectors that forms a basis for span$\{\mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{3}, \mathbf{v}_{4}\}$, and express those vectors not in the basis as linear combinations of basis vectors.
\end{exer}

\begin{sol}

\begin{verbatim}

v1 = [1 2 4 -6 11 23 -14 0 2 2]';
v2 = [3 1 -1 7 9 13 -12 8 6 -30]';
v3 = [5 5 7 -5 31 59 -40 8 10 -26]';
v4 = [5 0 -6 20 7 3 -10 16 10 -62]';

% Construct A whose column space is W=span(v1,v2,v3,v4).
A = [v1 v2 v3 v4];

% Find the reduced row echelon form R of A and the pivot columns of A.
[R, pivotcols] = rref(A);

format short;

disp('The pivot columns of the reduced row echelon form of A are');
disp(pivotcols);

% From the result, the leading 1's in R occur in columns 1 and 2.
% (i.e., the pivot columns of A are 1 and 2.)
% Hence, the basis vectors for W are v1 and v2.

disp('The reduced row echelon form R of A is'); disp(R);

% Furthermore, from the reduced row echelon form R of A,
% we can see that v3 = 2*v1 + v2, and v4 = -v1 + 2*v2.
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The pivot columns of the reduced row echelon form of A are
     1     2

The reduced row echelon form R of A is
     1     0     2    -1
     0     1     1     2
     0     0     0     0
     0     0     0     0
     0     0     0     0
     0     0     0     0
     0     0     0     0
     0     0     0     0
     0     0     0     0
     0     0     0     0
\end{verbatim}
\end{outputs}

\end{sol}

\vspace{3mm}

\begin{exer}(\textit{Finding Bases for the Fundamental Spaces})\\
Consider the matrix

\begin{displaymath}
A = \left[\begin{array}{rrrr} 1& \hspace{5mm} 3& \hspace{5mm} 2&\hspace{5mm} 1\\ -2 & -6 & 0 & -6 \\ 3 & 9 & 1 & 8 \\ -1 & -3 & -3 & -6 \\ 1 & 3 & 2 & 1 \\ 4 & 12 & 1 & 11 \end{array} \right].
\end{displaymath}

\begin{enumerate}

\item[(a)] Use Algorithm 1 in Section 7.6 to find a subset of the column vectors of $A$ that forms a basis for the column space of $A$, and express each column vector of $A$ that is not in that basis as a linear combination of the basis vectors.

\vspace{2mm}

\item[(b)] Use Algorithm 2 in Section 7.6 to find a basis for the null space of the matrix $A^{T}$.
\end{enumerate}

\end{exer}

\begin{sol}
\begin{verbatim}

A = [1 3 2 1; -2 -6 0 -6; 3 9 1 8; -1 -3 -3 -6; 1 3 2 1; 4 12 1 11];

% Find the reduced row echelon form R of A and the pivot columns of A.
[R, pivotcols] = rref(A);

format short;

disp('The pivot columns of the reduced row echelon form of A are');
disp(pivotcols);
% From the result, the leading 1's in R occur in columns 1, 3, and 4.
% (i.e., the pivot columns of A are 1, 3, and 4.)
% Hence, the columns 1, 3, and 4 of A are a basis for the column space of A.

disp('The reduced row echelon form R of A is'); disp(R);
% Furthermore, from the reduced row echelon form R of A,
% we can see that v2 = 3*v1, where v1 = A(:, 1), and v2 = A(:, 2).
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The pivot columns of the reduced row echelon form of A are
     1     3     4

The reduced row echelon form R of A is
     1     3     0     0
     0     0     1     0
     0     0     0     1
     0     0     0     0
     0     0     0     0
     0     0     0     0
\end{verbatim}
\end{outputs}

\end{sol}

% 7.7
\section{The Projection Theorem and Its Implications}

\begin{exer}(\textit{Standard Matrix for an Orthogonal Projection})\\
One way to find the standard matrix for the orthogonal projection onto a subspace $W$ spanned by a set of vectors $\{\mathbf{v}_{1}, \mathbf{v}_{2}, ..., \mathbf{v}_{k}\}$ is first to find a basis for $W$, then create a matrix $A$ that has the basis vectors as columns, and then use the Formula (27) in the Section 7.7.

\vspace{2mm}

\begin{enumerate}

\item[(a)] Find the standard matrix for the orthogonal projection of $\mathbb{R}^{4}$ onto the subspace $W$ spanned by

\begin{displaymath}
\begin{array}{ll} \vspace{1mm} \mathbf{v}_{1} = (1, \hspace{1mm} 2, \hspace{1mm} 3, \hspace{1mm} -4), & \hspace{2mm} \mathbf{v}_{2} = (2, \hspace{1mm}3, \hspace{1mm} -4, \hspace{1mm} 1), \\ \vspace{1mm} \mathbf{v}_{3} = (2, \hspace{1mm} -5, \hspace{1mm} 8, \hspace{1mm} -3), & \hspace{2mm} \mathbf{v}_{4} = (5, \hspace{1mm} 26, \hspace{1mm} -9, \hspace{1mm} -12), \\ \vspace{1mm} \mathbf{v}_{5} = (3, \hspace{1mm} -4, \hspace{1mm} 1, \hspace{1mm} 2). \end{array}
\end{displaymath}

\vspace{1mm}

\item[(b)] Use the matrix obtained in part (a) to find $\mathrm{proj}_{\tiny{W}} \mathbf{x}$, where $\mathbf{x} = (1, \hspace{1mm} 0, \hspace{1mm} -3, \hspace{1mm} 7)$.

\vspace{3mm}

\item[(c)] Find $\mathrm{proj}_{\tiny{W^{\perp}}} \mathbf{x}$ for the vector in part (b).

\end{enumerate}
\end{exer}

\begin{sol}
\begin{verbatim}

v1 = [1 2 3 -4]'; v2 = [2 3 -4 1]'; v3 = [2 -5 8 -3]';
v4 = [5 26 -9 -12]'; v5 = [3 -4 1 2]';

% Set A that has v1,v2,v3,v4 and v5, as column vectors.
A = [v1 v2 v3 v4 v5];

% Find the reduced row echelon form R of A and the pivot columns of A.
[R, pivotcols] = rref(A);

% M is the matrix whose columns are a basis for the column space of A.
M = A(:, pivotcols);

% By (27) in section 7.7, find the standard matrix.
P = M * inv(M'* M) * M';

format short;
disp('The standard matrix for the orthogonal projection of R^4 onto W=col(A) is');
disp(P);

x = [1 0 -3 7]';
xproj = P*x;
xperp = x - xproj;
disp('The projection of x onto W=col(A) is'); disp(xproj');

disp('The projection of x onto the orthogonal complement of W=col(A) is');
disp(xperp');

% As a check, the dot product of the two projections should be zero.
disp('The dot product of the two projections is'); disp(dot(xproj, xperp));
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The standard matrix for the orthogonal projection of R^4 onto W=col(A) is
    0.9992   -0.0144   -0.0161   -0.0195
   -0.0144    0.7551   -0.2737   -0.3314
   -0.0161   -0.2737    0.6941   -0.3703
   -0.0195   -0.3314   -0.3703    0.5517

The projection of x onto W=col(A) is
    0.9110   -1.5127   -4.6907    4.9534

The projection of x onto the orthogonal complement of W=col(A) is
    0.0890    1.5127    1.6907    2.0466

The dot product of the two projections is
 -5.3291e-015
\end{verbatim}
\end{outputs}

\end{sol}

% 7.8
\section{Best Approximation and Least Squares}

\begin{exer}
Make a function file \verb"LinearSolver.m" to find a least squares solution of $A\mathbf{x}=\mathbf{b}$ where $A$ has full column rank. Complete the missing part referring to the comments. Using this function file, solve the linear system
$$\begin{cases}
x-y=4\\
3x+2y=1\\
-2x+4y=3
\end{cases}
$$
and compare the output with the result of the MATLAB syntax $A\backslash \mathbf{b}$.
\end{exer}

\begin{sol}

\begin{verbatim}

%--- This is a function file 'LinearSolver.m' ---%
function [rank_A sol]=LinearSolver(A, b)
  [m,n]=size(A);
  rank_A=rank(A);

  % Check that A has full column rank.
  if rank_A<n
    fprintf('rank(A)=%d < %d  -> Not full column rank\n', rank_A, n);
    return; % If A does not have full column rank, then return.
  else
    fprintf('rank(A)=%d = %d -> Full column rank\n', rank_A, n);
  end

  % From the reduced row echelon form of [A'*A |A'*b],
  % find a solution to the normal equation A'Ax=A'b.

  Aug=[A'*A A'*b];
  rref_Aug=rref(Aug);
  sol=rref_Aug(:,n+1);

  fprintf('The least squares solution is');disp(sol');
end
\end{verbatim}

You execute the followings:
\begin{verbatim}
A=[1 -1; 3 2; -2 4];
b=[4; 1; 3];
LinearSolver(A, b);
A\b
\end{verbatim}

\begin{outputs}
\begin{verbatim}

rank(A)=2 = 2 -> Full column rank
The least squares solution is    0.1789    0.5018
ans =
    0.1789
    0.5018
\end{verbatim}

\end{outputs}

\end{sol}

\vspace{3mm}

\begin{exer}
The least squares method can be used to estimate the center $(h,k)$ of a circle $(x-h)^2+(y-k)^2=r^2$ using measured data points on its circumference. Suppose that the data points are
$$(x_1, y_1), (x_2, y_2), \cdots, (x_n, y_n)$$
and rewrite the equation of the circle in the form
\begin{equation}\label{eq1}\centering  2xh+2yk+s=x^2+y^2
\end{equation}
where
\begin{equation}\label{eq2}\centering s=r^2-h^2-k^2\end{equation}
Substituting the data points in \eqref{eq1} yields a linear system in the unknowns $h$, $k$, and $s$, which can be solved by least squares to estimate their values. Equation~\eqref{eq2} can then be used to estimate $r$. Use this method to approximate the center and radius of a circle from the measured data points on the circumference given in the accompanying table.

\begin{table}[h]
\centering
\caption{Data points of Problem 7(b)}
\label{my-label}
\begin{tabular}{|l|l|l|l|l|l|l|l|}
\hline
$\mathbf{x}$ & 19.880 & 20.919 & 21.735 & 23.375 & 24.361 & 25.375 & 25.979 \\ \hline
$\mathbf{y}$ & 68.874 & 67.676 & 66.692 & 64.385 & 62.908 & 61.292 & 60.277 \\ \hline
\end{tabular}
\end{table}
Graph the circle you obtained and plot the data points with red circles in the same figure.
\end{exer}

\begin{sol}
You execute the followings:
\begin{verbatim}
format short;

% given data
x=[19.880 20.919 21.735 23.375 24.361 25.375 25.979];
y=[68.874 67.676 66.692 64.385 62.908 61.292 60.277];

% number of data points.
[m,n]=size(x);

% construct the system matrix
A=[2*x' 2*y' ones(n,1)]; b=x.^2+y.^2;

% solve the normal equation
hks=inv(A'*A)*A'*b'
h=hks(1); k=hks(2); s=hks(3);

% compute the radius
r=sqrt(s+h^2+k^2);

figure;
theta=0:0.01:2*pi;
xx=h+r*cos(theta);
yy=k+r*sin(theta);

% plot the obtained circle
plot(xx,yy);
hold on;

% plot the data points
plot(x, y, 'o');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

hks =
  -18.3534
   35.4513
  986.5129
\end{verbatim}
\begin{figure}[h]
\centering
\includegraphics[width=9cm]{leastSq.jpg}
\end{figure}

\end{outputs}

\end{sol}

% 7.9
\section{Orthonormal Bases and the Gram Schmidt Process}

\begin{exer} (\textit{Gram-Schmidt Process})\\
Perform the Gram-Schmidt process to transform the vectors given in the Example 9 of the Section 7.9 to obtain an orthonormal basis for $\mathbb{R}^{3}$.

In this problem, use a nested loop and the MATLAB command \textit{norm}.
\end{exer}

\begin{sol}

\begin{verbatim}

w1 = [1 1 1]'; w2 = [0 1 1]'; w3 = [0 0 1]';
A = [w1 w2 w3]; % Construct a matrix A whose columns are w1, w2, and w3.
format short; [m, n] = size(A);
Q = zeros(m, n); % Initialize the matrix Q as an m*n zero matrix.

% Find an orthonormal basis for the column space of A.
for j = 1 : n
    v = A(:, j); % v begins as jth column of A.
    for i = 1 : (j-1)
        temp = Q(:, i)' * A(:, j);
        % Subtract each component of orthogonal projection of v
        % onto the subspace spanned by the vector Q(:, i).
        v = v - temp * Q(:, i);
    end
    Q(:, j) = v / norm(v); % Normalize v by its 2-norm.
end
disp('The orthonormal basis {q1,q2,q3} for R^3 from {w1,w2,w3} are as follows:')
disp('q1='); disp(Q(:,1)'); disp('q2='); disp(Q(:,2)'); disp('q3='); disp(Q(:,3)');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The orthonormal basis {q1,q2,q3} for R^3 from {w1,w2,w3} are as follows:
q1=
    0.5774    0.5774    0.5774

q2=
   -0.8165    0.4082    0.4082

q3=
   -0.0000   -0.7071    0.7071
\end{verbatim}
\end{outputs}

\end{sol}

\vspace{3mm}
\begin{exer}\label{GS} (\textit{Orthonormal Bases for the Four Fundamental Spaces})\\
Find orthonormal bases for the four fundamental spaces of the matrix

\begin{displaymath}
A = \left[\begin{array}{rrrr} 2& \hspace{2mm} -1& \hspace{5mm} 3&\hspace{4mm} 5\\ 4 & -3 & 1 & 3 \\ 3 & -2 & 3 & 4 \\ 4 & -1 & 15 & 17 \\ 7 & -6 & -7 & 0  \end{array} \right].
\end{displaymath}
\end{exer}

\begin{sol}

\begin{verbatim}

%--- The following is the function file 'GramSchmidt.m'. ---%

% Find an orthonormal basis for col(A) when A has full column rank.
function Q = GramSchmidt(A)

  [m, n] = size(A);

  % Initialize the matrix Q as an m*n zero matrix.
  Q = zeros(m, n);

  for j = 1 : n
    % v begins as jth column of A.
    v = A(:, j);
    for i = 1 : (j-1)
      temp = Q(:, i)' * A(:, j);
      % Subtract each component of orthogonal projection of v
      % onto the subspace spanned by the vector Q(:, i).
      v = v - temp * Q(:, i);
    end
    Q(:, j) = v / norm(v); % Normalize v by its 2-norm.
  end
end
% Q is an m*n matrix whose columns form an orthonormal basis for col(A).
\end{verbatim}

The following commands are performed in the command window of MATLAB.

\begin{verbatim}
A = [2 -1 3 5; 4 -3 1 3; 3 -2 3 4; 4 -1 15 17; 7 -6 -7 0];
format short;

% Find the reduced row echelon form of A.
rref_A = rref(A);

% (1). Find an orthonormal basis for the row space of A.
% From the result of rref_A, the first three nonzero rows in rref_A form
% a basis for the row space of A.

% Construct a matrix R_A whose columns are a basis for the row space of A.
R_A = rref_A(1:3, :)';

% Find an orthonormal basis for the column space of R_A by Gram-Schmidt process,
% which is the same as finding an orthonormal basis for the row space of A.
Orth_R_A = GramSchmidt(R_A);

a1 = Orth_R_A(:, 1); a2 = Orth_R_A(:, 2); a3 = Orth_R_A(:, 3);

disp('An orthonormal basis {a1, a2, a3} for the row space of A is');
disp('a1 = '); disp(a1'); disp('a2 = '); disp(a2'); disp('a3 = '); disp(a3');

% (2). Find an orthonormal basis for the column space of A.
% From the result of rref_A, the first three columns of A are the pivot columns
% which form a basis for the column space of A.

% Construct a matrix C_A whose columns are a basis for the column space of A.
C_A = A(:, 1:3);

% Find an orthonormal basis for the column space of C_A by Gram-Schmidt process,
% which is the same as finding an orthonormal basis for the column space of A.
Orth_C_A = GramSchmidt(C_A);

b1 = Orth_C_A(:, 1); b2 = Orth_C_A(:, 2); b3 = Orth_C_A(:, 3);

disp('An orthonormal basis {b1, b2, b3} for the column space of A is');
disp('b1 = '); disp(b1'); disp('b2 = '); disp(b2'); disp('b3 = '); disp(b3');

% (3). Find an orthonormal basis for the null space of A.
% In addition, from the result of rref_A,
% we can easily see that {[-6 -7 0 1]'} is a basis for N(A).

% Construct a matrix N_A whose columns are a basis for the null space of A.
N_A = [-6 -7 0 1]';

% Find an orthonormal basis for the column space of N_A by Gram-Schmidt process,
% which is the same as finding an orthonormal basis for the null space of A.
Orth_N_A = GramSchmidt(N_A);

c1 = Orth_N_A(:, 1);
disp('An orthonormal basis {c1} for the null space of A is');
disp('c1 = '); disp(c1');

% (4). Find an orthonormal basis for the null space of A transpose.
[L U P] = lu(A);
temp = [0 0 0 0 1]';
\% Make L a square matrix of order 5.
L = [L temp];

% Make U have the same size of A.
U(5, :) = 0;

% Then, we have P*A = L*U, which is the same result as above.
% Note that L^(-1)*P*A = U, where U is an upper triangular matrix.

E = L^(-1)*P;

% Since E = L^(-1)*P is a product of elementary matrices s.t. E*A=U,
% E represents a set of elementary row operations
% that makes A become a row echelon form U.

% ref_par_A is the resulting partitioned matrix [U E].
ref_par_A = [U E];

% From the result of ref_par_A, we can see that ref_par_A([4:5], [1:4]) = 0.
% Thus, the row vectors of E2 form a basis for null(A'),
% where E2 = ref_par_A([4:5], [5:9]).

% Construct a matrix N_Atrans whose columns are a basis for
% the null space of A transpose.
N_Atrans = ref_par_A(4:5, 5:9)';

% Find an orthonormal basis for the column space of N_Atrans by Gram-Schmidt process,
% which is the same as finding an orthonormal basis for the null space of A transpose.
Orth_N_Atrans = GramSchmidt(N_Atrans);

d1 = Orth_N_Atrans(:, 1); d2 = Orth_N_Atrans(:, 2);
disp('An orthonormal basis {d1, d2} for the null space of the transpose of A is');
disp('d1 = '); disp(d1'); disp('d2 = '); disp(d2');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

An orthonormal basis {a1, a2, a3} for the row space of A is
a1 =
    0.1644         0         0    0.9864

a2 =
   -0.7446    0.6559         0    0.1241

a3 =
     0     0     1     0

An orthonormal basis {b1, b2, b3} for the column space of A is
b1 =
    0.2063    0.4126    0.3094    0.4126    0.7220

b2 =
    0.1873   -0.0887    0.0493    0.8378   -0.5027

b3 =
   -0.3699    0.5812    0.5878   -0.1321   -0.4029

An orthonormal basis {c1} for the null space of A is
c1 =
   -0.6470   -0.7548         0    0.1078

An orthonormal basis {d1, d2} for the null space of the transpose of A is
d1 =
    0.8649    0.3089         0   -0.3089   -0.2471

d2 =
    0.1936   -0.6234    0.7458   -0.1224    0.0512
\end{verbatim}
\end{outputs}

\end{sol}

% section 7.10

\section{$QR-$Decomposition; Householder Transformations}

\begin{exer}(\textit{$QR-$Decomposition})\\
\begin{enumerate}
\item[(a)] Make a function file \verb"myQR.m" to find a $QR$-decomposition of a given matrix.
You may use your function file \verb"GS_process.m" from the \textbf{Exercise}~\ref{GS}.
\vspace{1mm}
\item[(b)] $$A=\begin{bmatrix} 1 & 1 & 1\\ 1& 0 & 2\\ 0 & 1& 2\end{bmatrix}.$$
Compare your result with the output produced by the MATLAB command \textit{qr}.
\end{enumerate}
\end{exer}

\begin{sol}
\begin{verbatim}

%(a)
%--- This is a function file myQR.m ---%

function [Q R]=myQR(A)
 Q=GS_process(A);
 R=Q'*A;
end

%(b)
A=[1 1 1; 1 0 2; 0 1 2];
[Q1 R1]=myQR(A); [Q R]=qr(A);

disp('my QR result'); disp('Q');disp(Q1); disp('R');disp(R1);
disp('MATLAB QR result'); disp('Q');disp(Q); disp('R');disp(R);
\end{verbatim}

\begin{outputs}
\begin{verbatim}

my QR result
Q
    0.7071    0.4082   -0.5774
    0.7071   -0.4082    0.5774
         0    0.8165    0.5774
R
    1.4142    0.7071    2.1213
    0.0000    1.2247    1.2247
    0.0000   -0.0000    1.7321

MATLAB QR result
Q
   -0.7071    0.4082   -0.5774
   -0.7071   -0.4082    0.5774
         0    0.8165    0.5774
R
   -1.4142   -0.7071   -2.1213
         0    1.2247    1.2247
         0         0    1.7321
\end{verbatim}

\end{outputs}

\noindent The results are the same.
\end{sol}

% 7.11

\section{Coordinates with Respect to a Basis}

\begin{exer} (\textit{Transition Matrices between Two Different Bases})\\

\begin{enumerate}

\item[(a)] Confirm that $B_{1} = \{\mathbf{u}_{1}, \mathbf{u}_{2}, \mathbf{u}_{3}, \mathbf{u}_{4}, \mathbf{u}_{5}\}$ and $B_{2} = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{3}, \mathbf{v}_{4}, \mathbf{v}_{5}\}$ are bases for $\mathbb{R}^{5}$, and find the transition matrices $P_{\tiny{B_{1} \rightarrow B_{2}}}$ and $P_{\tiny{B_{2} \rightarrow B_{1}}}$, where

\begin{displaymath}
\begin{array}{lllllll}
\vspace{1mm} \mathbf{u}_{1} & = & (3, \hspace{1mm} 1, \hspace{1mm} 3, \hspace{1mm} 2, \hspace{1mm} 6) & \hspace{4mm} & \mathbf{v}_{1} & = & (2, \hspace{1mm} 6, \hspace{1mm} 3, \hspace{1mm} 4, \hspace{1mm} 2) \\ \vspace{1mm} \mathbf{u}_{2} & = & (4, \hspace{1mm} 5, \hspace{1mm} 7, \hspace{1mm} 2, \hspace{1mm} 4) & \hspace{4mm} & \mathbf{v}_{2} & = & (3, \hspace{1mm} 1, \hspace{1mm} 5, \hspace{1mm} 8, \hspace{1mm} 3) \\ \vspace{1mm} \mathbf{u}_{3} & = & (3, \hspace{1mm} 2, \hspace{1mm} 1, \hspace{1mm} 5, \hspace{1mm} 4) & \hspace{4mm} & \mathbf{v}_{3} & = & (5, \hspace{1mm} 1, \hspace{1mm} 2, \hspace{1mm} 6, \hspace{1mm} 7) \\ \vspace{1mm} \mathbf{u}_{4} & = & (2, \hspace{1mm} 9, \hspace{1mm} 1, \hspace{1mm} 4, \hspace{1mm} 4) & \hspace{4mm} & \mathbf{v}_{4} & = & (8, \hspace{1mm} 4, \hspace{1mm} 3, \hspace{1mm} 2, \hspace{1mm} 6) \\ \vspace{1mm} \mathbf{u}_{5} & = & (3, \hspace{1mm} 3, \hspace{1mm} 6, \hspace{1mm} 6, \hspace{1mm} 7) & \hspace{4mm} & \mathbf{v}_{5} & = & (5, \hspace{1mm} 5, \hspace{1mm} 6, \hspace{1mm} 3, \hspace{1mm} 4) \\
\end{array}
\end{displaymath}

\item[(b)] Find the coordinate matrices with respect to $B_{1}$ and $B_{2}$ of $\mathbf{w} = (1, \hspace{1mm} 1, \hspace{1mm} 1, \hspace{1mm} 1, \hspace{1mm} 1)$.

\end{enumerate}

\end{exer}

\begin{sol}
\begin{verbatim}

u1 = [3 1 3 2 6]'; v1 = [2 6 3 4 2]';
u2 = [4 5 7 2 4]'; v2 = [3 1 5 8 3]';
u3 = [3 2 1 5 4]'; v3 = [5 1 2 6 7]';
u4 = [2 9 1 4 4]'; v4 = [8 4 3 2 6]';
u5 = [3 3 6 6 7]'; v5 = [5 5 6 3 4]';

U = [u1 u2 u3 u4 u5];
V = [v1 v2 v3 v4 v5];

format short;

% Initialization.
P_B1B2 = zeros(5);
P_B2B1 = zeros(5);

for j = 1:5
  % Find the coordinate vector of U(:, j) in B1 with respect to B2.
  P_B1B2(:, j) = V\U(:, j);
  % Find the coordinate vector of V(:, j) in B2 with respect to B1.
  P_B2B1(:, j) = U\V(:, j);
end

disp('The transition matrix from B1 to B2 is'); disp(P_B1B2);
disp('The transition matrix from B2 to B1 is'); disp(P_B2B1);

w = [1 1 1 1 1]';

% Find the coordinate matrix of w with respect to B1.
w_B1 = U\w;

% Find the coordinate matrix of w with respect to B2.
w_B2 = P_B1B2 * w_B1;

disp('The coordinate matrix of w with respect to B1 is'); disp(w_B1');
disp('The coordinate matrix of w with respect to B2 is'); disp(w_B2');
\end{verbatim}

\begin{outputs}
\begin{verbatim}

The transition matrix from B1 to B2 is
   -0.4992   -0.2531    0.4843    1.8286   -0.2123
   -0.7830   -0.3679    0.1604   -0.8019   -0.5849
    1.3019    0.2925    0.4623    0.6887    1.4906
   -0.9096   -0.6116    0.1918   -0.2091   -1.4104
    1.4230    1.8082   -0.4591   -0.2044    1.8019

The transition matrix from B2 to B1 is
   -0.6889   -1.3556    0.6222    1.2667   -0.0444
    0.4067    0.3591    0.0278    1.2083    1.0873
    0.3151    1.2675    1.3444    1.6833    0.6540
    0.3615   -0.5433   -0.2056   -0.1417   -0.0746
    0.2571    0.9714   -0.2000   -1.8000   -0.3429

The coordinate matrix of w with respect to B1 is
   -0.0222    0.1508    0.1841   -0.0016   -0.0286

The coordinate matrix of w with respect to B2 is
    0.0653    0.0094    0.0566    0.0039    0.1053
\end{verbatim}
\end{outputs}

\end{sol}

_codes/intro/Learning MATLAB with Linear Algebra (Jeon, Lee)/section7.tex

files/intro/Learning MATLAB with Linear Algebra (Jeon, Lee)/section7.tex


\section{Integer Linear Programming}

Integer Linear Program (ILP): $max\{c^Tx:x \in P  \cap Z^n\}$ with $P = \{x \in R^n : Ax \leq b\}$ \\
\textit{All variables are integer}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/integerlp.png}
\caption{ILP}
\end{figure}

Mixed Integer LP (MIP): $max\{c^T x:x\in P \cap Z_K^n\}$ with $P = \{x \in R^n : Ax \leq b\}$ where $K  \subseteq \{1, ..., n\}$ and $Z_K^n = \{x \in R^n : x_j \in Z$ for $j \in K\}$ \\
\textit{Some variables are integer, others are not}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/mlp.png}
\caption{MIP}
\end{figure}

It is not possible to just run a linear programming algorithm and then round the result up or down.
\begin{itemize}
    \item Problem 1: Rounding the LP Solution may yield a non-feasible solution.
    \item Problem 2: The optimal LP Solution is maybe far away from the optimal ILP solution.
\end{itemize}

\clearpage

\subsection{Relaxations}

\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{figures/relaxations.png}
\caption{Relaxations}
\end{figure}

$\Rightarrow f'(x)$ need to be higher or equal to $f(x)$ within the solution space.

\subsection{Branch and Bound Method}
Branch-and-Bound (B\&B) is a general solution method (independent from ILP).

Basic Procedure
\begin{enumerate}
    \item Split solution space iteratively into smaller subspaces ("Branch"). In each subproblem:
    \item Calculate an upper bound ("Bound") e.g. through relaxation
    \item Calculate a feasible solution $\Rightarrow$ Lower Bound ("Bound").
    \item Then use this information to "cut away" ("prune") certain subproblems.
\end{enumerate}

\clearpage
\subsubsection{Himmalaya Example}

\begin{itemize}
    \item We seperate the full solution space in several subspaces
    \item We send some helicopters to find the upper bound of these subspaces
    \item From the bottom, there are searching some sherpas for the lower bound
    \item At the end, we compare the values we got from each subspace.
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/himmalayaExample.png}
\caption{Branch and Bound - Himmalaya Example}
\end{figure}

\begin{itemize}
    \item Stopping by Dominance means, that the upper bound is already lower than an already found maximum in the queue.
    \item Stopping by Optimum means, that we found the optimum in this area (upper and lower bound are the same).
\end{itemize}

\clearpage
\subsubsection{Knap-Sack Problem Example}

\begin{itemize}
    \item Items $j \in J$ have a volume $a_j$ and a benefit $c_j$.
    \item The knapsack has a capacity of $b$
    \item $max\{c^Tx:a^Tx\leq b, x \in \{0,1\}^j\}$
\end{itemize}
LP Solution easy:
\begin{enumerate}
    \item Sort the items by decreasing benefit per volume $\frac{c_j}{a_j}$
    \item Choose items in this order, until the knapsack is full. The last item fractionally.
    \item After doing this, create two new branches. One branch with $x = 0$ for the fractional item and one branch with $x = 1$ for the fractional item. Continue until you discovered all options.
\end{enumerate}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{figures/bbknapsack.png}
\caption{Branch and Bound - Knapsack Example}
\end{figure}

\clearpage

\subsection{Cutting Plane Method}

\begin{itemize}
    \item Let $P$ and $P'$ be two formulations for the ILP: $max\{c^Tx:x\in P \subseteq Z^n\}$. $P'$ is called a better solution is $P' \subseteq P$.
    \item And if $P' \subseteq P$ then $max\{c^Tx:x \in P'\} \leq max\{c^Tx:x \in P\}$
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/optimalilpsolution.png}
\caption{Cutting Plane Method - Optimal ILP Solution}
\end{figure}

\textbf{The best formulation for $P'$} would be the convex hull, which is the thightest polyhedrun around $P$, but this is nearly impossible to find.

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/convexhull.png}
\caption{Cutting Plane Method - Behavior Convex Hull}
\end{figure}

\textbf{The integer hull of a polyhedrun is the convex hull, but only with integers.}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{figures/cuttingplanemethod.png}
\caption{Cutting Plane Method}
\end{figure}

\subsubsection{Gomory-Chvatal-Cut}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{figures/gomory.png}
\caption{Gomory-Chvatal-Cut}
\end{figure}

\clearpage


\subsection{Indifference curves}

\subsubsection{Marginal rate of substitution}

We can consider how much of one good a consumer is willing to give up to get one of another.

Consider the utility function:

\(U=f(x,y)\)

We can examine the change in utility following changes in inputs by taking the total differential.

\(dU=\dfrac{\delta f}{\delta x}dx+\dfrac{\delta f}{\delta y}dy\)

We want to examine changes where \(dU\) is \(0\), so:

\( \dfrac{\delta f}{\delta x}dx+\dfrac{\delta f}{\delta y}dy=0\)

\( \dfrac{\delta f}{\delta x}dx+\dfrac{\delta f}{\delta y}dy=0\)

\( MU_xdx+MU_ydy=0\)

\( \dfrac{dy}{dx}=-\dfrac{MU_x}{MU_y}\)

This is the form of the indifference curve.  We refer to \(\dfrac{MU_x}{MU_y}\) as the marginal rate of substitution.

ADD GRAPH TO SHOW

\subsubsection{Multiple choices}

If the set of choices is more complex, say there are now apples and bananas, we have to be more careful with a representative function.

The agent still prefers more apples and more bananas, but the following imply different choices:

\(f=x^2.y^2\)

\(f=ln(x)+ln(y)\)

In the first example, the agent would always swap an apple for a banana, if they had more apples, whereas the opposite is true in the second case. Functional form is important with multiple goods.

\subsubsection{Solving}


\subsection{Linear second-order Ordinary Differential Equations}

These are of the form

\(\dfrac{d^2y}{dt^2}+p(t)\dfrac{dy}{dt}+q(t)y=g(t)\)

There are two types. Homogenous equations are where \(g(t)=0\). Otherwise they are heterogenous.

We explore the case with constants:

\(a\dfrac{d^2y}{dt^2}+b\dfrac{dy}{dt}+cy=0\)


\chapter{Elemental matrices}

\section{Deformation modes of four-nodes finite elements}

\subsection{Plane stress}
For plane-stress conditions the stiffness matrix of a four-nodes element reads
\begin{equation}
  K = \frac{E}{24(1 - \nu^2)} \hat{K}
\end{equation}
with
\begin{equation}
  \hat{K} =
  \begin{bmatrix}
    12 - 4\nu & 3 \nu + 3 & - 2 \nu - 6 & 9 \nu - 3 & 2 \nu - 6 & - 3 \nu - 3 & 4 \nu & 3 - 9\nu\\
    3 \nu + 3 & 12 - 4\nu & 3 - 9\nu & 4 \nu & - 3 \nu - 3 & 2 \nu - 6 & 9 \nu - 3 & - 2 \nu - 6\\
    - 2 \nu - 6 & 3 - 9\nu & 12 - 4\nu & - 3 \nu - 3 & 4 \nu & 9 \nu - 3 & 2 \nu - 6 & 3 \nu + 3\\
    9 \nu - 3 & 4 \nu & - 3 \nu - 3 & 12 - 4\nu & 3 - 9\nu & - 2 \nu - 6 & 3 \nu + 3 & 2 \nu - 6\\
    2 \nu - 6 & - 3 \nu - 3 & 4 \nu & 3 - 9\nu & 12 - 4\nu & 3 \nu + 3 & - 2 \nu - 6 & 9 \nu - 3\\
    - 3 \nu - 3 & 2 \nu - 6 & 9 \nu - 3 & - 2 \nu - 6 & 3 \nu + 3 & 12 - 4\nu & 3 - 9\nu & 4 \nu\\
    4 \nu & 9 \nu - 3 & 2 \nu - 6 & 3 \nu + 3 & - 2 \nu - 6 & 3 - 9\nu & 12 - 4\nu & - 3 \nu - 3\\
    3 - 9\nu & - 2 \nu - 6 & 3 \nu + 3 & 2 \nu - 6 & 9 \nu - 3 & 4 \nu & - 3 \nu - 3 & 12 - 4\nu
  \end{bmatrix} .
  \label{eq:stiff}
\end{equation}

The characteristic polynomial is given by
\[\det(K - \lambda I)= 0\enspace ,\]
or
\begin{equation}
  \frac{\lambda^3(E - \lambda\nu - \lambda)^2 (E + \lambda \nu - \lambda) (\nu E - 3E - 6\nu^2\lambda + 6\lambda)^2}{36 (\nu - 1)^3 (1 + \nu)^4} = 0 \enspace .
\end{equation}

What gives as eigenvalues
\[\lambda \in \left\{0,
	\frac{E(3 - \nu)}{6(1 - \nu^2)},
	\frac{E}{1 + \nu},
	\frac{E}{1 - \nu} \right\} \enspace .
\]

And the corresponding eigenvectors are
\begin{equation}
\begin{array}{ll}
\lambda_1 = 0,
	&u^{(1)} = (1, 0, 1, 0, 1, 0, 1, 0)\\
	&u^{(2)} = (0, 1, 0, 1, 0, 1, 0, 1)\\
	&u^{(3)} = (0, 1, 0, -1, 2, -1, 2, 1)\\
\lambda_2 = \frac{E(3 - \nu)}{6(1 - \nu^2)},
	&u^{(4)} = (1, 0, -1, 0, 1, 0, -1, 0)\\
	&u^{(5)} = (0, 1, 0, -1, 0, 1, 0, -1)\\
\lambda_3 = \frac{E}{1 + \nu},
	&u^{(6)} = (1, 0, 0, -1, -1, 0, 0, 1)\\
	&u^{(7)} = (0, 1, 1, 0, 0, -1, -1, 0)\\
\lambda_4 = \frac{E}{1 - \nu},
	&u^{(8)} = (1, 1, -1, 1, -1, -1, 1, -1)
\end{array}
\end{equation}

\subsection{Plane strain}
In the case of plane strain the stiffness matrix of a four-nodes element reads
\begin{equation}
  K = \frac{E}{24(1 - 2\nu)(1 + \nu)} \hat{K}
\end{equation}
with
\begin{equation}
  \hat{K} =
  \begin{bmatrix}
    12 - 16\nu & 3 & 4 \nu - 6 & 12 \nu - 3 & 8 \nu - 6 & -3 & 4 \nu & 3 - 12\nu\\
    3 & 12 - 16\nu & 3 - 12\nu & 4 \nu & -3 & 8 \nu - 6 & 12 \nu - 3 & 4 \nu - 6\\4 \nu - 6 & 3 - 12\nu & 12 - 16\nu & -3 & 4 \nu & 12 \nu - 3 & 8 \nu - 6 & 3\\
    12 \nu - 3 & 4 \nu & -3 & 12 - 16\nu & 3 - 12\nu & 4 \nu - 6 & 3 & 8 \nu - 6\\
    8 \nu - 6 & -3 & 4 \nu & 3 - 12\nu & 12 - 16\nu & 3 & 4 \nu - 6 & 12 \nu - 3\\
    -3 & 8 \nu - 6 & 12 \nu - 3 & 4 \nu - 6 & 3 & 12 - 16\nu & 3 - 12\nu & 4 \nu\\4 \nu & 12 \nu - 3 & 8 \nu - 6 & 3 & 4 \nu - 6 & 3 - 12\nu & 12 - 16\nu & -3\\
    3 - 12\nu & 4 \nu - 6 & 3 & 8 \nu - 6 & 12 \nu - 3 & 4 \nu & -3 & 12 - 16\nu
  \end{bmatrix}
\end{equation}

The characteristic polynomial is given by
\begin{equation}
  \frac{\lambda^3 (E - \nu\lambda - \lambda)^2(E + 2\nu^2\lambda + \nu\lambda - \lambda)(4\nu E - 3E - 12\nu^2\lambda - 6\nu\lambda + 6\lambda)^2}{36(\nu + 1)^5\ (2\nu - 1)^3}=0 \enspace .
\end{equation}

What gives as eigenvalues
\[\lambda \in \left\{0,
	\frac{E(3 - 4\nu)}{6(1 - 2\nu)(1 + \nu)},
	\frac{E}{1 + \nu},
	\frac{E}{(1 + \nu)(1 - 2\nu)} \right\} \enspace .
\]

And the corresponding eigenvectors are
\begin{equation}
\begin{array}{ll}
\lambda_1 = 0,
	&u^{(1)} = (1, 0, 1, 0, 1, 0, 1, 0)\\
	&u^{(2)} = (0, 1, 0, 1, 0, 1, 0, 1)\\
	&u^{(3)} = (0, 1, 0, -1, 2, -1, 2, 1)\\
\lambda_2 = \frac{E(3 - 4\nu)}{6(1 - 2\nu)(1 + \nu)},
	&u^{(4)} = (1, 0, -1, 0, 1, 0, -1, 0)\\
	&u^{(5)} = (0, 1, 0, -1, 0, 1, 0, -1)\\
\lambda_3 = \frac{E}{1 + \nu},
	&u^{(6)} = (1, 0, 0, -1, -1, 0, 0, 1)\\
	&u^{(7)} = (0, 1, 1, 0, 0, -1, -1, 0)\\
\lambda_4 = \frac{E}{(1 + \nu)(1 - 2\nu)},
	&u^{(8)} = (1, 1, -1, 1, -1, -1, 1, -1)
\end{array}
\end{equation}

As expected, the eigenvectors are the same while the eigenvalues differ.