\section{The Gram-Schmidt orthogonalization procedure}
\label{sec:gram-schmidt}

\begin{outcome}
  \begin{enumerate}
  \item Use the Gram-Schmidt procedure to find an orthogonal basis
    of a subspace of an inner product space.
  \item Find an orthonormal basis of a subspace.
  \end{enumerate}
\end{outcome}

Although we have already seen some potential uses for orthogonal
bases, we have not yet seen very many examples of such bases. In this
section, we will look at the Gram-Schmidt orthogonalization procedure,
a method for turning any basis into an orthogonal one.

The basic idea is very simple: if two vectors $\vect{v}_1,\vect{v}_2$
are not orthogonal, then we can make them orthogonal by replacing
$\vect{v}_2$ by a vector of the form $\vect{u}_2 = \vect{v}_2 -
t\vect{v}_1$, for a suitable parameter $t$.
\begin{equation*}
  \begin{tikzpicture}[scale=1.25]
    \draw[thick,red] (1.5,0) -- (1.5,2) -- (0,2);
    \draw[thick,blue,->] (0,0) -- node[left]{$\vect{v}_1$} (0,2.7);
    \draw[thick,blue,->] (0,0) -- node[left, pos=0.6]{$\vect{v}_2$} (1.5,2);
    \draw[thick,green!50!black,->] (0,0) -- node[below right, pos=0.3]{$\vect{u}_2 = \vect{v}_2 - t\vect{v}_1$} (1.5,0);
  \end{tikzpicture}
\end{equation*}
But what is the correct value of $t$? It turns out that this value is
uniquely determined by the requirement that $\vect{v}_1$ and
$\vect{u}_2$ must be orthogonal. We calculate
\begin{equation*}
  \iprod{\vect{v}_1,\vect{u}_2}
  ~=~ \iprod{\vect{v}_1,\vect{v}_2-t\vect{v}_1}
  ~=~ \iprod{\vect{v}_1,\vect{v}_2}-t\iprod{\vect{v}_1,\vect{v}_1}.
\end{equation*}
Setting this equal to $0$ yields the unique solution
\begin{equation*}
  t = \frac{\iprod{\vect{v}_1,\vect{v}_2}}{\iprod{\vect{v}_1,\vect{v}_1}}
\end{equation*}
Note that this is exactly the same thing as the Fourier coefficient of
$\vect{v}_2$ in the direction of $\vect{v}_1$. The following
proposition summarizes what we have found so far. For consistency with
our later notation, we also rename the first basis vector $\vect{v}_1$
to $\vect{u}_1$.

\begin{proposition}{Gram-Schmidt orthogonalization procedure for 2 vectors}{gram-schmidt-2}
  Let $\set{\vect{v}_1,\vect{v}_2}$ be a basis for some subspace $W$
  of an inner product space $V$. Define vectors
  $\vect{u}_1,\vect{u}_2$ as follows:
  \begin{eqnarray*}
    \vect{u}_1 &=& \vect{v}_1, \\
    \vect{u}_2 &=& \vect{v}_2 ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1.
  \end{eqnarray*}
  Then $\set{\vect{u}_1,\vect{u}_2}$ is an orthogonal basis of $W$.
\end{proposition}

\begin{example}{Gram-Schmidt orthogonalization procedure for 2 vectors}{gram-schmidt-2}
  In $\R^3$ with the usual dot product, find an orthogonal basis for
  \begin{equation*}
    \sspan\set{
      \begin{mymatrix}{c} 1 \\ 1 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 0 \\ 1 \\ 1 \end{mymatrix}
    }.
  \end{equation*}
\end{example}

\begin{solution}
  Let $\vect{v}_1 = \begin{mymatrix}{c} 1 \\ 1 \\ 0 \end{mymatrix}$
  and $\vect{v}_2 = \begin{mymatrix}{c} 0 \\ 1 \\ 1 \end{mymatrix}$.
  We calculate
  \begin{eqnarray*}
    \vect{u}_1
    &=& \vect{v}_1
        ~=~ \begin{mymatrix}{c} 1 \\ 1 \\ 0 \end{mymatrix}, \\
    \vect{u}_2
    &=& \vect{v}_2 ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~=~ \begin{mymatrix}{c} 0 \\ 1 \\ 1 \end{mymatrix}
    ~-~ \frac{1}{2}\begin{mymatrix}{c} 1 \\ 1 \\ 0 \end{mymatrix}
    ~=~ \begin{mymatrix}{c} -1/2 \\ 1/2 \\ 1 \end{mymatrix}.
  \end{eqnarray*}
  Therefore the desired orthogonal basis is
  $\set{\begin{mymatrix}{c} 1 \\ 1 \\ 0 \end{mymatrix},
    \begin{mymatrix}{c} -1/2 \\ 1/2 \\ 1 \end{mymatrix}}$.
\end{solution}

The procedure for finding an orthogonal basis of a $k$-dimensional
space is very similar. We adjust each basis vector $\vect{v}_i$ by
subtracting a suitable linear combination of previous orthogonal basis
vectors.

\begin{proposition}{Gram-Schmidt orthogonalization procedure for $k$ vectors}{gram-schmidt-k}
  Let $\set{\vect{v}_1,\ldots,\vect{v}_k}$ be a basis for some subspace $W$
  of an inner product space $V$.%
  \index{Gram-Schmidt procedure}%
  \index{orthogonalization}%
  \index{orthogonal basis!Gram-Schmidt procedure}
  Define vectors
  $\vect{u}_1,\ldots,\vect{u}_k$ as follows:
  \begin{eqnarray*}
    \vect{u}_1
    &=& \vect{v}_1,
    \\
    \vect{u}_2
    &=& \vect{v}_2
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1,
    \\
    \vect{u}_3
    &=& \vect{v}_3
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_3}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_3}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2,
    \\
    &\vdots&
    \\
    \vect{u}_k
    &=& \vect{v}_k
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_k}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_k}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_{k-1},\vect{v}_k}}{\iprod{\vect{u}_{k-1},\vect{u}_{k-1}}}\vect{u}_{k-1}.
  \end{eqnarray*}
  Then $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is an orthogonal basis of $W$.
\end{proposition}

\begin{proof}
  First, it is clear that $\set{\vect{v}_1,\ldots,\vect{v}_k}$ and
  $\set{\vect{u}_1,\ldots,\vect{u}_k}$ span the same subspace, as each
  $\vect{v}_i$ is a linear combination of
  $\vect{u}_1,\ldots,\vect{u}_i$ and conversely, each $\vect{u}_i$ is
  a linear combination of $\vect{v}_1,\ldots,\vect{v}_i$. So the only
  thing we must check is that $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is
  an orthogonal set. In other words, we must show that
  $\iprod{\vect{u}_j,\vect{u}_i}=0$ for all $j<i$. We prove this by
  induction on $i$, i.e., we assume it is already true for all pairs
  of indices smaller than $i$. To show
  $\iprod{\vect{u}_j,\vect{u}_i}=0$, we calculate:
  \begin{eqnarray*}
    \iprod{\vect{u}_j,\vect{u}_i}
    &=& \textstyle
        \iprod{\vect{u}_j, \vect{v}_i
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_i}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_j,\vect{v}_i}}{\iprod{\vect{u}_j,\vect{u}_j}}\vect{u}_j
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_{i-1},\vect{v}_i}}{\iprod{\vect{u}_{i-1},\vect{u}_{i-1}}}\vect{u}_{i-1}}
    \\
    &=& \textstyle
        \iprod{\vect{u}_j, \vect{v}_i}
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_i}}{\iprod{\vect{u}_1,\vect{u}_1}}\iprod{\vect{u}_j,\vect{u}_1}
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_j,\vect{v}_i}}{\iprod{\vect{u}_j,\vect{u}_j}}\iprod{\vect{u}_j,\vect{u}_j}
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_{i-1},\vect{v}_i}}{\iprod{\vect{u}_{i-1},\vect{u}_{i-1}}}\iprod{\vect{u}_j,\vect{u}_{i-1}}
    \\
    &=& \textstyle
        \iprod{\vect{u}_j, \vect{v}_i}
        ~-~ 0
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_j,\vect{v}_i}}{\iprod{\vect{u}_j,\vect{u}_j}}\iprod{\vect{u}_j,\vect{u}_j}
        ~-~ \ldots
        ~-~ 0
    \\
    &=& \iprod{\vect{u}_j, \vect{v}_i}
        ~-~ \iprod{\vect{u}_j,\vect{v}_i}
    \\
    &=& 0.
  \end{eqnarray*}
  It follows that the set $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is
  orthogonal, as desired.
\end{proof}

\begin{example}{Gram-Schmidt orthogonalization procedure}{gram-schmidt-r4}
  In $\R^4$, find an orthogonal basis for
  \begin{equation*}
    \sspan\set{
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},~
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
    }.
  \end{equation*}
\end{example}

\begin{solution}
  Let
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},\quad
    \vect{v}_2 = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix},\quad\mbox{and}\quad
    \vect{v}_3 = \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}.
  \end{equation*}
  We calculate
  \begin{eqnarray*}
    \vect{u}_1
    &=& \vect{v}_1
        ~=~ \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},
    \\
    \vect{u}_2
    &=& \vect{v}_2 ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
    ~=~ \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix}
    ~-~ \frac{3}{4}\begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    ~=~ \begin{mymatrix}{c} 1/4 \\ 1/4 \\ 1/4 \\ -3/4 \end{mymatrix},
    \\
    \vect{u}_3
    &=& \vect{v}_3
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_3}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_3}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2
    \\
    &=& \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
    ~-~ \frac{2}{4}\begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    ~-~ \frac{1/2}{3/4}\begin{mymatrix}{c} 1/4 \\ 1/4 \\ 1/4 \\ -3/4 \end{mymatrix}
    ~=~ \begin{mymatrix}{c} 1/3 \\ 1/3 \\ -2/3 \\ 0 \end{mymatrix}.
  \end{eqnarray*}
  Therefore the orthogonal basis is
  $\set{\vect{u}_1,\vect{u}_2,\vect{u}_3} = \set{
    \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},~
    \begin{mymatrix}{c} 1/4 \\ 1/4 \\ 1/4 \\ -3/4 \end{mymatrix},~
    \begin{mymatrix}{c} 1/3 \\ 1/3 \\ -2/3 \\ 0 \end{mymatrix}
  }$.
\end{solution}

The Gram-Schmidt procedure is sensitive to reordering the vectors. For
example, if we order the original basis vectors in
Example~\ref{exa:gram-schmidt-r4} in the opposite order, we end up
with a different orthogonal basis at the end. Sometimes this can
simplify the calculations, as the following example shows.

\begin{example}{Gram-Schmidt orthogonalization procedure: reordering the vectors}{gram-schmidt-r4-b}
  In $\R^4$, find an orthogonal basis for
  \begin{equation*}
    \sspan\set{
      \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    }.
  \end{equation*}
\end{example}

\begin{solution}
  Note that these are the same basis vectors as in
  Example~\ref{exa:gram-schmidt-r4}, but listed in a different order.
  Let
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix},\quad
    \vect{v}_2 = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix},\quad\mbox{and}\quad
    \vect{v}_3 = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  We calculate
  \begin{eqnarray*}
    \vect{u}_1
    &=& \vect{v}_1
        ~=~ \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix},
    \\
    \vect{u}_2
    &=& \vect{v}_2 ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
    ~=~ \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix}
    ~-~ \frac{2}{2}\begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
    ~=~ \begin{mymatrix}{c} 0 \\ 0 \\ 1 \\ 0 \end{mymatrix},
    \\
    \vect{u}_3
    &=& \vect{v}_3
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_3}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_3}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2
    \\
    &=& \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    ~-~ \frac{2}{2}\begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
    ~-~ \frac{1}{1}\begin{mymatrix}{c} 0 \\ 0 \\ 1 \\ 0 \end{mymatrix}
    ~=~ \begin{mymatrix}{c} 0 \\ 0 \\ 0 \\ 1 \end{mymatrix}.
  \end{eqnarray*}
  This time, we end up with the orthogonal basis
  $\set{\vect{u}_1,\vect{u}_2,\vect{u}_3} = \set{
    \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix},~
    \begin{mymatrix}{c} 0 \\ 0 \\ 1 \\ 0 \end{mymatrix},~
    \begin{mymatrix}{c} 0 \\ 0 \\ 0 \\ 1 \end{mymatrix}
  }$.
\end{solution}

In the next example, we will consider $\R^n$, but with a non-standard
inner product.

\begin{example}{Gram-Schmidt orthogonalization procedure, non-standard inner product}{gram-schmidt-non-standard}
  Let
  \begin{equation*}
    A = \begin{mymatrix}{ccc}
      1 & 2 & -2 \\
      2 & 6 & -1 \\
      -2 & -1 & 9 \\
    \end{mymatrix},
  \end{equation*}
  and consider the vector space $\R^3$ with the inner product given by
  $\iprod{\vect{v},\vect{w}} = \vect{v}^T A\vect{w}$.
  Let
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix},
    \quad
    \vect{v}_2 = \begin{mymatrix}{c} 0 \\ 1 \\ 0 \end{mymatrix},
    \quad\mbox{and}\quad
    \vect{v}_3 = \begin{mymatrix}{c} 0 \\ 0 \\ 1 \end{mymatrix}.
  \end{equation*}
  Apply the Gram-Schmidt procedure to
  $\vect{v}_1,\vect{v}_2,\vect{v}_3$ to find an orthogonal basis
  $\set{\vect{u}_1,\vect{u}_2,\vect{u}_3}$ for $\R^3$ with respect to
  the above inner product.
\end{example}

\begin{solution}
  As usual, we start with
  \begin{eqnarray*}
    \vect{u}_1 &=& \vect{v}_1 = \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix}.
  \end{eqnarray*}
  Next, we calculate
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}_2} &=& \vect{u}_1^TA\vect{v}_2 = 2, \\
    \iprod{\vect{u}_1,\vect{u}_1} &=& \vect{u}_1^TA\vect{u}_1 = 1.
  \end{eqnarray*}
  Therefore,
  \begin{eqnarray*}
    \vect{u}_2
    &=& \vect{v}_2
        - \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}} \vect{u}_1
        = \begin{mymatrix}{c} 0 \\ 1 \\ 0 \end{mymatrix}
    - \frac{2}{1} \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} -2 \\ 1 \\ 0 \end{mymatrix}.
  \end{eqnarray*}
  Finally, we calculate
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}_3} &=& \vect{u}_1^TA\vect{v}_3 = -2, \\
    \iprod{\vect{u}_2,\vect{v}_3} &=& \vect{u}_2^TA\vect{v}_3 = 3, \\
    \iprod{\vect{u}_2,\vect{u}_2} &=& \vect{u}_2^TA\vect{u}_2 = 2.
  \end{eqnarray*}
  Therefore,
  \begin{eqnarray*}
    \vect{u}_3
    &=& \vect{v}_3
        - \frac{\iprod{\vect{u}_1,\vect{v}_3}}{\iprod{\vect{u}_1,\vect{u}_1}} \vect{u}_1
        - \frac{\iprod{\vect{u}_2,\vect{v}_3}}{\iprod{\vect{u}_2,\vect{u}_2}} \vect{u}_2
        = \begin{mymatrix}{c} 0 \\ 0 \\ 1 \end{mymatrix}
    - \frac{-2}{1} \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix}
    - \frac{3}{2} \begin{mymatrix}{c} -2 \\ 1 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 5 \\ -3/2 \\ 1 \end{mymatrix}.
  \end{eqnarray*}
  So the desired orthogonal basis is
  \begin{equation*}
    \set{\vect{u}_1,\vect{u}_2,\vect{u}_3}
    = \set{
      \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix},
      \begin{mymatrix}{c} -2 \\ 1 \\ 0 \end{mymatrix},
      \begin{mymatrix}{c} 5 \\ -3/2 \\ 1 \end{mymatrix}
    }.
  \end{equation*}
  Note that it is not orthogonal with respect to the dot product, but
  with respect to the inner product defined above.
\end{solution}

\begin{example}{Legendre polynomials}{legendre-polynomials}
  Consider the vector space $\Poly$ of polynomials, with the inner product
  \begin{equation*}
    \iprod{p,q} = \int_{-1}^{1} p(x)q(x)\,dx.
  \end{equation*}
  Use the Gram-Schmidt procedure to find an orthogonal basis for
  $\sspan\set{1,x,x^2,x^3}$.%
  \index{Legendre polynomial}%
  \index{polynomial!Legendre polynomial}
\end{example}

\begin{solution}
  Let $\vect{v}_1=1$, $\vect{v}_2=x$, $\vect{v}_3=x^2$, and
  $\vect{v}_4=x^3$. We follow the Gram-Schmidt procedure:
  \begin{eqnarray*}
    \vect{u}_1
    &=& \vect{v}_1
        ~=~ 1.
  \end{eqnarray*}
  Before we calculate $\vect{u}_2$, we have to evaluate two integrals:
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}_2}
    &=& \int_{-1}^{1} 1\cdot x\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{2}x^2}_{-1}^{1}
    ~=~ 0,
    \\
    \iprod{\vect{u}_1,\vect{u}_1}
    &=& \int_{-1}^{1} 1\cdot 1\,dx
    ~=~ \bigbracket{x}_{-1}^{1}
    ~=~ 2.
  \end{eqnarray*}
  Then
  \begin{eqnarray*}
    \vect{u}_2
    &=& \vect{v}_2 ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
    ~=~ x ~-~ \frac{0}{2}\cdot 1 ~=~ x.
  \end{eqnarray*}
  To calculate $\vect{u}_2$, we first evaluate three integrals:
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}_3}
    &=& \int_{-1}^{1} 1\cdot x^2\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{3}x^3}_{-1}^{1}
    ~=~ \frac{2}{3},
    \\
    \iprod{\vect{u}_2,\vect{v}_3}
    &=& \int_{-1}^{1} x\cdot x^2\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{4}x^4}_{-1}^{1}
    ~=~ 0,
    \\
    \iprod{\vect{u}_2,\vect{u}_2}
    &=& \int_{-1}^{1} x\cdot x\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{3}x^3}_{-1}^{1}
    ~=~ \frac{2}{3}.
  \end{eqnarray*}
  Then
  \begin{eqnarray*}
    \vect{u}_3
    &=& \vect{v}_3
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_3}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_3}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2
        ~=~ x^2 ~-~ \frac{2/3}{2}\cdot 1 ~-~ 0\cdot x
        ~=~ x^2 - \frac{1}{3}.
  \end{eqnarray*}
  To calculate $\vect{u}_3$, we first evaluate four integrals:
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}_4}
    &=& \int_{-1}^{1} 1\cdot x^3\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{4}x^4}_{-1}^{1}
    ~=~ 0,
    \\
    \iprod{\vect{u}_2,\vect{v}_4}
    &=& \int_{-1}^{1} x\cdot x^3\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{5}x^5}_{-1}^{1}
    ~=~ \frac{2}{5},
    \\
    \iprod{\vect{u}_3,\vect{v}_4}
    &=& \int_{-1}^{1}\textstyle (x^2-\frac{1}{3})\cdot x^3\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{6}x^6 - \frac{1}{12}x^4}_{-1}^{1}
    ~=~ 0,
    \\
    \iprod{\vect{u}_3,\vect{u}_3}
    &=& \int_{-1}^{1}\textstyle (x^2-\frac{1}{3})^2\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{5}x^5-\frac{2}{9}x^3+\frac{1}{9}x}_{-1}^{1}
    ~=~ \frac{8}{45}.
  \end{eqnarray*}
  Then
  \begin{eqnarray*}
    \vect{u}_4
    &=& \vect{v}_4
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_4}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_4}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2
        ~-~ \frac{\iprod{\vect{u}_3,\vect{v}_4}}{\iprod{\vect{u}_3,\vect{u}_3}}\vect{u}_3
    \\
    &=& x^3 ~-~ \frac{0}{2}\cdot 1 ~-~ \frac{2/5}{2/3}\cdot x ~-~ \frac{0}{8/45}\cdot(x^2-\frac{1}{3})
        ~=~ x^3 - \frac{3}{5}x.
  \end{eqnarray*}
  Thus, we obtain the orthogonal basis $\set{\vect{u}_1,\vect{u}_2,\vect{u}_3,\vect{u}_4} =
  \set{1,~ x,~ x^2-\frac{1}{3},~ x^3 - \frac{3}{5}x}$.
\end{solution}

The orthogonal polynomials from Example~\ref{exa:legendre-polynomials}
are known (up to scalar multiples) as \textbf{Legendre polynomials}%
\index{Legendre polynomial}%
\index{polynomial!Legendre polynomial}.  We can continue in the same
fashion applying the Gram-Schmidt procedure to the polynomials
$1, x, x^2, x^3, x^4, x^5, x^6, \ldots$ to get an infinite sequence of
orthogonal polynomials. The first few elements of this sequence are:
\begin{eqnarray*}
  p_0(x) &=& 1, \\
  p_1(x) &=& x, \\
  p_2(x) &=& x^2 - \frac{1}{3}, \\
  p_3(x) &=& x^3 - \frac{3}{5}x, \\
  p_4(x) &=& x^4 - \frac{6}{7}x^2 + \frac{3}{35}, \\
  p_5(x) &=& x^5 - \frac{10}{9}x^3 + \frac{5}{21}x, \\
  p_6(x) &=& x^6 - \frac{15}{11}x^4 + \frac{5}{11}x^2 - \frac{5}{231}, \\
  p_7(x) &=& x^7 - \frac{21}{13}x^5 + \frac{105}{143}x^3 - \frac{35}{429}x, \\
  p_8(x) &=& x^8 - \frac{28}{15}x^6 + \frac{14}{13}x^4 - \frac{28}{143}x^2 + \frac{7}{1287}.
\end{eqnarray*}
\begin{center}
  \begin{tikzpicture}[domain=1:-1, scale=4]
    \draw[thick,color=fourier1,samples=50] plot (\x,{abs(\x^6)-15/11*abs(\x^4)+5/11*abs(\x^2)-5/231}) node[left=2ex,color=fourier1t] {$x^6 - \frac{15}{11}x^4 + \frac{5}{11}x^2 - \frac{5}{231}$};
    \draw[thick,color=fourier2,samples=50] plot (\x,{\x^5 - 10/9*\x^3+5/21*\x}) node[left=2ex,color=fourier2t] {$x^5 - \frac{10}{9}x^3 + \frac{5}{21}x$};
    \draw[thick,color=fourier3,samples=50] plot (\x,{abs(\x^4)-6/7*abs(\x^2)+3/35}) node[left=2ex,color=fourier3t] {$x^4 - \frac{6}{7}x^2 + \frac{3}{35}$};
    \draw[thick,color=fourier4,samples=50] plot (\x,\x^3-3/5*\x) node[left=2ex,color=fourier4t] {$x^3 - \frac{3}{5}x$};
    \draw[thick,color=fourier5,samples=50] plot (\x,{abs(\x^2)-1/3}) node[left=2ex,,color=fourier5t] {$x^2-\frac{1}{3}$};
    \draw[thick,color=fourier6] plot (\x,\x) node[left=2ex,color=fourier6t] {$x$};
    \draw[thick,color=fourier7] plot (\x,1) node[left=2ex,color=fourier7t] {$1$};
    \draw[->] (-1.2,0) -- (1.2,0) node[right] {$x$};
    \draw[->] (0,-1.2) -- (0,1.2) node[above] {$p(x)$};
    \draw (1,0) -- (1,-0.05) node[below] {$1$};
    \draw (-1,0) -- (-1,-0.05) node[below] {$-1$};
  \end{tikzpicture}
\end{center}

The Gram-Schmidt procedure yields an {\em orthogonal} basis. If we
want to compute an {\em orthonormal} basis, we also have to normalize
each basis vector. Since normalization usually involves dividing by a
square root, it is best to do this at the end, i.e., after the entire
Gram-Schmidt procedure is complete, rather than normalizing each
$\vect{u}_i$ immediately after it is found. Note that the Gram-Schmidt
procedure itself does not involve computing any square roots.

\begin{example}{Finding an orthonormal basis}{finding-orthonormal-basis-r4}
  In $\R^4$, find an orthogonal basis for
  \begin{equation*}
    \sspan\set{
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},~
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
    }.
  \end{equation*}
\end{example}

\begin{solution}
  In Example~\ref{exa:gram-schmidt-r4}, we already found an orthogonal basis
  \begin{equation*}
    \set{\vect{u}_1,\vect{u}_2,\vect{u}_3} ~=~
    \set{
      \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},~
      \begin{mymatrix}{c} 1/4 \\ 1/4 \\ 1/4 \\ -3/4 \end{mymatrix},~
      \begin{mymatrix}{c} 1/3 \\ 1/3 \\ -2/3 \\ 0 \end{mymatrix}
    }
  \end{equation*}
  for this space. So all that is left to do is to normalize each
  vector. The orthonormal basis is
  \begin{equation*}
    \set{
      \frac{\vect{u}_1}{\norm{\vect{u}_1}},
      \frac{\vect{u}_2}{\norm{\vect{u}_2}},
      \frac{\vect{u}_3}{\norm{\vect{u}_3}}
    }
    ~=~
    \set{
      \frac{1}{2}\begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 1 \end{mymatrix},~
      \frac{2}{\sqrt{3}}\begin{mymatrix}{c} 1/4 \\ 1/4 \\ 1/4 \\ -3/4 \end{mymatrix},~
      \frac{\sqrt{3}}{\sqrt{2}}\begin{mymatrix}{c} 1/3 \\ 1/3 \\ -2/3 \\ 0 \end{mymatrix}
    }.
  \end{equation*}
  Alternatively, we could have also normalized the orthogonal basis we
  found in Example~\ref{exa:gram-schmidt-r4-b}. In that case, we
  obtain the orthonormal basis
  \begin{equation*}
    \set{
      \frac{1}{\sqrt{2}}
      \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 0 \\ 0 \\ 1 \\ 0 \end{mymatrix},~
      \begin{mymatrix}{c} 0 \\ 0 \\ 0 \\ 1 \end{mymatrix}
    }.
  \end{equation*}
\end{solution}

\begin{example}{Orthonormal basis of polynomials}{legendre-polynomials2}
  Find an orthonormal basis for the space of
  Example~\ref{exa:legendre-polynomials}.
\end{example}

\begin{solution}
  In Example~\ref{exa:legendre-polynomials}, we found the orthogonal
  basis
  $\set{\vect{u}_1,\vect{u}_2,\vect{u}_3,\vect{u}_4} = \set{1,~ x,~
    x^2-\frac{1}{3},~ x^3 - \frac{3}{5}x}$.  We also computed
  \begin{equation*}
    \iprod{\vect{u}_1,\vect{u}_1} = 2,\quad
    \iprod{\vect{u}_2,\vect{u}_2} = \frac{2}{3},\quad\mbox{and}\quad
    \iprod{\vect{u}_3,\vect{u}_3} = \frac{8}{45}.
  \end{equation*}
  We also need to compute $\iprod{\vect{u}_4,\vect{u}_4}$:
  \begin{equation*}
    \iprod{\vect{u}_4,\vect{u}_4}
    ~=~ \int_{-1}^{1}{\textstyle (x^3 - \frac{3}{5}x)^2}\,dx
    ~=~ \int_{-1}^{1}{\textstyle x^6 - \frac{6}{5}x^4 + \frac{9}{25}x^2}\,dx
    ~=~ \bigbracket{\textstyle\frac{1}{7}x^7-\frac{6}{25}x^5+\frac{3}{25}x^3}_{-1}^{1}
    ~=~ \frac{8}{175}.
  \end{equation*}
  Therefore, the orthonormal basis is:
  \begin{equation*}
    \set{
      \frac{\vect{u}_1}{\norm{\vect{u}_1}},
      \frac{\vect{u}_2}{\norm{\vect{u}_2}},
      \frac{\vect{u}_3}{\norm{\vect{u}_3}},
      \frac{\vect{u}_4}{\norm{\vect{u}_4}}
    }
    = \set{
      \frac{1}{\sqrt{2}},~
      \sqrt{\frac{3}{2}} x,~
      \sqrt{\frac{45}{8}} (x^2-\frac{1}{3}),~
      \sqrt{\frac{175}{8}} (x^3 - \frac{3}{5}x)
    }.
  \end{equation*}
\end{solution}

We finish this section by remarking that the formula
\begin{equation*}
  \frac{\iprod{\vect{u},\vect{v}}}{\iprod{\vect{u},\vect{u}}}\vect{u}
\end{equation*}
is exactly what we called the \textbf{projection of $\vect{v}$ onto
  $\vect{u}$}%
\index{vector!projection of}%
\index{projection!in inner product space}%
\index{projection!vector to vector} in Section~\ref{ssec:projections},
except that we have generalized this concept from $\R^n$ to an
arbitrary inner product space. We can define
\begin{equation*}
  \proj_{\vect{u}}(\vect{v})
  = \frac{\iprod{\vect{u},\vect{v}}}{\iprod{\vect{u},\vect{u}}}\vect{u}.
\end{equation*}
With this definition, the Gram-Schmidt procedure can also be expressed
more succinctly as follows.
\begin{equation*}
  \begin{array}{rcl}
    \vect{u}_1
    &=& \vect{v}_1,
    \\
    \vect{u}_2
    &=& \vect{v}_2
        ~-~ \proj_{\vect{u}_1}(\vect{v}_2),
    \\
    \vect{u}_3
    &=& \vect{v}_3
        ~-~ \proj_{\vect{u}_1}(\vect{v}_3)
        ~-~ \proj_{\vect{u}_2}(\vect{v}_3),
    \\
    &\vdots&
    \\
    \vect{u}_k
    &=& \vect{v}_k
        ~-~ \proj_{\vect{u}_1}(\vect{v}_k)
        ~-~ \proj_{\vect{u}_2}(\vect{v}_k)
        ~-~ \ldots
        ~-~ \proj_{\vect{u}_{k-1}}(\vect{v}_k).
  \end{array}
\end{equation*}


\lab{Applications}{Voronoi Diagrams and Delaunay Triangulations}{Voronoi Diagrams and Delaunay Triangulations}
\label{lab:voronoi}

\objective{Introduce Voronoid Diagrams and Delaunay Triangulations and discuss their applications}

\section*{Voronoi Diagrams}

In this lab we will discuss some applications of Voronoi Diagrams.
In the abstract sense, a Voronoi diagram is a partition of a plane into regions that lie closest to different points.
The easiest way to understand this is to look at some examples.
Figure \ref{voronoi_ex_1} is a voronoi diagram generated from 100 random points with $x$ and $y$ values between 0 and 1.
Notice how each point has a small  cell around it that is made up of the points that lie closest to it.

\begin{figure}
\includegraphics[width=\textwidth]{voronoi_example_1.pdf}
\caption{A Voronoi diagram of the square $[0,1]\times [0,1]$ for 100 randomly generated points.}
\label{voronoi_ex_1}
\end{figure}

One possible way to compute this sort of diagram would be by brute force, for example:
\begin{lstlisting}
import numpy as np
from numpy.random import rand
from matplotlib imort pyplot as plt
num = 10
res = 201
pts = rand(num, 2)
X = np.linspace(0, 1, res)
Y = X.copy()
X, Y = np.meshgrid(X, Y)
Z = np.empty_like(X)
indices = np.zeros_like(X)
for i in xrange(res):
    for j in xrange(res):
        #note: we don't need to do the sqare root
        #since we really just need to compare the distances
        mn = (X[i,j] - pts[0,0])**2 + (Y[i,j] - pts[0,1])**2
        for k in xrange(1,num):
            dist = (X[i,j] - pts[k,0])**2 + (Y[i,j] - pts[k,1])**2
            if dist < mn:
                indices[i,j] = k
                mn = dist
plt.pcolormesh(X,Y,indices)
plt.scatter(pts[:,0],pts[:,1])
plt.xlim((0,1))
plt.ylim((0,1))
plt.show()
\end{lstlisting}
This algorithm is good because it can work regardless of the metric space we are using, but it is terribly slow.
Figure \ref{voronoi_1norm} shows a similar diagram using the 1-norm and figure \ref{voronoi_supnorm} shows a diagram generated using the supremum norm.
It is linear in the number of points added and linear in the number of pixels used to represent the diagram.
This can be a terrible limitation, but if you are not working in a well behaved metric space, this may be the simplest approach.
\begin{figure}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{voronoi_1norm.png}
\caption{Voronoi diagram in the 1-norm}
\label{voronoi_1norm}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{voronoi_supnorm.png}
\caption{Voronoi diagram in the supremum norm}
\label{voronoi_supnorm}
\end{minipage}
\end{figure}

It is also possible to make voronoi diagrams based off of shapes and lines as well as points.

There are several data structures used in the field of computational geometry that can store the exact edges of diagrams like this.
We would prefer to not have to worry about sampling the domain in this way.
There are a variety of algorithms that can compute Voronoi diagrams in $\mathcal{O}\left( n \log\left(n\right)\right)$ time (where $n$ is the number of points).
Fortune's algorithm is a linesweep algorithm that can do this.

The Qhull library is commonly used to compute voronoi diagrams, and delaunay triangulations.
SciPy includes a wrapper of the Qhull library in the \li{scipy.spatial} module.
It currently only supports 2 dimensional voronoi diagrams under the Euclidean norm.
It allows you to generate a voronoi diagram from a set of points, add points to a voronoi diagram, find the nearest point to any given point, and plot a voronoi diagram.

A plot similar to the one we just generated can be made using SciPy like this:
\begin{lstlisting}
import numpy as np
from numpy.random import rand
import scipy.spatial as st
from matplotlib import pyplot as plt
G = rand(100,2)
G = st.Voronoi(G)
st.voronoi_plot_2d(G)
plt.xlim((0, 1))
plt.ylim((0, 1))
plt.show()
\end{lstlisting}
Notice how much clearer the plot is and how much faster it is generated.
Try running the last bit of code for 1000 points.
Though this is a little slower than it was for 10 points, this is a graph that would not display well if we had been using the brute force method.

Table \ref{voronoi_attributes} shows the different attributes of the \li{Voronoi} object.
For more details, see \url{http://docs.scipy.org/doc/scipy-dev/reference/spatial.html}
\begin{table}[h!]
\begin{center}
	\begin{tabular}{|l|p{12cm}|}
    \hline

    \li{points} & coordinates of input points (centers of cells)\\

    \li{vertices} & vertices of graph\\

    \li{ridge_points} & indices of the input points between which each ridge on the diagram lies\\

    \li{ridge_vertices} & indices of the vertices at the end of each ridge\\

    \li{regions} & indices of the vertices corresponding to each voronoi cell\\

    \li{point_region} & indices of the regions corresponding to each input point\\

    \hline

    \end{tabular}
\end{center}
\caption{Various summarizing functions}
\label{voronoi_attributes}
\end{table}

Unfortunately, the \li{Voronoi} objects do not allow us to find out, given the coordinates of a new point, which of our original input points lies the closest to it.
To do something like that we would need to use a different algorithm or data structure.
One possible way of solving such a problem is to us a KD tree.
KD trees will be discussed in further detail in Volume II.
Scipy includes built in KD trees in both Python and C.
The version in C is generally faster.
You can make a KD tree like this
\begin{lstlisting}
import numpy as np
from numpy.random import rand
import scipy.spatial as st
A = rand(1000,2)
kd = st.cKDTree(A)
\end{lstlisting}
In this case we used the \li{cKDTree} object.
That is a KDTree implemented in C.
\li{KDTree} is a Python-based version that is also included in scipy.
To find the nearest neighbor of a point \li{pt} and how far away it is we can now use the following line of code:
\begin{lstlisting}
kd.query(pt)
\end{lstlisting}

\section*{Applications of Voronoi Diagrams}

Though the computation of a Voronoi Diagram does not provide a quick way to run nearest neighbor queries, there are still several applications for Voronoi Diagrams.
The first, and probably most obvious, is the representation of data.
If you want to look for visual patterns in data, a Voronoi Diagram can be very useful.
One historically significant example was the containment of the London Cholera outbreak of 1854.
The English Mathematician John Snow plotted where the cholera outbreaks had all happened and after some consideration, noticed that they were all relatively close to a certain water pump in that portion of the city.
Upon noticing this, he plotted the voronoi cell of that particular water pump and proposed that the cholera outbreak was linked to contaminated water.
He recommended that the pump handle be removed so that people would have to use other pumps in the city.
Once the contaminated pump was shut down, the cholera outbreak was stopped.

Voronoi diagrams can also be used to solve problems involving the points furthest from those already on the graph.
Examples include determining where to drill next when searching for oil, or where to put a new branch of a major company.

The file \li{edge_intersections.py} included with this lab includes a helper function, \li{edge_intersections}, that will be helpful for the rest of the lab.
It takes a Voronoi diagram object as input as well as two tuples (xlims and ylims) representing the limits of a rectangle.
It returns an array containing the intersections of the edges of the voronoi diagram.
It also returns two lists of pairs of indices.
In the first list of indices, the first index in each pair corresponds to one of the vertices of the Voronoi diagram and the second index corresponds to one of the intersections found by this function.
The second list of indices accounts for the case that no endpoint of an edge intersecting the region lies within the region.
In the second list of indices, both indices of each pair correspond to intersections with the boundaries of the region.

\begin{problem}
\label{FurthestPoints}
Write a function that, given a list of points and a rectangular region of the plain and an integer $n$, finds the $n$ points in the region that lie farthest from all the points already given.
Use the helper function given to test the intersections of the edges of the voronoi diagram with the intersections of the rectangular region.
\end{problem}

Another possible application is navegation through a field of obstacles.

\begin{problem}
Write a function that, given a list of points, forms the adjacency matrix corresponding to the connections between each node of the Voronoi diagram.
Have this function take a threshold value to limit which ridges are included.
Do not include a ridge in the adjacency matrix if the distance from the ridge to the center points of the Voronoi cells is less than the threshold.
You can find this distance by measuring the distance between each pair of points referenced in the \li{ridge_points} attribute of the Voronoi object and then dividing by two.
Note: the infinite ridges of the graph have $-1$ listed as one of the indices in the \li{ridge_vertices} attribute.
Make sure you do not include these ridges.

Use your solution to Lab \ref{lab:MarkovGraph} Problem \ref{maze_prob} to find a path from any given node to any other node.
Write another function that plots the Voronoi diagram and the path between any two given nodes.

Note: if you have a working implementation of Djikstra's algorithm, you may use it and use the graph weights corresponding to the distance between nodes instead.
Using a modified version of Djikstra's Algorithm you could maximize the distance from any node to the path, but we won't ask you to do that here.
A fuller solution of this problem would also account for the bounds of the region.
That could be done with the helper function included with this lab.
\end{problem}

Another possible application of voronoi diagrams is in the estimation of total rainfall, size of ore deposits, or other similar problems.
One simple way to do this is to take a weighted average of all known measurements where each measurement is given the weight corresponding to the size of its voronoi cell.

\begin{comment}
% This problem is cool, but it would probably make the lab too long
% I'll leave it here in case we want to add it later

\begin{problem}
\label{AverageRainfall}
Write a function that, given a square region and measurement values at different nodes, computes the weighted average of the measurements over the region.
Weight each measurement according to the area of the voronoi cell of each node.

Hint: You can find the area of each voroni cell by considering the triangles formed between its vertices and its center point. One way to compute the area of a triangle given the coordinates of its vertices is $A = \sqrt{s\left(s-a\right) \left(s-b\right) \left(s-c\right)}$ where $s = \frac{a+b+c}{2}$ and $a$, $b$, and $c$ are the lengths of the edges of the triangle.
This is known as Herron's Formula.
\end{problem}

\end{comment}

\section*{Delaunay Triangulation}

A concept related to Voronoi diagrams is that of the Delaunay Triangulation.
Delaunay Triangulations also have a wide variety of applications.
One such application is the automatic division of a region into triangles for use in finite element analysis for the numerical solution of partial differential equations.
In general, the delaunay triangulation divides the smallest convex region containing all the given points (the convex hull) into triangles that obey certain rules.

You can make a Delaunay Triangulation from a list of points and plot it like this:
\begin{lstlisting}
A = rand(100, 2)
D = st.Delaunay(A)
plt.triplot(A[:,0] ,A[:,1], D.simplices)
plt.show()
\end{lstlisting}

\begin{problem}
Use a Delaunay triangulation to write a function that breaks up the unit square into right triangles.
Have the only argument to your function be the number of nodes you want along each edge of the square.
Plot your results.
\end{problem}

%other possible applications:
%Use Delaunay Triangulation to tesselate a 3d surface.
%Use tesselation to rerun ore/rainfall problem and compare results.


\input{../../assignment-header}

%========================================================================
\title{Assignment 2:  Simulation Basics}
\date{Assigned:  Jan. 23  ---  Due:  Feb. 1}
\author{Optimal Control For Robotics}
%========================================================================
\begin{document}
\maketitle
%=================================================

\section*{Introduction}\

In this assignment you will implement Euler's method and
use it to simulate two simple dynamical systems.
The first system is an ideal passive pendulum,
one of the simplest non-linear systems that corresponds to a physical mechanism.
The second system is the Lorenz system,
which is highly non-linear and generates the famous Lorenz Attractor.

\par This assignment includes a set of template files to guide you in writing the
simulation function, dynamics functions, and top-level code to generate the plots.
\vspace{-0.6em} \begin{itemize}  \setlength\itemsep{0em}
  \item \texttt{EulerMethodSimulate.m} --- simulate a dynamical system using Euler's method
  \item \texttt{PendulumDynamics.m} --- compute the system dynamics for a simple pendulum
  \item \texttt{LorenzDynamics.m} --- compute the system dynamics for the Lorenz system
  \item \texttt{prob\_03\_studentName.m} --- run the simulation of the pendulum and make plots
  \item \texttt{prob\_04\_studentName.m} --- run the simulation of the Lorenz system and make plots
\end{itemize}

You will need to complete each of these files by replacing the \texttt{\%\%\%\% TODO} markings with
correct implementations in each case. There are detailed comments throughout the template code
to indicate what each section of the code should do. The template files are described in
detail at the end of this assignment, and included as stand-alone Matlab files as well.

\section*{Write-Up}

You will submit a single (short) write-up for this assignment:
\vspace{-0.0em} \begin{itemize}  \setlength\itemsep{0em} \setlength\itemindent{18pt}
  \item Header: full name, studentName, date, assignment name and number
  \item List any other students that you worked with.
  \item How long did this assignment take you?
\end{itemize}

\section*{Deliverables}
\begin{itemize}
  \item Fully implement all five template files and submit them.
  \item Submit the plots that you generated for each simulation: \\
        \texttt{prob\_03\_studentName.pdf}   and   \texttt{prob\_04\_studentName.pdf}
  \item Submit the write-up for the assignment:   \texttt{hw\_02\_studentName.txt}
  \item All code should be clearly written and well documented. Figures should include labels and titles.
  \item All files should besubmitted through Tufts Trunk (as a single compressed file)
\end{itemize}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\pagebreak
\section*{\texttt{EulerMethodSimulation.m}}

The key line that you will write in this file will implement Euler's Method for integration,
given below, where $\bm{z}_k$ is the state at the current time step, $\bm{z}_{k+1}$ is the state
at the next time step, $h$ is the duration of the time step, and $\dot{\bm{z}}_k$ is the time-derivative
of the state at the current time step.
Notice that this function is general-purpose: it can be used to simulate any dynamical system.

\begin{equation}
  \bm{z}_{k+1} = \bm{z}_k + h \dot{\bm{z}}_k
\end{equation}

\lstinputlisting{EulerMethodSimulation.m}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\pagebreak
\section*{\texttt{PendulumDynamics.m}}

The dynamics of a simple pendulum can be modeled as:

\begin{equation}
  \ddot{q} = -\sin(q)
\end{equation}

Euler's Method for simulation, like most simulation methods, requires the dynamical system to be
in first-order form. To to this, we will introduce a new variable $\omega = \dot{q}$ to represent
angular rate. This allows us to write a second-order equation as a system of two first-order equations:

\begin{align}
  &  \dot{q} = \omega \\
  &  \dot{\omega} = -\sin(q)
\end{align}

You will notice a tilde (\texttt{\~}) in the argument list for this function.
This tells Matlab that our function does not use this argument
and allows it the compiler to do some optimization to improve run-time speed.
We keep the argument here beause simulation functions
(such as \texttt{EulerMethodSimulate()} and \texttt{ode45()})
often require the first argument of the dynamics function to be time.

\lstinputlisting{PendulumDynamics.m}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\pagebreak
\section*{\texttt{LorenzDynamics.m}}

This function implements the equations for the Lorenz system. For a special combination of
parameters this system produces a famous fractal known as the Lorenz attractor.
The equations for this system are given below, where $x$, $y$, and $z$ are the state of the
system, and $\sigma=10$, $\rho=29$, and $\beta = 8/3$ are system parameters.

\begin{align}
  &  \dot{x} = \sigma (y-x) \\
  &  \dot{y} = x (\rho - z) - y \\
  &  \dot{z} = x y - \beta z
\end{align}

\lstinputlisting{LorenzDynamics.m}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\pagebreak
\section*{\texttt{prob\_03\_studentName.m}}

This function sets up and runs a simulation of the simple pendulum system and then generates a plot
of the resulting simulation. Let's assume that the pendulum is using SI units.
Follow the directions in the template code for simulation and plotting details.
When you submit this file you should replace \texttt{studentName} with your full name.

\lstinputlisting{prob_03_studentName.m}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\pagebreak
\section*{\texttt{prob\_04\_studentName.m}}

This function sets up and runs a simulation of the Lorenz System and then generates a plot of the
Lorenz attractor. We will use dimensionless units.
Follow the directions in the template code for simulation and plotting details.

\lstinputlisting{prob_04_studentName.m}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%
\pagebreak

%=================================================
\end{document}


\hypertarget{sorting-algorithms}{%
\section{Sorting Algorithms}\label{sorting-algorithms}}

\begin{itemize}
\tightlist
\item
  sorting can be \textbf{comparison-based or non-comparison-based}
\item
  the fundamental operation of comparison-based sorting is
  compare-exchange
\item
  the lower bound on any comparison-based sequential sorting of n
  numbers is $\mathcal{O}(n * log (n))$
\item
  we focus here on comparison-based sorting algorithms
\end{itemize}

\hypertarget{compare-exchangesplit}{%
\subsection{Compare-Exchange/Split}\label{compare-exchangesplit}}

\hypertarget{compare-exchange}{%
\subsubsection{Compare Exchange}\label{compare-exchange}}

\begin{figure}[H]
\centering
\includegraphics[width=0.6\textwidth]{figures/compareExchange.png}
\caption{Compare Exchange process}
\end{figure}

\hypertarget{compare-split}{%
\subsubsection{Compare Split}\label{compare-split}}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/comparesplit.png}
\caption{Compare Split process}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/comparesplitExample.png}
\caption{Compare Split example}
\end{figure}

\hypertarget{odd-even-transposition-sort}{%
\subsection{Odd-Even Transposition
Sort}\label{odd-even-transposition-sort}}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/transpositionsort.png}
\caption{Odd-Even Transposition Sort}
\end{figure}

\hypertarget{parallel-shellsort}{%
\subsection{Parallel Shellsort}\label{parallel-shellsort}}

\begin{itemize}
\tightlist
\item
  let n be the number of elements to be sorted and p be the number of
  processes
\item
  during the first phase, processes that are far away from each other in
  the array compare split their elements
\item
  during the second phase, the algorithm switches to an odd-even
  transposition sort with $2l \leq p$ iterations
\end{itemize}

In phase 1:

\begin{itemize}
\tightlist
\item
  initially, each process sorts its block of $n/p$ elements internally
\item
  each process is now paired with its corresponding process in the
  reverse order of the array: that is, process $P_i$ is paired with
  process $P_{p-i-1}$, where $i < p/2$
\item
  all pairs perform a compare-split operation
\item
  the processes are split into two groups of size $p/2$ each and the phase
  1 is repeated in each group until a group contains one process only
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.35\textwidth]{figures/parallelshellsort.png}
\caption{Parallel Shellsort}
\end{figure}

\begin{itemize}
\tightlist
\item
  In the first phase, each process performs $d = log(p)$ compare-split
  operations
\item
  with $\mathcal{O}(p)$ bisection width, each communication can be performed in time   $\Theta(n/p)$ for a total time of $\Theta ( \frac{n * log (p)}{p})$
\item
  in the second phase, l odd and l even iterations are performed, each
  requiring time $\Theta(n/p)$
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/runtimeShellsort.png}
\caption{Parallel runtime of Shellsort}
\end{figure}

\clearpage
\hypertarget{sorting-networks}{%
\subsection{Sorting Networks}\label{sorting-networks}}

\begin{itemize}
\tightlist
\item
  a sorting network is a network of comparators designed specifically
  for sorting
\item
  a comparator is a device with

  \begin{itemize}
  \tightlist
  \item
    2 inputs: $x$ and $y$
  \item
    2 outputs: $x'$ and $y'$
  \end{itemize}
\item
  an increasing comparator $\oplus : x' = min\{x,y\} and y' = max\{x,y\}$
\item
  a decreasing comparator $\ominus : x' = max\{x,y\} and y' = min\{x,y\}$
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/sortingNetwork.png}
\caption{Sorting Networks}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/sortingNetwork2.png}
\caption{Sorting Network Example}
\end{figure}

\clearpage
\hypertarget{bitonic-sort}{%
\subsection{Bitonic Sort}\label{bitonic-sort}}

A bitonic sequence has two tones (sequences), one increasing and one
decreasing or vice versa. Any cyclic rotation fo a bitonic sequence is
also bitonic (repeated sequence). E.g. \{1,2,4,7,6,0\} is bitonic or
\{8,9,2,1,0,4\} is bitonic (because it's repeated).

For the bitonic sort, we

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  build a single bitonic sequence from the given unsorted sequence
\item
  rearrange the bitonic sequence into a sorted sequence
\end{enumerate}

Parallel Runtime (p = n): a bitonic sorting network sorts n elements in
$\Theta(log^2 (n))$ time.

\hypertarget{bitonic-merge}{%
\subsubsection{Bitonic Merge}\label{bitonic-merge}}

\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{figures/bitonicSequenceTheory.png}
\caption{Theory Bitonic Merge}
\end{figure}

In other words: At the beginning, there is one big bitonic sequence. In
splitting this bitonic sequence into two, you have two new sequences
(lower and upper sequence). If you repeate this splitting, you will have
a sorted sequence at the end.

\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{figures/bitonic-merge.png}
\caption{Bitonic Merge}
\end{figure}

\clearpage
\hypertarget{building-a-bitonic-sequence}{%
\subsubsection{Building a Bitonic
Sequence}\label{building-a-bitonic-sequence}}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/create-bitonic-sequence.png}
\caption{Creation of a bitonic sequence}
\end{figure}

\hypertarget{bitonic-sort-block-of-elements-per-process}{%
\subsubsection{Bitonic Sort: Block of Elements Per
Process}\label{bitonic-sort-block-of-elements-per-process}}

You are also able to implement bitonic sort if there are less processes
than elements.

\begin{itemize}
\tightlist
\item
  the first step is a local sort of the local block
\item
  each subsequent compare-exchange operation is replaced by a
  compare-split operation
\item
  the bitonic network consists of $(1 + log (p))*\frac{log (p)}{2} = \mathcal{O}(log^2 (p))$
  steps
\end{itemize}

\hypertarget{parallel-quicksort}{%
\subsection{Parallel Quicksort}\label{parallel-quicksort}}

The following figure describes the serial sequence of the quick sort.

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/quicksort-serial.png}
\caption{Quicksort in Serial}
\end{figure}

\clearpage
\hypertarget{recursive-decomposition}{%
\subsubsection{Recursive Decomposition}\label{recursive-decomposition}}

\begin{itemize}
\tightlist
\item
  the array is partitioned serially and each of the sub-problems is
  handled by a different process
\item
  the time for this algorithm is lower-bounded by $\Omega(n)$, because of the serial partitioning
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/quicksort-parallel.png}
\caption{Quicksort in parallel}
\end{figure}

\hypertarget{pram}{%
\subsubsection{PRAM}\label{pram}}

PRAM makes use of CRCW (concurrent read, concurrent write).

\begin{itemize}
\tightlist
\item
  process $P_i$ is assigned element $A[i]$
\item
  build binary tree
\item
  traverse tree and count elements in left and right sub-trees and
  compute correct element positions
\item
  each process writes its element to the correct position in A
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.4\textwidth]{figures/quicksortPRAM.png}
\caption{Quicksort PRAM pseudo-code}
\end{figure}

\begin{itemize}
\tightlist
\item
  The arbitrary choice of root (by concurrent write of all processes) is
  the new pivot
\item
  The root doesn't work anymore.
\item
  All other processes save their value into the proper array of the
  existing two halfs
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.7\textwidth]{figures/binarytreeQuicksortExample.png}
\caption{Quicksort Example Binary tree}
\end{figure}

\begin{itemize}
\tightlist
\item
  As soon as the binary tree is created, the values need to be written
  to the goal array
\item
  Each node needs to know his final position in the goal array
\item
  The position can be calculated from the left childs of the current
  node
\item
  The number of childs can be calculated recursively by asking his child
  how many child it has (which asks his childs again\ldots{})
\end{itemize}

\clearpage
\hypertarget{shared-address-space-sas-parallelizing-quicksort}{%
\subsubsection{Shared Address Space (SAS): Parallelizing
Quicksort}\label{shared-address-space-sas-parallelizing-quicksort}}

Consider an array of size n equally divided across p processes.

\begin{itemize}
\tightlist
\item
  Pivot Selection

  \begin{itemize}
  \tightlist
  \item
    a pivot is selected by one of the processes and made known to all
    processes
  \item
    can be made local from one process
  \end{itemize}
\item
  Local Rearrangement

  \begin{itemize}
  \tightlist
  \item
    each process partitions its array block into two, say S\_i and L\_i
    , based on the selected and communicated pivot
  \end{itemize}
\item
  Global Rearrangement

  \begin{itemize}
  \tightlist
  \item
    all of the S\_i arrays are merged to S and all of the L\_i arrays
    are merged to L separately
  \end{itemize}
\item
  Processor Partitioning

  \begin{itemize}
  \tightlist
  \item
    the set of processes is partitioned into two (in proportion of the
    size of arrays S and L)
  \end{itemize}
\item
  the algorithm is recursively repeated for each process group and
  sub-array, until a sub-array is assigned to a single process, in which
  case it proceeds to sort it locally
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{figures/quicksortSASExample.png}
\caption{SAS Quicksort Example}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=0.6\textwidth]{figures/sasQuicksortGlobalRearrangement.png}
\caption{SAS Quicksort Global Rearrangement}
\end{figure}

\hypertarget{bucket-sort}{%
\subsection{Bucket Sort}\label{bucket-sort}}

The sequential bucket sort divides the values which need to be sorted
into different buckets. After that, one can sort the buckts itself.

\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{figures/bucket-sort.png}
\caption{Bucketsort}
\end{figure}

\clearpage
\hypertarget{parallel-bucket-sort}{%
\subsubsection{Parallel Bucket Sort}\label{parallel-bucket-sort}}

The pre-sorting into different buckets can also be done in parallel.

\begin{itemize}
\tightlist
\item
  Input

  \begin{itemize}
  \tightlist
  \item
    each process is assigned a block of $n/p$ elements
  \item
    the number of buckets $m = p$
  \item
    each process knows the range $[a, b]$
  \end{itemize}
\item
  Algorithm

  \begin{itemize}
  \tightlist
  \item
    each process partitions its block of $n/p$ elements into p sub-blocks, one for each of the $p$ buckets: $\Theta(n/p)$
  \item
    each process sends $p- 1$ sub-blocks to the appropriate processes
    using a single all-to-all personalized communication: $\Theta(p * n/p^2) = \Theta(n/p)$
  \item
    each process sorts all the elements it receives by using an optimal
    sequential sorting algorithm: $\Theta((n/p) * log (n/p))$
  \end{itemize}
\item
  Parallel Runtime

  \begin{itemize}
  \tightlist
  \item
    $T_P = \Theta(n/p) + \Theta(n/p) + \Theta((n/p) *log (n/p)) = \Theta((n/p) * log (n/p))$
  \end{itemize}
\end{itemize}

\clearpage
\hypertarget{sample-sort}{%
\subsection{Sample Sort}\label{sample-sort}}

\begin{tcolorbox}[colback=red!5!white,colframe=red!75!black]
Random Sampling: If I don't know the distribution of my data, I can take a random sequence of this data ond check the distribution of this sequence, with the assumption that it is the same as the rest of the data.
\end{tcolorbox}

\begin{itemize}
\tightlist
\item
  Similar to Bucket Sort

  \begin{itemize}
  \tightlist
  \item
    without the unrealistic assumption of uniformly distributed elements
  \item
    a sample is selected from the n elements, and the range of the
    buckets is determined by choosing m - 1 elements (splitters) from
    the sample
  \end{itemize}
\item
  Splitter Selection Method

  \begin{itemize}
  \tightlist
  \item
    the n elements are divided into m blocks of size n/m each
  \item
    each block is sorted by using an optimal sequential sorting
    algorithm
  \item
    from each sorted block m - 1 evenly spaced elements are chosen
  \item
    the m(m - 1) elements selected from all the blocks represent the
    sample used to determine the buckets
  \item
    from the sorted sample choose m - 1 evenly spaced splitters
  \item
    this scheme guarantees that the number of elements ending up in each
    bucket is less than 2n/m if the elements don't contain duplicates
  \end{itemize}
\end{itemize}

\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{figures/samplesort.png}
\caption{Samplesort}
\end{figure}

\clearpage
\hypertarget{parallel-sample-sort}{%
\subsubsection{Parallel Sample Sort}\label{parallel-sample-sort}}

\begin{itemize}
\tightlist
\item
  Parallelized Splitter Selection Scheme

  \begin{itemize}
  \tightlist
  \item
    choose $m = p$
  \item
    each processor generates the $p - 1$ local splitters in parallel
  \item
    all processors share their splitters using a single all-to-all
    broadcast operation
  \item
    each processor sorts (merges) the same $p*(p- 1)$ sorted sample
    elements it receives and selects the same $p - 1$ uniformly spaces
    splitters from them
  \end{itemize}
\item
  Parallel Algorithm

  \begin{itemize}
  \tightlist
  \item
    each process partitions its block of $n/p$ elements into p sub-blocks
    according to the $p - 1$ splitters, one for each of the p buckets
  \item
    each process sends p sub-blocks to the appropriate processes (a
    single all-to-all personalized communication is useful only if all
    sub-blocks are of almost equal size)
  \item
    each process merges all the received p sub-blocks
  \end{itemize}
\end{itemize}

\hypertarget{analysis-of-sample-sort}{%
\subsubsection{Analysis of Sample Sort}\label{analysis-of-sample-sort}}

\begin{figure}[H]
\centering
\includegraphics[width=0.85\textwidth]{figures/analysis-samplesort.png}
\caption{Analysis of the sample-sort}
\end{figure}

\textbf{Total} $= \mathcal{O}(\frac{n}{p}*log(\frac{n}{p}))+\mathcal{O}(p^2 * log(p)) + \mathcal{O}(p * log(\frac{n}{p})) + \mathcal{O}(\frac{n}{p} * log(p))$ \\
$= \mathcal{O}(\frac{n}{p}*log(n)) + \mathcal{O}(p^2*log(p)) + \mathcal{O}(p*log(\frac{n}{p}))$\\ \\
Da gilt: $\mathcal{O}(\frac{n}{p}*log (\frac{n}{p})) = \mathcal{O}(\frac{n}{p}*log(n) - \frac{n}{p}*log(p))$


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS622: Theory of Formal Languages
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Define the mapping $f:A^*\rightarrow A^*$ by

\begin{equation}\label{assum1}
f(\lambda)=\lambda
\end{equation}

\begin{equation}\label{assum2}
f(ax)=xa
\end{equation}

for every $x \in A^*$ and $a \in A$.
Prove that $f$ is a \textit{bijection}, that is, it is one-to-one and onto.

\subsection*{Solution}

To prove $f$ is a bijection, it is first proven that $f$ is one-to-one, using proof by contradiction.
Let us assume that $f: A^* \rightarrow A^*$ is not one-to-one, that is, there is at least an element $y$ in $A^*$ to which two different elements $u, v \in A^*$, $u \neq v$ are mapped by $f$.
That is following assumptions are made.

\begin{eqnarray}
f(u) = y \label{assum3} \\
f(v) = y \label{assum4} \\
u \neq v \label{assum5}
\end{eqnarray}

Two possible cases on the nature of $y$ arise:

\begin{enumerate}[label=(\alph*)]

\item
$y = \lambda$

Using assumption \eqref{assum1},

\begin{equation}\label{assum6}
y=\lambda=f(\lambda)=f(y)
\end{equation}

Substituting \eqref{assum6} in \eqref{assum3} and \eqref{assum4} gives

\begin{equation}
f(u) = f(y) = f(v) \Rightarrow u = v
\end{equation}

which is in contrast to our assumption \eqref{assum5}.

\item
$y \neq \lambda$

In this case $y$ has at least one symbol and hence can be shown by $y=xa$ where $a \in A$ and $x \in A^*$.
$u$ and $v$ must also have at least one symbol as, if otherwise, Based on \eqref{assum1}, $f(\lambda)$ would never be mapped to a symbol.
Thus $u$ and $v$ can be represented as $u = by$ and $v = cz$ where $\{b,c\}\in A$ and $\{y,z\}\in A^*$.

From \eqref{assum3} and \eqref{assum4}:
\begin{eqnarray}
f(u) = f(by) = y = xa \label{2eq8}\\
f(v) = f(cz) = y = xa \label{2eq9}
\end{eqnarray}

Based on \eqref{assum2} following would also be true.
\begin{eqnarray}
f(u) = f(by) = yb \label{2eq10}\\
f(v) = f(cz) = zc \label{2eq11}
\end{eqnarray}

From \eqref{2eq8} and \eqref{2eq10}

\begin{equation}\label{2eq12}
xa = yb
\end{equation}

And from \eqref{2eq9} and \eqref{2eq11}

\begin{equation}\label{2eq13}
xa = zc
\end{equation}

Since $a$, $b$ and $c$ are symbols, \eqref{2eq12} and \eqref{2eq13} can only happen if

\begin{eqnarray}
a = b = c & x = y = z
\end{eqnarray}

Thus

\begin{equation}
by = cz \Rightarrow u = v
\end{equation}

which contradicts our earlier assumption \eqref{assum5}.

\end{enumerate}

Based on contradictions occurred in both possible cases, one-to-one property of $f:A^*\rightarrow A^*$ holds true.

A similar approach will be conducted to prove the onto property of $f:A^*\rightarrow A^*$ mapping.
By proof of contradiction, it is assumed that $f$ is not onto; suggesting that there is at least one element $y$ in $A^*$ that is not mapped to by any element in $A^*$.
Two possible cases for $y$ would arise.

\begin{enumerate}[label=(\alph*)]
\item $y = \lambda$

Based on \eqref{assum1}, there exists an element $\lambda$ in $A^*$ that when mapped by $f$ is mapped to $y = \lambda$.
\item $y \neq \lambda$

In this case $y$ has at least one symbol and hence can be shown by $y=xa$ where $a \in A$ and $x \in A^*$.
Based on \eqref{assum2} there is always an element $z \in A^*$ where $z = ax$ and $f(z) = f(ax) = xa = y$.

\end{enumerate}

As both possible cases contradict with our earlier assumption that $y$ is not mapped to by any element in $A^*$, contradicted assumption is false and $f:A^*\rightarrow A^*$ holds onto.


\subsection{Cournot model}

With competition, the elasticity of demand refers to the whole market, not just a single producer. Instead we have:

\(\epsilon = \dfrac{p}{Q}\dfrac{\delta Q}{\delta p}\)

\(Q=\sum_j q_j\)

We now get:

\(p[1+\dfrac{q}{Q}\dfrac{\delta Q}{\delta q}\dfrac{Q}{p}\dfrac{\delta p}{\delta Q}]=MC\)

\(p[1+\dfrac{\mu }{\epsilon }]=MC\)

Using the firm's size elasticity: \(\mu = \dfrac{q}{Q}\dfrac{\delta Q}{\delta q}\)

With monopoly this is:

\(\mu = 1\)

In this model this is:

\(\dfrac{p-MC}{p}=-\dfrac{\mu}{\epsilon }\)


\section{Source Code}

To save pages, only codes that are relevant to the text are given below. I will upload all the codes to \href{https://github.com/iChunyu/signal-process-demo}{my GitHub repository} after I get score of \textsl{Digital Signal Processing} to avoid being judged cheating.

\subsection{LPSD Algorithm}

{\noindent \bfseries Python Version}

\begin{lstlisting}[language=Python]
'''
Personal fuction library in Python
Contents
    (1) pxx,f = lpsd(data,fs,Jdes=1000)

Last Update: 2021-02-10
'''

# %%
import numpy as np
import matplotlib.pyplot as plt

# %% function01: LPSD
'''
LPSD: Logarithmic frequency axis Power Spectral Density
pxx,f = lpsd(data,fs,Jdes=1000)
   data --- Input data series, one dimension vector
   fs   --- Sample frequency, unit: Hz
   Jdes --- Desired frequency points, default: 1000
   pxx  --- One-sided PSD, unit: *^2/Hz
   f    --- Frequency points related to PSD points, unit: Hz
   Default window function is hanning window.

Ref: Improved spectrum estimation from digitized timeseries on a logarithmic frequency axis
     Article DOI: 10.1016/j.measurement.2005.10.010

Remark: Unfamiliar with Python by now, the functiondoesn't support matrix data by now.
    [MATLAB version supports matrix data.]

XiaoCY 2021-02-10
'''

# subfuction
# calculate frequency points
def getFreqs(N,fs,Jdes,Kdes,ksai):
    fmin = fs/N
    fmax = fs/2
    r_avg = fs/N*(1+(1-ksai)*(Kdes-1))

    g = (N/2)**(1/(Jdes-1))-1

    f = np.zeros(Jdes)-1
    L = np.copy(f)
    m = np.copy(f)
    j = 0
    fj = fmin
    while fj < fmax:
        rj = fj*g
        if rj < r_avg:
            rj = np.sqrt(rj*r_avg)
        if rj < fmin:
            rj = fmin

        Lj = np.floor(fs/rj)
        rj = fs/Lj
        mj = fj/rj

        f[j] = fj
        L[j] = Lj
        m[j] = mj

        fj = fj+rj
        j += 1

    idx = f<0
    f = np.delete(f,idx)
    L = np.delete(L,idx)
    m = np.delete(m,idx)
    return f,L,m

# main
def lpsd(data,fs,Jdes=1000):
    Kdes = 100
    ksai = 0.5
    N = len(data)

    f,L,m = getFreqs(N,fs,Jdes,Kdes,ksai)

    J = len(f)
    pxx = np.zeros(J)
    for j in range(J):
        Dj = np.floor((1-ksai)*L[j])
        Kj = np.floor((N-L[j])/Dj+1)
        w = np.hanning(L[j])
        C_PSD = 2/fs/np.dot(w,w)
        l = np.arange(0,L[j])
        W1 = np.cos(-2*np.pi*m[j]/L[j]*l)
        W2 = np.sin(-2*np.pi*m[j]/L[j]*l)
        A = 0.0
        for k in range(int(Kj)):
            G = data[int(k*Dj):int(k*Dj+L[j])].copy()
            G = G-np.mean(G)
            G = G*w
            A += np.dot(G,W1)**2 + np.dot(G,W2)**2
        pxx[j] = A/Kj*C_PSD

    return pxx,f

if __name__ =='__main__':
    fs = 10
    x = np.random.randn(1000)*np.sqrt(fs/2)
    pxx,f = lpsd(x,fs)

    plt.figure
    plt.loglog(f,np.sqrt(pxx))
    plt.grid()
    plt.xlabel('Frequency (Hz)')
    plt.ylabel(r'PSD ($\rm*/\sqrt{Hz}$)')
\end{lstlisting}

{\noindent \bfseries MATLAB Version}

\begin{lstlisting}[language=Matlab]
% Use LPSD mothod to plot power spectral density
% [Pxx,f] = iLPSD(Data,fs,Jdes)
%    Data --- Input data, processed by column
%    fs   --- Sample frequency, unit: Hz
%    Jdes --- Desired frequency points, default: 1000
%    Pxx  --- One-sided PSD, unit: *^2/Hz
%    f    --- Frequency points related to PSD points,unit: Hz
%    Default window function is hanning window.
% Demo:
%    iLPSD(data,fs)
%       Plot PSD using default settings.
%    h = iLPSD(data,fs,Jdes)
%       Plot PSD with desired points of 1000
%    [Pxx,f] = iLPSD(data,fs)
%       Return PSD points, not plot any figure

% Ref: Improved spectrum estimation from digitizedtime series on a logarithmic frequency axis
%      Article DOI: 10.1016/j.measurement.2005.10.010

% XiaoCY 2020-04-21

%% Main
function varargout = iLPSD(varargin)

    nargoutchk(0,2);
    narginchk(2,3);

    data = varargin{1};
    fs = varargin{2};
    if nargin == 3
        Jdes = varargin{3};
    else
        Jdes = 1000;
    end

    Kdes = 100;
    ksai = 0.5;

    [N,nCol] = size(data);
    if N==1 && nCol~=1
       data = data';
       N = nCol;
       nCol = 1;
    end

    [f,L,m] = getFreqs(N,fs,Jdes,Kdes,ksai);

    J = length(f);
    P = zeros(J,nCol);
    for j = 1:J
        Dj = floor((1-ksai)*L(j));
        Kj = floor((N-L(j))/Dj+1);
        w = hann(L(j));
        C_PSD = 2/fs/sum(w.^2);
        l = (0:L(j)-1)';
        W1 = cos(-2*pi*m(j)/L(j).*l);
        W2 = sin(-2*pi*m(j)/L(j).*l);
        A = zeros(1,nCol);
        for k = 0:Kj-1
            G = data(k*Dj+1:k*Dj+L(j),:);
            G = G-mean(G);        % G = detrend(G);
            G = G.*w;
            A = A + sum(G.*W1).^2+sum(G.*W2).^2;
        end
        P(j,:) = A/Kj*C_PSD;
    end

    switch nargout
        case 0
            PlotPSD(P,f)
        case 1
            varargout{1} = PlotPSD(P,f);
        case 2
            varargout{1} = P;
            varargout{2} = f;
        otherwise
            % Do Nothing
    end
end

%% Subfunctions
% get logarithmic frequency points
function [f,L,m] = getFreqs(N,fs,Jdes,Kdes,ksai)
    fmin = fs/N;
    fmax = fs/2;
    r_avg = fs/N*(1+(1-ksai)*(Kdes-1));

    g = (N/2)^(1/(Jdes-1))-1;

    f = zeros(Jdes,1)-1;
    L = f;
    m = f;
    j = 1;
    fj = fmin;
    while fj < fmax
        rj = fj*g;
        if rj < r_avg
            rj = sqrt(rj*r_avg);
        end
        if rj < fmin
            rj = fmin;
        end

        Lj = floor(fs/rj);
        rj = fs/Lj;
        mj = fj/rj;

        f(j) = fj;
        L(j) = Lj;
        m(j) = mj;

        fj = fj+rj;
        j = j+1;
    end
    f(f<0) = [];
    L(L<0) = [];
    m(m<0) = [];
end

% plot PSD
function varargout = PlotPSD(P,f)
    hLine = loglog(f,sqrt(P));
    grid on
    xlabel('Frequency (Hz)')
    ylabel('PSD ([Unit]/Hz^{1/2})')

    if nargout == 1
        varargout{1} = hLine;
    end
end
\end{lstlisting}

\subsection{Main Code}

{\noindent \bfseries Python Version}

\begin{lstlisting}[language=Python]
'''
Main code for DSP project
(See ../DSP_Project_Reqirement&Guidance(2020 Fall).pdfProject-1 for detail.)

XiaoCY 2021-02-18
'''

#%%
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as signal
import springlib as sp

savepdf = False
N = 1000
fs = 1000.0
pi = np.pi

t = np.arange(N)/fs
x1 = np.cos(2*pi*50*t)
x2 = 0.3*np.cos(2*pi*100*t + pi/3)
x3 = 0.2*np.cos(2*pi*150*t - pi/3)
x4 = 0.1*np.sin(2*pi*250*t)
xn = 0.05*np.random.randn(N)
x = x1+x2+x3+x4+xn

#%% design IIR filter
Wp = 110./(fs/2)             # passband cornerfrequency (normalized)
Ws = 130./(fs/2)             # stopband cornerfrequency (normalized)
Rp = 1.                      # passband ripple (dB)
Rs = 60.                     # stopband attenuation(dB)

Nz,Wn = signal.cheb2ord(Wp,Ws,Rp,Rs)
b,a = signal.cheby2(Nz,Rs,Wn)

f = np.linspace(0,fs/2,500)
_,g = signal.freqz(b,a,f,fs=fs)
gdb = 20*np.log10(np.abs(g))

plt.figure()
plt.plot(f,gdb,label='Chebyshev Type II')
plt.grid()
plt.legend()
plt.xlabel('Frequency (Hz)')
plt.ylabel('Gain (dB)')
plt.xlim(0,fs/2)
if savepdf:
    plt.savefig('prjFilter.pdf')
plt.show()

#%% filter data
y = signal.filtfilt(b,a,x)

plt.figure()
plt.plot(t,x,label='original')
plt.plot(t,y,label='filtered')
plt.grid()
plt.legend()
plt.xlabel('Time (s)')
plt.ylabel('Signal (A)')
plt.xlim(0,0.2)
if savepdf:
    plt.savefig('prjSignal.pdf')
plt.show()

pxx,fx = sp.lpsd(x,fs)
pyy,fy = sp.lpsd(y,fs)

plt.figure()
plt.semilogy(fx,np.sqrt(pxx),label='original')
plt.semilogy(fy,np.sqrt(pyy),label='filtered')
plt.grid()
plt.legend()
plt.xlabel('Frequency (Hz)')
plt.ylabel(r'Signal ($\rm A/\sqrt{Hz}$)')
plt.xlim(0,fs/2)
if savepdf:
    plt.savefig('prjPSD.pdf')
plt.show()
\end{lstlisting}

{\noindent \bfseries MATLAB Version}

\begin{lstlisting}[language=Matlab]
% Main code for DSP project
% (See ../DSP_Project_Reqirement&Guidance(2020 Fall).pdf Project-1 for detail.)

% XiaoCY 2021-02-18

%%
clear;clc

N = 1e3;
fs = 1e3;

t = (0:N-1)'/fs;
x1 = cos(2*pi*50*t);
x2 = 0.3*cos(2*pi*100*t + pi/3);
x3 = 0.2*cos(2*pi*150*t - pi/3);
x4 = 0.1*sin(2*pi*250*t);
xn = 0.05*randn(N,1);
x = x1+x2+x3+x4+xn;

%%
Wp = 110/(fs/2);                % passband corner frequency (normalized)
Ws = 130/(fs/2);                % stopband corner frequency (normalized)
Rp = 1;                         % passband ripple (dB)
Rs = 60;                        % stopband attenuation (dB)

[Nz,Wn] = cheb2ord(Wp,Ws,Rp,Rs);
[b,a] = cheby2(Nz,Rs,Wn);

f = linspace(0,fs/2,500);
g = freqz(b,a,f,fs);
gdb = 20*log10(abs(g));

figure('Name','filter')
plot(f,gdb,'DisplayName','Chebyshev Type II')
grid on
legend
xlabel('Frequency (Hz)')
ylabel('Gain (dB)')

%%
y = filtfilt(b,a,x);

figure('Name','time')
plot(t,x,'DisplayName','original')
hold on
grid on
plot(t,y,'DisplayName','filtered')
legend
xlabel('Time (s)')
ylabel('Signal (A)')
xlim([0,0.1])

figure('Name','PSD')
[pxx,f] = iLPSD([x,y],fs);
semilogy(f,sqrt(pxx))
grid on
legend('original','filtered')
xlabel('Frequency (Hz)')
ylabel('PSD (A/Hz^{1/2})')
\end{lstlisting}


\subsection{Trend stationary}

If we can remove the trend as a function, eg linear or non-linear growth, and the rest is stationary, then the process is trend stationary


% !Mode:: "TeX:UTF-8"
% !TEX program  = xelatex
\subsection{Computer Field}
In the field of computers, the summation formula for the geometric progression is quite important for reducing the computing complexity. For example, we can reduce the summation complexity of the arithmetic progression from $O(n)$ to $O(1)$, and the geometric progression from $O(n^2)$ to $O(n)$, which maximizes memory utilization. You may refer to appendix~\ref{A:python-1} for more information. Here comes an example written in \texttt{IPython}\cite{PER-GRA:2007}:
\begin{pylist}{Comparison}
base = 2
number = 1024

%%timeit
s = 0
for i in range(number):
    s += base**i

%%timeit
s = sum(base**i for i in range(number))

%%timeit
s = base**number - 1
\end{pylist}

The running time of the first block is $656 \mu s ± 10.4\mu s$ per loop, the second block is $628 \mu s ± 15.8 \mu s$, and the third block is $765 ns ± 9.5 ns$ per loop. Obviously, the formula works efficiently.


\section{The dot product}
\label{sec:dot-product}

\begin{outcome}
  \begin{enumerate}
  \item Compute the dot product of vectors geometrically and
    algebraically.
  \item Use properties of the dot product, including the
    Cauchy-Schwarz inequality and the triangle inequality, to prove
    further equalities and inequalities.
  \item Determine whether two vectors are orthogonal.
  \item Compute the scalar and vector projection of one vector onto
    another.
  \item Decompose a vector into orthogonal components.
  \end{enumerate}
\end{outcome}

There are two ways of multiplying vectors that are useful in
applications. The first of these is called the \textbf{dot product},
and the second is called the \textbf{cross product}. We will consider
the dot product here, and the cross product in the next section.


\lab{Interior Point I}{Interior Point I}
\objective{Learn About Interior Point Methods for Linear Constrained Optimization.}

\section*{Interior Point Methods: Overview}
Although the Simplex algorithm was long the only practically competitive method for linear programming, the past 30 years have seen the
discovery and widespread adoption of a new family of algorithms that rival and in some cases outperform the Simplex algorithm, collectively
called Interior Point methods. One of the major shortcomings of the Simplex algorithm is that the number of steps required to solve the
problem can grow exponentially in the size of the linear system. Thus, for certain large linear programs, the Simplex algorithm is simply
not viable. Interior Point methods offer and alternative approach and guarantee much better theoretical convergence properties.

Recall that a linear program is a constrained optimization problem with a linear objective function and linear constraints.
The linear constraints define a set of allowable points called the \emph{feasible region}, the boundary of which forms a geometric
object known as a \emph{polytope}. The theory of convex optimization ensures that the optimal point for the objective function
can be found among the vertices of the feasible polytope. The Simplex Method tests a sequence of such vertices until it finds
the optimal point. Provided the linear program is neither unbounded nor infeasible, the algorithm is certain to produce the correct
answer after a finite number of steps, but it does not guarantee an efficient path along the polytope toward the minimizer. Interior
point methods do away with the feasible polytope and instead generate a sequence of points that cut through the interior (or
exterior) of the feasible region and converge iteratively to the optimal point. Although it is computationally more expensive to
compute such interior points, each step results in significant progress toward the minimizer. See Figure \ref{fig:intPath} for
a visualization of an Interior Point method. In general, the Simplex Method requires many more, albeit less expensive, computations.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{interiorPath.pdf}
\caption{A path traced by an Interior Point algorithm.}
\label{fig:intPath}
\end{figure}

\section*{Primal-Dual Interior Point Methods}
Some of the most popular and successful types of Interior Point methods nowadays are known as Primal-Dual Interior Point methods. To
describe this approach, let us consider the following linear program:
\begin{align*}
\text{minimize }\qquad &c^Tx\\
\text{subject to }\qquad &Ax = b\\
&x \geq 0.
\end{align*}
Here, $x, c \in \mathbb{R}^n$, $b \in \mathbb{R}^m$, and $A$ is an $m \times n$ matrix with full row rank. By $x \geq 0$, we
simply mean that each coordinate of $x$ is nonnegative. Note that this formulation is quite general, as any linear program can be
posed in this manner, after appropriate transformations. This is the primal problem, and its dual takes the form
\begin{align*}
\text{maximize }\qquad &b^T\lambda\\
\text{subject to }\qquad &A^T\lambda + s = c\\
&s \geq 0,
\end{align*}
where $\lambda \in \mathbb{R}^m$ and $s \in \mathbb{R}^n$.

The theory of convex optimization gives us necessary and sufficient
conditions for the solutions to the primal and dual problems via the Karush-Kuhn-Tucker (KKT) conditions.
These conditions are derived as follows. First, we calculate the so-called \emph{Lagrangian function} for our linear program.
The Lagrangian function for a general constrained optimization problem of the form
\begin{align*}
\text{minimize }\qquad &f(x)\\
\text{subject to }\qquad &c_i(x) = 0, \quad i = 1,2,\ldots,k,\\
&d_i(x) \geq 0, \quad i = 1,2,\ldots,j,
\end{align*}
is defined to be
\[
\mathcal{L}(x, \lambda, s) := f(x) - \displaystyle\sum_{i=1}^k\lambda_ic_i(x) - \displaystyle\sum_{i=1}^js_id_i(x).
\]
From here, the KKT conditions are
\begin{align*}
\nabla_x\mathcal{L}(x,\lambda,s) &= 0,\\
c_i(x) &= 0,\quad i = 1,2,\ldots,k,\\
s_id_i(x) &= 0,\quad i=1,2,\ldots,j,\\
d_i(x) &\geq 0,\quad i=1,2,\ldots,j,\\
s_i &\geq 0,\quad i=1,2,\ldots,j.
\end{align*}
For our linear program, $f(x) = c^Tx$, $c_i(x) = A_i^Tx - b_i$ (where $A_i$ denotes the $i$-th row of $A$), and
$d_i(x) = x_i$. Hence, you can check that our Lagrangian function is
\[
\mathcal{L}(x,\lambda,s) = c^Tx - \lambda^T(Ax - b) - s^Tx.
\]
We then calculate
\[
\nabla_x\mathcal{L}(x,\lambda,s) = (c^T - \lambda^TA - s^T)^T = c - A^T\lambda - s.
\]
The KKT conditions can now be expressed as follows:
\begin{align*}
A^T\lambda + s &= c\\
Ax &= b\\
x_is_i &= 0, \quad i = 1,2,\ldots,n,\\
x, s &\geq 0.
\end{align*}
In fact, it is convenient to write these conditions in a more compact manner, by defining an almost-linear function
$F$ and setting it equal to zero:
\begin{align*}
F(x,\lambda,s) :=
\begin{bmatrix}
A^T\lambda + s - c\\
Ax - b\\
XSe\\
\end{bmatrix}
&= 0,\\
(x,s) &\geq 0,
\end{align*}
where $X = \text{diag}(x_1,x_2,\ldots,x_n)$, $S = \text{diag}(s_1,s_2,\ldots,s_n)$, and $e = (1,1,\ldots,1)^T$.

A Primal-Dual Interior Point method is a line search method that starts with an initial guess $(x_0, \lambda_0, s_0)$
and produces a sequence of points that converge to $(x^*, \lambda^*, s^*)$, the solution to the KKT equations and hence
the solution to the original linear program.

\subsection*{Search Direction and Step Length}
We now describe how to select the search direction and step length at each iteration of the algorithm. Because this is
a constrained problem, matters are more complicated than is usual in unconstrained line search methods.

In the spirit of
Newton's Method, we can form a linear approximation of the system $F(x,\lambda,s) = 0$ centered around our current
point $(x, \lambda, s)$, and
calculate the direction $(\triangle x, \triangle \lambda, \triangle s)$ in which to step to set the linear approximation
equal to 0. This equates to solving the linear system
\[
J(x,\lambda,s)
\begin{bmatrix}
\triangle x\\
\triangle \lambda\\
\triangle s\\
\end{bmatrix}
= - F(x,\lambda,s).
\]
Here $J(x,\lambda,s)$ denotes the Jacobian matrix of $F$. We can calculate this matrix block-wise by obtaining the
partial derivatives of each block entry of $F(x,\lambda, s)$ with respect to $x$, $\lambda$, and $s$, respectively.
For example, consider the topmost block entry of $F(x,\lambda,s)$, given by
\[
A^T\lambda + s - c.
\]
We calculate the partial derivatives of this expressions as follows:
\begin{align*}
\frac{\partial}{\partial x}[A^T\lambda + s - c] &= 0,\\
\frac{\partial}{\partial \lambda}[A^T\lambda + s - c] &= A^T,\\
\frac{\partial}{\partial s}[A^T\lambda + s - c] &= I.\\
\end{align*}
Hence, the topmost block row of $J(x,\lambda,s)$ is given by
\[
\begin{bmatrix}
0 \quad A^T \quad I\\
\end{bmatrix}.
\]

Continuing in this manner, we can calculate the entire Jacobian, which leads us to the following linear system:
\begin{equation}
\begin{bmatrix}
0 & A^T & I\\
A & 0 & 0\\
S & 0 & X
\end{bmatrix}
\begin{bmatrix}
\triangle x\\
\triangle \lambda\\
\triangle s
\end{bmatrix}
=
\begin{bmatrix}
-r_c\\
-r_b\\
-XSe
\end{bmatrix},
\label{eq:affine}
\end{equation}
where $r_b = Ax - b$ and $r_c = A^T\lambda + s - c$.
This Newton direction is too greedy, however, and even small steps in this direction may cause us to violate the nonnegativity
condition (the last line of the KKT conditions). We need to find a new search direction.

Choosing an appropriate search direction is a tricky task, and various approaches exist. We will follow a popular strategy
known as the \emph{Predictor-Corrector Algorithm}. The idea is to actually calculate two directions. We first calculate
the standard Newton direction described above; that is, we obtain the solution to Equation \ref{eq:affine}. This is known as
the \emph{predictor step}, since it gives us a direction in which the objective does indeed decrease, hence predicting our
final search direction. Denote the solution to this system by $(\triangle x^a, \triangle \lambda^a, \triangle s^a)$.
As discussed, however, we must deviate from this direction somewhat, and so we additionally solve the
following linear system of equations:
\begin{equation}
\begin{bmatrix}
0 & A^T & I\\
A & 0 & 0\\
S & 0 & X
\end{bmatrix}
\begin{bmatrix}
\triangle x\\
\triangle \lambda\\
\triangle s
\end{bmatrix}
=
\begin{bmatrix}
-r_c\\
-r_b\\
-XSe - \triangle X^a\triangle S^ae + \sigma \mu e
\end{bmatrix}.
\label{eq:centering}
\end{equation}
The solution to this system, which we denote by $(\triangle x, \triangle \lambda, \triangle s)$, is our final search direction.

We now describe the new pieces of Equation \ref{eq:centering}.
Note that
\begin{align*}
\triangle X^a &= \text{diag}(\triangle x_1^a,\ldots,\triangle x_n^a),\\
\triangle S^a &= \text{diag}(\triangle s_1^a,\ldots,\triangle s_n^a).
\end{align*}
Further,
define $\mu := x^Ts/n$. This quantity, called the \emph{duality measure}, tells us roughly how close we are
to the optimal solution, with values closer to 0 being more desirable. It is present in Equation \ref{eq:centering}
for the purpose of obtaining a search direction that leads to a decrease towards 0 in the duality measure.

The formula for $\sigma$ is somewhat more complicated, and
is based on a heuristic approach. First, make the following calculations (which give the maximum allowable step lengths in
the Newton search direction for $x$ and $s$ before the nonnegativity condition is violated):
\begin{align*}
\alpha_a^p &:= \min\left(1, \displaystyle\min_{i : \triangle x_i^a < 0}-\frac{x_i}{\triangle x_i^a}\right)\\
\alpha_a^d &:= \min\left(1, \displaystyle\min_{i : \triangle s_i^a < 0}-\frac{s_i}{\triangle s_i^a}\right).
\end{align*}
Next, define
\[
\mu_a := \frac{1}{n}(x+\alpha_a^p\triangle x^a)^T(s+\alpha_a^d\triangle s^a),
\]
which is simply the duality measure for the point obtained by taking a full step in the Newton direction.
Finally, calculate $\sigma$ by the formula
\[
\sigma = \left(\frac{\mu_a}{\mu}\right)^3.
\]

Now that we have our search direction, it remains to choose our step length. We wish to step nearly as far as possible without
violating the nonnegativity condition, thus remaining in the interior of the feasible region. First, make the following
calculations, which give the maximum allowable step lengths for $x$ and $s$, respectively:
\begin{align*}
\beta^p &:= \displaystyle\min_{i : \triangle x_i < 0}-\frac{x_i}{\triangle x_i}\\
\beta^d &:= \displaystyle\min_{i : \triangle s_i < 0}-\frac{s_i}{\triangle s_i}.
\end{align*}
Next, we back off from these maximum step lengths slightly:
\begin{align*}
\alpha^p &:= \min(1, 0.95\beta^p)\\
\alpha^d &:= \min(1, 0.95\beta^d).
\end{align*}
These are our final step lengths. That is, our next point $(x', \lambda', s')$ is given by
\begin{align*}
x' &= x + \alpha^p\triangle x\\
(\lambda', s') &= (\lambda, s) + \alpha^d(\triangle \lambda, \triangle s).
\end{align*}
We summarize the entire procedure in Algorithm \ref{alg:predcorr}.
\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Predictor-Corrector Algorithm}{}
    \State \textrm{Choose initial point } $(x_0, \lambda_0, s_0)$.
    \For{$k = 0, 1, 2, \ldots$}
        \State \textrm{Solve for } $(\triangle x^a, \triangle \lambda^a, \triangle s^a)$.
        \State \textrm{Calculate } $\alpha_a^p, \alpha_a^d, \mu_a$, \textrm{and} $\sigma$.
        \State \textrm{Solve for } $(\triangle x, \triangle \lambda, \triangle s)$.
        \State \textrm{Calculate the step lengths } $\alpha^p, \alpha^d$.
        \State $x_{k+1} = x_k + \alpha^p\triangle x$,\\
        $\qquad\quad(\lambda_{k+1}, s_{k+1}) = (\lambda_k, s_k) + \alpha^d(\triangle \lambda, \triangle s)$.
    \EndFor
\EndProcedure
\end{algorithmic}
\caption{Predictor-Corrector Algorithm}
\label{alg:predcorr}
\end{algorithm}

A few notes on the implementation of this algorithm are in order.
In each iteration, by far the most expensive operations are solving Equations \ref{eq:affine} and
\ref{eq:centering}. Fortunately, the only difference between the two equations is the right-hand side.
Thus, we can avoid repetitious calculation by first factorizing the block matrix
\[
\begin{bmatrix}
0 & A^T & I\\
A & 0 & 0\\
S & 0 & X
\end{bmatrix}
\]
once at the beginning of the iteration, and then using the factorization twice to solve both equations.
For convenience, consider using the functions \li{lu_factor} and \li{lu_solve} in the \li{scipy.linalg}
module. It is possible to speed up these calculations further, but for simplicity we won't pursue the issue
further.

Next, as mentioned above, the duality measure $\mu$ tells us in some sense how close our current point is to the minimizer.
The closer $\mu$ is to 0, the closer we are to the optimal point. Thus, by printing the value of $\mu$ at
each iteration, you can track how your algorithm is progressing and detect when you have converged.

Another potentially tricky calculation that comes up in each iteration has the following form:
\[
\min\left(1, \displaystyle\min_{i : u_i < 0}-\frac{v_i}{u_i}\right),
\]
where $\mathbf{u} = (u_1, \ldots, u_n)^T$ and $\mathbf{v} = (v_1, \ldots, v_n)^T$ are vectors.
This can be done in a vectorized fashion in Python as follows (assuming that the arrays \li{u} and \li{v}
have already been initialized):
\begin{lstlisting}
>>> mask = u < 0
>>> if np.any(mask):
>>>     myMin = min(1, (-v/u)[mask].min())
>>> else:
>>>     myMin = 1
\end{lstlisting}
We need the if-statement to deal with the case where no entry of $u$ is negative.

Finally, the choice of initial point $(x_0, \lambda_0, s_0)$ is an important, nontrivial one.
A naively or randomly chosen initial point may cause the algorithm to fail to converge.
We provide code to calculate an appropriate initial point. You should include a call to this function in your algorithm to obtain the initial point.

\begin{lstlisting}
def startingPoint(A, b, c):
    """
    Calculate an initial guess to the solution of the
    linear program min c^T x, Ax = b, x>=0.
    Inputs:
        A -- array of shape (m,n) with linearly independent rows
        b -- array of length m
        c -- array of length n
    Returns:
        x -- array of length n
        lam -- array of length m
        s -- array of length n
    Ref: Nocedal and Wright, p. 410
    """
    # first calculate x, lam, s of minimal norm satisfying the primal and dual constraints
    B = la.inv(A.dot(A.T))
    x = A.T.dot(B.dot(b))
    lam = B.dot(A.dot(c))
    s = c - A.T.dot(lam)

    # perturb x and s so they are nonnegative
    dx = max((-3./2)*x.min(), 0)
    ds = max((-3./2)*s.min(), 0)
    x += dx*np.ones(x.shape)
    s += ds*np.ones(s.shape)

    # perturb x and s so they are not too close to zero, not too dissimilar
    dx = .5*(x*s).sum()/s.sum()
    ds = .5*(x*s).sum()/x.sum()
    x += dx*np.ones(x.shape)
    s += ds*np.ones(s.shape)

    return x, lam, s
\end{lstlisting}

\begin{problem}
Write a function \li{interiorPoint} that implements the interior point method described above.
The function should accept $A$, $b$, and $c$ as parameters, along with keyword arguments
\li{niter} that gives the number of iterations to perform, and \li{verbose} that indicates
whether to print out the value of the objective function and $\mu$ at each iteration.
The function should return the optimal point $x$ along with the value of the objective function at this
point.
\end{problem}
\section*{Least Absolute Deviations}
We now return to the familiar problem of fitting a line (or hyperplane) to a set of data. We have previously approached this
problem by minimizing the sum of the squares of the errors between the data points and the line, an approach known as \emph{least
squares}. The least squares solution can be obtained analytically when fitting a linear function, or through a number of optimization
methods (such as Conjugate Gradient) when fitting a nonlinear function.

The method of least absolute deviations also seeks to find a best fit line to a set of data, but the error between the data and
the line is measured differently. In particular, suppose we have a set of data points $(y_1, \mathbf{x}_1), (y_2, \mathbf{x}_2), \ldots,
(y_m, \mathbf{x}_m)$, where $y_i \in \mathbb{R}$, $\mathbf{x}_i \in \mathbb{R}^n$ for $i = 1, 2, \ldots, m$. Here, the $\mathbf{x}_i$ vectors
are the explanatory variables and the $y_i$ values are the response variables, and we assume the following linear model:
\[
y_i = \beta^T\mathbf{x}_i + b, \qquad i = 1, 2, \ldots, m,
\]
where $\beta\in\mathbb{R}^n$ and $b \in \mathbb{R}$. The error between the data and the proposed linear model is given by
\[
\sum_{i=1}^n |\beta^T\mathbf{x}_i + b - y_i|,
\]
and we seek to choose the parameters $\beta, b$ so as to minimize this error.

Before we explore how to solve this problem, a discussion of the differences between least absolute deviations and least squares is in order.
The most prominent difference between these two approaches is how they respond to outliers in the data. Least absolute deviations is
robust in the presence of outliers, meaning that one (or a few) errant data points won't severely affect the fitted line. Indeed, in most cases,
the best fit line is guaranteed to pass through at least two of the data points.
This is a desirable property when the outliers may be ignored (perhaps because they are due to measurement error or corrupted data).
Least squares, on the other hand,
is much more sensitive to outliers, and so is the better choice when outliers cannot be dismissed. See Figure \ref{fig:leastAbsDev}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{leastAbsDev.pdf}
\caption{Fitted lines produced by least absolute deviations (top) and least squares (bottom). The presence of an outlier accounts for the
stark difference between the two lines.}
\label{fig:leastAbsDev}
\end{figure}

While least absolute deviations is robust with respect to outliers, small horizontal perturbations of the data points can lead to very different
fitted lines. Hence, the least absolute deviations solution is less stable than the least squares solution. In some cases there are even infinitely
many lines that minimize the least absolute deviations error term. However, one can expect a unique solution in most cases.

The least absolute deviations solution arises naturally when we assume that the residual terms $\beta^T\mathbf{x}_i + b - y_i$ have a particular
statistical distribution (the Laplace distribution). Ultimately, however, the choice between least absolute deviations and least squares depends on the
nature of the data at hand, as well as your own good judgment.

\subsection{Least Absolute Deviations as a Linear Program}
We can formulate the least absolute deviations problem as a linear program, and then solve it using our interior point method.
For $i = 1, 2, \ldots, m$ we introduce the artificial variable $u_i$ to take the place of the error term $|\beta^T\mathbf{x}_i + b - y_i|$,
and we require this variable to satisfy $u_i \geq |\beta^T\mathbf{x}_i + b - y_i|$. This constraint is not yet linear, but we can split it into
an equivalent set of two linear constraints:
\begin{align*}
u_i &\geq \beta^T\mathbf{x}_i + b - y_i,\\
u_i &\geq y_i - \beta^T\mathbf{x}_i - b.
\end{align*}
Clearly, the $u_i$ are implicitly constrained to be nonnegative.

Our linear program can now be stated as follows:
\begin{align*}
\text{minimize }\qquad &\sum_{i=1}^m u_i\\
\text{subject to }\qquad &u_i \geq \beta^T\mathbf{x}_i + b - y_i,\\
&u_i \geq y_i - \beta^T\mathbf{x}_i - b.
\end{align*}
This is not yet in the form that we need in order to use our interior point method.
For each inequality constraint, we bring all variables ($u_i, \beta, b$) to the left hand side and
introduce a nonnegative slack variable to transform the constraint into an equality:
\begin{align*}
u_i  - \beta^T\mathbf{x}_i - b - s_{2i-1}&= -y_i,\\
u_i +\beta^T\mathbf{x}_i + b - s_{2i}&= y_i,\\
s_{2i-1}, s_{2i}&\geq 0.
\end{align*}

Notice that the variables $\beta, b$ are not assumed to be nonnegative, but in our interior point method, all variables are assumed
to be nonnegative. We can fix this situation by writing these variables as the difference of nonnegative variables:
\begin{align*}
  \beta &= \beta_1 - \beta_2,\\
  b &= b_1 - b_2,\\
  &\beta_1, \beta_2, b_1, b_2 \geq 0.
\end{align*}
Substituting these values into our constraints, we have the following system of constraints:
\begin{align*}
u_i  - \beta_1^T\mathbf{x}_i + \beta_2^T\mathbf{x}_i - b_1 + b_2 - s_{2i-1}&= -y_i,\\
u_i + \beta_1^T\mathbf{x}_i - \beta_2^T\mathbf{x}_i + b_1 - b_2 - s_{2i}&= y_i,\\
u_i, \beta_1, \beta_2, b_1, b_2, s_{2i-1}, s_{2i}&\geq 0.
\end{align*}
Writing $\mathbf{y} = (-y_1, y_1, -y_2, y_2, \ldots, -y_m, y_m)^T$ and $\beta_i = (\beta_{i,1}, \ldots, \beta_{i,n})^T$ for $i = 1, 2$,
we can aggregate all of our variables into one vector as follows:
\[
\mathbf{v} = (u_1,\ldots, u_m, \beta_{1,1},\ldots, \beta_{1,n}, \beta_{2,1},\ldots, \beta_{2,n}, b_1, b_2, s_1,\ldots,s_{2m})^T.
\]
Defining $\mathbf{c} = (1, 1, \ldots, 1, 0, \ldots, 0)^T$ (where only the first $m$ entries are equal to 1), we can write our
objective function as
\[
\sum_{i=1}^m u_i = \mathbf{c}^T\mathbf{v}.
\]
Hence, the final form of our linear program is
\begin{align*}
  \text{minimize }\qquad &\mathbf{c}^T\mathbf{v}\\
  \text{subject to }\qquad A\mathbf{v} &= \mathbf{y},\\
  \mathbf{v} &\geq 0,
\end{align*}
where $A$ is a matrix containing the coefficients of the constraints.
Our constraints are now equalities, and the variables are all nonnegative, so we are ready to use our interior point method to obtain
the solution.

Let's work through an example. We start with an array \li{data}, each row of which consists of the values $y_i, x_{i,1},\ldots,x_{i,n}$,
where $\mathbf{x}_i = (x_{i,1}, x_{i,2}, \ldots, x_{i,n})^T$. We will have $3m + 2(n+1)$ variables in our linear program. Below, we initialize
the vectors $\mathbf{c}$ and $\mathbf{y}$.
\begin{lstlisting}
>>> m = data.shape[0]
>>> n = data.shape[1] - 1
>>> c = np.zeros(3*m + 2*(n + 1))
>>> c[:m] = 1
>>> y = np.empty(2*m)
>>> y[::2] = -data[:, 0]
>>> y[1::2] = data[:, 0]
>>> x = data[:, 1:]
\end{lstlisting}

The hardest part is initializing the constraint matrix correctly. It has $2m$ rows and $3m + 2(n+1)$ columns. Try writing out the
constraint matrix by hand for small $m, n$, and make sure you understand why the code below is correct.
\begin{lstlisting}
>>> A = np.ones((2*m, 3*m + 2*(n + 1)))
>>> A[::2, :m] = np.eye(m)
>>> A[1::2, :m] = np.eye(m)
>>> A[::2, m:m+n] = -x
>>> A[1::2, m:m+n] = x
>>> A[::2, m+n:m+2*n] = x
>>> A[1::2, m+n:m+2*n] = -x
>>> A[::2, m+2*n] = -1
>>> A[1::2, m+2*n+1] = -1
>>> A[:, m+2*n+2:] = -np.eye(2*m, 2*m)
\end{lstlisting}

Now we can calculate the solution by calling our interior point function.
\begin{lstlisting}
>>> sol = interiorPoint(A, y, c, niter=10, verbose=False)[0]
\end{lstlisting}

The variable \li{sol}, however, holds the value for the vector
\[
\mathbf{v} = (u_1,\ldots, u_m, \beta_{1,1},\ldots, \beta_{1,n}, \beta_{2,1},\ldots, \beta_{2,n}, b_1, b_2, s_1,\ldots,s_{2m+1})^T.
\]
We want the values of $\beta = \beta_1-\beta_2$ and $b = b_1 - b_2$, and we accomplish this by the
following code:
\begin{lstlisting}
>>> beta = sol[m:m+n] - sol[m+n:m+2*n]
>>> b = sol[m+2*n] - sol[m+2*n+1]
\end{lstlisting}

\begin{problem}
The file \li{simdata.txt} contains two columns of data. The first gives the values of the response
variables, and the second column gives the values of the explanatory variables. Find the least absolute
deviations line for this data set, and plot it together with the data.
\end{problem}