\section{Planes}
\label{sec:planes}

\begin{outcome}
  \begin{enumerate}
  \item Find the vector and parametric equations of a plane in $\R^n$.
  \item Find the normal and standard equations of a plane in $\R^3$.
  \item Find the intersection of two planes, or of a line and a plane.
  \item Find the angle between two planes, or between a line and a plane.
  \item Find the shortest distance between a point and a plane.
  \end{enumerate}
\end{outcome}

Much like the above discussion with lines, vectors can be used to
determine planes in $\R^n$. Consider a point $P$ and two direction
vectors $\vect{d}$ and $\vect{e}$ that are not parallel to each
other. Then there is a unique plane passing through $P$ and containing
$\vect{d}$ and $\vect{e}$:
\begin{center}
  \begin{tikzpicture}[rotate=-10]
    \filldraw[draw=red!80,fill=red!10](-2,0,-5) -- (4,0,-5) -- (4,0,2) -- (-2,0,2) -- cycle;
    \draw[->,thick,blue!80!black](0,0,0) -- node[below left] {$\vect{d}$} (2,0,0);
    \draw[->,thick,blue!80!black](0,0,0) -- node[above left] {$\vect{e}$} (0,0,-3);
    \fill (0,0) circle [radius=2.2pt] node [left=3pt] {$P$};
  \end{tikzpicture}
\end{center}
The plane is infinite in each direction, although in the picture, we
have only shown a small part of it. If $\vect{p}$ is the position
vector of $P$ and $\vect{q}$ is the position vector of some other
point in the plane, we have
\begin{equation*}
  \vect{q} = \vect{p} + t\,\vect{d} + s\,\vect{e}
\end{equation*}
for some real numbers $t$ and $s$. This is called the \textbf{vector
  equation}%
\index{vector equation!of a plane}%
\index{plane!vector equation} of the plane.

\begin{definition}{Vector equation of a plane}{vector-equation-of-plane}
  Let $\vect{p}$ be a vector and let $\vect{d},\vect{e}$ be non-zero,
  non-parallel vectors. Then
  \begin{equation*}
    \vect{q} = \vect{p} + t\,\vect{d} + s\,\vect{e}
  \end{equation*}
  is the \textbf{vector equation}%
  \index{vector equation!of a plane}%
  \index{plane!vector equation} of a plane.
\end{definition}

The vector equation of a plane can also be written in
\textbf{component form}%
\index{vector equation!of a plane!component form}%
\index{component form!plane}%
\index{plane!component form}
\begin{equation*}
  \begin{mymatrix}{c} x_1 \\ x_2 \\ \vdots \\ x_n \end{mymatrix}
  = \begin{mymatrix}{c} p_1 \\ p_2 \\ \vdots \\ p_n \end{mymatrix}
  + t \begin{mymatrix}{r} d_1 \\ d_2 \\ \vdots \\ d_n \end{mymatrix},
  + s \begin{mymatrix}{r} e_1 \\ e_2 \\ \vdots \\ e_n \end{mymatrix}
\end{equation*}
and in \textbf{parametric form}
\begin{equation*}
  \begin{array}{c@{~}c@{~}c}
    x_1 &=& p_1 + t\,d_1 + s\,e_1, \\
    x_2 &=& p_2 + t\,d_2 + s\,e_2, \\
        &\vdots&             \\
    x_n &=& p_n + t\,d_n + s\,e_n.
  \end{array}
\end{equation*}
The latter set of equations are also called the \textbf{parametric
  equations}%
\index{parametric equations!of a plane}%
\index{plane!parametric equations} of the plane.

\begin{example}{Vector and parametric equations}{plane-from-three-points}
  Find vector and parametric equations for the plane through the
  points $P = (1,2,0,0)$, $Q = (2,2,0,1)$, and $R = (0,1,1,0)$.
\end{example}

\begin{solution}
  We can use $P$ as the base point and $\longvect{PQ}$ and
  $\longvect{PR}$ as the direction vectors. We have
  \begin{equation*}
    \longvect{PQ} =
    \begin{mymatrix}{c} 2\\2\\0\\1 \end{mymatrix}
    - \begin{mymatrix}{c} 1\\2\\0\\0 \end{mymatrix}
    = \begin{mymatrix}{c} 1\\0\\0\\1 \end{mymatrix}
    \quad\mbox{and}\quad
    \longvect{PR} =
    \begin{mymatrix}{c} 0\\1\\1\\0 \end{mymatrix}
    - \begin{mymatrix}{c} 1\\2\\0\\0 \end{mymatrix}
    = \begin{mymatrix}{c} -1\\-1\\1\\0 \end{mymatrix}.
  \end{equation*}
  Therefore the vector equation is
  \begin{equation*}
    \begin{mymatrix}{c} x\\y\\z\\w \end{mymatrix}
    = \begin{mymatrix}{c} 1\\2\\0\\0 \end{mymatrix}
    + t\,\begin{mymatrix}{c} 1\\0\\0\\1 \end{mymatrix}
    + s\,\begin{mymatrix}{c} -1\\-1\\1\\0 \end{mymatrix}.
  \end{equation*}
  We can also write this as a system of parametric equations:
  \begin{equation*}
    \begin{array}{c@{~}c@{~}l}
      x &=& 1 + t - s, \\
      y &=& 2 - s, \\
      z &=& s, \\
      w &=& t.
    \end{array}
  \end{equation*}
\end{solution}

Note that the vector and parametric equations of a plane are not
unique. For example, in Example~\ref{exa:plane-from-three-points}, we
could have equally used $Q$ or $R$ as the base point, and/or used
$\longvect{QR}$ as one of the direction vectors. In each case we would
have obtained a different equation for the same plane.

\begin{example}{Determine whether a point is on a plane}{point-on-plane-parametric}
  Determine whether the point $S=(4,4,-2,1)$ lies on the plane through
  the points $P = (1,2,0,0)$, $R = (2,2,0,1)$, and
  $Q = (0,1,1,0)$.
\end{example}

\begin{solution}
  We already found the parametric equations for this plane in
  Example~\ref{exa:plane-from-three-points}. To determine whether the
  point $S=(4,4,-2,1)$ lies on this plane, we must substitute its
  coordinates into the parametric equations:
  \begin{equation*}
    \begin{array}{r@{~}c@{~}l}
      4 &=& 1 + t - s, \\
      4 &=& 2 - s, \\
      -2 &=& s, \\
      1 &=& t.
    \end{array}
  \end{equation*}
  This is a system of linear equations. We solve it to find that it
  has the unique solution $(t,s) = (1,-2)$. Therefore, the point $S$
  lies on the given plane, and more specifically, it is the point that
  corresponds to the parameters $t=1$ and $s=-2$.
\end{solution}

In the special case of 3 dimensions, a plane can also be described by
a point and a normal vector. A \textbf{normal vector}%
\index{normal vector of a plane}%
\index{vector!normal vector}%
\index{plane!normal vector} of a plane is a vector that is
perpendicular to the plane.
\begin{center}
  \begin{tikzpicture}[rotate=-10]
    \filldraw[draw=red!80,fill=red!10](-3,0,-3.5) -- (3,0,-3.5) -- (3,0,3.5) -- (-3,0,3.5) -- cycle;
    \draw[->,thick,blue!80!black](0,0,0) -- node[left, pos=0.4] {$\vect{n}$} (100:2);
    \draw[dashed,gray](0,0,0) -- (2,0,-2);
    \fill (0,0,0) circle [radius=2.2pt] node [left=3pt] {$P$};
    \fill (2,0,-2) circle [radius=2.2pt] node [right=3pt] {$Q$};
  \end{tikzpicture}
\end{center}
Given a non-zero vector $\vect{n}$ in $\R^3$ and a point $P$, there
exists a unique plane that contains $P$ and has $\vect{n}$ as a normal
vector. We wish to find an equation for this plane. If $Q$ is an
arbitrary point on the plane, then by definition, the normal vector is
orthogonal to the vector $\longvect{PQ}$. Writing this as a formula,
we have $\vect{n} \dotprod \longvect{PQ} = 0$. If $\vect{p}$ and
$\vect{q}$ are the position vectors of $P$ and $Q$, respectively, we
have $\longvect{PQ} = \vect{q}-\vect{p}$, and therefore the equation
of the plane can be written as
\begin{equation*}
  \vect{n} \dotprod (\vect{q}-\vect{p}) = 0,
\end{equation*}
or equivalently,
\begin{equation*}
  \vect{n} \dotprod \vect{q} = \vect{n} \dotprod \vect{p}.
\end{equation*}
This is called the \textbf{normal equation} of the plane. Note that in
this equation, $\vect{n}$ and $\vect{p}$ are given and fixed, whereas
$\vect{q}$ is a variable ranging over the position vectors of all
points on the plane.

\begin{definition}{Normal equation of a plane in $\R^3$}{normal-equation-plane}
  Let $\vect{n}$ be a non-zero vector in $\R^3$, and let $P$ be a
  point with position vector $\vect{p}$. Then there is a unique plane through
  $P$ with normal vector $\vect{n}$. It is described by the equation
  \begin{equation*}
    \vect{n} \dotprod \vect{q} = \vect{n} \dotprod \vect{p}.
  \end{equation*}
  This equation is called the \textbf{normal equation}%
  \index{plane!normal equation}%
  \index{plane!normal equation}%
  \index{normal equation of a plane} of the plane.
\end{definition}

\begin{example}{Finding the normal equation of a plane}{normal-equation}
  Find the normal equation of the plane through the point $P=(1,3,0)$
  and orthogonal to $\vect{n}=\mat{2,1,1}^T$.
\end{example}

\begin{solution}
  Let $\vect{p}=\mat{1,3,0}^T$ be the position vector of $P$, and let
  $\vect{q}=\mat{x,y,z}^T$ be the position vector of some arbitrary
  point $Q$ in the plane. The normal equation is
  $\vect{n} \dotprod \vect{q} = \vect{n} \dotprod \vect{p}$, which we
  can write in component form:
  \begin{equation*}
    \begin{mymatrix}{c}2\\1\\1\end{mymatrix}
    \dotprod
    \begin{mymatrix}{c}x\\y\\z\end{mymatrix}
    =
    \begin{mymatrix}{c}2\\1\\1\end{mymatrix}
    \dotprod
    \begin{mymatrix}{c}1\\3\\0\end{mymatrix}.
  \end{equation*}
  We can pre-compute the dot product on the right-hand side:
  $\vect{n} \dotprod \vect{p} = 1(2)+3(1)=0(1) = 5$. Therefore, the
  normal equation can also be written as
  \begin{equation*}
    \begin{mymatrix}{c}2\\1\\1\end{mymatrix}
    \dotprod
    \begin{mymatrix}{c}x\\y\\z\end{mymatrix}
    = 5.
  \end{equation*}
\end{solution}

Notice that the last equation in Example~\ref{exa:normal-equation} can
also be written in the form
\begin{equation*}
  2x + y + z = 5.
\end{equation*}
This last form is called the \textbf{standard equation} of the plane.

\begin{definition}{Standard equation of a plane in $\R^3$}{standard-equation-plane}
  Let $\vect{n} = \mat{a,b,c}^T$ be the normal vector for a plane that
  contains the point $P = (x_0, y_0, z_0)$. The \textbf{standard
    equation}%
  \index{plane!standard equation}%
  \index{standard equation of a plane} of the plane is given by
  \begin{equation*}
    ax + by + cz = d,
  \end{equation*}
  where $a,b,c,d \in \R$ and $d = ax_0 + by_0 + cz_0$.
\end{definition}

\begin{example}{Normal and standard equations}{normal-from-three-points}
  Find normal and standard equations for the plane through the points
  $P = (0,1,3)$, $Q=(2,-1,0)$, and $R=(1,2,2)$.
\end{example}

\begin{solution}
  We first need to find a normal vector for the plane. Since the
  normal vector must be perpendicular to the plane, it must be
  orthogonal to both $\longvect{PQ}$ and $\longvect{PR}$. We can
  therefore use the cross product to compute a normal vector for the
  plane:
  \begin{equation*}
    \vect{n}
    ~=~
    \longvect{PQ} \times \longvect{PR}
    ~=~
    \begin{mymatrix}{r} 2 \\ -2 \\ -3 \end{mymatrix}
    \times
    \begin{mymatrix}{r} 1 \\ 1 \\ -1 \end{mymatrix}
    ~=~
    \begin{mymatrix}{r} 5 \\ -1 \\ 4 \end{mymatrix}.
  \end{equation*}
  \begin{center}
    \begin{tikzpicture}[rotate=-10]
      \filldraw[draw=red!80,fill=red!10](-3,0,-3.5) -- (3,0,-3.5) -- (3,0,3.5) -- (-3,0,3.5) -- cycle;
      \draw[->,thick,blue!80!black](0,0,0) -- node[left, pos=0.4] {$\vect{n}$} (100:2);
      \draw[->,thick,blue!80!black](0,0,0) -- (2,0,1);
      \draw[->,thick,blue!80!black](0,0,0) -- (1,0,-2);
      \fill (0,0,0) circle [radius=2.2pt] node [left=3pt] {$P$};
      \fill (2,0,1) circle [radius=2.2pt] node [right=3pt] {$Q$};
      \fill (1,0,-2) circle [radius=2.2pt] node [right=3pt] {$R$};
    \end{tikzpicture}
  \end{center}
  Now we can easily obtain the normal equation from any point on the
  plane (say $P$) and the normal vector we just calculated:
  \begin{equation*}
    \begin{mymatrix}{r} 5 \\ -1 \\ 4 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{r} x \\ y \\ z \end{mymatrix}
    =
    \begin{mymatrix}{r} 5 \\ -1 \\ 4 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{r} 0 \\ 1 \\ 3 \end{mymatrix}
  \end{equation*}
  We get the standard equation by computing the dot products on the
  left- and right-hand sides:
  \begin{equation*}
    5x - y + 4z = 11.
  \end{equation*}
  It is worthwhile to double-check the answer by substituting each of
  the three original points $P$, $Q$, and $R$ into this equation.
  For example, for $Q=(2,-1,0)$, we obtain $5(2)-(-1)+4(0)$, which is
  indeed $11$.
\end{solution}

\begin{example}{Find the normal vector of a plane}{find-normal}
  Find a normal vector for the plane $2x+3y-z=7$.
\end{example}

\begin{solution}
  The standard equation $2x+3y-z=7$ can be rewritten as a normal
  equation
  \begin{equation*}
    \begin{mymatrix}{r} 2\\3\\-1 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{c} x\\y\\z \end{mymatrix}
    = 7.
  \end{equation*}
  Therefore,
  \begin{equation*}
    \vect{n} = \begin{mymatrix}{r} 2\\3\\-1 \end{mymatrix}
  \end{equation*}
  is a normal vector for the plane.
\end{solution}

\begin{example}{Determine whether a point is on a plane}{point-plane-normal}
  Let $\vect{n} = \mat{1,2,3}^T$ be the normal vector for a plane
  which contains the point $P = (2,1,4)$. Determine if the point
  $Q = (5,4,1)$ is in this plane.
\end{example}

\begin{solution}
  By Definition~\ref{def:normal-equation-plane}, $Q$ is a point in the
  plane if and only if
  \begin{equation*}
    \vect{n} \dotprod \vect{q} = \vect{n} \dotprod \vect{p},
  \end{equation*}
  where $\vect{p}$ and $\vect{q}$ are the position vectors of $P$ and
  $Q$, respectively.  Given $\vect{n}$, $P$, and $Q$ as above, this
  equation becomes
  \begin{equation*}
    \begin{mymatrix}{r} 1 \\ 2 \\ 3 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{r} 5 \\ 4 \\ 1 \end{mymatrix}
    ~=~
    \begin{mymatrix}{r} 1 \\ 2 \\ 3 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{r} 2 \\ 1 \\ 4 \end{mymatrix}.
  \end{equation*}
Since both sides of the equation are equal to $16$, the equation is
  true. So the point $Q$ is indeed in the plane determined by
  $\vect{n}$ and $P$.
\end{solution}

\begin{example}{Vector equation from normal equation}{vector-from-normal}
  Find a vector equation for the plane $x+3y-2z=7$.
\end{example}

\begin{solution}
  This is the same thing as finding the general solution of a system
  of one linear equation in 3 variables. Since there is only a single
  equation $x+3y-2z=7$, it is already in {\ef}. The variables
  $y$ and $z$ are free, so we set them equal to parameters: $z=t$ and
  $y=s$. The variable $x$ is a pivot variable, and we get
  $x=7+2t-3s$. So the general solution of the equation is
  \begin{equation*}
    \begin{mymatrix}{c} x\\y\\z \end{mymatrix}
    = \begin{mymatrix}{r} 7\\0\\0 \end{mymatrix}
    + t\,\begin{mymatrix}{r} 2\\0\\1 \end{mymatrix}
    + s\,\begin{mymatrix}{r} -3\\1\\0 \end{mymatrix}.
  \end{equation*}
  This is also a vector equation for the plane.
\end{solution}

\begin{example}{Intersection of two planes}{intersection-planes}
  Find the intersection of the planes $x-2y+z=0$ and
  $2x-3y-z=4$.%
  \index{intersection!of two planes}
\end{example}

\begin{solution}
  Finding the intersection means finding all of the points $(x,y,z)$
  that are on both planes simultaneously. This is the same as solving
  the system of equations
  \begin{eqnarray*}
    x-2y+z &=& 0, \\
    2x-3y-z &=& 4.
  \end{eqnarray*}
  We solve the system by Gauss-Jordan elimination:
  \begin{equation*}
    \begin{mymatrix}{ccc|c}
      1 & -2 & 1 & 0 \\
      2 & -3 & -1 & 4
    \end{mymatrix}
    \stackrel{R_2\rowop R_2-2R_1}{\roweq}
    \begin{mymatrix}{ccc|c}
      1 & -2 & 1 & 0 \\
      0 & 1 & -3 & 4
    \end{mymatrix}
    \stackrel{R_1\rowop R_1+2R_2}{\roweq}
    \begin{mymatrix}{ccc|c}
      1 & 0 & -5 & 8 \\
      0 & 1 & -3 & 4
    \end{mymatrix}.
  \end{equation*}
  Therefore, the general solution is
  \begin{equation*}
    \begin{mymatrix}{c} x\\y\\z \end{mymatrix}
    =
    \begin{mymatrix}{c} 8\\4\\0 \end{mymatrix}
    + t\,\begin{mymatrix}{c} 5\\3\\1 \end{mymatrix},
  \end{equation*}
  where $t$ is a parameter. This is the parametric equation of a
  line. Therefore, the two planes intersect in a line. Specifically,
  the intersection is the line through the point $(8,4,0)$ with
  direction vector $\mat{5,3,1}^T$.
\end{solution}

\begin{example}{Intersection of a line and a plane}{intersection-line-plane}
  Find the intersection of the line
  \begin{equation*}
    \begin{mymatrix}{r} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{r} 1 \\ 2 \\ 0 \end{mymatrix}
    + t \begin{mymatrix}{r} -1 \\ 1 \\ 2 \end{mymatrix}
  \end{equation*}
  and the plane $2x+2y-z = 2$.%
  \index{intersection!of a line and a plane}
\end{example}

\begin{solution}
  Let us write
  \begin{equation*}
    \vect{p}=\begin{mymatrix}{c}1\\2\\0\end{mymatrix},
    \quad
    \vect{d}=\begin{mymatrix}{c}-1\\1\\2\end{mymatrix},
    \quad
    \vect{n}=\begin{mymatrix}{c}2\\2\\-1\end{mymatrix},
    \quad\mbox{and}\quad
    \vect{q}=\begin{mymatrix}{c}x\\y\\z\end{mymatrix}.
  \end{equation*}
  Then the equation of the line is $\vect{q}=\vect{p}+t\,\vect{d}$ and
  the equation of the plane is
  $\vect{n}\dotprod\vect{q}=2$. Substituting the first equation into
  the second one, we get $\vect{n}\dotprod(\vect{p}+t\,\vect{d}) = 2$.
  Using distributivity of the dot product, we can write this last
  equation as
  $\vect{n}\dotprod\vect{p} + t(\vect{n}\dotprod\vect{d}) = 2$.  By
  computing the dot products $\vect{n}\dotprod\vect{p}=6$ and
  $\vect{n}\dotprod\vect{d} = -2$, this equation simplifies to
  $6-2t=2$, or $t=2$. Therefore, the line intersects the plane when
  $t=2$, or at the point
  \begin{equation*}
    \begin{mymatrix}{r} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{r} 1 \\ 2 \\ 0 \end{mymatrix}
    + 2 \begin{mymatrix}{r} -1 \\ 1 \\ 2 \end{mymatrix}
    = \begin{mymatrix}{r} -1 \\ 4 \\ 4 \end{mymatrix}.
  \end{equation*}
  An alternative method is to directly substitute the parametric
  equation of the line, $x=1-t$, $y=2+t$, and $z=2t$, into the
  equation of the plane, $2x+2y-z = 2$. In this case, we get
  $2(1-t)+2(2+t)-(2t)=2$, which we can solve for $t$ to obtain $t=2$.
\end{solution}

The next few examples are concerned with calculating angles between
planes, angles between lines and planes, and finding the distance
between points and planes.

\begin{example}{Find the angle between two planes}{angle-planes}
  Find the angle between the planes $7x-y=5$ and $4x+3y+5z=3$.%
  \index{angle!between two planes}%
  \index{plane!angle between}
\end{example}

\begin{solution}
  The angle between two planes is the same thing as the angle between
  their normal vectors.
  \begin{center}
    \begin{tikzpicture}
      \filldraw[draw=red!80,fill=red!10] (-3,-1,-3) -- (0,0,-3) -- (0,0,3) -- (-3,-1,3) -- cycle;
      \filldraw[draw=red!80,fill=red!15] (-3,1,-3) -- (3,-1,-3) -- (3,-1,3) -- (-3,1,3) -- cycle;
      \filldraw[draw=red!80,fill=red!10] (0,0,-3) -- (3,1,-3) -- (3,1,3) -- (0,0,3) -- cycle;
      \filldraw[fill=green!20,draw=green!50!black] (0,0,3) -- +(-18.4:12mm) arc (-18.4:18.4:12mm) -- cycle;
      \filldraw[fill=green!20,draw=green!50!black] (0,0,0) -- +(71.6:12mm) arc (71.6:108.4:12mm) -- cycle;
      \draw[red!80] (3,1,3) -- (-3,-1,3);
      \draw[red!80] (3,-1,3) -- (-3,1,3);
      \draw[->,thick,blue!80!black](0,0,0) -- node[left, near end] {$\vect{n}_1$} (-1,3,0);
      \draw[->,thick,blue!80!black](0,0,0) -- node[right, near end] {$\vect{n}_2$} (1,3,0);
      \fill (0,0,0) circle [radius=2.2pt];
      \node at (0,0,0) [above=5mm] {$\theta$};
      \node at (0,0,3) [right=5mm] {$\theta$};
    \end{tikzpicture}
  \end{center}
  The normal vectors are $\vect{n}_1 = \mat{7,-1,0}^T$ and $\vect{n}_2
  = \mat{4,3,5}^T$. The angle between them is given by
    \begin{equation*}
    \cos\theta =
    \frac{\vect{n}_1\dotprod\vect{n}_2}{\norm{\vect{n}_1}\norm{\vect{n}_2}}
    = \frac{25}{50} = \frac{1}{2}.
  \end{equation*}
  Therefore, the angle is $\arccos(\frac{1}{2}) = \pi/3$, or 60 degrees.
\end{solution}

\begin{example}{Find the angle between a line and a plane}{angle-line-plane}
  Find the angle between the line
  \begin{equation*}
    \begin{mymatrix}{r} x \\ y \\ z \end{mymatrix}
    = \begin{mymatrix}{r} 1 \\ 2 \\ 0 \end{mymatrix}
    + t \begin{mymatrix}{r} 2 \\ -1 \\ -2 \end{mymatrix}
  \end{equation*}
  and the plane $2x+2y-z = 2$.%
  \index{angle!between line and plane}
\end{example}

\begin{solution}
  To get the angle $\theta$ between the plane and the line, we can
  compute the angle $\phi$ between the direction vector of the line
  and the normal vector of the plane, and then take
  $\theta = \frac{\pi}{2}-\phi$.
  \begin{center}
    \begin{tikzpicture}[rotate=-10]
      \draw[thick, red!80] (50:-3) -- (0,0);
      \filldraw[draw=red!80,fill=red!10](-3,0,-3.5) -- (3,0,-3.5) -- (3,0,3.5) -- (-3,0,3.5) -- cycle;
      \filldraw[fill=yellow!20,draw=yellow!50!black] (0,0) -- (0:12mm) arc (0:50:12mm) -- cycle;
      \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (50:12mm) arc (50:100:12mm) -- cycle;
      \draw[thick, red!80] (0,0) -- (50:3.5);
      \begin{scope}
        \clip (-3,0,-3.5) -- (3,0,-3.5) -- (3,0,3.5) -- (-3,0,3.5) -- cycle;
        \draw[thick, red!30] (50:-3) -- (0,0);
      \end{scope}
      \draw[->,thick,blue!80!black](0,0,0) -- node[left=3pt, pos=0.9] {$\vect{n}$} (100:2);
      \draw[->,thick,blue!80!black](0,0) -- node[above left, pos=0.7] {$\vect{d}$} (50:3);
      \draw[dashed] (0,0,0) -- (2.5,0,0);
      \draw[dashed] (50:3) -- +(100:-2.3);
      \fill (0,0,0) circle [radius=2.2pt];
      \node at (25:8mm) {$\theta$};
      \node at (75:7mm) {$\phi$};
    \end{tikzpicture}
  \end{center}
  The direction vector of the line is $\mat{2,-1,-2}^T$ and the normal
  vector of the plane is $\mat{2,2,-1}$. We have
  \begin{equation*}
    \cos\phi =
    \frac{\vect{n}\dotprod\vect{d}}{\norm{\vect{n}}\norm{\vect{d}}}
    = \frac{4}{9},
  \end{equation*}
  and therefore $\phi = \arccos(\frac{4}{9}) \approx 1.11$ radians. We
  have $\theta = \frac{\pi}{2} - \phi \approx 0.46$ radians, or about
  26.4 degrees.
\end{solution}

\begin{example}{Shortest distance from a point to a plane}{shortest-distance-plane}
  Find the shortest distance from the point $P = (3,2,3)$ to the plane
  given by $2x + y + 2z = 2$, and find the point $Q$ on the plane that
  is closest to $P$.%
  \index{distance!point to plane}%
  \index{projection!point to plane}
\end{example}

\begin{solution}
  In this problem, we are going to use the projection of one vector
  onto another, which was introduced in
  Section~\ref{ssec:projections}.  Pick an arbitrary point $R$ on
  the plane. Then, it follows that
  \begin{equation*}
    \longvect{QP} = \proj_{\vect{n}}\,\longvect{RP}
  \end{equation*}
  and $\norm{\longvect{QP}}$ is the shortest distance from $P$ to the
  plane. Further, the position vector of the point $Q$ can be computed
  as $\vect{q} = \vect{p} - \longvect{QP}$, where $\vect{p}$ is the
  position vector of $P$.
  \begin{center}
    \begin{tikzpicture}[x={(1cm,-0.2cm)},y={(0.5cm,0.5cm)},z={(0cm,1cm)}]
      \filldraw[draw=red!80,fill=red!10](-3,-2,0) -- (4,-2,0) -- (4,2,0) -- (-3,2,0) -- cycle;
      \draw[dashed](0,0,0) -- (2,0,0) -- (2,0,3) -- (0,0,3) -- cycle;
      \draw[->,thick,blue!80!black](0,0,0) -- node[left=3pt] {$\vect{n}$} (0,0,2);
      \draw[->,thick,blue!80!black](0,0,0) -- node[left=3pt, pos=0.7] {$\longvect{RP}$} (2,0,3);
      \draw[->,thick,blue!80!black](2,0,0) -- node[right=3pt, pos=0.7] {$\longvect{QP}$} (2,0,3);
      \fill (0,0,0) circle [radius=2.2pt] node [left=3pt] {$R$};
      \fill (2,0,3) circle [radius=2.2pt] node [right=3pt] {$P$};
      \fill (2,0,0) circle [radius=2.2pt] node [right=3pt] {$Q$};
    \end{tikzpicture}
  \end{center}
  From the above scalar equation, we have that $\vect{n} =
  \begin{mysmallmatrix}{c} 2 \\ 1 \\ 2 \end{mysmallmatrix}$.  Now, choose any
  point on the plane, for example, $R = (1,0,0)$ (notice that this
  satisfies $2x+y+2z=2$).  Then,
  \begin{equation*}
    \longvect{RP} = \begin{mymatrix}{c} 3 \\ 2 \\ 3 \end{mymatrix}
    - \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix} =
    \begin{mymatrix}{c} 2 \\ 2 \\ 3 \end{mymatrix}.
  \end{equation*}
  Next, compute $\longvect{QP} = \proj_{\vect{n}}\longvect{RP}$.
  \begin{equation*}
    \longvect{QP} ~=~ \proj_{\vect{n}}\longvect{RP}
    ~=~ \paren{\frac{\vect{n}\dotprod\longvect{RP}}{\norm{\vect{n}}^2}}\vect{n}
    ~=~ \frac{12}{9} \begin{mymatrix}{r} 2 \\ 1 \\ 2 \end{mymatrix}
    ~=~ \frac{4}{3} \begin{mymatrix}{r} 2 \\ 1 \\ 2 \end{mymatrix}.
  \end{equation*}
  Then, $\norm{\longvect{QP}} = 4$ so the shortest distance from $P$
  to the plane is $4$.  To find the point $Q$ on the plane that is
  closest to $P$, we have
  \begin{equation*}
    \vect{q} ~=~ \vect{p} - \longvect{QP}
    ~=~ \begin{mymatrix}{r} 3 \\ 2 \\ 3 \end{mymatrix}
    -
    \frac{4}{3} \begin{mymatrix}{r} 2 \\ 1 \\ 2 \end{mymatrix}
    ~=~
    \frac{1}{3}
    \begin{mymatrix}{r} 1 \\ 2 \\ 1 \end{mymatrix}.
  \end{equation*}
  Therefore, $Q = (\frac{1}{3},\frac{2}{3},\frac{1}{3})$.
\end{solution}


\chapter{Fundamental Lemma of Calculus of Variations}\label{app:fundamental-lemma-of-calculus-of-variations}

The \textit{Fundamental Lemma of Calculus of Variations}\footnote{\url{http://en.wikipedia.org/wiki/Fundamental_lemma_of_calculus_of_variations}} says that, for continuous functions, the condition
\begin{align}
  \int_{-\infty}^\infty dx f(x)\eta(x)
=
  0
\text{ for all $\eta(x)$}
\end{align}
holds only when $f(x)=0$ for all $x$.
We can see this by considering the case $\eta(x)=f(x)$.
Since $f(x)^2$ is nonnegative everywhere, the integral yields a positive number whenever $f(x)\neq 0$ on a finite range of $x$ values.


\documentclass[paper.tex]{subfiles}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{multicol}
\usepackage{algpseudocode}
\usepackage{algorithm}

\usepackage{setspace}

\usepackage{pgfplots}
\pgfplotsset{compat=1.16}

% Begin Document
\begin{document}

\section{Experimental Data}

Utilizing the two algorithms outlined above, we set out to test them with some randomized, undirected graphs.
We did this by generating random adjacency matrices.
Each node had a connection to itself for ease of comparsion.
Each node then had a chance to be connected to each other node, which we termed the graph density.
We used a value of $30\%$ for this graph density.

Then for each randomly generated graph, we ran both the brute force and approximation algorithms on it.
We did starting with a graph of 20 nodes, and ran these algorithms on a new randomized graph of incrementally larger size until the runtime exceeded 24 hours.
This happened at the 37-node mark with our implementation and hardware.
The data is below:

\singlespacing

\begin{center}
\begin{tabular}{c | r r | r r}

     & \multicolumn{2}{c |}{\textbf{Minimum Set}} & \multicolumn{2}{c}{\textbf{Approximate Set}} \\
     \textbf{\# of Vertices} & \textbf{Size} & \textbf{Time (seconds)} & \textbf{Size} & \textbf{Time (seconds)} \\ \hline

     20 &         4 &          0.46 &    7 &          8.6 e-6     \\
     21 &         4 &          1.00 &    5 &          6.5 e-6     \\
     22 &         4 &          2.09 &    5 &          6.6 e-6     \\
     23 &         4 &          4.46 &    7 &          9.2 e-6     \\
     24 &         4 &          9.44 &    8 &          1.0 e-5    \\
     25 &         5 &          20.15 &    7 &          9.6 e-6     \\
     26 &         5 &          42.07 &    5 &          8.9 e-6     \\
     27 &         4 &          90.59 &    6 &          1.0 e-5    \\
     28 &         4 &          233.23 &    8 &          1.2 e-5    \\
     29 &         4 &          395.63 &    8 &          1.2 e-5    \\
     30 &         4 &          852.31 &    7 &          1.3 e-5     \\
     31 &         4 &          1,834.03 &    7 &          1.4 e-5    \\
     32 &         4 &          3,869.44 &    6 &          1.4 e-5    \\
     33 &         4 &          8,145.86 &    6 &          1.4 e-5    \\
     34 &         4 &          17,353.50 &    7 &          1.5 e-5    \\
     35 &         5 &          35,943.60 &    10&          1.9 e-5   \\
     36 &         4 &          75,798.40 &    7 &          1.8 e-5  \\
     37 &         4 &          161,439.00  &    8 &          2.0 e-5

\end{tabular}
\end{center}

\onehalfspacing

The doubling of the time needed to run each incrementally larger graph through the brute force algorithm is easily spotted in the data.
While a 37-node graph is not quite double the size of a 20-node graph, the time required ballooned from a half-second to nearly 45 hours.
As the trend will continue due to the nature of the brute force approach, we can extrapolate that a 38-node graph will take approximately 90 hours to compute the minimum set.
Further extrapolation reveals a 50-node graph will take over 42 \textit{years}.

On the other hand, the approximation algorithm ran near instantaneously for each graph, with its growth much lower.
The runtime of the approximation algorithm was measured in microseconds.
As is apparent, the approximation algorithm runs in orders of magnitude faster time than the brute force approach.

The question remains though, just how efficient is the approximation algorithm in terms of the size of the minimum set?
Below is a chart showing the relation between the minimum size and the approximate size for the data recorded above.

\vspace{5mm}

\begin{figure}[H]
\begin{center}
\begin{tikzpicture}
    \begin{axis}
        [
            xlabel = {\# of Vertices},
            ylabel = {Size of Set},
            xmin=19,xmax=38,
            ymin=0,ymax=16,
            xtick={20,25,30,35},
            ytick={2,4,6,8,10}
        ]

        \addplot[mark=*,blue] plot coordinates {
            (20,4)
            (21,4)
            (22,4)
            (23,4)
            (24,4)
            (25,5)
            (26,5)
            (27,4)
            (28,4)
            (29,4)
            (30,4)
            (31,4)
            (32,4)
            (33,4)
            (34,4)
            (35,5)
            (36,4)
            (37,4)
        };
        \addlegendentry{Minimum Set}

        \addplot[mark=*,red] plot coordinates {
            (20,7)
            (21,5)
            (22,5)
            (23,7)
            (24,8)
            (25,7)
            (26,5)
            (27,6)
            (28,8)
            (29,8)
            (30,7)
            (31,7)
            (32,6)
            (33,6)
            (34,7)
            (35,10)
            (36,7)
            (37,8)
        };
        \addlegendentry{Approximate Set}

    \end{axis}
\end{tikzpicture}
\end{center}
\end{figure}

As this chart shows, the approximate solution only found a minimum solution one time.
This indicates that it is entirely possible for the approximate solution to agree with the brute force approach.
However, the chart also clearly shows that the majority of cases in our test data were much larger than the minimal solution, often times double the size.

\end{document}


\chapter{Benchmarks} \label{benchmarks}

\note{
Some programs over numbers, some over lists, some over lists of lists and some over trees (What kind of trees?).
For every program, try to get a sample implementation. \\
Types needed: \lstinline?Int, [a], Tree a? \\
Basic components needed:
arithmetic (\lstinline?+, -, *, /?),
relation (\lstinline?<, <=, ==, /=, >=, >?),
\note{(maybe we do not need relations)}
}

\begin{enumerate}
	\item max of two numbers\\
	(hopefully) the easiest program \\
	\begin{lstlisting}
max :: Int -> Int -> Int
max 0 0 == 0
max 1 0 == 1
max 0 1 == 1
max x y = if x > y then x else y
	\end{lstlisting}
	\note{We don't care about conditionals, we cannot synthesize this.}\\
	This is the only function that requires a conditional branch.
%
	\item square a number
	\begin{lstlisting}
square :: Int -> Int -> Int
square 0 == 0
square 1 == 1
square 2 == 4
square 3 == 9
square x = x * x
	\end{lstlisting}
	That is, basic arithmetic operations like \lstinline!+ - * /! should be provided
%
	\item tetrahedral numbers \\
	\begin{lstlisting}
tetrahedral :: Int -> Int
tetrahedral 1 == 1
tetrahedral 2 == 4
tetrahedral 3 == 10
	\end{lstlisting}
	closed form solution
	\begin{lstlisting}
tetrahedral n = n * (n+1) * (n+2) / 6
	\end{lstlisting}
	iterative solution
	\begin{lstlisting}
tetrahedral n = scanl1 (+) (scanl1 (+) [0..]) !! n
	\end{lstlisting}
	Another iterative solution (without infinite lists)
	\begin{lstlisting}
tetrahedral n = foldl1 (+) (scanl1 (+) (enumFromTo 1 n))
	\end{lstlisting}
	Components needed: \lstinline?scanl1, !!? \\
	Interestingly the iterative version is much faster than the closed form solution
%
	\item prime test \\
	I think this is too difficult
	\begin{lstlisting}
prime :: Int -> Int
prime 1 == 0
prime 2 == 1
prime 3 == 1
prime 4 == 0
prime 25 == 0
prime 29 == 1
prime n = minimum (1 : (map (mod n) (enumFromTo 2 (subtract 1 n))))
	\end{lstlisting}
	Components needed: \lstinline?map, mod, minimum, enumFromTo, subtract?
%
	\item average
	\begin{lstlisting}
average :: [Int] -> Int
average [1] == 1
average [1,3] == 2
average [1,2,3,6] == 6
average xs = (sum xs) `div` (length xs)
	\end{lstlisting}
%
	\item movingAverage (forward)
	\begin{lstlisting}
movingAverage :: Int -> [Int] -> [Int]
movingAverage 1 [1,2,3] == [1,2,3]
movingAverage 2 [1,2,3] == [2,2,3]
movingAverage 3 [3,2,4,1,5,2] == [3,2,3,2,3,2]
movingAverage n xs = map (average . take n) (init $ tails xs)
	\end{lstlisting}
	Components needed: \lstinline?tails? from \lstinline?Data.List? and  \lstinline?average? (one of the benchmarks), as well as \lstinline?map, take? and \lstinline?init? from \lstinline?Prelude?.
%
	\item movingSum (backward)
	\begin{lstlisting}
movingSum :: Int -> [Int] -> [Int]
movingSum 1 [1,2,3] == [1,2,3]
movingSum 2 [1,2,3] == [1,3,5]
movingSum 3 [4,8,6,-1,-2,-3,-1,3,4,5] == [4,12,18,13,3,-6,-6,-1,6,12]
movingSum n xs = scanl1 (+) (zipWith (-) xs (replicate n 0 ++ xs))
	\end{lstlisting}
%
	\item waterflow problem \\
	Given an array of "wall" heights, determine the volume of the puddles that can form if it rains.
	\begin{lstlisting}
water :: [Int] -> Int
water [1,2,3] == 0
water [5,2,5] == 3
water [2,3,1,6,1] == 2
water h = sum $
      zipWith (-)
        (zipWith min (scanl1 max h) (scanr1 max h))
        h
	\end{lstlisting}
%
	\item horner schema to evaluate polynomials
	\begin{lstlisting}
horner :: [Int] -> Int -> Int
horner [1,2,3] 1 == 6
horner [1,2,3] 2 == 11
horner [4,3,2] 3 == 47
horner p x = foldl1 ((+) . (x *)) p
	\end{lstlisting}
	Problem: we do not generate lambda's. Do we generate functions like \lstinline?(x *)??
%
	\item sum-under, sum all integers up to the argument
	\begin{lstlisting}
sum_under :: Int -> Int
sum_under 0 == 0
sum_under 1 == 1
sum_under 2 == 3
sum_under 3 == 6
sum_under 4 == 10
sum_under n = sum [1..n]
	\end{lstlisting}
	Components needed: \lstinline?sum, enumFromTo?
%
	\item factorial \\
	\begin{lstlisting}
factorial :: Int -> Int
factorial 0 == 0
factorial 1 == 1
factorial 3 == 6
factorial 5 == 120
factorial n = product [1..n]
	\end{lstlisting}
	interesting for intermediate states
%
	\item maximum of a list\\
	I don't know (yet) how to specify a "global property" like greater or smaller than all other elements in a list in \textsc{Synquid}. Moreover, it seems a difficult property to extract from input-output examples.
	\begin{lstlisting}
maximum :: [Int] -> Int
maximum [1,3,2] == 3
maximum [4,2,1] == 4
maximum [1,3,5] == 5
maximum xs = foldr max (head xs) xs
	\end{lstlisting}
	Or just use the \lstinline?maximum? function from \lstinline?Prelude?, if it is given as a component
%
	\item append two lists\\
	The specification given by Nadia does not synthesize the usual append function. Maybe it's better to let her know...\\
	Although it's possible to synthesize append in \textsc{Synquid}.
	\begin{lstlisting}
append :: [a] -> [a] -> [a]
append [1,2,3] [4,5,6] == [1,2,3,4,5,6]
append [1,2] [6,2,3] == [1,2,6,2,3]
append xs ys = foldr (:) ys xs
	\end{lstlisting}
	Or use \lstinline?++? from \lstinline?Prelude?, if we decide to provide it for this example too.
%
	\item length of a list \\
	Can be also interesting for intermediate states
	\begin{lstlisting}
length :: [a] -> Int
length [1,2,3] == 3
length [2,2,2] == 3
length [] == 0
length [5] == 1
length xs = sum $ map (const 1) xs
	\end{lstlisting}
%
	\item list reversal
	\begin{lstlisting}
reverse :: [a] -> [a]
reverse [1,2,3] == [3,2,1]
reverse [5,2,3] == [3,2,5]
reverse [6,2,3,1] == [1,3,2,6]
reverse xs = foldl (flip (:)) [] xs
	\end{lstlisting}
%
	\item bagsum: \lstinline![far,bar,gar,bar,bar,far] -> [(bar,3),(far,2),(gar,1)]!\\
	Seems difficult and maybe intermediate states can be helpful. Should I take \lstinline?Int? instead of \lstinline?a?? I mean, \lstinline?a? should belong to the typeclass \lstinline?Ord?, otherwise we cannot yield a sorted output list. And we do not have any other base types anyway. But actually lists of integers are also ordable. Hence also lists of lists of integers and lists of lists of lists of integers and so on.
	\begin{lstlisting}
bagsum :: [a] -> [(a, Int)]
bagsum [1,1,1] == [(1,3)]
bagsum [4,4,2,1,2,1] == [(1,2),(2,2),(4,2)]
bagsum [3,2,1,3,2,3] == [(1,1),(2,2),(3,3)]
bagsum xs = map (head &&& length) (group (sort xs))
	\end{lstlisting}
	Components needed: \lstinline?&&&? from \lstinline?Control.Arrow?. And we also need \lstinline?Tuples? for this one.
%
	\item stutter \\
	Repeat every list element twice.
	\begin{lstlisting}
stutter :: [a] -> [a]
stutter [] == []
stutter [1,2,3] == [1,1,2,2,3,3]
stutter xs = concatMap (replicate 2) xs
	\end{lstlisting}
%
	\item map \\
	Isn't it a higher order function? I thought we synthesize only first order functions.\\
	How can we provide examples? I mean, we have to write functions as well.\\
	\begin{lstlisting}
map :: (a -> b) -> [a] -> [b]
map f xs = foldr ((:) . f) [] xs
	\end{lstlisting}
%
	\item zipWith \\
	it's a higher order function as well. We need Tuples.\\
	How do we provide the examples?
	\begin{lstlisting}
zipWith :: (a -> b -> c) -> [a] -> [b] -> [c]
zipWith f xs ys = map (uncurry f) (zip xs ys)
	\end{lstlisting}
	Components needed: \lstinline?map, uncurry, zip?.
%
	\item list drop
	\begin{lstlisting}
drop :: Int -> [a] -> [a]
drop 0 [1,2,3] == [1,2,3]
drop 1 [1,2,3] == [2,3]
drop 5 [5,4,2,5] == []
drop 3 [4,2,3,1] == [1]
drop n xs = snd (splitAt n xs)
	\end{lstlisting}
%
	\item droplast, drop the last element of a list
	\begin{lstlisting}
droplast :: [a] -> [a]
droplast [] == []
droplast [1] == []
droplast [1,2,3] == [1,2]
droplast [3,2,1]== [3,2]
droplst = init
droplast' xs = map fst (zip xs (enumFromTo 1 (subtract 1 (length xs))))
droplast'' xs = take (subtract 1 (length xs)) xs
	\end{lstlisting}
%
	\item dropmax, drop the greatest element of a list\\
	$\lambda^2$ takes much more time to synthesize droplast than dropmax. Why?
	\begin{lstlisting}
dropmax :: [Int] -> [Int]
dropmax [1,2] == [1]
dropmax [2,1] == [1]
dropmax [1,2,3] == [1,2]
dropmax [3,2,1] == [2,1]
dropmax [2,3,1] == [2,1]
dropmax xs = filter (/= (maximum xs)) xs
	\end{lstlisting}
%
	\item dedup, remove duplicates from a list \\
	$\lambda^2$ requires more time\\
	\TODO{Find a non-recursive implementation that preserves the order of the elements} Either implement it with \lstinline?groupBy? or say you cannot implement it. Say it's not possible to get the usual semantics of deleting all but the first occurrence of some program.
%
	\item sort by length (on lists of lists)
	\begin{lstlisting}
sortByLength :: [[a]] -> [[a]]
sortByLength [[1,2],[1,2,3],[1]] == [[1],[1,2],[1,2,3]]
sortByLength = sortBy (curry ((uncurry compare) . (length *** length)))
	\end{lstlisting}
	\TODO{Is there a shorter implementation?} What's wrong with this one? \\
	Components needed: \lstinline?(***)? from \lstinline?Control.Arrow?
%
	\item dropmins \\
	$\lambda^2$ required more time to synthesize it
	\begin{lstlisting}
dropmins :: [[Int]] -> [[Int]]
dropmins [[1,2,3],[2,3,1],[2,3],[1]] == [[2,3],[2,3],[3],[]]
dropmins = map dropmin
	\end{lstlisting}
	\TODO{Is there an implementation without the auxiliary function \lstinline?dropmin? and without lambda expressions?} Yes, there is one, but you don't want to see it. With \lstinline?ap? and \lstinline?join?.
%
	\item lasts, last element of every list \\
	another program on nested lists
	\begin{lstlisting}
lasts :: [[a]] -> [a]
lasts [[1,2,3],[3,2],[4,2,4,1],[2]] == [3,2,1,2]
lasts = map last
	\end{lstlisting}
%
	\item member of the tree\\
	Something with trees. Membership seems a difficult thing to learn from input-output examples.
%
	\item count leaves it a tree
%
	\item nodes at level
	The standard Haskell tree is a rose tree. Defined in \lstinline?Data.Tree?.

\note{Nadia has more complicated examples with Red-Black-Trees, AVL-trees and different sorting algorithms}
\end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "thesis"
%%% End:


\paragraph{Answer 3.}

\begin{enumerate}

  \item The first regular expression can be simplified in the
    following way:
   \begin{align*}
     \lparen \epsilon \, \disjM{} \, a\kleeneM{} \, \disjM{} \,
     b\kleeneM{} \, \disjM{} \, a \, \disjM{} \, b \rparen\kleeneM
     &= \lparen \epsilon \, \disjM{} \, a\kleeneM{} \, \disjM{} \,
     b\kleeneM{} \, \disjM{} \, b\rparen\kleeneM, & \text{since}
     \, L(a) \subset L(a\kleeneM);\\
     &= \lparen \epsilon \, \disjM{} \, a\kleeneM{} \, \disjM{} \,
     b\kleeneM\rparen\kleeneM, & \text{since} \,
     L(b) \subset L(b\kleeneM);\\
     &= \lparen \epsilon \, \disjM{} \, a\plusM{} \, \disjM{} \,
     b\plusM\rparen\kleeneM, & \text{since} \, \{\epsilon\}
     \subset L(x\kleeneM);\\
     &= \lparen a\plusM{} \, \disjM{} \, b\plusM\rparen\kleeneM, &
     \text{since} \, \lparen\epsilon \, \disjM{} \,
     x\rparen\kleeneM = x\kleeneM.
   \end{align*}
   Words in \(L(\lparen a\plusM{} \, \disjM{} \,
   b\plusM\rparen\kleeneM)\) are of the form \(\epsilon\) or
   \((a\ldots a)\) \((b \ldots b)\) \((a\ldots a)\) \((b\ldots
    b)\ldots\), where the ellipsis stands for `none or many times'.
   So we recognise \(\lparen a \, \disjM{} \,
   b\rparen\kleeneM\). Therefore \(\lparen \epsilon \, \disjM{}
   \, a\kleeneM{} \, \disjM{} \, b\kleeneM{} \, \disjM{} \, a \,
   \disjM{} \, b \rparen\kleeneM = \lparen a \, \disjM{} \,
   b\rparen\kleeneM\).

  \item The second regular expression can be simplified in the
    following way. We note first that the expression is made of the
    disjunction of three regular sub-expressions (\emph{i.e.,} it is a
    union of three sub-languages). The simplest idea is then to check
    whether one of these sub-languages is redundant, \emph{i.e.,} if
    one is included in another. If so, we can simply remove it from
    the expression.
  \begin{align*}
     a \lparen a \, \disjM{} \, b \rparen\kleeneM{} b
     \, \disjM{} \, \lparen a b \rparen\kleeneM{}
     \, \disjM{} \, \lparen b a \rparen\kleeneM
     &= a \lparen a \, \disjM{} \, b \rparen\kleeneM{} b
     \, \disjM{} \, \epsilon
     \, \disjM{} \, \lparen a b \rparen\plusM{}
     \, \disjM{} \, \lparen b a \rparen\kleeneM,\\
     & \qquad \text{since} \, \lparen ab \rparen\kleeneM =
     \epsilon \, \disjM{}\, \lparen ab \rparen\plusM;\\
     &= a \lparen a \, \disjM{} \, b \rparen\kleeneM{} b
     \, \disjM{} \, \lparen a b \rparen\plusM{}
     \, \disjM{} \, \lparen b a \rparen\kleeneM,\\
     & \qquad \text{since} \, \{\epsilon\} \subset L(\lparen ba
     \rparen\kleeneM).
  \end{align*}
  We have:
  \begin{align*}
    \lparen ab\rparen\plusM
   &= \lparen ab\rparen \lparen ab\rparen \ldots \lparen ab \rparen\\
   &= a\lparen ba\rparen\lparen ba\rparen\ldots\lparen  ba\rparen b \;
    \disjM{} \; ab\\
   &= a\lparen ba \rparen\kleeneM b.
   \end{align*}
   Also \(L(\lparen ba \rparen) \subset L(\lparen a \, \disjM{}
   \, b\rparen\kleeneM)\) and then \(L(\lparen ba
   \rparen\kleeneM) \subset L(\lparen a \, \disjM{} \,
   b\rparen\kleeneM)\), because \(\lparen a \, \disjM{} \,
   b\rparen\kleeneM\) denotes all the words. Therefore
   \begin{align*}
      L(a\lparen ba \rparen\kleeneM b)
    &\subset L(a \lparen a \, \disjM{} \, b\rparen\kleeneM b)\\
      L(\lparen ab\rparen\plusM)
    &\subset L(a \lparen a \, \disjM{} \, b\rparen\kleeneM b)
   \end{align*}

   As a consequence, one possible answer is
   \begin{equation*}
     a \lparen a \, \disjM{} \, b \rparen\kleeneM{} b
     \, \disjM{} \, \lparen a b \rparen\kleeneM{}
     \, \disjM{} \, \lparen b a \rparen\kleeneM
     = a \lparen a \, \disjM{} \, b\rparen\kleeneM b
     \, \disjM{} \, \lparen ba \rparen\kleeneM.
   \end{equation*}
   The intersection between \(L(a \lparen a \, \disjM{} \,
   b\rparen\kleeneM b)\) and \(L(\lparen ba \rparen\kleeneM)\)
   is empty because all the words of the former start with
     \(a\), while all the words of the other start with \(b\) (or is
     \(\epsilon\)).  Therefore we cannot simply further this way.

\end{enumerate}


\section{Introduction}

\begin{frame}{\emph{``Does Watching Football on TV Cause Hair Loss?''}}

    \begin{itemize}
        \item 296 British subjects were asked about their hair loss and how much football they watch on television, \cite{DMJSDS2003}.
    \end{itemize}

    \begin{block}{Contingency Table}

    One can represent the responses in a $3\times 3$ \emph{contingency table}:

    \begin{table}[h]
    \vspace{12pt}
        \begin{tabular}{@{}lcccc@{}}
        & &  \multicolumn{3}{c}{Hair Amount}\\\cmidrule{3-5}
        TV Hours & & lots & medium & balding\\ \midrule
        $\leq 2$h & & 51 & 45 & 33 \\
        $2-6$ h & & 28 & 30 & 29 \\
        $\geq 6$h & & 15 & 27 & 38\\\bottomrule
        \end{tabular}
    \end{table}
    \end{block}

    \begin{itemize}
        \item Is there an association between the variables, or are they independent?
    \end{itemize}

\end{frame}

\begin{frame}{\emph{``Does Watching Football on TV Cause Hair Loss?''}}
    $$ M = \begin{bmatrix}
    51 & 45 & 33 \\
    28 & 30 & 29 \\
    15 & 27 & 38
    \end{bmatrix} $$

    \begin{block}{Null Hypothesis}
    \vspace*{-18pt}
        \begin{equation*}
            H_{0}: \qquad \text{\emph{Football on TV and Hair Loss are Independent.}}
        \end{equation*}
    \vspace*{-18pt}
    \end{block}

    \begin{itemize}
        \item Independence $\implies$ odds ratios all equal one:
            $$ \text{Odds Ratio} = \frac{ m_{ij}\cdot m_{(i+1)(j+1)} } { m_{(i+1)j} \cdot m_{i(j+1)} } = 1, \quad (\text{for all } i,j). $$
        \item But (for example): $51 \cdot 30 - 45 \cdot 28 = 1530 - 1260 = 270 \neq 0\, \ldots\, $?
    \end{itemize}

\end{frame}

\begin{frame}{\emph{``Does Watching Football on TV Cause Hair Loss?''}}

    \begin{itemize}
        \item A better explanation is obtained by identifying a certain \emph{hidden variable}, which is the \emph{gender identification} of the respondents:
    \end{itemize}

    $$ M  = M_{m} + M_{f} =
    \underbrace{\begin{bmatrix}
    3 & 9 & 15 \\
    4 & 12 & 20 \\
    7 & 21 & 35
    \end{bmatrix}}_{126 \text{ male}} +
    \underbrace{\begin{bmatrix}
    48 & 36 & 18 \\
    24 & 18 & 9 \\
    8 & 6 & 3
    \end{bmatrix}}_{170 \text{ female}}.  $$

    \begin{block}{Alternative Hypothesis}
        Instead, we include \emph{conditional independence}:
        \begin{equation*}
            H_{1}: \text{ \emph{Football on TV \& Hair Loss are Independent} given \emph{Gender.}}
        \end{equation*}
    \end{block}

\end{frame}


\section{Delayed Neutrons}
Most neutrons born in the core are prompt neutrons, about 99\%.
The rest are \textit{deyaled neutrons} and come after a fission
fragment beta decays,
%FIXME insert fission fragment figure here.
\underline{Call} $\beta$ the fraction [iunitless] of neutrons that are delayed, ie
that are not prompt.  Typically $\beta$ is chopped up into $I$ groups the each represent
a different mean generation time for thatgroup. Say $i\in I$, then,
\[ \beta = \sum_{i=1}^I \beta_i \]
and by convention, $beta_1$ has the longest generation time and $\beta_I$ has the shortest
generation time. $I=6$ is pretty standard.

Thus the shource of prompt neutrons, rather than being $\nu\Sigma_f\phi$ is now,
\[ \mathrm{propmpt source} = (1 - \beta) \nu \Sigma_f \phi \]
Call $C_i(t)$ the \underline{concentration} [1/cm$^3$] of gthe ith group and
$\lambda_i = \ln(2) / T_{1/2, i}$i sthe \underline{decay constant} [1/s] of the
ith group. for the half-life [s] of the ith group. The total delayed source is then
\[ \mathrm{delayed source} = \sum_{i=1}^I \lambda_i C_i(t)\]
Given that $\beta_i\nu\Sigma_f\phi$
is the number of ith group neutrons that are born each generation, the concentration ith group
delayed neutrons at any time is,
\[ \frac{dC_i(t)}{dt} = \beta_i\nu\Sigma_f\phi(t) - \lambda_i C_i(t)\]
Recasting the prompt equation as,
\[ \frac{1}{v} \frac{d\phi}{dt} = (1-\beta)\nu\Sigma_f\phi(t) + \sum_{i=1}^I \lambda_i C_i(t) - \Sigma_a\phi(t) - DB^2\phi(t) \]
Thus we have a system of $I+1$ equations and $I+1$ unknowns.


\section{The algebra of linear transformations}

\begin{outcome}
  \begin{enumerate}
  \item Use algebraic properties of linear transformations to
    manipulate expressions.
  \end{enumerate}
\end{outcome}

Two linear transformations are considered to be equal if they act in
the same way on all vectors. This is the content of the following
definition.

\begin{definition}{Equal transformations}{equal-transformations}
  Let $S$ and $T$ be linear transformations from $V$ to $W$. We say
  that $S$ and $T$ are \textbf{equal}%
  \index{linear transformation!equality of}%
  \index{equality!of linear transformations}, and we write $S = T$, if
  for all $\vect{v}\in V$,
  \begin{equation*}
    S(\vect{v}) = T(\vect{v}).
  \end{equation*}
\end{definition}

We now consider several operations on linear transformations. These
include addition and scalar multiplication of linear transformations,
as well as the zero transformation.

\begin{definition}{Addition and scalar multiplication of linear transformations}{addition-linear-transformations}
  Let $V$ and $W$ be vector spaces over a field $K$.
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item The \textbf{zero transformation}%
      \index{zero transformation}%
      \index{linear transformation!zero transformation} $0:V\to W$ is
      defined by $0(\vect{v})=\vect{0}$ for all $\vect{v}\in V$.
    \item If $T,S:V\to W$ are linear transformations, then their
      \textbf{sum}%
      \index{addition!of linear transformations}%
      \index{linear transformation!addition}
      $T+S:V\to W$ is the linear transformation defined by
      \begin{equation*}
        (T+S)(\vect{v}) = T(\vect{v}) + S(\vect{v})
      \end{equation*}
      for all $\vect{v}\in V$.
    \item If $T:V\to W$ is a linear transformation and $k\in K$, the
      linear transformation%
      \index{scalar multiplication!of linear transformation}%
      \index{linear transformation!scalar multiplication}
      $kT : V\to W$ is defined by
      \begin{equation*}
        (kT)(\vect{v}) = k(T(\vect{v}))
      \end{equation*}
      for all $\vect{v}\in V$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{definition}

\begin{proposition}{Addition and scalar multiplication of linear transformations}{addition-linear-transformations}
  If $T,S:V\to W$ are linear transformations, then so are $T+S$ and $kT$.
\end{proposition}

\begin{proof}
  To show that $T+S$ is linear, we must verify that it preserves
  addition and scalar multiplication. Let $\vect{v},\vect{u}\in
  V$. Then we have
  \begin{equation*}
    \begin{array}{r@{~}c@{~}ll}
      (T+S)(\vect{v}+\vect{u})
      &=& T(\vect{v}+\vect{u}) + S(\vect{v}+\vect{u})
      & \mbox{by definition of $T+S$,} \\
      &=& (T(\vect{v}) + T(\vect{u})) + (S(\vect{v}) + S(\vect{u}))
      & \mbox{by linearity of $T$ and $S$,} \\
      &=& (T(\vect{v}) + S(\vect{v})) + (T(\vect{u}) + S(\vect{u}))
      & \mbox{by the associative and commutative laws of vectors,} \\
      &=& (T+S)(\vect{v}) + (T+S)(\vect{u})
      & \mbox{by definition of $T+S$.}
    \end{array}
  \end{equation*}
  Therefore, $T+S$ preserves addition. Also, for $\vect{v}\in V$ and
  $\ell\in K$, we have
  \begin{equation*}
    \begin{array}{r@{~}c@{~}ll}
      (T+S)(\ell\vect{v})
      &=& T(\ell\vect{v}) + S(\ell\vect{v})
      & \mbox{by definition of $T+S$} \\
      &=& \ell T(\vect{v}) + \ell S(\vect{v})
      & \mbox{by linearity of $T$ and $S$} \\
      &=& \ell(T(\vect{v}) + S(\vect{v}))
      & \mbox{by the distributive law over vector addition} \\
      &=& \ell((T+S)(\vect{v}))
      & \mbox{by definition of $T+S$.}
    \end{array}
  \end{equation*}
  Therefore, $T+S$ preserves scalar multiplication, and hence $T+S$ is
  a linear transformation. The proof that $kT$ is a linear
  transformation is left as an exercise.
\end{proof}

There operations satisfy the following properties:

\begin{proposition}{Properties of addition and scalar multiplication of linear transformations}{properties-addition-linear-transformation}
  Let $V,W$ be vector spaces over a field $K$, let $T,S,R:V\to W$
  be linear transformations, and let $k,\ell\in K$ be scalars. Then
  the following hold:%
  \index{properties of addition!linear transformations}%
  \index{linear transformation!properties of addition}%
  \index{linear transformation!addition!properties}%
  \index{properties of scalar multiplication!linear transformation}%
  \index{linear transformation!properties of scalar multiplication}%
  \index{linear transformation!scalar multiplication!properties}
  \begin{itemize}\setlength\itemsep{0em}
  \item Commutative law of addition:
    $T+S = S+T$.
  \item Associative law of addition:
    $(T+S)+R = T+(S+R)$.
  \item The existence of an additive unit: there exists an element $\vect{0}\in
    V$ such that for all $T$,
    $T + 0 = T$.
  \item The law of additive inverses:
    $T + (-T) = 0$.
  \item The distributive law over vector addition:
    $k(T + S) = kT + kS$.
  \item The distributive law over scalar addition:
    $(k + \ell) T = k T + \ell T$.
  \item The associative law for scalar multiplication:
    $k(\ell T) = (k \ell)T$.
  \item The rule for multiplication by one:
    $1T=T$.
  \end{itemize}
\end{proposition}

But these 8 properties are just the vector space laws (A1)--(A4) and
(SM1)--(SM4)! Therefore, the set of linear transformations from $V$ to
$W$, with the above operations of addition and scalar multiplication,
forms a vector space.

\begin{definition}{Vector space of linear transformations}{vector-space-linear-transformations}
  Let $V,W$ be vector spaces over a field $K$. We define $\Lin_{V,W}$%
  \index{vector space!of linear transformations}%
  \index{LinVW@$\Lin_{V,W}$} to be the vector space of all linear
  transformations from $V$ to $W$, with the above operations of
  addition and scalar multiplication.
\end{definition}

Another important operation is the composition of linear
transformations. We have already encountered this in
Definition~\ref{def:composite-transformations} for the case of
$\R^n$. Here, we generalize it to linear transformations of arbitrary
vector spaces.  We also consider the identity transformation on a
vector space, which forms the unit for composition.

\begin{definition}{Composition of linear transformations}{composition-linear-transformation}
  Let $V,U,W$ be vector spaces over a field $K$, and let $S:V\to U$
  and $T:U\to W$ be linear transformations.
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item The
      \textbf{composition}%
      \index{linear transformation!composition}%
      \index{composition of linear transformations} $T\circ S:V\to W$ is
      the linear transformation defined by
      \begin{equation*}
        (T\circ S) (\vect{v}) = T(S(\vect{v}))
      \end{equation*}
      for all $\vect{v}\in V$.
    \item The \textbf{identity transformation}%
      \index{identity transformation}%
      \index{linear transformation!identity transformation}
      $1_V:V\to V$ is defined by $1_V(\vect{v})=\vect{v}$ for all
      $\vect{v}\in V$. We often omit the subscript and just write
      $1:V\to V$ when $V$ is clear from the context.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{definition}

Composition of linear transformations satisfies the following properties:

\begin{proposition}{Properties of composition of linear transformations}{properties-composition}
  Composition of linear transformations satisfies the following
  properties, for all $R,S,T$ as appropriate for each law:%
  \index{properties of composition!linear transformations}%
  \index{linear transformation!properties of composition}%
  \index{linear transformation!composition!properties}%
  \begin{itemize}\setlength\itemsep{0em}
  \item Associative law of composition:
    $(T\circ S)\circ R = T\circ (S\circ R)$.
  \item Unit laws of composition: $T\circ 1_V = T$ and $1_W\circ T = T$, where
    $T:V\to W$.
  \item Distributive laws: $(T+S)\circ R = T\circ R + S\circ R$ and
    $P\circ (T+S) = P\circ T + P\circ S$.
  \item Zero laws: $0\circ R = 0$ and $R\circ 0 = 0$.
  \item Compatibility with scalar multiplication:
    $(kT)\circ S = k(T\circ S) = T\circ (kS)$, where $k\in K$.
  \end{itemize}
\end{proposition}

Finally, we consider the notion of the inverse of a linear
transformation.

\begin{definition}{Inverse of a linear transformation}{inverse-linear-transformation}
  Let $V,U$ be vector spaces over a field $K$, and let $S:V\to U$ and
  $T:U\to V$ be linear transformations.  We say that $S$ and $T$ are
  \textbf{inverses}%
  \index{inverse!of a linear transformation}%
  \index{linear transformation!inverse} if
  \begin{equation*}
    T\circ S = 1_V
    \quad\mbox{and}\quad
    S\circ T = 1_U.
  \end{equation*}
  In this case, we also write $S=T^{-1}$ and $T=S^{-1}$.
\end{definition}

\begin{proposition}{Uniqueness of inverses}{uniqueness-inverses-linear-transformation}
  Inverses are unique. In other words, if $S$ and $S'$ are two inverses
  of $T$, then $S=S'$.
\end{proposition}

\begin{proof}
  Suppose both $S$ and $S'$ are inverses of $T$. Consider
  $S\circ T\circ S'$. Since $S$ and $T$ are inverses, this is equal to
  $S'$, but since $T$ and $S'$ are inverses, it is also equal to
  $S$. Therefore, $S=S'$.
\end{proof}

\begin{proposition}{Properties of inverses of linear transformations}{properties-inverse-linear-transformation}
  \begin{itemize}
  \item If $S:V\to U$ and $T:U\to W$ are both invertible, then so is
    $T\circ S$, and we have
    \begin{equation*}
      (T\circ S)^{-1} = S^{-1}\circ T^{-1}.
    \end{equation*}
  \item $1^{-1} = 1$.
  \item If $S:V\to U$ is invertible and $k$ is a non-zero scalar, then
    \begin{equation*}
      (kS)^{-1} = k^{-1}S^{-1}.
    \end{equation*}
  \end{itemize}
\end{proposition}


Implements the SMT compliant \emph{Fourier-Motzkin algorithm}.
Hence, this module can decide the consistency of any conjunction
consisting only of linear real arithmetic constraints. Furthermore,
it might also find the consistency of a conjunction of constraints
even if they are not all linear e.g. in the case of a monomial $x^i$
(where i is a positive integer) only occuring in the shape of this
monomial. Such a monomial is subsequently eliminated as common linear
variables are eliminated. One can tune a threshold parameter in order to
determine when, regarding the size of the considered constraints,
this module shall call the backends. In the latter case, the backends are called
with the constraint set that is obtained after eliminating a certain
number of variables.

\paragraph{Integer arithmetic} One can also use this approach for (linear) integer arithmetic
as unsatisfiability over the real domain implies unsatisfiability over the integer
domain. In addition to that, one can heuristically try to construct integer solutions
by considering the lowest upper and the highest lower bound of a variable that can be derived
from the respective elimination step. Note that this approach is incomplete.


\documentclass[]{report}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{listings}

\graphicspath{ {images/} }

\title{CSCI 567 HW \# 5}
\author{Mohmmad Suhail Ansari \\ USC ID: 8518586692\\e-mail: mohmmada@usc.edu}

\begin{document}

\maketitle

\paragraph{Sol. 1.1}
	Given
	\[ D = \sum_{n=1}^N \sum_{k=1}^K r_{nk}  {\| x_n - \mu_k \|}_2^2 \]
	taking partial derivative w.r.t. $\mu_k$, we get
	\[ \frac{\partial{D}}{\partial{\mu_k}} = \sum_{n=1}^N r_{nk} [-2 (x_n - \mu_k)] = 0 \]
	\[ = \sum_{n=1}^N r_{nk} x_n - \mu_k \sum_{n=1}^N r_{nk} = 0\]
	\[ \mu_k = \frac{\sum_{n=1}^N r_{nk} x_n}{\sum_{n=1}^N r_{nk}}\]

\paragraph{Sol. 1.2}
	Given
	\[ D = \sum_{n=1}^N \sum_{k=1}^K r_{nk}  {\| x_n - \mu_k \|}_1 \]
	We want find an optimal $\mu_k$, such that it minimizes $D$ and that optimal $\mu$ is equal to the
	median of $x_n = [x_{n1}, x_{n2} \hdots x_{nD}]$.

	Now, let us assume that, $\mu_k$ is optimal and for a given vector $x_n, x_n \in R^D$, so fro the definition of
	``closeness'' in the text we get that
	\[ L = \sum_{i=1}^D |x_{ni} - \mu_k|\],
	Now if there are $l$ numbers to the left of $\mu_k$ and $r$ number to the right, then if we were to shift $\mu_k$ by a distance $d$
	to the left, the $L$ increases by $(l - r)d \quad if(l > r)$. Similarly if, we were to shift $\mu_k$ to the right by a distance of $d$, then again the
	measurement of $L$ increases by $(r - l)d \quad if (r > l)$. Therefore $L$ will achieve if minimum value when $l = r$, i.e. $\mu_k$ is the median of the
	vector $x_n$.

	Now we derive the above conclusion to multi-dimensionalities. We can see only when  is the elementwise median of cluster k, $L_k = \sum_{i=1}^{N_k} |x_i - \mu_k |$ has the minimal value. Suppose there are K clusters, the overall loss is
	\[ L = \sum_{k=1}^K L_k = \sum_{k=1}^K \sum_{n=1}^N |x_n - \mu_k| \]
	which will also be minimal, which equals to
	\[ D = \sum_{k=1}^K L_k = \sum_{k=1}^K \sum_{n=1}^N r_{nk} |x_n - \mu_k| \]

\paragraph{Sol 1.3}
	Given
	\begin{equation}
		\tilde{D} = \sum_{n=1}^N \sum_{k=1}^K r_{nk} {\| \phi (x_n) - \tilde{\mu}_k \|}_2^2
	\end{equation}

	where,
	\begin{equation}
		\tilde{\mu}_k = \frac{1}{N_k} \sum_{n=1}^N r_{nk} \phi (x_n)
	\end{equation}

	We can write $\frac{r_{nk}}{N_k} = \gamma_{nk}$, then we can rewrite $\tilde{\mu}$ as
	\[ \tilde{\mu}_k = \sum_{n=1}^N \gamma_{nk} \phi (x_n) \]

	Then,
	\[ {\| \phi(x_n) - \tilde{\mu}\|}^2 = {\| \phi(x_n) - \sum_{n=1}^N \gamma_{nk} \phi (x_n) \|}^2 \]
	\[ = [\phi(x_n) - \sum_{n=1}^N \gamma_{nk} \phi (x_n)] \cdot [\phi(x_n) - \sum_{n=1}^N \gamma_{nk} \phi (x_n)] \]
	\[ = K(x, x) - 2 \sum_{i=1}^N \gamma_{ik} K(x, x_i) + \sum_{i=1}^N \sum_{j=1}^N \gamma_{ik} \gamma_{jk} K(x_i, x_j) \]

	Therefore, we can write
	\[ \tilde{D} = \sum_{n=1}^N \sum_{k=1}^K r_{nk} \big[ K(x_n, x_n) - 2 \sum_{i=1}^N \gamma_{ik} K(x_n, x_i) + \sum_{i=1}^N \sum_{j=1}^N \gamma_{ik} \gamma_{jk} K(x_i, x_j) \big] \]

	To assign a point to a cluster $k$, we initialize (randomly or through other methods) $\tilde{\mu}_k$ and for each iteration, assign, $x_n$ to k where
	\[ argmax_{k \in K} K(x_n, x_n) - 2 \sum_{i=1}^N \gamma_{ik} K(x_n, x_i) + \sum_{i=1}^N \sum_{j=1}^N \gamma_{ik} \gamma_{jk} K(x_i, x_j) \]

	The pseudo-code
	\newpage

\begin{lstlisting}
Kernel_Kmeans():
  G = GramMatrix(X)
  Assignment = init_random_assigmnment()
  Means = random.sample(X, k)
  for max_iterations:
  	for k in K:
  	  for i, x in enumerate(X):
	        distance[i, k] =  G[x, x]
	        distance[i, k] -= (2 * sum(G[(Assignment == k), i])
	        distance[i, k] += sum(G[Assignment == k, Assignment == k])
    Assignment = argmin(distance)
    for k in K:
      NewMeans[k] = mean(X[Assignment == k])
    if convereged(Means, NewMeans):
      return Assignment, NewMeans
    else:
      Means = NewMeans
\end{lstlisting}

\paragraph{Sol 2.1}
	We can write the likelihood function as
	\[ p(x | \alpha) = \frac{\alpha}{\sqrt{2 \pi}} exp(-\frac{1}{2} x^2) + \frac{1 - \alpha}{\sqrt{\pi}} exp(- x^2)\]
	and for observed sample $x_1$, we can write
	\[ p(x_1 | \alpha) = (\frac{1}{\sqrt{2 \pi}} exp(-\frac{1}{2}x_1^2) - \frac{1}{\sqrt(\pi)} exp(-x_1^2)) \alpha + \frac{1}{\sqrt{\pi}} exp(-x^2) \]
	We observe the likelihood function for an observed sample $x_1$ is a linear function of $\alpha$ where the slope of the line is determined by the values of the gaussian probabilities. So, for maximum likelihood, if the gaussian probability $N(x_1 | 0, 1) > N(x_1|0, 0.5)$ then we choose $\alpha = 1$ for maximum likelihood, else we choose $\alpha = 0$.

\paragraph{Sol 3.1}
	Let us define the hidden variable as $z_i = 1$ when a person in the sample has taken insurance, and $z_i = 0$ otherwise. Now, if $x_i > 0$, then clearly $z_i = 1$, however, when $x_i = 0$, then $z_i = 1$ or $0$.

	Therefore our likelihood function can be given as
	\[
		L = \prod_{i=1}^N \pi^{1-u_i} \big[ (1 - \pi) \frac{e^{-\lambda} \lambda^{x_i}}{x_i!} \big]^{u_i}
	\]

	where $u_i = 1$ if $x_i > 0$ and $u_i = z_i$ if $x_i = 0$.

\paragraph{Sol 3.2} N/A

\paragraph{Sol 4.1}
	\begin{center}
		\includegraphics[width=\textwidth]{blob-2}
		Blob, K = 2
		\includegraphics[width=\textwidth]{blob-3}
		Blob, K = 3
		\includegraphics[width=\textwidth]{blob-5}
		Blob, K = 5
		\includegraphics[width=\textwidth]{circle-2}
		Circle, K = 2
		\includegraphics[width=\textwidth]{circle-3}
		Circle, K = 3
		\includegraphics[width=\textwidth]{circle-5}
		Circle, K = 5
	\end{center}

	For $\texttt{circle.csv}$ and $K = 2$, since, both the clusters are concentric circles and their centroid falls at the same point and hence for a point the difference between minimum distances is small or equal.

\paragraph{Sol 4.2}
	Using the feature transformation
	\[ \phi(x, y) = [x, y, 2(x^2 + y^2)] \]

	\begin{center}
		\includegraphics[width=\textwidth]{Kernel-Kmeans-2}
		k = 2
	\end{center}

\paragraph{Sol 4.3}

	\begin{center}
		\includegraphics[width=\textwidth]{LogLikelihood}
		Log-Likehood Plot
		\includegraphics[width=\textwidth]{EM-Scatterplot}
		Cluster Assignment Plot
	\end{center}
\[
	Means = \begin{bmatrix}
		K=1 &   0.75896032 & 0.67976983 \\
		K=2 &  -0.32591595 & 0.97133268 \\
		K=3 &  -0.63946222 & 1.47460006
	\end{bmatrix}
\]

\[
	Covariance[k = 1] = \begin{bmatrix}
		0.02717056  & -0.00840045 \\
	 	-0.00840045 &  0.040442
	\end{bmatrix}
\]
\[
	Covariance[k = 2] = \begin{bmatrix}
		0.03604869 & 0.01463998 \\
 		0.01463998 & 0.01629099
	\end{bmatrix}
\]
\[
	Covariance[k = 3] = \begin{bmatrix}
		0.03596703 & 0.01549264  \\
 		0.01549264 & 0.01935347
	\end{bmatrix}
\]

\end{document}


\lab{Plotting With matplotlib and Mayavi}{Plotting}
\objective{Introduce some of the basic plotting functions available in matplotlib and Mayavi.}
\label{lab:Matplotlib_and_Mayavi}

\section*{matplotlib}
matplotlib is one of Python's many plotting libraries.
It has an active development community and was designed to resemble MATLAB.
Because of its high quality and its helpful user interface toolkits, matplotlib serves as our primary plotting library for 2D graphs in this text.

This section seeks to introduce many of the basic plotting functions we will access using matplotlib, but matplotlib has many more.
For further reference and information, visit \url{http://matplotlib.org}.

\subsection*{Simple Plots}
To get started, we import pyplot from matplotlib.
\begin{lstlisting}
from matplotlib import pyplot as plt
\end{lstlisting}

The basic line plotting function in matplotlib is \li{plot()}.
It takes a set of data points and plots the line that is formed between those points.
These plots are pieced together using a \emph{state machine environment}, which means that we can run several different functions that will display or modify the plot we are creating.

To plot and display our function we typically use an outline similar to the following:
\begin{lstlisting}
# Definition of x and y coordinates (typically NumPy arrays or lists).
plt.plot(x,y)
plt.show()
\end{lstlisting}
Note that x and y must be the same length in order for \li{plot(x,y)} to work.

When we define our x and y coordinates, we often use NumPy's \li{linspace(<start>, <stop>, num=50)} function since it returns evenly spaced values over a given interval.
These values are returned in an \li{ndarray}.
So, after importing \li{numpy} and \li{pyplot}, we can generate a basic plot of the function $e^x$ with the following lines of code:
\begin{lstlisting}
# Increase number of samples from default 50 to 501.
x = np.linspace(-2, 3, 501)
y = np.exp(x)
plt.plot(x, y)
plt.show()
\end{lstlisting}

This should display a plot similar to the one shown in Figure \ref{fig:exp_plot}.

\begin{figure}
\includegraphics[width=\textwidth]{exp_plot.pdf}
\caption{A simple plot of $e^x$.}
\label{fig:exp_plot}
\end{figure}

\subsection*{Multiple Plots}
We can also use our interface to display multiple plots at once.
The following code will plot lines with random values at integers from 1 to 10.
Note that NumPy's \li{random.rand} function creates an array of a specified shape and fills it with random samples from a uniform distribution over the interval [0,1).
The given dimensions of the array should be positive and, if no argument is given, a single float is returned.

\begin{lstlisting}
x = np.linspace(1, 10, 10)
y = np.random.rand(10, 10)
plt.plot(x, y[0], x, y[1], x, y[2], x, y[3], x, y[4], x, y[5], x, y[6], x, y[7], x, y[8], x, y[9])
plt.show()
\end{lstlisting}

We just used the plot function to plot several different lines at once.
Alternatively, we can plot each \li{(x, y[i])} pair separately and overlay the plots to produce the same result.
We use a loop to efficiently overlay create each plot.
\begin{lstlisting}
x = np.linspace(1, 10, 10)
y = np.random.rand(10, 10)
for n in y:
    plt.plot(x, n)
plt.show()
\end{lstlisting}
A plot that was generated by this code will be similar to the one shown
in Figure \ref{fig:statemachine}.

\begin{figure}
\includegraphics[width=\textwidth]{statemachine.pdf}
\caption{A plot of $10$ lines with randomly generated $y$ values.}
\label{fig:statemachine}
\end{figure}

Note that a suitable domain and range for your plot is automatically chosen unless you specify otherwise.

\begin{problem} Plot the function $\sin(x)$ from $0$ to $2\pi$ with a red dashed line.
Then plot the function $\cos(x)$ on the same domain with a blue dotted line.
Implement both with a single call to the \li{plot()} function.
Information on how to do this can be found in Appendix \ref{mpltables}.
\end{problem}

\begin{comment}
There are also many functions that we may use to set different values in
the plotting environment. A few examples are shown in Table
\ref{mpl:useful_functions}.

\begin{table}
\begin{center}
\begin{tabular}{|l|p{6cm}|p{4cm}|}

    \hline

    Function & Description & Usage\\

    \hline

    \li{annotate} & adds a commentary at a given point on the plot &
    annotate('text',(x,y))\\

    \li{arrow} & draws an arrow from a given point on the plot &
    arrow(x,y,dx,dy)\\

    \li{axhline} & draws a horizontal line at y from xmin to xmax &
    axhline(y=0, xmin=0, xmax=1)\\

    \li{axvline} & draws a vertical line at x from ymin to ymax &
    axvline(x=0, ymin=0, ymax=1)\\

    \li{axhspan} & draws a rectangle from xmin to xmax and ymin to ymax,
    if no xmin and xmax are given it goes across the plot &
    axhspan(ymin, ymax, xmin=0, xmax=1)\\

    \li{axvspan} & draws a rectangle from ymin to ymax and xmin to xmax,
    if no ymin and ymax are given it goes across the entire plot &
    axvspan(xmin, xmax, ymin=0, ymin=1)\\

    \li{figlegend} & place a legend in the plot & figlegend(handles,
    labels, loc)\\

    \li{grid} & add gridlines & grid()\\

    \li{text} & add text at a given position on the plot &
    text(x,y,'text')\\

    \li{title} & add a title to the plot & title('text')\\

    \li{xlim} & set the x limits, returns current limits if no arguments
    are given & xlim(xmin,xmax)\\

    \li{ylim} & set the y limits, returns current limits if no arguments
    are given & ylim(ymin,ymax)\\

    \li{xticks} & set the location of the tick marks on the x axis,
    returns current locations if no arguments are given & xticks(x)\\

    \li{yticks} & set the location of the tick marks on the y axis,
    returns current locations if no arguments are given & yticks(y)\\

    \li{xlabel} & add a label to the x axis & xlabel('text')\\

    \li{ylabel} & add a label to the y axis & ylabel('text')\\

    \hline

    \end{tabular}
    \end{center}
    \caption{Some Functions to Set Plotting Options}
    \label{mpl:useful_functions}
    \end{table}

	\end{comment}

\begin{problem}
Plot the curve $1/(x-1)$ from $-2$ to $6$ with a magenta dashed line.
Force the plot to only show $y$ values that are between $-6$ and $6$.
Make your line width 5. Once again, more information on how to do this can be found in Appendix \ref{mpltables}.
By default, \li{plot()} will try to make the graph connected.
Correct this so that the graph \emph{appears} to be discontinuous at $x=1$ (as it should be) by modifying the input values.

Your plot should look like the figure below.

\begin{figure}[H]
\includegraphics[width=\textwidth]{soln2.pdf}
\label{fig:problem2}
\end{figure}
\end{problem}

\begin{problem} Plot the curve $\sin(x)\frac{1}{x+1}$ from $0$ to $10$.

Use blue shading under the curve when it is positive and red when it is negative (you may want to consider using the \li{fill_between} command.
Make the line dotted.
Label the x-axis ``x-axis'', the y-axis ``y-axis'',and the plot ``My Plot''.
Enable the grid lines.

Finally, use the \li{scatter} command to include a scatter plot of half of the value of the function at each of its maxima and minima in the range.
Display these points as upward-pointing triangles.
Don't forget to make sure the x limits of the plot are still 0 and 10.

\emph{Helpful Hint}: Since you are working with arrays of discrete values, you will want to find the index values where your $x$ and $y$ values are closest to the actual maxima and minima. As you work, consider the following:
\begin{itemize}
\item How would you manually find maxima and minima of a function?
\item How could you do something similar with your $x$ and $y$ arrays?
\end{itemize}
Your plot should look like the figure below.

\begin{figure}[H]
\includegraphics[width=\textwidth]{soln3.pdf}
\label{fig:problem3}
\end{figure}
\end{problem}

\subsection*{pcolormesh}
Pcolormesh is a function in pyplot which produces a pseudocolor plot of a 2D array.
Think of it as a 3D plot that assigns color rather than height to the result $C$ of a function in $x$ and $y$.
Since our \emph{domain} is now two-dimensional, instead of one-dimensional as before, we use NumPy's \li{meshgrid} function.
This function takes two one-dimensional arrays that represent our input values for $x$ and $y$ respectively, and returns a grid of $(x, y)$ coordinates.
$X$ represents the $x$-coordinates of this grid and $Y$ represents the $y$-coordinates.
We then evaluate our function at these points and return the result in a two-dimensional array $C$, which the pcolormesh function uses to assign colors to the points in our $xy$ domain.

We now use the pcolormesh function to represent the surface $z=\sin(x)\sin(y)$:
\begin{lstlisting}
n = 401
x = np.linspace(-6, 6, n)
y = np.linspace(-6, 6, n)
# Returns a coordinate matrix given coordinate vectors.
X, Y = np.meshgrid(x, y)
C = np.sin(X) * np.sin(Y)
plt.pcolormesh(X, Y, C)
plt.show()
\end{lstlisting}
This plot is shown in Figure \ref{fig:pcmexample}
\begin{figure}
\includegraphics[width=\textwidth]{sinxsiny.png}
\caption{Color plot of $\sin\left(x\right)\times\sin\left(y\right)$.}
\label{fig:pcmexample}
\end{figure}

\begin{problem} Use plt.pcolormesh to plot the absolute value of the function $x^3 +2x^2 -x +3$ on the complex plane with 0 $\leq$ \li{x} $\leq$ 2 and 0 $\leq$ \li{y} $\leq$ 2.
\emph{Helpful Hint}: First create your domain arrays, then convert these to a single array of complex variables to evaluate the function.

Your plot should look like Figure \ref{fig:pcolormesh}.

\begin{figure}[H]
\includegraphics[width=\textwidth]{pcolor2.png}
\caption{Another example of a colorplot.}
\label{fig:pcolormesh}
\end{figure}
\end{problem}

\begin{comment}

Matplotlib can also be used for 3D plotting. The following is an example
of how to use matplotlib to plot the function $z=\sin(x)\sin(y)$ with
both x and y ranging from -6 to 6. The resulting plot is shown in Figure
\ref{mpl:3dplot}. If you change the number of sample points used you
will notice that graphs look much nicer with large numbers of sample
points, but it will also take much longer for your computer to render
the image.

\begin{lstlisting} from mpl_toolkits.mplot3d import Axes3D from
matplotlib import pyplot as plt import numpy as np fig = plt.figure() ax
= fig.gca(projection='3d') x = np.linspace(-6, 6, 301) y =
np.linspace(-6, 6, 301) X, Y = np.meshgrid(x, y) Z = np.sin(X) *
np.sin(Y) ax.plot_surface(X, Y, Z) plt.show() \end{lstlisting}

\begin{figure} \includegraphics[width=\textwidth]{3dplot.pdf} \caption{A
3D plot of $\sin\left(x\right)\times\sin\left(y\right)$.}
\label{mpl:3dplot} \end{figure}

\begin{problem} Plot the function \begin{equation*}
\frac{\cos\left(\sqrt{x^2 + y^2}\right)}{\frac{x^2 + y^2}{10} + 1}
\end{equation*} on $[-10, 10] \times [-10, 10]$. \end{problem}

Matplotlib also allows us to make interactive graphs as follows:

% This example is largely based on one of the examples in the matplotlib
% docs. I have simplified it and changed the way the libraries are
% imported, but we could do a citation anyway.
%
\begin{lstlisting} import numpy as np from matplotlib import pyplot as
plt from matplotlib import widgets as wg ax = plt.subplot(111)
plt.subplots_adjust(bottom=.25) t = np.arange(0., 1., .001) a0 = 5. f0 =
3. s = a0 * np.sin(2 * np.pi * f0 * t) l = plt.plot(t, s)[0]
plt.axis([0, 1, -10, 10]) axfreq = plt.axes([.25, .05, .65, .03]) axamp
= plt.axes([.25, .1, .65, .03]) sfreq = wg.Slider(axfreq, 'Freq', .1,
30., valinit=f0) samp = wg.Slider(axamp, 'Amp', .1, 10., valinit=a0) def
update(val): amp = samp.val freq = sfreq.val l.set_ydata(amp * np.sin(2
* np.pi * freq * t)) plt.draw() sfreq.on_changed(update)
samp.on_changed(update) plt.show() \end{lstlisting} The resulting plot
is shown in Figure \ref{mpl:interact}.

\begin{figure} \includegraphics[width=\textwidth]{interact.pdf}
\caption{A snapshot of an interactive plot made using Matplotlib.}
\label{mpl:interact} \end{figure}

\begin{problem} Modify the code above to add a third slider to
manipulate the phase of the wave shown. Have it range from 0 to $2\pi$
and set the default value to zero. \end{problem}

\end{comment}

\section*{subplots}

We can plot multiple different images within the same figure using the \li{plt.subplot} command.
It takes three arguments: the total number of rows for the figure, the total number of columns for the figure, and the index of the current subplot.
This index starts at 1 and increments across rows first.
Preface the code for each subplot with the following command:
\begin{lstlisting}
plt.subplot(numrows, numcols, fignum)
\end{lstlisting}
If all the argument values are less than 10, commas and spaces can be omitted.

The following example shows plots of $sin x$ and $cos x$ on two different axes within the same figure.
Figure \ref{fig:subplots} is the output from this code.
\begin{lstlisting}
x = np.linspace(-np.pi, np.pi, 400)
y1 = np.sin(x)
y2 = np.cos(x)
# Commas and spaces are omitted between arguments.
plt.subplot(211)
plt.plot(x, y1)
plt.subplot(212)
plt.plot(x, y2)
plt.show()
\end{lstlisting}

\begin{figure}
\includegraphics[width=\textwidth]{subplots.pdf}
\caption{An example of the use of subplots in matplotlib.}
\label{fig:subplots}
\end{figure}

\subsection*{Titles and Labels}

We can add titles to each subplot using the \li{plt.title} function.
The \li{plt.suptitle} function allows us to give the entire figure a title as well.

\begin{problem}
Make a plot with 4 subplots.
In the subplots place graphs of $e^x$, $sin(x)$, $cos(x)$, and $x^2$.
Plot each graph over the interval $(-\pi,\pi)$.
Title each graph accordingly and title the entire figure ``My Different Plots."
\end{problem}

\begin{comment}
%Table \ref{mpl:basics}
\begin{table}
\begin{center}
\begin{tabular}{|l|p{7cm}|p{3cm}|}

    \hline

    Function & Description & Usage\\

    \hline

    \li{bar} & makes a bar graph & bar(left,height)\\

    \li{barh} & makes a horizontal bar graph & barh(bottom,width)\\

    \li{fill} & plots lines with shading under the curve & fill(x,y)\\

    \li{fill\_between} & plots lines with shading between two given y
    values & fill\_ between(x,y1, y2=0)\\

    \li{hist} & plots a histogram from data & hist(data)\\

    \li{pie} & make a pie chart & pie(x)\\

    \li{plot} & plots lines and data on standard axes & plot(x,y)\\

    \li{polar} & plots lines and data on polar axes & polar(theta,r)\\

    \li{loglog} & plots lines and data on logarithmic x and y axes &
    loglog(x,y)\\

    \li{scatter} & plots data, has more options for scatter plots than
    the plot function & scatter(x,y)\\

    \li{semilogx} & plots lines and data with a log scaled x axis &
    semilogx(x,y)\\

    \li{semilogy} & plots lines and data with a log scaled y axis &
    semilogy(x,y)\\

    \li{specgram} & make a spectogram from data & specgram(x)\\

    \li{spy} & plot the sparsity pattern of a 2D array & spy(Z)\\

    \li{triplot} & plot triangulation between given points &
    triplot(x,y)\\

    \hline

    \end{tabular}
    \end{center}
    \caption{Some basic functions in Matplotlib.}
    \label{mpl:basics}
    \end{table}

    \end{comment}

\section*{Mayavi}

matplotlib is designed primarily for 2D plotting.
Although matplotlib can also create basic 3D plots, it is better to create 3D plots using a library that was specifically designed to do so: Mayavi.
Not only is Mayavi designed primarily for 3D plotting, it further has the benefits of easy integration with Python scientific libraries and a user interface that allows for interaction with all data and objects.

We encourage you to visit the formal documentation at \url{http://docs.enthought.com/mayavi/mayavi/} for more information.

Within Mayavi, we will be using the \li{mlab} API which is imported like so:
\begin{lstlisting}
from mayavi import mlab
\end{lstlisting}

\li{mlab} has many plotting functions, which are detailed at \url{http://docs.enthought.com/mayavi/mayavi/auto/mlab_helper_functions.html}.
\begin{comment}
\begin{table}
\begin{center}
\begin{tabular}
{|c|l|}
\hline
Function & Description \\
\hline
\li{barchart} & Produces 3D histogram-like plots\\
\li{contour3d} & Plots level surfaces of functions of three variables\\
\li{flow} & Creates a trajectory of particles following the flow of a vector field\\
\li{imshow} & Use a colormap to view a 2D array as an image\\
\li{mesh} & Plot a surface using \li{(x,y,z)} coordinates supplied as three 2D arrays\\
\li{plot3d} & Draws lines between points\\
\li{points3d} & Plots glyphs (like points) at the coordinates supplied\\
\li{quiver3d} & Generate 3D vector fields\\
\li{surf} & Plot a surface with a 2D array as elevation data\\
\hline
\end{tabular}
\end{center}
\caption{Some plotting functions in \li{mlab}.}
\label{table:mlab_functions}
\end{table}
\end{comment}
All these functions can be ``tested" (which provides an example figure for the function) with the command:
\begin{lstlisting}
mlab.test_<plotting function>()
mlab.show()
\end{lstlisting}
Note that pressing tab after typing \li{mlab.test_} will provide a list of available commands and that, like matplotlib, we use the \li{show()} command to actually view our plot.

For example, to test the \li{fancy_mesh} function, we run the following:
\begin{lstlisting}
mlab.test_fancy_mesh()
mlab.show()
\end{lstlisting}

\begin{comment}
The result should resemble Figure \ref{fig:fancymesh}.

\begin{figure}
\includegraphics[width=\textwidth]{fancymesh.png}
\caption{An example of the fancy mesh plotting function}
\label{fig:fancymesh}
\end{figure}
\end{comment}

In this lab, we will focus our attention on introducing the \li{mesh}, \li{plot3d}, and \li{points3d} functions.

Like matplotlib, \li{mlab} takes a set of data points and plots it.
However, instead of passing just \li{x} and \li{y} coordinates as we did in matplotlib, we also send \li{z} coordinates to produce a 3D plot.

\begin{comment}
\subsection*{surf}
The \li{surf} function is great for simple structures like orthogonal grids
because it will create efficient data structures.
It is common to use 2D arrays returned by \li{np.meshgrid}.

For more complex structures, the \li{mesh} function is more appropriate.
\end{comment}

\subsection*{mesh}
\li{mesh} plots a surface using grid-spaced data.
It expects three 2D NumPy arrays (i.e., a grid of \li{(x,y,z)} coordinates).
As with pcolormesh in matplotlib, it is important that all of the given arrays have the same shape.
The following illustrates a hyperbolic paraboloid, using the colormap \li{RdYlGn}.
\begin{lstlisting}
import numpy as np
from mayavi import mlab

x = np.linspace(-4,4,300)
y = np.linspace(-4,4,300)
X, Y = np.meshgrid(x,y)

Z = X**2/4-Y**2/4

mlab.mesh(X, Y, Z, colormap='RdYlGn')
mlab.show()
\end{lstlisting}

This will produce Figure \ref{fig:mesh_example}
\begin{figure}
\includegraphics[width=\textwidth]{mesh_example.png}
\caption{A hyperbolic paraboloid}
\label{fig:mesh_example}
\end{figure}

\subsection*{plot3d}
\li{plot3d} is used to draw lines between points.
It expects three one-dimensional NumPy arrays to provide the position of the line.

The following illustrates a simple floral design.
\begin{lstlisting}
num = np.pi/1000
pts = np.arange(0, 2*np.pi + num, num)
x = np.cos(pts) * (1 + np.cos(pts*6))
y = np.sin(pts) * (1 + np.cos(pts*6))
z = np.sin(pts*6/11)
mlab.plot3d(x, y, z)
mlab.show()
\end{lstlisting}
This will produce Figure \ref{fig:plot3d}
\begin{figure}
\includegraphics[width=\textwidth]{plot3d.png}
\caption{A simple floral.}
\label{fig:plot3d}
\end{figure}

\subsection*{points3d}
\li{points3d} is similar to \li{plot3d}, except that it plots glyphs (the three-dimensional analogue of points) at the positions of the supplied data, and does not connect them like \li{plot3d} does.
It takes the same arguments, three 1D NumPy arrays that form a line given by a list of \li{(x,y,z)} coordinates.
For this example, we will also use a keyword argument, \li{s}, which provides an associated scalar value for each point given.
This modifies the size and color of the glyphs.
We also use a scale factor to adjust the size of the points so that all of them are visible.

The following illustrates a simple heart - given the right perspective.
\begin{lstlisting}
pts = np.linspace(0, 4 * np.pi, 30)
x = np.sin(2 * pts)
y = np.cos(pts)
z = np.cos(2 * pts)
s = 2+np.sin(pts)
mlab.points3d(x, y, z, s, scale_factor=.15)
mlab.show()
\end{lstlisting}
This will produce Figure \ref{fig:points3d}.

\begin{figure}
\includegraphics[width=\textwidth]{points3d.png}
\caption{A simple heart.}
\label{fig:points3d}
\end{figure}

\subsection*{Changing Looks}
The color of your object can be explicitly defined using the \li{color} keyword argument.
It is specified as a triplet (red, green, blue) of floating points ranging from 0 to 1.
For example, (1.0, 1.0, 1.0) is white.

If, however, you would prefer to vary the colors across your visualization, we recommend using a colormap.
For example, in Figure \ref{fig:mesh_example} we used the colormap \li{RdYlGn} to make our hyperbolic paraboloid a blend from red to cream to green.
Visit \url{http://docs.enthought.com/mayavi/mayavi/mlab_changing_object_looks.html} for a list of all previously established colormaps.
\begin{comment}
\begin{table}
\begin{center}
\begin{tabular}{|l|c|r|}
\hline
\multicolumn{2}{c}
{Colormaps} \\
\hline
\li{Accent} & \li{autumn} & \li{black-white} \\
\li{blue-red} & \li{Blues} & \li{bone} \\
\li{BrBG} & \li{BuGn} & \li{BuPu} \\
\li{cool} & \li{copper} & \li{Dark2} \\
\li{flag} & \li{gist_earth} & \li{gist_gray} \\
\li{gist_heat} & \li{gist_ncar} & \li{gist_rainbow} \\
\li{gist_stern} & \li{gist_yarg} & \li{GnBu} \\
\li{gray} & \li{Greens} & \li{Greys} \\
\li{hot} & \li{hsv} & \li{jet} \\
\li{Oranges} & \li{OrRd} & \li{Paired} \\
\li{Pastel1} & \li{Pastel2} & \li{pink} \\
\li{PiYG} & \li{PRGn} & \li{prism} \\
\li{PuBu} & \li{PuBuGn} & \li{PuOr} \\
\li{PuRd} & \li{PurpLes} & \li{RdBu} \\
\li{RdGy} & \li{RdPu} & \li{RdYlBu} \\
\li{RdYlGn} & \li{Reds} & \li{Setl} \\
\li{Set2} & \li{Set3} & \li{Spectral} \\
\li{spring} & \li{summer} & \li{winter} \\
\li{YlGnBu} & \li{YlGn} & \li{YlOrRd} \\
\li{YlOrBr} \\
\hline
\end{tabular}
\end{center}
\caption{Colormaps}
\label{table:colormaps}
\end{table}
\end{comment}

\subsection*{Useful Features}
Another useful feature in \li{mayavi} is the record feature.
This can be found by clicking the pipeline view in the top left corner of a plotting window.
In the pipeline view there is a round, red button.
Clicking it opens a recorder that keeps track of all the changes made interactively to the visualization, using valid lines of Python code.

\begin{problem}
Provided Grand Canyon topological radar data from NASA, do the following.

\begin{itemize}
\item Reshape the data to be 3601x3601.
\item Cast the data type as \li{float32}.
\item Slice the data, taking the first 1000 rows and columns 900-1900.
\item There is some missing data, so set the minimum of your data equal to the minimum of the the positive data points.
\item Preset the figure using the following commands: \li{mlab.figure(size=(400,320)}, \li{bgcolor = (.16, .28, .46))}
\item Now plot with \li{mlab.surf}, using the colormap \li{gist_earth}, with a \li{warp_scale=.2}, \li{vmin=1200}, and \li{vmax=1610}.
\item Take a smaller view of the canyon using \li{mlab.view(-5.9, 83, 570, [5.3, 20, 238])}.
\end{itemize}

This should produce Figure \ref{fig:GrandCanyon}

\end{problem}

\begin{figure}
\includegraphics[width=\textwidth]{GrandCanyon.png}
\caption{A view of the Grand Canyon}
\label{fig:GrandCanyon}
\end{figure}


\documentclass[11pt]{article}

\usepackage{microtype}

\title{\textbf{Markov chain text generation with inferred parts of speech}}
\author{Travis Mick}

\begin{document}

\maketitle

\begin{abstract}
%%
Markov chains have commonly been used in text generators for applications
such as chat bots, typically implemented by utilizing the likelihood of $k$-word
sequences to inform the iterative production of words.
%
In this document, we outline a technique to improve the quality of such systems'
outputs by leveraging inferred knowledge about the parts of speech for each
word in a sequence.
%
\end{abstract}

\section{Introduction}

%%
Traditional Markov chaining text generation leverages a database indexing the
frequency $F_W(w)$ of each observed $k$-word sequence $w = (w_0, w_1, \dots, w_{k-1})$.
%
The database can be queried for a likely sequence given some prefix.
%
Given a prefix $w' = (w_0, w_1, \dots, w_{j-1})$ of length $j < k$, a candidate
sequence $c$ with prefix $w'$ can be assigned a score as
%
\begin{equation}
\label{eqn:wordscore}
S_W(c) = \frac{F_W(c)}{\sum_{c' \in C_W(w')} F_W(c')}
\end{equation}
where $C_W(w')$ is the set of all possible sequences with the prefix $w'$.
%
Ostensibly, the highest scored sequence is returned by the text generation algorithm,
and an output string can be obtained by iterating this process while maintaining a
rolling window of $k$ words.
%

%%
Unfortunately, the choice of $k$ introduces a trade-off between overfitting and underfitting.
%
Too low of $k$ would result in unnatural sequences of words being produced, and a $k$ too large
would yield few novel results.
%
Therefore, we suggest augmenting the $k$-word Markov chain with additional information about
the parts of speech of the constituent words to increase output quality given small $k$.
%

\section{Generation model}

%%
In addition to a word-sequence frequency database, we will utilize a database of frequencies for
part of speech sequences, as well as a database of frequencies of observations of a word being
used as a particular part of speech.
%

%%
First, let $F_P(p)$ give the frequency of observations of $k$-length part of speech sequences
$p = (p_0, p_1, \dots, {p_{k-1}})$.
%
Then, let $P(w_i, p_i)$ give the frequency of observations of word $w_i$ used as
part of speech $p_i$.
%
We can now define the likelihood of a word sequence $w$ corresponding to a part of speech
sequence $p$ as
%
\begin{equation}
\label{eqn:probcorr}
P_A(w, p) =  \frac{F_P(p)}{\sum_{p' \in X^k} F_P(p')} \cdot \prod_{0 \leq i < k} \frac{P(w_i, p_i)}{\sum_{p'_i \in X} P(w_i, p'_i) }
\end{equation}
%
Where $X$ is the set of all parts of speech and $X^k$ is the set of all possible part of speech
sequences of length $k$.
%
Note that we have utilized information about the correspondence between parts of speech
and words, as well as the likelihood of the part of speech sequence itself.
%
We can leverage this to produce an augmented score for a candidate word sequence $c$
with prefix $w'$.
%
\begin{equation}
\label{eqn:augscore}
S_A(c) = S_W(c) \cdot \max_{p \in X^k} \left( P_A(c, p) \right)
\end{equation}
%
Note that we score the word sequence based on the best possible part of speech assignment
as informed by Eqn.~\ref{eqn:probcorr}.
%

\section{Learning model}

%%
In many applications, Markov chain text generators attempt to learn from observations in
real time.
%
While the learning process to inform $F_W$ is straightforward, we must define a mechanism
to label observed word sequences with parts of speech in order to update $F_P$.

%%
When attempting to assign a part of speech sequence to a word sequence, we must come up with
a scoring mechanism to implement inference, as was done with word sequences in
Eqn.~\ref{eqn:wordscore}.
%
Given a part of speech sequence prefix $p' = (p_0, p_1, \dots, p_{j-1})$ of length $j < k$
and a corresponding word sequence $w$, a candidate part of speech sequence $c$ with prefix $p'$
can be assigned a score as
%
\begin{equation}
\label{eqn:posscore}
S_P(c) = P_A(w, c) \cdot \frac{F_P(c)}{\sum_{c' \in C_P(p')} F_P(c')}
\end{equation}
%
where $C_P(p')$ is the set of all possible sequences with the prefix $p'$.
%
We have again leveraged Eqn.~\ref{eqn:probcorr} to provide weight to both observed
part of speech sequences and the correspondence between parts of speech and words.
%
Note that because our learning model requires prior observations in order to infer
parts of speech for new observations, the system must be trained prior to being
exposed to input.
%

\end{document}