\lab{NumPy and SciPy}{NumPy and SciPy}
\objective{Create and manipulate NumPy arrays and learn features available in NumPy and SciPy.}
\label{lab:NumPyArrays}

\section*{Introduction}

NumPy and SciPy\footnote{SciPy is also the name of a Python coding environment that includes the NumPy and SciPy libraries, as well as IPython, matplotlib, and other tools.}
are the two Python libraries most used for scientific computing.
NumPy is a package for manipulating vectors and arrays, and SciPy is a higher-level library built on NumPy.
The basic object in NumPy is the \emph{array}, which is conceptually similar to a matrix.
However, unlike a matrix, which has two dimensions, a NumPy \li{array} can have arbitrarily many dimensions.
NumPy is optimized for fast array computations.

The convention is to import NumPy as follows.

\begin{lstlisting}
>>> import numpy as np
\end{lstlisting}

\section*{Learning NumPy}
The strategies discussed in the section ``Learning Python'' of Lab \ref{lab:Essential_Python} will also help you learn NumPy and SciPy.
The following online resources are specific to SciPy.
\begin{itemize}
\item Official SciPy Documentation (\url{http://docs.scipy.org/doc/})
\item Sections 1.3 and 1.5 of the SciPy Lecture Notes (\url{http://scipy-lectures.github.io/})
\end{itemize}
The remainder of this lab is a brief summary of the tools available in NumPy and SciPy, beginning with NumPy arrays.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Arrays}

Conceptually, a 1-dimensional array (called a 1-D array) is just a list of numbers.
An $n$-dimensional array (or $n$-D array) is an array of $(n-1)$-dimensional arrays.
Thus, any 2-D array is conceptually a matrix, and a 3-D array is a list of matrices, which can be visualized as a cube of numbers.
Each dimension is called an \emph{axis}.
When a 2-D array is printed to the screen, the 0-axis indexes the rows and the 1-axis indexes the columns.

The NumPy array class is called \li{ndarray}.
The simplest way to create an \li{ndarray} is to define it explicitly using nested lists.
\begin{lstlisting}
# Create a 1-D array
>>> np.array([0, 3, 8, 6, 3.14])
array([0, 3, 8, 6, 3.14])

# Create a 2-D array
>>> ex1 = np.array([[1, 1, 2], [3, 3, 4]])
>>> ex1
array([[1, 1, 2],
       [3, 3, 4]])

\end{lstlisting}

You can view the length of each dimension with the \li{shape} command, and change the shape of an array with the \li{np.reshape()} function.
The number of arguments passed to \li{reshape} tells NumPy the dimension of the new array, and the arguments specify the length of each dimension.
An argument of \li{-1} tells NumPy to make that dimension as long a necessary.
\begin{lstlisting}
# The 0-axis of ex1 has length 2
>>> ex1.shape
(2, 3)
>>> ex1.reshape(3, 2)
array([[1, 1],
       [2, 3],
       [3, 4]])
>>> ex1.reshape(-1)
array([1, 1, 2, 3, 3, 4])
\end{lstlisting}

Array objects also support the usual binary operators, including addition \li{+} and componentwise multiplication \li{*}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Why Use Arrays?}
NumPy arrays are drastically more efficient than nested Python lists for large computations.
In this section we will compare matrix multiplication in Python and NumPy.

\begin{problem}\label{prob:multiply_problem}
A matrix in NumPy is just a 2-D array. How can you multiply two 2-D NumPy arrays as matrices?
Hint: Use the strategies outlined in the section ``Learning Numpy''.
\end{problem}

After doing the previous problem, you should know how to implement matrix multiplication in NumPy.
On the other hand, a matrix in Python can be implemented as a list of lists.
The following function will multiply two such matrices.

\lstinputlisting[style=fromfile]{arr_mult.py}

Table \ref{table:square_times} documents how long\footnote{You can replicate this experiment yourself. In IPython, you can find the execution time of a line of code by prefacing it with \li{\%timeit}.
If you aren't using IPython, you will need
to use the timeit function documented here: \url{https://docs.python.org/2/library/timeit.html}.}
one computer took to square a $k \times k$ matrix in both Python (using the function \li{arr_mult}) and Numpy (using the method you found in Problem \ref{prob:multiply_problem}) for various values of $k$.
As you can see, NumPy is much faster. One reason for this is that algorithms in NumPy are usually implemented in C or Fortran.

\begin{table}
 \begin{tabular}{|c|l|l|} \hline Data Structure & $k$ & Time (s) \\ \hline
 Python List  	& $10\times10$ 	& 0.0002758503 \\
 \cline{2-3} 	& $100\times100$ 	& 0.1336028576 \\
 \cline{2-3} 	& $1000\times1000$ & 200.4009799957 \\
 %& $1\times1$ 		& 0.0000181198 \\
\hline \hline
 NumPy Array 	& $10\times10$ 	& 0.0000109673 \\
 \cline{2-3} 	& $100\times100$ 	& 0.0009210110 \\
 \cline{2-3} 	& $1000\times1000$ & 2.1682999134 \\
 %& $1\times1$ 		& 0.0000298023 \\
 \hline \end{tabular}
 \caption{Time for one computer to square a $k \times k$ matrix in Python and NumPy.}
\label{table:square_times}
\end{table}

%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Data Types}
Unlike Python containers, a NumPy array requires all its elements to have the same data type.
The data types used by NumPy arrays are machine-native and avoid the overhead of Python objects, meaning that they are faster to compute with.
A NumPy \li{int} and a Python \li{int} are not the same; the former has been optimized to speed up numerical computations.
Datatypes supported by NumPy are shown in Table \ref{table:numpytypes}.

\begin{table}
\begin{tabular}{l|l}
Data type & Description
\\ \hline
\li{bool} & Boolean \\
\li{int8} & 8-bit integer \\
\li{int16} & 16-bit integer \\
\li{int32} & 32-bit integer \\
\li{int64} & 64-bit integer \\
\li{int} & Platform integer (depends on platform) \\
\li{uint8} & Unsigned 8-bit integer \\
\li{uint16} & Unsigned 16-bit integer \\
\li{uint32} & Unsigned 32-bit integer \\
\li{uint64} & Unsigned 64-bit integer \\
\li{float16} & Half precision float \\
\li{float32} & Single precision float \\
\li{float64} & Double precision float (also \li{float}) \\
\li{complex64} & Complex number represented by two single precision floats \\
\li{complex128} & Complex number represented by two double precision floats (also \li{complex})
\end{tabular}
\caption{Native numerical data types available in NumPy.}
\label{table:numpytypes}
\end{table}

Here are some examples of how to manipulate data types in NumPy.
\begin{lstlisting}
# Access the data type of an array
>>> ex2 = np.array(range(5))
>>> ex2.dtype
dtype('int64')

# Specify the data type of an array
>>> ex3 = np.array(range(5), dtype=np.float)
>>> ex3.dtype
dtype('float64')
\end{lstlisting}

\begin{comment}
Most floating point numbers cannot be represented perfectly
as a binary fraction and are thus an approximation when stored.
Consider the following example where \li{x_1, x_2, x_3} are all
increasingly better approximations of 1/3, but no matter how
many more digits you're willing to append, the value will never be
exactly 1/3.
\begin{lstlisting}
>>> x_1 = .333
>>> x_2 = .33333
>>> x_3 = .3333333
>>> x_1 == x_2
False
>>> x_2 == x_3
False
\end{lstlisting}
It is almost impossible to accurately test the equality
of elements within two arrays. NumPy provides a special function,
\li{np.allclose}, to check if two arrays are \emph{almost} the same (or
within some specified tolerances).
\begin {lstlisting}
>>> np.allclose(x_1, x_2)
False
>>> np.allclose(x_1, x_2, .001)
True
>>> np.allclose(x_2, x_3)
True
>>> np.allclose(x_2, x_3, .000001)
False
\end{lstlisting}
\emph{Please note that in some rare
cases} \li{np.allclose(a, b)} \emph{will not match} \li{np.allclose(b,
a)}. This is because the equation the function uses for checking
closeness is not symmetric ($\abs{a-b} \leq \mbox{atol} +
\mbox{rtol}*\abs{b}$).

NumPy also allows bitwise operations on arrays
using the standard Python bitwise operators: \li{&}, \li{|}, and \li{^},
as well as \li{&=}, \li{|=}, and \li{^=}.
\end{comment}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Creating Arrays}
In addition to \li{np.array()}, NumPy provides efficient ways to create special kinds of arrays. The function \li{np.arange([start], stop, [step])} is similar to the Python function \li{range()}.

\begin{lstlisting}
>>> np.arange(10, 20, 2)
array([10, 12, 14, 16, 18])
\end{lstlisting}

Use \li{np.linspace(start, stop, num=50)} to create an array of \li{num} numbers evenly spaced in the interval from \li{start} to \li{stop}.
\begin{lstlisting}
>>> np.linspace(0, 32, 4)
array([  0.        ,  10.66666667,  21.33333333,  32.        ])
\end{lstlisting}

We can even create arrays of random values chosen
from probability distributions. These probability distributions are stored
in the submodule \li{np.random}.
\begin{lstlisting}
>>> np.random.rand(5) # uniformly distributed values in [0, 1)
array([ 0.21845499,  0.73352537,  0.28064456,  0.66878454,  0.44138609])
\end{lstlisting}
Some other commonly used functions are \li{np.random.normal}, which samples from the normal distribution, and \li{np.random.randint}, which randomly selects integers from a range.

There are many functions for creating arrays besides these, some of which are described in Table \ref{table:numpycreate}.
See \url{http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html} for more details.

\begin{table}
\begin{tabular}{l|l}
Function & Description
\\ \hline
\li{diag} & Extract a diagonal or construct a diagonal array.\\
\li{empty} & Return a new array of given shape and type, without initializing entries. \\
\li{empty_like} & Return a new array with the same shape and type as a given array. \\
\li{eye} & Return a 2-D array with ones on the diagonal and zeros elsewhere. \\
\li{identity} & Return the identity array. \\
\li{meshgrid} & Return coordinate matrices from two coordinate vectors.\\
\li{ones} & Return a new array of given shape and type, filled with ones. \\
\li{ones_like} & Returns an array of ones with the same shape and type as a given array. \\
\li{zeros} & Return a new array of given shape and type, filled with zeros. \\
\li{zeros_like} & Return an array of zeros with the same shape and type as a given array. \\
\end{tabular}
\caption{Some functions for creating arrays in NumPy.}
\label{table:numpycreate}
\end{table}

\begin{comment}
We can create arrays that
consist entirely of ones or zeros using \li{np.ones()} and
\li{np.zeros()} respectively. We can also allocate an ``empty'' array without initializing its values. The syntax for these commands is similar.
\begin{lstlisting}
>>> np.zeros([2, 4])
array([[ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.]])

>>> np.empty(5)
array([  0.00000000e+000,   1.30586451e-316,   1.17126324e-316,
0.00000000e+000,   2.37151510e-322])
\end{lstlisting}

Note that the final array is not really ``empty''; it just has garbage entries.

The following functions can also be useful for creating arrays.
See the documentation for more information and examples.

If \li{a} is an array, we can create a new empty array with the same shape and data type with the function \li{np.empty_like()}.
The analogous functions \li{np.ones_like} and
\li{np.zeros_like} for create arrays of ones or zeros, respectively.

The function \li{np.identity(n)} returns an $n \times n$ identity matrix.

The function \li{np.eye(N, M, k)} returns an $N \times M$ matrix with ones on the diagonal specified by \li{k} and zeros elsewhere.

If \li{v} is a 1-D array, the function \li{np.diag(v, k)} returns a 2-D array with \li{v} on the diagonal specified by \li{k}.
If \li{v} is a 2-D array, \li{np.diag(v, k)} returns the diagonal of \li{v} specified by \li{k}.

The function \li{np.tile()} constructs an array by repeating an existing array
 in a specified pattern.

We finish this section by demonstrating the function \li{meshgrid}, which is used to create arrays that represent a
two-dimensional grid of coordinates.
\begin{lstlisting}
>>> x = np.arange(3)
>>> y = np.arange(4, 8)
>>> X, Y = np.meshgrid(x, y)
>>> X
array([[0, 1, 2],
       [0, 1, 2],
       [0, 1, 2],
       [0, 1, 2]])
>>> Y
array([[4, 4, 4],
       [5, 5, 5],
       [6, 6, 6],
       [7, 7, 7]])
\end{lstlisting}
\li{X} is the set of x-coordinates of pairs in the cartesian product of \li{x} with \li{y}, and \li{Y} is the set of y-coordinates. That is, \li{(X[i, j], Y[i, j]) = (x[i], y[j])}.
\end{comment}

\begin{comment}
When
creating large grids of points this can use large amounts of RAM, so the
meshgrid function includes the \li{copy} argument which, when set to
false, returns arrays that are views of the original arrays instead of
copies (views and copies are discussed later in this lab). For example,
instead of running \li{np.meshgrid(x, y)} you could run
\li{np.meshgrid(x, y, copy=False)}. This can be much faster, but should
probably only be used if you do not intend to make any additional
changes to the coordinates grid independent of the values stored in the
original arrays.

Every NumPy array has five flags that give important information about
the array. We can check if an array is read-only by looking at its
flags, or we can check how the array's contents are laid out in memory.
Only the \texttt{WRITEABLE} and \texttt{ALIGNED} flags can be modified.
The other flags are read-only. The \texttt{OWNDATA} flag lets us know if
the array is a view or not. We will explain array views later in this
lab. \begin{lstlisting}
>>> i.flags
  C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE :
  True ALIGNED : True UPDATEIFCOPY : False \end{lstlisting} NumPy has
  two different memory orderings for an array. Many array constructors
  allow you to specify an \li{order} keyword that determines the memory
  layout of the array. \begin{description} \item[Row-major:] Arrays are
  stored by rows in continuous memory. Languages such as C and Python
  use row-major indexing. NumPy arrays by default use this indexing
  convention. When an array is stored in memory the addresses to its
  values are stored linearly. In simplest terms, the ordering of an
  array determines whether its rows or its columns are stored in
  contiguous blocks (for example: row 0, row 1, row 2, ... as opposed to
  column 1, column 2, column 3, ...). For an array where the rows are in
  contiguous blocks in memory, performing any sort of operation along a
  column will be slower than performing that same operation along a row
  of the same length. This difference is because of the irregular memory
  access pattern. In NumPy, row-major arrays are identified as \emph{C
  contiguous} (\li{order=`C'}). \item[Column-major:] Arrays are stored
  by columns in contiguous memory. Languages like FORTRAN, MATLAB, and R
  use column-major indexing. For a column major array, operations that
  run along rows are slower. In NumPy, column-major arrays are
  identified as \emph{FORTRAN contiguous} (\li{order=`F'}).
  \end{description} Paying attention to how your arrays are indexed will
  be beneficial to the performance of your algorithms. Speed is not
  usually a critical concern, but it is good to know these things when
  speed does become an issue.
\end{comment}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Indexing and Slicing}
Indexing for a 1-D NumPy array works exactly like indexing for a Python list.
To access a single entry of a multi-dimenional array, say a 3-D array, you should use the syntax \li{f[i, j, k]}.
While the syntax \li{f[i][j][k]} will also work, it is significantly slower because each bracket returns an array slice.
Similarly, slicing an array works just like slicing a list, but with more dimensions.
\begin{lstlisting}
>>> ex4 = np.arange(25).reshape((5,5))
>>> ex4
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
>>> ex4[4, -2]
23

# Extract the lower right 2x2 subarray.
>>> ex4[3:, 3:]
array([[18, 19],
       [23, 24]])

# Extract the second column. The returned array is 1-D.
>>> ex4[:, 1]
array([ 1,  6, 11, 16, 21])

# Reverse the order of the columns.
>>> ex4[:, ::-1]
array([[ 4,  3,  2,  1,  0],
       [ 9,  8,  7,  6,  5],
       [14, 13, 12, 11, 10],
       [19, 18, 17, 16, 15],
       [24, 23, 22, 21, 20]])
\end{lstlisting}

Fancy indexing is a second way to access elements of an array.
There are two types of fancy indexing: boolean and integer.
Boolean indexing uses an array of \li{True} or \li{False} values to
determine which elements of the array to take.
You can create true-false valued arrays by using logical operations on arrays.
See \url{https://scipy-lectures.github.io/intro/numpy/operations.html#other-operations} for examples and \url{http://docs.scipy.org/doc/numpy/reference/routines.logic.html} for a full list of such operations.

\begin{lstlisting}
# Logic operations on arrays result in true-false valued arrays.
>>> ex4mask = np.logical_and(ex4<23, ex4>15)
>>> ex4mask
array([[False, False, False, False, False],
       [False, False, False, False, False],
       [False, False, False, False, False],
       [False,  True,  True,  True,  True],
       [ True,  True,  True, False, False]], dtype=bool)

>>> ex4[ex4mask]
array([16, 17, 18, 19, 20, 21, 22])
\end{lstlisting}

Integer indexing uses Python lists to determine what array values to access.
\begin{lstlisting}
# Return an array of elements ex4[0,0], ex4[0, 0], ex4[2,3], and ex4[4,1].
>>> ex4[[0, 0, 2, 4], [0, 0, 3, 1]]
array([ 0,  0, 13, 21])
# Take the first (0) and last (-1) columns.
>>> ex4[:, [0, -1]]
array([[ 0,  4], [ 5,  9], [10, 14], [15, 19], [20, 24]])
\end{lstlisting}

Fancy indexing can
be used for assignment. For example, we can set all values of an array
that are less than \li{10} to \li{0} in the following way.
\begin{lstlisting}
>>> ex4[ex4<10] = 0
>>> ex4
array([[ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24]])
\end{lstlisting}

% \begin{problem} Generate a random $1000 \times 1000$ array \li{A}. Now
% create an uninitialized array \li{B} with all the same attributes as
% \li{A}. Now do the following 100 times: \begin{itemize} \item
% Overwrite \li{B} so that it is an array of new random values like
% \li{A}. This can be done like this: \li{B[:] = rand(1000,1000)} \item
% Use fancy indexing to make \li{A} the maximum of \li{A} and \li{B}.
% \end{itemize} Now take \li{exp(A)} and have NumPy store the output
% directly in \li{A}. Take the maximum along the vertical axis and
% average the result. The final number should be very close to $e$.
% \end{problem}
%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Array Views and Copies}
NumPy has two ways of returning an array. Slice operations and indexing always return
a \emph{view} and fancy indexing always returns a \emph{copy}.
Understand that even though they may look the same, views and copies are different.

A view of an array is a distinct object from the original array in Python, but it references the same place in memory.
Thus, when you change elements in a view, you also change the array it references.
\begin{lstlisting}
>>> ex5 = np.arange(5)

# Slicing produces a view of k.
>>> view_ex5 = ex5[:]
>>> view_ex5

# Check that m and k are distinct objects in Python.
>>> id(view_ex5) == id(ex5)
False

# Change the third element of view_ex5 to 500
# Changing view_ex5 also changes ex5.
>>> view_ex5[2] = 500
>>> view_ex5
array([  0,   1, 500,   3,   4])
>>> ex5
array([  0,   1, 500,   3,   4])
\end{lstlisting}

A copy of an array is a separate array with its own memory.
Thus, when you change a copy of an array, you do not affect the original array.
Because copying an array uses more memory and also more time, it should only be done when necessary.
An array can be copied using the \li{np.copy()} function (also available as a method of
the array object).

\begin{lstlisting}
>>> copy_ex5 = np.copy(ex5)

# Check that j and n are distinct objects in Python.
>>> id(copy_ex5) == id(ex5)
False

# Change the third element of copy_ex5 to 1000
# Changing copy_ex5 does not affect ex5.
>>> copy_ex5[2] = 500
>>> copy_ex5
array([  0,   1, 1000,   3,   4])
>>> ex5
array([  0,   1, 500,   3,   4])
\end{lstlisting}

Whenever possible, the function \li{np.reshape()} returns a view.
See the documentation for more information.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{More Methods of NumPy Arrays}
Some of the more common methods of NumPy arrays are described in Table \ref{table:ndarraymethods}.
A more comprehensive list can be found at
\url{http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.
html}.

\begin{table}
\centering
\begin{tabular}{l|p{10cm}}
    \hline
    Function & Description \\
    \hline
    \li{all} & returns True if all elements evaluate to True \\
    \li{any} & returns True if any elements evaluate to True \\
    \li{argmax} & indices of maximum value(s) \\
    \li{argmin} & indices of minimum value(s) \\
    \li{argsort} & indices that would sort the array \\
    \li{astype} & casts a copy of an array to a different data type \\
    \li{clip} & restrict values in an array to fit within a given range\\
    \li{conj} & return the complex conjugate of the array \\
    \li{copy} & return a copy of the array\\
    \li{diagonal} & return a given diagonal of the array \\
    \li{dot} & matrix multiplication \\
    \li{max} & max element of the array \\
    \li{mean} & average of the array \\
    \li{min} & minimum element of the array \\
    \li{vstack}
    \li{prod} & product of elements of the array \\
    \li{ravel} & make a flattened version of an array, return a view if
    possible \\
    \li{reshape} & return a view of the array with a changed shape \\
    \li{round} & return a rounded version of the array \\
    \li{sort} & sort the array in place \\
    \li{std} & compute the standard deviation \\
    \li{sum} & sum the elements of the array \\
    \li{swapaxes} & return a view with the given axes swapped \\
    \li{tolist} & return the array represented as a list or nested list\\
    \li{trace} & return the sum of the elements along the main diagonal\\
    \li{var} & return the variance of the array \\
    \hline
    \end{tabular} \caption{A few of the methods of NumPy arrays.}
    \label{table:ndarraymethods} \end{table}

Many of these methods have the option to operate \emph{along an axis}.
When called in this way on an $n$-D array, these methods return an $(n-1)$-D array (the specified axis is collapsed in the evaluation process).

\begin{lstlisting}
>>> ex6 = np.arange(9).reshape(3, 3)
>>> ex6
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])

# Return the maximum value in the array
>>> ex6.max()
8

# Return the maximum values evaluated along the 0-axis
>>> ex6.max(axis=0)
array([6, 7, 8])

# Return the maximum values evaluated along the 1-axis
>>> ex6.max(axis=1)
array([2, 5, 8])
\end{lstlisting}

\begin{problem}
% There should be more problems like this in the vectorization lab. I'll
% include this one here for now though.
Write a function which accepts an integer $n$ as input and does the following:
\begin{enumerate}
\item Creates an $n\times n$ array of \li{floats} randomly chosen from a normal distribution
\item Computes the mean of each row (use a built-in command)
\item Computes the variance of these means (use a built-in command).
\end{enumerate}
As you increase $n$, what happens to the output of
your function? This illustrates one version of
the Law of Large Numbers, about which you will learn more later on.
\end{problem}

\begin{comment}
The transpose \li{np.T} is another efficient NumPy operation that returns an array
view.

\begin{lstlisting}
>>> b = np.arange(16).reshape((4,4))
>>> b.T
array([[ 0,  4,  8, 12],
       [ 1,  5,  9, 13],
       [ 2,  6, 10, 14],
       [ 3,  7, 11, 15]])
\end{lstlisting}

We can also manipulate the axes of an existing array using
\li{np.swapaxes} and \li{np.rollaxis}. Functions can also be applied
across one or more axes using \li{np.apply_across_axis} or
\li{np.apply_across_axes}. The function \li{np.unique} will return the
sorted unique elements of the input array. There are also methods for
constructing arrays from individual subarrays. While they may be useful,
use them very carefully as they can have a very negative impact on
performance. Functions like \li{np.hstack} and \li{np.vstack} will
horizontally or vertically stack the input arrays into a new NumPy
array.

\end{comment}

\begin{comment}
\begin{problem}
Operations that create new arrays are often slower than
operations that create views because allocating an array can be time
consuming.
\begin{enumerate}
\item Create an $1000 \times 1000$ array \li{A} of random floating point values.
\item Compare the speed of the operations \li{A.reshape(A.size)}
and \li{A.flatten()}. Note that we are calling the methods of the arrays.
They are the same as \li{np.reshape(A, A.size)}, and \li{np.flatten(A)}
respectively.
\item Why is there such a difference in speed?
\end{enumerate}
\end{problem}
\end{comment}

\begin{problem}
One good application of array slicing is the Jacobi
method for solving Laplace's equation, which is used to model
steady-state heat flow on a square. This problem will help you implement the Jacobi method.

Make a function that accepts an array and a tolerance as input and does the
following:
\begin{enumerate}
\item Makes a copy of the array.
\item Creates a variable to track the difference between the arrays. Initialize
it as the tolerance parameter your function accepts.
\item While the difference is greater than or equal
to the tolerance
\begin{enumerate}
\item Sets all points that are not on an edge of the new array equal to the average of their 4 immediate neighbors.
Use the values from the old array for this computation.
This should only take one line and should be based entirely on array slicing, NOT iterating through the array.
(Hint: given a 2D array \li{A}, the slice \li{A[1:-1,1:-1]} references
all non-edge entries, \li{A[:-2,1:-1]} references the upper neighbors,
and \li{A[1:-1,2:]} references the right neighbors.)
\item Updates the difference to be the maximum of the absolute value of the new array
minus the old one.
\item Copies the values from the new array into the old
one (without creating a new array).
\end{enumerate}
\end{enumerate}

Now use the following code to generate a plot of your results.
\lstinputlisting[style=fromfile]{laplace_plot.py}
It should resemble the following figure.

\begin{figure} [H]
\includegraphics[width=.75\textwidth]{laplace.pdf}
\end{figure}
\end{problem}

\begin{comment}
\section*{Saving Arrays} It is often useful to save an array as a file.
NumPy provides several easy methods for saving and loading array data.

\begin{table*}
\begin{tabular}{l|l}
\hline
\li{np.save(file, arr)} & Save an array to a binary file \\
\li{np.savez(file, *arrs)} & Save multiple arrays to a binary file \\
\li{np.savetxt(file, arr)} & Save an array to a text file \\
\hline
\end{tabular}
\end{table*}

\begin{table*}
\begin{tabular}{l|l}
\hline
\li{np.load(file)} & Load and return an array from a binary file \\
\li{np.loadtxt(file)} & Load and return an array from text file \\
\hline
\end{tabular}
\end{table*}

Let's practice saving an array to a file and loading it again.
Note that, when saving an array, NumPy automatically appends the extension \li{.npy} if it is not already present.
\begin{lstlisting}
a = np.arange(30)
np.save('test_arr', a)
new_a = np.load('test_arr.npy')
np.savez('test_multi', a=a, new_a=new_a)
arrs = np.load('test_multi.npz')
\end{lstlisting}
The variable \li{arrs} points to a dictionary object with the keys \li{a} and \li{new_a} which reference the arrays that have been saved.
The \li{.npz} file extension is the file type used to store multiple arrays.
\end{comment}

\begin{comment}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection*{Manipulating Arrays}
Sometimes it is best to work on the entire array in single dimension.
NumPy provides several ways to do this, each optimized for a different purpose.

The function \li{np.ravel()} returns a 1-D view of an array. It behave exactly like calling \li{reshape(-1)} on the array, but it executes faster.On the other hand, the array method \li{flatten()} returns a 1-D copy of the array.

Finally, the array method \li{flat} lets you iterate over the array as if it were 1-dimensional. Here is an example.
\begin{lstlisting}
>>> a = np.arange(4).reshape(2, 2)

# Iterate through the rows of a
>>> for i in a:
...     print i,
...
[0 1] [2 3]

# Iterate through the entries of a
>>> for i in a.flat:
...     print i,
...
0 1 2 3
\end{lstlisting}
\end{comment}

\subsection*{Iterating Through Arrays}

Iterating through an array undoes most speed advantages of NumPy.
You should avoid doing this whenever possible.
You can often avoid iterating through arrays by using \emph{array broadcasting} and \emph{universal functions}, discussed in the next section.

It is occasionally valid to iterate through an array.
The function \li{np.nditer()} will create an object that iterates through an array as quickly as possible.

\section*{NumPy and SciPy}
We now introduce some additional features of NumPy and SciPy.

\subsection*{Array Broadcasting}
Many matrix operations make sense only when the two operands have the same shape.
Two examples are addition and component-wise multiplication.
Broadcasting is NumPy's way of extending such operations to accept some (not all) operands with different shapes.
Broadcasting happens automatically whenever it is necessary.

To understand broadcasting, let us look at an example.
\begin{lstlisting}
>>> A = np.ones(3);
>>> A
array([ 1.,  1.,  1.])
>>> B = np.vstack([1, 2, 3])
>>> B
array([[1],
       [2],
       [3]])
>>> A+B
array([[ 2.,  2.,  2.],
       [ 3.,  3.,  3.],
       [ 4.,  4.,  4.]])
\end{lstlisting}
We will now describe the algorithm used to obtain the result for \li{A+B} above.
First, the shapes of \li{A} and \li{B} are lined up, starting at the far right, and 1's are prepended to the shorter tuple. So
\begin{lstlisting}
A 	(1-D array):     3
B	(2-D array): 3 x 1
\end{lstlisting}
becomes
\begin{lstlisting}
A 	(1-D array): 1 x 3
B	(2-D array): 3 x 1
\end{lstlisting}
For broadcasting to work, the dimensions must be compatible; that is, in a given axis, the lengths are equal, or one of the lengths is 1.
Second, the arrays \li{A} and \li{B} are ``stretched'' one axis at a time until the lengths of their axes are the same.
In each axis, if the lengths are different, the smaller array is copied along that axis (or ``stretched''), until it is the size of the larger array.
Conceptually, we are creating new $3 \times 3$ arrays $A'$ and $B'$ where
\[
A' = \left[ \begin{array}{ccc}
1 & 1 & 1\\
1 & 1 & 1\\
1 & 1 & 1 \end{array} \right] \qquad \text{and} \qquad B' =  \left[ \begin{array}{ccc}
1 & 1 & 1\\
2 & 2 & 2\\
3 & 3 & 3\end{array} \right].
\]
Finally, NumPy returns the sum $A'+B'$.

We emphasize that the ``stretching'' in this example is only conceptual, and no new array $A'$ or $B'$ is created. However,
you should still be careful when broadcasting large arrays because you can fill the
RAM on your computer, which can sometimes freeze the system.
For a more detailed description of array broadcasting rules, see
\url{http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html}.

\begin{problem}
Create a $100\times100\times3$ array of random integers taking values in the range
[0, 256]. Such an array can represent an RGB image of $100\times100$ pixels,
where each pixel is associated with an array of three integers indicating the
amounts of red, green, and blue color present in that pixel.
Use array broadcasting to multiply the red and green values by $0.5$.
(Such an operation would tone down the red and green colors and make the
image appear more blue.)
\end{problem}

\subsection*{Universal Functions}

A universal function, or \li{ufunc}, operates on an array elementwise.
It outputs an array of the same shape and datatype as the input array.
Using a universal function is usually much faster than iterating through the array yourself.

Many scalar functions from the Python standard library have a universal analog that operates on arrays.
For example, \li{math.sin()} operates on scalars, and \li{numpy.sin()} operates on arrays.
If you have a simple operation that you want to perform elementwise on an array, you should see if SciPy has a universal function that will do it (it probably does).
For a list of available \li{ufuncs}, see \url{http://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs}.

Most universal functions also allow you to specify an output array, which must have the same shape as the input array.
Doing so can reduce memory allocation.

\begin{lstlisting}
>>> ex7 = np.arange(3, dtype=float)

# Take exp(ex7) and store the result in ex7.
>>> np.exp(ex7, out=ex7)
>>> ex7
array([ 1.        ,  2.71828183,  7.3890561 ])
\end{lstlisting}

\begin{comment}
Other useful examples are \li{max}, \li{min}, \li{absolute}, and \li{average}.
Each of these operations also allows you to specify whether you want to
operate across a particular axis or over the entire array.
For example:

The above example returns a row of A which represents the maximum of all the
rows of A. If we had set \li{axis=1}, it would have taken the maximum of all
the columns. If, for purposes of broadcasting (discussed later) you need the
output of one of these functions to have the same number of dimensions as the
original array, you can also include the argument \li{keepdims=True}.

\end{comment}

Although universal functions also accept scalar inputs, they can be much slower than the corresponding standard library function.
Thus, use standard library functions on scalars and universal functions on arrays.

\begin{comment}

\begin{lstlisting}
>>> timeit np.sin(.5)
1000000 loops, best of 3: 1.37 muµs per loop
>>> timeit np.math.sin(.5)
10000000 loops, best of 3: 144 ns per loop
\end{lstlisting}

We can see that performance increases when we know how to use a
\texttt{ufunc}. However, they should only be used on arrays as they are
not designed to handle single values efficiently.
\end{comment}

\subsection*{Linear Algebra}
Both NumPy and SciPy have a linear algebra library, but the SciPy library is larger. The SciPy linear algebra library is typically imported as follows.

\begin{lstlisting}
from scipy import linalg as la
\end{lstlisting}

\begin{comment}
To shorten the amount of typing, it can be aliased as
\li{from scipy import linalg as la}.
\end{comment}

The linear algebra library contains several functions to construct special
matrices, located in
\li{linalg.special_matrices}. There are also functions that will invert matrices, find determinants and norms, solve linear systems and least squares problems, and find special matrix decompositions. You can read more about the linear algebra capabilities of SciPy in the
documentation for the \li{linalg} module found at
(\url{http://docs.scipy.org/doc/scipy/reference/linalg.html}).

Finally, the \li{scipy.linalg} library has a \li{matrix} class that is very
similar to a 2-D NumPy array. The matrix class can be convenient when doing matrix
operations. However, in such situations we still recommend using NumPy arrrays, which have many of the same features and are also compatible with all other SciPy operations.

\begin{comment}
\begin{problem}
Block ciphers are ciphers that encode blocks of input symbols at a time
instead of one symbol at a time. In the days before computers, the Hill
cipher was the first cipher that allowed practical encoding of more than
three symbols at a time. It was invented by Lester Hill in 1929.
The Hill cipher is considered a classical substitution cipher.
The entire cipher is based on linear algebra and uses a matrix key.
All substitution ciphers work with the 26 letters.
Thus, all our operations will be done mod 26 (modulo 26).
To do this, we introduce you to the \li{\%} operator in Python.
This new operator allows us to do modular arithmetic.
When applied to an array it takes the elementwise mod.

This problem has a number of parts.  You will write a function that
accepts \emph{plaintext} and returns the encoded \emph{ciphertext}.
You will also write a decoder that will accept ciphertext and return
plaintext.

The Encoder: \begin{enumerate}
\item We must first gather the plaintext to encode and a block size, $n$.
We must split this plaintext into blocks removing any spaces.  We need to
convert each character to a number. We use the index of \li{string.lowercase}
(found in the \li{string} module of the Python standard library).
We can easily build a lookup table that will let you easily find the index.
With a lookup table, we map each letter to its index.

\begin{lstlisting}
from string import lowercase
lut = {a:i for i, a in enumerate(lowercase)}
s = "this is a message"
s = "".join(s.split()) #remove all whitespace
map(lut.__getitem__, s) #return a list of indices
[19, 7, 8, 18, 8, 18, 0, 12, 4, 18, 18, 0, 6, 4]
\end{lstlisting}

Another way is to use \li{lowercase.index()} in a loop to find the index
each time.
% \begin{lstlisting}
% >>> indices = []
% >>> for letter in s:
%         indices.append(lowercase.index(letter))
% \end{lstlisting}
We need to split the list of indices into $n$-length arrays and store them
in a list. If the input is not a multiple of $n$, you will need to pad the
input until it is a multiple of $n$. Pick any character to pad the input
(typically it is a rarely used letter). The \li{itertools} module is useful
for this.  One of the common recipes for doing this task is available in
the \li{itertools} documentation.

\begin{lstlisting}
def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return itertools.izip_longest(fillvalue=fillvalue, *args)
\end{lstlisting}

It will be useful to wrap all of this step in a separate function as we will
need to do the same thing when decoding (except for removing whitespace).

\item Find a suitable cipher key.  The keys of a Hill cipher are square
matrices.  Let $K$, be our key. $K$ must be invertible mod 26.
Remember from linear algebra, that the determinant of square matrix will
tell you if that matrix is invertible. To find a matrix that is invertible
mod 26, we need to find a matrix with a determinant that is relatively prime
to 26 (they share no common factors). The Euclidean algorithm can be used to
determine if two numbers are co-prime (i.e. $\gcd(d, 26) = 1$). Write a
function that generate random integer matrices using NumPy, checking the
determinant, and returns a suitable key, $K$.

\item Write a function that will accept a message and a key matrix.
The message should already be broken into $n$ length blocks (you can do this
inside the encode function if needed) and the matrix should be $n \times n$.
A Hill cipher is the dot product of the block with the key.  Return a
ciphertext that is letters (the numbers correspond the indices in
\li{string.lower}).

\item Write a function that will compute $K^{-1} \pmod{26}$.  This will
necessarily be an integer inverse.
Use \li{linalg.inv} to find the inverse of $K$.
Then \[K^{-1} = \det(K)K^{-1}\det(K)^{-1}  \pmod{26}\] where $\det(K)^{-1}$
is the inverse of the determinant mod 26.
You will need to round the determinant to the nearest integer before doing
these steps.  You will also need to round the results of each of your
multiplications.
You can check that you have the integer inverse by checking $KK^{-1} = I$.

\item Write a function that will decode a message given a ciphertext and the
key. You will need to invert the key before decoding.  Break the message into
blocks of size $n$ and calculate the dot product of each block with the
inverted key. Return a plaintext that is letters (the numbers, again,
correspond to indices in \li{string.lower}).
\end{enumerate}

Experiment with your Hill cipher.  If you are in a classroom setting, try
sending encoded messages to friends (they will need the key you used to
encode).
\end{problem}
\end{comment}

\subsection*{Polynomials}
The \li{np.poly1d} object represents a polynomial in NumPy. The constructor is called with the coefficients of the desired polynomial.

\begin{lstlisting}
>>> poly_array = np.poly1d([3, 5, 1, 2, 0, 1])
>>> print poly_array
   5     4     3     2
3 x + 5 x + 1 x + 2 x + 1
\end{lstlisting}

The object \li{poly_array} represents the polynomial $3x^5+5x^4+x^3+2x^2+1$.
NumPy provides many functions to operate on \li{poly1d} objects (see \url{http://docs.scipy.org/doc/numpy/reference/routines.polynomials.polynomial.html}).

\begin{comment}
\begin{table}
\centering
\begin{tabular}{l|l}
Function & Description \\
\hline
\li{np.polyadd} & Add two polynomial arrays \\
\li{np.polyder} & Find the derivative of a polynomial array \\
\li{np.polydiv} & Divide two polynomial arrays \\
\li{np.polyfit} & Find a least squares polynomial fit \\
\li{np.polyint} & Find the integral of a polynomial array \\
\li{np.polymul} & Multiply two polynomial arrays \\
\li{np.polysub} & Subtract two polynomial arrays \\
\li{np.polyval} & Evaluate a polynomial at specific points
\end{tabular} \caption{A few methods of NumPy polynomial arrays.}
\label{table:poly1dmethods}\end{table}
\end{comment}

Here is an example using the \li{poly1d} class. Recall that
\[
e^x = \sum_{n=0}^{\infty} \frac{x^n}{n!}.
\]
The following function evaluates the $n^{th}$ partial sum of this series at the value $a$.

\lstinputlisting[style=fromfile]{exp.py}
\begin{comment}
\begin{lstlisting}
from scipy.misc import factorial
n = 18 # number of terms
p = 1. / factorial(np.arange(18, -1, -1)) # compute coefficients
X = np.random.rand(10000) # where to evaluate the series
P = np.poly1d(p) # make polynomial object
P(X)
\end{lstlisting}
\end{comment}
The last two lines can be condensed by using the following command.

\begin{lstlisting}
np.polyval(p, a)
\end{lstlisting}

\begin{problem}
\leavevmode
\begin{enumerate}
\item Use NumPy's polynomial objects to approximate the following series.
\[
\arcsin x = \sum_{n=0}^{\infty} \frac{\left(2 n\right) ! x^{2 n + 1}}{\left(2 n + 1\right)\left(n!\right)^2 4^n}
\]
This series converges on $(-1, 1)$. Use your series approximation to approximate $\pi$. Hint: think of the powers of $x$ that
are not included in the series as having zero coefficients.

\item The lambert W function is the inverse of $x e^x$.
Its Taylor series is below (note the index starts at 1).
\[
W(x) = \sum_{n=1}^{\infty} \frac{\left(-n\right)^{n-1} x^n}{n!}
\]
This series has a radius of convergence of $\frac{1}{e}$.
Use the series to approximate a number $x$ such that $x e^x = \frac{1}{4}$.
Verify that your approximation is close.

\end{enumerate}
\end{problem}

\begin{comment}
\section*{Useful Functions}
The following table contains a list of useful NumPy functions.
For more information please refer to the NumPy documentation.
\begin{table}
\centering
\begin{tabular}{l|l}
Function & Description \\
\hline
\li{np.intersect1d} & Return the intersection of two flattened arrays. \\
\li{np.union} & Return the union of two flattened arrays. \\
\li{np.diff} & Calculates a discrete difference of order $n$. \\
\li{np.absolute} & Return the elementwise absolute value of an array. \\
\li{np.pad} & \\
\li{np.nonzero} & \\
\li{np.count_nonzero} & \\
\li{np.select} & \\
\li{np.nan} & Represent IEEE NAN (not-a-number). \\
\li{np.inf} & Represent IEEE INF (infinity). \\
\li{np.who} & Print information about defined NumPy arrays in a variable scope. \\
\li{np.unique} & Return a sorted array of unique elements of an array. \\
\end{tabular}
\end{table}
\end{comment}


Source: waynedisonitau123

\subsubsection{Kirchhoff's Theorem}
Denote $L$ be a $n \times n$ matrix as the Laplacian matrix of graph $G$, where $L_{ii} = d(i)$, $L_{ij} = -c$ where $c$ is the number of edge $(i, j)$ in $G$.
\begin{itemize}
    \itemsep-0.5em
    \item The number of undirected spanning in $G$ is $\lvert \det(\tilde{L}_{11}) \rvert$.
    \item The number of directed spanning tree rooted at $r$ in $G$ is $\lvert \det(\tilde{L}_{rr}) \rvert$.
\end{itemize}

\subsubsection{Tutte's Matrix}
Let $D$ be a $n \times n$ matrix, where $d_{ij} = x_{ij}$ ($x_{ij}$ is chosen uniformly at random) if $i < j$ and $(i, j) \in E$, otherwise $d_{ij} = -d_{ji}$. $\frac{rank(D)}{2}$ is the maximum matching on $G$.

\subsubsection{Cayley's Formula}
\begin{itemize}
    \itemsep-0.5em
  \item Given a degree sequence $d_1, d_2, \ldots, d_n$ for each \textit{labeled} vertices, there are $\frac{(n - 2)!}{(d_1 - 1)!(d_2 - 1)!\cdots(d_n - 1)!}$ spanning trees.
  \item Let $T_{n, k}$ be the number of \textit{labeled} forests on $n$ vertices with $k$ components, such that vertex $1, 2, \ldots, k$ belong to different components. Then $T_{n, k} = kn^{n - k - 1}$.
\end{itemize}

\subsubsection{Erdős–Gallai Theorem}
A sequence of non-negative integers $d_1 \geq d_2 \geq \ldots \geq d_n$ can be represented as the degree sequence of a finite simple graph on $n$ vertices if and only if $d_1 + d_2 + \ldots + d_n$ is even and
$$ \sum_{i = 1}^{k}d_i \leq k(k - 1) + \sum_{i = k + 1}^{n}\min(d_i, k) $$
holds for all $1 \leq k \leq n$.


\documentclass[a4paper]{article}

\usepackage{amsmath}
\usepackage{amssymb}

\hbadness=99999
\voffset=-1in
\oddsidemargin=5pt
\textwidth=450pt
\textheight=700pt

\begin{document}
\title{Linear Algebra: Week 3 Notes and Exercises}
\author{Christopher Aytona}
\maketitle

\section{Notes}
Exercises\\

1) Find the length of the vectors $a = \begin{bmatrix}
-10\\
5
\end{bmatrix}$ and $b = \begin{bmatrix}
3\\
3
\end{bmatrix}$

$||\vec{a}|| = \sqrt{(-10)^2+5^2}$

$||\vec{a}|| = \sqrt{125}$

$||\vec{b}|| = \sqrt{(3)^2+(3)^2}$

$||\vec{b}|| = \sqrt{18}$\\

2) Find the magnitude of the vectors $r = \begin{bmatrix}
7\\
-3
\end{bmatrix}$ and $q = \begin{bmatrix}
-3\\
7
\end{bmatrix}$

$||\vec{r}|| = \sqrt{(7)^2+(-3)^2}$

$||\vec{r}|| = \sqrt{58}$

$||\vec{q}|| = \sqrt{(-3)^2+(7)^2}$

$||\vec{q}|| = \sqrt{58}$\\

3) Normalize the vectors in $\mathbb{R}^2$ $a = \begin{bmatrix}
4\\
5
\end{bmatrix}$ and $b = \begin{bmatrix}
5\\
-4
\end{bmatrix}$

$||\vec{a}|| = \sqrt{(4)^2+(5)^2}$

$||\vec{a}|| = \sqrt{41}$

$\frac{\vec{a}}{||\vec{a}||} = \frac{\begin{bmatrix}
4&5
\end{bmatrix}}{\sqrt{41}} = \begin{bmatrix}
\frac{4}{\sqrt{41}}&\frac{5}{\sqrt{41}}
\end{bmatrix}$

$||\vec{b}|| = \sqrt{(5)^2+(-4)^2}$

$||\vec{b}|| = \sqrt{41}$

$\frac{\vec{b}}{||\vec{b}||} = \frac{\begin{bmatrix}
5&-4
\end{bmatrix}}{\sqrt{41}} = \begin{bmatrix}
\frac{5}{\sqrt{41}}&\frac{-4}{\sqrt{41}}
\end{bmatrix}$\\

4) Normalize the vectors in $\mathbb{R}^2$ $q = \begin{bmatrix}
-3\\
6
\end{bmatrix}$ and $p = \begin{bmatrix}
-1\\
-1
\end{bmatrix}$

$||\vec{q}|| = \sqrt{(-3)^2+(6)^2}$

$||\vec{q}|| = \sqrt{41}$

$\frac{\vec{q}}{||\vec{q}||} = \begin{bmatrix}
\frac{-3}{\sqrt{41}}&\frac{6}{\sqrt{41}}
\end{bmatrix}$

$||\vec{p}|| = \sqrt{(-1)^2+(-1)^2}$

$||\vec{p}|| = \sqrt{2}$

$\frac{\vec{p}}{||\vec{p}||} = \begin{bmatrix}
\frac{-1}{\sqrt{2}}&\frac{-1}{\sqrt{2}}
\end{bmatrix}$

5) Find the distance between the points $a = \begin{bmatrix}
-10\\
5
\end{bmatrix}$ and $b = \begin{bmatrix}
3\\
3
\end{bmatrix}$

$\vec{a}-\vec{b} = \begin{bmatrix}
-13\\
2
\end{bmatrix}$

$||\vec{a}-\vec{b}|| = \sqrt{(-13)^2+(2)^2}$

$||\vec{a}-\vec{b}|| = \sqrt{173}$\\

6) Find the distance between the points $q = \begin{bmatrix}
-3\\
7
\end{bmatrix}$ and $r = \begin{bmatrix}
7\\
-3
\end{bmatrix}$

$\vec{q}-\vec{r} = \begin{bmatrix}
-10\\
10
\end{bmatrix}$

$||\vec{q}-\vec{r}|| = \sqrt{(-10)^2+(10)^2}$

$||\vec{q}-\vec{r}|| = \sqrt{200}$\\

10) Find the dot product of the two vectors. Are the two vectors perpendicular?

a) $$v_1 = \begin{bmatrix}
-1\\
-1
\end{bmatrix} v_2 =
\begin{bmatrix}
2\\
-2
\end{bmatrix}$$

$v_1 \cdot v_2 = \begin{bmatrix}
-1\times2+(-1)\times-2
\end{bmatrix} = \begin{bmatrix}
0
\end{bmatrix}$

$||\vec{v_1}|| = \sqrt{(-1)^2+(-1)^2}$

$||\vec{v_1}|| = \sqrt{2}$

$||\vec{v_2}|| = \sqrt{(2)^2+(-2)^2}$

$||\vec{v_2}|| = \sqrt{8}$

$\cos\theta = \frac{\vec{v_1}\cdot\vec{v_2}}{||\vec{v_1}||\cdot||\vec{v_2}||} = \frac{0}{\sqrt{2}\cdot\sqrt{8}} = \frac{0}{\sqrt{16}} = 0$

$\therefore$ it is perpendicular.\\

b) $$a = \begin{bmatrix}
-10\\
5
\end{bmatrix} b = \begin{bmatrix}
3\\
3
\end{bmatrix}$$

$a \cdot b = \begin{bmatrix}
-10\times3+5\times3
\end{bmatrix} =
\begin{bmatrix}
-15
\end{bmatrix}$

$||\vec{a}|| = \sqrt{(-10)^2+(5)^2}$

$||\vec{a}|| = \sqrt{125}$

$||\vec{b}|| = \sqrt{(3)^2+(3)^2}$

$||\vec{b}|| = \sqrt{18} $

$\cos\theta = \frac{\vec{a}\cdot\vec{b}}{||\vec{a}||\cdot||\vec{b}||} = \frac{-15}{\sqrt{125}\cdot\sqrt{18}}$

$\cos\theta = -\frac{15}{\sqrt{2250}}$

$\therefore$ it is not perpendicular.\\

c) $$v = \begin{bmatrix}
3\\
6
\end{bmatrix} w = \begin{bmatrix}
1\\
2
\end{bmatrix}$$

$v \cdot w = \begin{bmatrix}
3\times1+6\times2
\end{bmatrix} = \begin{bmatrix}
15
\end{bmatrix}$

$||\vec{v}|| = \sqrt{(3)^2+(6)^2}$

$||\vec{v}|| = \sqrt{45}$

$||\vec{w}|| = \sqrt{(1)^2 + (2)^2}$

$||\vec{w}|| = \sqrt{5}$

$\cos\theta = \frac{\vec{v}\cdot\vec{w}}{||\vec{v}||\cdot||\vec{w}||} = \frac{15}{\sqrt{45}\cdot\sqrt{5}}$

$\cos\theta = \frac{15}{\sqrt{225}} = 1$

$\therefore$ it is not perpendicular.\\

11) Compute the cosine of the angle formed by the vectors $r = \begin{bmatrix}
7\\
-3
\end{bmatrix}$ and $m = \begin{bmatrix}
2\\
-5
\end{bmatrix}$

$r \cdot m = \begin{bmatrix}
7\times2+(-3)\times-5
\end{bmatrix} = \begin{bmatrix}
29
\end{bmatrix}$

$||\vec{r}|| = \sqrt{(7)^2 + (-3)^2}$

$||\vec{r}|| = \sqrt{58}$

$||\vec{m}|| = \sqrt{(2)^2 + (-5)^2}$

$||\vec{m}|| = \sqrt{29}$

$\cos\theta = \frac{29}{\sqrt{58}\cdot\sqrt{29}} = \frac{29}{1682} = \frac{1}{58} > 0$

$\therefore$ the angle is going to be acute.\\

12) Are the following angles acute, obtuse or right angle?

$\cos\theta_1 = -0.3$, $\cos\theta_2 = 0$, $\cos\theta_3 = 0.75$

$\theta_1$ is an obtuse angle, $\theta_2$ is a right angle, $\theta_3$ is an acute angle.\\

13) Given the target box $(min_1, min_2) = (4,3)$, $(max_1, max_2) = (6,5)$. Find the local point $(\frac{1}{2}, \frac{1}{2})$ in the target box.

$x_1 = (1 - u_1)min_1 + u_1max_1$

$x_1 = (1 - \frac{1}{2})4 + \frac{1}{2}6$

$x_1 = 5$

$x_2 = (1-u_2)min_2 + u_2max_2$

$x_2 = (1-\frac{1}{2})3+\frac{1}{2}5$

$x_2 = 4$

$\therefore$ the local point $(\frac{1}{2},\frac{1}{2})$ in the target box is $(5,4)$.\\

14) Given the target box $(min_1, min_2) = (1,3)$, $(max_1, max_2) = (6,8)$. Find the local point $(\frac{1}{2},\frac{1}{4})$ in the target box.

$x_1 = (1 - \frac{1}{2})1+\frac{1}{2}6$

$x_1 = 3\frac{1}{2}$

$x_2 = (1-\frac{1}{4})3+\frac{1}{4}8$

$x_2 = 1\frac{1}{2}+2$

$x_2 = 3\frac{1}{2}$

$\therefore$ the local point $(\frac{1}{2},\frac{1}{4})$ in the target box is $(3\frac{1}{2}, 3\frac{1}{2})$.\\

\end{document}


\chapter{Generative models for discrete data}

\section{Generative classifier}
\begin{equation}\label{eqn:Generative-classifier}
p(y=c|\vec{x},\vec{\theta})=\dfrac{p(y=c|\vec{\theta})p(\vec{x}|y=c,\vec{\theta})}{\sum_{c'}{p(y=c'|\vec{\theta})p(\vec{x}|y=c',\vec{\theta})}}
\end{equation}

This is called a \textbf{generative classifier}, since it specifies how to generate the data using the \textbf{class conditional density} $p(\vec{x}|y=c)$ and the class prior $p(y=c)$. An alternative approach is to directly fit the class posterior, $p(y=c|\vec{x})$ ;this is known as a \textbf{discriminative classifier}.

\section{Bayesian concept learning}
Psychological research has shown that people can learn concepts from positive examples alone (Xu and Tenenbaum 2007).

We can think of learning the meaning of a word as equivalent to \textbf{concept learning}, which in turn is equivalent to binary classification. To see this, define $f(\vec{x})=1$ if xis an example of the concept $C$, and $f(\vec{x})=0$ otherwise. Then the goal is to learn the indicator function $f$, which just defines which elements are in the set $C$.

\subsection{Likelihood}
\begin{equation}
p(\mathcal{D}|h) \triangleq \left(\dfrac{1}{\text{size}(h)}\right)^N=\left(\dfrac{1}{|h|}\right)^N
\end{equation}

This crucial equation embodies what Tenenbaum calls the \textbf{size principle}, which means the model favours the simplest (smallest) hypothesis consistent with the data. This is more commonly known as \textbf{Occam’s razor}\footnote{\url{http://en.wikipedia.org/wiki/Occam\%27s_razor}}.

\subsection{Prior}
The prior is decided by human, not machines, so it is subjective. The subjectivity of the prior is controversial. For example, that a child and a math professor will reach different answers. In fact, they presumably not only have different priors, but also different hypothesis spaces. However, we can finesse that by defining the hypothesis space of the child and the math professor to be the same, and then setting the child’s prior weight to be zero on certain “advanced” concepts. Thus there is no sharp distinction between the prior and the hypothesis space.

However, the prior is the mechanism by which background knowledge can be brought to bear on a problem. Without this, rapid learning (i.e., from small samples sizes) is impossible.

\subsection{Posterior}
The posterior is simply the likelihood times the prior, normalized.
\begin{equation}
p(h|\mathcal{D}) \triangleq \dfrac{p(\mathcal{D}|h)p(h)}{\sum_{h' \in \mathcal{H}}p(\mathcal{D}|h')p(h')}=\dfrac{\mathbb{I}(\mathcal{D} \in h)p(h)}{\sum_{h' \in \mathcal{H}}\mathbb{I}(\mathcal{D} \in h')p(h')}
\end{equation}
where $\mathbb{I}(\mathcal{D} \in h)p(h)$ is 1 \textbf{iff}(iff and only if) all the data are in the extension of the hypothesis $h$.

In general, when we have enough data, the posterior $p(h|\mathcal{D})$ becomes peaked on a single concept, namely the MAP estimate, i.e.,
\begin{equation}
p(h|\mathcal{D}) \rightarrow \hat{h}^{MAP}
\end{equation}
where $\hat{h}^{MAP}$ is the posterior mode,
\begin{equation}\begin{split}
\hat{h}^{MAP} & \triangleq \arg\max\limits_h p(h|\mathcal{D})=\arg\max\limits_h p(\mathcal{D}|h)p(h) \\
    & =\arg\max\limits_h [\log p(\mathcal{D}|h) + \log p(h)]
\end{split}\end{equation}

Since the likelihood term depends exponentially on $N$, and the prior stays constant, as we get more and more data, the MAP estimate converges towards the \textbf{maximum likelihood estimate} or \textbf{MLE}:
\begin{equation}
\hat{h}^{MLE} \triangleq \arg\max\limits_h p(\mathcal{D}|h)=\arg\max\limits_h \log p(\mathcal{D}|h)
\end{equation}

In other words, if we have enough data, we see that the \textbf{data overwhelms the prior}.

\subsection{Posterior predictive distribution}
The concept of \textbf{posterior predictive distribution}\footnote{\url{http://en.wikipedia.org/wiki/Posterior_predictive_distribution}} is normally used in a Bayesian context, where it makes use of the entire posterior distribution of the parameters given the observed data to yield a probability distribution over an interval rather than simply a point estimate.
\begin{equation}
p(\tilde{\vec{x}}|\mathcal{D}) \triangleq \mathbb{E}_{h|\mathcal{D}}[p(\tilde{\vec{x}}|h)] = \begin{cases}
\sum_h p(\tilde{\vec{x}}|h)p(h|\mathcal{D}) \\
\int p(\tilde{\vec{x}}|h)p(h|\mathcal{D})\mathrm{d}h
\end{cases}
\end{equation}

This is just a weighted average of the predictions of each individual hypothesis and is called \textbf{Bayes model averaging}(Hoeting et al. 1999).

\section{The beta-binomial model}

\subsection{Likelihood}
Given $X \sim \text{Bin}(\theta)$, the likelihood of $\mathcal{D}$ is given by
\begin{equation}
p(\mathcal{D}|\theta)= \text{Bin}(N_1|N,\theta)
\end{equation}

\subsection{Prior}
\begin{equation}
\text{Beta}(\theta|a,b) \propto \theta^{a-1}(1-\theta)^{b-1}
\end{equation}

The parameters of the prior are called \textbf{hyper-parameters}.

\subsection{Posterior}
\begin{equation}\begin{split}\label{eqn:beta-binomial-posterior}
p(\theta|\mathcal{D}) & \propto \text{Bin}(N_1|N_1+N_0,\theta)\text{Beta}(\theta|a,b) \\
    & =\text{Beta}(\theta|N_1+a,N_0b)
\end{split}\end{equation}

Note that updating the posterior sequentially is equivalent to updating in a single batch. To see this, suppose we have two data sets $\mathcal{D}_a$ and $\mathcal{D}_b$ with sufficient statistics $N_1^a,N_0^a$ and $N_1^b,N_0^b$. Let $N_1=N_1^a+N_1^b$ and $N_0=N_0^a+N_0^b$ be the sufficient statistics of the combined datasets. In batch mode we have
\begin{align*}
p(\theta|\mathcal{D}_a,\mathcal{D}_b)& = p(\theta,\mathcal{D}_b|\mathcal{D}_a)p(\mathcal{D}_a) \\
               &\propto p(\theta,\mathcal{D}_b|\mathcal{D}_a) \\
               & = p(\mathcal{D}_b,\theta|\mathcal{D}_a) \\
			   & = p(\mathcal{D}_b|\theta)p(\theta|\mathcal{D}_a) \\
			   & \text{Combine Equation \ref{eqn:beta-binomial-posterior} and \ref{eqn:binomial-pmf}} \\
			   & =\text{Bin}(N_1^b|\theta, N_1^b+N_0^b)\text{Beta}(\theta|N_1^a+a,N_0^a+b) \\
			   & =\text{Beta}(\theta|N_1^a+N_1^b+a,N_0^a+N_0^b+b)
\end{align*}

This makes Bayesian inference particularly well-suited to \textbf{online learning}, as we will see later.

\subsubsection{Posterior mean and mode}
\label{sec:beta-binomial-Posterior-mean-and-mode}
From Table \ref{tab:beta-distribution}, the posterior mean is given by
\begin{equation}
\bar{\theta}=\dfrac{a+N_1}{a+b+N}
\end{equation}

The mode is given by
\begin{equation}
\hat{\theta}_{MAP}=\dfrac{a+N_1-1}{a+b+N-2}
\end{equation}

If we use a uniform prior, then the MAP estimate reduces to the MLE,
\begin{equation}
\hat{\theta}_{MLE}=\dfrac{N_1}{N}
\end{equation}

We will now show that the posterior mean is convex combination of the prior mean and the MLE, which captures the notion that the posterior is a compromise between what we previously believed and what the data is telling us.

\subsubsection{Posterior variance}
The mean and mode are point estimates, but it is useful to know how much we can trust them. The variance of the posterior is one way to measure this. The variance of the Beta posterior is given by
\begin{equation}
\text{var}(\theta|\mathcal{D})=\dfrac{(a+N_1)(b+N_0)}{(a+N_1+b+N_0)^2(a+N_1+b+N_0+1)}
\end{equation}

We can simplify this formidable expression in the case that $N \gg a, b$, to get
\begin{equation}
\text{var}(\theta|\mathcal{D}) \approx \dfrac{N_1N_0}{NNN}=\dfrac{\hat{\theta}_{MLE}(1-\hat{\theta}_{MLE})}{N}
\end{equation}

\subsection{Posterior predictive distribution}
So far, we have been focusing on inference of the unknown parameter(s). Let us now turn our attention to prediction of future observable data.

Consider predicting the probability of heads in a single future trial under a Beta$(a, b)$posterior. We have
\begin{align}
p(\tilde{x}|\mathcal{D})& =\int_0^1 p(\tilde{x}|\theta)p(\theta|\mathcal{D})\mathrm{d}\theta \nonumber \\
                        & =\int_0^1 \theta\text{Beta}(\theta|a,b)\mathrm{d}\theta \nonumber \\
						& =\mathbb{E}[\theta|\mathcal{D}]=\dfrac{a}{a+b}
\end{align}

\subsubsection{Overfitting and the black swan paradox}
Let us now derive a simple Bayesian solution to the problem. We will use a uniform prior, so $a=b=1$. In this case, plugging in the posterior mean gives \textbf{Laplace’s rule of succession}
\begin{equation}
p(\tilde{x}|\mathcal{D})=\dfrac{N_1+1}{N_0+N_1+1}
\end{equation}

This justifies the common practice of adding 1 to the empirical counts, normalizing and then plugging them in, a technique known as \textbf{add-one smoothing}. (Note that plugging in the MAP parameters would not have this smoothing effect, since the mode becomes the MLE if $a=b=1$, see Section \ref{sec:beta-binomial-Posterior-mean-and-mode}.)

\subsubsection{Predicting the outcome of multiple future trials}
Suppose now we were interested in predicting the number of heads, $\tilde{x}$, in $M$ future trials. This is given by
\begin{align}
p(\tilde{x}|\mathcal{D})& =\int_0^1 \text{Bin}(\tilde{x}|M,\theta)\text{Beta}(\theta|a,b)\mathrm{d}\theta \\
                        & =\dbinom{M}{\tilde{x}}\dfrac{1}{B(a,b)}\int_0^1 \theta^{\tilde{x}}(1-\theta)^{M-\tilde{x}}\theta^{a-1}(1-\theta)^{b-1}\mathrm{d}\theta
\end{align}

We recognize the integral as the normalization constant for a Beta$(a+\tilde{x}, M−\tilde{x}+b)$ distribution. Hence
\begin{equation}
\int_0^1 \theta^{\tilde{x}}(1-\theta)^{M-\tilde{x}}\theta^{a-1}(1-\theta)^{b-1}\mathrm{d}\theta=B(\tilde{x}+a,M-\tilde{x}+b)
\end{equation}

Thus we find that the posterior predictive is given by the following, known as the (compound) \textbf{beta-binomial distribution}:
\begin{equation}
Bb(x|a,b,M) \triangleq \dbinom{M}{x}\dfrac{B(x+a,M-x+b)}{B(a,b)}
\end{equation}

This distribution has the following mean and variance
\begin{equation}
\text{mean}=M\dfrac{a}{a+b} \text{ , var}=\dfrac{Mab}{(a+b)^2}\dfrac{a+b+M}{a+b+1}
\end{equation}

This process is illustrated in Figure \ref{fig:beta-binomial-distribution}. We start with a Beta$(2,2)$ prior, and plot the posterior predictive density after seeing $N_1 =3$ heads and $N_0 =17$ tails. Figure \ref{fig:beta-binomial-distribution}(b) plots a plug-in approximation using a MAP estimate. We see that the Bayesian prediction has longer tails, spreading its probability mass more widely, and is therefore less prone to overfitting and blackswan type paradoxes.

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.60]{beta-binomial-distribution-a.png}} \\
\subfloat[]{\includegraphics[scale=.60]{beta-binomial-distribution-b.png}}
\caption{(a) Posterior predictive distributions after seeing $N_1=3,N_0=17$. (b) MAP estimation.}
\label{fig:beta-binomial-distribution}
\end{figure}

\section{The Dirichlet-multinomial model}
In the previous section, we discussed how to infer the probability that a coin comes up heads. In this section, we generalize these results to infer the probability that a dice with $K$ sides comes up as face $k$.

\subsection{Likelihood}
Suppose we observe $N$ dice rolls, $\mathcal{D}=\{x_1,x_2,\cdots,x_N\}$, where $x_i \in \{1,2,\cdots,K\}$. The likelihood has the form
\begin{equation}
p(\mathcal{D}|\vec{\theta}) = \dbinom{N}{N_1 \cdots N_k} \prod\limits_{k=1}^K\theta_k^{N_k} \quad \text{where } N_k=\sum\limits_{i=1}^N \mathbb{I}(y_i=k)
\end{equation}
almost the same as Equation \ref{eqn:multinomial-pmf}.

\subsection{Prior}
\begin{equation}
\text{Dir}(\vec{\theta}|\vec{\alpha}) = \dfrac{1}{B(\vec{\alpha})}\prod\limits_{k=1}^K \theta_k^{\alpha_k-1}\mathbb{I}(\vec{\theta} \in S_K)
\end{equation}

\subsection{Posterior}
\begin{align}
p(\vec{\theta}|\mathcal{D})& \propto p(\mathcal{D}|\vec{\theta})p(\vec{\theta}) \\
     & \propto \prod\limits_{k=1}^K\theta_k^{N_k}\theta_k^{\alpha_k-1} = \prod\limits_{k=1}^K\theta_k^{N_k+\alpha_k-1}\\
	 & =\text{Dir}(\vec{\theta}|\alpha_1+N_1,\cdots,\alpha_K+N_K)
\end{align}

From Equation \ref{eqn:Dirichlet-properties}, the MAP estimate is given by
\begin{equation}\label{eqn:Dir-MAP}
\hat{\theta}_k=\dfrac{N_k+\alpha_k-1}{N+\alpha_0-K}
\end{equation}

If we use a uniform prior, $\alpha_k=1$, we recover the MLE:
\begin{equation}\label{eqn:Dirichlet-multinomial-posterior-MLE}
\hat{\theta}_k=\dfrac{N_k}{N}
\end{equation}

\subsection{Posterior predictive distribution}
The posterior predictive distribution for a single multinoulli trial is given by the following expression:
\begin{align}
p(X=j|\mathcal{D})& =\int p(X=j|\vec{\theta})p(\vec{\theta}|\mathcal{D})\mathrm{d}\vec{\theta} \\
    & =\int p(X=j|\theta_j)\left[\int p(\vec{\theta}_{-j}, \theta_j|\mathcal{D})\mathrm{d}\vec{\theta}_{-j}\right]\mathrm{d}\theta_j \\
	& =\int \theta_jp(\theta_j|\mathcal{D})\mathrm{d}\theta_j=\mathbb{E}[\theta_j|\mathcal{D}]=\dfrac{\alpha_j+N_j}{\alpha_0+N}
\end{align}
where $\vec{\theta}_{-j}$ are all the components of \vec{\theta} except $\theta_j$.

The above expression avoids the zero-count problem. In fact, this form of Bayesian smoothing is even more important in the multinomial case than the binary case, since the likelihood of data sparsity increases once we start partitioning the data into many categories.

\section{Naive Bayes classifiers}
\label{sec:NBC}
Assume the features are \textbf{conditionally independent} given the class label, then the class conditional density has the following form
\begin{equation}
p(\vec{x}|y=c,\vec{\theta})=\prod\limits_{j=1}^D p(x_j|y=c,\vec{\theta}_{jc})
\end{equation}

The resulting model is called a \textbf{naive Bayes classifier}(NBC).

The form of the class-conditional density depends on the type of each feature. We give some possibilities below:
\begin{itemize}
\item{In the case of real-valued features, we can use the Gaussian distribution: $p(\vec{x}|y,\vec{\theta})=\prod_{j=1}^D \mathcal{N}(x_j|\mu_{jc},\sigma_{jc}^2)$, where $\mu_{jc}$ is the mean of feature $j$ in objects of class $c$, and $\sigma_{jc}^2$ is its variance.}
\item{In the case of binary features, $x_j \in \{0,1\}$, we can use the Bernoulli distribution: $p(\vec{x}|y,\vec{\theta})=\prod_{j=1}^D \text{Ber}(x_j|\mu_{jc})$, where $\mu_{jc}$ is the probability that feature $j$ occurs in class $c$. This is sometimes called the \textbf{multivariate Bernoulli naive Bayes} model. We will see an application of this below.}
\item{In the case of categorical features, $x_j \in \{a_{j1},a_{j2},\cdots, a_{jS_j}\}$, we can use the multinoulli distribution: $p(\vec{x}|y,\vec{\theta})=\prod_{j=1}^D \text{Cat}(x_j|\vec{\mu}_{jc})$, where $\vec{\mu}_{jc}$ is a histogram over the $K$ possible values for $x_j$ in class $c$.}
\end{itemize}

Obviously we can handle other kinds of features, or use different distributional assumptions. Also, it is easy to mix and match features of different types.

\subsection{Optimization}
\label{sec:NBC-Optimization}
We now discuss how to “train” a naive Bayes classifier. This usually means computing the MLE or the MAP estimate for the parameters. However, we will also discuss how to compute the full posterior, $p(\vec{\theta}|\mathcal{D})$.

\subsubsection{MLE for NBC}
The probability for a single data case is given by
\begin{equation}\begin{split}
p(\vec{x}_i,y_i|\vec{\theta}) & =p(y_i|\vec{\pi})\prod\limits_j p(x_{ij}|\vec{\theta}_j) \\
  & =\prod\limits_c \pi_c^{\mathbb{I}(y_i=c)} \prod\limits_j\prod\limits_c p(x_{ij}|\vec{\theta}_{jc})^{\mathbb{I}(y_i=c)}
\end{split}\end{equation}

Hence the log-likelihood is given by
\begin{equation}
p(\mathcal{D}|\vec{\theta})=\sum\limits_{c=1}^C{N_c\log\pi_c}+ \sum\limits_{j=1}^D{\sum\limits_{c=1}^C{\sum\limits_{i:y_i=c}{\log p(x_{ij}|\vec{\theta}_{jc})}}}
\end{equation}
where $N_c \triangleq \sum\limits_i \mathbb{I}(y_i=c)$ is the number of feature vectors in class $c$.

We see that this expression decomposes into a series of terms, one concerning $\vec{\pi}$, and $DC$ terms containing the $\theta_{jc}$’s. Hence we can optimize all these parameters separately.

From Equation \ref{eqn:Dirichlet-multinomial-posterior-MLE}, the MLE for the class prior is given by
\begin{equation}
\hat{\pi}_c=\dfrac{N_c}{N}
\end{equation}

The MLE for $\theta_{jc}$’s depends on the type of distribution we choose to use for each feature.

In the case of binary features, $x_j \in \{0,1\}$, $x_j|y=c \sim \text{Ber}(\theta_{jc})$, hence
\begin{equation}
\hat{\theta}_{jc}=\dfrac{N_{jc}}{N_c}
\end{equation}
where $N_{jc} \triangleq \sum\limits_{i:y_i=c} \mathbb{I}(y_i=c)$ is the number that feature $j$ occurs in class $c$.

In the case of categorical features, $x_j \in \{a_{j1},a_{j2},\cdots, a_{jS_j}\}$, $x_j|y=c \sim \text{Cat}(\vec{\theta}_{jc})$, hence
\begin{equation}
\hat{\vec{\theta}}_{jc}=(\dfrac{N_{j1c}}{N_c},\dfrac{N_{j2c}}{N_c}, \cdots, \dfrac{N_{jS_j}}{N_c})^T
\end{equation}
where $N_{jkc} \triangleq \sum\limits_{i=1}^N \mathbb{I}(x_{ij}=a_{jk}, y_i=c)$ is the number that feature $x_j=a_{jk}$ occurs in class $c$.

\subsubsection{Bayesian naive Bayes}
\label{sec:Bayesian-naive-Bayes}
Use a Dir$(\vec{\alpha})$ prior for $\vec{\pi}$.

In the case of binary features, use a Beta$(\beta0,\beta1)$ prior for each $\theta_{jc}$; in the case of categorical features, use a Dir$(\vec{\alpha})$ prior for each  $\vec{\theta}_{jc}$. Often we just take $\vec{\alpha}=\vec{1}$ and $\vec{\beta}=\vec{1}$, corresponding to \textbf{add-one} or \textbf{Laplace smoothing}.

\subsection{Using the model for prediction}
The goal is to compute
\begin{equation}\begin{split}
y=f(\vec{x}) & =\arg\max\limits_{c}{P(y=c|\vec{x},\vec{\theta})} \\
   & =P(y=c|\vec{\theta})\prod_{j=1}^D P(x_j|y=c,\vec{\theta})
\end{split}\end{equation}

We can the estimate parameters using MLE or MAP, then the posterior predictive density is obtained by simply plugging in the parameters $\bar{\vec{\theta}}$(MLE) or $\hat{\vec{\theta}}$(MAP).

Or we can use BMA, just integrate out the unknown parameters.

\subsection{The log-sum-exp trick}
when using generative classifiers of any kind, computing the posterior over class labels using Equation \ref{eqn:Generative-classifier} can fail due to \textbf{numerical underflow}. The problem is that $p(\vec{x}|y=c)$ is often a very small number, especially if \vec{x} is a high-dimensional vector. This is because we require that $\sum_{\vec{x}}p(\vec{x}|y)=1$, so the probability of observing any particular high-dimensional vector is small. The obvious solution is to take logs when applying Bayes rule, as follows:
\begin{equation}
\log p(y=c|\vec{x},\vec{\theta})=b_c-\log\left(\sum\limits_{c'}e^{b_{c'}}\right)
\end{equation}
where $b_c \triangleq \log p(\vec{x}|y=c,\vec{\theta})+\log p(y=c|\vec{\theta})$.

We can factor out the largest term, and just represent the remaining numbers relative to that. For example,
\begin{equation}\begin{split}
\log(e^{-120}+e^{-121}) & =\log(e^{-120}(1+e^{-1})) \\
   & =\log(1+e^{-1})-120
\end{split}\end{equation}

In general, we have
\begin{equation}
\sum\limits_{c}e^{b_{c}}=\log\left[(\sum e^{b_c-B})e^B\right]=\log\left(\sum e^{b_c-B}\right)+B
\end{equation}
where $B \triangleq \max\{b_c\}$.

This is called the \textbf{log-sum-exp} trick, and is widely used.

\subsection{Feature selection using mutual information}
Since an NBC is fitting a joint distribution over potentially many features, it can suffer from overfitting. In addition, the run-time cost is $O(D)$, which may be too high for some applications.

One common approach to tackling both of these problems is to perform \textbf{feature selection}, to remove “irrelevant” features that do not help much with the classification problem. The simplest approach to feature selection is to evaluate the relevance of each feature separately, and then take the top K,whereKis chosen based on some tradeoff between accuracy and complexity. This approach is known as \textbf{variable ranking}, \textbf{filtering}, or \textbf{screening}.

One way to measure relevance is to use mutual information (Section \ref{sec:Mutual-information}) between feature $X_j$ and the class label $Y$
\begin{equation}
\mathbb{I}(X_j,Y)=\sum\limits_{x_j}{\sum\limits_{y}{p(x_j,y)\log \dfrac{p(x_j,y)}{p(x_j)p(y)}}}
\end{equation}

If the features are binary, it is easy to show that the MI can be computed as follows
\begin{equation}
\mathbb{I}_j = \sum\limits_c \left[\theta_{jc}\pi_c\log{\dfrac{\theta_{jc}}{\theta_j}}+(1-\theta_{jc})\pi_c\log{\dfrac{1-\theta_{jc}}{1-\theta_j}}\right]
\end{equation}
where $\pi_c=p(y=c)$, $\theta_{jc}=p(x_j=1|y=c)$, and $\theta_j=p(x_j=1)=\sum_{c} \pi_c\theta_{jc}$.

\subsection{Classifying documents using bag of words}
\textbf{Document classification} is the problem of classifying text documents into different categories.

\subsubsection{Bernoulli product model}
One simple approach is to represent each document as a binary vector, which records whether each word is present or not, so $x_{ij} =1$ iff word $j$ occurs in document $i$, otherwise $x_{ij}=0$. We can then use the following class conditional density:
\begin{equation}\begin{split}
p(\vec{x}_i|y_i=c,\vec{\theta}) & =\prod\limits_{j=1}^D \mathrm{Ber}(x_{ij}|\theta_{jc}) \\
  & =\prod\limits_{j=1}^D \theta_{jc}^{x_{ij}}(1-\theta_{jc})^{1-x_{ij}}
\end{split}\end{equation}

This is called the \textbf{Bernoulli product model}, or the \textbf{binary independence model}.

\subsubsection{Multinomial document classifier}
However, ignoring the number of times each word occurs in a document loses some information (McCallum and Nigam 1998). A more accurate representation counts the number of occurrences of each word. Specifically, let $\vec{x}_i$ be a vector of counts for document $i$, so $x_{ij} \in \{0,1,\cdots,N_i\}$, where $N_i$ is the number of terms in document $i$(so $\sum\limits_{j=1}^D x_{ij}=N_i$). For the class conditional densities, we can use a multinomial distribution:
\begin{equation}\label{eqn:Multinomial-document-classifier}
p(\vec{x}_i|y_i=c,\vec{\theta})=\text{Mu}(\vec{x}_i|N_i,\vec{\theta}_c)=\dfrac{N_i!}{\prod_{j=1}^D x_{ij}!}\prod\limits_{j=1}^D \theta_{jc}^{x_{ij}}
\end{equation}
where we have implicitly assumed that the document length $N_i$ is independent of the class. Here $θ_{jc}$ is the probability of generating word $j$ in documents of class $c$; these parameters satisfy the constraint that $\sum_{j=1}^D \theta_{jc}=1$ for each class c.

Although the multinomial classifier is easy to train and easy to use at test time, it does not work particularly well for document classification. One reason for this is that it does not take into account the \textbf{burstiness} of word usage. This refers to the phenomenon that most words never appear in any given document, but if they do appear once, they are likely to appear more than once, i.e., words occur in bursts.

The multinomial model cannot capture the burstiness phenomenon. To see why, note that Equation \ref{eqn:Multinomial-document-classifier} has the form $\theta_{jc}^{x_{ij}}$, and since $\theta_{jc} \ll 1$ for rare words, it becomes increasingly unlikely to generate many of them. For more frequent words, the decay rate is not as fast. To see why intuitively, note that the most frequent words are function words which are not specific to the class, such as “and”, “the”, and “but”; the chance of the word “and” occuring is pretty much the same no matter how many time it has previously occurred (modulo document length), so the independence assumption is more reasonable for common words. However, since rare words are the ones that matter most for classification purposes, these are the ones we want to model the most carefully.

\subsubsection{DCM model}
Various ad hoc heuristics have been proposed to improve the performance of the multinomial document classifier (Rennie et al. 2003). We now present an alternative class conditional density that performs as well as these ad hoc methods, yet is probabilistically sound (Madsen et al. 2005).

Suppose we simply replace the multinomial class conditional density with the \textbf{Dirichlet Compound Multinomial} or \textbf{DCM} density, defined as follows:
\begin{equation}\begin{split}
p(\vec{x}_i|y_i=c,\vec{\alpha}) & =\int \text{Mu}(\vec{x}_i|N_i,\vec{\theta}_c)\text{Dir}(\vec{\theta}_c|\vec{\alpha}_c) \\
   & =\dfrac{N_i!}{\prod_{j=1}^D x_{ij}!}\prod\limits_{j=1}^D\dfrac{B(\vec{x}_i+\vec{\alpha}_c)}{B(\vec{\alpha}_c)}
\end{split}\end{equation}

(This equation is derived in Equation TODO.) Surprisingly this simple change is all that is needed to capture the burstiness phenomenon. The intuitive reason for this is as follows: After seeing one occurence of a word, say wordj, the posterior counts on θj gets updated, making another occurence of wordjmore likely. By contrast, ifθj is fixed, then the occurences of each word are independent. The multinomial model corresponds to drawing a ball from an urn with Kcolors of ball, recording its color, and then replacing it. By contrast, the DCM model corresponds to drawing a ball, recording its color, and then replacing it with one additional copy; this is called the \textbf{Polya urn}.

Using the DCM as the class conditional density gives much better results than using the multinomial, and has performance comparable to state of the art methods, as described in (Madsen et al. 2005). The only disadvantage is that fitting the DCM model is more complex; see (Minka 2000e; Elkan 2006) for the details.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage[letterpaper, total={7.5in, 9in}]{geometry}

\title{Linear Algebra}
\author{Grant Smith}
\date{Spring 2022}

\begin{document}

\maketitle

\section{The Four Subspaces}

First of all, we know that there is a row space and a column space. We also know that the column space of $A$ is the row space of $A^T$ and the row space of $A$ is the column space of $A^T$. Next are questions about the null space.  But first, I want to discuss how Gaussian Elimination changes (or doesn't change) the row space.
\subsection{Gaussian Elimination}
Gaussian Elimination does not change the row space. We'll prove this in a couple steps. First of all, we have to prove that Gaussian Elimination doesn't generate rows outside of the original row space, and then we have to prove that it doesn't remove anything from the original space. First, we show that it doesn't add anything to the space. This is because whenever Gaussian Elimination is being used, the rows are just being added and scaled. Which means all of the row vectors in the end are linear combinations of the original row vectors we had. So we know that it didn't add any row vectors that weren't there previously. Next we need to prove that it doesn't take anything away. And right now, I don't know how to prove this.

Also, just to be clear, Gaussian Elimination does indeed change the column space, but it doesn't change it's dimension. We'll prove the part about it not changing the dimension, but the best way to see that it does change the column space is to think about a matrix:

$$\begin{bmatrix}
  1  & 5 \\
  2  & 10 \\
   \end{bmatrix}$$

And with Gaussian Elimination, that is:

$$\begin{bmatrix}
    1  & 5 \\
    0  & 0 \\
\end{bmatrix}$$

Which certainly has a different column space, but the column space has the same dimension, which, again, we will show soon.
\subsection{The Row Space and the Column Space dimensions are equal}
First of all, let's start with an $m \times n$ matrix called $A$.

$$A = \begin{bmatrix}
    a_{1,1} & \dots & a_{1,n} \\
    \vdots & \ddots &  \\
    a_{m,1}&  & a_{m,n} \\
\end{bmatrix}$$

Notice that the dimension of the row space and the column space are both at most $\min (m,n)$. This is because if $m$ is bigger, then we have $m$ rows, but they are only $n$ wide, so they can't span a space with higher dimension than $n$.  And for the columns, we only have $n$ columns, so they can't span a space of dimension $> n$. The same argument would work for if $n$ was bigger. This is nice because it sets a ceiling on the two dimensions, and they are the same for the rows and columns. So this is just a hint and a nice fact, but it doesn't prove that they're the same.

Next, let's say the column space is $b$ dimensional, and $b \leq \min (m,n)$. Then there is a basis of $b$ column vectors (each of which is $m$-dimensional) that can be used to create our matrix. So let's arrange those into a matrix $B$.  And since those column vectors span the row space of $A$, we can write $A = BC$ where the columns of $C$ are the linear weightings of $B$'s columns that make $A$'s columns.

$$A = \begin{bmatrix}
    a_{1,1} & \dots & a_{1,b}  & \dots & a_{1,n} \\
    \vdots & \ddots &  \vdots & \ddots & \vdots \\
    a_{b,1} & \dots & a_{b,b} & \dots & a_{b,n} \\
    \vdots & \ddots & \vdots & \ddots & \vdots \\
    a_{m,1}& \dots & a_{m,b} & \dots & a_{m,n} \\
\end{bmatrix} = \begin{bmatrix}
    b_{1,1} & \dots & b_{1,b}\\
    \vdots & \ddots &  \vdots \\
    b_{b,1} & \dots & b_{b,b} \\
    \vdots & \ddots & \vdots \\
    b_{m,1}& \dots & b_{m,b} \\
\end{bmatrix}\begin{bmatrix}
    c_{1,1} & \dots & c_{1,b}  & \dots & c_{1,n} \\
    \vdots & \ddots &  \vdots & \ddots & \vdots \\
    c_{b,1} & \dots & c_{b,b} & \dots & c_{b,n} \\

\end{bmatrix}$$

But we could also view this as a statement about $C$'s rows spanning $A$'s rows. So that means that $dim (row(A)) \leq dim(col A)$

Let's do a similar argument, but by spanning the row space initially instead of the column space. let's say the row space is $e$ dimensional. Then we can write the rows of $A$ as linear combinations of $e$ row vectors, and let's put those into a matrix $E$. Now we can say that $A = DE$ where $D$ contains the weightings of $E$'s row vectors that make $A$'s row vectors.  Thus, we have

$$A = \begin{bmatrix}
    a_{1,1} & \dots & a_{1,e}  & \dots & a_{1,n} \\
    \vdots & \ddots &  \vdots & \ddots & \vdots \\
    a_{e,1} & \dots & a_{e,e} & \dots & a_{e,n} \\
    \vdots & \ddots & \vdots & \ddots & \vdots \\
    a_{m,1}& \dots & a_{m,e} & \dots & a_{m,n} \\
\end{bmatrix} = \begin{bmatrix}
    d_{1,1} & \dots & d_{1,e}\\
    \vdots & \ddots &  \vdots \\
    d_{e,1} & \dots & d_{e,e} \\
    \vdots & \ddots & \vdots \\
    d_{m,1}& \dots & d_{m,e} \\
\end{bmatrix}\begin{bmatrix}
    e_{1,1} & \dots & e_{1,e}  & \dots & e_{1,n} \\
    \vdots & \ddots &  \vdots & \ddots & \vdots \\
    e_{e,1} & \dots & e_{e,e} & \dots & e_{e,n} \\

\end{bmatrix}$$

Which means that $A$'s columns can be spanned by $e$ or fewer columns, where $e$ was the dimension of the row space. This means that $dim (row(A)) \geq dim(col A)$ but now we have both sides of the inequality, so the two dimensions are equal.

And this means that $e = b$, so the two factorizations above are the same dimension. They don't have to be the exact same because there are many available bases for the two spaces, but they are at least the same dimension.

Thanks to Sean Owen on Quora for this explanation.

\subsection{Back To Gauss Jordan}

Now that we know the dimension of the row space is equal to the dimension of the column space, We can combine that with what we know about Gaussian Elimination not changing the row space: if the row space dimension and the column space dimension of any matrix are equal, and Gaussian Elimination does not change the row space at all (let alone its dimension), then Gaussian Elimination doesn't change the column space dimension.

\subsection{Back to the end of the Row Space and The Column Space Being Equal Section}

At this point, we could make one of those two matrices orthogonal or orthonormal. That would be nice because orthogonal and orthonormal bases are great. The problem is we can only really do one at a time. For example, if we make $C (or E)$ orthonormal (and there are infinite options of orthonormal bases), then that specifies the weightings on $B (or D)$.  And vice versa if we chose to make the left matrix orthonormal, then that would change the numbers in the right matrix.

But! If you choose your orthonormal vectors just right on either side (i.e. choose the right one of the infinite options), then you can allow the vectors on the other side to be orthogonal as well, and you can factor out their sizes to make them orthonormal. And this is the SVD.  If we choose just the right orthonormal basis of the row space, then we can actually make the column space orthonormal too.  And same for the other way around. We'll talk more about how to do this, but for now just know that there is something coming.
\newpage
\section{LU Decomposition}

We will do the LU decomposition of:

$$A = \begin{bmatrix}
  1  & 0 & 3 \\
   1 & -1 & -3 \\
  5  & -1 & 6 \\
   \end{bmatrix}
$$

$$
\begin{bmatrix}
    1 & 0 & 0 \\
    0 & 1 & 0 \\
    0 & 0 & 1 \\
\end{bmatrix}
\begin{bmatrix}
    1  & 0 & 3 \\
    1 & -1 & -3 \\
    5  & -1 & 6 \\
\end{bmatrix} =
\begin{bmatrix}
    1 & 0 & 0 \\
    0 & 1 & 0 \\
    0 & 0 & 1 \\
\end{bmatrix}
\begin{bmatrix}
    1  & 0 & 3 \\
    1 & -1 & -3 \\
    5  & -1 & 6 \\
\end{bmatrix}
  $$

  $$
  \begin{bmatrix}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & 1 \\
  \end{bmatrix}
  A =
  \begin{bmatrix}
      1 & 0 & 0 \\
      0 & 1 & 0 \\
      0 & 0 & 1 \\
  \end{bmatrix}
  \begin{bmatrix}
    0 & 0 & 1 \\
    0 & 1 & 0 \\
    1 & 0 & 0 \\
\end{bmatrix}
  \begin{bmatrix}
    5  & -1 & 6 \\
      1 & -1 & -3 \\
      1  & 0 & 3 \\
  \end{bmatrix}
    $$

    $$
    \begin{bmatrix}
        1 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & 1 \\
    \end{bmatrix}
    A =
    \begin{bmatrix}
      0 & 0 & 1 \\
      0 & 1 & 0 \\
      1 & 0 & 0 \\
  \end{bmatrix}
    \begin{bmatrix}
      5  & -1 & 6 \\
        1 & -1 & -3 \\
        1  & 0 & 3 \\
    \end{bmatrix}
      $$

      $$
      \begin{bmatrix}
        0 & 0 & 1 \\
        0 & 1 & 0 \\
        1 & 0 & 0 \\
    \end{bmatrix}
      \begin{bmatrix}
          1 & 0 & 0 \\
          0 & 1 & 0 \\
          0 & 0 & 1 \\
      \end{bmatrix}
      A =
      \begin{bmatrix}
        0 & 0 & 1 \\
        0 & 1 & 0 \\
        1 & 0 & 0 \\
    \end{bmatrix}
      \begin{bmatrix}
        0 & 0 & 1 \\
        0 & 1 & 0 \\
        1 & 0 & 0 \\
    \end{bmatrix}
      \begin{bmatrix}
        5  & -1 & 6 \\
          1 & -1 & -3 \\
          1  & 0 & 3 \\
      \end{bmatrix}
        $$

        $$
        \begin{bmatrix}
          0 & 0 & 1 \\
          0 & 1 & 0 \\
          1 & 0 & 0 \\
      \end{bmatrix}
        A =
        \begin{bmatrix}
            1 & 0 & 0 \\
            0 & 1 & 0 \\
            0 & 0 & 1 \\
        \end{bmatrix}
        \begin{bmatrix}
          5  & -1 & 6 \\
            1 & -1 & -3 \\
            1  & 0 & 3 \\
        \end{bmatrix}
          $$
          $$
          P
          A =
          \begin{bmatrix}
            1 & 0 & 0 \\
            0 & 1 & 0 \\
            0 & 0 & 1 \\
        \end{bmatrix}
          \begin{bmatrix}
              1 & 0 & 0 \\
              \frac{1}{5} & 1 & 0 \\
              \frac{1}{5} & 0 & 1 \\
          \end{bmatrix}
          \begin{bmatrix}
            5  & -1 & 6 \\
              0 & -\frac{4}{5} & -\frac{21}{5} \\
              0  & \frac{1}{5} & \frac{9}{5} \\
          \end{bmatrix}
            $$
            $$
            P
            A =
            \begin{bmatrix}
                1 & 0 & 0 \\
                \frac{1}{5} & 1 & 0 \\
                \frac{1}{5} & 0 & 1 \\
            \end{bmatrix}
            \begin{bmatrix}
              5  & -1 & 6 \\
                0 & -\frac{4}{5} & -\frac{21}{5} \\
                0  & \frac{1}{5} & \frac{9}{5} \\
            \end{bmatrix}
              $$
We don't need to permute here because $|-4/5| > |1/5|$.
$$
P
A =
\begin{bmatrix}
    1 & 0 & 0 \\
    \frac{1}{5} & 1 & 0 \\
    \frac{1}{5} & 0 & 1 \\
\end{bmatrix}
\begin{bmatrix}
    1 & 0 & 0 \\
    0 & 1 & 0 \\
    0 & -\frac{1}{4} & 1 \\
\end{bmatrix}
\begin{bmatrix}
  5  & -1 & 6 \\
    0 & -\frac{4}{5} & -\frac{21}{5} \\
    0  & 0 & \frac{3}{4} \\
\end{bmatrix}
  $$
  $$
  PA =
  \begin{bmatrix}
      1 & 0 & 0 \\
      \frac{1}{5} & 1 & 0 \\
      \frac{1}{5} & -\frac{1}{4} & 1 \\
  \end{bmatrix}
  \begin{bmatrix}
    5  & -1 & 6 \\
      0 & -\frac{4}{5} & -\frac{21}{5} \\
      0  & 0 & \frac{3}{4} \\
  \end{bmatrix}
    $$

Now we need to solve $Ax = b$ where $b = [-1, 2, 1]^T$.  To do this, we first note that $PAx = Pb$, so $LUx = Pb$.  So we first solve $Lc = Pb$ for $c$, then solve $Ux = c$ for $x$.

$$Pb = \begin{bmatrix}
    1  \\
    2  \\
    -1  \\
\end{bmatrix}$$

So

$$ \begin{bmatrix}
    1 & 0 & 0 \\
    \frac{1}{5} & 1 & 0 \\
    \frac{1}{5} & -\frac{1}{4} & 1 \\
\end{bmatrix}c = \begin{bmatrix}
    1  \\
    2  \\
    -1  \\
\end{bmatrix}$$

$$ \begin{bmatrix}
    1 & 0 & 0 \\
    0 & 1 & 0 \\
    \frac{1}{5} & -\frac{1}{4} & 1 \\
\end{bmatrix}c = \begin{bmatrix}
    1  \\
    2  - \frac{1}{5} = \frac{9}{5} \\
    -1  \\
\end{bmatrix}$$
$$ \begin{bmatrix}
    1 & 0 & 0 \\
    0 & 1 & 0 \\
    0 & 0 & 1 \\
\end{bmatrix}c = \begin{bmatrix}
    1  \\
    \frac{9}{5} \\
    -\frac{20}{20} -\frac{4}{20} + \frac{9}{20} = -\frac{3}{4}\\
\end{bmatrix} = c = \begin{bmatrix}
    1  \\
    \frac{9}{5} \\
    -\frac{3}{4}\\
\end{bmatrix}$$

Now we need to solve $Ux = c$

$$\begin{bmatrix}
    5  & -1 & 6 \\
      0 & -\frac{4}{5} & -\frac{21}{5} \\
      0  & 0 & \frac{3}{4} \\
  \end{bmatrix}x = \begin{bmatrix}
    1  \\
    \frac{9}{5} \\
    -\frac{3}{4}\\
\end{bmatrix}$$

Scaling all to have 1 as a pivot:

$$\begin{bmatrix}
  1  & \frac{-1}{5} & \frac{6}{5} \\
    0 & 1 & \frac{21}{4} \\
    0  & 0 & 1 \\
\end{bmatrix}x = \begin{bmatrix}
  \frac{1}{5}  \\
  -\frac{9}{4} \\
  -1\\
\end{bmatrix}$$

$$\begin{bmatrix}
  1  & \frac{-1}{5} & \frac{6}{5} \\
    0 & 1 & 0 \\
    0  & 0 & 1 \\
\end{bmatrix}x = \begin{bmatrix}
  \frac{1}{5}  \\
  3 \\
  -1\\
\end{bmatrix}$$

$$\begin{bmatrix}
  1  & 0 & 0 \\
    0 & 1 & 0 \\
    0  & 0 & 1 \\
\end{bmatrix}x = \begin{bmatrix}
  \frac{1}{5} + \frac{3}{5} + \frac{6}{5}  \\
  3 \\
  -1\\
\end{bmatrix}$$

$$\begin{bmatrix}
  1  & 0 & 0 \\
    0 & 1 & 0 \\
    0  & 0 & 1 \\
\end{bmatrix}x = \begin{bmatrix}
  2  \\
  3 \\
  -1\\
\end{bmatrix}$$

$$x = \begin{bmatrix}
  2  \\
  3 \\
  -1\\
\end{bmatrix}$$

Which matches what I got by using RREF on the augmented matrix in Julia.

\end{document}


\section{Appendix}

% ===
\subsection{Tips and Tricks}

\emph{Complete the square:}
\\
If $p(\bm x) \propto \exp(-\frac12 \bm x\!^\top \!\bm{{\color{orange} A}x} + \bm x\!^\top {\color{orange} \bm b})$,\\
then $p(\bm x) = \Gauss{\bm x \mid \bm A^{-1} \bm b, \bm A^{-1}}$

\emph{Constrained optimisation:}
\\
\textit{primal}: \enspace \highlight{$\min_{\bm x} f(\bm x)$ \enspace s.t. \enspace $g_i(\bm x) = 0$; \enspace $h_j(\bm x) \leq 0$}
\\
\textbf{Lagrangian:} \enspace with each $\alpha_j \geq 0$\\
\enspace $\mathcal L(\bm x,\lambda,\alpha) = f(\bm x) + \sum_i \lambda_i g_i(\bm x) + \sum_j \alpha_j h_j(\bm x)$
\\
Solve: \: $\pderiv{\mathcal L}{\bm x} = 0$; \: $g_i(\bm x) = 0$; \: $\alpha_j \geq 0$; \: $h_j(\bm x) \leq 0$
\\
If \textbf{Slater's cond.} holds, $\exists \bm x : g_i(\bm x) = 0, h_j(\bm x) {\color{red}\,<\,} 0$, then we can solve the \textit{dual} instead:\\
\enskip \highlight{$\max_{\bm \lambda, \bm \alpha} \brace{ \min_{\bm x} \mathcal L(\bm x, \bm \lambda, \bm \alpha) }$ \: s.t. \: $\alpha_j \geq 0$}\\
Solve: \: $\pderiv{\mathcal L}{\bm x} = 0$; \: $\pderiv{\mathcal L}{\bm \lambda} = 0$; \: $\alpha_j h_j(\bm x) {\color{red}\,=\,} 0$; \: $\alpha_j \geq 0$

\iffalse
    \emph{Lagrange Multipliers:}
    \\
    Problem $\mathcal P : \begin{cases}
        \min f(\bm x),      & \bm x\in\mathbb R^d \\
        \text{s.t. } g_i(\bm x)=0,      & i\leq m \\
        \phantom{\text{s.t. }} h_j(\bm x) \leq 0,       & j\leq n
    \end{cases}$
    \\
    Lagrangian: $\mathcal L(\bm x,\lambda,\alpha) = f(\bm x) + \sum_{i\leq m} \lambda_i g_i(\bm x) + \sum_{j\leq n} \alpha_j h_j(\bm x)$ with each $\alpha_j \geq 0$.
    \\
    Solution must satisfy
    $\pderiv{\mathcal L}{\bm x} = 0$ and $\pderiv{\mathcal L}{\lambda} = 0$,
    $\alpha_j h_j(\bm x) = 0$, and
    $\alpha_j \geq 0, \forall j\leq n$.
\fi

\emph{Euler-Lagrange:}
Find extrema of functional $\mathcal F[f] = \int G(x, f(x), f’(x)) \diff x$,
thus $\pderiv{\mathcal F}{f} \overset!= 0$.
\\
If $G$ is twice diff'able, then
\\
$\pderiv{\mathcal F}{f} = \pderiv{G}{f(x)} - \deriv{}{x} \paren*{ \pderiv{G}{f'(x)} } \overset{(\ast)}= \pderiv{G}{f(x)}$.
\\
$(\ast)$ : when $G$ does not depend on $f'$.

% ===
\subsection{Approximations}

\emph{Laplace Approximation:}
$\deriv{f}{x} \big\vert_{x_0} \!= 0$\\
$\implies \int_{\mathbb R} \eu^{C f(x)} \diff x \approx \sqrt{2\pi} C \cdot \abs{f''(x_0)} \cdot \eu^{C f(x_0)}$

% ===


If you haven't downloaded and unzipped \href{https://libaoj.in/courses/2021f/MATH3341/zip/Math.3341.zip}{\texttt{Math.3341.zip}}. Download and unzip it under \verb|H:| (H Drive if you are working on the Remote Lab). Change the current working directory by typing \verb|cd H:\Math.3341\Math.3341.Lab.04| in the Command Window, and type \verb|edit lab_04_script| in the Command Window to edit \verb|lab_04_script.m|.

%---------------------------------------------
\section{Basics of Plotting Functions}
%---------------------------------------------
\begin{enumerate}[(a)]
    \item \label{enu:1a} Plot $y = x^3$. Define a vector \verb|x|, of which the range is from $-10$ to $10$ with step size $4$, then define \verb|y| by aforementioned $y$. Plot in \verb|subplot(2, 2, 1)|. Add labels \verb|$x$|, \verb|$y$| to $x$, $y$ axis, respectively, and add title \verb|$y = x^{3}$ (step size = 4)|.
    \item Repeat (\ref{enu:1a}) but change the step size of vector \verb|x| to $0.1$, and put the plot in \verb|subplot(2, 2, 2)|. Observe the difference between two plots.
    \item \label{enu:1c} Plot the curve $(x(t), y(t))$ whose parametrization is
        \begin{equation}
        \label{eq:heart}
        \begin{cases}
        x(t) = 13 \sin^{3}{t}, \\
        y(t) = 13 \cos{t} - 5 \cos{2t} - 2 \cos{3t} - \cos{4t},
        \end{cases}
        t \in [0, 2\pi].
        \end{equation}
        First, define a vector \verb|t| using \verb|linspace|, then define \verb|x|, \verb|y| by \eqref{eq:heart}. Plot in \verb|subplot(2, 2, 3)| with \emph{red dash-dot} line. Add labels, title as shown in the third plot of Figure \ref{fig:1}.
    \item \label{enu:1c} Plot the curve $(x(t), y(t))$ whose parametrization is
        \begin{equation}
        \label{eq:fancy}
        \begin{cases}
        x(t) = 4 \sin{\frac{24t}{25}}, \\
        y(t) = 3 \sin{t},
        \end{cases}
        t \in [-25 \pi, 25\pi].
        \end{equation}
        First, define a vector \verb|t| using \verb|linspace| with $5000$ entries, then define \verb|x|, \verb|y| by \eqref{eq:fancy}. Plot in \verb|subplot(2, 2, 4)|. Add labels, title as shown in the fourth plot of Figure \ref{fig:1}.
\begin{figure}[!hbtp]
    \centering
    \includegraphics[height=0.27\textheight]{./fig/lab_04_plot_1.pdf}
    \caption{Expected Result for Part 1}
    \label{fig:1}
\end{figure}
\end{enumerate}
%---------------------------------------------
\section{Set Properties for Plotting}
%---------------------------------------------
\begin{enumerate}[(a)]
    \item Define \verb|x|, which ranges from $0$ to $2\pi$ with $1000$ points, and define \verb|y1|, \verb|y2|, and \verb|y3| as follows
        $$
        y_1 = \sin(x/2), \quad
        y_2 = \sin(x), \quad
        y_3 = \sin(2x).
        $$
    \item Plot \verb|y1|, \verb|y2|, \verb|y3| versus \verb|x| in the same figure window with line style (\verb|'LineWidth'|, 2), legend, labels, grid, and title in Figure \ref{fig:2}. Change the range of $x$-axis to $[0, 2\pi]$, and that of $y$-axis to $[-1, 1]$.
\item Use \verb|set| to set the following properties:
    \begin{itemize}
        \item \verb|XTick| to \verb|[0, pi / 2, pi, 3 * pi / 2, 2 * pi]|;
        \item \verb|XTickLabel| to \verb|{'0', '$\pi/2$', '$\pi$', '$3 \pi/2$', '$2\pi$'}|;
        \item \verb|GridLineStyle| to \verb|'--'|;
        \item \verb|Box| to \verb|'on'|;
        \item \verb|BoxStyle| to \verb|'full'|.
    \end{itemize}
\begin{figure}[!hbtp]
    \centering
    \includegraphics[height=0.27\textheight]{./fig/lab_04_plot_2.pdf}
    \caption{Expected Result for Part 2}
    \label{fig:2}
\end{figure}
\end{enumerate}
%---------------------------------------------
\section{Plotting Piecewise Function on Different Scales}
%---------------------------------------------
\begin{enumerate}[(a)]
    \item Define \verb|x| to be a vector from $0$ to $10$ with step size $0.01$, and the piecewise function \verb|y| as below
        $$
        y =
        \begin{cases}
            \frac{e^{8}}{8} x & x \leq 8, \\
            e^{x} & 8 < x.
        \end{cases}
        $$
    \item \label{enu:3b} In \verb|subplot(2, 2, 1)|, use \verb|plot| to plot \verb|y| versus \verb|x|. Set \verb|grid minor|, add labels and title as shown in the first plot in Figure \ref{fig:3}.
    \item Repeat (\ref{enu:3b}), then set $y$-axis to log scale using \verb|set(gca, 'YScale', 'log');|.
    \item Repeat (\ref{enu:3b}), but use \verb|semilogy| to plot \verb|y| versus \verb|x| instead.
    \item Combine the first and the third figure in \verb|subplot(2, 2, 4)| using \verb|plotyy|, then add labels, title, etc. as shown in the fourth plot in Figure \ref{fig:3}.

\begin{figure}[!hbtp]
    \centering
    \includegraphics[height=0.25\textheight]{./fig/lab_04_plot_3.pdf}
    \caption{Expected Result for Part 3}
    \label{fig:3}
\end{figure}
\end{enumerate}
%---------------------------------------------
\section{Save Plots}
Use the following script to save the figures.

\begin{lstlisting}[style=MATLAB]
prefix = 'lab_04_plot_';
for i = 1:3
    name = strcat(prefix, num2str(i));       % Set filename for figure i
    fig = figure(i);                         % Set figure i as current figure window
    set(fig, 'PaperPositionMode', 'auto');   % Set paper position mode to 'auto'
    pos = get(fig, 'PaperPosition');         % Get figure window paper position
    set(fig, 'PaperSize', [pos(3) pos(4)]);  % Set figure paper size
    print(fig, '-dpdf', name);               % Save figure
end
\end{lstlisting}

Once you finish, upload the script file \verb|lab_04_script.m| to the folder \verb|src|, figure files \verb|lab_04_plot_1.pdf|,  \verb|lab_04_plot_2.pdf|, and \verb|lab_04_plot_3.pdf| to the folder \verb|figure| on Overleaf. Recompile, and submit the generated \verb|.pdf| file to WyoCourses.