\lab{Introduction to NumPy}{Introduction to NumPy}
\label{lab:NumPy}
\objective{NumPy is a powerful Python package for manipulating data with multi-dimensional vectors.
Its versatility and speed makes Python an ideal language for applied and computational mathematics.
In this lab we introduce basic NumPy data structures and operations as a first step to numerical computing in Python.}

\section*{Arrays} % ===========================================================

In many algorithms, data can be represented mathematically as a \emph{vector} or a \emph{matrix}.
Conceptually, a vector is just a list of numbers and a matrix is a two-dimensional list of numbers (a list of lists).
However, even basic linear algebra operations like matrix multiplication are cumbersome to implement and slow to execute when data is stored this way.
The \emph{NumPy} module\footnote{NumPy is \emph{not} part of the standard library, but it is included in most Python distributions.} \cite{oliphant2006guide,ascher2001numerical,oliphant2007python} offers a much better solution.

The basic object in NumPy is the \emph{array}, which is conceptually similar to a matrix.
The NumPy array class is called \li{ndarray} (for ``$n$-dimensional array'').
The simplest way to explicitly create a 1-D \li{ndarray} is to define a list, then cast that list as an \li{ndarray} with NumPy's \li{array()} function.

\begin{lstlisting}
>>> import numpy as np

# Create a 1-D array by passing a list into NumPy's array() function.
>>> np.array([8, 4, 6, 0, 2])
array([8, 4, 6, 0, 2])

# The string representation has no commas or an array() label.
>>> print(np.array([1, 3, 5, 7, 9]))
[1 3 5 7 9]
\end{lstlisting}
%
The alias ``\li{np}'' for numpy is standard in the Python community. % Make footnote?

An \li{ndarray} can have arbitrarily many dimensions.
A 2-D array is a 1-D array of 1-D arrays (like a list of lists), a 3-D array is a 1-D array of 2-D arrays (a list of lists of lists), and, more generally, an $n$-dimensional array is a 1-D array of $(n-1)$-dimensional arrays (a list of lists of lists of lists...).
Each dimension is called an \emph{axis}.
For a 2-D array, the $0$-axis indexes the rows and the $1$-axis indexes the columns.
Elements are accessed using brackets and indices, with the axes separated by commas.

\begin{lstlisting}
# Create a 2-D array by passing a list of lists into array().
>>> A = np.array( [ [1, 2, 3],[4, 5, 6] ] )
>>> print(A)
[[1 2 3]
 [4 5 6]]

# Access elements of the array with brackets.
>>> print(A[0, 1], A[1, 2])
2 6

# The elements of a 2-D array are 1-D arrays.
>>> A[0]
array([1, 2, 3])
\end{lstlisting}

\begin{problem} % Matrix Multiplication
There are two main ways to perform matrix multiplication in NumPy: with NumPy's \li{dot()} function (\li{np.dot(A, B)}), or with the \li{@} operator (\li{A @ B}).
Write a function that defines the following matrices as NumPy arrays.
\begin{align*}
A = \left[\begin{array}{rrr}
3 & -1 &  4 \\
1 &  5 & -9 \end{array}\right]
&&
B = \left[\begin{array}{cccc}
2 &  6 & -5 &  3\\
5 & -8 &  9 &  7\\
9 & -3 & -2 & -3\end{array}\right]
\end{align*}
Return the matrix product $AB$.
% Solution:
% array([[ 37,  14, -32, -10],
%        [-54,  -7,  58,  65]])

For examples of array initialization and matrix multiplication, use object introspection in IPython to look up the documentation for \li{np.ndarray}, \li{np.array()} and \li{np.dot()}.
\begin{lstlisting}
<g<In [1]:>g> import numpy as np

<g<In [2]:>g> np.array?       # press 'enter'
\end{lstlisting}
\end{problem}

\begin{warn} % The @ operator.
The \li{@} operator was not introduced until Python 3.5.
It triggers the \li{__matmul__()} magic method,%
\footnote{See the lab on Object Oriented Programming for an overview of magic methods.}
which for the \li{ndarray} is essentially a wrapper around \li{np.dot()}.
If you are using a previous version of Python, always use \li{np.dot()} to perform basic matrix multiplication.
\end{warn}

\subsection*{Basic Array Operations} % ----------------------------------------

NumPy arrays behave differently with respect to the binary arithmetic operators \li{+} and \li{*} than Python lists do.
For lists, \li{+} concatenates two lists and \li{*} replicates a list by a scalar amount (strings also behave this way).

\begin{lstlisting}
# Addition concatenates lists together.
>>> [1, 2, 3] + [4, 5, 6]
[1, 2, 3, 4, 5, 6]

# Mutliplication concatenates a list with itself a given number of times.
>>> [1, 2, 3] * 4
[1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3]
\end{lstlisting}

NumPy arrays act like mathematical vectors and matrices: \li{+} and \li{*} perform component-wise addition or multiplication.
\begin{comment}
\[
\left[\begin{array}{c}1 \\ 2 \\ 3\end{array}\right]
+
\left[\begin{array}{c}4 \\ 5 \\ 6\end{array}\right]
=
\left[\begin{array}{c}1+4 \\ 2+5 \\ 3+6\end{array}\right]
=
\left[\begin{array}{c}5 \\ 7 \\ 9\end{array}\right]
\qquad
4*
\left[\begin{array}{c}1 \\ 2 \\ 3\end{array}\right]
=
\left[\begin{array}{c}4*1 \\ 4*2 \\ 4*3\end{array}\right]
=
\left[\begin{array}{c}4 \\ 8 \\ 12\end{array}\right]
\]
\end{comment}

\begin{lstlisting}
>>> x, y = np.array([1, 2, 3]), np.array([4, 5, 6])

# Addition or multiplication by a scalar acts on each element of the array.
>>> x + 10                          # Add 10 to each entry of x.
array([11, 12, 13])
>>> x * 4                           # Multiply each entry of x by 4.
array([ 4,  8, 12])

# Add two arrays together (component-wise).
>>> x + y
array([5, 7, 9])

# Multiply two arrays together (component-wise).
>>> x * y
array([ 4, 10, 18])
\end{lstlisting}

% Subtraction and division are also component-wise operations for NumPy arrays.
% Apart from being syntactically convenient, arithmetic operations on NumPy arrays also execute significantly faster than component-wise list operations.

\begin{comment}
\begin{warn} % Don't mix up * and np.dot().
Matrix multiplication is executed via \li{np.dot()} or \li{@} and scalar (component-wise) multiplication is executed via the \li{*} operator.
The equation below is an example of scalar multiplication, which uses \li{*}.
\begin{align*}
2 \left[\begin{array}{rrr}
1 & 2 & 3\\
4 & 5 & 6 \end{array}\right]
= \left[\begin{array}{rrr}
2 & 4 & 6\\
8 & 10 & 12 \end{array}\right]
\end{align*}
\end{warn}
\end{comment}

\begin{problem} % Simple Matrix Equation 2 (Cayley Hamilton)
Write a function that defines the following matrix as a NumPy array.
\[
A = \left[\begin{array}{rrr}
3 & 1 & 4\\
1 & 5 & 9 \\
-5 & 3 & 1 \end{array}\right]
\]
Return the matrix $-A^3 + 9A^2 - 15A$.

In this context, $A^2 = AA$ (the matrix product, not the component-wise square).
The somewhat surprising result is a demonstration of the Cayley-Hamilton theorem.
\end{problem}

\subsection*{Array Attributes} % ----------------------------------------------
An \li{ndarray} object has several attributes, some of which are listed below.

\begin{table}[H] % Important ndarray Attributes
\centering
\begin{tabular}{c|l}
    Attribute & Description \\
    \hline \li{dtype} & The type of the elements in the array. \\
    \li{ndim} & The number of axes (dimensions) of the array. \\
    \li{shape} & A tuple of integers indicating the size in each dimension. \\
    \li{size} & The total number of elements in the array. \\
\end{tabular}
\end{table}

\begin{lstlisting}
>>> A = np.array([[1, 2, 3],[4, 5, 6]])

# 'A' is a 2-D array with 2 rows, 3 columns, and 6 entries.
>>> print(A.ndim, A.shape, A.size)
2 (2, 3) 6
\end{lstlisting}

Note that \li{ndim} is the number of entries in \li{shape}, and that
the \li{size} of the array is the product of the entries of \li{shape}.

\begin{comment} % This is being severely shortened.
\subsection*{Data Types} % ----------------------------------------------------

Unlike native Python data structures, \textbf{all elements of a NumPy array must have the same data type}.
The data types used by NumPy arrays are machine-native and avoid the overhead of Python objects, meaning that they are faster to compute with.
Thus NumPy's \li{np.float64} and Python's standard \li{float} are not the same; the former has been optimized to speed up numerical computations.
NumPy's more common numerical data types are listed below in Table \ref{table:numpytypes}.

\begin{table}[H] % Numpy Data Types
\centering
\begin{tabular}{l|l}
Data type & Description
\\ \hline
\li{bool_} & Boolean \\
\li{int8} & 8-bit integer \\
\li{int16} & 16-bit integer \\
\li{int32} & 32-bit integer \\
\li{int64} & 64-bit integer \\
\li{uint8} & Unsigned 8-bit integer \\
\li{uint16} & Unsigned 16-bit integer \\
\li{uint32} & Unsigned 32-bit integer \\
\li{uint64} & Unsigned 64-bit integer \\
\li{float16} & Half-precision float \\
\li{float32} & Single-precision float \\
\li{float64} & Double-precision float (default type for most computations)\\
\li{complex64} & Complex number represented by two single-precision floats \\
\li{complex128} & Complex number represented by two double-precision floats
\end{tabular}
\caption{Native numerical data types available in NumPy. The numbers on these types, such as the 16 on \li{int16}, indicates the number of bits used by the machine to store the number. Thus \li{float64} is more precise, but more computationally expensive, than \li{float16}.}
\label{table:numpytypes}
\end{table}

We specify the data type of an array as it is created with the keyword argument \li{dtype}.
If unspecified, then the type is determined as the minimum type required to hold the objects in the provided sequence.
To change an existing array's type, use the array's \li{astype()} method.

\begin{lstlisting}
# A list of integers becomes an array of integers.
>>> x = np.array([0, 1, 2, 3, 4])
>>> print(x)
[0 1 2 3 4]
>>> x.dtype
<<dtype('int64')>>

# Change the data type to one of NumPy's float types.
>>> x = x.astype(np.float64)
>>> print(x)
[ 0.  1.  2.  3.  4.]
>>> x.dtype
<<dtype('float64')>>
\end{lstlisting}

\begin{warn}
The default data type for most NumPy operations is \li{np.float64}.
Using smaller data types can speed up computations, but can result in unexpected problems due to overflow.

For example, an integer represented in binary with 8 bits (including one bit for the sign) can range from $-128$ to $127$.
Casting a number outside of this range as an 8-bit integer can cause some unexpected overflow issues.

\begin{lstlisting}
>>> np.int8(200)
-56
\end{lstlisting}
\end{warn}
\end{comment}

\subsection*{Array Creation Routines} % ---------------------------------------

In addition to casting other structures as arrays via \li{np.array()}, NumPy provides efficient ways to create certain commonly-used arrays.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
Function & Returns \\
\hline \li{arange()} & Array of sequential integers (like \li{list(range())}).\\
% \li{empty()} & Array of given shape and type, without initializing entries. \\
% \li{empty_like()} & Array with the same shape and type as a given array. \\
\li{eye()} & 2-D array with ones on the diagonal and zeros elsewhere. \\
% \li{identity()} & The square identity array. \\
\li{ones()} & Array of given shape and type, filled with ones. \\
\li{ones_like()} & Array of ones with the same shape and type as a given array. \\
\li{zeros()} & Array of given shape and type, filled with zeros. \\
\li{zeros_like()} & Array of zeros with the same shape and type as a given array. \\
\li{full()} & Array of given shape and type, filled with a specified value. \\
\li{full_like()} & Full array with the same shape and type as a given array.
\end{tabular}
\end{table}

Each of these functions accepts the keyword argument \li{dtype} to specify the data type.
Common types include \li{np.bool_}, \li{np.int64}, \li{np.float64}, and \li{np.complex128}.

\begin{lstlisting}
# A 1-D array of 5 zeros.
>>> np.zeros(5)
array([ 0.,  0.,  0.,  0.,  0.])

# A 2x5 matrix (2-D array) of integer ones.
>>> np.ones((2,5), dtype=<<np.int>>)    # The shape is specified as a tuple.
array([[1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1]])

# The 2x2 identity matrix.
>>> I = np.eye(2)
>>> print(I)
[[ 1.  0.]
 [ 0.  1.]]

# Array of 3s the same size as 'I'.
>>> np.full_like(I, 3)              # Equivalent to np.full(I.shape, 3).
array([[ 3.,  3.],
       [ 3.,  3.]])
\end{lstlisting}

Unlike native Python data structures, \textbf{all elements of a NumPy array must be of the same data type}.
To change an existing array's data type, use the array's \li{astype()} method.

\begin{lstlisting}
# A list of integers becomes an array of integers.
>>> x = np.array([0, 1, 2, 3, 4])
>>> print(x)
[0 1 2 3 4]
>>> x.dtype
<<dtype('int64')>>

# Change the data type to one of NumPy's float types.
>>> x = x.astype(np.float64)        # Equivalent to x = np.float64(x).
>>> print(x)
[ 0.  1.  2.  3.  4.]               # Floats are displayed with periods.
>>> x.dtype
<<dtype('float64')>>
\end{lstlisting}

The following functions are for dealing with the diagonal, upper, or lower portion of an array.

\begin{table}[H]
\centering
\begin{tabular}{c|l}
Function & Description \\ \hline
\li{diag()} & Extract a diagonal or construct a diagonal array.\\
\li{tril()} & Get the lower-triangular portion of an array by replacing entries above\\&the diagonal with zeros.\\
\li{triu()} & Get the upper-triangular portion of an array by replacing entries below\\&the diagonal with zeros.
\end{tabular}
\label{table:numpycreate2}
\end{table}

\begin{lstlisting}
>>> A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

# Get only the upper triangular entries of 'A'.
>>> np.triu(A)
array([[1, 2, 3],
       [0, 5, 6],
       [0, 0, 9]])

# Get the diagonal entries of 'A' as a 1-D array.
>>> np.diag(A)
array([1, 5, 9])

# diag() can also be used to create a diagonal matrix from a 1-D array.
>>> np.diag([1, 11, 111])
array([[  1,   0,   0],
       [  0,  11,   0],
       [  0,   0, 111]])
\end{lstlisting}

See \url{http://docs.scipy.org/doc/numpy/reference/routines.array-creation.html} for the official documentation on NumPy's array creation routines.

\begin{problem} % Simple Matrix Equation 3
Write a function that defines the following matrices as NumPy arrays using the functions presented in this section (not \li{np.array()}).
Calculate the matrix product $ABA$.
Change the data type of the resulting matrix to \li{np.int64}, then return it.
\begin{align*}
A = \left[\begin{array}{rrrrrrr}
1 & 1 & 1 & 1 & 1 & 1 & 1\\
0 & 1 & 1 & 1 & 1 & 1 & 1\\
0 & 0 & 1 & 1 & 1 & 1 & 1\\
0 & 0 & 0 & 1 & 1 & 1 & 1\\
0 & 0 & 0 & 0 & 1 & 1 & 1\\
0 & 0 & 0 & 0 & 0 & 1 & 1\\
0 & 0 & 0 & 0 & 0 & 0 & 1\end{array}\right]
&&
B = \left[\begin{array}{rrrrrrr}
-1 &  5 &  5 &  5 &  5 &  5 &  5\\
-1 & -1 &  5 &  5 &  5 &  5 &  5\\
-1 & -1 & -1 &  5 &  5 &  5 &  5\\
-1 & -1 & -1 & -1 &  5 &  5 &  5\\
-1 & -1 & -1 & -1 & -1 &  5 &  5\\
-1 & -1 & -1 & -1 & -1 & -1 &  5\\
-1 & -1 & -1 & -1 & -1 & -1 & -1\end{array}\right]
\end{align*}
% Solution:
% array([[ -7,  -8,  -3,   8,  25,  48,  77],
%        [ -6, -12, -12,  -6,   6,  24,  48],
%        [ -5, -10, -15, -14,  -7,   6,  25],
%        [ -4,  -8, -12, -16, -14,  -6,   8],
%        [ -3,  -6,  -9, -12, -15, -12,  -3],
%        [ -2,  -4,  -6,  -8, -10, -12,  -8],
%        [ -1,  -2,  -3,  -4,  -5,  -6,  -7]])
\end{problem}

\section*{Data Access} % ======================================================

\subsection*{Array Slicing} % -------------------------------------------------

Indexing for a 1-D NumPy array uses the slicing syntax \li{x[start:stop:step]}.
If there is no colon, a single entry of that dimension is accessed.
With a colon, a range of values is accessed.
For multi-dimensional arrays, use a comma to separate slicing syntax for each axis.

\begin{lstlisting}
# Make an array of the integers from 0 to 10 (exclusive).
>>> x = np.arange(10)
>>> x
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

# Access elements of the array with slicing syntax.
>>> x[3]                            # The element at index 3.
3
>>> x[:3]                           # Everything before index 3.
array([0, 1, 2])
>>> x[3:]                           # Everything from index 3 on.
array([3, 4, 5, 6, 7, 8, 9])
>>> x[3:8]                          # Everything between index 3 and index 8 (exclusive).
array([3, 4, 5, 6, 7])

>>> A = np.array([[0,1,2,3,4],[5,6,7,8,9]])
>>> A
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])

# Use a comma to separate the dimensions for multi-dimensional arrays.
>>> A[1, 2]                         # The element at row 1, column 2.
7
>>> A[:, 2:]                        # All of the rows, from column 2 on.
array([[2, 3, 4],
       [7, 8, 9]])
\end{lstlisting}

% See Appendix \ref{appendix:numpy-visual-guide} for visual examples of slicing.

\begin{info} % Views vs. Copies.
Indexing and slicing operations return a \emph{view} of the array.
Changing a view of an array also changes the original array.
In other words, \textbf{arrays are mutable}.
To create a copy of an array, use \li{np.copy()} or the array's \li{copy()} method.
Changes to a copy of an array does not affect the original array, but copying an array uses more time and memory than getting a view.
\end{info}

\subsection*{Fancy Indexing} % ------------------------------------------------

So-called \emph{fancy indexing} is a second way to access or change the elements of an array.
Instead of using slicing syntax, provide either an array of indices or an array of boolean values (called a \emph{mask}) to extract specific elements.

\begin{lstlisting}
>>> x = np.arange(0, 50, 10)        # The integers from 0 to 50 by tens.
>>> x
array([ 0, 10, 20, 30, 40])

# An array of integers extracts the entries of 'x' at the given indices.
>>> index = np.array([3, 1, 4])     # Get the 3rd, 1st, and 4th elements.
>>> x[index]                        # Same as np.array([x[i] for i in index]).
array([30, 10, 40])

# A boolean array extracts the elements of 'x' at the same places as 'True'.
>>> mask = np.array([True, False, False, True, False])
>>> x[mask]                         # Get the 0th and 3rd entries.
array([ 0, 30])
\end{lstlisting}

Fancy indexing is especially useful for extracting or changing the values of an array that meet some sort of criterion.
Use comparison operators like \li{<} and \li{==} to create masks.

\begin{lstlisting}
>>> y = np.arange(10, 20, 2)        # Every other integers from 10 to 20.
>>> y
array([10, 12, 14, 16, 18])

# Extract the values of 'y' larger than 15.
>>> mask = y > 15                   # Same as np.array([i > 15 for i in y]).
>>> mask
<<array([False, False, False,  True,  True], dtype=bool)>>
>>> y[mask]                         # Same as y[y > 15]
array([16, 18])

# Change the values of 'y' that are larger than 15 to 100.
>>> y[mask] = 100
>>> print(y)
[10 12 14  100  100]
\end{lstlisting}

While indexing and slicing always return a view, fancy indexing always returns a copy.

\begin{problem} % Slicing / Fancy Indexing
Write a function that accepts a single array as input.
Make a copy of the array, then use fancy indexing to set all negative entries of the copy to $0$.
Return the copy.
\end{problem}

\section*{Array Manipulation} % ===============================================

\subsection*{Shaping} % -------------------------------------------------------

An array's \li{shape} attribute describes its dimensions.
Use \li{np.reshape()} or the array's \li{reshape()} method to give an array a new shape.
The total number of entries in the old array and the new array must be the same in order for the shaping to work correctly.
Using a \li{-1} in the new shape tuple makes the specified dimension as long as necessary.
% Whenever possible, \li{np.reshape()} returns a view.

\begin{lstlisting}
>>> A = np.arange(12)               # The integers from 0 to 12 (exclusive).
>>> print(A)
[ 0  1  2  3  4  5  6  7  8  9 10 11]

# 'A' has 12 entries, so it can be reshaped into a 3x4 matrix.
>>> A.reshape((3,4))                # The new shape is specified as a tuple.
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])

# Reshape 'A' into an array with 2 rows and the appropriate number of columns.
>>> A.reshape((2,-1))
array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11]])
\end{lstlisting}

Use \li{np.ravel()} to flatten a multi-dimensional array into a 1-D array and \li{np.transpose()} or the \li{T} attribute to transpose a 2-D array in the matrix sense.

\begin{lstlisting}
>>> A = np.arange(12).reshape((3,4))
>>> A
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])

# Flatten 'A' into a one-dimensional array.
>>> np.ravel(A)                     # Equivalent to A.reshape(A.size)
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])

# Transpose the matrix 'A'.
>>> A.T                             # Equivalent to np.transpose(A).
array([[ 0,  4,  8],
       [ 1,  5,  9],
       [ 2,  6, 10],
       [ 3,  7, 11]])
\end{lstlisting}

\begin{info} % All 1-D Numpy arrays are flat!
By default, all NumPy arrays that can be represented by a single dimension, including column slices, are automatically reshaped into ``flat'' 1-D arrays.
For example, by default an array will have 10 elements instead of 10 arrays with one element each.
Though we usually represent vectors vertically in mathematical notation, NumPy methods such as \li{dot()} are implemented to purposefully work well with 1-D ``row arrays''.

\begin{lstlisting}
>>> A = np.arange(10).reshape((2,5))
>>> A
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])

# Slicing out a column of A still produces a "flat" 1-D array.
>>> x = A[:,1]                      # All of the rows, column 1.
>>> x
array([1, 6])                       # Not array([[1],
>>> x.shape                         #            [6]])
(2,)
>>> x.ndim
1
\end{lstlisting}

However, it is occasionally necessary to change a 1-D array into a ``column array''.
Use \li{np.reshape()}, \li{np.vstack()}, or slice the array and put \li{np.newaxis} on the second axis.
Note that \li{np.transpose()} does not alter 1-D arrays.

\begin{lstlisting}
>>> x = np.arange(3)
>>> x
array([0, 1, 2])

>>> x.reshape((-1,1))               # Or x[:,np.newaxis] or np.vstack(x).
array([[0],
       [1],
       [2]])
\end{lstlisting}

Do not force a 1-D vector to be a column vector unless necessary.
\end{info}

\subsection*{Stacking} % ------------------------------------------------------

NumPy has functions for \emph{stacking} two or more arrays with similar dimensions into a single block matrix.
Each of these methods takes in a single tuple of arrays to be stacked in sequence.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
    Function & Description\\
    \hline
    % % Shaping methods, already discussed
    % \li{reshape()} & Return a view of the array with a changed shape.\\
    % \li{ravel()} & Make a flattened version of an array, return a view if possible.\\
    % \li{transpose()} & Permute the dimensions of the array (also \li{ndarray.T}).\\
    % \hline
    \li{concatenate()} & Join a sequence of arrays along an existing axis\\
    \li{hstack()} & Stack arrays in sequence horizontally (column wise).\\
    \li{vstack()} & Stack arrays in sequence vertically (row wise).\\
    \li{column_stack()} & Stack 1-D arrays as columns into a 2-D array.
\end{tabular}
\end{table}

\begin{lstlisting}
>>> A = np.arange(6).reshape((2,3))
>>> B = np.zeros((4,3))

# vstack() stacks arrays vertically (row-wise).
>>> np.vstack((A,B,A))
array([[ 0.,  1.,  2.],             # A
       [ 3.,  4.,  5.],
       [ 0.,  0.,  0.],             # B
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  1.,  2.],             # A
       [ 3.,  4.,  5.]])

>>> A = A.T
>>> B = np.ones((3,4))

# hstack() stacks arrays horizontally (column-wise).
>>> np.hstack((A,B,A))
array([[ 0.,  3.,  1.,  1.,  1.,  1.,  0.,  3.],
       [ 1.,  4.,  1.,  1.,  1.,  1.,  1.,  4.],
       [ 2.,  5.,  1.,  1.,  1.,  1.,  2.,  5.]])

# column_stack() stacks arrays horizontally, including 1-D arrays.
>>> np.column_stack((A, np.zeros(3), np.ones(3), np.full(3, 2)))
array([[ 0.,  3.,  0.,  1.,  2.],
       [ 1.,  4.,  0.,  1.,  2.],
       [ 2.,  5.,  0.,  1.,  2.]])
\end{lstlisting}
%
See \url{http://docs.scipy.org/doc/numpy-1.10.1/reference/routines.array-manipulation.html} for more array manipulation routines and documentation.

\begin{problem} % Simple Matrix Equation 4
Write a function that defines the following matrices as NumPy arrays.
\begin{align*}
A = \left[\begin{array}{rrr}
0 & 2 & 4\\
1 & 3 & 5\end{array}\right]
&&
B = \left[\begin{array}{rrr}
3 & 0 & 0\\
3 & 3 & 0\\
3 & 3 & 3\end{array}\right]
&&
C = \left[\begin{array}{rrr}
-2 & 0 & 0\\
0 & -2 & 0\\
0 & 0 & -2\end{array}\right]
\end{align*}
Use NumPy's stacking functions to create and return the block matrix:
\begin{align*}
\left[\begin{array}{ccc}
\0 & A\trp & I\\
A & \0 & \0 \\
B & \0 & C \end{array}\right],
\end{align*}
where $I$ is the $3\times 3$ identity matrix and each $\0$ is a matrix of all zeros of appropriate size.

A block matrix of this form is used in the interior point method for linear optimization.
\end{problem}

\subsection*{Array Broadcasting} % --------------------------------------------

Many matrix operations make sense only when the two operands have the same shape, such as element-wise addition.
\emph{Array broadcasting} extends such operations to accept some (but not all) operands with different shapes, and occurs automatically whenever possible.

Suppose, for example, that we would like to add different values to the columns of an $m\times n$ matrix $A$.
Adding a 1-D array $x$ with the $n$ entries to $A$ will automatically do this correctly.
To add different values to the different rows of $A$, first reshape a 1-D array of $m$ values into a column array.
Broadcasting then correctly takes care of the operation.

Broadcasting can also occur between two 1-D arrays, once they are reshaped appropriately.

\begin{lstlisting}
>>> A = np.arange(12).reshape((4,3))
>>> x = np.arange(3)
>>> A
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11]])
>>> x
array([0, 1, 2])

# Add the entries of 'x' to the corresponding columns of 'A'.
>>> A + x
array([[ 0,  2,  4],
       [ 3,  5,  7],
       [ 6,  8, 10],
       [ 9, 11, 13]])

>>> y = np.arange(0, 40, 10).reshape((4,1))
>>> y
array([[ 0],
       [10],
       [20],
       [30]])

# Add the entries of 'y' to the corresponding rows of 'A'.
>>> A + y
array([[ 0,  1,  2],
       [13, 14, 15],
       [26, 27, 28],
       [39, 40, 41]])

# Add 'x' and 'y' together with array broadcasting.
>>> x + y
array([[ 0,  1,  2],
       [10, 11, 12],
       [20, 21, 22],
       [30, 31, 32]])
\end{lstlisting}

\section*{Numerical Computing with NumPy} % ===================================

\subsection*{Universal Functions} % -------------------------------------------

A \emph{universal function} is one that operates on an entire array element-wise.
Universal functions are significantly more efficient than using a loop to operate individually on each element of an array.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
    Function & Description \\
    \hline
    \li{<<abs()>>} or \li{absolute()} & Calculate the absolute value element-wise. \\
    % \li{conj()} & Return the complex conjugate of the array.\\
    \li{exp()} / \li{log()} & Exponential ($e^x$) / natural log element-wise.\\
    \li{maximum()} / \li{minimum()}& Element-wise maximum / minimum of two arrays.\\
    % \li{<<round()>>} & Return a rounded version of the array.\\
    \li{sqrt()} & The positive square-root, element-wise.\\
    \li{sin()}, \li{cos()}, \li{tan()}, etc. & Element-wise trigonometric operations.
\end{tabular}
\label{table:ufuncs}
\end{table}

\begin{lstlisting}
>>> x = np.arange(-2,3)
>>> print(x, np.<<abs>>(x))             # Like np.array([abs(i) for i in x]).
[-2 -1  0  1  2] [2 1 0 1 2]

>>> np.sin(x)                       # Like np.array([math.sin(i) for i in x]).
array([-0.90929743, -0.84147098,  0.        ,  0.84147098,  0.90929743])
\end{lstlisting}

See \url{http://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs} for a more comprehensive list of universal functions.

\begin{warn}
The \li{math} module has many useful functions for numerical computations.
However, most of these functions can only act on single numbers, not on arrays.
NumPy functions can act on either scalars or entire arrays, but \li{math} functions tend to be a little faster for acting on scalars.
\begin{lstlisting}
>>> import math

# Math and NumPy functions can both operate on scalars.
>>> print(math.exp(3), np.exp(3))
20.085536923187668 20.0855369232

# However, math functions cannot operate on arrays.
>>> x = np.arange(-2, 3)
>>> np.tan(x)
array([ 2.18503986, -1.55740772,  0.        ,  1.55740772, -2.18503986])
>>> math.tan(x)
<<Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: only length-1 arrays can be converted to Python scalars>>
\end{lstlisting}
Always use universal NumPy functions, not the \li{math} module, when working with arrays.
\end{warn}

\subsection*{Other Array Methods} % -------------------------------------------

The \li{np.ndarray} class itself has many useful methods for numerical computations.

\begin{table}[H]
\centering
\begin{tabular}{r|l}
    Method & Returns \\
    \hline
    \li{<<all()>>} & \li{True} if all elements evaluate to \li{True}.\\
    \li{<<any()>>} & \li{True} if any elements evaluate to \li{True}.\\
    \li{argmax()} & Index of the maximum value.\\
    \li{argmin()} & Index of the minimum value.\\
    \li{argsort()} & Indices that would sort the array.\\
    \li{clip()} & restrict values in an array to fit within a given range\\
    \li{<<max()>>} & The maximum element of the array.\\
    \li{mean()} & The average value of the array.\\
    \li{<<min()>>} & The minimum element of the array.\\
    \li{sort()} & Return nothing; sort the array in-place.\\
    \li{std()} & The standard deviation of the array.\\
    \li{<<sum()>>} & The sum of the elements of the array.\\
    % \li{trace()} & return the sum of the elements along the main diagonal\\
    \li{var()} & The variance of the array.\\
\end{tabular}
\label{table:ndarraymethods}
\end{table}

Each of these \li{np.ndarray} methods has an equivalent NumPy function.
For example, \li{A.<<max>>()} and \li{np.<<max>>(A)} operate the same way.
The one exception is the \li{sort()} function: \li{np.sort()} returns a sorted copy of the array, while \li{A.sort()} sorts the array in-place and returns nothing.

Every method listed can operate \emph{along an axis} via the keyword argument \li{axis}.
If \li{axis} is specified for a method on an $n$-D array, the return value is an $(n-1)$-D array, the specified axis having been collapsed in the evaluation process.
If \li{axis} is not specified, the return value is usually a scalar.
Refer to the NumPy Visual Guide in the appendix for more visual examples.

\begin{lstlisting}
>>> A = np.arange(9).reshape((3,3))
>>> A
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])

# Find the maximum value in the entire array.
>>> A.<<max>>()
8

# Find the minimum value of each column.
>>> A.<<min>>(axis=0)                   # np.array([min(A[:,i]) for i in range(3)])
array([0, 1, 2])

# Compute the sum of each row.
>>> A.<<sum>>(axis=1)                   # np.array([sum(A[i,:]) for i in range(3)])
array([3, 12, 21])
\end{lstlisting}

See \url{http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html} for a more comprehensive list of array methods.

\begin{problem} % Row stochastic matrices.
A matrix is called \emph{row-stochastic}\footnote{Similarly, a matrix is called \emph{column-stochastic} if its columns each sum to $1$.} if its rows each sum to $1$.
Stochastic matrices are fundamentally important for finite discrete random processes and some machine learning algorithms.

Write a function than accepts a matrix (as a 2-D array).
Divide each row of the matrix by the row sum and return the new row-stochastic matrix.
Use array broadcasting and the \li{axis} argument instead of a loop.
\end{problem}

\begin{problem} % Project Euler #11
This problem comes from \url{https://projecteuler.net}.

In the $20 \times 20$ grid below, four numbers along a diagonal line have been marked in red.
%
\begin{align*}
\centering
\arraycolsep=1.4pt\begin{array}{cccccccccccccccccccc}
08&02&22&97&38&15&00&40&00&75&04&05&07&78&52&12&50&77&91&08\\
49&49&99&40&17&81&18&57&60&87&17&40&98&43&69&48&04&56&62&00\\
81&49&31&73&55&79&14&29&93&71&40&67&53&88&30&03&49&13&36&65\\
52&70&95&23&04&60&11&42&69&24&68&56&01&32&56&71&37&02&36&91\\
22&31&16&71&51&67&63&89&41&92&36&54&22&40&40&28&66&33&13&80\\
24&47&32&60&99&03&45&02&44&75&33&53&78&36&84&20&35&17&12&50\\
32&98&81&28&64&23&67&10&\textcolor{red}{26}&38&40&67&59&54&70&66&18&38&64&70\\
67&26&20&68&02&62&12&20&95&\textcolor{red}{63}&94&39&63&08&40&91&66&49&94&21\\
24&55&58&05&66&73&99&26&97&17&\textcolor{red}{78}&78&96&83&14&88&34&89&63&72\\
21&36&23&09&75&00&76&44&20&45&35&\textcolor{red}{14}&00&61&33&97&34&31&33&95\\
78&17&53&28&22&75&31&67&15&94&03&80&04&62&16&14&09&53&56&92\\
16&39&05&42&96&35&31&47&55&58&88&24&00&17&54&24&36&29&85&57\\
86&56&00&48&35&71&89&07&05&44&44&37&44&60&21&58&51&54&17&58\\
19&80&81&68&05&94&47&69&28&73&92&13&86&52&17&77&04&89&55&40\\
04&52&08&83&97&35&99&16&07&97&57&32&16&26&26&79&33&27&98&66\\
88&36&68&87&57&62&20&72&03&46&33&67&46&55&12&32&63&93&53&69\\
04&42&16&73&38&25&39&11&24&94&72&18&08&46&29&32&40&62&76&36\\
20&69&36&41&72&30&23&88&34&62&99&69&82&67&59&85&74&04&36&16\\
20&73&35&29&78&31&90&01&74&31&49&71&48&86&81&16&23&57&05&54\\
01&70&54&71&83&51&54&69&16&92&33&48&61&43&52&01&89&19&67&48
\end{array}
\end{align*}

The product of these numbers is $26 \times 63 \times 78 \times 14 = 1788696$.
Write a function that returns the greatest product of four adjacent numbers in the same direction (up, down, left, right, or diagonally) in the grid.

For convenience, this array has been saved in the file \texttt{grid.npy}.
Use the following syntax to extract the array:

\begin{lstlisting}
>>> grid = np.load("grid.npy")
\end{lstlisting}

One way to approach this problem is to iterate through the rows and columns of the array, checking small slices of the array at each iteration and updating the current largest product.
Array slicing, however, provides a much more efficient solution.

The na\"{i}ve method for computing the greatest product of four adjacent numbers in a horizontal row might be as follows:

\begin{lstlisting}
>>> winner = 0
>>> for i in range(20):
...     for j in range(17):
...         winner = max(np.prod(grid[i,j:j+4]), winner)
...
>>> winner
48477312
\end{lstlisting}

Instead, use array slicing to construct a single array where the $(i,j)$th entry is the product of the four numbers to the right of the $(i,j)$th entry in the original grid.
Then find the largest element in the new array.

\begin{lstlisting}
>>> np.<<max>>(grid[:,:-3] * grid[:,1:-2] * grid[:,2:-1] * grid[:,3:])
48477312
\end{lstlisting}

Use slicing to similarly find the greatest products of four vertical, right diagonal, and left diagonal adjacent numbers.
\\
(Hint: Consider drawing the portions of the grid that each slice in the above code covers, like the examples in the visual guide.
Then draw the slices that produce vertical, right diagonal, or left diagonal sequences, and translate the pictures into slicing syntax.)
\end{problem}

\begin{warn}
All of the examples in this lab use NumPy arrays, objects of type \li{np.ndarray}.
NumPy also has a ``matrix'' data structure called \li{np.matrix} that was built specifically for MATLAB users who are transitioning to Python and NumPy.
It behaves slightly differently than the regular array class, and can cause some unexpected and subtle problems.

For consistency (and your sanity), \textbf{never} use a NumPy matrix; \textbf{always} use NumPy arrays.
If necessary, cast a matrix object as an array with \li{np.array()}.
\end{warn}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Random Sampling} % -----------------------------------------------

The submodule \li{np.random} holds many functions for creating arrays of random values chosen from probability distributions such as the uniform, normal, and multinomial distributions.
It also contains some utility functions for getting non-distributional random samples, such as random integers or random samples from a given array.

\begin{table}[H] % The np.random submodule.
\begin{tabular}{r|l}
Function & Description\\
\hline
\li{choice()} & Take random samples from a 1-D array.\\
\li{random()} & Uniformly distributed floats over [0, 1).\\
\li{randint()} & Random integers over a half-open interval.\\
\li{random_integers()} & Random integers over a closed interval.\\
\li{randn()} & Sample from the standard normal distribution.\\
\li{permutation()} & Randomly permute a sequence / generate a random sequence.\\
\\
Function & Distribution\\
\hline
\li{beta()} & Beta distribution over [0, 1].\\
\li{binomial()} & Binomial distribution.\\
\li{exponential()} & Exponential distribution.\\
\li{gamma()} & Gamma distribution.\\
\li{geometric()} & Geometric distribution.\\
\li{multinomial()} & Multivariate generalization of the binomial distribution.\\
\li{multivariate_normal()} & Multivariate generalization of the normal distribution.\\
\li{normal()} & Normal / Gaussian distribution.\\
\li{poisson()} & Poisson distribution.\\
\li{uniform()} & Uniform distribution.
\end{tabular}
\end{table}

Note that many of these functions have counterparts in the standard library's \li{random} module.
These NumPy functions, however, are much better suited for working with large collections of random samples.

\begin{lstlisting}
# 5 uniformly distributed values in the interval [0, 1).
>>> np.random.random(5)
array([ 0.21845499,  0.73352537,  0.28064456,  0.66878454,  0.44138609])

# A 2x5 matrix (2-D array) of integers in the interval [10, 20).
>>> np.random.randint(10, 20, (2,5))
array([[17, 12, 13, 13, 18],
       [16, 10, 12, 18, 12]])
\end{lstlisting}

\subsection*{Saving and Loading Arrays} % -------------------------------------

It is often useful to save an array as a file for later use.
NumPy provides several easy methods for saving and loading array data.

\begin{table}[H]
\begin{tabular}{r|l}
Function & Description\\
\hline
\li{save()} & Save a single array to a \texttt{.npy} file.\\
\li{savez()} & Save multiple arrays to a \texttt{.npz} file.\\
\li{savetxt()} & Save a single array to a \texttt{.txt} file.\\
\hline
\li{load()} & Load and return an array or arrays from a \texttt{.npy} or \texttt{.npz} file.\\
\li{loadtxt()} & Load and return an array from a text file.
\end{tabular}
\end{table}

\begin{lstlisting}
# Save a 100x100 matrix of uniformly distributed random values.
>>> x = np.random.random((100,100))
>>> np.save("uniform.npy", x)       # Or np.savetxt("uniform.txt", x).

# Read the array from the file and check that it matches the original.
>>> y = np.load("uniform.npy")      # Or np.loadtxt("uniform.txt").
>>> np.allclose(x, y)               # Check that x and y are close entry-wise.
<<True>>
\end{lstlisting}

To save several arrays to a single file, specify a keyword argument for each array in \li{np.savez()}.
Then \li{np.load()} will return a dictionary-like object with the keyword parameter names from the save command as the keys.

\begin{lstlisting}
# Save two 100x100 matrices of normally distributed random values.
>>> x = np.random.randn(100,100)
>>> y = np.random.randn(100,100)
>>> np.savez("normal.npz", first=x, second=y)

# Read the arrays from the file and check that they match the original.
>>> arrays = np.load("normal.npz")
>>> np.allclose(x, arrays["first"])
<<True>>
>>> np.allclose(y, arrays["second"])
<<True>>
\end{lstlisting}

\begin{comment}
\subsection*{Polynomials} % ---------------------------------------------------

The \li{np.poly1d} object represents a polynomial in NumPy.
The constructor is called with the coefficients of the desired polynomial.

\begin{lstlisting}
>>> poly = np.poly1d([3, 5, 1, 2, 0, 1])
>>> print(poly)
   5     4     3     2
3 x + 5 x + 1 x + 2 x + 1
\end{lstlisting}

The object \li{poly} represents the polynomial $3x^5+5x^4+x^3+2x^2+1$.
NumPy provides many functions to operate on \li{poly1d} objects (see \url{http://docs.scipy.org/doc/numpy/reference/routines.polynomials.polynomial.html}).

Recall that
\[
e^x = \sum_{n=0}^{\infty} \frac{x^n}{n!}.
\]
The following function evaluates the $N$th partial sum of this series at the value $a$.

\begin{lstlisting}
>>> from scipy.special import factorial
>>> def exp(a, N=25):
...     """Construct an array in reverse order from n to 0."""
...     n = np.arange(N, -1, -1)
...     # Use broadcasting to compute coefficients
...     coeffs = 1. / factorial(n)
...     poly = np.poly1d(coeffs)        # Make a polynomial object.
...     return poly(a)
...
\end{lstlisting}

The last two lines can be condensed by using the following command:

\begin{lstlisting}
np.polyval(p, a)
\end{lstlisting}

\begin{problem}
\leavevmode
\begin{enumerate}
\item Use NumPy's polynomial objects to approximate the following series.
\[
\arcsin x = \sum_{n=0}^{\infty} \frac{\left(2 n\right) ! x^{2 n + 1}}{\left(2 n + 1\right)\left(n!\right)^2 4^n}
\]
This series converges on $(-1, 1)$. Use your series approximation to approximate $\pi$. Hint: think of the powers of $x$ that
are not included in the series as having zero coefficients.

\item The lambert W function is the inverse of $x e^x$.
Its Taylor series is below (note the index starts at 1).
\[
W(x) = \sum_{n=1}^{\infty} \frac{\left(-n\right)^{n-1} x^n}{n!}
\]
This series has a radius of convergence of $\frac{1}{e}$.
Use the series to approximate a number $x$ such that $x e^x = \frac{1}{4}$.
Verify that your approximation is close.
\end{enumerate}
\end{problem}
\end{comment}

\begin{comment}
\subsection*{Iterating Through Arrays} % --------------------------------------

Iterating through an array (using a \li{for} loop) negates most of the advantages of using NumPy.
Avoid iterating through arrays as much as possible by using array broadcasting and universal functions.
When absolutely necessary, use \li{np.nditer()} to create an efficient iterator for the array.
See \url{http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html} for details.
\end{comment}


\section{Fourier Sampling}

The Nyquist-Shannon sampling theorem states that a signal that contains
no frequencies higher than $B$ is completely determined by sampling it
at a rate of $1/(2B)$.

In the context of simulating observations with an interferometer, this
means that if our instrument samples only to a spatial frequency of
\math{\umax}, then an image sampled at $1/({2\umax})$ will contain all the
information that the interferometer can detect.

Simulated images in \climax\ will therefore be pixelized with $\mdx =
1/({2\umax})$, with $x_{\rm max}$ given by the specified correlation
length in the Fourier plane.


\documentclass[11pt, a4paper]{article}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{pgfplots}

\begin{document}

\title{LOGISTIC REGRESSION}
\date{}
\maketitle

The classification problem can be written as follows:

\begin{align*}
	\mathbf{Y} = \operatorname*{argmax}_{y_k}\left\{
	\mathbf{P}(\mathbf{Y}=y_k\ |\ x_1,\ x_2,\ ...\ x_p)
	\right.
\end{align*}

``Generative'' classifiers like the Naive Bayes classifier solve this problem using

\begin{align*}
	\mathbf{P}(\mathbf{Y}=y_k\ |\ x_1,\ x_2,\ ...\ x_p) = \frac{\mathbf{P}(x_1,\ x_2,\ ...\ x_p|\mathbf{Y}=y_k) \times \mathbf{P}(\mathbf{Y}=y_k)}{\mathbf{P}(x_1,\ x_2,\ ...\ x_p)} \\
\end{align*}

and modelling

\begin{align*}
	\mathbf{P}(x_1,\ x_2,\ ...\ x_p|\mathbf{Y}=y_k)
\end{align*}

On the other hand, ``Discriminative'' classifiers like Logistic Regression model

\begin{align*}
	\mathbf{P}(\mathbf{Y}=y_k\ |\ x_1,\ x_2,\ ...\ x_p)
\end{align*}

directly.

\section{Probability Model}

Assume the binary classification problem. Logistic Regression assumes:

\begin{align*}
	\mathbf{P}(\mathbf{Y}=1\ |\ x_1,\ x_2,\ ...\ x_p) = \frac{e^{\beta_0 + \beta_1 x_1 +\  ...\  + \beta_p x_p}}{1 + e^{\beta_0 + \beta_1 x_1 +\  ...\  + \beta_p x_p}}
\end{align*}

To simplify this equation, the following notation is used,

\begin{align*}
	\boldsymbol{\beta} = \begin{pmatrix} \beta_0 \\ \beta_1 \\ . \\ . \\ .  \\ \beta_p \end{pmatrix} \ \text{and}\
	\boldsymbol{x} = \begin{pmatrix} 0           \\ x_1 \\ . \\ . \\ .  \\ x_p \end{pmatrix}
\end{align*}

The equation now becomes,

\begin{align*}
	\mathbf{P}(\mathbf{Y}=1|\mathbf{X} = \boldsymbol{x}) = \frac{e^{\boldsymbol{\beta}^T\boldsymbol{x}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x}}}
\end{align*}

The other half of this equation is,

\begin{align*}
	\mathbf{P}(\mathbf{Y}=0|\mathbf{X} = \boldsymbol{x}) = \frac{1}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x}}}
\end{align*}

These half equations can be combined as follows,

\begin{align*}
	\mathbf{P}(\mathbf{Y}=y|\mathbf{X} = \boldsymbol{x}) = \left(\frac{e^{\boldsymbol{\beta}^T\boldsymbol{x}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x}}}\right) ^y \left(\frac{1}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x}}}\right)^{1-y}
\end{align*}

\section{Maximum Likelihood estimation}

If the form of a Probability Density Function(PDF) is known and some independent and identically distributed(i.i.d.) observations are given, Maximum Likelihood Estimation(MLE) can be used to estimate the optimal parameters of the PDF.

For example, consider a random variable whose PDF is known to be Gaussian,

\begin{align*}
	f(x|\mu, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}
\end{align*}

If some i.i.d. observations $x_1, x_2, \ ...\ x_n$ are given, likelihood can be written as follows,

\begin{align*}
	l(\mu, \sigma^2) & = P(x_1, x_2,\ ... \ x_n|\mu, \sigma^2) \\
	                 & = \prod_i P(x_i|\mu, \sigma^2)
\end{align*}

Log likelihood is more desirable to work with most of the times and expressed as follows,

\begin{align*}
	L(\mu, \sigma^2) & = \sum_i log(P(x_i|\mu, \sigma^2))                                        \\
	                 & = -\frac{n}{2} log(2 \pi \sigma^2) - \sum_i \frac{(x_i-\mu)^2}{2\sigma^2}
\end{align*}

To maximize log likelihood, its derivative must be equated to zero.

\begin{align*}
	\frac{\partial L(\mu, \sigma^2)}{\partial \mu}                                                                    & = 0 \\
	\frac{\partial}{\partial \mu}\left(-\frac{n}{2} log(2 \pi \sigma^2) - \sum_i \frac{(x_i-\mu)^2}{2\sigma^2}\right) & = 0 \\
	\sum_i (x_i - \mu)                                                                                & = 0 \\
	\mu = \frac{1}{n} \sum_i x_i
\end{align*}

So, in this case, MLE estimates $\mu$ to be average of the observations.

\section{MLE application to Logistic Regression}

\begin{align*}
	L(\boldsymbol{\beta}) & = \sum_i log\left\{ \left(\frac{e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}\right) ^{y_i} \left(\frac{1}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}\right)^{1-y_i}\right\} \\
	                      & = \sum_i \left(y_i \boldsymbol{\beta}^T\boldsymbol{x_i} - log(1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}})\right)
\end{align*}

Taking the derivative and equating it to zero,

\begin{align*}
	\frac{\partial \boldsymbol{L}(\boldsymbol{\beta})} {\partial \boldsymbol{\beta}}                                                                          & = 0 \\
	\sum_i \left(y_i \boldsymbol{x_i} - \frac{e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}} \boldsymbol{x_i}\right) & = 0 \\
\end{align*}

Unfortunately, no closed form solution exists in this case and hence this equation must be solved by numerical methods like the Newton-Raphson method. The objective function is always convex, so there are no problems of local maxima.

\section {Newton-Raphson Method}

Newton-Raphson method successively finds better approximations to the root of a real valued function.

\subsection {Single varable}

\begin{figure}[htbp]
	\centering
	\begin{tikzpicture}
		\draw[-latex] (0,0) -- (8,0) node[right]{x};
		\draw[-latex] (0,0) -- (0,5) node[left]{y};

		\draw[line width=1pt] (1,-1) .. controls (5,1) .. (6,4);

		\draw[] (5.5,2.3) -- (4,0);
		\draw[] (5,1.6) -- (5,0);

		\draw (5, 4) node [above right] {$y = f(x)$};'
		\draw (4, 0) node [below] {$x_{n+1}$};
		\draw (5, 0) node [below] {$x_n$};
		\draw (5, 1.6) node [right] {($x_n, f(x_n))$};
	\end{tikzpicture}
\end{figure}

\begin{align*}
	f(x+\Delta x) \approx f(x) + f'(x) \Delta x
\end{align*}

Setting $f(x+\Delta x) = 0$ to find a better approximation of the root,

\begin{align*}
	f(x) + f'(x) \Delta x & \approx 0                    \\
	\Delta x              & \approx - \frac{f(x)}{f'(x)} \\
	x_{n+1} - x_{n}       & \approx - \frac{f(x)}{f'(x)}
\end{align*}

The method starts with an initial guess $x_0$ and then applies the above updation logic to come up with $x_1,\ x_2,\ ...$ until convergence.

\subsection{Multiple Variables}

\begin{align*}
	f_1(x+\Delta x, y+\Delta y) \approx f_1(x, y) + \begin{pmatrix} \frac{\partial f_1}{\partial x}, \frac{\partial f_1}{\partial y} \end{pmatrix} \begin{pmatrix} \Delta x \\ \Delta y \end{pmatrix} \\
	f_2(x+\Delta x, y+\Delta y) \approx f_2(x, y) + \begin{pmatrix} \frac{\partial f_2}{\partial x}, \frac{\partial f_2}{\partial y} \end{pmatrix} \begin{pmatrix} \Delta x \\ \Delta y \end{pmatrix}
\end{align*}

These two equations can be written in matrix form as follows,

\begin{align*}
	\begin{pmatrix} f_1(x+\Delta x, y+\Delta y) \\ f_2(x+\Delta x, y+\Delta y)\end{pmatrix} \approx \begin{pmatrix} f_1(x, y) \\ f_2(x, y) \end{pmatrix} + \begin{pmatrix} \frac{\partial f_1}{\partial x}, \frac{\partial f_1}{\partial y} \\ \frac{\partial f_2}{\partial x}, \frac{\partial f_2}{\partial y}\end{pmatrix} \begin{pmatrix} \Delta x \\ \Delta y \end{pmatrix}
\end{align*}

Setting the l.h.s to zero and solving,

\begin{align*}
	\begin{pmatrix} \Delta x \\ \Delta y \end{pmatrix}  \approx -\begin{pmatrix} \frac{\partial f_1}{\partial x}, \frac{\partial f_1}{\partial y} \\ \frac{\partial f_2}{\partial x}, \frac{\partial f_2}{\partial y}\end{pmatrix}^{-1} \begin{pmatrix} f_1(x, y) \\ f_2(x, y) \end{pmatrix}
\end{align*}

The $2\times2$ matrix of partial derivatives is called Jacobian and denoted by $J$. The same idea can be extended to any number of variables.

\section{Newton-Raphson application to Logistic Regression}

To find optimal parameter $\beta$ for Logistic Regression using MLE, the roots of the following equation must be found,

\begin{align*}
	\frac{\partial \boldsymbol{L}(\boldsymbol{\beta})} {\partial \boldsymbol{\beta}} & = 0 \\
\end{align*}

The Newton-Raphson update rule for this has the form,

\begin{align*}
	\Delta \boldsymbol{\beta} & = -\left(\frac{\partial^2 L(\boldsymbol{\beta})}{\partial \boldsymbol{\beta} \partial \boldsymbol{\beta}^T}\right)^{-1} \frac{\partial \boldsymbol{L}(\boldsymbol{\beta})} {\partial \boldsymbol{\beta}}                                                                                \\
	\Delta \boldsymbol{\beta} & = \left[ \sum_i \left(\frac{e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}\right) \left(\frac{1}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}\right) \boldsymbol{x_i} \boldsymbol{x_i}^T \right]^{-1} \left[ \sum_i \left(y_i \boldsymbol{x_i} - \frac{e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}} \boldsymbol{x_i}\right)\right]
\end{align*}

Using the following notation

\begin{align*}
	\boldsymbol{X} & = \begin{pmatrix} \boldsymbol{x_1}, \boldsymbol{x_2}, .., \boldsymbol{x_N}\end{pmatrix}         \\
	P_i            & = \frac{e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}} \\
	\boldsymbol{Y} & = \begin{pmatrix} y_1                                                                           \\ y_2 \\ . \\ . \\ . \\ y_N \end{pmatrix}
	\boldsymbol{P} = \begin{pmatrix} P_1 \\ P_2 \\ . \\ . \\ . \\ P_N \end{pmatrix}
\end{align*}

the update rule can be rewritten as

\begin{align*}
	\Delta \boldsymbol{\beta} = [\boldsymbol{X}\boldsymbol{W}\boldsymbol{X}^T]^{-1}\boldsymbol{X}(\boldsymbol{Y}-\boldsymbol{P})
\end{align*}

where $\boldsymbol{W}$ is a $N\times N$ diagonal matrix of $P_i(1-P_i)$s with N as total number of $\boldsymbol{x_i}^T, y_i$ pairs. This equation bears a surprising resemblance with the OLS solution of the problem of linear regression. In fact $\Delta \boldsymbol{\beta}$ is solutions to the following linear regression problem.

\begin{align*}
	(\boldsymbol{Y}-\boldsymbol{P}) = \boldsymbol{W}\boldsymbol{X}^T \Delta \boldsymbol{\beta}
\end{align*}

Hence, the update in each iteration of Newton-Raphson method for Logistic Regression is the solution of a weighted least square linear regression problem where the response is the difference between the observed response and estimated probability of the Logistic Regression model.

\section{Interpretation}

\subsection{Coefficients}

Rewriting the base equations for Logistic Regression

\begin{align*}
	\mathbf{P}(\mathbf{Y}=1\ |\ x_1,\ x_2,\ ...\ x_p) = \frac{e^{\beta_0 + \beta_1 x_1 +\  ...\  + \beta_p x_p}}{1 + e^{\beta_0 + \beta_1 x_1 +\  ...\  + \beta_p x_p}} \\
	\mathbf{P}(\mathbf{Y}=0\ |\ x_1,\ x_2,\ ...\ x_p) = \frac{1}{1 + e^{\beta_0 + \beta_1 x_1 +\  ...\  + \beta_p x_p}}
\end{align*}

Dividing both equations

\begin{align*}
	\frac{\mathbf{P}(\mathbf{Y}=1\ |\ x_1,\ x_2,\ ...\ x_p)}{\mathbf{P}(\mathbf{Y}=0\ |\ x_1,\ x_2,\ ...\ x_p) } = e^{\beta_0 + \beta_1 x_1 +\  ...\  + \beta_p x_p}
\end{align*}

The left side of the equation: the ratio of the probability of an event occurring to the probability of it not occurring is called the ``odds" for the event. Hence, the odds of the event $\mathbf{Y}=1$  is equal to $e$ raised to a linear function of the input features. Every other thing remaining equal, if one of the input features is increased by one unit, the odds are multiplied by $e$ raised to the coefficient of the input features. Positive coefficients have a multiplicative effect on the odds, but negative coefficients have a divisive effect.

\subsection{Coordinate Freedom}

Rewriting the MLE solution equation again,

\begin{align*}
	\sum_i \left(y_i \boldsymbol{x_i} - \frac{e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}}{1 + e^{\boldsymbol{\beta}^T\boldsymbol{x_i}}} \boldsymbol{x_i}\right) & = 0 \\
\end{align*}

or

\begin{align*}
	\sum_i \left(y_i \boldsymbol{x_i} - P_i \boldsymbol{x_i}\right) & = 0 \\
\end{align*}

or

\begin{align*}
	\boldsymbol{X}(\boldsymbol{Y}-\boldsymbol{P}) = 0
\end{align*}

If $\boldsymbol{P}$ satisfies $ \boldsymbol{X}(\boldsymbol{Y}-\boldsymbol{P}) = 0$, then it will also satisfy $\boldsymbol{M}\boldsymbol{X}(\boldsymbol{Y}-\boldsymbol{P}) = 0$ for non singular matrix $\boldsymbol{M}$. Since $\boldsymbol{M}\boldsymbol{X}$ is related to $\boldsymbol{X}$ by linear transformations, it is clear that linear combinations of input features like rescaling and combination will not have any effect on the probabilities estimated by a Logistic Regression model. Hence, Logistic regression is a coordinate free model.

\subsection{Marginal Probability Preservation}

Since the Logistic Regression model demands,

\begin{align*}
	\sum_i \left(y_i \boldsymbol{x_i} - P_i \boldsymbol{x_i}\right) & = 0 \\
\end{align*}

For every feature, the sum of the input feature values corresponding to positive response will be equal to the weighted sum of the all the input feature values where the probabilities estimated by the model serve as the weights. Hence, Logistic Regression preserves marginal probabilities of the input.

\end{document}


\documentclass[a4paper,11pt]{article}

\usepackage{physics}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{amsthm, mathtools}
%\usepackage{hyperref}
\usepackage{color}
\usepackage{jheppub}
\usepackage[T1]{fontenc} % if needed

\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\bes}{\begin{equation*}}
\newcommand{\ees}{\end{equation*}}
\newcommand{\bea}{\begin{flalign*}}
\newcommand{\eea}{\end{flalign*}}

%\linespread{1.0}
%\setlength{\parindent}{0em}
%\setlength{\parskip}{0.8em}

\title{\textbf{Singular Perturbation Theory}}
\author{Aditya Vijaykumar}
\affiliation{International Centre for Theoretical Sciences, Bengaluru, India.}
\emailAdd{aditya.vijaykumar@icts.res.in}
\abstract{Course Project - Dynamical Systems}

\begin{document}
\maketitle
\section{Introduction}
We take a tour of the theory of singular perturbations and the methods to solve them. Main references for the material in this report are :-
\begin{itemize}
\item Bender, Carl M., and Steven A. Orszag. 2013. \textit{Advanced mathematical methods for scientists and engineers}. New York: Springer.
\item O'Malley, Robert E. 2013. \textit{Singular perturbation methods for ordinary differential equations}.  Springer-Verlag New York.
\item Lin, C. C., Lee A. Segel, and G. H. Handelman. 2006. \textit{Mathematics applied to deterministic problems in the natural sciences}. Philadelphia, Pa: Society for Industrial and Applied Mathematics.
\end{itemize}
\section{Singular Perturbation theory for Algebraic Equations}
Consider first the equation $ \epsilon m^2 + 2m + 1 =0 $, where $ \epsilon << 1 $ is a small parameter. We note that with $ \epsilon= 0$ the equation has just one solution $ m = -1/2 $, whereas the full equation has two solutions by virtue of being quadratic. If what we desire are perturbative solutions around the unperturbed solutions, we won't be able to obtain two solutions by usual power series substitution.

So where exactly are we failing to capture the problem? As this problem is exactly solvable, let's look at the roots -
\begin{equation*}
m_\pm = \dfrac{-1 \pm \sqrt{1 - \epsilon}}{\epsilon} \approx\dfrac{-1 \pm (1 - \epsilon/2)}{\epsilon} \approx -\dfrac{1}{2}, - \dfrac{2}{\epsilon}
\end{equation*}
We see that the roots are diverging at $ \epsilon \rightarrow 0 $, which explains why the perturbative approach fails! So  from the two preceding observations that \textit{ignoring} the $ \epsilon $ for obtaining the unperturbed solution is not the best of ideas.

Let's then try to do the next best thing - trying to see what terms have comparable orders as the $ \epsilon m^2 $ term. We adopt a trial and error approach. Let's first try $ \epsilon m^2 \sim m \implies m \sim \frac{1}{\epsilon} \implies 2m  = \order{\frac{1}{\epsilon}} $. This means that now the $ \epsilon m^2 $ term cannot be ignored. Let's make the substitution $ m = \frac{x}{\epsilon}$ to, in some sense, normalize $ \epsilon  $ term.
\begin{align*}
\implies \epsilon \qty(\dfrac{x}{\epsilon})^2 + 2 \dfrac{x}{\epsilon} + 1 = 0 \\
\implies \dfrac{x^2}{\epsilon} + 2 \dfrac{x}{\epsilon} + 1 = 0 \\
\implies x^2 + 2 x + \epsilon = 0
\end{align*}
Let's make the substitution $  x = \sum_{n=0}^{\infty} x_n \epsilon^n$. Upto first order in $ \epsilon $, we get the following equations,
\begin{align*}
x_0^2 + 2 x_0 = 0 &\implies x_0 = 0 , -2 \\
2 x_0 x_1 + 2 x_1 + 1 = 0 &\implies x_1 = -\dfrac{1}{2}, x_1 = \dfrac{1}{2} \qq{pairwise for } x_0= 0 , -\frac{1}{2}
\end{align*}
Hence we have the two roots, perturbatively to $ \order{\epsilon} $,
\begin{equation*}
x = -\dfrac{\epsilon}{2}, -2 + \dfrac{\epsilon}{2} \implies m = -\dfrac{1}{2}, -\dfrac{2}{\epsilon}
\end{equation*}
So this procedure works! The procedure is actually called the \textit{method of dominant balance}. Let's look at a more complicated equation, $ \epsilon x^4 + \epsilon x^3 - x^2 + 2x - 1 =0 $. A usual characteristic of a singular perturbation problem is that the highest order term is multiplied by an infinitesimal quantity. Let's apply the above procedure to this equation -
\begin{itemize}
	\item Substituting $ x = \sum_{n} a_n \epsilon^n $, we get,
	\begin{align*}
	a_0^2 -2a_0 +1 = 0 \implies a_0 = 1,-1 \\
	a_0^4 +a_0^3 - 2 a_0 a_1 + 2 a_1 = 0 \implies
	\end{align*}
	\item $ \epsilon x^4 \sim x^2 \implies x \sim \frac{1}{\sqrt{\epsilon}} \sim \frac{1}{\mu} $. Substituting $ x = \frac{m}{\mu} $, $ m^4 + \mu m^3 - m^2 + 2 \mu m - \mu^2 = 0$. Substituting $ m = \sum_{n=0}^{\infty} a_n \mu^n $ and using sympy to do the algebraic manipulations, we get the following equations,
	\begin{align*}
	a_0^4 - a_0^2 = 0 &\implies a_0 = 1,-1 \\
	4 a_0^3 a_1 + a_0^3 - 2 a_0 a_1 + 2 a_0 = 0 &\implies a_1 = -\frac{3}{2}, -\frac{3}{2} \\
	4 a_0^3 a_2 + 6 a_0^2 a_1^2 + 3 a_0^2 a_1 -2 a_0 a_2 - a_1^2 +2 a_1 - 1 = 0 &\implies a_2 = -\dfrac{11}{8}, \dfrac{11}{8}
	\end{align*}
	Hence, we get two roots as,
	\begin{align*}
	x &= \frac{1 -\frac{3}{2}\mu - \frac{11}{8} \mu^2}{\mu}, \frac{-1 -\frac{3}{2}\mu + \frac{11}{8} \mu^2}{\mu} \\
	\implies x &\approx \frac{1}{\sqrt{\epsilon}} -\frac{3}{2}, -\frac{1}{\sqrt{\epsilon}} -\frac{3}{2}
	\end{align*}
	\item $ \epsilon x^4 + \epsilon x^3 = (x-1)^2 \implies x = 1 \pm \sqrt{\epsilon} \sqrt{x^3 + x^4}$. We can use this to set up an iterative scheme in $ \sqrt{\epsilon} $. Therefore, two more roots are,
	\begin{equation*}
	x = 1 + \sqrt{2} \sqrt{\epsilon} + \dfrac{7}{2} \epsilon, 1 - \sqrt{2} \sqrt{\epsilon} + \dfrac{7}{2} \epsilon,
	\end{equation*}
\end{itemize}

\section{Singular Perturbation Theory for Ordinary Differential Equations}
\subsection{Boundary Value Problem}
\begin{equation*}
\epsilon y'' + 2 y' + y = 0 \qq{,} y' = \dv{y}{x} \qq{,} y(0) = 0 \qq{,} y(1)=1
\end{equation*}
The unperturbed differential equation reads $ 2y' + y = 0  $. We see now, if we take both boundary conditions into account, that this is an overdetermined problem. Anyway, this unperturbed equation has the general solution,
\begin{equation*}
y = A \exp(-\dfrac{x}{2}) \implies y = 0 \qq{OR} y = \exp(\dfrac{1-x}{2})
\end{equation*}
How do we reconcile the differences? Let's first solve the differential equation exactly by substituting $ y = \exp(mx) $ :-
\begin{align*}
\epsilon m^2 + 2m + 1 = 0 &\implies m = \dfrac{-1 \pm \sqrt{1 - \epsilon}}{\epsilon} \approx -\dfrac{1}{2}, -\dfrac{2}{\epsilon} \\
\implies y = B \exp(-\dfrac{x}{2}) +  C \exp(-\dfrac{2x}{\epsilon}) &\implies y \approx e^{1/2} \qty(\exp(-\dfrac{x}{2}) - \exp(-\dfrac{2x}{\epsilon}) )
\end{align*}
So, we see that there is a term which becomes large as $ x\rightarrow 0 $, but dies of faster as $ x $ increases. This infinitesimal region near $x =0 $ is popularly called the the \textit{boundary layer}, mainly inspired by fluid mechanics problems.

What does this tell us? We can clearly see that there are two regions in the problem - one where the effects of the $ \epsilon $ term are significant, and one where the effects are insignificant. This suggests we should really solve this equation in these regions separately and then adopt some kind of matching procedure. So let's do exactly that, first for the region where $ x \sim \order{\epsilon} $. Substituting $ t = \frac{x}{\delta} $ in the governing equation, we get,
\begin{equation*}
\dfrac{\epsilon}{\delta^2} \ddot{y} + \dfrac{2}{\delta} \dot{y} + y = 0 \qq{,} \dot{y} = \dv{y}{t}
\end{equation*}
Let's now apply a dominant balance like method here. If $ \frac{\epsilon}{\delta^2} \sim \frac{2}{\delta}  \implies \epsilon \sim \delta$, after rearranging terms and neglecting $ \order{\epsilon} $ terms, we have
\begin{equation*}
\ddot{y} + 2 \dot{y} = 0 \implies y = A + B \exp(-2t)
\end{equation*}
We note that this solution is only valid when $ \delta \sim \epsilon $, \textit{ie} when $ x \rightarrow 0 $. Hence, imposing only the boundary condition $ y(0) = 0 $, we have $ y_{in} = A(1 - e^{-2t}) $. We have already obtained the solution far away from zero, which is just $ y_{out} =e^{\frac{1-x}{2}}  $. So we have now got solutions at two extremities, and need to match these solutions at some intermediate points. If $ \theta(\epsilon) $ is the scale of the intermediate region, we can say it should satisfy the following,
\begin{equation*}
\lim\limits_{\epsilon \rightarrow 0} \dfrac{\theta}{\epsilon} = \infty \qq{and} \lim\limits_{\epsilon \rightarrow 0} \theta = 0
\end{equation*}
So it's natural to now consider another scale in the system $ x \sim \theta $ and substitute $ x = \eta \theta \implies t = \frac{\eta \theta}{\delta}$. Matching condition requires,
\begin{align*}
\lim\limits_{\epsilon \rightarrow 0} y_{out}(x=\eta \theta) &= \lim\limits_{\epsilon \rightarrow 0} y_{in}( x =\eta \theta) \\
\implies \lim\limits_{\epsilon \rightarrow 0}e^{\frac{1 - \eta \theta}{2}} &= \lim\limits_{\epsilon \rightarrow 0}A( 1 - e^{-2 \eta \theta/\delta}) \\
\implies e^{1/2} &= A
\end{align*}
Hence, we can now say that $ y_{in} =   e^{1/2}(1 - e^{-2x/\epsilon})$. To obtain a seemingly uniform solution, what we can do is add the inner and outer solutions and subtract the common part \textit{i.e.} $ e^{1/2} $,
\begin{equation*}
\implies y_{uni} = e^{1/2}(e^{-x/2} - e^{-2x/\epsilon})
\end{equation*}
which exactly agrees with our initial approximation.

This was a problem whose exact solution was known to us. Let's try to do a slightly trickier problem. Consider,
\begin{equation*}
\epsilon y''' - y' + xy = 0 \qq{,} y(0) = y'(0) = y(1)= 1
\end{equation*}
The unperturbed equation has solution $ y_{out} = A_0 e^{x^2/2} $. Let us consider the boundary layer of thickness $ \delta $ near $ x=0 $. Substituting $ t = \frac{x}{\delta} $, the equation becomes,
\begin{align*}
\dfrac{\epsilon}{\delta^3} \dddot{y} - \dfrac{1}{\delta} \dot{y} + xy = 0 &\qq{,} \dfrac{\epsilon}{\delta^3} \sim \dfrac{1}{\delta} \implies \delta \sim \sqrt{\epsilon} \\
\dddot{y} - \dot{y} = 0 \implies \dot{y} = A e^t + B e^{-t} &\implies y_{in} = A e^t - B e^{-t} + C \implies y_{in}' = \dfrac{A}{\sqrt{\epsilon}} e^{\frac{x}{\sqrt{\epsilon}}} + \dfrac{B}{\sqrt{\epsilon}} e^{-\frac{x}{\sqrt{\epsilon}}}
\end{align*}
We now impose boundary conditions $ y(0)= y'(0) = 1 \implies  $ and get $ y_{in}  = A (e^\frac{x}{\sqrt{\epsilon}} + e^{-\frac{x}{\sqrt{\epsilon}}}) + (1- 2A )$, where we have retained only the leading terms. If we try to match the inner and outer solutions, we see that there is now way that the final form of the solution satisfies the boundary condition $ y(1) = 1 $ . Hence we expect to have another layer at $ x=1 $. Now using t = $\frac{1-x}{\delta}$ and doing a similar procedure again, we obtain,
\begin{equation*}
y_{1in} = A' e^{\frac{1-x}{\sqrt{\epsilon}}} + B' e^{-\frac{1-x}{\sqrt{\epsilon}}} + (1 - A' - B')
\end{equation*}
Matching this with the outer solution gives $ A'=0, B'=1, A_0 = 1  \implies y_{out} = e^{x^2/2} $ and $ y_{1in} = e^{\frac{x-1}{\sqrt
\epsilon}} $. The perturbative solution upto leading order is, hence,
\begin{equation*}
y_{uni} = e^{x^2/2} (1 + \sqrt{\epsilon}) + e^{\frac{x-1}{\sqrt{\epsilon}}} (\sqrt{\epsilon }-1) - \sqrt{\epsilon} e^{-\frac{x}{\sqrt{\epsilon}}} + constant
\end{equation*}

\subsection{Initial Value Problem}
We consider a problem which is prevalant in chemical kinetics. without going into the details of it, we note that the governing equations are,
\begin{align*}
\dot{x}=-x + (x + \kappa - \lambda )y \qq{,} x(0)=1\\
\epsilon \dot{y} = x - (x + \kappa )y \qq{,} y(0)=0
\end{align*}
Here, $ \epsilon $ is typically $ \order{10^{-6}} $, and $ \kappa, \lambda $ are positive constants. The unperturbed equation is,
\begin{align*}
0 = x - (x + \kappa )y  &\implies y_{out} = \dfrac{x_{out}}{x_{out}+ \kappa}\\
\dot{x}=-x + (x + \kappa - \lambda )y = -\dfrac{\lambda x}{x + \kappa} &\implies x_{out} + \kappa \log x_{out} = - \lambda t + C
\end{align*}

To get a perturbative solution, we take inspiration from the boundary layer problems and say that there should be a initial time layer of thickness $ \delta $. Hence, changing variables $ \tau = t/\delta $, we see, with $ x' = \dv{x}{\tau} $,
\begin{equation*}
\dfrac{1}{\delta} x' = -x + (x + \kappa - \lambda )y \qq{and} \dfrac{\epsilon}{\delta} y' = x - (x + \kappa) y
\end{equation*}
Comparing scales, we see that $ \delta \sim \epsilon $, and hence
\begin{align*}
 x' = -\epsilon x + \epsilon (x + \kappa - \lambda )y \qq{and} y' = x - (x + \kappa) y
\end{align*}
As the boundary layer is near $ t = \tau = 0 $, the boundary conditions $ x(0)=1, y(0)=0  $ should be satisfied inside the boundary layer. Solving the above equations,
\begin{equation*}
x_{in } =1 \qq{and} y_{in} = \dfrac{1 - e^{- (\kappa + 1) t/\epsilon}}{\kappa + 1}
\end{equation*}

We now need to match solutions at intermediate times. Let $ \eta  $ be our matching scale in $ t $. Here,
\begin{equation*}
\lim\limits_{\epsilon \rightarrow 0} \dfrac{\eta}{\epsilon} = \infty \qq{and} \lim\limits_{\epsilon \rightarrow 0} \eta = 0
\end{equation*}
Matching the $ x$ solutions, we get $ C=1 $. To obtain the \textit{uniform solution}, we again add the inner and outer solutions and subtract the common part. Therefore,
\begin{align*}
x_{uni} = 1 + x_{out} -1 = x_{out} \\
y_{uni} = \dfrac{1 - e^{- (\kappa + 1) t/\epsilon}}{\kappa + 1} + \dfrac{x_{out}}{x_{out} + \kappa}
\end{align*}
where $ x_{out} $ is given implicitly as stated before.

\section{Method of Multiple Scales}
Consider the damped harmonic oscillator with the following governing equation,
\begin{equation*}
\ddot{x} + 2 \epsilon \dot{x} + x = 0 \qq{,}  x(0) = 0 \qq{,} \dot{x}(0) = 1 \implies x(t) = e^{-\epsilon t} \dfrac{\sin(t \sqrt{1-\epsilon^2})t}{\sqrt{1-\epsilon^2}}
\end{equation*}
We can see in this solution that there are two timescales governing the variation in the solution. One is the \textit{slow} timescale $ \tau = \epsilon t $ during which the solution has an exponential decay, and the other is the \textit{fast} timescale $ T = t \sqrt{1 - \epsilon^2} $ in which the solution varies sinusoidally. We could say then that a formulation of the problem should exist in $ \tau, T, \epsilon $. This is the philosophy behind the method of multiple scales.

We could solve this damped harmonic oscillator problem by multiple scales, but since it is easy and exactly solvable, let us look at a somewhat more complicated \textit{non-linear} equation namely the Duffing equation with damping given by,
\begin{equation*}
\ddot{x} + x + \epsilon(2 \lambda \dot{x} - x^3) = 0 \qq{,} x(0)=1 \qq{,} \dot{x}(0) = 0
\end{equation*}
We introduce two timescales into this problem,
\begin{equation*}
\tau = \epsilon t \qq{and} T = \qty(1 + \sum_{n=2}^\infty \epsilon^n \omega_n ) t \qq{and also} x(t) = X(\tau,T, \epsilon) = \sum_{n=0}^{\infty} \epsilon^n X_n(\tau,T)
\end{equation*}
As we have changed the timescales, we need to rewrite the time derivatives in the governing equation,
\begin{align*}
\dv{t} = \epsilon \dv{\tau} &+ \qty(1 + \sum_{n=2}^\infty \epsilon^n \omega_n )\dv{T} \\
\implies \epsilon^2 X_{\tau \tau} + \qty(1 + \sum_{n=2}^\infty \epsilon^n \omega_n )^2 X_{TT} &+ 2 \epsilon \qty(1 + \sum_{n=2}^\infty \epsilon^n \omega_n ) X_{T \tau} +
\\2 \epsilon \lambda \qty[\epsilon X_\tau + \qty(1 + \sum_{n=2}^\infty \epsilon^n \omega_n ) X_{T}]  &- \epsilon X^3 = 0
\end{align*}
Substituting expansion for $ X $ in the second equation above, one obtains the following,
\begin{align*}
X_{0TT} + X_0 = 0 \qq{,} X_{1TT}  + X_1 + 2 X_{0T \tau} + 2 \lambda X_{0T} - X_0^3 = 0 \\
X_{2TT} + X_2 + 2 \omega_2 X_{0TT} + 2 X_{1 T \tau} + X_{0 \tau \tau } + 2 \lambda (X_{1T} + X_{0 \tau}) - 3 X_0^2 X_1 = 0
\end{align*}
The initial conditions in terms of $ X, \dot{X}$ become,
\begin{align*}
X_0(0,0) = 1 &\qq{,} X_n(0,0 ) = 0 , \forall n \ne 0 \\
X_{0T}(0,0) = X_{0 \tau}(0,0) + X_{1 T}(0,0) &= X_{2T}(0,0) + \omega_2 X_{0T}(0,0) + X_{1T}(0,0) = 0
\end{align*}
Solving the first differential equation for $ X_0 $ and applying appropriate boundary conditions, we get,
\begin{equation*}
X_0(T, \tau) = A_0(\tau) \cos(T + \phi_0(\tau))  \qq{,} A_0(0) = 1 \qq{,} \phi_0( 0) = 0
\end{equation*}
With this solution, we can write the equation for $ X_1 $ as (with $ \theta_0 = T + \phi_0 $),
\begin{align*}
X_{1TT} + X_1 -2 A_0'\sin \theta_0 -2 A_0 \phi_0' \sin \theta_0 + 2 \lambda A_0 \sin \theta_0 - A_0^3 \cos^3\theta_0 = 0 \\
X_{1TT} + X_1 -2 A_0'\sin \theta_0 -2 A_0 \phi_0' \cos \theta_0 - 2 \lambda A_0 \sin \theta_0 - A_0^3 \dfrac{\cos 3\theta_0}{4} - 3A_0^3 \dfrac{\cos \theta_0}{4}   = 0
\end{align*}
Now, we want an overall periodic solution for $ X $, and hence $ X_1 $ should also exhibit periodic behaviour. For this to hold,
\begin{equation*}
A_0' + \lambda A_0 = 0 \qq{and} \phi_0' = - \dfrac{3}{8} A_0^2 \implies A_0 = e^{-\lambda \tau} \qq{and} \phi_0 = \dfrac{3}{16 \lambda} ( 1 - e^{-2 \lambda \tau})
\end{equation*}
With these substitutions, $ X_1 $ becomes,
\begin{equation*}
X_1(T,\tau) = A_1(\tau) \cos (T + \phi_1(\tau) ) + \dfrac{1}{32} e^{-3 \lambda \tau} \cos(3\{T + \phi_0(\tau)\})
\end{equation*}
The boundary conditions for $ X_1  $ are $ X_1(0,0)= 0 , X_{1T}(0,0) = - A'(0) $. Substituting gives,
\begin{equation*}
A_1(0) \cos(\phi_1(0)) = -\dfrac{1}{32} \qq{and} A_1(0) \sin(\phi_1(0)) = \lambda
\end{equation*}
Similarly, we substitute $ X_1 , X_0 $ in the equation for $ X_2 $. The calculations are long, but the conclusion derived after all that is $ \omega_2 = -\frac{1}{\lambda^2} $. So finally, we have out answer,
\begin{equation*}
x \sim e^{-\lambda\tau} \cos\qty[T + \dfrac{3}{16 \lambda}(1- e^{-2 \lambda \tau})] + \epsilon \qty{ A_1(\tau) \cos(T + \phi_1(\tau)) + \dfrac{1}{32} e^{-3 \lambda \tau}  \cos\qty[3T + \dfrac{9}{16 \lambda}(1- e^{-2 \lambda \tau}) ]}
\end{equation*}

We can apply this method to a whole range of differential equations.
\end{document}


\section{Figure and Output}
\subsection{Figure file: \lstinline[style=Plain]{lab_12_figure.pdf}}
\begin{figure}[!hbtp]
\centering
\includegraphics[width=\textwidth]{../src/lab_12_figure.pdf}
\caption{Romberg Integration error vs. $n$}
\end{figure}
\newpage
\subsection{Output file: \lstinline[style=Plain]{lab_12_output.txt}}
\lstinputlisting[style=Plain]{../src/lab_12_output.txt}
\newpage
\section{Function and Script}
\subsection{Function file:  \lstinline[style=Plain]{lab_12_romberg.m}}
\lstinputlisting[style=MATLAB]{../src/lab_12_romberg.m}
\newpage
\subsection{Script file: \lstinline[style=Plain]{lab_12_script.m}}
\lstinputlisting[style=MATLAB]{../src/lab_12_script.m}


\section{Multigrid for finite element methods}
By the definition of convolution \eqref{con1}, we can rewrite \eqref{2d-fe0} and \eqref{2d-fe1} as follows:
\begin{equation}\label{conA}
A\ast:  \mathbb{R}^{m\times n}\rightarrow \mathbb{R}^{m\times n},~~~A\ast u=f,~~\Leftrightarrow~~ u=\argmin J(v)=\argmin \left(\frac{1}{2}(A\ast v,v)-(f,v)_{l^2}\right)
\end{equation}
where $u=(u_{ij})$, $f=(f_{ij})$,
\begin{equation}\label{fe0_Ka}
A=\left\{
\begin{array}{ll}
	\begin{pmatrix}
	0 &-1&0\\
	-1& 4&-1\\
	0 &-1& 0
	\end{pmatrix}
&\text{for linear finite element,}	\\
%	\begin{equation}\label{fe1_Ka}
%	K_A=
	\begin{pmatrix}
	-1 &-1&-1\\
	-1& 8&-1\\
	-1 &-1& -1
	\end{pmatrix}
&\text{for bilinear finite element.}
\end{array}\right.
	\end{equation}

It is easy to see that
$$
\nabla J(v)= A\ast v - f := - r,\quad r=f-A\ast v.
$$
Applying the gradient descent method to \eqref{minProblem}, we obtain the following iterative method:
\begin{equation}\label{iterativeAst}
u^{k+1} = u^k + \eta r^k,\quad r^k = f-A\ast u^k.
\end{equation}
Here we can clearly see that gradient descent method is equivalent to
damped Jacobi method. Usually, we call this as smoother.
\begin{lemma}
The gradient descent method \eqref{iterativeAst} for linear finite element converges if $\eta={1\over 8} %\in (0,1/4)
$, and the one for bilinear finite element converges  if $\eta={1\over 16}% \in (0,1/8)
$. Furthermore, the high frequence in $u-u^k$ are damped very rapidly.
\end{lemma}
\begin{proof}
According to \eqref{iterativeAst} ,
$$
u^{k+1}-u=(I-\eta A)\ast (u^k-u).
$$
The gradient descent method \eqref{iterativeAst} converges if $\rho((I-\eta A)\ast)<1$, namely $\eta\rho(A\ast)<2$. For linear finite element, if $\eta= {1\over 8}$, $\rho(A\ast )<8$, thus the gradient descent method \eqref{iterativeAst} converges.
For bilinear finite element, if $\eta= {1\over 16}$, $\rho(A\ast )<16$, thus the gradient descent method \eqref{iterativeAst} converges.

For linear finite element, let $(\lambda_i, v_i)$ satisfy $A\ast v_i=\lambda_i v_i$ and $0<\lambda_1\leq \lambda_2\leq\cdots\lambda_{N}$ with $N=n^2$. Expand the error $u^k-u$ in terms of eigenvectors $v_i$, namely,
$$
u^k-u=\sum_{i=1}^{N} a_i^kv_i.
$$
Then
$$
u^{k}-u=\sum_{i=1}^{N} a_i^k(1-\eta \lambda_i)v_i=\sum_{i=1}^{N} a_i^0(1-\eta \lambda_i)^{k}v_i.
$$
According to Proposition \ref{prop:A}, it is easy to see that
$$
1-{1\over 8}\lambda_N\approx {\pi^2\over 4(n+1)^2} \ll 1.
$$
For $\eta={1\over 8}$, the coefficient $a_N^0(1-{1\over 8} \lambda_N)^{k}$ of $v_N$ approximates to zero much faster. This means that high frequency in the error will damp rapidly.
\end{proof}

For an initial guess $u^0$, the left picture in Fig \ref{fig:smooth} plots the error $u-u^0$ and the the right one plots the error $u-u^1$. Fig \ref{fig:smooth} shows that the high frequency in the error of the initial guess $u^0$ is damped after one step of smoothing and results in a smoother error $u-u^1$.
\begin{figure}
\centering
\includegraphics[width=5.5cm,height=5cm]{pictures/smooth0.jpg} \quad
\includegraphics[width=5.5 cm,height=5cm]{pictures/smooth10.jpg}\quad
\includegraphics[width=5.5cm,height=5cm]{pictures/smooth50.jpg}\quad
\includegraphics[width=5.5cm,height=5cm]{pictures/smooth100.jpg}
\caption{\footnotesize{The errors of an random initial guess $u^0$, $u^{10}$, $u^{50}$ and  $u^{100}$.}}
\label{fig:smooth}
\end{figure}
%\begin{figure}
%\includegraphics[width=12cm,height=8cm]{pictures/smoothing.pdf}
%\end{figure}
Next, we make the following particular choice:
\begin{equation}
  \label{eta}
\eta={1\over 8}.
\end{equation}

The gradient descent method can be written in terms of $S_{0}:\mathbb
R^{m\times n}\mapsto \mathbb R^{m\times n}$ satisfying
\begin{equation}
\label{jacobi1}
u^1=(S_{0}f)={1\over 8} f,
\end{equation}
for equation \eqref{2d-fe0} with initial guess zero.
If we apply this method twice, then
$$
u^2=S_1(f) = S_{0} f + S_0(f - A\ast(S_{0}f)),
$$
with element-wise form
\begin{equation}
\begin{aligned}
u^2_{i,j} &={3\over 16}f_{i,j} + {1\over 64}(f_{i+1,j}+f_{i-1,j}+f_{i,j+1}+f_{i,j-1}).
\end{aligned}
\end{equation}
Then by the definition of convolution \eqref{con1}, we have
 \begin{equation}\label{eq:convS}
u^1= S_{0}\ast f \quad u^2 = S_1 \ast f.
\end{equation}
with
\begin{equation}\label{eq:kernel-S}
S_{0} = {1 \over 8},
\end{equation}
and
\begin{equation}\label{eq:kernel-S2}
S_1={1\over 64} \begin{pmatrix}
0 & 1 & 0 \\
1 & 12 & 1  \\
0 &1  & 0
\end{pmatrix}.
\end{equation}

Next, we make the following particular choice for the bilinear case:
\begin{equation}
  \label{eta}
\eta={1\over 16}.
\end{equation}

The gradient descent method can be written in terms of $S_{0}:\mathbb
R^{m\times n}\mapsto \mathbb R^{m\times n}$ satisfying
\begin{equation}
\label{jacobi1}
u^1=(S_{0}f)_{i,j}={1\over 16} f_{i,j},
\end{equation}
for equation \eqref{2d-fe1} with initial guess zero.
If we apply this method twice, then
$$
u^2=S_1(f) = S_{0} f + S_0(f - A\ast(S_{0}f)),
$$
with element-wise form
\begin{equation}
\begin{aligned}
u^2_{i,j} &={3\over 32}f_{i,j} + {1\over 256}(f_{i+1,j}+f_{i-1,j}+f_{i,j+1}+f_{i,j-1}+f_{i+1,j+1}+f_{i-1,j-1}+f_{i-1,j+1}+f_{i+1,j-1}).
\end{aligned}
\end{equation}
Then by the definition of convolution \eqref{con1}, we have
 \begin{equation}\label{eq:convS}
u^1= S_{0}\ast f \quad u^2 = S_1 \ast f.
\end{equation}
with
\begin{equation}\label{eq:kernel-S}
S_{0} = {1 \over 16},
\end{equation}
and
\begin{equation}\label{eq:kernel-S2}
S_1={1\over 256} \begin{pmatrix}
1 & 1 & 1 \\
1 & 24 & 1  \\
1 &1  & 1
\end{pmatrix}.
\end{equation}

We note the gradient descent method \eqref{iterativeAst} can be written as
\begin{equation}\label{GD0}
u^k=u^{k-1} + S_0\ast (f-A\ast u^{k-1}).
\end{equation}
\begin{equation}\label{GD1}
u^{2k}=u^{2(k-1)} + S_1\ast (f-A\ast u^{2(k-1)}).
\end{equation}
We can sometimes use $S_1$ and the above identity to define a smoother
directly, namely
\begin{equation}\label{GD2}
u^{k}=u^{k-1} + S_1\ast (f-A\ast u^{k-1}).
\end{equation}

Similarly, with the smoother obtained by \eqref{eq:convS}, we can define
$S^{\ell}: \mathbb{R}^{m_\ell \times n_\ell} \mapsto \mathbb{R}^{m_\ell \times n_\ell}$.

First solve the problem on the fine grid $\mathcal T_{\ell}$, denote the solution by $u^{\ell}$,
\begin{equation}\label{eq:smoothing0}
u^{\ell} \leftarrow u^{\ell} + S^\ell \ast (f^\ell - A_\ell \ast u^{\ell}).
\end{equation}
Denote the error by $e^\ell=u-u^{\ell}$ and the residual
\begin{equation}\label{eq:formresidual}
r^{\ell}=f-A_{\ell}\ast u^{\ell}.
\end{equation}
It is obvious that $A_{\ell}\ast e^\ell=r^{\ell}.$
We need to solve the residual equation
\begin{equation}\label{erreq}
A_{\ell}\ast e^\ell=r^{\ell}.
\end{equation}
The idea of multigrid method is to solve this residual equation on the coarse grid space $\mathcal V_{\ell+1}$
and repeat the process until coarsest grid.

We note that the restriction of \eqref{erreq} to the coarse level $\ell+1$ is
\begin{equation}\label{coarse:correc}
A_{\ell+1}\ast e^{\ell+1}=r^{\ell+1}
\end{equation}
with $r^{\ell+1}=R_\ell^{\ell+1}r^\ell=R\ast r^\ell$.

Denote the finite element approximation to $e_\ell$ on the coarse grid $\mathcal T_{\ell+1}$ by $e_{\ell+1}$. Interpolate the error $e_{\ell+1}$ back to the fine space $\mathcal V_{\ell}$ and add the resulting residual to $u_{\ell+1}$, that is
\begin{equation}\label{eq:prolongation00}
u^\ell \leftarrow u^{\ell}+P^{\ell}_{\ell+1}e^{\ell+1}
\end{equation}
%as defined in \eqref{mg-prolong} and restriction $R_{\ell}^{\ell+1} = (P_{\ell+1}^{\ell})^T$.

Now using the smoother $S^\ell$, prolongation $P^{\ell}_{\ell+1}$, restriction $R_{\ell}^{\ell+1}$ and mapping
$A^\ell$ as given in \eqref{eq:def_coarse}, we can formulate the following algorithm
 as a major component of a multigrid algorithm.
\begin{breakablealgorithm}%[!htb]
	\caption{$(u^{1}, u^2, \cdots, u^J) = {\text{MG0}}(f; u^0; J,\nu_1, \cdots, \nu_J)$}
	\label{alg:L-Slash0}
	\begin{algorithmic}
		\State Set up
		$$
		f^1 = f, \quad u^{1}=u^0.
		$$
		\State Smoothing and restriction from fine to coarse level (nested)
		\For{$\ell = 1:J$}
		\For{$i = 1:\nu_\ell$}
		\State
		\begin{equation}\label{eq:smoothing}
		u^{\ell} \leftarrow u^{\ell} + S^\ell \ast (f^\ell - A_\ell \ast u^{\ell}).
		\end{equation}
		\EndFor
		\State Form restricted residual and set initial guess:
		$$
		u^{\ell+1,0} \leftarrow 0, \quad f^{\ell+1} \leftarrow R \ast_2 (f^\ell -  A_\ell \ast u^{\ell}), A_{\ell+1} = R \ast_2 A_\ell \ast (R\ast_2^\top).
		$$
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}
Here $S^\ell$ can be chosen as $S_0$ or $S_1$ as definition in \eqref{eq:kernel-S} and \eqref{eq:kernel-S2}.

Using the above algorithm, there are different multigrid algorithms such as: $\backslash$-cycle, V-cycle and W-cycle.
Let us now only give one special form of multigrid algorithm for solving \eqref{laplace} as follows.
\begin{breakablealgorithm}%[!htb]
	\caption{$u = {\text{MG1}}(f; u^0; J,\nu_1, \cdots, \nu_J)$}
	\label{alg:L-Slash1}
	\begin{algorithmic}
		\State
		$$
		u \leftarrow u^0.
		$$
		\State
		$$
		(u^{1}, u^2, \cdots, u^J) = {\text{MG0}}(f; u; J,\nu_1, \cdots, \nu_J).
		$$
		\State Prolongation and restriction from coarse to fine level
		\For{$\ell = J-1:1$}
		\State
		$$
		u^{\ell} \leftarrow u^{\ell} + R  \ast_2^{\top} u^{\ell+1}.
		$$
%		%		\IF{V-cycle}
%		\For{$i = 1:\nu_\ell$}
%		\State
%		$$
%		u^{\ell,i} \leftarrow u^{\ell,i-1} + [B^{\ell,i}]^T (f^\ell - A^{\ell} u^{\ell,i-1})
%		$$
%		\EndFor
%		%		\ENDIF
		\EndFor
		\State
		$$
		u \leftarrow u^{1}.
		$$
	\end{algorithmic}
\end{breakablealgorithm}

If we add the post-smoothing with a symmetric form which means we
use $[S^\ell \ast ]^\top$ as the smoother, then we can get the V-cycle
version multigrid algorithm.

\begin{breakablealgorithm}%[!htb]
	\caption{$u = {\text{MG2}}(f; u^0; J,\nu_1, \cdots, \nu_J; \nu'_1, \cdots, \nu'_J )$}
	\label{alg:V-cycle}
	\begin{algorithmic}
		\State
		$$
		(u^{1}, u^2, \cdots, u^J) = {\text{MG0}}(f; u^0; J,\nu_1, \cdots, \nu_J).
		$$
		\State Prolongation and restriction from coarse to fine level
		\For{$\ell = J-1:1$}
		\State
		$$
		u^{\ell} \leftarrow u^{\ell} + R  \ast_2^{\top} u^{\ell+1}.
		$$
		\For{$i = 1:\nu'_\ell$}
		\State
		$$
		u^{\ell} \leftarrow u^{\ell} + S^\ell \ast (f^\ell - A_\ell \ast u^{\ell})
		$$
		\EndFor
				%		\ENDIF
		\EndFor
		\State
		$$
		u = u^{1}.
		$$
	\end{algorithmic}
\end{breakablealgorithm}
Here $S^\ell $ means the central symmetry of kernel for smoother $S^\ell$ as in the
definition of \eqref{eq:def_tildeK} in Lemma \ref{lemm:tilde-K}.

We note that
$$
\text{MG1}(f; u^0; J,\nu_1, \cdots, \nu_J) = {\text{MG2}}(f; u^0; J,\nu_1, \cdots, \nu_J; 0, \cdots, 0).
$$

Either MG1 or MG2 only represents one cycle in a multigrid process. There many  different ways
to use this basis multigrid cycle. For a given iterate $u$, we need to define a metric to measure the
accuracy of $u$. One way to define it is:
$$
\text{error}(u) = \|f - A\ast u\|  \big/ \|f - A\ast u^0\|.
$$
Sometimes, when we debug a code, we can first try to find the exact solution. $u_{\text{exact}}$, and
then define
$$
\text{error}(u) = \| u_{\text{exact}} - u\|_A.
$$
Below is one example fo algorithm for application of the basic multigrid cycle, say MG1:
\begin{breakablealgorithm}%[!htb]
	\caption{$u = {\text{multigrid1}}(f; u^0; J,\nu_1, \cdots, \nu_J;  \text{tol})$;}
	\label{alg:multigrid-1}
	\begin{algorithmic}
		\State
		$$
		u \leftarrow u^0.
		$$
		\While{$\text{error}(u) \ge tol $}
		\State
		$$
		u \leftarrow u + \text{MG1}(f-Au;0;\nu_1, \cdots, \nu_J).
		$$
		\EndWhile
	\end{algorithmic}
\end{breakablealgorithm}

A slightly more general multigrid method.

\begin{breakablealgorithm}\label{alg:multigrid-Pi}
\begin{enumerate}
\item Initialization of inputs
		$$
		g_1 \leftarrow g, \quad
                u_{1}\leftarrow {\rm random}.
		$$
	\vspace{-.6mm}
\item Smoothing and restriction
  \begin{itemize}
  \item  For $\ell = 1:J$
    \begin{itemize}%3
    \item For $i = 1:\nu_\ell$
		\begin{equation}\label{eq:smoothing}
		u_{\ell} \leftarrow u_{\ell} + S_\ell \ast (g_\ell - A_\ell \ast u_{\ell}).
		\end{equation}
\item  Form restricted residual and set initial guess:
\begin{equation*}
u_{\ell+1,0} \leftarrow\Pi_\ell^{\ell+1}u_{\ell},  \quad g_{\ell+1} \leftarrow R_\ell \ast_2 (g_\ell -  A_\ell \ast u_{\ell}) + A_{\ell+1} \ast u_{\ell+1}^0,
\end{equation*}
    \end{itemize} %3
  \end{itemize}
\item  Prolongation with post-smoothing
  \begin{itemize} %4
  \item For {$\ell = J-1:1$}
		$$
		u_{\ell} \leftarrow u_{\ell} + R_\ell  \ast_2^{\top} (u_{\ell+1}-u_{\ell+1}^0).
		$$

                \begin{itemize}
  \item For $i = 1:\nu'_\ell$
		$$
		u_{\ell} \leftarrow u_{\ell} + S_\ell' \ast (g_\ell - A_\ell \ast u_{\ell})
		$$
                \end{itemize}
                \end{itemize} %4
\item Output
		$$
u_1
		$$
\end{enumerate} %1
\end{breakablealgorithm}

\section{Numerical examples}
We consider to solve
\begin{equation}
\label{laplacenu}
-\Delta u = f,  \mbox{ in } \Omega,\quad
u=0  \mbox{ on } \partial\Omega,\quad
\Omega=(0,1)^2.
\end{equation}
For the $x$ direction and the $y$ direction, we consider the partition:
\begin{equation}\label{partitionyx}
 0=x_0<x_1<\cdots<x_{n+1}=1, \quad x_i=\frac{i}{n+1},\quad (i=0,\cdots,n+1);
 \end{equation}
 \begin{equation}\label{partitiony}
 0=y_0<y_1<\cdots<y_{n+1}=1, \quad y_j=\frac{j}{n+1},\quad (j=0,\cdots,n+1).
\end{equation}
We use linear finite to discretize the 2D Laplacian equation \eqref{laplacenu}.  The size of
unknowns is $n^2$. We use gradient descent method as smoother in the multigrid method.
\begin{table}\label{table:multivsGS}%[htdp]
\begin{center}
\begin{tabular}{|c||c|c|}
\hline \hline
Size of Unkowns & Gauss-Seidel for $A$ & Multigrid for $A$ \\ \hline\hline %& V + PCG for $A$\\ \hline %\hline
225  &  226/0.04s     &    13/0.004s \\ \hline %&\onslide<3->{\brown{5/0.003s}}  \\ \hline
961   &  910/0.26s      & 13/0.007s
\\ \hline %& \onslide<5->{\brown{5/0.005s}} \\ \hline
3969   &  3,044/2.40s   &13/0.021s  \\ \hline %&  \onslide<7->{\brown{5/0.016s}} \\ \hline
16,129   & 9,869/31.45s    &13/0.08s \\ \hline %& \onslide<9->{\brown{5/0.06s}}  \\ \hline
65,025   &  30,226/347.38s  &13/0.3s\\ \hline %& \onslide<11->{\brown{5/0.22s}} \\   \hline \hline
\end{tabular}
\caption{Number of iterations for $\| Ax - b \|/ \|b\| \leq 10^{-6}$.}
\end{center}
\end{table}
\vspace{-30pt}
\begin{figure}[!ht]
\centering
%\setlength{\abovecaptionskip}{0pt}
%\setlength{\belowcaptionskip}{0pt}
\includegraphics[width=10cm]{figures/mgcompare.png}
\caption{ Comparison GD with Multigrid.}
\label{fig:Hmesh}
\end{figure}

From the Table \ref{table:multivsGS} and Figure \ref{fig:Hmesh}, we can see that the multigrid method is much faster than Gauss-Seidel method and is uniform with respect to the size of unknowns.

497-6DL/8 Convolutional Multigrid Method/8.12-FEM-MG-CONV.tex


%!TEX root = ../notes.tex
\section{January 31, 2022}
\subsection{Linear Combinations \emph{continued}}
Recall from last time that we proposed that
\[\text{greatest common divisor} \leq \text{least linear combination}.\]
\begin{example}
    $\gcd(2024, 748)=44$ because we have
    \begin{align*}
        2024 & = 748\cdot 2 + 528                                  \\
        748  & = 528\cdot 1 + 220                                  \\
        528  & = 220\cdot 2 + 88                                   \\
        220  & = 88\cdot 2 + \boxed{44} \leftarrow \gcd(2024, 748) \\
        88   & = 44\cdot 2 + 0
    \end{align*}
    We determine which linear combinations or $2024$ and $748$ we can create:
    \begin{align*}
        2024 & = 1\cdot 2024 + 0\cdot 748                         \\
        748  & = 0\cdot 2024 + 1\cdot 748                         \\
        528  & = 1\cdot 2024 + (-2)\cdot 748                      \\
        220  & = 748 - 1\cdot 528                                 \\
             & = 748 - 1\cdot (1\cdot 2024 + (-2)\cdot 748)       \\
             & = -1\cdot 2024 + 3\cdot 748                        \\
        88   & = 528 - 2\cdot 220                                 \\
             & = \underbrace{[1\cdot 2024 + (-2)\cdot 748]}_{528}
        - 2\cdot \underbrace{[-1\cdot 2024 + 3\cdot 748]}_{220}   \\
             & = 3\cdot 2024 - 8\cdot 748                         \\
        44   & = 220 - 2\cdot 88                                  \\
             & = [-1\cdot 2024 + 3\cdot 748]
        - 2\cdot [3\cdot 2024 - 8\cdot 748]                       \\
             & = -7\cdot 2024 + 19\cdot 748
    \end{align*}
\end{example}
Following this example, we have shown that every common divisor of $a$ and $b$ can be written as a linear combination of $a$ and $b$, and since the greatest common divisor has to be less than the least linear combination (as shown last time), the greatest common divisor \emph{is} the least linear combination\footnote{Assume for contradiction that the gcd were any less, then that would also be a linear combination. $\lightning$}.

We realize that there is a \emph{recurrence} happening here. If we call every set of coefficients $x, y$ and $z, w$ for $a$ and $b$ respectively, such that
\begin{align*}
    a & = x\cdot a_0 + y\cdot b_0 \\
    b & = z\cdot a_0 + y\cdot b_0
\end{align*}
where $a_0$ and $b_0$ are the original numbers, we can use a sliding window approach\footnote{Updating our iterators on every loop by sliding our window of coefficients down.} again to determine the next set of $x, y, z, w, a, b$.

Recall from last time we had
\begin{align*}
    a' & = b       \\
    b' & = a\mod b
\end{align*}
We can extend this algorithm for our new coefficients:
\begin{align*}
    x' & = z                                               \\
    y' & = w                                               \\
    z' & = w - \left\lfloor\frac{a}{b}\right\rfloor\cdot z \\
    w' & = y - \left\lfloor\frac{a}{b}\right\rfloor\cdot w \\
\end{align*}
where $\left\lfloor\frac{a}{b}\right\rfloor$ are the quotients from our Euclidean Algorithm. Note that initially, we have
\begin{align*}
    a & = 1\cdot a_0 + 0\cdot b_0 \\
    b & = 0\cdot a_0 + 1\cdot b_0
\end{align*}
so we have initial values of $x=1, y=0, z=0, w=0$.

so our code for the \emph{extended Euclidean Algorithm} is now
\begin{algorithm}[Extended Euclidean Algorithm]
    ~\lstinputlisting[numbers=none,language=Python]{code/ext_gcd.py}
\end{algorithm}

\subsection{Modular Arithmetic}
% *** Create recall environment
\recall We used a substitution/shift cipher to encrypt text:
\[\begin{array}{ccc}
        \mathtt{Y} & \mathtt{E} & \mathtt{S} \\
        \downarrow & \downarrow & \downarrow \\
        \mathtt{D} & \mathtt{J} & \mathtt{X}
    \end{array}\]
by incrementing 5 letters for each lecture.

$a = 0$, $b = 1$, \dots, $z=25$.

We had this notion of
\begin{align*}
    \text{ciphertext} & = \text{plaintext} + 5 \\
    \mathtt{d}        & = \mathtt{y} + 5       \\
    3                 & = 24+5 = 29
\end{align*}

\begin{definition}
    We say $a\equiv b\mod{m}$ if $m\mid a - b$.

    We say ``$a$ is congruent\footnote{Congruence is a ``behave like'' equality.} to $b$ modulo $m$''.
\end{definition}
\begin{example}
    \begin{align*}
        24 + 5 \equiv 3\mod{26} \\
        22 + 2 \equiv 1\mod{12} \\
    \end{align*}
    The first example is from our shift sipher, the second example is equivalent to ``two hours after 11:00, it is 1:00''.
\end{example}

\begin{proposition}
    If we have
    \begin{align}
        a_1          & \equiv a_2\mod{m} \nonumber                    \\
        b_1          & \equiv b_2\mod{m} \nonumber                    \\
        \intertext{Then we have the following: }
        a_1 + b_1    & \equiv a_2 + b_2\mod{m} \label{premod-add}     \\
        a_1 - b_1    & \equiv a_2 - b_2\mod{m} \label{premod-sub}     \\
        a_1\cdot b_1 & \equiv a_2\cdot b_2\mod{m} \label{premod-mult}
    \end{align}
\end{proposition}
\begin{proof}
    For \cref{premod-add}, realize that we have
    \[(a_1 + b_1) - (a_2 + b_2) = (a_1 - a_2) + (b_1 - b_2)\]
    and the two terms on the right are each divisible by $m$ by our premise. We can also write out
    \begin{align*}
        a_1 + b_1 & = (a_2 + \alpha m) + (b_2 + \beta m)     \\
                  & = (a_2 + b_2) + (\alpha + \beta)\cdot m.
    \end{align*}
    Similarly, for \cref{premod-sub}, we have
    \begin{align*}
        a_1 - b_1 & = a_2 + \alpha m - (b_2 + \beta m)      \\
                  & = a_2 - b_2 + (\alpha - \beta) \cdot m.
    \end{align*}
    and for \cref{premod-mult}, we have
    \begin{align*}
        a_1\cdot b_1 & = (a_2 + \alpha m)\cdot (b_2 + \beta m)                           \\
                     & = a_2\cdot b_2 + \alpha m b_2 + \beta m a_2 + \alpha\beta m^2     \\
                     & = a_2\cdot b_2 + (\alpha b_2 + \beta a_2 + \alpha\beta m)\cdot m.
    \end{align*}
    which concludes the proofs of the premod rules.
\end{proof}

\begin{proposition}\label{existence-of-inverse}
    There exists $b$ with
    \[a\cdot b\equiv 1\mod{m}\]
    if and only if $\gcd(a, m) = 1$.
\end{proposition}
\begin{proof}
    We can write linear combination equation
    \[a\cdot b + m\cdot k = 1\]
    and we have that the following are equivalent (we cascade down the list and can easily prove the iff relations):
    \begin{enumerate}[i.]
        \item such a $b$ exists,
        \item there is a solution $b, k$ to this equation,
        \item $1$ is a linear combination of $a$ and $m$,
        \item $1$ is the \emph{least} linear combination of $a$ and $m$,
        \item $1 = \gcd(a, m)$.
    \end{enumerate}
    so we have that $1 = \gcd(a, m)$ if and only if $a$'s inverse $b$ exists.
\end{proof}