\documentclass{stdlocal}
\begin{document}
\section{Implementation} % (fold)
\label{sec:implementation}
  In the section \ref{sec:pseudorandom_number_generators}, we have already discussed the mathematical foundations of PRNGs, as well as some examples, like the MT19937.
  In this section, we will first implement a scalar variant of the chosen generators to better understand their structure and design.
  Moreover, they serve as a testing facility to make sure the output of the vectorized PRNGs is correct.
  After explaining the critical points in the implementation, we will show how to vectorize the given generators.
  For this, it is helpful to first discuss which technique to use for the vectorization.
  Afterwards a theoretical analysis of the used vector intrinsics concerning their latency and throughput will follow to pinpoint bottlenecks of the implementation possibly resulting in future work.

  \subsection{MT19937} % (fold)
  \label{sub:mersenne_twister}
    The MT19937 is the de facto standard of applications using random numbers to compute their results.
    In the original paper, \citeauthor{matsumoto1998} have given a standard implementation in the C programming language.
    % This implementation is easily adjustable to work in C++.
    To map the C-style functional programming to modern C++ paradigms, we introduce a functor that stores the complete state of the Mersenne twister.
    The functional operator is then used to represent the application of the transition and generator function to receive a new random number.
    The introduction of the functor grants us the complete power of the C++ template, type, and class system while preserving an easy-to-use interface.

    \inputCodeBlock[title = Scalar MT19937 Structure]{code/mt19937/sisd/struct.hpp}
    In \textcite{kneusel2018}, a special seeding routine was used, such that the actual initialization process only needs one truly random number.
    In the STL of the C++ programming language the MT19937 is already implemented.
    The corresponding class is called \code{std::mt19937} and offers a constructor with only one integral number of 32-bit length as seeding value.
    This constructor uses the initialization routine described by \textcite{kneusel2018}.
    We want to make sure, that our implementation gives exactly the same output as the standard variant while keeping the advantages of our own API.
    As a consequence, we have to introduce another structure \code{default_seeder} which can be interpreted as a PRNG and should also be able to serve as a default seeder for the MT19937.
    The implementation of \code{default_seeder} is constructed in the same way as it was done for the MT19937 itself.
    First, we introduce a functor with the appropriate state while using the function operator as advancing routine.
    The initialization from \textcite{kneusel2018} was slightly changed to adapt to the new functor definitions.
    By designing a fast-forward constructor, it is possible to seed our own MT19937 with any other RNG including the default seeder.

    \inputCodeBlock[title = Scalar MT19937 Seeding]{code/mt19937/sisd/seeding.hpp}
    The types and parameters of the MT19937 will be defined as compile-time constants inside the functor to make sure the code stays maintainable by not inserting magic numbers in the implementation of the advancing routine.
    To guarantee that there is no runtime overhead, we define those variables as \code{static constexpr}.
    Please note that we could use a class template to parameterize such member types and constants as it is done in the STL.
    However, in consideration of the fact that we only want to vectorize the MT19937 of the Mersenne Twister family, it is completely sufficient to use non-template types and constants.
    The code is given in the appendix in listing \ref{code:mt19937-scalar-constants}.

    Advancing the state of the MT19937 by asking for a random number is a more complicated algorithm.
    The recurrence of the Mersenne twister is linear and gives us an equation on how to get to the next element based on the 624 current elements.
    According to the mathematics, after the computation we would have to move every element of the state one step further, as can be seen in figure \ref{fig:mersenne-twister-transition-generation}.
    But accessing each element in the state vector for every random number would greatly reduce the performance due to cache misses and the time to move every element.
    Instead, \citeauthor{matsumoto1998} cache the index of the current element in the state and are precomputing 624 steps of the transition function.
    This guarantees that newly generated elements do not have to be moved to the end of the state but instead can reside at their current positions.
    Because we are transforming every state element by applying the transition function 624 times, the precomputation step has to loop over all elements.
    To make sure the transformation is only accessing elements that lie inside the state, the loop has to be split into three distinct parts.
    The transition of a single element depends on two other elements in the state.
    Every dependency introduces a split because it could lie outside the state by shifting forward.
    In figure \ref{fig:mt-loop-scheme}, this process is shown schematically.

    \begin{figure}
      \center
      \begin{subfigure}[b]{0.26\textwidth}
        \center
        \includegraphics[width=0.95\textwidth]{figures/mt19937_transition_short.pdf}
      \end{subfigure}
      \begin{subfigure}[b]{0.71\textwidth}
        \center
        \includegraphics[width=0.95\textwidth]{figures/mt19937_loop_scheme.pdf}
      \end{subfigure}
      \caption[Mersenne Twister Loop Scheme]{
        Implementation of the Mersenne twister by precomputing its transition through a loop over the elements in the state.
        The left image shows a shorthand notation for a single call to the transition function.
        The right image visualizes the loop over all elements by using the simpler notation for a simplified version of the Mersenne Twister with a state size of eight and a shift size of five.
      }
      \label{fig:mt-loop-scheme}
    \end{figure}

    While the cached index is small enough, advancing the state only results in incrementing the index and returning the state value at this position by applying the generator function.
    Only if the index reaches the end, the complete state vector will be regenerated with respect to the recursive relation.
    In our code, the regeneration of the state vector uses a lambda expression to reduce code duplication and improve code transparency.
    The lambda expression keeps the code local in contrast to helper or member functions.
    Hence, the compiler will probably inline all calls to the lambda expression optimizing the scalar code in the best possible way.

    \inputCodeBlock[title = Scalar MT19937 Advancing]{code/mt19937/sisd/advance.hpp}
    The design of our Mersenne Twister ensures that the output will be the same as the output of the MT19937 from the STL of the C++ programming language.
    Additionally, the interface provides a more general seeding variant which improves statistical performance for seeding generators other than the default seeder.
    The new initialization facility is easier to use and gives a better {}insight into the used seeding values to the reader of the code.
    The function to advance the state is easier to understand, maintainable and generalizable.
    Last but not least, as we will show, our MT19937 implementation is able to provide a better performance in an actual application.

    Because the state of the MT19937 is very large in comparison to the size of the SSE and AVX vector registers, we rely on a typical vectorization technique described by \textcite{fog2015} that does not need to initialize multiple instances of the same generator.
    The scalar implementation offers a large degree of data independence.
    After the regeneration of the complete state vector, the generator function has to be applied on each individual element to provide random numbers.
    This problem describes the perfect use case for SIMD intrinsics.
    In the AVX case, we can read eight 32-bit values at the same time by the usage of a vector register and apply the generator function on each element in parallel by using the respective intrinsics.
    In the SSE case, we would only read four elements.
    The number of elements in the state vector is divisible by eight and four without remainder.
    Hence, we do not have to take care of the boundaries of the state vector.

    \inputCodeBlock[title = AVX MT19937 Structure]{code/mt19937/simd256/struct.hpp}
    The structure of the vectorized version does not introduce major changes to the underlying design.
    We make sure that the state vector will be at least 32-byte aligned to exploit faster loading and storing operations while advancing the state.
    To further optimize the structure, we could force 64-byte alignment to provide improved caching possibilities by letting the state start at a cache line boundary.

    The moment every package of eight or four elements in the state vector is read, the regeneration has to be triggered.
    The computation of the new state vector exhibits some data dependencies we have to take care of.
    In the scalar variant of the regeneration process the loop of all elements was split into three parts --- one for the first 227 elements, one for the following 396 elements and one for the last element.
    The splitting followed from the dependencies given by the recurrence defining the Mersenne twister algorithm.
    All elements in the first loop have to access the elements at the same position, one position ahead and 397 steps ahead.
    Reaching the 227th~element will then forbid us to go 397 steps further, because the position would not lie inside the state vector.
    But the requested element has already been produced at position zero.
    Therefore all elements in the second loop need to access the elements at the same position, one position ahead and 227 steps backwards.
    For the last element the successive element would again lie outside the state vector.
    Hence, the last elements uses the elements at the last position and positions zero and 396.

    \begin{figure}
      \center
      \includegraphics[width=0.95\textwidth]{figures/mt19937_vector_loop_scheme.pdf}
      \caption[Mersenne Twister Vector Loop Scheme]{
        Vectorization of the loop of Mersenne twister implementation by adding a buffer (gray) with the size of an SIMD vector at the end of the state.
        Here, the same simplified version as in figure \ref{fig:mt-loop-scheme} is used.
        It is assumed SIMD vectors are able to contain two elements of the state at once.
        Vectorizable operations are shown in boxes.
        By copying the first vector containing the new state elements into the buffer, no SIMD vector will overlap the first and the second loop.
      }
      \label{fig:mt-vector-loop-scheme}
    \end{figure}

    Thus, all newly generated elements in the first loop are independent of each other and can therefore be vectorized.
    This is also true for the second loop after the first loop has been executed.
    The problem lies in the fact that the number 397, as well as 227, is not divisible by eight or four without remainder.
    This leads to the appearance of a boundary vector term for the last element at the end of the state vector and one between the first and the second loop which needs to access successive and previous elements at the same time.
    Handling these boundary terms explicitly through vector intrinsics will result in complex loading, storing and blend operations.
    We have chosen another path by slightly changing the layout of the underlying data structure.
    Instead of saving only 624 elements for the state vector, we append values at the end of the state vector to get a buffer that should reduce complexity.
    The number of values we append is given by the amount of elements that fit into a vector register.
    After the generation of the first vector unit in the first loop, we will copy these values to the end into the buffer.
    In this case, the last value in the state vector will no longer be a boundary term because it can access the consecutive elements through the buffer.
    The boundary term between the first and the second loop can now be completely handled by the vectorized first loop.
    The new vectorization scheme of the complete loop is shown in figure \ref{fig:mt-vector-loop-scheme}.
    This method makes the code implementation easier by making the state bigger.
    But the number of elements we put into the buffer is small in comparison to the MT19937 state size and therefore an acceptable price to pay.

    % \inputCodeBlock[title = AVX MT19937 Advancing]{code/mt19937/simd256/advance.hpp}

    \inputCodeBlock[title = AVX MT19937 Advancing]{code/mt19937/simd256/advance.hpp}
    Because the state vector is aligned, we can exploit this by using aligned load operations for elements that will be used without a shift.
    For all other elements, we still have to use unaligned load operations.
    All parts in the vectorized advancing routine are then directly translated into their corresponding vector intrinsics.

    Except \code{_mm256_cmpgt_epi32}, all AVX intrinsics which are used in the code exhibit a latency of $1$ on the typical microarchitectures.
    \code{_mm256_cmpgt_epi32} has a latency of $3$ and a throughput of $1$ and therefore represents some kind of bottleneck that cannot be avoided.
    The throughput for loading and storing vector values is $0.25$.
    Hence, the loading of the three dependent values in the transition function should run in parallel.
    To even tweak the code further, we could always use two AVX vector registers at the same time by interleaving their operations.
    This would exploit the lower throughput of other intrinsics.
    In this case, only the shift operations in the generation function, like \code{_mm256_slli_epi32}, would bound the overall performance due to their throughput of $1$.
    \autocite{intel-intrinsics-guide,fog2019d}

    In listing \ref{code:mt19937-avx-advance-asm}, the assembly code of the vectorized advancing routine is shown.
    We have compiled the source code through the use of \citetitle{compiler-explorer} with the highest degree of optimization enabled \autocite{compiler-explorer}.
    It can be seen that the AVX intrinsics that were used for the implementation could directly be translated to their according assembler instructions.
    Besides, the optimization of the compiler is still active and helps to further exploit latencies and throughputs automatically.
    Hence, the use of SIMD intrinsics in C++ code can be seen as superior in comparison to writing inline assembly code.
  % subsection mersenne_twister (end)

  \subsection{Xoroshiro128+} % (fold)
  \label{sub:xoroshiro}
    The scalar implementation of the Xoroshiro128+ follows the same rules as the scalar implementation of the MT19937.
    \citeauthor{vigna-xoroshiro} provides the C code for the generator on his website \autocite{vigna-xoroshiro}.
    We again use a functor which saves the state and advances it by calling the function operator.
    Compile-time constants will again be declared as \code{static constexpr} to reduce the runtime overhead.
    In comparison to the MT19937, the state of the Xoroshiro128+ can be easily described by two 64-bit unsigned integer values.
    Hence, we only show the data structure in the appendix in code snippet \ref{code:xoroshiro-scalar-struct}.
    For the advancing routine, the generator needs a utility which is rotating the bits of a 64-bit unsigned integer.
    % \inputCodeBlock[title = Scalar Xoroshiro128+ Structure]{code/xoroshiro128plus/sisd/struct.hpp}
    Furthermore, the Xoroshiro128+ offers a jump-ahead feature for discarding $2^{64}$ or $2^{96}$ elements of the output.
    In listing \ref{code:xoroshiro-scalar-jump}, we are only providing the implementation of the smaller jump.
    Both jumping routines differ only in their masking numbers.
    % The testing of the jump functions seems to be problematic because we have no possibility of generating $2^{64}$ random numbers in a short time to check if state of the PRNG is the same after a call to the jump function.
    % One workaround is to check each jump against each other by executing $2^{32}$ small jumps

    \inputCodeBlock[title = Scalar Xoroshiro128+ Advancing]{code/xoroshiro128plus/sisd/advance.hpp}
    Due to the small state size of the Xoroshiro128+, the vectorization has to use multiple instances of the same generator to exploit the full capabilities of the SIMD intrinsics.
    \citeauthor{vigna-xoroshiro} only gives one parameter set for the PRNG.
    As a consequence, we have to use different seeds for all instances possibly causing overlapping subsequences or the jump-ahead feature to make sure the instances do not overlap for at least $2^{64}$ or $2^{96}$ values.
    For the AVX implementation, we will use four instances of the Xoroshiro128+.
    The underlying structure therefore does not really change and uses the SIMD types instead of the 64-bit unsigned integer types.
    The layout of the data structure for the AVX version can be seen in figure \ref{fig:xoroshiro-vector-layout}.

    \begin{figure}
      \center
      \includegraphics[width=0.95\textwidth]{figures/xrsr128p_vector_layout.pdf}
      \caption[Xoroshiro128+ Vector Layout]{%
        Memory layout concerning SIMD vectors of four instances of the Xoroshiro128+.
        SIMD vectors are shown as blue box.
      }
      \label{fig:xoroshiro-vector-layout}
    \end{figure}

    % \inputCodeBlock[title = AVX Xoroshiro128+ Structure]{code/xoroshiro128plus/simd256/struct.hpp}
    With this approach, every instance of the generator is completely independent of the other instances.
    This is again perfect for the application of SIMD intrinsics.
    The AVX implementation of the advancing routine is therefore straightforward because we are able to interchange every scalar instruction with its corresponding vector intrinsic.

    \inputCodeBlock[title = AVX Xoroshiro128+ Advancing]{code/xoroshiro128plus/simd256/advance.hpp}
    Due to the latencies and throughputs of the AVX intrinsics used in this implementation, the advancing routine does not introduce a major bottleneck in the generation of random numbers.
    But to exploit the throughput of every operation, we would need to use multiple independent SIMD vector registers at once.
    This would increase the number of SIMD registers in the return type of the function as well.

    The jump function, given in listing \ref{code:xoroshiro-avx-jump}, uses a branch in the inner loop to decide which code path to execute.
    We do not want this branch to slow down the code by switching to scalar-execution mode.
    Hence, we will map the branch to vector intrinsics by executing the inner computation in every case and masking the result based on the vectorized evaluation of the branch condition.
    Seeding the AVX version of the Xoroshiro128+ with the usage of the vectorized jump routine can then be done by first initializing the four instances with the same random seed.
    Afterwards, three vectorized jumps have to be executed.
    For every step, we have to use masking operations to make sure the jumping is only executed for the instances in the AVX register that have to advance further.
    So after the computation of the first jump, the first instance has to be masked.
    For the second step, only the third and the fourth instance should change and therefore the first and the second instance are masked.
    The third jump then only has to be done for the fourth instance.

    In the code \ref{code:xoroshiro-avx-advance-asm}, \ref{code:xoroshiro-avx-advance2-asm}, and \ref{code:xoroshiro-avx-advance4-asm}, we provide the assembly code generated by \citetitle{compiler-explorer} for calling the advancing routine once, twice, and four times, respectively \autocite{compiler-explorer}.
    As before, all AVX intrinsics were correctly translated into their respective assembler instructions.
    Looking at the listings \ref{code:xoroshiro-avx-advance2-asm} and \ref{code:xoroshiro-avx-advance4-asm}, we even see that the compiler completely inlined multiple calls to the advancing routine automatically.
    In contrast to the MT19937, the AVX implementation of the Xoroshiro128+ enables us to keep using the vector registers without cache or memory communication while generating new random numbers.
    This property will provide a higher performance as we will see in the evaluation process.

  % subsection xoroshiro (end)

  \subsection{MSWS} % (fold)
  \label{sub:middle_square_weyl_generator}
    The MSWS is a simple, modern, and non-linear PRNG that uses a state which is based on 64-bit unsigned integer values but is returning only a 32-bit unsigned integer.
    The scalar implementation will again use the same approach as the MT19937 and the Xoroshiro128+ as it is given as a C implementation in \textcite{widynski2019}.
    The seeding routine for the MSWS is more complicated and requires a more sophisticated algorithm.

    \inputCodeBlock[title = Scalar MSWS]{code/msws/sisd/struct.hpp}
    The MSWS does not provide different parameters or a jump-ahead feature.
    Its small state size requires us to use multiple instances of independent generators to exploit the size of the vector registers.
    Thus, all instances have to be initialized with different seeds possibly generating overlapping subsequences.
    But because its output is only given by a 32-bit value, we have to generate eight random numbers at once.
    We can decide to call four instances of the same generator twice or instead use eight instances.
    While benchmarking the generators, the variant for calling each instance twice reduced the actual throughput of the vectorized PRNG and was therefore discarded from the implementation and measurement.
    In our vectorized implementation, we use eight instances of the MSWS.
    A visualization of the data structure layout is given in figure \ref{fig:msws-vector-layout}.

    \begin{figure}
      \center
      \includegraphics[width=0.95\textwidth]{figures/msws_vector_layout.pdf}
      \caption[MSWS Vector Layout]{%
        Memory layout concerning SIMD vectors of eight instances of the MSWS.
        SIMD vectors are shown as blue box.
      }
      \label{fig:msws-vector-layout}
    \end{figure}

    % \inputCodeBlock[title = AVX MSWS Structure]{code/msws/simd256/struct.hpp}
    Advancing the state of the vectorized MSWS introduces difficulties because we have to compute the square of the first 64-bit value.
    Neither the AVX nor the SSE instruction set is providing a multiplication or squaring operation for 64-bit unsigned integer numbers.
    Accordingly, we have to implement our own function for squaring the 64-bit values in a vector register based on given intrinsics.
    For this, we will map the 64-bit multiplication to 32-bit multiplications.
    Let $x\in\setInteger_{2^{64}}$ be a 64-bit unsigned integer and let $x_0,x_1\in\setInteger_{2^{32}}$ be its 32-bit representation, such that
    \[
      x = x_1 2^{32} + x_0 \ .
    \]
    We are then able to formulate the squaring of this number in 64-bit arithmetic by using modulo and the binomial formula.
    \begin{align*}
      x^2
      &= x_1^2 2^{64} + 2x_1x_0 2^{32} + x_0^2 \\
      &\equiv 2x_1x_0 2^{32} + x_0^2 \mod 2^{64}
    \end{align*}
    Due the modulo, the product $x_1x_0$ only needs to use a 32-bit multiplication.
    The value $x^2_0$ can be fully expressed by a 64-bit integer type.
    Hence, we have to use a 64-bit multiplication for 32-bit number $x_0$ to deal with the carry bits of the computation $x_0^2$.
    The SSE and AVX instruction sets provide this operation, such that the added complexity of the computation can be used to find an optimal implementation.

    Afterwards, the first part of the vectorization is again straightforward because all the instances are independent so that every statement can directly be interchanged with its corresponding vector intrinsic.
    At the end of the first part, two variables of SIMD type will provide 64-bit values as random numbers.
    The MSWS algorithm forces us to use only the lower 32-bit of the given values resulting in only one vector type with the doubled amount of values as result.
    To do that, both variables have to be convoluted by shuffle or permutation operations.
    Two variants are shown in figure \ref{fig:msws-vector-merge-scheme}.
    % The squaring function and the permutation operations at the end will probably exhibit a high latency and therefore reduce the speed-up of the vectorization.

    \begin{figure}
      \center
      \begin{subfigure}[b]{\textwidth}
        \center
        \includegraphics[width=0.95\textwidth]{figures/msws_merge.pdf}
        \caption{Sequential.}
      \end{subfigure}

      \begin{subfigure}[b]{\textwidth}
        \center
        \includegraphics[width=0.95\textwidth]{figures/msws_merge2.pdf}
        \caption{Interleaved.}
      \end{subfigure}
      \caption[MSWS Vector Merge Scheme]{%
        Two variants of merging the two output vectors of the vectorized MSWS containing 64-bit integers into one SIMD vector containing 32-bit integers by means of AVX vector registers.
      }
      \label{fig:msws-vector-merge-scheme}
    \end{figure}

    \inputCodeBlock[title = AVX MSWS Advancing]{code/msws/simd256/advance.hpp}
    In the AVX implementation, we use \code{_mm256_mul_epu32} and \code{_mm256_mullo_epi32} to compute the square of the first element in the state.
    They exhibit a latency of $5$ and $10$ and a throughput of $0.5$ and $1$ on the typical microarchitectures, respectively.
    Thus, in comparison to the other intrinsics, they tremendously reduce the performance of the advancing routine.
    To exploit their throughput, we would have to run ten of these instructions for independent AVX vectors consecutively.
    One 256-bit SIMD vector contains four 64-bit MSWS generators.
    Therefore at least 40 instances of the PRNG have to be called in parallel to remove the latency bottleneck of the multiplication operations.
    Due to the risk of lowering the statistical quality, we have only used eight generators in parallel.
    \autocite{intel-intrinsics-guide,fog2019d}
  % subsection middle_square_weyl_generator (end)

  \subsection{Uniform Distribution Functions for Reals} % (fold)
  \label{sub:uniform_real_distribution}
    The uniformly distributed transformation of random 32- and 64-bit integers to single- and double-precision floating-point values is described in \textcite{vigna-xoroshiro} and \textcite{kneusel2018}.
    Here, the IEEE 754 floating-point standard is used to ensure that bit manipulations will result in the same number on every machine supporting this standard.
    Because we only want to deal with normalized numbers, a value in the interval $[1,2)$ should be generated firstly.
    Such a value is positive and therefore the sign bit of the floating-point value is set to $0$.
    The actual number representing the exponent of the floating-point encoding should be zero.
    Due to the offset applied to the exponent value, the content of the exponent has to be equal to the offset.
    Then the fraction can be filled with random bits to construct a uniformly distributed floating-point value in the interval $[1,2)$.
    Low-order bits of a pseudorandom integer tend to provide weak randomness properties.
    Hence, a shift is used to fill the fraction with the most significant bits of the random integer.
    In figure \ref{fig:real-uniform-implementation-scheme}, this can be seen for a 32-bit integer and a single-precision floating-point value.
    After this computation, the remaining part has to subtract $1$ from the floating-point value to guarantee the uniform distribution in the unit interval $[0,1)$.
    The scalar implementation is given in the following code snippet.

    \begin{figure}[h]
      \center
      \includegraphics[width=0.95\textwidth]{figures/uniform_implementation_scheme.pdf}
      \caption[Real Uniform Distribution Function Implementation Scheme]{%
        Generating a uniformly distributed floating-point number by shifting bits of a given random integer of the same size.
        According to the IEEE 754 floating-point standard, sign and exponent bits have to be set, such that the number will be in the interval $[1,2)$.
      }
      \label{fig:real-uniform-implementation-scheme}
    \end{figure}

    \inputCodeBlock[title = Scalar Uniform Function]{code/uniform/sisd/uint32.hpp}
    \newpage
    \noindent
    The vectorization of the uniform routine is straightforward.
    Every operation can be replaced by its respective intrinsic.
    Analyzing the function, \code{_mm256_sub_ps} and \code{_mm256_sub_pd} exhibit the highest latency with a value ranging from $3$ to $4$ on typical microarchitectures \autocite{intel-intrinsics-guide,fog2019d}.
    On the other hand, their throughput ranges from $0.5$ to $1$.
    So to further improve the performance of the uniform function, multiple independent values should be computed resulting in better exploitation of the processor pipeline.

    \inputCodeBlock[title = AVX Uniform Function]{code/uniform/simd256/__m256i.hpp}
    % \inputCodeBlock[title = AVX Uniform Function with Bounds]{code/uniform/simd256/bounds.hpp}

  % subsection uniform_real_distribution (end)

  % \subsection{Uniform Distribution Functions for Integers} % (fold)
  % \label{sub:uniform_distribution_functions_for_integers}

  % % subsection uniform_distribution_functions_for_integers (end)
% section implementation (end)
\end{document}


\documentclass[twocolumn]{article}
\usepackage{amsmath,cite}
\usepackage{graphicx}

\usepackage{amssymb}

\usepackage{url}

\usepackage[framed,numbered,autolinebreaks,useliterate]{../common/mcode}

\graphicspath{{figures/}}

\title{SCATNET Architecture Document}

\begin{document}

\maketitle

\section{Filters}

\subsection{Filter Structure}

Filters are defined by a signal size \mcode{[N,M]}, a filter type (Morlet, Gabor, spline), and wavelet-specific parameters. For one-dimensional signals, $M = 1$.

Filter parameters are specified in a parameters structure, \mcode{fparam}, containing the following fields:
\begin{itemize}
	\item \mcode{fparam.filter_type}: The wavelet type, such as `morlet\_1d', `morlet\_2d', `spline\_1d', for example.
	\item \mcode{fparam.precision}: The numeric precision of the filters. Either \mcode{'double'} or \mcode{'single'}.
\end{itemize}
In addition, the \mcode{fparam} structure would contain parameters specific to the wavelet type chosen (see below).

Once filter parameters are entered, the \mcode{filter_bank} function is called
to generate the filter bank:
\begin{lstlisting}
	filters = filter_bank([N M], fparam);
\end{lstlisting}
This function will then call the appropriate filter bank function (\mcode{spline_filter_bank_1d}, \mcode{morlet_filter_bank_2d}, etc.) depending on the value of \mcode{fparam.filter_type} and put it in a cell array.

If one of the parameters is an array (except for \mcode{fparam.filter_type} and \mcode{fparam.precision}, which have to be cell arrays), \mcode{filter_bank} will create multiple filter banks and output them. For example, if \mcode{fparam.filter_type} equals \mcode{\{'gabor\_1d','spline_1d'\}}, \mcode{filter_bank} will output a cell array of two filter banks, one with Gabor wavelets and one with spline wavelets. If parameter fields are of different sizes, the shorter ones are extended by concatenating the last value the required number of times.

The returned structure, \mcode{filters}, contains the filters $\psi$ and $\phi$ that form the filter bank, as well as meta information. Specifically, the fields are:
\begin{itemize}
	\item \mcode{filters.psi}: A set of wavelet filters $\psi_\lambda$ (for definition, see below)
	\item \mcode{filters.phi}: A set of lowpass filter(s) $\phi$ (for definition, see below)
	\item The parameters given in \mcode{fparam} and the signal size $\mcode{[N,M]}$. For example, \mcode{filter.filter_type} gives the type of filters in \mcode{filters.psi} and \mcode{filters.phi}.
\end{itemize}

Each filter set (be it \mcode{filters.phi} or \mcode{filters.psi}), is a structure \mcode{fset} containing the following:
\begin{itemize}
	\item \mcode{fset.filter}: A cell array of the actual filter coefficients. These coefficients are implementation-dependent and can encode the filter spatially, in the Fourier domain, at different resolutions, etc.
	\item \mcode{fset.meta}: Contains meta information on the filters. Specifically, it has two fields: \mcode{fset.meta.j}, which is the scale indices, and \mcode{fset.meta.theta}, which is the angle indices (for two-dimensional filters). Both \mcode{fset.meta.j} and \mcode{fset.meta.theta} are of the same length as \mcode{fset.filter}.
\end{itemize}

The scale and angle indices are non-negative integers. The scale index rises with increasing scale, while the angle index rises with increasing angle (counter-clockwise).

\subsection{Morlet/Gabor filter bank}

In addition to the parameters listed above, the Morlet/Gabor filter bank has the following options:
\begin{itemize}
	\item \mcode{fparam.Q}: The number of wavelets per octave. By default $1$.
	\item \mcode{fparam.J}: The number of wavelet scales.
	\item \mcode{fparam.B}: The reciprocal octave bandwidth of the wavelets. By default \mcode{fparam.Q}.
	\item \mcode{fparam.sigma_psi}: The standard deviation of the mother wavelet in space. By default calculated from \mcode{fparam.B}.
	\item \mcode{fparam.sigma_phi}: The standard deviation of the scaling function in space. By default calculated from \mcode{fparam.B}.
	\item \mcode{fparam.slant}: The slant of the mother wavelet ellipse in frequency.
	\item \mcode{fparam.nb_angle}: The number of wavelet angles.
\end{itemize}
The maximal wavelet bandwidth (in space) is determined by $2^{J/Q}$ times the bandwidth of the mother wavelet, which is proportional to \mcode{sigma\_psi}. If \mcode{sigma\_psi} is smaller than a certain threshold, a number of constant-bandwidth filters are added, linearly spaced, to cover the low frequencies.

Again, we can specify different filter banks by setting \mcode{fparam.Q} and \mcode{fparam.J}, etc. to arrays instead of scalars. This is often useful if the nature of the signal is different at different orders, which is usually the case in audio.

To calculate the appropriate \mcode{J} necessary for a given window size \mcode{T} and filter parameters \mcode{Q}, etc, there is the conversion function \mcode{T_to_J}, which is called using:
\begin{lstlisting}
	fparam.J = T_to_J(T, fparam.Q);
\end{lstlisting}
This will ensure that the maximum wavelet scale, and thus the averaging scale of the lowpass filter, will be approximately \mcode{T}.

\subsection{Spline filter bank}

In addition to the parameters listed above, the spline filter bank has the following options:
\begin{itemize}
	\item \mcode{fparam.J}: The number of wavelet scales.
	\item \mcode{fparam.spline_order}: The order of the splines. Only linear (spline order $1$) and cubic (spline order $3$) are supported.
\end{itemize}
The maximal bandwidth is specified here by $2^J$.

\section{Wavelet and Scattering Transforms}

\subsection{Wavelet Transform}

Given a signal \mcode{x}, a filter bank \mcode{filters}, and an options structure \mcode{options}, the wavelet transforms \mcode{wavelet_1d} and \mcode{wavelet_2d} decompose the former into a lowpass part \mcode{x_phi} and a cell array of its wavelet coefficients \mcode{x_psi}. The 1D version is called in the following manner:
\begin{lstlisting}
	[x_phi, x_psi] = wavelet_1d(x, filters, options)
\end{lstlisting}
The outputs are sampled according to their frequency bandwidth.

This transform can also be applied on a layer of coefficients, representing layers of a scattering transform. These layers are defined as structures with fields:
\begin{itemize}
	\item \mcode{layer.signal}: A cell array of signals.
	\item \mcode{layer.meta}: The meta information on the signals, such as their path, their resolutions, etc.
\end{itemize}
The signals are one-dimensional or two-dimensional arrays while \mcode{meta} contains the fields \mcode{meta.j} and \mcode{meta.theta}, which are two-dimensional
arrays. The first dimension has length corresponding \mcode{layer.signal} while the second dimension as length corresponding to the order of the coefficients.
The path of the $p$th coefficient is thus encoded in \mcode{meta.j(:,p)} and \mcode{meta.theta(:,p)}, respectively.

The corresponding layer transforms are \mcode{wavelet_layer_1d} and \mcode{wavelet_layer_2d}, the former being applied to the layer \mcode{U} by calling
\begin{lstlisting}
	[U_phi, U_psi] = wavelet_layer_1d(U, filters, options);
\end{lstlisting}

In addition, there is a function \mcode{modulus_layer}, which computes the complex modulus of the coefficients in the layer.

Combining the wavelet transform with the modulus operator, we can construct a scattering transform. First, we initialize $U\{1\}$ with the input signal, then
iterate the following calls:
\begin{lstlisting}
	[S{m}, W] = wavelet_layer_1d(U{m}, filters, options);
	U{m+1} = modulus_layer(W);
\end{lstlisting}
Then \mcode{S\{m+1\}} and \mcode{U\{m+1\}} will contain the $m$th-order scattering and wavelet modulus coefficients, respectively.

In the next section, we see how this can be automated.

\subsection{Scattering Transform}

By alternating wavelet transforms and modulus operators, we obtain the scattering transform. Specifically, the \mcode{scat} function takes a signal, an options
structure, a structure defining the wavelet cascade, and returns the scattering coefficients (or wavelet modulus coefficients, if desired). The scattering
coefficients are output as a cell array of layers \mcode{S}.

To create this cascade structure, a factory function, \mcode{wavelet_factory_1d} or \mcode{wavelet_factory_2d} is used, which takes as input the signal size, a
 filter parameter structure, a scattering options structure, and the order of the transform. The scattering transform could be used like the following

\begin{lstlisting}
	fparam.filter_type = {'gabor_1d', 'morlet_1d'};
	fparam.Q = [8 1];
	fparam.J = T_to_J(8192,fparam.Q);

	options = struct();

	wavelet = wavelet_factory_1d(N, fparam, options, 2);

	S = scat(x, wavelet);
\end{lstlisting}

The above code will compute a filter bank for signals of length $N$, with the first filter bank consisting of $8$ filters per octave. The second filter bank
will only have one wavelet per octave, with the corresponding bandwidth, and $13$ octaves of wavelets. The averaging scale of both filter banks is set to $T = 8192$.
These filters are then used to specify the wavelet transforms that are stored in the cascade structure.

To obtain the wavelet modulus coefficients \mcode{U}, \mcode{scat} is called for two outputs, as in
\begin{lstlisting}
[S, U] = scat(x, wavelet);
\end{lstlisting}
The structure of \mcode{U} follows that of \mcode{S}. That is it consists of a cell array representing each layer of wavelet modulus coefficients. The first layer
corresponds to the zeroth-order wavelet modulus coefficients, which is the original signal. The second layer contains the first-order wavelet modulus coefficients,
which are the signal convolved with the filters with the modulus applied, and so on. The coefficients in \mcode{S\{m+1\}} are thus the $m$th-order scattering
coefficients obtained from smoothing the coefficients in \mcode{U\{m+1\}}.

\section{Manipulating, Displaying, Formatting}

\subsection{Renormalization and Logarithm}
Often, second- and higher-order coefficients will reproduce information from their parent coefficients. That is, they will be highly correlated. To reduce this,
the \mcode{renorm\_scat} function renormalizes each coefficient by dividing it by its parent coefficient. Similarly, the \mcode{log\_scat} function calculates
the logarithm of each coefficient.

These functions both act on the output of \mcode{scat}, and so can be called on the scattering transform \mcode{S} like:
\begin{lstlisting}
	S_renorm = renorm_scat(S);
	S_renorm_log = log_scat(S_renorm);
\end{lstlisting}

\subsection{Display (1D)}
Two functions are available to display scattering coefficients, \mcode{display_slice} and \mcode{display_multifractal}. They both take as input a scattering transform \mcode{S} and a time point \mcode{t}.

\subsection{Display (2D)}
To display the scattering of 2d functions,
\begin{lstlisting}
image_scat(sx);
\end{lstlisting}
opens one figure per layer. In each figure, images corresponding to each node of the layer are concatenated.

To display only coefficients of a certain order \mcode{m}, one can use \mcode{image_scat_layer}
\begin{lstlisting}
m = 2;
image_scat_layer(sx{m+1});
\end{lstlisting}

To arrange the coefficients in a specified order (lexicographic order on $j$ for rows and lexicographic order on $\theta$ for columns), one can also use \mcode{image_scat_layer_order}
\begin{lstlisting}
var_y{1}.name = 'j';
var_y{1}.index = 1;
var_y{2}.name = 'j';
var_y{2}.index = 2;
var_x{1}.name = 'theta';
var_x{1}.index = 1;
var_x{2}.name = 'theta';
var_x{2}.index = 2;
big_img = image_scat_layer_order(S{3},var_x, var_y,1);
\end{lstlisting}

\subsection{Formatting}
In order to use the scattering coefficients for classification, they need to
be in a vector format. This is obtained using the function
\mcode{format_scat}, which arranges scattering coefficients into a
two-dimensional table, the first dimension corresponding to a scattering
coefficient index and the second dimension corresponding to time/space. It
also returns a meta structure that specifies the order, scale, etc. of each
scattering coefficient. The following example illustrates its usage:

\begin{lstlisting}
	[sc_table, meta] = format_scat(S);

	% plot the 2nd-order coefficients
	% corresponding to scale (3, 7)
	ind = find(meta.order==2 & meta.j(1,:)==3 & meta.j(2,:)==7);
	plot(sc_table(:,ind));

	% calculate the energy of 1st-order
	% coefficients
	ind = find(meta.order==1);
	E2 = norm(sc_table(:,ind),'fro')^2;
\end{lstlisting}

Note that if two or more filter banks are used when calculating \mcode{S}, formatting is only possible if the lowpass filters $\phi$ have the same bandwidth, since otherwise scattering coefficients of different orders will have different resolutions and so cannot be fitted into the same matrix without resampling. Using the \mcode{T_to_J} function will ensure that this is always the case.

\section{Classification}

\subsection{Batch Computation}

To compute scattering coefficients for a database of signals, we first define a source. A source is a set of files, each file containing a number of objects to be classified. Each object has a class associated with it and a location within the file.

The source can be created using \mcode{create_src(directory, extract_objects_fun)} (or one of its wrappers, such as \mcode{gtzan_src}). Given a directory, this function recursively traverses it, looking for `.jpg', `.wav', or `.au' files. For each such file, it calls \mcode{extract_objects_fun} to determine its constituent objects and their classes. To add a new database, it suffices to write the corresponding \mcode{extract_objects_fun} function.

The \mcode{extract_objects_fun} function has the following signature
\begin{lstlisting}
[objects, classes] = extract_objects_fun(file);
\end{lstlisting}
Given a file, it returns a structure array \mcode{objects} of the objects it contains and a cell array of their class names \mcode{classes}.

Most often, a file will only contain one object, so \mcode{objects} is a  structure of size one, but this can vary from database to database (for example in speech recognition a file corresponds to a recording of a phrase, which can contain different phones to be classified).

The objects structure has the fields \mcode{objects.u1} and \mcode{objects.u2} which correspond to the lower-left and upper-right corners (start- and endpoints) of the object in a 2D (or 1D) signal. These bounds are inclusive. For the case where a file only contains one object, the bounds will simply be the bounds of the signal.

Given a directory and this function, \mcode{create_src} returns a source structure \mcode{src}, which has the following fields
\begin{itemize}
	\item \mcode{src.classes}: A cell array of the class names.
	\item \mcode{src.files}: A cell array of the file names.
	\item \mcode{src.objects}: A struct array of the objects returned by applying \mcode{objects_fun} to each filename, but with two supplementary fields: \mcode{src.objects.ind}, corresponding to the index of its parent filename in \mcode{src.files} and \mcode{src.objects.class}, corresponding to the index of its class in \mcode{src.classes}.
\end{itemize}

The scattering coefficients of the objects in \mcode{src} can now be calculated using the function \mcode{prepare_database}. This function takes for input the \mcode{src} structure, a cell array of function handles, \mcode{feature_fun}, which contain the functions calculating the feature representations, and an options structure.

A ``feature function'' takes as an input the file data and a structure array of its constituent objects and returns the corresponding feature vectors as a 2D matrix. Its signature is thus
\begin{lstlisting}
features = feature_fun(signal, objects);
\end{lstlisting}
Since this flexibility is often not necessary, a simplified feature function is also allowed, which only takes inputs \mcode{obj_signal} in the form of a 2D matrix, each column corresponding to a signal to be transformed, and returns the associated feature vectors \mcode{obj_features}. It has the following signature
\begin{lstlisting}
obj_features = feature_fun(obj_signal);
\end{lstlisting}
In this case, the function \mcode{feature_wrapper} is called, which extracts the objects from the file data \mcode{signal}, applies the feature function to each object, and collects the output features.

Note that a feature function is not required to output only one feature vector for each object. If more than one feature vector is associated to a single object, the classifier will classify each feature vector separately and aggregate the results, either through averaging the approximation error (for the affine space classifier) or through voting (for the SVM classifier).

The following code sets up a feature function array for the normalized log-scattering transform:
\begin{lstlisting}
fparam.filter_type = {'gabor_1d', 'morlet_1d'};
fparam.Q = [8 1];
fparam.J = T_to_J(8192,fparam.Q);

options = struct();

cascade = wavelet_factory_1d(N, fparam, options, 2);

feature_fun = {@(x)(format_scat( ...
	log_scat(renorm_scat(scat(x, cascade)))))};
\end{lstlisting}

The feature vectors are then computed by calling \mcode{prepare_database}, as in:
\begin{lstlisting}
database = prepare_database(src, feature_fun);
\end{lstlisting}
The resulting \mcode{database} structure then contains three fields:
\begin{itemize}
	\item \mcode{database.src}: The original \mcode{src} structure from which the features were computed.
	\item \mcode{database.features}: The feature vectors, arranged in a 2D matrix, each feature vector forming a column.
	\item \mcode{database.indices}: The indices corresponding to each object in \mcode{database.src}. Specifically, \mcode{database.features(:,database.indices\{k\})} contains the features vector(s) calculated from \mcode{database.src.objects(k)}.
\end{itemize}

\subsection{Training/Testing}

Once the database of feature vectors is constructed, models can be trained and tested. To create a train/test partition, the function \mcode{create_partition} is available. Given a source \mcode{src} and a ratio, it partitions the objects so that each class is divided evenly into the training and testing set according to the ratio. For example, the following code defines a train/test partition with $80\%$ of the objects in the training set:
\begin{lstlisting}
[train_set, test_set] = create_partition(src, 0.8);
\end{lstlisting}
Here, \mcode{train_set} will contain the indices in \mcode{src.objects} that correspond to the training instances, while \mcode{test_set} will contain those corresponding to testing instances.

Alternatively, a partition can be defined manually by traversing the \mcode{src.objects} array and recording the desired indices.

Given a partition, a model can be trained using the appropriate training function. We will denote the training function by \mcode{train}, but in reality it will be either \mcode{affine_train} or \mcode{svm_train}. The training is then done by:
\begin{lstlisting}
model = train(database, train_set, train_opt);
\end{lstlisting}
The structure \mcode{train_opt} contains various parameters for training the model, which will depend on the type of classifier used.

To test the model, the functions \mcode{affine_test} and \mcode{svm_test} are used. Here, we denote the testing function by \mcode{test}. The labels obtained for the testing instances specified by \mcode{test_set} are then obtained by:
\begin{lstlisting}
labels = test(database, model, test_set);
\end{lstlisting}
To calculate the classification error, the \mcode{classif_err} is used:
\begin{lstlisting}
err = classif_err(labels, test_set, src);
\end{lstlisting}
Note that the original \mcode{src} structure is necessary here to verify the class membership of the testing instances.

\subsection{Affine Classifier}

The affine classifier is defined by the functions \mcode{affine_train} and \mcode{affine_test}. The former takes as an option the number of dimensions, \mcode{train_opt.dim} used for the affine space model of each class.

Multiple dimensions can be specified in order to test the difference in performance for different dimensions. In this case, the labels returned by \mcode{affine_test} form a 2D matrix, with the first dimension corresponding to the dimension and the second dimension corresponding to the testing instance index.

The following code calculates the minimum error for an affine space classifier of dimension between $0$ and $160$:
\begin{lstlisting}
train_opt.dim = 0:160;
model = affine_train(database, train_set, train_opt);
labels = affine_test(database, model, test_set);
err = classif_err(labels, test_set, src);
[min_err, dim_ind] = min(err);
min_dim = train_opt.dim(dim_ind);
\end{lstlisting}

For an affine space classifier, the model structure contains the dimensions for which it is defined \mcode{model.dim}, the centers of the classes \mcode{model.mu} and the direction vectors defining the affine space \mcode{model.v}.

\subsection{Support Vector Machine}

\textbf{NOTE: Requires the LIBSVM library, see \url{http://www.csie.ntu.edu.tw/~cjlin/libsvm/}}

The support vector machine classifier is defined by the functions \mcode{svm_train} and \mcode{svm_test}. The training options consist of:
\begin{itemize}
	\item \mcode{train_opt.kernel_type}: The kernel type used in the SVM. Can be \mcode{'linear'} for a linear kernel $u^Tv$ or \mcode{'gaussian'} for a Gaussian kernel $e^{-\gamma\|u-v\|^2}$.
	\item \mcode{train_opt.C}: The slack factor $C$ used for training the SVM.
	\item \mcode{train_opt.gamma}: The regularity constant $\gamma$ used in the case of a Gaussian SVM kernel.
\end{itemize}

The following code calculates the error for an SVM classifier with a linear kernel and a slack factor $C = 8$:
\begin{lstlisting}
train_opt.kernel_type = 'linear';
train_opt.C = 8;
model = svm_train(database, train_set, train_opt);
labels = svm_test(database, model, test_set);
err = classif_err(labels, test_set, src);
\end{lstlisting}

If a linear kernel is used, we can extract the discriminant vector $w$ and bias $\rho$ from the model using the function \mcode{svm_extract_w}. The function takes as input the database \mcode{database} and the SVM model \mcode{model}. It outputs the $w$s corresponding to each pair of classes in the model, arranged in the order $1$vs$2$,$1$vs$3$,\ldots$1$vs$N$,$2$vs$3$,\ldots,$2$vs$N$,\ldots,$(N-1)$vs$N$, where $N$ is the number of classes. The function is called as in:
\begin{lstlisting}
[w,rho] = svm_extract_w(database, model);
\end{lstlisting}

\end{document}


%%% lecture 02 %%%
\documentclass{beamer}
\usepackage[utf8]{inputenc}
\usepackage{algorithm2e, amsmath, amssymb, amsfonts, graphicx}
% allow section.equation numbering
\numberwithin{equation}{section}
% use boadilla theme
\usetheme{Boadilla}
% remove navigation symbols
\usenavigationsymbolstemplate{}
% get numbered figure captions
\setbeamertemplate{caption}[numbered]
% changes itemize to circle + other things
\useoutertheme{split}
\useinnertheme{circles}

% title page stuff. brackets content displayed in footer bar
\title[Math for Data Science, Part 2]{Math for Data Science, Part 2}
% metadata. content in brackets is displayed in footer bar
\author[Derek Huang (BAC Advanced Team)]{Derek Huang}
\institute{BAC Advanced Team}
\date{May 20, 2021}

% change "ball" bullet to numbered bullet and section title for section
\setbeamertemplate{section in toc}{\inserttocsectionnumber.~\inserttocsection}
% change ball to gray square (copied from stackoverflow; \par needed for break)
\setbeamertemplate{subsection in toc}{
    \hspace{1.2em}{\color{gray}\rule[0.3ex]{3pt}{3pt}}~\inserttocsubsection\par}
% use default enumeration scheme
\setbeamertemplate{enumerate items}[default]
% required line that fixes the problem of \mathbf, \bf not working in beamer
% for later (post-2019) TeX Live installations. see the issue on GitHub:
% https://github.com/josephwright/beamer/issues/630
\DeclareFontShape{OT1}{cmss}{b}{n}{<->ssub * cmss/bx/n}{}

\begin{document}

% title slide
\begin{frame}
    \titlepage
    \centering
    \includegraphics[scale = 0.1]{../bac_logo1.png}
\end{frame}

% table of contents slide
\begin{frame}{Overview}
    \tableofcontents
\end{frame}

\section{Linear algebra}

\subsection{Matrix essentials}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        \textit{Definition.} A \textit{matrix} $ \mathbf{A} $ of shape
        $ m \times n $ over a field\footnotemark\footnotetext{
            It turns out that it doesn't matter if $ \mathbb{F} = \mathbb{R} $
            or $ \mathbb{F} = \mathbb{C} $, $ \mathbb{C} $ the complex numbers
            \cite{jacob_linalg}.
        } $ \mathbb{F} $ is a collection of indexed elements of $ \mathbb{F} $
        arranged in a rectangular array. The $ i, j$th element of
        $ \mathbf{A} $ is denoted as $ A_{ij} $ or $ a_{ij} $.
        \begin{equation*}
            \mathbf{A} = \begin{bmatrix}
                \ a_{11} & \ldots & a_{1n} \ \\
                \ \vdots & \ddots & \vdots \ \\
                \ a_{m1} & \ldots & a_{mn} \
            \end{bmatrix}
        \end{equation*}

        \item
        \textit{Definition.} A \textit{column vector} $ \mathbf{a} $ with
        $ n $ elements is a $ n \times 1 $ matrix over a field $ \mathbb{F} $
        with elements denoted as $ a_1, \ldots a_n $.
        \begin{equation*}
            \mathbf{a} = \begin{bmatrix}
                \ a_1 \ \\ \ \vdots \ \\ \ a_n \
            \end{bmatrix}
        \end{equation*}
    \end{itemize}
\end{frame}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        \textit{Definition.} The $ n \times n $ \textit{identity matrix}
        $ \mathbf{I} $ is such that
        \begin{equation*}
            \mathbf{I} \triangleq \begin{bmatrix}
                \ 1 & \ldots & 0 \ \\
                \ \vdots & \ddots & \vdots \ \\
                \ 0 & \ldots & 1 \
            \end{bmatrix}
        \end{equation*}
        If context is not clear, we write $ n \times n $ identity as
        $ \mathbf{I}_n $.

	    \item
        \textit{Definition.} The \textit{transpose} of a matrix
        $ \mathbf{A} \in \mathbb{R}^{m \times n } $ is $ \mathbf{A}^\top
        \in \mathbb{R}^{n \times m} $, where the $ i, j $th element of
        $ \mathbf{A}^\top $ is the $ j, i $th element of $ \mathbf{A} $.
        \begin{equation*}
            \mathbf{A}^\top \triangleq \begin{bmatrix}
                \ a_{11} & \ldots & a_{m1} \ \\
                \ \vdots & \ddots & \vdots \ \\
                \ a_{1n} & \ldots & a_{mn} \
            \end{bmatrix}
        \end{equation*}

        \item
        \textit{Theorem.} For $ \mathbf{A} \in \mathbb{F}^{m \times p} $,
        $ \mathbf{B} \in \mathbb{F}^{p \times n} $,
        $ (\mathbf{AB})^\top = \mathbf{B}^\top\mathbf{A}^\top $
        \cite{jacob_linalg}.
    \end{itemize}
\end{frame}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        \textit{Definition.} The \textit{sum} of matrices
        $ \mathbf{A}, \mathbf{B} \in \mathbb{F}^{m \times n }$ is such that
        \begin{equation*}
            \mathbf{A} + \mathbf{B} = \begin{bmatrix}
                \ a_{11} + b_{11} & \ldots & a_{1n} + b_{1n} \ \\
                \ \vdots & \ddots & \vdots \ \\
                \ a_{m1} + b_{m1} & \ldots & a_{mn} + b_{mn} \
            \end{bmatrix}
        \end{equation*}

        \item
        \textit{Properties of matrix addition} \cite{jacob_linalg}. For
        $ \mathbf{A}, \mathbf{B}, \mathbf{C} \in \mathbb{F}^{m \times n} $,
        $ a, b \in \mathbb{F} $
        \begin{enumerate}
            \item
            $ \mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A} $
            (commutative).

            \item
            $ (\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} +
            (\mathbf{B} + \mathbf{C}) $ (associative).

            \item
            $ a(\mathbf{A} + \mathbf{B}) = a\mathbf{A} + a\mathbf{B} $ (scalar
            distribution).

            \item
            $ (a + b)\mathbf{A} = a\mathbf{A} + b\mathbf{A} $ (scalar
            distribution).

            \item
            $ a(b\mathbf{A}) = (ab)\mathbf{A} $ (scalar associativity).
        \end{enumerate}
    \end{itemize}
\end{frame}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        \textit{Definition.} The \textit{product} of $ \mathbf{A} \in
        \mathbb{F}^{m \times d} $, $ \mathbf{B} \in \mathbb{F}^{d \times n} $
        is such that
        \begin{equation*}
            \mathbf{AB} = \begin{bmatrix}
                \ \sum_{k = 1}^da_{1k}b_{k1} & \ldots &
                    \sum_{k = 1}^da_{1k}b_{kn} \ \\
                \ \vdots & \ddots & \vdots \ \\
                \ \sum_{k = 1}^da_{mk}b_{k1} & \ldots &
                    \sum_{k = 1}^da_{mk}b_{kn} \
            \end{bmatrix} \in \mathbb{F}^{m \times n}
        \end{equation*}

        \item
	    \textit{Properties of matrix multiplication} \cite{jacob_linalg}. For
	    matrices $ \mathbf{A}, \mathbf{B} \in \mathbb{F}^{m \times d} $,
	    $ \mathbf{C}, \mathbf{D} \in \mathbb{F}^{d \times n} $, and a scalar
	    $ a \in \mathbb{F} $,
	    \begin{enumerate}
	        \item
            $ a(\mathbf{AC}) = (a\mathbf{A})\mathbf{C} $ (scalar
            associativity).

            \item
            $ a(\mathbf{AC}) = \mathbf{A}(a\mathbf{C}) $ (scalar
            commutativity + associativity).

            \item
            $ (\mathbf{A} + \mathbf{B})\mathbf{C} =
            \mathbf{AC} + \mathbf{BC} $ (matrix distribution).

            \item
            $ \mathbf{AI}_d = \mathbf{A} $, $ \mathbf{I}_d\mathbf{C} =
            \mathbf{C} $ (identity).
	    \end{enumerate}

        \item
        \textit{Theorem.} For $ \mathbf{A} \in \mathbb{F}^{m \times p} $,
        $ \mathbf{B} \in \mathbb{F}^{p \times q} $, $ \mathbf{C} \in
        \mathbb{F}^{q \times n} $, $ (\mathbf{AB})\mathbf{C} =
        \mathbf{A}(\mathbf{BC}) $ \cite{jacob_linalg}.
    \end{itemize}
\end{frame}

%\begin{frame}{Matrix essentials}
%    \begin{itemize}
%        \item
%        \textit{Definition.} Given matrices $ \mathbf{A}, \mathbf{B} \in
%        \mathbb{F}^{m \times n} $, their \textit{Hadamard product} is
%        \begin{equation*}
%            \mathbf{A} \odot \mathbf{B} \triangleq \begin{bmatrix}
%                \ a_{11}b_{11} & \ldots & a_{1n}b_{1n} \ \\
%                \ \vdots & \ddots & \vdots \ \\
%                \ a_{m1}b_{m1} & \ldots & a_{mn}b_{mn} \
%            \end{bmatrix}
%        \end{equation*}
%
%        \item
%        \textit{Definition.} Given matrices $ \mathbf{A}, \mathbf{B} \in
%        \mathbb{F}^{m \times n} $, \textit{Hadamard division} is
%        \begin{equation*}
%            \mathbf{A} \oslash \mathbf{B} \triangleq \begin{bmatrix}
%                \ a_{11} / b_{11} & \ldots & a_{1n} / b_{1n} \ \\
%                \ \vdots & \ddots & \vdots \ \\
%                \ a_{m1} / b_{m1} & \ldots & a_{mn} / b_{mn} \
%            \end{bmatrix}
%        \end{equation*}
%
%        \item
%        \textit{Definition.} Given matrix $ \mathbf{A} \in
%        \mathbb{F}^{m \times n} $, $ b \in \mathbb{R} $, the
%        \textit{Hadamard power} is
%        \begin{equation*}
%            \mathbf{A}^{\odot b} \triangleq \begin{bmatrix}
%                \ a_{11}^b & \ldots & a_{1n}^b \ \\
%                \ \vdots & \ddots & \vdots \ \\
%                \ a_{m1}^n & \ldots & a_{mn}^b \
%            \end{bmatrix}
%        \end{equation*}
%    \end{itemize}
%\end{frame}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        \textit{Definition.} $ \mathbf{A} \in \mathbb{F}^{m \times n} $
        is \textit{diagonal} if $ a_{11}, \ldots a_{qq}  \in \mathbb{F} $,
        $ q \triangleq \min\{m, n\} $ are the only nonzero elements in
        $ \mathbf{A} $. If $ \mathbf{A} \in \mathbb{F}^{n \times n} $, i.e.
        square, for $ a_1, \ldots a_n \in \mathbb{F} $, $ \mathbf{A} $ can
        be written as $ \operatorname{diag}(a_1, \ldots a_n) $.

        \item
        \textit{Definition.} If $ \mathbf{A} \in \mathbb{F}^{n \times n} $ is
        \textit{invertible}, $ \exists! \mathbf{A}^{-1} \in
        \mathbb{F}^{n \times n} $ such that
        \begin{equation*}
            \mathbf{A}^{-1}\mathbf{A} = \mathbf{AA}^{-1} = \mathbf{I}
        \end{equation*}
        The $ \exists! $ notation means ``uniquely exists''.

        \item
        \textit{
            Some properties of invertible matrices\footnotemark\footnotetext{
                We'll discuss rank later in the slides. Don't worry about
                eigenvalues for now.
            }.
        } If $ \mathbf{A} \in
        \mathbb{F}^{n \times n} $ invertible,
        \begin{itemize}
            \item
            $ |\mathbf{A}| \ne 0 $ (nonzero determinant).

            \item
            $ \forall \mathbf{b} \in \mathbb{F}^n $, $ \exists! \mathbf{x} \in
            \mathbb{F}^n $ s.t. $ \mathbf{Ax} = \mathbf{b} $ (unique solution
            to linear system).

            \item
            $ \operatorname{rank}(\mathbf{A}) = n $ (full rank).

            \item
            All eigenvalues of $ \mathbf{A} $ are nonzero.
        \end{itemize}

        \item
        \textit{Theorem.} For invertible $ \mathbf{A}, \mathbf{B} \in
        \mathbb{F}^{n \times n} $, $ (\mathbf{AB})^{-1} =
        \mathbf{B}^{-1}\mathbf{A}^{-1} $ \cite{jacob_linalg}.

        \item
        \textit{Theorem.} For $ \mathbf{A} \triangleq
        \operatorname{diag}(a_1, \ldots a_n) $, $ \mathbf{A}^{-1} \triangleq
        \operatorname{diag}\left(a_1^{-1}, \ldots a_n^{-1}\right) $.
    \end{itemize}

    % skip for footnote spacing
    \medskip
\end{frame}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        \textit{Definition.} $ \mathbf{A} \in \mathbb{R}^{n \times n} $ is
        \textit{symmetric} if $ \mathbf{A} = \mathbf{A}^\top $.

        \item
        \textit{Definition.} Symmetric $ \mathbf{A} \in
        \mathbb{R}^{n \times n} $ is \textit{positive definite} if
        $ \forall \mathbf{x} \in \mathbb{R}^n $, $ \mathbf{x}^\top
        \mathbf{Ax} > 0 $. $ \mathbf{A} $ is \textit{positive semidefinite}
        if $ \forall \mathbf{x} \in \mathbb{R}^n,
        \mathbf{x}^\top\mathbf{Ax} \ge 0 $. $ \mathbf{A} \succ \mathbf{0} $
        means $ \mathbf{A} $ positive definite, while $ \mathbf{A}
        \succeq \mathbf{0} $ means $ \mathbf{A} $ positive semidefinite.

        \item
        \textit{Some properties of positive definite matrices.} For
        $ \mathbf{A} \succ \mathbf{0} $,
        \begin{enumerate}
            \item
            $ \exists \mathbf{A}^{-1} \succ \mathbf{0} $ (positive definite
            inverse).

            \item
            All eigenvalues are real and positive.

            \item
            $ |\mathbf{A}| > 0 $ (positive determinant).
        \end{enumerate}

        \item
        \textit{Remark.} For $ \mathbf{A} \in \mathbb{R}^{n \times n} $,
        $ \mathbf{A} \succeq \mathbf{0} $ does \textbf{not} mean that all
        elements of $ \mathbf{A} $ are nonnegative. For example,
        $ \mathbf{Q} \triangleq \mathbf{I}_n - \frac{1}{n^2}\mathbf{11}^\top
        \succ \mathbf{0} $, as $ |\mathbf{Q}| > 0 $.

        \item
        \textit{Some properties of positive semidefinite matrices.} For
        $ \mathbf{A} \succeq \mathbf{0} $,
        \begin{enumerate}
            \item
            $ \exists \mathbf{A}^{-1} \succ \mathbf{0} \Leftrightarrow $ all
            eigenvalues positive.

            \item
            All eigenvalues are real and nonnegative.

            \item
            $ |\mathbf{A}| \ge 0 $ (nonnegative determinant).
        \end{enumerate}
    \end{itemize}
\end{frame}

\begin{frame}{Matrix essentials}
    \begin{itemize}
        \item
        Matrices and vectors are very useful for applied modeling.

        \item
        \textit{Examples.}
        \begin{itemize}
            \item
            \textit{Representing data.} Given data points $ \mathbf{x}_1,
            \ldots \mathbf{x}_N \in \mathbb{R}^d $, we can define the matrix
            $ \mathbf{X} \in \mathbb{R}^{N \times d} $ where each $ k $th row
            is $ \mathbf{x}_k^\top $.

            \item
            \textit{Linear dynamical system.} Suppose $ \mathbf{x}_k \in
            \mathbb{R}^n $ represents the state of a system at time step
            $ k \in \mathbb{N} $. Suppose system state evolution depends only
            on fixed linear transformation of $ \mathbf{x}_k $ and
            independent noise. Then,
            \begin{equation*}
                \mathbf{x}_{k + 1} = \mathbf{Qx}_k + \mathbf{z}_k
            \end{equation*}
            Here $ \mathbf{Q} \in \mathbb{R}^{n \times n} $, $ \mathbf{z}_k $
            i.i.d. where $ \mathbb{E}[\mathbf{z}_k] = \mathbf{0} $,
            $ \mathbf{C}_\mathbf{z} = \operatorname{diag}(\sigma_1, \ldots
            \sigma_n) \succ \mathbf{0} $.

            \item
            \textit{Multivariate quadratic function.} Given symmetric
            $ \mathbf{Q} \in \mathbb{R}^{n \times n} $, $ \mathbf{a} \in
            \mathbb{R}^n $, $ b \in \mathbb{R} $, a quadratic function
            $ f : \mathbb{R}^n \rightarrow \mathbb{R} $ can be represented
            such that
            \begin{equation*}
                f(\mathbf{x}) \triangleq \mathbf{x}^\top\mathbf{Qx} +
                \mathbf{a}^\top\mathbf{x} + b
            \end{equation*}
            $ \mathbf{Q} $ gives the quadratic coefficients, $ \mathbf{a} $
            the linear coefficients.
        \end{itemize}
    \end{itemize}
\end{frame}

\subsection{Vector spaces}

\begin{frame}{Vector spaces}
    \begin{itemize}
        \item
        \textit{Definition.} A \textit{field} $ \mathbb{F} $ is a set equipped
        with addition and multiplication with elements $ 0, 1 \in \mathbb{F} $
        s.t. $ \forall x, y, z \in \mathbb{F} $ \cite{jacob_linalg},
        \begin{enumerate}
            \item
            $ x + y \in \mathbb{F} $ (closure under addition).

            \item
            $ x + y = y + x $ (commutativity of addition).

            \item
            $ (x + y) + z = x + (y + z) $ (associativity of addition).

            \item
            $ 0 + x = x $ (contains additive identity).

            \item
            $ xy \in \mathbb{F} $ (closure under multiplication).

            \item
            $ xy = yx $ (commutativity of multiplication).

            \item
            $ (xy)z = x(yz) $ (associativity of multiplication).

            \item
            $ 1x = x $ (contains multiplicative identity).

            \item
            $ x(y + z) = xy + xz $ (distribution of multiplication).

            \item
            $ \exists!x' \in \mathbb{F} $ s.t. $ x + x' = 0 $ (unique additive
            inverses).

            \item
            If $ x \ne 0 $, $ \exists!\tilde{x} \in \mathbb{F} $ s.t.
            $ x\tilde{x} = 1 $ (unique multiplicative inverses).
        \end{enumerate}

        \item
        \textit{Remark.} We will use $ \mathbb{F} $ to denote arbitrary fields.

        \item
        \textit{Examples.} $ \mathbb{Q} $, $ \mathbb{R} $, $ \mathbb{C} $. We
        focus mostly on $ \mathbb{R} $.
    \end{itemize}
\end{frame}

\begin{frame}{Vector spaces}
    \begin{itemize}
        \item
        \textit{Definition.} A \textit{vector space} $ V $ over a field
        $ \mathbb{F} $ is a set of \textit{vectors} containing the
        \textit{zero vector} $ 0 \in V $ equipped with \textit{vector
        addition} and \textit{scalar multiplication} of $ v \in V $ by
        $ x \in \mathbb{F} $. $ \forall u, v, w \in V $, $ \forall \alpha,
        \beta \in \mathbb{F} $ \cite{jacob_linalg},
        \begin{enumerate}
            \item
            $ u + v \in V $ (closure under vector addition).

            \item
            $ u + v = v + u $ (commutativity of vector addition).

            \item
            $ 0 + u = u $ (contains vector addition identity).

            \item
            $ (u + v) + w = u + (v + w) $ (associativity of vector addition).

            \item
            $ \alpha u \in V $ (closure under scalar multiplication).

            \item
            $ 1u = u $, $ 1 \in \mathbb{F} $ ($ \mathbb{F} $ has scalar
            multiplicative identity).

            \item
            $ \alpha(u + v) = \alpha u + \alpha v $ (distribution of scalar
            multiplication).

            \item
            $ (\alpha + \beta)u  = \alpha u + \beta u $ (distribution of
            scalar multiplication).

            \item
            $ (\alpha\beta)u = u(\alpha\beta) $ (commutativity of scalar
            multiplication).
        \end{enumerate}

        \item
        \textit{Remark.} These properties should be natural; no need to
        memorize.

        \item
        \textit{Example.} $ \mathbb{R}^n $, i.e. $ n $\textit{-dimensional
        Euclidean space}. For any field $ \mathbb{F} $, $ \mathbb{F}^n $ is
        the $ n $\textit{-fold Cartesian product}, for example
        $ \mathbb{C}^n $.

        \item
        \textit{Remark.} We will denote elements of $ \mathbb{R}^n $ in bold.
    \end{itemize}
\end{frame}

\begin{frame}{Vector spaces}
    \begin{itemize}
        \item
        \textit{Definition.} For $ v_1, \ldots v_n \in V $, $ V $ a vector
        space over $ \mathbb{F} $, and $ a_1, \ldots a_n \in \mathbb{F} $,
        $ w = \sum_{k = 1}^na_kv_k $ is a \textit{linear combination}.

        \item
        \textit{Definition.} The \textit{span} of $ v_1, \ldots v_n \in V $,
        denoted $ \operatorname{span}\{v_1, \ldots v_n\} $, is the set of all
        linear combinations of $ v_1, \ldots v_n $ \cite{jacob_linalg}.

        \item
        \textit{Definition.} $ v_1, \ldots v_n \in V $ are \textit{linearly
        independent} if $ \sum_{k = 1}^na_kv_k = 0 \in V \Leftrightarrow
        a_1 = \ldots a_n = 0 \in \mathbb{F} $, else \textit{linearly
        dependent}.

        \item
        \textit{Definition.} A set $ \{v_1, \ldots v_n\} \subset V $ is called
        a \textit{basis} for $ V $ if \cite{jacob_linalg}
        \begin{enumerate}
            \item
            $ v_1, \ldots v_n $ are linearly independent.

            \item
            $ V = \operatorname{span}\{v_1, \ldots v_n\} $.
        \end{enumerate}

        \item
        \textit{Remark.} Although unordered, bases are often treated as
        ordered.

        \item
        \textit{Examples.}
        \begin{itemize}
            \item
            \textit{The standard Euclidean basis.} The standard basis for
            $ \mathbb{R}^n $ is $ \{\mathbf{e}_1, \ldots \mathbf{e}_n\} $,
            where the $ k $th element of $ \mathbf{e}_k $ is $ 1 $ and all
            other elements are zero.

            \item
            \textit{Non-standard} $ \mathbb{R}^2 $ \textit{basis.}
            $ \left\{[ \ -1 \ \ 1 \ ]^\top, [ \ 2 \ \ 0 \ ]^\top\right\} $ is
            a basis for $ \mathbb{R}^2 $.
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Vector spaces}
    \begin{itemize}
        \item
        \textit{Lemma.} If $ \{v_1, \ldots v_n\} \subset V $,
        $ \{w_1, \ldots w_m\} \subset V $ are both finite bases of a vector
        space $ V $, then $ m = n $ \cite{jacob_linalg}.

        \item
        \textit{Definition.} If vector space $ V $ has a basis $ B \subset V $
        where $ |B| = n $, $ n \in \mathbb{N} $, then $ V $ is said to be
        $ n $\textit{-dimensional}.

        \item
        \textit{Remark.} $ \mathbb{F}^n $ is often referred to as
        $ n $-dimensional, but in general, for a vector space $ V $ on field
        $ \mathbb{F} $, dimension $ \ne $ number of elements of $ \mathbb{F} $
        in a vector $ v \in V $. A vector \textit{subspace} $ U \subset
        \mathbb{R}^n $ that has $ k < n $ elements in its basis has dimension
        $ k $ although $ \mathbf{u} \in U $ contains $ n $ elements of
        $ \mathbb{R} $.

        \item
        \textit{Definition.} Given ordered basis $ B \subset \mathbb{F}^n $,
        denote $ \mathbf{B} \in \mathbb{F}^{n \times n} $ as the matrix whose
        columns hold the elements of $ B $. For $ \mathbf{x} \in
        \mathbb{F}^n $, its \textit{coordinates} in $ B $, denoted by
        $ (\mathbf{x})_B $ are such that
        $ \mathbf{x} = \mathbf{B}(\mathbf{x})_B $.

        \item
        \textit{Theorem.} $ \mathbf{B} $ is invertible and
        $ \forall \mathbf{u} \in \mathbb{F}^n $,
        $ (\mathbf{u})_B = \mathbf{B}^{-1}\mathbf{u} $ \cite{jacob_linalg}.

        \item
        \textit{Remark.} Bases and coordinates are important in the context of
        principal components analysis, which performs a \textit{change of
        basis}.
    \end{itemize}
\end{frame}

\begin{frame}{Vector spaces}
    \begin{itemize}
        \item
        \textit{Definition.} The \textit{rank} of $ \mathbf{A} \in
        \mathbb{F}^{m \times n} $ is the maximal number of linearly
        independent columns/rows of $ \mathbf{A} $, denoted as
        $ \operatorname{rank}(\mathbf{A}) $.

        \item
        \textit{Definition.} $ \mathbf{A} \in \mathbb{R}^{m \times n} $ is
        \textit{full rank} if
        $ \operatorname{rank}(\mathbf{A}) = \min\{m, n\} $.

        \item
        Rank can be thought of as a matrix's ``information content''.

        \item
        \textit{Examples.}
        \begin{itemize}
            \item
            \textit{Feature redundancy.} Suppose $ \mathbf{X} \in
            \mathbb{R}^{N \times d} $, each $ k $th row a transposed data
            example $ \mathbf{x}_k^\top $. Assuming $ N > d $, if
            $ \operatorname{rank}(\mathbf{X}) < d $, at least one column is
            linearly dependent and can be safely removed.

            \item
            \textit{Square linear system.} Suppose $ \mathbf{A} \in
            \mathbb{R}^{n \times n} $, $ \mathbf{b} \in \mathbb{R}^n $. We
            want to find $ \mathbf{x} \in \mathbb{R}^n $ s.t. $ \mathbf{Ax} =
            \mathbf{b} $. If $ \operatorname{rank}(\mathbf{A}) = n $, each row
            (linear equation) helps to determine one of the $ n $ unknown
            elements of $ \mathbf{x} $. If
            $ \operatorname{rank}(\mathbf{A}) < n $, there isn't enough
            information $ \Rightarrow  $ no solution or infinitely many
            solutions.
        \end{itemize}
    \end{itemize}
\end{frame}

\subsection{Norms and inner products}

\begin{frame}{Norms and inner products}
    \begin{itemize}
        \item
        \textit{Definition.} A \textit{norm} on a vector space $ V $ over
        $ \mathbb{F} \in \{\mathbb{R}, \mathbb{C}\} $ is a function
        $ \Vert\cdot\Vert : V \rightarrow \mathbb{R} $ that satisfies the
        following properties \cite{jacob_linalg}:
        \begin{enumerate}
            \item
            $ \forall v \in V $, $ \Vert v\Vert \ge 0 $, where
            $ \Vert v\Vert = 0 \Leftrightarrow v = 0 $.

            \item
            $ \forall k \in \mathbb{F} $, $ \forall v \in V $,
            $ \Vert kv\Vert = |k|\Vert v\Vert $ (absolute homogeneity).

            \item
            $ \forall u, v \in V $, $ \Vert u + v\Vert \le
            \Vert u\Vert + \Vert v\Vert $ (triangle inequality).
        \end{enumerate}

        \item
        A norm gives a sense of the ``size'' of a vector in a vector space.

        \item
        \textit{Definition.} $ \Vert\mathbf{x}\Vert_p $, the finite
        $ \ell^p $\textit{-norm}, or $ p $\textit{-norm}, of
        $ \mathbf{x} \in \mathbb{R}^n $ is s.t.
        \begin{equation*}
            \Vert\mathbf{x}\Vert_p \triangleq
            \left(\sum_{k = 1}^n|x_k|^p\right)^{1 / p}
        \end{equation*}
        When $ p = 2 $, $ \Vert\cdot\Vert_2 $ called the \textit{Euclidean},
        or $ \ell^2 $ norm. When $ p = 1 $, $ \Vert\cdot\Vert_1 $ called the
        $ \ell^1 $, \textit{Manhattan}, or \textit{taxicab} norm. When
        $ p = \infty $, $ \Vert\mathbf{x}\Vert_\infty \triangleq
        \max\{|x_1|, \ldots |x_n|\} $, and $ \Vert\cdot\Vert_\infty $ called
        the \textit{max norm}.
    \end{itemize}
\end{frame}

\begin{frame}{Norms and inner products}
    \begin{itemize}
        \item
        \textit{Definition.} An \textit{inner product} on a vector space $ V $
        over $ \mathbb{R} $ is a function $ \langle\cdot, \cdot\rangle :
        V \times V \rightarrow \mathbb{R} $ that satisfies the following
        properties \cite{jacob_linalg}:
        \begin{enumerate}
            \item
            $ \forall u, v \in V $, $ \langle u, v\rangle =
            \langle v, u\rangle $ (symmetric).

            \item
            $ \forall k \in \mathbb{R} $, $ \forall u, v \in V $,
            $ \langle ku, v\rangle = k\langle u, v\rangle $ (bilinear).

            \item
            $ \forall u, v, w \in V $, $ \langle u + v, w\rangle =
            \langle u, w\rangle + \langle v, w\rangle $ (bilinear).

            \item
            $ \forall v \in V $, $ \langle v, v\rangle \ge 0 $, where
            $ \langle v, v\rangle = 0 \Leftrightarrow v = 0 $.
        \end{enumerate}

        \item
        \textit{Definition.} A vector space $ V $ over $ \mathbb{R} $ equipped
        with an inner product $ \langle\cdot, \cdot\rangle $ is called a
        \textit{real inner product space} \cite{jacob_linalg}.

        \item
        \textit{Examples.}
        \begin{itemize}
            \item
            $ \mathbb{R}^n $. $ \mathbb{R}^n $ equipped with the dot product
            $ \cdot $, where $ \forall \mathbf{u}, \mathbf{v} \in
            \mathbb{R}^n $, $ \mathbf{u}\cdot\mathbf{v} =
            \mathbf{u}^\top\mathbf{v} $, is a finite-dimensional inner product
            space.

            \item
            $ L^2(\mathbb{R}) $. The $ L^2 $ function space on $ \mathbb{R} $,
            i.e. the vector space of all functions $ f : \mathbb{R}
            \rightarrow \mathbb{R} $ s.t.
            $ \int_\mathbb{R}f(x)^2\,dx < \infty $ where $ \forall f, g
            \in L^2(\mathbb{R}) $,
            \begin{equation*}
                \langle f, g\rangle \triangleq \int_\mathbb{R}f(x)g(x)\,dx
            \end{equation*}
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Norms and inner products}
    \begin{itemize}
        \item
        Inner products can be thought of as measuring vector similarity.

        \item
        \textit{Theorem.} If $ \langle\cdot, \cdot\rangle : V \times V
        \rightarrow \mathbb{R} $ is an inner product on real vector $ V $
        space, i.e. a vector space over $ \mathbb{R} $, then
        $ \Vert\cdot\Vert : V \rightarrow \mathbb{R} $ where
        $ \forall v \in V $, $ \Vert v\Vert \triangleq
        \sqrt{\langle v, v\rangle} $, is a norm on $ V $ \cite{jacob_linalg}.
        \begin{itemize}
            \item
            In particular, if $ V = \mathbb{R}^n $, $ \forall \mathbf{u},
            \mathbf{v} \in \mathbb{R}^n $, $ \langle\mathbf{u},
            \mathbf{v}\rangle  = \mathbf{u}^\top\mathbf{v} $,
            $ \Vert\mathbf{u}\Vert_2 = \sqrt{\mathbf{u}^\top\mathbf{u}} $.
        \end{itemize}

        \item
        \textit{Definition.} $ u \in V $ is a \textit{unit vector} if
        $ \Vert u\Vert = 1 $.

        \item
	    \textit{Definition.} $ u, v \in V $ are \textit{orthogonal} if
	    $ \langle u, v\rangle = 0 $, denoted as $ u \perp v $.

        \item
        \textit{Definition.} $ u, v \in V $ are \textit{orthonormal} if
        $ \langle u, v\rangle = 0 $, $ \Vert u\Vert = \Vert v\Vert = 1 $.

        \item
        \textit{Theorem.} $ \forall \mathbf{u}, \mathbf{v} \in \mathbb{R}^n
        \setminus \{\mathbf{0}\} $, $ \mathbf{u}^\top\mathbf{v} =
        \Vert\mathbf{u}\Vert_2\Vert\mathbf{v}\Vert_2\cos\theta $,
        $ \theta \in [0, \pi] $. $ \theta $ is the angle between
        $ \mathbf{u}, \mathbf{v} $, giving the similarity of $ \mathbf{u},
        \mathbf{v} $ in direction.

        \item
        \textit{Corollary.} $ \theta = 0 \Leftrightarrow
        \mathbf{u} = k\mathbf{v} $, $ k \in (0, \infty) $, and $ \mathbf{u} =
        \mathbf{v} \Rightarrow \theta = 0 $.

        \item
        \textit{Corollary.} $ \mathbf{u}^\top\mathbf{v} = 0 \Leftrightarrow
        \theta = \pi / 2 $.
    \end{itemize}
\end{frame}

\begin{frame}{Norms and inner products}
    \begin{itemize}
        \item
        \textit{Definition.} A basis $ B \triangleq \{v_1, \ldots v_n\} \subset
        V $ is called \textit{orthogonal}
        if $ \forall v_i, v_j $, $ i \ne j $, $ i, j \in \{1, \ldots n\} $,
        $ \langle v_i, v_j\rangle = 0 $.

        \item
        \textit{Definition.} An orthogonal basis $ B \triangleq
        \{v_1, \ldots v_n\} \subset V $, where $ \Vert v\Vert = 1 $,
        $ k \in \{1, \ldots n\} $, is called an \textit{orthonormal basis}.

        \item
        \textit{Definition.} $ \mathbf{Q} \in \mathbb{R}^{n \times n} $ is
        \textit{orthogonal}\footnotemark\footnotetext{
            Should really be called orthonormal; this is a misnomer.
        } if $ \mathbf{Q}^\top\mathbf{Q} = \mathbf{QQ}^\top
        = \mathbf{I} $. That is, the rows and columns of $ \mathbf{Q} $ are
        orthonormal.

        \item
        \textit{Definition.} $ \mathbf{Q} \in \mathbb{R}^{m \times n} $ has
        \textit{orthonormal columns} if $ \mathbf{Q}^\top\mathbf{Q} =
        \mathbf{I}_n $ and \textit{orthonormal rows} if $ \mathbf{QQ}^\top =
        \mathbf{I}_m $.

        \item
        \textit{Remark.} Orthogonal matrices have appealing analytical and
        numerical properties. For example, linear ordinary least squares
        problems are often solved in practice using the singular value
        decomposition\footnotemark\footnotetext{
            \href{
https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lstsq.html
            }{\alert{\texttt{scipy.linalg.lstsq}}} wraps LAPACK's
            \href{
            http://www.netlib.org/lapack/explore-html/db/d6a/dgelsd_8f.html
            }{\alert{\texttt{gelsd}}} routine.
        }.
    \end{itemize}
\end{frame}

\section{Vector calculus}

\subsection{Gradient, Jacobian, Hessian}

\begin{frame}{Gradient, Jacobian, Hessian}
    \begin{itemize}
        \item
        \textit{Definition.} For $ f : \mathbb{R}^n \rightarrow \mathbb{R} $,
        the \textit{gradient} $ \nabla f : \mathbb{R}^n \rightarrow
        \mathbb{R}^n $ is such that
        \begin{equation*}
            \nabla f(\mathbf{x}') \triangleq \begin{bmatrix}
                \ \frac{\partial f}{\partial x_1}(\mathbf{x}') \ \\
                \ \vdots \ \\
                \ \frac{\partial f}{\partial x_n}(\mathbf{x}') \
            \end{bmatrix} \triangleq \left[
                \frac{\partial f}{\partial \mathbf{x}}(\mathbf{x}')
            \right]^\top
        \end{equation*}
        Here $ \mathbf{x}' \in \mathbb{R}^n $ and
        $ \frac{\partial f}{\partial \mathbf{x}} : \mathbb{R}^n \rightarrow
        \mathbb{R}^{1 \times n} $ is the \textit{vector
        derivative}\footnotemark\footnotetext{
            The fact that this is a \textbf{row} vector is important in matrix
            calculus.
        }.

        \item
        \textit{Definition.} For $ \mathbf{f} : \mathbb{R}^n \rightarrow
        \mathbb{R}^m $, its \textit{Jacobian} is
        \begin{equation*}
            \nabla\mathbf{f}(\mathbf{x}') \triangleq \begin{bmatrix}
                \ \frac{\partial f_1}{\partial x_1}(\mathbf{x}') \ & \ldots &
                    \frac{\partial f_1}{\partial x_n}(\mathbf{x}') \ \\
                \ \vdots & \ddots & \vdots \ \\
                \frac{\partial f_m}{\partial x_1}(\mathbf{x}') \ & \ldots &
                    \frac{\partial f_m}{\partial x_n}(\mathbf{x}') \
            \end{bmatrix} \triangleq \begin{bmatrix}
                \ \frac{\partial f_1}{\partial\mathbf{x}}(\mathbf{x}') \ \\
                \ \vdots \ \\
                \ \frac{\partial f_m}{\partial\mathbf{x}}(\mathbf{x}') \
            \end{bmatrix} \in \mathbb{R}^{m \times n}
        \end{equation*}
        Here $ \mathbf{x}' \in \mathbb{R}^n $, $ \mathbf{f} \triangleq
        [ \ f_1 \ \ldots \ f_ m \ ]^\top $, $ f_k : \mathbb{R}^n
        \rightarrow \mathbb{R} $, $ \forall k \in \{1, \ldots n\} $.
    \end{itemize}

    % adjust spacing for footnote
    \bigskip
\end{frame}

\begin{frame}{Gradient, Jacobian, Hessian}
    \begin{itemize}
        \item
        Gradient is linear approximation of function change along axes.

        \item
        \textit{Definition.} For $ f : \mathbb{R}^n \rightarrow \mathbb{R} $,
        the \textit{Hessian} $ \nabla^2f : \mathbb{R}^n \rightarrow
        \mathbb{R}^{n \times n} $ is s.t.
        \begin{equation*}
            \nabla^2 f(\mathbf{x}) \triangleq \begin{bmatrix}
                \ \frac{\partial^2 f}{\partial x_1^2}(\mathbf{x}) & \ldots &
                    \frac{\partial^2 f}{\partial x_1x_n}(\mathbf{x}) \ \\
                \ \vdots & \ddots & \vdots \ \\
                \ \frac{\partial^2 f}{\partial x_nx_1}(\mathbf{x}) & \ldots &
                    \frac{\partial^2 f}{\partial x_n^2}(\mathbf{x}) \
            \end{bmatrix}
        \end{equation*}
        $ \forall \mathbf{x} \in \mathbb{R}^n $, $ \nabla^2f(\mathbf{x})
        = \nabla^2f(\mathbf{x})^\top $ ($ \nabla^2f(\mathbf{x}) $ symmetric).

        \item
        The Hessian approximates the local curvature of $ f $ at a point.

        \item
        Provides a way to check if $ f $ is locally \textit{convex} at a
        point. If $ f $ is globally convex, then $ \forall \mathbf{x} \in
        \operatorname{dom}f $, $ \nabla^2f(\mathbf{x}) \succeq \mathbf{0} $
        \cite{bv_convex_opt}.

        \item
        For large $ n $, computing and storing the Hessian is expensive, so
        in practice repeatedly computing the Hessian is not a good idea.
    \end{itemize}
\end{frame}

\subsection{Derivative cookbook}

\begin{frame}{Derivative cookbook}
    \begin{itemize}
        \item
	    Gradients, Jacobians, and Hessians of some common expressions.

	    \item
	    \textit{Squared} $ \ell^2 $\textit{-norm.} For $ \mathbf{x} \in
	    \mathbb{R}^n $, $ \nabla \Vert\mathbf{x}\Vert_2^2 =
	    \nabla\mathbf{x}^\top\mathbf{x} = 2\mathbf{x} $.
	    \begin{equation*}
            \mathbf{x}^\top\mathbf{x} = \sum_{k = 1}^nx_k^2 \Rightarrow
            \nabla \mathbf{x}^\top\mathbf{x} = \begin{bmatrix}
                \ 2x_1 \ \\ \ \vdots \ \\ \ 2x_n \
            \end{bmatrix} \triangleq 2\mathbf{x}
	    \end{equation*}

        \item
        \textit{Linear transformation.} For $ \forall \mathbf{A} \in
        \mathbb{R}^{m \times n} $, $ \nabla \mathbf{Ax} = \mathbf{A} $.
        \begin{equation*}
            \mathbf{Ax} = \begin{bmatrix}
                \ \sum_{k = 1}^na_{1k}x_k \ \\ \ \vdots \ \\
                \ \sum_{k = 1}^na_{mk}x_k \
            \end{bmatrix} \in \mathbb{R}^m \Rightarrow
            \nabla\mathbf{Ax} = \begin{bmatrix}
                \ a_{11} & \ldots & a_{1n} \ \\ \ \vdots \ \\
                \ a_{m1} & \ldots & a_{mn} \
            \end{bmatrix} \triangleq \mathbf{A}
        \end{equation*}
    \end{itemize}
\end{frame}

\begin{frame}{Derivative cookbook}
    \begin{itemize}
        \item
        \textit{Quadratic form.} $ \forall \mathbf{Q} \in
        \mathbb{R}^{n \times n} $ symmetric,
        \begin{equation*}
            \begin{split}
                \mathbf{x}^\top\mathbf{Qx} & = \sum_{j = 1}^n\sum_{k = 1}^n
	                q_{jk}x_jx_k = \sum_{j = 1}^nq_{jj}x_j^2 +
	                2\sum_{j = 1}^n\sum_{k = j + 1}^nq_{jk}x_jx_k \\
                & \Rightarrow \nabla \mathbf{x}^\top\mathbf{Qx} =
                \begin{bmatrix}
	                \ \sum_{k = 1}^n2q_{1k}x_k \ \\ \ \vdots \ \\
	                \ \sum_{k = 1}^n2q_{nk}x_k \
	            \end{bmatrix} \triangleq 2\mathbf{Qx} \\
	            & \Rightarrow \nabla^2\mathbf{x}^\top\mathbf{Qx} =
	            \nabla\left(\nabla\mathbf{x}^\top\mathbf{Qx}\right) =
	            \begin{bmatrix}
                    \ 2q_{11} & \ldots & 2q_{1n} \ \\
                    \ \vdots & \ddots & \vdots \ \\
                    \ 2q_{n1} & \ldots & 2q_{nn} \
                \end{bmatrix} \triangleq 2\mathbf{Q}
            \end{split}
        \end{equation*}
    \end{itemize}
\end{frame}

\begin{frame}{Derivative cookbook}
    \begin{itemize}
        \item
        \textit{Multivariate chain rule.} Suppose $ h : \mathbb{R}^n
        \rightarrow \mathbb{R} $ is the \textit{composition} of $ f :
        \mathbb{R}^m \rightarrow \mathbb{R} $, $ g : \mathbb{R}^n
        \rightarrow \mathbb{R}^m $, i.e. $ h \triangleq f \circ g $. Then,
        $ \forall \mathbf{x} \in D \subseteq \mathbb{R}^n $, $ D $ the set of
        points where $ \nabla h : \mathbb{R}^n \rightarrow \mathbb{R}^n $ is
        defined,
        \begin{equation*}
            \nabla h(\mathbf{x}) = [\nabla g(\mathbf{x})]^\top\nabla f(g(\mathbf{x}))
        \end{equation*}
        Here $ \nabla f(g(\mathbf{x})) \in \mathbb{R}^m $,
        $ \nabla g(\mathbf{x}) \in \mathbb{R}^{m \times n} $.
        In the special case that $ m = 1 $, we simply have
        $ \nabla h(\mathbf{x}) = f'(g(\mathbf{x}))\nabla g(\mathbf{x}) $.

        \item
        \textit{Remark.} The fact that the $ n \times 1 $ gradient is a
        transposed $ 1 \times n $ Jacobian can lead to some confusion,
        especially with the overloading of the $ \nabla $ operator. Context
        is important.
    \end{itemize}
\end{frame}

% BibTeX slide for references. should use either acm or ieeetr style
\begin{frame}{References}
    \bibliographystyle{acm}
    \bibliography{../master_bib}
\end{frame}

\end{document}