\section{Regular expressions}

In \Pascal, an identifier is a letter followed by zero or more letters
or digits, that is, and identifier is a member of the set defined by
\(L(L \cup D)^{*}\). The notation we introduced so far is comfortable
for mathematics but not for computers. Let us introduce another
notation, called \emph{regular expressions}, for describing the same
languages and define its meaning in terms of the mathematical
notation. With this notation, we might define \Pascal identifiers as
\begin{center}
\term{letter} \lparen\term{letter} \disj \term{digit}\rparen\kleene
\end{center}
where the vertical bar means `or', the parentheses group
subexpressions, the star means `zero or more instances of' the
previous expression and juxtaposition means concatenation. A regular
expression~\(r\) is built up out of simpler regular expressions using
a set of rules, as follows. Let~\(\Sigma\) be an alphabet and \(L(r)\)
the language denoted by~\(r\). Then
\begin{enumerate*}

   \item \(\epsilon\) is a regular expression that denotes
     \(\{\varepsilon\}\);\label{regexp_empty}

   \item if \(a \in \Sigma\), then \(a\)~is a regular expression that
     denotes~\(\{a\}\). This is ambiguous: \(a\)~can denote a
     language, a word or a letter --~it depends on the
     context; \label{regexp_sym}

   \item assume \(r\)~and~\(s\) denote the languages \(L(r)\)~and
     \(L(s)\); \(a\)~denotes a letter. Then \label{regexp_rec}
   \begin{enumerate*}

     \item \(r\) \disj \(s\) is a regular expression
     denoting \(L(r) \cup L(s)\);

     \item \(r s\) is a regular expression denoting \(L(r) L(s)\);

     \item \(r\)\kleene{} is a regular expression
     denoting \((L(r))^{*}\);

     \item \lparen\(r\)\rparen{} is a regular expression
     denoting \(L(r)\);

     \item \(\overline{a}\) is a regular expression denoting
       \(\Sigma\backslash \{a\}\).

   \end{enumerate*}

\end{enumerate*}
A language described by a regular expression is a \emph{regular
  language}. Rules~\ref{regexp_empty} and~\ref{regexp_sym} form the
base of the definition. Rule~\ref{regexp_rec} provides the inductive
step. Unnecessary parentheses can be avoided in regular expressions if
\begin{itemize*}

  \item the unary operator \kleene{} has the highest precedence and
  is left associative,

  \item concatenation has the second highest precedence and is left
  associative,

  \item \disj{} has the lowest precedence and is left associative.

\end{itemize*}
Under those conventions, \lparen\(a\)\rparen{} \disj
\lparen\lparen\(b\)\rparen\kleene\lparen\(c\)\rparen\rparen{} is
equivalent to \(a\) \disj \(b\)\kleene\(c\). Both expressions denote
the language containing either the string \(a\) or zero or more
\(b\)'s followed by one \(c\): \(\{a, c, bc, bbc, bbbc, \dots\}\). For
example,
\begin{itemize*}

  \item the regular expression \(a\) \disj \(b\) denotes the set
    \(\{a, b\}\);

  \item the regular expression \lparen\(a\) \disj
    \(b\)\rparen\lparen\(a\) \disj \(b\)\rparen{} denotes \(\{aa, ab,
    ba, bb\}\), the set of all strings of \(a\)'s and \(b\)'s of
    length two. Another regular expression for the set is \(aa\) \disj
    \(ab\) \disj \(ba\) \disj \(bb\);

  \item the regular expression \(a\)\kleene{} denotes the set of all
    strings of zero or more \(a\)'s, i.e. \(\{\varepsilon, a, aa, aaa,
    \dots\}\);

  \item the regular expression \lparen\(a\) \disj
    \(b\)\rparen\kleene{} denotes the set of all strings containing
    zero of more instances of an \(a\) or \(b\), that is the language
    of all words made of \(a\)'s and \(b\)'s. Another expression is
    \lparen\(a\)\kleene\(b\)\kleene\rparen\kleene.

\end{itemize*}
If two regular expressions \(r\)~and~\(s\) denote the same language,
we say that \(r\)~and~\(s\) are \emph{equivalent} and write \(r =
s\). In \fig~\vref{fig:laws},
\begin{figure}
\centering
\begin{tabular}{c|l}
\toprule
  \multicolumn{1}{c}{\textsc{Law}}
& \multicolumn{1}{c}{\textsc{Description}}\\
\midrule
  \(r\) \disj \(s\) = \(s\) \disj \(r\)
& \disj is commutative\\
\hline
  \(r\) \disj \lparen\(s\) \disj \(t\)\rparen{}
  = \lparen\(r\) \disj \(s\)\rparen{} \disj \(t\)
& \disj is associative\\
\hline
  \lparen\(rs\)\rparen \(t\) = \(r\)\lparen\(st\)\rparen
& concatenation is associative\\
\hline
  \(r\)\lparen\(s\) \disj \(t\)\rparen{} = \(rs\) \disj \(rt\)
& concatenation distributes over \disj\\
  \lparen\(s\) \disj \(t\)\rparen \(r\) = \(sr\) \disj \(tr\)
&\\
\hline
  \(\epsilon r = r\)
& \(\epsilon\) is the identity element\\
  \(r \epsilon = r\)
& for the concatenation\\
\hline
  \(r\)\kleene\kleene = \(r\)\kleene
& Kleene closure is idempotent\\
\hline
  \(r\)\kleene = \(r\)\plus \disj \(\epsilon\)
& Kleene closure and positive closure\\
  \(r\)\plus = \(r r\)\kleene
& are closely linked\\
\bottomrule
\end{tabular}
\caption{Algebraic laws on regular languages\label{fig:laws}}
\end{figure}
we show useful algebraic laws on regular languages.

\subsection*{Regular definitions}

It is convenient to give names to regular expressions and define new
regular expressions using these names as if they were symbols. If
\(\Sigma\) is an alphabet, then a \emph{regular definition} is a
series of definitions of the form
\begin{align*}
    d_1 &\rightarrow r_1\\
    d_2 &\rightarrow r_2\\
    &\cdots\\
    d_n &\rightarrow r_n
\end{align*}
where each \(d_i\) is a distinct name and each \(r_i\) is a regular
expression over the alphabet \(\Sigma \cup \{d_1, d_2, \dots,
d_{i-1}\}\), \emph{i.e.,} the basic symbols and the previously defined
names. The restriction to \(d_j\) such \(j < i\) allows to construct a
regular expression over \(\Sigma\) only by repeatedly replacing all
the names in it. For instance, as we have stated, the set of \Pascal
identifiers can be defined by the regular definitions
\begin{align*}
\term{letter} & \rightarrow \text{\exc{A} \disj \exc{B} \disj
  \ldots \disj \exc{Z} \disj \exc{a} \disj \exc{b} \disj \ldots \disj
  \exc{z}} \\
\term{digit} & \rightarrow \text{\exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{id} & \rightarrow \text{\term{letter} \lparen\term{letter}
  \disj \term{digit}\rparen\kleene}
\intertext{Unsigned numbers in \Pascal are strings like
\texttt{5280}, \texttt{39.37}, \texttt{6.336E4}
or \texttt{1.894E-4}.}
\term{digit} & \rightarrow \text{ \exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{digits} & \rightarrow \text{\term{digit} \term{digit}\kleene}\\
\term{optional\_fraction} & \rightarrow \text{\exc{.} \term{digits}
  \disj} \, \epsilon\\
\term{optional\_exponent} & \rightarrow \text{\lparen\exc{E} \lparen
  \exc{+} \disj \exc{-} \disj} \, \epsilon \, \text{\rparen{}
  \term{digits}\rparen{} \disj} \, \epsilon\\
\term{num} & \rightarrow \text{\term{digits} \term{optional\_fraction}
  \term{optional\_exponent}}
\end{align*}
Certain constructs occur so frequently in regular expressions that it
is convenient to introduce notational shorthands for them:
\begin{itemize}

  \item \emph{Zero or one instance.} The unary operator `\opt{}' means
    `zero or one instance of.' Formally, by definition, if~\(r\) is a
    regular expression then \(r\)\opt = \(r\) \disj \(\epsilon\). In
    other words, \lparen\(r\)\rparen\opt{} denotes the language \(L(r)
    \cup \{\varepsilon\}\).
\begin{align*}
\term{digit} & \rightarrow \text{\exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{digits} & \rightarrow \text{\term{digit}\plus}\\
\term{optional\_fraction} & \rightarrow \text{\lparen\exc{.}
  \term{digits}\rparen\opt}\\
\term{optional\_exponent} & \rightarrow \text{\lparen\exc{E} \lparen
  \exc{+} \disj \exc{-}\rparen\opt{} \term{digits}\rparen\opt}\\
\term{num} & \rightarrow \text{\term{digits} \term{optional\_fraction}
  \term{optional\_exponent}}
\end{align*}

 \item It is also possible to write:
\begin{align*}
\term{digit} & \rightarrow \text{\exc{0} \disj \exc{1} \disj \exc{2}
  \disj \exc{3} \disj \exc{4} \disj \exc{5} \disj \exc{6} \disj
  \exc{7} \disj \exc{8} \disj \exc{9}}\\
\term{digits} & \rightarrow \text{\term{digit}\plus}\\
\term{fraction} & \rightarrow \text{\exc{.} \term{digits}}\\
\term{exponent} & \rightarrow \text{\exc{E} \lparen \exc{+} \disj
  \exc{-}\rparen\opt{} \term{digits}}\\
\term{num} & \rightarrow \text{\term{digits} \term{fraction}\opt{}
  \term{exponent}\opt}
\end{align*}

\end{itemize}
If we want to specify the characters `\texttt{?}', `\texttt{*}',
`\texttt{+}', `\texttt{|}', we write them with a preceding backslash,
\emph{e.g.,} `\verb+\?+', or between double-quotes, \emph{e.g.,}
\verb+"?"+. Then, of course, the character double-quote must have a
backslash: \verb+\"+. It is also sometimes useful to match against end
of lines and end of files: \verb+\n+ stands for the control character
`end of line' and \term{\$} is for `end of file'.

\subsection*{Non-regular languages}

Some languages cannot be described by any regular expression. For
example, the language of balanced parentheses cannot be recognised by
any regular expression: \lparen\rparen, \lparen\lparen\rparen\rparen,
\lparen\rparen\lparen\rparen,
\lparen\lparen\lparen\rparen\rparen\lparen\rparen\rparen{}
etc. Another example is the C programming language: it is not a
regular language because it contains embedded blocs between `\verb+{+'
and `\verb+}+'. Therefore, a lexer cannot recognise valid C programs:
we need a parser.

\paragraph{Exercises}

\input{regexp_question_01}
\input{regexp_answer_01}
\input{regexp_question_02}
\input{regexp_answer_02}
\input{regexp_question_03}
\input{regexp_answer_03}


\subsubsection{Snapshots and Transformations to a Frequency Domain}

While it is often useful to know the exact state of the system at any time step $q$, this actually leads to problems in practical implementations:

\begin{description}
  \item[Memory Consumption] Assuming that each value is a complex value with double precision leads to a space consumption of $2\cdot\SI{64}{bit}$ = \SI{128}{bit} or \SI{16}{byte} for every single point of $\mathbf{E}$ and $\mathbf{H}$. A cubic system with a side length of \SI{200}{nm} and a grid size of \SI{0.5}{nm} already has 64 million points which corresponds to about \SI{976}{mb} of memory consumption just for the $\mathbf{E}$ field for a single point in time.
  \item[Mixed $\mathbf{E}$ and $\mathbf{H}$ Fields] If the discretisation $\Delta_t$ is not corresponding to the period of the electromagnetic wave there will be a mixture of $\mathbf{E}$ and $\mathbf{H}$ at any given timestep $q$ which is undesired in most cases because only the $\mathbf{E}$ field might be of interest. The same problem arises for the spatial propagation of the wave.
\end{description}

To solve these problems most implementations of FDTD transform the electric field $\mathbf{E}(\mathbf{x},t)$ with a Fourier transformation to $\mathbf{E}(\mathbf{x}, \nu)$ with $\nu$ being the frequency of the electric field. In case only a specific frequency $\nu$ is of interest all other values can already be discarded while the simulation is running which is greatly reducing the memory needed to perform a calculation.

This also solves the problem of mixed fields: Over many time steps $q$ the calculated $\mathbf{E}$ values will average out the error that would be present by just using a single iteration.


% !TEX root = index.tex

\section{Vector spaces}

\subsection{Motivation}
\label{section:VectorSpaces}
A \emph{scalar} is a real number.
A (column) \emph{vector} $\vec{v}$ of size $n$ is a column of $n$ scalars
\begin{align*}
  \vec{v} =
  \begin{bmatrix}
    v_1 \\
    v_2 \\
    \vdots \\
    v_n
  \end{bmatrix}
\end{align*}
The scalars $v_i$ are called the \textit{coordinates} of $\vec{v}$.
The set of all (column) vectors is called the \emph{Euclidean space} of dimension $n$ and is denoted by $\bbr^n$.
Denote by $\vec{0}$ the vector with all coordinates 0.
$\vec{e}_i$ is the vector whose $i^{th}$ coordinate is 1 and all other coordinates are 0.
The vector $\vec{e}_i$ is called the \emph{$i^{th}$ standard basis vector} and the collection $\calb= \set{\vec{e}_1, \dots, \vec{e}_n}$ is called the \emph{standard basis}.

The spaces $\bbr^1$, $\bbr^2$, $\bbr^3$ are very commonly studied objects, but unfortunately, most of us are unable to visualize the higher dimensions.
Linear algebra is the language that lets us do exactly this.

\begin{figure}[H]
  \centering
  \begin{subfigure}[b]{0.45\textwidth}
    \input{images/r1.tex}
    \caption*{$\bbr^1$ is the real line,}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.45\textwidth}
    \input{images/r2.tex}
    \caption*{$\bbr^2$ is the Euclidean plane,}
  \end{subfigure}
\end{figure}
\begin{figure}[H]
  \centering
  \begin{subfigure}[b]{0.45\textwidth}
    \input{images/r3.tex}
    \caption*{$\bbr^3$ is the three dimensional space,}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.45\textwidth}
    {\Huge \begin{align*}
      ?
    \end{align*}}
    \caption*{What does $\bbr^n$ look like?}
  \end{subfigure}
\end{figure}

There are two operations we can perform on vectors in $\bbr^n$:
\begin{enumerate}
  \item\textbf{Scalar multiplication:} Given a scalar $c$ and a vector $\vec{v}$, the vector $c \vec{v}$ is the vector obtained by multiplying every coordinate of $\vec{v}$ by $c$.

  Geometrically, this is scaling the vector $\vec{v}$ by a factor of $c$.\footnote{If $c$ is negative, we ``flip'' $\vec{v}$ across the origin.}

  \item\textbf{Addition:} Given two vectors $\vec{v}$, $\vec{w}$, the sum $\vec{v} + \vec{w}$ is the vector obtained by adding the respective coordinates.

  Geometrically, vector addition is given by the Parallelogram Law.
\end{enumerate}
\begin{qbox}
  \label{q:parallelogramLaw}
    Let $\vec{v} = \begin{bmatrix}v_1 \\ v_2\end{bmatrix}$ and $\vec{w} = \begin{bmatrix}w_1 \\ w_2\end{bmatrix}$ be vectors in $\bbr^2$.
    Verify that
    \begin{align*}
      \vec{0}, \: \vec{v}, \: \vec{v} + \vec{w}, \: \vec{w}
    \end{align*}
    form the vertices of a parallelogram.\tablefootnote{Here we are thinking of vectors as points. It can be a bit confusing at first, to switch between points and arrows, but this flexibility is very useful once you get used to it.}
    This is called the \textit{parallelogram law of vector addition}. The parallelogram law holds even for $\bbr^n$.
    \begin{figure}[H]
      \input{images/ParallelogramLaw.tex}
      \caption{Parallelogram law of vector addition.}
    \end{figure}
\end{qbox}

\subsubsection{Lines and planes in $\bbr^3$}
A line in $\bbr^2$ is given by an equation $y = mx + c$. What about a line in $\bbr^3$?
One way to decribe a line in $\bbr^3$ is using vector notation.
\begin{ex}
  \label{example:lineAsSpan}
  Every line $L$ in $\bbr^3$ passing through the origin can be written as
  \begin{align*}
    L = \set{c \vec{v} : c \in \bbr}
  \end{align*}
  where $\vec{v}$ is a vector in $\bbr^3$.
  We can then \emph{define} a line passing through the origin in $\bbr^n$ as $\set{c \vec{v} : c \in \bbr}$ for some $\vec{v} \in \bbr^n$.
\end{ex}

What about a plane? We needed one vector to describe a line. We will need two to describe a plane.

\begin{qbox}
  \label{q:planeAsSpan}
    Let $\vec{v}_1$ and $\vec{v}_2$ be non-zero vectors in $\bbr^2$ which are not scalar multiples of each other.
    Using the Parallelogram Law from Q. \ref{q:parallelogramLaw} , argue that for every vector $\vec{v}$ in $\bbr^2$ there exist scalars $c_1, c_2$ such that \begin{equation*}
      \vec{v} = c_1 \vec{v}_1 + c_2 \vec{v}_2.
  \end{equation*}
    \begin{figure}[H]
      \input{images/planeBasis.tex}
    \end{figure}
    So that we can write $\bbr^2$ as
    \begin{align*}
      \bbr^2 = \set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}
    \end{align*}
\end{qbox}
\begin{qbox}
  What is the set $\set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}$, if $\vec{v}_1$ is a scalar multiple of $\vec{v}_2$, for $\vec{v}_1$, $\vec{v}_2 \in \bbr^2$.
\end{qbox}

One can similarly check that if $\vec{v}_1$ and $\vec{v}_2$ are non-zero vectors in $\bbr^3$ which are not scalar multiples of each other, the set $\set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}$ describes planes passing through the origin.
We then \emph{define} a plane in $\bbr^n$ passing through the origin, to be a set of the form $\set{c_1 \vec{v}_1 + c_2 \vec{v}_2 : c_1, c_2 \in \bbr}$.

\begin{qbox}
  How would you define lines and planes in $\bbr^n$ not necessarily passing through the origin (using vector notation)?
\end{qbox}

Our goal then is to generalize this entire story to higher dimensions, by defining ``\emph{$m$ dimensional subspaces of $\bbr^n$}''.
We'll first define a subspace, then slowly move toward defining dimension\footnote{Which is a surprisingly difficult thing to define!}.

\subsection{Subspaces}
\begin{definition}
  A \emph{subspace} of $\bbr^n$ is a \emph{non-empty} subset $V \subseteq \bbr^n$ satisfying the following conditions.
  \begin{enumerate}
    \item (closed under scalar multiplication) For every real number $c$ and vector $\vec{v}$ in $V$, the vector $c \vec{v}$ is in $V$.
    \item (closed under addition) For every $\vec{v}$ and $\vec{w}$ in $V$, the vector $\vec{v} + \vec{w}$ is in $V$.
  \end{enumerate}
\end{definition}
Subspaces, and more generally abstract vector spaces, are the primary objects of study in linear algebra.

\begin{qbox}
  \begin{enumerate}
    \item Show that $\bbr^n$ is a subspace of $\bbr^n$.
    \item Show that the set $V = \{\vec{0}\}$ is a subspace of $\bbr^n$.
    \item For any subspace $V \subseteq \bbr^n$, show that the vector $\vec{0}$ is in $V$.
  \end{enumerate}
\end{qbox}

\begin{qbox}
  Determine, with proof, all the subspaces of $\bbr^1$.
\end{qbox}

We already know two subspaces of $\bbr^2$: $\set{0}$ and $\bbr^2$.
$\bbr^2$ has one other family of subspaces given by lines.
\begin{qbox}
  When is the line in $\bbr^2$ a subspace of $\bbr^2$? What about $\bbr^3$?
\end{qbox}

\begin{qbox}
  When is a plane in $\bbr^3$ a subspace of $\bbr^3$?
\end{qbox}

\begin{qbox}
  Make a guess as to what all the subspaces of $\bbr^2$ and $\bbr^3$ are. How would you prove this?\tablefootnote{We will give a rigorous proof of this tomorrow.}
\end{qbox}

\begin{qbox}
  Let $V$ and $W$ be subspaces of $\bbr^n$.
  \begin{enumerate}
    \item Show that the intersection $V \cap W$ is also a subspace.
    \item What can you say about the union $V \cup W$?
  \end{enumerate}
\end{qbox}

\subsection{Span, Linear Independence, and Basis}
$\spn$ is a general procedure for constructing subspaces of $\bbr^n$. It generalizes the concepts in Example \ref{example:lineAsSpan} and Q. \ref{q:planeAsSpan}.

\begin{definition}
  For a subset $S$ of  $\bbr^n$, the span of $S$ is defined to be the set of finite \emph{linear combinations} of elements of $S$.
  \begin{align*}
    \spn(S) := \{ c_1 \vec{v}_1 + \dots + c_k \vec{v}_k \quad : \quad
    & c_1, \dots, c_k \mbox{ are real numbers}, \\
    & \vec{v}_1, \dots, \vec{v}_k \mbox{ are vectors in } S \}
  \end{align*}
  For the empty set $ \varnothing \subseteq \bbr^n$ we define $\spn(\varnothing):=\{\vec{0}\}$.
  We think of $S$ as being a \emph{generating set} of the subspace $\spn(S)$.
\end{definition}

\begin{qbox}
  Let $S$ be a subset of $\bbr^n$. Show that $\spn(S)$ is a subspace of $\bbr^n$.
\end{qbox}

\begin{qbox}[Practice problems]
  \label{q:span}
  Find simple descriptions of $\spn$s of the following subsets of $\bbr^3$.
  \begin{multicols}{2}
    \begin{enumerate}
      \item $\set{ \vec{e_1}}$
      \item $\set{\vec{e}_1, \vec{e}_{2}}$
      \item $\set{\vec{e}_1, \vec{e}_2 , \vec{e}_3}$
      \item $\bbr^3$
      % \item $\set{\begin{bmatrix} 1 \\ 0 \\ 0\end{bmatrix}, \begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}}$
      \item $\set{\vec{e}_1,\vec{e}_1 + \vec{e}_2, \vec{e}_3}$
      \item $\set{\vec{e}_1-\vec{e}_2, \vec{e}_2-\vec{e}_3, \vec{e}_3-\vec{e}_1}$
      \item The plane $z = 0$
      \item The plane $z = 1$
      % \item $\set{\begin{bmatrix} 1 \\ 0 \\ 0\end{bmatrix}, \begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}, \begin{bmatrix} 0 \\ 2 \\ 0\end{bmatrix}}$
    \end{enumerate}
  \end{multicols}
\end{qbox}

\begin{qbox}
  Let $S$ be a subset of $\bbr^n$ and $V$ be a subspace of $\bbr^n$.
  Show that
  \begin{equation*}
    \mbox{if } S \subseteq V \mbox{ then }\spn(S) \subseteq V.
  \end{equation*}
  Thus $\spn(S)$ is the \emph{smallest vector space} containing $S$.
\end{qbox}

The notion of $\spn$ allows for redundancies. For example, the set $\set{\vec{e}_1, \vec{e}_2 , \vec{e}_3}$ has 3 elements and the set $\bbr^3$ has infinitely many, but both of these sets have the same $\spn$.
This redundancy is precisely captured by \emph{linear dependence}.

\begin{definition}
  \label{def:linearIndependence}
  A finite set of vectors $\cals = \set{\vec {v}_1, \dots, \vec{v}_k} \subseteq \bbr^n$ is \emph{linearly dependent} if there exist real numbers $c_1, \dots, c_k$, not all 0, satisfying
  \begin{equation*}
    c_1 \vec{v}_1 + \dots + c_k \vec{v}_k = 0.
  \end{equation*}
  $\cals$ is called \emph{linearly independent} otherwise i.e. $\cals$ is \emph{linearly independent} if the only real numbers $c_1, \dots, c_k$ for which
  \begin{equation*}
    c_1 \vec{v}_1 + \dots + c_k \vec{v}_k = 0.
  \end{equation*}
  are $c_1 = \dots = c_k = 0$.

  An infinite set $\cals \subseteq \bbr^n$ is said to be \emph{linearly dependent} if it contains a finite subset which is linearly dependent, it is called \emph{linearly independent} otherwise.
\end{definition}

\begin{qbox}
  Let $\vec{v}_1$, $\vec{v}_2$ be vectors in $\bbr^n$.
  \begin{enumerate}
    \item When is the set $\cals = \set{\vec{v}_1}$ linearly independent?
    \item When is the set $\cals' = \set{\vec{v}_1, \vec{v}_2}$ linearly independent?
  \end{enumerate}
\end{qbox}

\begin{qbox}
  \sout{Is the empty set $\varnothing \subset \bbr^n$ linearly dependent or independent?}
\end{qbox}

\begin{qbox}
  \begin{enumerate}
    \item Show that a set $\cals$ is linearly independent if and only if every subset of $\cals$ is linearly independent.
    \item Is the statement still true if we replace linear independence with linear dependence?
  \end{enumerate}
\end{qbox}

\begin{definition}
  For a subspace $V$ of $\bbr^n$, a set $\calb \subseteq V$ is said to be a \emph{basis} of $V$ if
  \begin{enumerate}
    \item $\spn(\calb) = V$,
    \item ${\calb}$ is linearly independent.
  \end{enumerate}
\end{definition}

\begin{qbox}
  For each of the sets $\cals$ in Q. \ref{q:span}, find a basis of $\spn(\cals)$.
\end{qbox}

The following theorem is what makes a basis extremely useful.

\begin{theorem}
  \label{theorem:generalizedCoordinates}
  Let $V$ be a subspace of $\bbr^n$ with a basis $\calb = \set{\vec{v}_1, \dots, \vec{v}_k}$.
  For every vector $\vec{v}$ in $V$ there exist unique scalars $c_1, \dots, c_k$ such that
  \begin{align*}
    \vec{v} = c_1 \vec{v}_1 + \dots + c_k \vec{v}_k.
  \end{align*}
\end{theorem}

\begin{qbox}
  Prove Theorem \ref{theorem:generalizedCoordinates}.\hint{First show that the scalars $c_i$ exist. Then argue that they are unique using a proof by contradiction.}
\end{qbox}

\begin{definition}
  Let $V$ be a subspace of $\bbr^n$ with a basis $\calb = \set{\vec{v}_1, \dots, \vec{v}_k}$.
  If $\vec{v} = c_1 \vec{v}_1 + \dots + c_k \vec{v}_k$ as in Theorem \ref{theorem:generalizedCoordinates}, we denote
  \begin{align*}
    \coords{v}{\calb}
    = \begin{bmatrix}
      c_1 \\
      \vdots \\
      c_k
      \end{bmatrix}.
  \end{align*}
  The $c_i$'s are the \emph{coordinates} of $\vec{v}$ in the basis $\calb$.
\end{definition}
Thus if the subspace $V$ has a basis $\calb$ of size $k$ then $V$ ``behaves'' like the Euclidean space $\bbr^k$.
\begin{ex}
  If $\vec{v}$
  is a vector in $\bbr^n$ and $\calb$ is the standard basis, then $\coords{v}{\calb} = \vec{v}$, hence the name \emph{standard basis}.
  The standard basis is a special basis that exists only for $\bbr^n$, for other vector spaces there is usually no natural choice of a basis.
  But we will show that \emph{a} basis always exists.
\end{ex}

\begin{qbox}
    Let $V$ be a subspace of $\bbr^n$ with a basis $\calb = \set{\vec{v}_1, \dots, \vec{v}_k}$.
    Let $\vec{v}$, $\vec{w}$ be vectors in $V$ and let $\alpha$ be a scalar.
    Show that
    \begin{enumerate}
      \item $[\alpha \vec{v}]_\calb = \alpha [\vec{v}]_\calb$
      \item $[\vec{v} + \vec{w}]_\calb = [\vec{v}]_\calb + [\vec{w}]_\calb$
      \item $\vec{v} = \vec{w}$ if and only if $[\vec{v}]_\calb = [\vec{w}]_\calb$.
    \end{enumerate}
\end{qbox}

\begin{qbox}[Practice problems]
  Find $\coords{v}{\calb}$ of the vector $\vec{v} = \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix} \in V$ for $V$ and $\calb$ as below.
    \begin{enumerate}
      \item $V = \bbr^3$, $\calb = \set{\vec{e}_1, \vec{e}_2, \vec{e}_3}$
      \item $V = \bbr^3$, $\calb = \set{\vec{e}_1, \vec{e}_3, \vec{e}_2}$
      \item $V = \bbr^3$, $\calb = \set{\vec{e}_1 + \vec{e}_2, \vec{e}_2, \vec{e}_3}$
      \item $V = \set{z = 0}$, $\calb = \set{\vec{e}_1, \vec{e}_2}$
      \item $V = \set{x + y + z = 0}$, $\calb = \set{\vec{e}_1 - \vec{e}_3, \vec{e}_2 - \vec{e}_3}$
      \item $V = \spn\set{\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}}$, $\calb = \set{\begin{bmatrix} 2 \\ -2 \\ 0 \end{bmatrix}}$
      \end{enumerate}
\end{qbox}

	Generalized coordinates are \emph{intrinsic} to the subspace and do not depend on the ambient vector space.
  However, they are not unique and depend upon the choice of a basis.

  We will show in the next section that the \emph{number} of generalized coordinates, on the other hand, does not depend on the basis. The dimension is then defined to equal this number.

\subsection{Optional: Abstract vector spaces}
We repeatedly used scalar multiplication and addition of vectors in the previous sections.
Everything we did naturally carries over to any structure that has scalar multiplication and addition.

\begin{definition}
  A \emph{vector space} over the real numbers is any set $V$ which has addition and scalar multiplication.
\end{definition}
\begin{ex}
  A subspace of $\bbr^n$ is an example of a vector space.
\end{ex}

\begin{qbox}
  Verify that the following are vector spaces.
  \begin{enumerate}
    \item Set of polynomials in a single variable $x$.
    \begin{align*}
      \mathrm{Poly} = \set{a_0 + a_1 x + \dots + a_n x^n \mid n \in \bbz_{\ge 0}, a_i \in \bbr}
    \end{align*}
    \item Set of polynomials $\mathrm{Poly}_n$ in a single variable $x$ of degree $\le n$.
    \begin{equation*}
      \mathrm{Poly}_n = \set{ a_0 + \dots + a_k x^k \mid k \le n, a_i \in \bbr}.
    \end{equation*}
    \item Set of functions $f: \bbr \rightarrow \bbr$.
    \begin{align*}
      \Maps(\bbr,\bbr) = \set{f:\bbr \rightarrow \bbr}
    \end{align*}
    \item Set of functions $f: \bbr \rightarrow \bbr$ with $f(0) = 0$.
    \begin{align*}
      \Maps_0(\bbr,\bbr) = \set{f:\bbr \rightarrow \bbr}
    \end{align*}
  \end{enumerate}
\end{qbox}

\begin{qbox}
  The set of polynomials in a single variable $x$ of degree $= n$ is \emph{not} a vector space. Why?
\end{qbox}

The notion of a subspace naturally extends to abstract vector spaces: $\mathrm{Poly}_n$ is a subspace of $\mathrm{Poly}$ and $\Maps_0(\bbr,\bbr)$ is a subspace of $\Maps(\bbr,\bbr)$.
The notions of spans, linear independence, and basis also naturally generalize.

\begin{qbox}
  Find bases for $\mathrm{Poly}_n$ and $\mathrm{Poly}$.
\end{qbox}

\begin{qbox}
  Can you find a basis for $\Maps(\bbr,\bbr)$? Are you sure?
\end{qbox}


\section{Subspaces}
\label{sec:vector-space-subspaces}

\begin{outcome}
  \begin{enumerate}
  \item Determine whether a set of vectors is a subspace of a given
    vector space.
  \item Determine whether two sets of vectors span the same subspace.
  \end{enumerate}
\end{outcome}

In this section we will consider subspaces of general vector spaces.

\begin{definition}{Subspace}{subspace-vector-space}
  Let $V$ be a vector space over a field $K$. A subset $W\subseteq V$
  is said to be a \textbf{subspace}%
  \index{subspace}%
  \index{subspace!of vector space} of\/ $V$ if the following conditions hold:
  \begin{enumerate}
  \item $\vect{0}\in W$, where $\vect{0}$ is the additive unit of\/ $V$.
  \item $W$ is \textbf{closed under addition}%
    \index{closed under!addition}%
    \index{addition!closed under}: Whenever $\vect{u}, \vect{v}\in W$,
    then $\vect{u} + \vect{v}\in W$.
  \item $W$ is \textbf{closed under scalar multiplication}%
    \index{closed under!scalar multiplication}%
    \index{scalar multiplication!closed under}: Whenever $k\in K$ and
    $\vect{u}\in W$, then $k\vect{u}\in W$.
  \end{enumerate}
\end{definition}

\begin{example}{Subspaces of $\R^3$}{subspaces-r3}
  As we have seen in Section~\ref{sec:subspaces-rn}, the subspaces of
  $\R^3$ are:
  \begin{itemize}
  \item the zero subspace%
    \index{zero subspace} $\set{\vect{0}}$;
  \item lines through the origin;
  \item planes through the origin;
  \item $\R^3$ itself.
  \end{itemize}
\end{example}

\begin{example}{Space of continuous functions}{subspace-continuous}
  Let $V = \Func_{\R,\R}$, the vector space of functions from real numbers to
  real numbers. Let $W\subseteq V$ be the subset of {\em continuous}
  functions%
  \index{continuous function}%
  \index{function!continuous}. Then $W$ is a subspace of\/ $V$.
\end{example}

\begin{proof}
  We know from calculus that:
  \begin{enumerate}
  \item the zero function, defined by $f(x)=0$ for all $x$, is
    continuous;
  \item if $f,g$ are continuous functions, then $f+g$ is
    continuous;
  \item if $f$ is a continuous function and $k$ a constant, then $kf$
    is continuous.
  \end{enumerate}
  It follows that $W$ contains $0$, and is closed under addition and
  scalar multiplication. Therefore, $W$ is a subspace of\/ $V$.
\end{proof}

\begin{example}{Space of differentiable functions}{subspace-differentiable}
  Let $V = \Func_{\R,\R}$. Recall from calculus that a function
  $f:\R\to\R$ is called \textbf{differentiable}%
  \index{differentiable function}%
  \index{function!differentiable} if the derivative $f'(x)$ exists for
  all $x\in\R$. Let $W\subseteq V$ be the subset of differentiable
  functions. Then $W$ is a subspace of\/ $V$.
\end{example}

\begin{proof}
  We know from calculus that:
  \begin{enumerate}
  \item The zero function, defined by $f(x)=0$ for all $x$, is
    differentiable. In fact, its derivative is $f'(x) = 0$.
  \item If $f,g$ are differentiable functions, then $h=f+g$ is
    differentiable. In fact, $h'(x) = f'(x) + g'(x)$.
  \item if $f$ is a differentiable function and $k$ a constant, then
    $h=kf$ is differentiable. In fact, $h' = kf'$.
  \end{enumerate}
  It follows that $W$ contains $0$, and is closed under addition and
  scalar multiplication. Therefore, $W$ is a subspace of\/ $V$.
\end{proof}

\begin{example}{Space of sequences satisfying a linear recurrence}{subspace-recurrence}
  Let $V=\Seq_{\R}$, the vector space of sequences of real
  numbers. Let
  \begin{equation*}
    W = \set{a \in \Seq_{\R} \mid \mbox{for all $n\geq 0$, $a_{n+2}=a_n+a_{n+1}$}}.
  \end{equation*}
  In other words, $W$ is the set of all sequences satisfying the
  recurrence%
  \index{recurrence!as a subspace} relation
  $a_{n+2}=a_n+a_{n+1}$. Then $W$ is a subspace of\/ $V$.
\end{example}

\begin{proof}
  \def\x#1{\makebox[1.2em][r]{$#1$}}
  \def\y#1{\makebox[3em][r]{$#1$}}
  \def\z#1{\makebox[0em][l]{$#1$}}
  Before we prove that $W$ is a subspace, let us first consider an
  example. The following sequences are elements of\/ $W$, because they
  both satisfy the recurrence:
  \begin{equation*}
    \begin{array}{rc@{}l}
      \y{a} &=& \x{(1},\x{1},\x{2},\x{3},\x{5},\x{8},\x{13},\x{21},\x{\ldots})\z{\,,} \\
      \y{b} &=& \x{(1},\x{3},\x{4},\x{7},\x{11},\x{18},\x{29},\x{47},\x{\ldots})\z{\,.}
    \end{array}
  \end{equation*}
  Note that if we add these sequences, we get
  \begin{equation*}
    \begin{array}{rc@{}l}
      \y{a+b} &=& \x{(2},\x{4},\x{6},\x{10},\x{16},\x{26},\x{42},\x{68},\x{\ldots})\z{\,,} \\
    \end{array}
  \end{equation*}
  which again satisfies the recurrence. Therefore, the set $W$ is
  closed under the addition of these particular sequences $a$ and
  $b$. We now prove the properties in general.
  \begin{enumerate}
  \item Let $z$ be the zero sequence%
    \index{zero sequence}%
    \index{sequence!zero sequence}, defined by $z_n=0$ for all $n$.
    Then $z$ satisfies the recurrence relation, since for all $n\geq 0$,
    $z_{n+2}=0=z_n+z_{n+1}$. Therefore $z\in W$.
  \item To show that $W$ is closed under addition, consider any two
    sequences $a,b\in W$, and let $c=a+b$. Then for all $n\geq 0$,
    \begin{equation*}
      c_{n+2}
      = a_{n+2} + b_{n+2}
      = (a_n + a_{n+1}) + (b_n + b_{n+1})
      = (a_n + b_n) + (a_{n+1} + b_{n+1})
      = c_n + c_{n+1},
    \end{equation*}
    so $c$ satisfies the recurrence. It follows that $c\in W$, and
    therefore $W$ is closed under addition.
  \item To show that $W$ is closed under scalar multiplication,
    consider any $k\in\R$ and $a\in W$, and let $c=ka$. Then for all
    $n\geq 0$,
    \begin{equation*}
      c_{n+2}
      = ka_{n+2}
      = k(a_n + a_{n+1})
      = ka_n + ka_{n+1}
      = c_n + c_{n+1},
    \end{equation*}
    so $c$ satisfies the recurrence. It follows that $c\in W$, and
    therefore $W$ is closed under scalar multiplication.
  \end{enumerate}
\end{proof}

\begin{example}{Solution space of a linear differential equation}{subspace-differential-equation}
  Let $V=\Func_{\R,\R}$. Recall from calculus that a
  \textbf{differential equation}%
  \index{differential equation} is an equation about an unknown
  function and its derivatives. For example
  \begin{equation*}
    f'' = -f
  \end{equation*}
  is a differential equation. The functions $f(x)=\sin x$,
  $f(x)=\cos x$, and $f(x)=0$ are examples of solutions of this
  differential equation.  Let $W$ be the set of all functions that are
  solutions of the differential equation $f'' = -f$. Then $W$ is a
  subspace of\/ $V$.
\end{example}

\begin{proof}
  \begin{enumerate}
  \item The zero function $f(x)=0$ is a solution of the differential
    equation, and therefore an element of\/ $W$.
  \item To show that $W$ is closed under addition, let $f,g\in W$ and
    consider $h=f+g$. Then $f''=-f$ and $g''=-g$, and therefore
    $h'' = f'' + g'' = -f+(-g) = -h$. Therefore, $h\in W$, and $W$ is
    closed under addition.
  \item To show that $W$ is closed under scalar multiplication, let
    $k\in\R$ and $f\in W$, and consider $h=kf$. Then $f''=-f$, and
    therefore $h'' = kf'' = k(-f) = -h$. It follows that $h\in W$, and
    therefore $W$ is closed under scalar multiplication.
  \end{enumerate}
\end{proof}

\begin{example}{Subspace of polynomials}{polynomial-subspace}
  Consider $\Poly_2$, the vector space of polynomials of degree at
  most 2, with coefficients in a field $K$. Fix some element $r\in K$,
  and let $W \subseteq \Poly_2$ be the subset of polynomials that have
  $r$ as a root. Then $W$ is a subspace of $\Poly_2$.
\end{example}

\begin{proof}
  We can express $W$ as follows:
  \begin{equation*}
    W = \set{p\in\Poly_2 \mid p(r)  = 0}.
  \end{equation*}
  We need to show that $W$ is a subspace.
  \begin{enumerate}
  \item The zero polynomial, given by $0(x)=0$, satisfies $0(r)=0$, so
    $0\in W$.
  \item To show that $W$ is closed under addition, assume
    $p, q\in W$, and let $s=p+q$. Then $p(r)=0$ and $q(r)=0$,
    therefore $s(r) = p(r)+q(r) = 0$. It follows that $s\in W$.
  \item To show that $W$ is closed under scalar multiplication, assume
    $p(x)\in W$ and $k$ be a scalar. Then $(kp)(r)=k(p(r))=k0=0$, and
    therefore $kp\in W$.
  \end{enumerate}
\end{proof}

\begin{example}{Trivial subspaces of\/ $V$}{trivial-subspaces}
  Let $V$ be an arbitrary vector space over a field $K$. Then
  $\set{\vect{0}}$ is a subspace of\/ $V$, called the \textbf{zero
    subspace}. Also, $V$ is a subspace of itself.
\end{example}

\begin{proof}
  Clearly $\set{\vect{0}}$ contains $\vect{0}$, and is closed under
  addition and scalar multiplication because
  $\vect{0}+\vect{0}=\vect{0}$ and $k\vect{0}=\vect{0}$ for all $k$.
  Similarly, $V$ contains $\vect{0}$ and is closed under addition and
  scalar multiplication, because addition and scalar multiplication
  are operations on $V$.  Therefore, both $\set{\vect{0}}$ and $V$ are
  subspaces of\/ $V$.
\end{proof}

The interest of subspaces lies in the fact that they are vector spaces
in their own right, as stated in the following proposition.

\begin{proposition}{Subspaces are vector spaces}{subspaces-are-vector-spaces}
  Let $W$ be a subspace of a vector space $V$. Then $W$ satisfies the
  vector space axioms (A1)--(A4) and (SM1)--(SM4), with respect to the
  same operations (addition and scalar multiplication) as those
  defined on $V$.
\end{proposition}

\begin{proof}
  Since $W$ is a subspace, it is closed under addition and scalar
  multiplication. This ensures that addition and scalar multiplication
  are well-defined operations on $W$. The axioms (A1), (A2), (A4), and
  (SM1)--(SM4) all obviously hold in $W$, because they hold in $V$
  (two elements of\/ $W$ are equal in $W$ if and only if they are equal
  in $V$). The axiom (A3) holds because $\vect{0}\in W$.
\end{proof}

We end this section with an observation about spans.

\begin{proposition}{Span is smallest subspace containing given vectors}{span-smallest-subspace}
  Let $V$ be a vector space over some field $K$, and consider a set of
  vectors $S\subseteq V$. Then $\sspan S$ is the smallest subspace of\/
  $V$ containing $S$. More explicitly, we have:
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item The set $\sspan S$ is a subspace of\/ $V$, and
      $S\subseteq \sspan S$.
    \item If $W$ is any other subspace of\/ $V$ such that
      $S\subseteq W$, then $\sspan S\subseteq W$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proposition}

\begin{proof}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item To show that $\sspan S$ is a subspace, first note that
      $\vect{0}\in\sspan S$, because $\vect{0}$ is the empty linear
      combination. Also, if $\vect{v},\vect{u}\in\sspan S$, then by
      definition of span, there exist
      $\vect{v}_1,\ldots,\vect{v}_k,\vect{u}_1,\ldots,\vect{u}_{\ell}\in
      S$ and $a_1,\ldots,a_k,b_1,\ldots,b_{\ell}\in K$ such that
      \begin{eqnarray*}
        \vect{v} &=& a_1\vect{v}_1 + \ldots + a_k\vect{v}_k, \\
        \vect{u} &=& b_1\vect{u}_1 + \ldots + b_{\ell}\vect{u}_{\ell}.
      \end{eqnarray*}
      Then
      \begin{eqnarray*}
        \vect{v}+\vect{u}
        &=& a_1\vect{v}_1 + \ldots + a_k\vect{v}_k
            + b_1\vect{u}_1 + \ldots + b_{\ell}\vect{u},
      \end{eqnarray*}
      and therefore $\vect{v}+\vect{u}\in\sspan S$. It follows that
      $\sspan S$ is closed under addition. The proof for scalar
      multiplication is similar. Finally, every $\vect{v}\in S$ is
      trivially a linear combination of itself, $\vect{v}=1\vect{v}$,
      and therefore $S\subseteq \sspan S$.
    \item Consider any other subspace $W$ of $V$ such that
      $S\subseteq W$. To show that $\sspan S\subseteq W$, consider an
      arbitrary element $\vect{v}\in\sspan S$. By definition of span,
      there exist $\vect{v}_1,\ldots,\vect{v}_k\in S$ and
      $a_1,\ldots,a_k\in K$ such that
      $\vect{v}=a_1\vect{v}_1 + \ldots + a_k\vect{v}_k$. By
      assumption, $\vect{v}_1,\ldots,\vect{v}_k\in W$. Since $W$ is
      closed under addition and scalar multiplication, it follows that
      $\vect{v}\in W$. Since $\vect{v}$ was an arbitrary element of
      $\sspan S$, it follows that $\sspan S\subseteq W$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proof}

While the last proposition looks technical, it can actually be useful
for proving that two sets of vectors span the same subspace. The
following in an example of this.

\begin{example}{Equal spans}{equal-span}
  Show that the sets $S=\set{x^2-2x+1,~ x-1}$ and
  $T=\set{x^2-1,~ x^2-x}$ span the same subspace of $\Poly_2$. In other
  words, show that
  \begin{equation*}
    \sspan S = \sspan T.
  \end{equation*}
\end{example}

\begin{solution}
  To show that two sets are equal, we must show that each is a subset
  of the other. So we will show $\sspan S \subseteq \sspan T$ and
  $\sspan T \subseteq \sspan S$. By
  Proposition~\ref{prop:span-smallest-subspace}, it is sufficient to show that
  $S\subseteq \sspan T$ and $T\subseteq \sspan S$, i.e., we must show
  that every element of $S$ is a linear combination of elements of $T$
  and vice versa.
  \begin{enumerate}
  \item $S \subseteq \sspan T$. We have
    \begin{equation*}
      \begin{array}{rclcl}
      x^2-2x+1 &=& (-1)(x^2-1) + 2(x^2-x) &\in& \sspan T, \\
      x-1 &=& 1(x^2-1) - 1(x^2-x) &\in& \sspan T.
      \end{array}
    \end{equation*}
    Since each element of $S$ is an element of $\sspan T$, it follows
    that $S\subseteq\sspan T$. By
    Proposition~\ref{prop:span-smallest-subspace}, this implies that
    $\sspan S\subseteq \sspan T$.
  \item $T \subseteq \sspan S$. We have
    \begin{equation*}
      \begin{array}{rclcl}
        x^2-1 &=& 1(x^2-2x+1) - 2(x-1) &\in& \sspan S, \\
        x^2-x &=& 1(x^2-2x+1) - 1(x-1) &\in& \sspan S.
      \end{array}
    \end{equation*}
    Since each element of $T$ is an element of $\sspan S$, it follows
    that $T\subseteq\sspan S$. By
    Proposition~\ref{prop:span-smallest-subspace}, this implies that
    $\sspan T\subseteq \sspan S$.
  \end{enumerate}
\end{solution}


\section{08/26}
\subsection{Birthday Paradox}
The probability that no two people share a birthday is
\begin{align*}\prob{E_1 \cap E_2 \cap \dots \cap E_n}
    &= \prob{E_1}\prob{E_2 \given E_1}\cdots \prob{E_n \given E_1 \cap E_2 \cap \dots \cap E_{n - 1}}\\
    &= 1 \times \left(1 - \frac{1}{365}\right) \times \left(1 - \frac{2}{365}\right) \times \cdots \times \left(1 - \frac{365 - n + 1}{365}\right)
    \shortintertext{Applying the inequality $1 - x \leq e^{-x}$ when $\abs{x} < 1$}\\
    &\leq 1 \cdot e^{-\frac{1}{365}} \cdot e^{-\frac{2}{365}} \cdots e^{-\frac{n - 1}{365}}\\
    &= e^{-\frac{n(n-1)}{2\cdot365}}
\end{align*}
Setting
\[e^{-\frac{n(n-1)}{2\cdot365}} \leq \frac{1}{2}\]
yields $n \geq 23$.

\subsection{Randomized Algorithms}
\begin{problem}{Toy Problem}{}
Let $A$ be an array of $n$ numbers with the property that at least one number
occurs at least $\sfrac{n}{2}$ times and the rest are distinct. Our goal is to
find the element with duplicate entries.
\end{problem}

Any deterministic algorithm will require at least $\sfrac{n}{2} + 1$ operations.
However, we can construct a randomized algorithm that only requires
$\bigO{\log{n}}$ operations.

We do this by \emph{random sampling}. Assume that choosing a single element in
$\set{a_1, a_2, \dots, a_n}$ requires $\bigO{1}$ time and that we sample
\emph{with replacement}. We will frequently use sampling with replacement
because it allows the samples to be independent, and therefore simplifies the
analysis. Now, consider the following algorithm:

\begin{algorithm}
    \caption{Randomized Algorithm for Toy Problem}
    \begin{algorithmic}[1]
        \Function{Rand-Duplicate-Toy}{\texttt{arr}}
            \Loop\label{line:toyloop}
                \State $\texttt{sample}_1 \gets \Call{Sample}{\texttt{arr}}$
                \State $\texttt{sample}_2 \gets \Call{Sample}{\texttt{arr}}$
                \If{$\texttt{sample}_1 \neq \texttt{sample}_2$}
                    \State \Return $\texttt{sample}_1$
                \EndIf
            \EndLoop
        \EndFunction
    \end{algorithmic}
\end{algorithm}

This algorithm simply chooses two elements (not necessarily distinct) at random
from the array and repeats until the two elements are identical, after which it
outputs the sampled element as the duplicate.

Let us analyze the probability that this algorithm outputs the duplicate value
after a single iteration. Suppose the two values chosen are $a_i$ and $a_j$.
Then
\begin{align*}\prob{a_i = a_j \text{ and } i \neq j}
    &= \frac{\sfrac{n}{2}}{n} \cdot \frac{\sfrac{n}{2} - 1}{n}\\
    &= \frac{1}{2} \left(\frac{1}{2} - \frac{1}{n}\right)\\
    &= \frac{1}{4} - \frac{1}{2n}\\
    &\geq \frac{3}{2} \hbox{ for $n \geq 5$}
\end{align*}
What is the probability, then, that our algorithm fails after $k$ iterations of
\cref{line:toyloop}? These events are independent, hence the probability is
bounded above by
\begin{align*}\prob{\text{failure}}
    &\leq \left(1 - \frac{3}{20}\right)^k\\
    &\leq e^{-\frac{3}{20}k}
    \shortintertext{Setting $k = \frac{20}{3}\ln{n}$,}\\
    &= e^{-\ln{n}}\\
    &=\frac{1}{n}
\end{align*}

\subsection{Bayes' Law}
\begin{theorem}{Bayes' Law}{}
    For any events $A$ and $B$,
    \[\prob{A \given B} = \frac{\prob{B \given A} \prob{A}}{\prob{B}}\]
\end{theorem}

\subsection{Concepts in Randomized Algorithms}
\begin{definition}{}{}
    An algorithm is \emph{Monte Carlo} if it has a non-zero probability of
    outputting an incorrect solution. If an algorithm will output the correct
    solution with probability 1, it is \emph{Las Vegas}.
\end{definition}

\begin{definition}{High Probability}{}
    Given an input of size $n$, we say an event occurs \emph{with high
    probability} (whp) if it occurs with probability $1 - \sfrac{1}{n^c}$ for
    some $c > 0$.
\end{definition}

\begin{problem}{Modified Toy Problem}{}
    Let $A$ be an array of $n$ numbers with the property that \emph{either} all
    elements are distinct \emph{or} $\sfrac{n}{2}$ elements are repeated.
\end{problem}

Let us define
\begin{align*}
    E_1 &= \hbox{event that exactly $\sfrac{n}{2}$ elements are distinct}\\
    E_2 &= \hbox{event that all $n$ elements are distinct}
\end{align*}

At first, our ``belief'' in $E_1$ and $E_2$ are approximately the same, i.e., we
have no information to determine which event is more likely. However, let us
assume we have sampled two elements with replacement and determined that they
are not equal (call this event $F$). We can update our ``belief'' via Bayes'
Law.
\begin{align*}\prob{E_1 \given F}
    &= \frac{\prob{F \given E_1} \prob{E_1}}{\prob{F}}\\
    &\leq \frac{\left(1 - \frac{3}{20}\right) \cdot \frac{1}{2}}{\left(1 - \frac{3}{20}\right) \cdot \frac{1}{2} + \frac{1}{2}}\\
    &= \frac{17}{37}
\end{align*}
Thus, we see that the probability has shifted from ``50-50'' to skewing \emph{in
favor} of the belief that $E_2$ has occurred.

\subsection{Law of Total Probability}
\begin{theorem}{Law of Total Probability}{}
    For any event $A$ and any countable partition of the sample space $B_1$,
    $B_2$, \dots, that is, the events $B_i$ are pairwise disjoint and their
    union forms the entire sample space, we have
    \[\prob{A} = \sum_i \prob{A \cap B_i} = \sum_i \prob{A \given B_i} \prob{B_i}\]
\end{theorem}

\subsection{Naive Bayesian Classifier}
A Naive Bayesian Classifier is a classic example of how we can apply
probabilistic techniques and Bayes' Law in practice. Suppose we have a
\emph{feature set} $e$, a vector of boolean $n$ boolean variable. These could
be, for example, whether a certain keyword appears in an email in an attempt to
detect spam. Then, assuming the independence of the boolean variables (this is
why it is referred to as \emph{naive}), we have
\begin{align*}\prob{E \given e = (b_1, b_2, \dots, b_n)}
    &= \frac{\prob{e = (b_1, b_2, \dots, b_n) \given E} \prob{E}}{\prob{e = (b_1, b_2, \dots, b_n)}}\\
    &= \frac{\prob{b_1 \given E} \cdot \prob{b_2 \given E} \cdots \prob{b_n \given E} \cdot \prob{event}}{\prob{e = (b_1, b_2, \dots, b_n)}}
\end{align*}


\chapter{Equations}
\section{Quadratic equations}
\begin{dfn}
A quadratic equation is an equation of the form
\begin{equation}
  \label{quad}
  ax^2 + bx + c = 0
\end{equation}
where \( a, b \) and \( c \) are constants and \( a \neq 0 \).
\end{dfn}
\begin{thm}
A quadratic equation (\ref{quad}) has two solutions for the variable \( x \):
\begin{equation}
  \label{root}
  x_{1,2} = \frac{-b \pm \sqrt{b^2-4ac}}{2a}
\end{equation}
\end{thm}
\begin{dfn}
\label{disc}
The \emph{discrimimant} of a quadratic equation (\ref{quad})
is called \( \Delta \) and it is defined as:
\[
  \Delta = b^2 - 4ac
\]
\end{dfn}
\begin{lem}
If the discrimimant \( \Delta \) is zero, then the equation (\ref{quad}) has a double solution: (\ref{root}) becomes:
\[
  x = - \frac{b}{2a}
\]
\end{lem}
\begin{proof}
The equation \eqref{quad} together with the definition \ref{disc} becomes:
\begin{align*}
  x_{1,2} &= \frac{-b \pm \sqrt{b^2-4ac}}{2a} \\
          &= \frac{-b \pm \sqrt{\Delta}}{2a} \\
          &= \frac{-b \pm \sqrt{0}}{2a} \\
          &= -\frac{b}{2a} \\
\end{align*}
and there exists only one solution.
\end{proof}


\subsection{Variation of Parameters for Systems}
\noindent
Let $X$ be a fundamental matrix for the homogeneous system.
\begin{equation*}
	\vec{x_h}' = A\vec{x_h}.
\end{equation*}
That is,
\begin{equation*}
	\vec{x_h} = X\vec{c}
\end{equation*}
where
\begin{equation*}
	\vec{c} = \begin{bmatrix}
	C_1 \\
	\vdots \\
	C_n
	\end{bmatrix}
\end{equation*}
and the entries of matrix $A$ can be any continuous functions of $t$.

\noindent
We are looking for the particular solution $\vec{x_p}$, to the system
\begin{equation*}
	\vec{x} = A\vec{x} + \vec{f}
\end{equation*}
where $\vec{x_p}$ is of the form
\begin{equation*}
	\vec{x_p} = X\vec{v}
\end{equation*}
where $\vec{v}$ is a vector of functions of $t$ that we'll have to find.

\noindent
Differentiating $\vec{x_p}$,
\begin{equation*}
	\vec{x_p}' = X\vec{v}' + X'\vec{v}.
\end{equation*}
From the system we're trying to solve we know that
\begin{equation*}
	X\vec{v}' + X'\vec{v} = A(X\vec{v}) + \vec{f}.
\end{equation*}
Since $X' = AX$,
\begin{equation*}
	X\vec{v}' = \vec{f}.
\end{equation*}

\noindent
Since the columns of $X$ are always linearly independent, we know that $X^{-1}$ always exists.
Multiplying by $X^{-1}$,
\begin{equation*}
	\vec{v}' = X^{-1}\vec{f}.
\end{equation*}
Integrating with respect to $t$,
\begin{equation*}
	\vec{v} = \int{X^{-1}\vec{f} \mathrm{d}t}.
\end{equation*}
So,
\begin{equation*}
	\vec{x_p} = X\int{X^{-1}\vec{f} \mathrm{d}t},
\end{equation*}
and
\begin{equation*}
	\vec{x} = X\vec{c} + X\int{X^{-1}\vec{f} \mathrm{d}t}.
\end{equation*}

\begin{example}
	Find the general solution to the system by variation of parameters
	\begin{equation*}
		\vec{x}' = \begin{bmatrix}
			2 & -3 \\
			1 & -2
		\end{bmatrix}\vec{x} + \begin{bmatrix}
			e^{2t} \\
			1
		\end{bmatrix}
	\end{equation*}
	given the fundamental matrix
	\begin{equation*}
		X = \begin{bmatrix}
			3e^t & e^{-t} \\
			e^t & e^{-t}
		\end{bmatrix}.
	\end{equation*}
\end{example}
\noindent
Finding $\vec{x_h}$ is simply multiplying $X$ by a vector, so we'll save that for the end and focus on $\vec{x_p}$. First we need to find $X^{-1}$. The details are left out, but the process is identical to that for matrices of numbers.
\begin{equation*}
	X^{-1} = \begin{bmatrix}
		\frac{1}{2}e^{-t} & -\frac{1}{2}e^{-t} \\
		-\frac{1}{2}e^t & \frac{3}{2}e^t
	\end{bmatrix}.
\end{equation*}
Multiplying by $\vec{f}$.
\begin{equation*}
	X\vec{f} = \begin{bmatrix}
		\frac{1}{2}e^t - \frac{1}{2}e^{-t} \\
		-\frac{1}{2}e^{3t} + \frac{3}{2}e^t
	\end{bmatrix}.
\end{equation*}
Integrating with respect to $t$,
\begin{equation*}
	\int{X^{-1}\vec{f} \mathrm{d}t} = \begin{bmatrix}
		\frac{1}{2}e^t + \frac{1}{2}e^{-t} \\
		-\frac{1}{6}e^{3t} + \frac{3}{2}e^t
	\end{bmatrix}.
\end{equation*}
Note that although we are doing an indefinite integral, we don't have a constant (or in this case a vector) of integration.
Multiplying by $X$ to obtain $\vec{x_p}$,
\begin{equation*}
	\vec{x_p} = X\int{X^{-1}\vec{f} \mathrm{d}t} = \begin{bmatrix}
		\frac{4}{3}e^{2t} + 3 \\
		\frac{1}{3}e^{2t} + 2
	\end{bmatrix}.
\end{equation*}
Finding $\vec{x_h}$,
\begin{equation*}
	\vec{x_h} = X\vec{c} = \begin{bmatrix}
		3C_1e^t + C_2e^{-t} \\
		C_1e^t + C_2e^{-t}
	\end{bmatrix}.
\end{equation*}
Putting $\vec{x_h}$ and $\vec{x_p}$ together to get the general solution,
\begin{equation*}
	\vec{x} = \begin{bmatrix}
		3C_1e^{3t} + C_2e^{-t} + \frac{4}{3}e^{2t} + 3 \\
		C_1e^t + C_2e^{-t} + \frac{1}{3}e^{2t} + 2
	\end{bmatrix}.
\end{equation*}


\subsection{Heteroskedasticity-Consistent (HC) standard errors}

\subsubsection{Variance of OLS estimators}

\(Var [\hat \theta ]=(X^TX)^{-1}X^T\Omega X(X^TX)^{-1}\)

\subsubsection{Robust standard errors for heteroskedasticity}

\(\Omega_{ij}=\delta_{ij}\epsilon_i\epsilon_j\)

These are also known as the Eicker-Huber-White standard errors, or the White correction.

These are also refered to as robust standard errors.


\subsection{Definition and properties}

When we take the dot product of two vectors, the result is a
scalar. For this reason, the dot product is also called the
\textbf{scalar product}. Sometimes it is also called the \textbf{inner
  product}. The definition is as follows%
\index{dot product}%
\index{vector!dot product}%
\index{scalar product|see{dot product}}%
\index{inner product|seealso{dot product}}.

\begin{definition}{Dot product}{dot-product}
  Let $\vect{u}=\begin{mymatrix}{c}
    u_1 \\
    u_2 \\
    \vdots \\
    u_n
  \end{mymatrix}$, $\vect{v}= \begin{mymatrix}{c}
    v_1 \\
    v_2 \\
    \vdots \\
    v_n
  \end{mymatrix}$ be two vectors in $\R^n$. We
  define their \textbf{dot product} as
  \begin{equation*}
    \vect{u}\dotprod \vect{v} = u_1v_1+u_2v_2+\ldots+u_nv_n.
  \end{equation*}
\end{definition}

\begin{example}{Compute a dot product}{dot-product}
  Find $\vect{u} \dotprod \vect{v}$ for $\vect{u}=\mat{1,2,0,-1}^T$
  and $\vect{v}=\mat{0,1,2,3}^T$.
\end{example}

\begin{solution}
  We have
  \begin{eqnarray*}
    \vect{u} \dotprod \vect{v}
    &=&
        (1)(0) + (2)(1) + (0)(2) + (-1)(3) \\
    &=&
        0 + 2 + 0 + -3 \\
    &=&
        -1.
  \end{eqnarray*}
\end{solution}

The dot product satisfies a number of important properties.

\begin{proposition}{Properties of the dot product}{properties-dot-product}
  \index{dot product!properties}%
  \index{properties of dot product}%
  \index{vector!dot product!properties}%
  \index{vector!properties of dot product}%
  The dot product satisfies the following properties, where
  $\vect{u},\vect{v},\vect{w}$ are vectors and $k,\ell$ are
  scalars.
  \begin{itemize}
  \item $\vect{u}\dotprod\vect{v}=\vect{v}\dotprod\vect{u}$.
  \item $\vect{u}\dotprod\vect{u}\geq 0$, and $\vect{u}\dotprod\vect{u}=0$ if and only if $\vect{u}=\vect{0}$.
  \item $(k\vect{u}+\ell\vect{v})\dotprod\vect{w}=k(\vect{u}\dotprod\vect{w})+\ell(\vect{v}\dotprod\vect{w})$.
  \item $\vect{u}\dotprod(k\vect{v}+\ell\vect{w})
    =k(\vect{u}\dotprod \vect{v})+\ell(\vect{u}\dotprod\vect{w})$.
  \item $\vect{u}\dotprod\vect{u}=\norm{\vect{u}}^2$.
  \end{itemize}
\end{proposition}

The proof is left as an exercise. Note that, by the last part of the
proposition, we can also use the dot product to find the length of a
vector.

\begin{example}{Length of a vector}{dot-product-length}
  Use a dot product to find $\norm{\vect{u}}$, where
  \begin{equation*}
    \vect{u}
    =
    \begin{mymatrix}{r}
      2 \\
      1 \\
      4 \\
      2
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  By the last part of Proposition~\ref{prop:properties-dot-product}, we have
  $\norm{\vect{u}} = \sqrt {\vect{u} \dotprod \vect{u}}$. We have
  $\vect{u} \dotprod \vect{u} = 2^2+1^2+4^2+2^2 = 25$, and therefore
  $\norm{\vect{u}} = \sqrt{\vect{u} \dotprod \vect{u}} = \sqrt{25} = 5$.
\end{solution}


\chapter{Basic Numerical Methods}

\section{Roundoff and Truncation Errors}

\newthought{Floating point numbers} can be represented using either single- (32 bits) or double-precision (64 bits). Single-precision usually stores out to six or seven decimal places while double-precision can usually store up to 16 decimal places. The main pitfall of roundoff error is when two numbers are treated as being equal when the user has not intended for this to be the case. For example, say $\epsilon$ is the smallest number in single-precision that we can store and $\xi<\epsilon$. If for two single-precision floating point numbers $a,b>\epsilon$ we attempt to perform the operation
\begin{equation}
	\frac{b}{(a+\xi)-a},
\end{equation}
we will have some serious problems as our computer will treat $a+\xi$ as $a$, resulting in $0$ in the denominator. Using double-precision often helps to alleviate these problems. Roundoff error depends on the hardware (and probably software) we are using. Thus, it can be persistent across many different parts of the program we are writing.

Truncation error, unlike roundoff error, is generally \textit{not} dependent on our choice of hardware, but rather on our choice of method (i.e. Euler, Runge-Kutta, etc.). Consider the Taylor series
\begin{equation}
	f(x+h) = f(x) + hf^{\prime}(x) + \frac{1}{2}h^2f^{\prime\prime}(x) + \ldots
\end{equation}
This can be rearranged to solve for the derivative of $f$,
\begin{equation}
	f^{\prime}(x)=\frac{f(x+h)-f(x)}{h} + \mathcal{O}(h),
	\label{eq:right_deriv}
\end{equation}
where $\mathcal{O}(h)$ represents the truncation error of order $h$. How is this applied in numerical integration? Consider the example of projectile motion. Our equations of motion can be written as
\begin{align}
	\frac{\mathrm{d}\mathbf{v}}{\mathrm{d}t} &= \mathbf{a}(\mathbf{r},\mathbf{v}), \\[0.5em]
	\frac{\mathrm{d}\mathbf{r}}{\mathrm{d}t} &= \mathbf{v},
\end{align}
where $\mathbf{a}$ is the acceleration. Let's rewrite these equations of motion in terms of Eq. \ref{eq:right_deriv},
\begin{align}
	\frac{\mathbf{v}(t+\tau) - \mathbf{v}(t)}{\tau} + \mathcal{O}(\tau) &= \mathbf{a}(\mathbf{r}(t),\mathbf{v}(t)) \\[0.5em]
	\frac{\mathbf{r}(t+\tau) - \mathbf{r}(t)}{\tau} + \mathcal{O}(\tau) &= \mathbf{v}(t)
\end{align}
and then solving for the updated values of $\mathbf{v}$ and $\mathbf{r}$,
\begin{align}
	\mathbf{v}(t+\tau) &= \mathbf{v}(t) + \tau\mathbf{a}(\mathbf{r}(t),\mathbf{v}(t)) + \mathcal{O}(\tau^2), \\[0.5em]
	\mathbf{r}(t+\tau) &= \mathbf{r}(t) + \tau\mathbf{v}(t) + \mathcal{O}(\tau^2).
\end{align}
Thus, the truncation error for an Euler scheme is quadratic in $\tau$, the timestep. However, we should be careful to draw the distinction between \textit{local} and \textit{global} truncation error. The global error is the sum of all the local truncation errors made at each iteration and is highly dependent on our choice of scheme. Consider a projectile motion problem on $t\in(0,T)$ solved using the Euler method such that the number of steps is $N_{\tau}=T/\tau$. The global error is thus expressed as
\begin{equation}
	\varepsilon_{G}\propto N_{\tau}\times \varepsilon_L.
\end{equation}
In the Euler method case, $\varepsilon\propto T/\tau\times\mathcal{O}(\tau^2)=\mathcal{O}(\tau)$. Thus, the global truncation error of the Euler scheme is $\mathcal{O}(\tau)$.

\section{Curve Fitting and Interpolation}

\newthought{Suppose we have $N$ data points} $(x_i,y_i)$ and we want to fit them to a curve $Y(x;\{a_j\})$ where $\{a_j\}$ is a set of $M$ adjustable parameters. If each point $y_i$ has an associated error $\sigma_i$, the fit parameters can be determined by minimizing the $\chi^2$-function
\begin{equation}
	\label{eq:chi_squared}
	\chi^2(\{a_j\})=\sum^{N}_{i=1}\left(\frac{\Delta_i}{\sigma_i}\right)= \sum^{N}_{i=1}\frac{[Y(x_i;\{a_j\}) - y_i]^2}{\sigma_i^2}.
\end{equation}
The set of parameters $\{a_j\}$ are determined by finding all $a_j$ which satisfy $\partial\chi^2/\partial a_j=0$. Now, say we want to fit $N$ data points to a function of the form
\begin{equation}
	\label{eq:gen_func_lsq}
	Y(x;\{a_j\}) = \sum_{j=1}^Ma_jY_j(x).
\end{equation}
To find $\{a_j\}$, we compute $\partial \chi^2/\partial a_j=0$ which for Eq. \ref{eq:gen_func_lsq} can be expressed as
\begin{align}
	\sum^N_{i=1}\frac{1}{\sigma_i^2}Y_j(x_i)\left(\sum_{k=1}^Ma_kY_k(x_i) - y_i\right) = 0 \\[0.5em]
	\sum^N_{i=1}\sum^M_{k=1}\frac{Y_jYk}{\sigma_i^2} = \sum^N_{i=1}\frac{Y_jy_i}{\sigma_i^2},
\end{align}
for $j=1,\ldots,M$. We can rewrite this equation as
\begin{equation}
	\sum^N_{i=1}\sum^M_{k=1}A_{ij}A_{ik}a_k = \sum^N_{i=1}A_{ij}\frac{y_i}{\sigma_i},
\end{equation}
where $A_{ij}=Y_j(x_i)/\sigma_i$ are the matrix elements of the so-called design matrix. In matrix form, this can be expressed as
\begin{equation}
	(\mathbf{A}^T\mathbf{A})\mathbf{a} = \mathbf{A}^T\mathbf{b},
\end{equation}
where $b_i=y_i/\sigma_i$. This can be easily inverted to solve for $\mathbf{a}$. The estimated errors for $a_j$ can be calculated using the equation
\begin{equation}
	\sigma_{a_j} = \sqrt{C_{jj}},
\end{equation}
where $\mathbf{C} = (\mathbf{A}^T\mathbf{A})^{-1}$ \cite{garcia_numerical_2000}. How does one evaluate whether the curve $Y$ is a good fit to the data $y$? On average, we can say that the difference between the data and the fit should be roughly equal to the error bar such that $\sigma_i\approx|y_i - Y(x_i)|$. Plugging this into Eq. \ref{eq:chi_squared} then gives that $\chi^2\approx N$. However, recalling that we have $M$ degrees of freedom, our condition should be revised such that $\chi^2\approx N-M$ implies an adequate fit. $\chi^2\gg N-M$ suggests that a poor choice of $Y$ or very small $\sigma_i$ and $\chi^2\ll N-M$ implies either overfitting or that the error bars may be too large \cite{garcia_numerical_2000}.
%
\par \hl{Still need something about interpolation}

\section{Numerical Integration}
\section{Basic Linear Algebra: Analytical and Numerical}
\section{Ordinary Differential Equations: Analytical and Numerical}
\section{Partial Differential Equations: Analytical and Numerical}


% !TEX root = lectures.tex
\section{Math Basics}
\bigskip

\subsection{Scalars, Vectors and Matrices}

A scalar is something with a value that is independent of coordinate system. Examples are mass, or the relative time between events. A vector has magnitude and direction. Under rotation, the magnitude stays the same but the direction changes. Scalars have no spatial index, whereas a three-dimensional vector has 3 indices, e.g. the position $\vec{r}$ has components $r_1,r_2,r_3$, which are often referred to as $x,y,z$.

There are several categories of changes of coordinate system. The observer can translate the origin, might move with a different velocity, or might rotate his/her coordinate axes. For instance, a particle's position vector changes when the origin is translated, but its velocity does not. When you study relativity you will find that quantities you thought of as scalars, such as time or electric potential, are actually parts of four-dimensional vectors and that changes of the velocity of the reference frame act in a similar way to rotations.

In addition to vectors and scalars, there are matrices, which have two indices. One also has objects with 3 or four indices. These are called tensors of rank $n$, where $n$ is the number of indices. A matrix is a rank-two tensor. The Levi-Civita symbol, $\epsilon_{ijk}$ used for cross products, is a third-rank tensor.

\subsubsection*{Unit Vectors}

Also known as basis vectors, unit vectors point in the direction of the coordinate axes, have unit norm, and are orthogonal to one another. Sometimes this is referred to as an orthonormal basis,
\begin{equation}
\hat{e}_i\cdot\hat{e}_j=\delta_{ij}=\left(\begin{array}{ccc}
1 & 0 & 0\\
0& 1 & 0\\
0 & 0 & 1
\end{array}\right).
\end{equation}
Here, $\delta_{ij}$ is unity when $i=j$ and is zero otherwise. This is called the unit matrix, because you can multiply it with any other matrix and not change the matrix. The ``dot'' denotes the dot product, $\vec{A}\cdot\vec{B}=A_1B_1+A_2B_2+A_3B_3=|A||B|\cos\theta_{AB}$. Sometimes the unit vectors are called $\hat{x}$, $\hat{y}$ and $\hat{z}$. Vectors can be decomposed in terms of unit vectors,
\begin{equation}
\vec{r}=r_1\hat{e}_1+r_2\hat{e}_2+r_3\hat{e}_3.
\end{equation}
The vector components $r_1$, $r_2$ and $r_3$ might be called $x$, $y$ and $z$ for a displacement of $v_x$, $v_y$ and $v_z$ for a velocity.

\subsubsection*{Rotations}

Here, we use rotations as an example of matrices and their operations. One can consider a different orthonormal basis $\hat{e}'_1$, $\hat{e}'_2$ and $\hat{e}'_3$. The same vector $\vec{r}$ mentioned above can also be expressed in the new basis,
\begin{equation}
\vec{r}=r'_1\hat{e}'_1+r'_2\hat{e}'_2+r'_3\hat{e}'_3.
\end{equation}
Even though it is the same vector, the components have changed. Each new unit vector $\hat{e}'_i$ can be expressed as a linear sum of the previous vectors,
\begin{equation}
\hat{e}'_i=\sum_j U_{ij}\hat{e}_j,
\end{equation}
and the matrix $U$ can be found by taking the dot product of both sides with $\hat{e}_k$,
\begin{eqnarray}
\nonumber
\hat{e}_k\cdot\hat{e}'_i&=&\sum_jU_{ij}\hat{e}_k\cdot\hat{e}_j\\
\label{eq:lambda_angles}
\hat{e}_k\cdot\hat{e}'_i&=&\sum_jU_{ij}\delta_{jk}=U_{ik}.
\end{eqnarray}
Thus, the matrix lambda has components $U_{ij}$ that are equal to the cosine of the angle between new unit vector $\hat{e}'_i$ and the old unit vector $\hat{e}_j$.
\begin{equation}
U = \left(\begin{array}{ccc}
\hat{e}'_1\cdot\hat{e}_1& \hat{e}'_1\cdot\hat{e}_2& \hat{e}'_1\cdot\hat{e}_3\\
\hat{e}'_2\cdot\hat{e}_1& \hat{e}'_2\cdot\hat{e}_2& \hat{e}'_2\cdot\hat{e}_3\\
\hat{e}'_3\cdot\hat{e}_1& \hat{e}'_3\cdot\hat{e}_2& \hat{e}'_3\cdot\hat{e}_3
\end{array}\right),~~~~~U_{ij}=\hat{e}'_i\cdot\hat{e}_j=\cos\theta_{ij}.
\end{equation}
Note that the matrix is not symmetric, $U_{ij}\ne U_{ji}$. One can also look at the inverse transformation, by switching the primed and unprimed coordinates,
\begin{eqnarray}
\label{eq:inverseU}
\hat{e}_i&=&\sum_jU^{-1}_{ij}\hat{e}'_j,\\
\nonumber
U^{-1}_{ij}&=&\hat{e}_i\cdot\hat{e}'_j=U_{ji}.
\end{eqnarray}
The definition of transpose of a matrix, $M^{t}_{ij}=M_{ji}$, allows one to state this as
\begin{eqnarray}
\label{eq:transposedef}
U^{-1}&=&U^{t}.
\end{eqnarray}
A tensor obeying Eq. (\ref{eq:transposedef}) defines what is known as a unitary, or orthogonal, transformation.

The matrix $U$ can be used to transform any vector to the new basis. Consider a vector
\begin{eqnarray}
\vec{r}&=&r_1\hat{e}_1+r_2\hat{e}_2+r_3\hat{e}_3\\
\nonumber
&=&r'_1\hat{e}'_1+r'_2\hat{e}'_2+r'_3\hat{e}'_3.
\end{eqnarray}
This is the same vector expressed as a sum over two different sets of basis vectors. The coefficients $r_i$ and $r'_i$ represent components of the same vector. The relation between them can be found by taking the dot product of each side with one of the unit vectors, $\hat{e}_i$, which gives
\begin{eqnarray}
r_i&=&\sum_j \hat{e}_i\cdot\hat{e}'_j~r'_j.
\end{eqnarray}
Using Eq. (\ref{eq:inverseU}) one can see that the transformation of $r$ can be also written in terms of $U$,
\begin{eqnarray}
\label{eq:rotateR}
r_i&=&\sum_jU^{-1}_{ij}~r'_j.
\end{eqnarray}
Thus, the matrix that transforms the coordinates of the unit vectors, Eq. (\ref{eq:inverseU}) is the same one that transforms the coordinates of a vector, Eq. (\ref{eq:rotateR}).

\example\label{ex:rotmat}
Find the rotation matrix $U$ for finding the components in the primed coordinate system given from those in the unprimed system, given that the unit vectors in the new system are found by rotating the coordinate system by and angle $\phi$ about the $z$ axis.\\
Solution:\\
In this case
\begin{eqnarray*}
\hat{e}'_1&=&\cos\phi \hat{e}_1-\sin\phi\hat{e}_2,\\
\hat{e}'_2&=&\sin\phi\hat{e}_1+\cos\phi\hat{e}_2,\\
\hat{e}'_3&=&\hat{e}_3.
\end{eqnarray*}

By inspecting Eq. (\ref{eq:lambda_angles}),
\[
U=\left(\begin{array}{ccc}
\cos\phi&-\sin\phi&0\\
\sin\phi&\cos\phi&0\\
0&0&1\end{array}\right).
\]
\exampleend

Under a unitary transformation $U$ (or basis transformation) scalars are unchanged, whereas vectors $\vec{r}$ and matrices $M$ change as
\begin{eqnarray}
r'_i&=&U_{ij}~ r_j, ~~({\rm sum~inferred})\\
\nonumber
M'_{ij}&=&U_{ik}M_{km}U^{-1}_{mj}.
\end{eqnarray}
Physical quantities with no spatial indices are scalars (or pseudoscalars if they depend on right-handed vs. left-handed coordinate systems), and are unchanged by unitary transformations. This includes quantities like the trace of a matrix -- the matrix itself had indices but none remain after performing the trace.
\begin{eqnarray}
{\rm Tr} M&\equiv& M_{ii}.
\end{eqnarray}
Because there are no remaining indices, one expects it to be a scalar. Indeed one can see this,
\begin{eqnarray}
{\rm Tr} M'&=&U_{ij}M_{jm}U^{-1}_{mi}\\
\nonumber
&=&M_{jm}U^{-1}_{mi}U_{ij}\\
\nonumber
&=&M_{jm}\delta_{mj}\\
\nonumber
&=&M_{jj}={\rm Tr} M.
\end{eqnarray}
A similar example is the determinant of a matrix, which is also a scalar.

\subsubsection*{Vector and Matrix Operations}
\begin{itemize}\itemsep=0pt
\item Scalar Product (or dot product): For vectors $\vec{A}$ and $\vec{B}$,
\begin{eqnarray*}
\vec{A}\cdot\vec{B}&=&A_iB_i=|A||B|\cos\theta_{AB},\\
|A|&\equiv& \sqrt{\vec{A}\cdot\vec{A}}.
\end{eqnarray*}
Note that the summation sign is inferred any time there are repeated indices. For example,
\begin{eqnarray}
A_iB_i&=&\sum A_iB_i.
\end{eqnarray}
\item Multiplying a matrix $C$ times a vector $\vec{A}$:
\[
(CA)_i=C_{ij}A_j.
\]
For the $i^{\rm th}$ element one takes the scalar product of the $i^{\rm th}$ row of $C$ with the vector $\vec{A}$.
\item Mutiplying matrices $C$ and $D$:
\[
(CD)_{ij}=C_{ik}D_{kj},
\]
This means the one obtains the $ij$ element by taking the scalar product of the $i^{\rm th}$ of $C$ with the $j^{\rm th}$ column of $D$.
\item Vector Product (or cross product) of vectors $\vec{A}$ and $\vec{B}$:
\begin{eqnarray*}
\vec{C}&=&\vec{A}\times\vec{B},\\
C_i&=&\epsilon_{ijk}A_jB_k.
\end{eqnarray*}
Here $\epsilon$ is the third-rank anti-symmetric tensor, also known as the Levi-Civita symbol. It is $\pm 1$ only if all three indices are different, and is zero otherwise. The choice of $\pm 1$ depends on whether the indices are an even or odd permutation of the original symbols. The permutation $xyz$ or $123$ is considered to be $+1$. For the 27 elements,
\begin{eqnarray}
\epsilon_{ijk}&=&-\epsilon_{ikj}=-\epsilon_{jik}=-\epsilon_{kji}\\
\nonumber
\epsilon_{123}&=&\epsilon_{231}=\epsilon_{312}=1,\\
\nonumber
\epsilon_{213}&=&\epsilon_{132}=\epsilon_{321}=-1,\\
\nonumber
\epsilon_{iij}&=&\epsilon_{iji}=\epsilon_{jii}=0.
\end{eqnarray}
You used cross products extensively when studying magnetic fields. Because the matrix is anti-symmetric, switching the $x$ and $y$ axes (or any two axes) flips the sign. If the coordinate system is right-handed, meaning the $xyz$ axes satisfy $\hat{x}\times\hat{y}=\hat{z}$, where you can point along the $x$ axis with your extended right index finger, the $y$ axis with your contracted middle finger and the $z$ axis with your extended thumb. Switching to a left-handed system flips the sign of the vector $\vec{C}=\vec{A}\times\vec{B}$. Note that $\vec{A}\times\vec{B}=-\vec{B}\times\vec{A}$. The vector $\vec{C}$ is perpendicular to both $\vec{A}$ and $\vec{B}$ and the magnitude of $\vec{C}$ is given by
\[
|C|=|A||B|\sin\theta_{AB}.
\]
Vectors obtained by the cross product of two real vectors are called pseudo-vectors because the assignment of their direction can be arbitrarily flipped by defining the Levi-Civita symbol to be based on left-handed rules. Examples are the magnetic field and angular momentum. If the direction of a real vector prefers the right-handed over the left-handed direction, that constitutes a violation of parity. For instance, one can polarize the spins (angular momentum) of nuclei with a magnetic field so that the spins preferentially point along the direction of the magnetic field. This does not violate parity because both are pseudo-vectors. Now assume these polarized nuclei decay and that electrons are one of the products. If these electrons prefer to exit the decay parallel vs. antiparallel to the polarizing magnetic field, this constitutes parity violation because the direction of the outgoing electron momenta are a real vector. This is precisely what is observed in weak decays.

\item Differentiation of a vector with respect to a scalar: For example, the acceleration is $d\vec{v}/dt$:
\[
(d\vec{v}/dt)_i=\frac{dv_i}{dt}.
\]

\item Angular velocity: Choose the vector $\vec{\omega}$ so that the motion is like your fingers wrapping around your thumb on your right hand with $\vec{\omega}$ pointing along your thumb. The magnitude is $d|\phi|/dt$.

\item Gradient operator $\nabla$: This is the derivative $\partial/\partial x$, $\partial/\partial y$ and $\partial/\partial z$, where $\partial _x$ means $\partial/\partial_x$. For taking the gradient of a scalar $\Phi$,
\[
{\rm\bf grad}~\Phi, (\nabla\Phi(x,y,z,t))_i=\partial/\partial r_i\Phi(\vec{r},t)=\partial_i\Phi(\vec{r},t).
\]
For taking the dot product of the gradient with a vector, sometimes called a divergence,
\[
div \vec{A}, \nabla\cdot\vec{A}=\partial_i A_i.
\]
For taking the vector product with another vector, sometimes called curl, $\nabla\times\vec{A}$,
\[
{\rm\bf curl}~\vec{A}, (\nabla\times\vec{A})_i=\epsilon_{ijk}\partial_j A_k(\vec{r},t).
\]
\item The Laplacian is referred to as $\nabla^2$ and is defined as
\[
\nabla^2=\nabla\cdot\nabla=\frac{\partial^2}{\partial x^2}+\frac{\partial^2}{\partial y^2}+\frac{\partial^2}{\partial z^2}.
\]
\end{itemize}

\subsubsection*{Some identities}

Here we simply state these, but you may wish to prove a few. They are useful for this class and will be essential when you study E\&M.
\begin{eqnarray}
\vec{A}\cdot(\vec{B}\times\vec{C})&=&\vec{B}\cdot(\vec{C}\times\vec{A})=\vec{C}\cdot(\vec{A}\times\vec{B})\\
\nonumber
\vec{A}\times(\vec{B}\times\vec{C})&=&(\vec{A}\cdot\vec{C})\vec{B}-(\vec{A}\cdot\vec{B})\vec{C}\\
\nonumber
(\vec{A}\times\vec{B})\cdot(\vec{C}\times\vec{D})&=&(\vec{A}\cdot\vec{C})(\vec{B}\cdot\vec{D})
-(\vec{A}\cdot\vec{D})(\vec{B}\cdot\vec{C})
\end{eqnarray}

\example
The height of a hill is given by the formula
\[
z(x,y)=2xy-3x^2-4y^2-18x+28y+12.
\]
Here $z$ is the height in meters and $x$ and $y$ are the east-west and north-south coordinates. Find the position $x,y$ where the hill is the highest, and give its height.

{\bf Solution:} The maxima or minima, or inflection points, are given when $\partial_x z=0$ and $\partial_yz=0$.
\begin{eqnarray*}
\partial_xz(x,y)&=&2y-6x-18=0,\\
\partial_yz(x,y)&=&2x-8y+28=0.
\end{eqnarray*}
Solving for two equations and two unknowns gives one solution $x=-2,y=3$. This then gives $z=72$. Although this procedure could have given a minimum or an inflection point you can look at the form for $z$ and see that for $x=y=0$ the height is lower, therefore it is not a minimum. You can also look and see that the quadratic contributions are negative so the height falls off to $-\infty$ far away in any direction. Thus, there must be a maximum, and this must be it. Deciding between maximum or minimum or inflection point for a general problem would involve looking at the matrix $\partial_i\partial_j z$ at the specific point, then finding the eigenvalues of the matrix and seeing whether they were both positive (minimum) both negative (maximum) or one positive and one negative (saddle point).

\exampleend

\subsubsection*{Gauss's Theorem and Stokes's Theorem}

For an integral over a volume $V$ confined by a surface $S$, Gauss's theorem gives
\[
\int_V dv~\nabla\cdot\vec{A}=\int_Sd\vec{S}\cdot\vec{A}.
\]
For a closed path $C$ which carves out some area $S$,
\[
\int_C d\vec{\ell}\cdot\vec{A}=\int_Sd\vec{s} \cdot(\nabla\times\vec{A})
\]
Stoke's law can be understood by considering a small rectangle, $-\Delta x<x<\Delta x$, $-\Delta y<y<\Delta y$. The path integral around the edges is
\begin{eqnarray}
\int_C d\vec{\ell}\cdot\vec{A}&=&2\Delta y[A_y(\Delta x,0)-A_y(-\Delta x,0)]-2\Delta x[A_x(0,\Delta y)-A_x(0,-\Delta y)]\\
\nonumber
&=&4\Delta x\Delta y\left\{
\frac{A_y(\Delta x,0)-A_y(-\Delta x,0)}{2\Delta x}-\frac{A_x(0,\Delta y)-A_x(0,-\Delta y)}{2\Delta y}\right\}\\
\nonumber
&=&4\Delta x\Delta y\left\{\frac{\partial A_y}{\partial x}-\frac{\partial A_x}{\partial y}\right\}\\
&=&\Delta S \cdot \nabla\times\vec{A}.
\end{eqnarray}
Here $\Delta S$ is the area of the surface element.

\subsubsection*{Some Notation}
From here on in, we may use bold face to denote vectors, e.g. ${\bf v}$ instead of $\vec{v}$. We also might use dots over quantities to represent time derivatives, e.g. $\dot{\bf v}=d{\bf v}/dt$. As mentioned above, repeated indices infer sums, e.g.,
\begin{eqnarray}
x_iy_i&=&\sum_i x_iy_i=\vec{x}\cdot\vec{y},\\
\nonumber
x_iA_{ij}y_j&=&\sum_{ij}x_iA_{ij}y_j,
\end{eqnarray}

\subsection{Exercises}

\begin{enumerate}

\item All physicists must become comfortable with thinking of oscillatory and wave mechanics in terms of expressions that include the form $e^{i\omega t}$.
\begin{enumerate}
\item Perform Taylor expansions in powers of $\omega t$ of the functions $\cos\omega t$ and $\sin\omega t$.
\item Perform a Taylor expansion of $e^{i\omega t}$.
\item Using parts (a) and (b) show that $e^{i\omega t}=\cos\omega t+i\sin\omega t$.
\item Show that $\ln(-1)=i\pi$.
\end{enumerate}

\item One of the many uses of the scalar product is to find the angle between two given vectors. Find the angle between the vectors $\vec{b}=(1,2,4)$ and $\vec{c}=(4,2,1)$ by evaluating their scalar product.

\item Use the product rule to show that
\[
\frac{d}{dt}(\vec{r}\cdot\vec{s})=\frac{d\vec{r}}{dt}\cdot\vec{s}+\vec{r}\cdot\frac{d\vec{s}}{dt}.
\]

\item Multiply the rotation matrix in Example \ref{ex:rotmat} by its transpose to show that the matrix is unitary or orthogonal, i.e. you get the unit matrix.

\item Find the matrix for rotating a coordinate system by 90 degrees about the $x$ axis.

\begin{comment}

\item Consider a rotation where in the new coordinate system the three axes are at angles $\alpha$, $\beta$ and $\gamma$ relative to the original $x$ axis. Show that
\[
\cos^2\alpha+\cos^2\beta+\cos^2\gamma=1.
\]

\item Find the rotation matrix for each of the three rotations, then find the matrix for the combined rotations.
First rotate by 90$^\circ$ around the $y$ axis to get to the primed system. Then rotate the primed system 90$^\circ$ about the new $z$ axis to get to the doubly primed system. Then find the matrix that rotates the doubly primed coordinate system 90$^\circ$ about the new $x$ axis to get to the triply primed system. Finally, find the matrix that performs all three operations sequentially.

\end{comment}

\item Consider a parity transformation which reflects about the $x=0$ plane. Find the matrix that performs the transformation. Find the matrix that performs the inverse transformation.

\item Show that the scalar product of two vectors is unchanged if both undergo the same rotation. Use the fact that the rotation matrix is unitary, $U_{ab}=U^{-1}_{ba}$.

\item Show that the product of two unitary matrices is a unitary matrix.

\item Show that
\[
\sum_k\epsilon_{ijk}\epsilon_{klm}=\delta_{il}\delta_{jm}-\delta_{im}\delta_{jl}.
\]

\item Consider a cubic volume $V=L^3$ defined by $0<x<L$, $0<y<L$ and $0<z<L$. Consider a vector $\vec{A}$ that depends arbitrarily on $x,y,z$. Show how Gauss's law,
\[
\int_V dv\nabla\cdot\vec{A}=\int_Sd\vec{S}\cdot\vec{A},
\]
is satisfied by direct integration. I.e., you should use the fact that $\int_a^b dx~(d/dx)f(x)=f(b)-f(a)$.

\item Consider the function $z=3x^2-4y^2+12xy-6x+24$. Find any maxima or minima and determine whether it is a maximum or a minimum or an inflection point.

\item A real $n-$dimensional symmetric matrix $\lambda$ can always be diagonalized by a unitary transformation, i.e. there exists some unitary matrix $U$ such that,
\begin{eqnarray}
U_{ij}\lambda_{jk}U^{-1}_{km}&=&\tilde{\lambda}_{im}=\left(\begin{array}{cccc}
\tilde{\lambda}_{11}&0&\cdots&0\\
0&\tilde{\lambda}_{22}&\cdots&0\\
\vdots& & \ddots & \vdots\\
0& \cdots &\cdots &\tilde{\lambda}_{nn}
\end{array}\right).
\end{eqnarray}
The values $\tilde{\lambda}_{ii}$ are referred to as eigenvalues. The set of $n$ eigenvalues are unique, but their ordering is not -- there exists a unitary transformation that permutes the indices.

Consider a function $f(x_1,\cdots,x_n)$ that has the property,
\begin{eqnarray}
\left.\partial_i f(\vec{x})\right|_{\vec{x}=0}=0,
\end{eqnarray}
for all $i$. Show that if this function is a minimum, and not a maximum or an inflection point, that the $n$ eigenvalues of the matrix
\begin{eqnarray}
\lambda_{ij}&\equiv&\left.\partial_i\partial_j f(\vec{x})\right|_{\vec{x}=0},
\end{eqnarray}
must be positive.

\item
\begin{enumerate}
\item For the unitary matrix $U$ that diagonalizes $\lambda$ as shown in the previous problem. Show that each row of the unitary matrix represents an orthogonal unit vector by using the definition of a unitary matrix.
\item Show that the vector
\begin{eqnarray}
x_i^{(k)}&\equiv& U_{ki}=(U_{k1},U_{k2},\cdots,U_{kn}),
\end{eqnarray}
has the property that
\begin{eqnarray}
\lambda_{ij}x^{(k)}_j&=&\tilde{\lambda}_{kk}x^{(k)}_i.
\end{eqnarray}
These vectors are known as eigenvectors, as they have the property that when multiplied by $\lambda$ the resulting vector is proportional ( same direction) as the original vector. Because one can transform to a basis, using $U$, where $\lambda$ is diagonalized, in the new basis the eigenvectors are simply the unit vectors.
\end{enumerate}
\end{enumerate}


\subsection{Classic regression trees}

In a classical regression tree, we follow a decison process as before, but the outcome is real number.

Within each leaf, all inputs are assigned that same number.

\subsubsection{Training}

With a regression problem we cannot split nodes the same way as we did for classification.

Instead by split by the residual sum of squares.