\section{Mathematical Symbols and Sets}
\subsection{Logical Statements and their Truth Value}
We start our discussion with the simplest mathematical concept: a \emph{proposition}. A proposition is simply a statement that might be either \true\ or \false.
\begin{example}{Truth of propositions}{}
	\begin{itemize}
		\item $3>1$ (\true)
		\item $-2=5-7$ (\true)
		\item $7<5$ (\false)
		\item The radius of the earth is bigger than that of the moon. (\true)
		\item The word `House' starts with the letter `G'. (\false)
	\end{itemize}
\end{example}

We can group together propositions using \emph{logical operators}. Two of the most common logical operators are \emph{AND} and \emph{OR}.

The \AND{} operator returns a \true\ statement only if \textbf{both} the statements it groups are themselves \true, otherwise it returns \false.

\begin{example}{The AND operator}{}
	\begin{itemize}
		\item $2+4=6$ is \true, $4-2=2$ is \true. $\left(2+4=6\ \text{\AND{}}\ 4-2=2\right)$ is therefore \true.
		\item $2+4=6$ is \true, $2>6$ is \false. $\left(2+4=6\ \text{\AND{}}\ 2>6\right)$ is therefore \false.
		\item $\frac{10}{2}=1$ is \false, $2^{4}=16$ is \true. $\left( \frac{10}{2}=1\ \text{\AND{}}\ 2^{4}=16 \right)$ is therefore \false.
		\item $7<5$ is \false, $10+2=13$ is \false. $\left( 7<5\ \text{\AND{}}\ 10+2=13 \right)$ is therefore \false.
	\end{itemize}
\end{example}

The \OR{} operator returns \true\ if \textbf{at least} one of the statements it groups is true.
\begin{example}{The OR operator}{}
	\begin{itemize}
		\item $2+4=6$ is \true, $4-2=2$ is \true. $\left(2+4=6\ \text{\OR{}}\ 4-2=2\right)$ is therefore \true.
		\item $2+4=6$ is \true, $2>6$ is \false. $\left(2+4=6\ \text{\OR{}}\ 2>6\right)$ is therefore \true.
		\item $\frac{10}{2}=1$ is \false, $2^{4}=16$ is \true. $\left( \frac{10}{2}=1\ \text{\OR{}}\ 2^{4}=16 \right)$ is therefore \true.
		\item $7<5$ is \false, $10+2=13$ is \false. $\left( 7<5\ \text{\OR{}}\ 10+2=13 \right)$ is therefore \false.
	\end{itemize}
\end{example}

The behaviour of both operators can be summarized using a \emph{truth table} (see \autoref{tab:AND_OR_truth_table} below).
\begin{table}
	\centering
	\caption{The truth table for the operators \AND{} and \OR{}.}
	\label{tab:AND_OR_truth_table}
	\begin{tabular}{llll}
		\toprule
		$A$ & $B$ & $A$ AND $B$ & $A$ OR $B$\\
		\midrule
		\true & \true & \true & \true \\
		\true & \false & \false & \true \\
		\false & \true & \false & \true \\
		\false & \false & \false & \false \\
		\midrule
	\end{tabular}
\end{table}

When writing, it is convenient to use \emph{notations} to represent operators: the \AND{} operator is denoted by $\opand$, while the \OR{} operator is denoted by $\opor$.

\begin{example}{Using the notations for \AND{} and \OR{}}{}
	\begin{align*}
		(\falseprop{2+2=5}) &\opand (\trueprop{1-1=0}) \Rightarrow \false\\
		(\falseprop{2+2=5}) &\opor  (\trueprop{1-1=0}) \Rightarrow \true
	\end{align*}
\end{example}

\subsection{Common mathematical notations}

Several more common mathematical notations are given in \autoref{tab:common_math_notations}.

\begin{table}
	\centering
	\caption{Common mathematical notations used in this Book.}
	\label{tab:common_math_notations}
	\begin{tabular}{ll}
		\toprule
		Symbol & In words\\
		\midrule
		$\neg a$ & \textbf{not} $a$\\
		$a \opand b$ & $a$ \textbf{and} $b$\\
		$a \opor b$ & $a$ \textbf{or} $b$\\
		$a \Rightarrow b$ & $a$ \textbf{implies} $b$\\
		$a \Leftrightarrow b$ & $a$ \textbf{is equivalent to} $b$\\
		$\forall x$ & \textbf{For all} $x$ (...)\\
		$\exists x$ & \textbf{There exists} $x$ \textbf{such that} (...)\\
		$a\defeq b$ & $a$ \textbf{is defined to be} $b$\\
		$a\equiv b$ & $a$ \textbf{is equivalent to} $b$\\
		\midrule
	\end{tabular}
\end{table}

The notation $\Rightarrow$ need a bit of clarification: implication means that we can directly derive a proposition from another proposition. For example, if $x=3$ then $x>2$. The opposite implication can be a \false{} statement, i.e. for the example above $x>2$ does not imply $x=3$ (denoted as $x>2 \nRightarrow x=3$). Sometimes implication is expressed by using the word \textit{if}: in the above example $x>2$ if $x=3$, but the other way around is not \true{}.

We say that two propositions are \emph{equivalent} when they imply each other. For example: $x=2$ implies that $\frac{x}{2}=1$, while $\frac{x}{2}=1$ implies that $x=2$. We can write this as
\[
	\frac{x}{2}=1 \Leftrightarrow x=2.
\]
Instead of the word \textit{equivalent}, the phrase \textit{if and only if} (sometimes shortened to \emph{iff}) is commonly used, e.g.
\[
	x=2\ \text{iff}\ \frac{x}{2}=1.
\]

\subsection{Sets and subsets}
The concept of \emph{sets} is perhaps one of the most basic ideas in modern mathematics. Much of the material covered in this book will be built upon sets and their properties. However, as with the rest of the material presented here - our description of sets will not be thorough nor precise.

For our purposes, a set is a collection of \emph{elements}. These elements can be any concept - be it physical (a chair, a bicycle, a tapir) or abstract (a number, an idea). However, we will consider only sets comprised of numbers. Sets can have finite of infinite number of elements in them.

We denote sets by using curly brackets, and if the number of elements in them is not too big - we display the elements, separated by commas, inside the brackets. In other cases we can express the sets as a sentence or a mathematical proposition.

\begin{example}{Simple sets}{}
	\[
		\left\{ 1,2,3,4 \right\}\qquad\left\{ -4,\frac{3}{7},0,\pi,0.13,-2.5,\frac{e}{3},2^{-\pi} \right\}\qquad\left\{ \text{all even numbers} \right\}
	\]
\end{example}

Sets have two important properties:
\begin{enumerate}
	\item Elements in a set do not repeat. i.e each element is unique.
	\item The order of elements in a set does not matter.
\end{enumerate}

\begin{example}{Important set properties}{}
	Examples demonstrating the two aforementioned important properties of sets:
	\begin{enumerate}
		\item The following is not a set:
		\[
			\left\{ 1,1,0,1,0,0,-1,0,0,-1,-1,1 \right\}
	\]

		\item The following sets are all identical:
		\[
			\left\{ 1,2,3,4 \right\}\qquad\left\{ 1,3,2,4  \right\}\qquad\left\{ 3,4,1,2 \right\}\qquad\left\{ 1,3,2,4 \right\}\qquad\left\{ 4,3,2,1 \right\}
		\]
	\end{enumerate}

\end{example}

Sets can be denoted using \emph{conditions}, with the symbol $|$ representing the phrase "such that".

\begin{example}{Defining a set using a condition}
	The following set contains all the odd whole numbers between $0$ and $10$, including both:
	\[
		\left\{ 0 < x < 10 \mid x\ \text{is an odd number}\right\}.
	\]
	The definition of this set can be read as

	\vspace{3mm}
	\centering
	\textit{all numbers $x$ that are bigger than $0$ and are smaller than $10$, such that $x$ is odd.}

	\flushleft{}
	(note that the requirement of $x$ to be an odd number means that it is necessarily a whole number as well)

	\vspace{1em}
	This set can be written explicitly as
	\[
		\left\{ 1,3,5,7,9 \right\}.
	\]
\end{example}

Sets are usually denoted with an uppercase Latin letter ($A,B,C,\dots$), while their elements are denoted as lowercase letters ($a,b,\alpha,\phi,\dots$). When we want to denote that an element belongs to a set we use the following symbol: $\in$. Conversely, $\notin$ is used to denote that an element \textit{does not} belong to a set.

\begin{example}{Elements in sets}{}
	For the two sets
	\[
		A = \left\{ 1,2,5,7 \right\},\quad B=\left\{ \text{even numbers} \right\},
	\]
	all the following propositions are \true{}:
	\begin{align*}
		&1\in A,\quad 2\in A,\quad 5\in A,\quad 7\in A,\\
		&2\in B,\quad 1\notin B,\quad 5\notin B,\quad 7\notin B.
	\end{align*}
\end{example}

The number of elements in a set, also called its \emph{cardinality} is denoted using two vertical bars (similar to the way absolute values are denoted).

\begin{example}{Cardinality}{}
	For $S=\left\{ -3,0,-2,7,1,\frac{1}{2},5 \right\},\ |S|=7$.
\end{example}

An important special set is the \emph{empty set}, which is the set containing no elements. It is denoted by $\emptyset$, and has the unique property that $|\emptyset|=0$.

\subsection{Intersection, union, difference and complement sets}

Two sets are equal if they both contain the exact same elements and only these elements, i.e.
\begin{equation}
	A = B \Longleftrightarrow x\in A \Leftrightarrow x\in B.
	\label{eq:set_equality}
\end{equation}
This proposition reads `The sets $A$ and $B$ are equal \underline{\textit{if and only if}} any element $x$ in $A$ is also in $B$, and any element $x$ in $B$ is also in $A$'. When all the elements of a set $B$ are also elements of another set $A$, we say that $B$ is a \emph{subset} of $A$, and we denote that as $B\subset A$. In mathematical notation, we write

\begin{equation}
	B\subset A \Leftrightarrow \forall x\in B, x\in A.
	\label{eq:subset_def}
\end{equation}
i.e. $B$ is a subset of $A$ \textbf{iff} the following is true: any element in $B$ is also an element in $A$.

\begin{note}{(not so) Surprising properties of subsets}{subset_properties}
	The definition of a subset (\autoref{eq:subset_def}) gives rise to two interesting properties:
	\begin{itemize}
		\item The empty set $\emptyset$ is a subset of any set.
		\item Any set is a  subset of itself.  %perhaps clarification of a proper vs improper subset? Unless that is outside the scope of the work.
	\end{itemize}
\end{note}

\begin{note}{The uniqueness of $\emptyset$}{}
	There is only a single empty set, as any set that has no elements is equivalent to any other set with no elements (i.e. they have the same elements). Due to the way subsets are defined, the empty set is a subset of any set (including itself!).
\end{note}

Of course, since we have a definition for a subset, the opposite concept also exists: if $B$ is a subset of $A$, then we say that $A$ is a \emph{superset} of $B$.

A very useful way of illustrating the relationship between two or more sets is by using \emph{Venn diagrams}, where sets are represented by circles (or other 2D shapes).

\begin{example}{Subsets and Venn diagrams}{}
	A Venn diagram depicting the set $\textcolor{xblue}{B=\left\{ 0,2 \right\}}$ as a subset of $\textcolor{xred}{A=\left\{ 0,1,2,3,4,\dots,9 \right\}}$:
	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\def\firstcircle{(0,0) circle (2)}
			\def\secondcircle{(0.3,0.8) circle (1)}
			\fill[xred!50]\firstcircle;
			\fill[xblue!50]\secondcircle;
			\Large
			\draw \firstcircle node[below left, xshift=-18mm, yshift=11mm] (A) {};
			\draw \secondcircle node[right of=A, xshift=7mm] (B) {};

			% Elements
			\small
			\tikzset{node distance={5mm}}
			\node at (0.57,1.02) {0};
			\node at (-1.04,-1.25) {1};
			\node at (0.16,0.4) {2};
			\node at (-0.1,-0.82) {3};
			\node at (-0.92,-0.49) {4};
			\node at (-1.36,0.06) {5};
			\node at (1.13,-1.16) {6};
			\node at (1.50,0.62) {7};
			\node at (-1.08,1.3) {8};
			\node at (1.32,-0.79) {9};
		\end{tikzpicture}
	\end{figure}
\end{example}

If for two sets $A,B$ both $A\subset B$ and $B\subset A$, then $A=B$. We can write this fact as a mathematical proposition:
\begin{equation}
	(A\subset B) \opand (B\subset A) \Leftrightarrow A=B.
	\label{eq:subset_equal}
\end{equation}

The \emph{intersection} of two sets $A$ and $B$, denoted $A\cup B$, is the set of all elements $x$ such that $x\in A$ \AND{} $x\in B$:
\begin{equation}
	A\cup B = \left\{ x \mid x\in A \opand x\in B \right\}.
	\label{eq:intersection}
\end{equation}

\begin{example}{Intersection of sets}{}
	The intersection of the sets $A=\left\{ 1,2,3,4 \right\}$ and $B=\left\{ 3,4,5,6 \right\}$ is the set $A\cup B=\left\{ 3,4 \right\}$.

	The intersection of the sets $C=\left\{ 0,1,2,6,7 \right\}$ and $D=\left\{ 3,9,-4,5 \right\}$ is the empty set $\emptyset$, since no element is in both sets.
\end{example}

The following Venn diagram depicts the intersection of two sets (the green area):
\begin{figure}
	\centering
	\begin{tikzpicture}
		\def\firstcircle{(0,0) circle (2)}
		\def\secondcircle{(2.3,0) circle (1.5)}
		\fill[xred!50]\firstcircle;
		\fill[xblue!50]\secondcircle;
		\begin{scope}
			\clip \firstcircle;
			\fill[xgreen!50]\secondcircle;
		\end{scope}
		\draw\firstcircle node[left] {$A$};
		\draw\secondcircle node[right] {$B$};
		\draw (1.4,-0.2) node[above] {$A\cap B$};
	\end{tikzpicture}
\end{figure}

\begin{note}{Disjoint sets}{}
	When the intersection of two sets is the empty set, we say that the set is \emph{disjoint}.
\end{note}

The \emph{union} of two sets (denoted using the symbol $\cup$) is the set composed of all the elements that belong to any of the sets, including elements that are in both sets:
\begin{equation}
	A\cup B = \left\{ x \mid x\in A \opor x\in B \right\}.
	\label{eq:union}
\end{equation}

\begin{example}{Union of sets}{}
	The union of the sets $A=\left\{ 1,2,3,4 \right\}$ and $B=\left\{ 3,4,5,6 \right\}$ is the set $A\cup B=\left\{ 1,2,3,4,5,6 \right\}$.

	The union of the sets $C=\left\{ 0,1,2,6,7 \right\}$ and $D=\left\{ 3,9,-4,5 \right\}$ is the set $C\cup D=\left\{ 0,1,2,3,-4,5,6,7,9 \right\}$.
\end{example}

The following Venn diagram depicts the union of two sets (the purple area):
\begin{figure}[H]
	\centering
	\begin{tikzpicture}
		\def\firstcircle{(0,0) circle (2)}
		\def\secondcircle{(2.3,0) circle (1.5)}
		\fill[xpurple!50, draw=black]\firstcircle;
		\fill[xpurple!50, draw=black]\secondcircle;
		\draw\firstcircle node {$A$};
		\draw\secondcircle node {$B$};
		\node[text=xpurple,font=\Large] at (1.5cm,2.5cm) {$A\cup B$};
	\end{tikzpicture}
\end{figure}

Naively, the number of elements of a union $A\cup B$ is simply the sum of the number of elements in $A$ and the number of elements in $B$. However, this naive approach might count the elements in both sets twice: once for $A$ and once for $B$ (see \autoref{fig:union_counting}) - this is exactly the set $A\cap B$. We therefore subtract the number of elements in $A\cap B$ and get
\begin{equation}
	|A\cup B| = |A|+|B|-|A\cap B|.
	\label{eq:number_of_elements_in_union}
\end{equation}

\begin{figure}
	\centering
	\begin{tikzpicture}
		\Large
		\tikzset{node distance=18mm}
		\draw[xred] (-1cm,0) circle (2) node (A) {};
		\draw[xblue] (1cm,0) circle (1.5) node (B) {};
		\node[xred, above of=A, yshift=5mm] {$A$};
		\node[xblue, above of=B]{$B$};

		\def\rdot{0.1}
		% A
		\fill[xred] (-1.15,-0.12) circle (\rdot);
		\fill[xred] (-1.82,-1.01) circle (\rdot);
		\fill[xred] (-1.28,-1.03) circle (\rdot);
		\fill[xred] (-1.68,+\rdot) circle (\rdot);
		\fill[xred] (-0.51,-1.25) circle (\rdot);
		\fill[xred] (-1.41,+0.25) circle (\rdot);
		\fill[xred] (-1.60,+1.25) circle (\rdot);
		%	B
		\fill[xblue] (1.70,0.54) circle (\rdot);
		\fill[xblue] (1.48,0.06) circle (\rdot);
		\fill[xblue] (1.51,-0.94) circle (\rdot);
		%fill[
		\fill[xgreen!75] (-0.16,0.24) circle (\rdot);
		\fill[xgreen!75] (0.13,0.53) circle (\rdot);
		\fill[xgreen!75] (0.06,-0.78) circle (\rdot);
	\end{tikzpicture}
	\caption{Counting the number of elements in the union of two sets: \textcolor{xred}{$A$} has 10 elements (\textcolor{xred}{red} + \textcolor{xgreen}{green} dots), while \textcolor{xblue}{$B$} has 6 elements (\textcolor{xblue}{blue} + \textcolor{xgreen}{green} dots). If we count both we get 16 elements, but this counts the joint elements (\textcolor{xgreen}{green dots}) twice. Therefore we should subtract the number of joint points, and get that there are only 13 elements in the union.}
	\label{fig:union_counting}
\end{figure}

When two sets $A,B$ are disjoint, then $|A\cap B|=0$, and so $|A\cup B| = |A|+|B|$.

The definitions of intersections and unions can be easily extended to any whole number of sets.

\begin{example}{Intersection and union of 3 sets}{}
	The intersection of 3 sets $A=\left\{ 1,2,3,4,5 \right\},\ B=\left\{ -2,-1,0,1,2 \right\}$ and $C=\left\{ 2,3,4,5,6 \right\}$ is the set of all elements that are in $A$ and in $B$ and in $C$, i.e.\ the set $A\cap B\cap C = \left\{2\right\}$.

	The union of these sets is the set of all elements that are in either of the sets, i.e. $A\cup B\cup C = \left\{ -2,-1,0,1,2,3,4,5,6 \right\}$.
\end{example}

The most general definition of an intersection of $n$ sets (where $n$ is a whole number), which we will call $A_{1},A_{2},A_{3},\cdots,A_{n}$ is
\begin{equation}
	A_{1}\cap A_{2}\cap A_{3}\cap \cdots \cap A_{n} = \left\{ x \mid (x\in A_{1}) \opand (x\in A_{2}) \opand (x\in A_{3}) \opand \cdots \opand (x \in A_{n})\right\}.
	\label{eq:intersection_n_sets}
\end{equation}

The left hand side of \autoref{eq:intersection_n_sets} can be written as
\begin{equation}
	A_{1}\cap A_{2}\cap A_{3}\cap \cdots \cap A_{n} = \bigcap\limits_{i=1}^{n}A_{i}.
	\label{eq:intersection_n_sets_big_notation}
\end{equation}
(clarifying the notation? i.e. indexing, etc.)

Similarly, the union of $n$ different sets is defined as
\begin{align}
	\bigcup\limits_{i=1}^{n}A_{i} &= A_{1}\cup A_{2}\cup A_{3}\cup \cdots \cup A_{n}\nonumber\\
	&= \left\{ x \mid (x\in A_{1}) \opor (x\in A_{2}) \opor (x\in A_{3}) \opor \cdots \opor (x\in A_{n})\right\}.
	\label{eq:union_n_sets}
\end{align}

\begin{example}{Venn diagrams: intersection and union of 3 sets}{}
	The following Venn diagram shows all possible intersections between three sets:
	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\tikzset{
				every node/.style={text=black, text opacity=1},
			}
			\def\sq{1.15}
			\Large
			\begin{scope}[blend group=hard light]
				\fill[xred!40!white] (-\sq,\sq) circle (2) node (A) {$A$};
				\fill[xblue!40!white] (\sq,\sq) circle (2) node (B) {$B$};
				\fill[xgreen!40!white] (0,-\sq) circle (2) node (C) {$C$};
			\end{scope}
			\footnotesize
			\node[yshift=5mm, rotate=90] at ($(A)!0.5!(B)$) {$A\cap B$};
			\node[xshift=-4mm, yshift=-2mm, rotate=+30] at ($(A)!0.5!(C)$) {$A\cap C$};
			\node[xshift=+4mm, yshift=-2mm, rotate=-30] at ($(B)!0.5!(C)$) {$B\cap C$};
			\node[align=center] at (0,4mm) {$A\cap B\cap C$};

		\end{tikzpicture}
	\end{figure}

	\vspace{2em}
	...and the following Venn diagram depicts the union of the same three sets:
	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\tikzset{
				every node/.style={text=black, text opacity=1},
			}
			\def\sq{1.15}
			\Large
			\fill[xpurple!40!white] (-\sq,+\sq) circle (2) node (A) {$A$};
			\fill[xpurple!40!white] (\sq,+\sq) circle (2) node (B) {$B$};
			\fill[xpurple!40!white] (0,-\sq) circle (2) node (C) {$C$};

			\node at (0,0) {$A\cup B\cup C$};
		\end{tikzpicture}
	\end{figure}
\end{example}

The \emph{difference} of two sets $A$ and $B$ (written $A-B$ or $A\setminus B$) is, in a sense, the opposite of their intersection: it is the set of all elements in $A$ that are \underline{not} in $B$. Note that $A-B$ doesn't necessarily equal $B-A$, i.e. it is not \emph{commutative}.

\begin{example}{Difference of two sets}{}
	Given the two sets $A=\{1,2,3,4,5\}$ and $B=\{3,4,5,6,7,8,9\}$,
	\begin{align*}
		A-B &= \{1,2\},\\
		B-A &= \{6,7,8,9\}.
	\end{align*}
	(note how in this case $A-B\neq B-A$)
\end{example}

Given a set $A$ and a subset of $A$, $B\subset A$, we can define the \emph{complement} of $B$ in relation to $A$ (notation: $\stcomp{B}$) as all the elements in $A$ that are \underline{not} in $B$. As the name suggest, the elements of $\stcomp{B}$ complete $B$: $B\cup\stcomp{B}=A$.

\begin{example}{Complement of a set}{}
	Given the set $A=\mathbb{Z}$ and $B=\{x\in \mathbb{Z} \mid x\ \text{is odd}\}$, the complement $\stcomp{B}$ in relation to $A$ is the set of all even numbers. The reason is that any integer number can be either odd (in which case it belongs in $B$) or even (in which case it belongs in $\stcomp{B}$).
\end{example}

Given a set $A$ with $|A|$ elements - how many different subsets does it have? We'll start by looking at a practical example: $A=\left\{ 1,2,3 \right\}$. We can imminently see that any set which contains just one of the elements of $A$ is a subset of $A$, i.e. $\{1\},\{2\},\{3\}$ are all subsets of $A$. In addition, any set which contains only two elements from $A$ is a subset of $A$, i.e. $\left\{ 1,2 \right\}, \left\{ 1,3 \right\}, \left\{ 2,3 \right\}$. Of course, we must not forget the empty set and $A$ itself - both subsets of $A$ (see \autoref{note:subset_properties}). Thus altogether $A$ has $8$ subsets:
\[
	\emptyset, \left\{ 1 \right\}, \left\{ 2 \right\}, \left\{ 3 \right\}, \left\{ 1,2 \right\}, \left\{ 1,3 \right\}, \left\{ 2,3 \right\}, \left\{ 1,2,3 \right\}.
\]

Generally, any set $A$ with $|A|$ elements has $2^{|A|}$ different subsets. The set of all these subsets is called the \emph{power set} of $A$, and is denoted as $P(A)$.

\begin{example}{Power set}{power_set}
	The power set of $A=\left\{ 1,2,3 \right\}$ is
	\[
		P(A) = \left\{ \emptyset, \left\{ 1 \right\}, \left\{ 2 \right\}, \left\{ 3 \right\}, \left\{ 1,2 \right\}, \left\{ 1,3 \right\}, \left\{ 2,3 \right\}, \left\{ 1,2,3 \right\}\right\}.
	\]
\end{example}

\subsection{Important number sets}
It is now time to introduce some important number sets. We begin with the simplest of these sets: the \emph{natural numbers}, denoted by $\mathbb{N}$. These are the numbers $1,2,3,4,\dots$. Adding the opposites to the natural numbers and adding $0$ to the set yields the \emph{integers}, denoted by $\mathbb{Z}$. Loosely speaking, we can define the integers as
\begin{equation}
	\mathbb{Z}=\left\{ 0,\pm1,\pm2,\pm3,\pm4,\dots \right\}.
	\label{eq:integers}
\end{equation}

This makes the integers a superset of the natural numbers, i.e.
\begin{equation}
	\mathbb{N} \subset \mathbb{Z}.
	\label{eq:naturals_subset_integers}
\end{equation}

One can think of the integers as all the number needed for solving an equation of the form $a+x=b$, where $a$ and $b$ are integers themselves, and $x$ is an unknown. No matter which integer values we put in $a$ and $b$, the unknown $x$ will always be an integer as well (whether it be positive, negative or zero depends on the values of $a$ and $b$). However, when one wishes to solve an equation of the sort $ax=b$, the integers are not longer sufficient: for example, if $a=2$ and $b=1$, then $x$ is not an integer.

To solve $ax=b$ (where $a,b\in\mathbb{Z}$) we must introduce the \emph{rational numbers}: numbers with values that are ratios of two integers. We denote the set of rational numbers with the symbol $\mathbb{Q}$, and write
\begin{equation}
	\mathbb{Q} = \left\{ \frac{a}{b} \ \middle\vert\ a,b\in\mathbb{Z} \opand b\neq0  \right\}.
	\label{eq:rationals}
\end{equation}
(TBW: discuss briefly why $b\neq0$)

For some combinations of $a$ and $b$ the ratio $\frac{a}{b}$ is an integer. For example: $\frac{3}{1},\ \frac{8}{4},\ \frac{-2}{2}$. This makes the integers a subset of the rational numbers, i.e.
\begin{equation}
	\mathbb{Z}\subset\mathbb{Q}.
	\label{eq:integers_subset_rationals}
\end{equation}

About 2500 years ago it was discovered that some numbers are not rational (and thus also not integers). The most famous example is the number $\sqrt{2}$ - there are not two integers $a,b$ such that $\frac{a}{b}=\sqrt{2}$. We call some of these numbers \emph{algebraic numbers} (denoted by $\mathbb{A}$), and what makes them special is that they are solutions to \emph{polynomial equations}, which we will not define yet (see section xxx). Instead, here is an example for a 2nd order polynomial equation (called a \emph{quadratic equation}):
\begin{equation}
	x^{2} - 2x - 1 = 0.
	\label{eq:quadratic_equation}
\end{equation}

Similar to what we saw before, the rational numbers are a subset of the algebraic numbers, i.e.
\begin{equation}
	\mathbb{Q}\subset\mathbb{A}.
	\label{eq:rationals_subset_algebraic}
\end{equation}

The algebraic numbers together with other non-rational numbers, such as $\pi$ and $e$, form the set of \emph{real numbers}, denoted as $\mathbb{R}$. The definition of real numbers is way beyond the scope of this book, but it is important to understand that the progression we used so far still holds, i.e.
\begin{equation}
	\mathbb{A}\subset\mathbb{R}.
	\label{eq:algebraic_subset_reals}
\end{equation}

The final set of numbers we will touch upon here is the set of \emph{complex numbers}, denoted $\mathbb{C}$, which we can define as
\begin{equation}
	\mathbb{C} = \left\{ a+\iu b \mid a,b\in\mathbb{R},\ \iu=\sqrt{-1} \right\}.
	\label{eq:complex_numbers_def}
\end{equation}

When $b=0$, \autoref{eq:complex_numbers_def} becomes just a single real number - and so
\begin{equation}
	\mathbb{R}\subset\mathbb{C}.
	\label{eq:reals_subset_complex}
\end{equation}
(\autoref{sec:complex numbers} is dedicated to complex numbers)

Equations \ref{eq:naturals_subset_integers}-\ref{eq:reals_subset_complex} can be merged together to the following single equation:
\begin{equation}
	\mathbb{N} \subset \mathbb{Z} \subset \mathbb{Q} \subset \mathbb{A} \subset \mathbb{R} \subset \mathbb{C}.
	\label{eq:numbers_subsets}
\end{equation}

There are more advanced constructions that generalize the complex numbers (i.e. create supersets of the complex number set). These include \emph{quaternions} and \emph{Clifford algebras}. However, as stated before, we will not consider them in this book.

\subsection{Intervals on the real number line}
An important concept that is easily defined over the set $\mathbb{R}$ is an \emph{interval}. A \emph{closed interval} $\left[ a,b \right]$ is a subset of $\mathbb{R}$ which is defined as
\begin{equation}
	\left[ a,b \right] = \left\{ x\in\mathbb{R} \mid a\leq x\leq b \right\}.
	\label{eq:closed_interval}
\end{equation}

An \emph{open interval} $\left( a,b \right)$ is a subset of $\mathbb{R}$ which is defined as
\begin{equation}
	\left( a,b \right) = \left\{ x\in\mathbb{R} \mid a<x<b \right\}.
	\label{eq:open_interval}
\end{equation}

The difference between closed and open intervals is the inclusion and exclusion, respectively, of the edge point: in a closed interval the points $a,b$ are included, while they are not included in an open interval. Of course, we can also create \emph{half open intervals}, i.e.
\begin{align}
	\left[ a,b \right) &= \left\{ x\in\mathbb{R} \mid a\leq x < b \right\},\nonumber\\
	\left( a,b \right] &= \left\{ x\in\mathbb{R} \mid a < x\leq b \right\},
	\label{eq:half_open_intervals}
\end{align}
where the first interval includes $a$ but not $b$, and the second interval includes $b$ but not $a$.

\vspace{2em}
\begin{example}{Intervals}{intervals}
	Intervals can be drawn as colored line segments on top of the real number line:
	\begin{figure}[H]
		\centering
		\begin{tikzpicture}[node distance=15mm]
			\begin{axis}[Interval]
				% NOTE: there must be a way to set this globally!
				\addplot[xred] table[y expr=4, meta index=1, header=false] {
				-2 c
				3 c
				} node [right] {$\left[ -2,3 \right]$};
				\addplot[xblue] table[y expr=3, meta index=1, header=false] {
				-4 o
				0 o
				} node [right] {$\left( -4,0 \right)$};
				\addplot[xgreen] table[y expr=2, meta index=1, header=false] {
				-1 c
				4 o
				} node [right] {$\left[ -1,4 \right)$};
				\addplot[xpurple] table[y expr=1, meta index=1, header=false] {
				1 o
				3 c
				} node [right] {$\left( 1,3 \right]$};
			\end{axis}
		\end{tikzpicture}
	\end{figure}

	Note how a full point denotes a closed edge, while an empty point denotes an open edge.
\end{example}

In some cases, it is necessary to use intervals that are infinite in one side, i.e. the left or the right edge are at infinity. In these cases, we use the symbol $\infty$ to denote infinity, and always keep the interval open at that end:
\begin{align}
	\left( -\infty,b \right) &= \left\{ x \mid x<b \right\},\nonumber\\
	\left( -\infty,b \right] &= \left\{ x \mid x\leq b \right\},\nonumber\\
	\left( a,\infty \right) &= \left\{ x \mid x>a \right\},\nonumber\\
	\left[ a,\infty \right) &= \left\{ x \mid x\geq a \right\}.
	\label{eq:infinite_intervals}
\end{align}

\subsection{Cartesian Products}
The \emph{Cartesian product}\index{Cartesian product} of two sets $A,B$ (denoted $A\times B$) is the set of all possible \emph{ordered} pairs, where the first component is an element of $A$ and the second component is an element of $B$:
\begin{equation}
	A\times B = \left\{ (a,b) \mid a\in A,\ b\in B \right\}.
	\label{eq:Cartesian_product}
\end{equation}

\begin{example}{Cartesian products}{Cartesian_products}
	Consider $A=\left\{ 1,2,3 \right\},\ B=\left\{ x, y \right\}$. Then
	\[
		A\times B = \left\{ \left( 1,x \right),\ \left( 1,y \right),\ \left( 2,x \right),\ \left( 2,y \right),\ \left( 3,x \right),\ \left( 3,y \right) \right\}
	\]
\end{example}

The concept of `ordered pairs' is paramount: if we reverse the order of the elements in a pair the result might not be in the Cartesian product. We therefore say that the Cartesian product is \emph{not commutative}.

\begin{example}{Non-commutativity of the Cartesian product}{}
	The elements $(x,1),\ (y,1),\ (x,2)$ and so on \textbf{are not} in the Cartesian product $A\times B$ as defined in the previous example, since in each one of the pairs the first element is from $B$ and the second element is from $A$.
\end{example}

The number of elements in a Cartesian product is the product of the number of elements in each of the sets it is composed of, i.e.
\begin{equation}
	|A\times B| = |A|\cdot|B|.
	\label{eq:number_of_elements_Cartesian_product}
\end{equation}

\begin{example}{Number of elements in a Cartesian product}{}
	The Cartesian product described in the previous two examples has in total $3\cdot2=6$ elements, as seen in \autoref{:Cartesian_products}.
\end{example}

As with intersections and unions, the definition of a Cartesian product can be expanded into any natural number of sets:
\begin{equation}
	A_{1}\times A_{2}\times \cdots \times A_{n} = \left\{ \left( x_{1},x_{2},\dots,x_{n} \right) \mid x_{1}\in A_{1},\ x_{2}\in A_{2},\ \dots, x_{n}\in A_{n} \right\}.
	\label{eq:Cartesian_product_multiple_sets}
\end{equation}

\begin{example}{Cartesian product of three sets}{}
	The Cartesian product of the sets $A=\{1,2,3\},\ B=\{x,y\},\ C=\{\alpha,\beta\}$ is
	\begin{align*}
		A\times B\times C = \{
			&(1,x,\alpha),\ (1,x,\beta),\ (1,y,\alpha),\ (1,y,\beta)\\
			&(2,x,\alpha),\ (2,x,\beta),\ (2,y,\alpha),\ (2,y,\beta)\\
			&(3,x,\alpha),\ (3,x,\beta),\ (3,y,\alpha),\ (3,y,\beta)
		\}.
	\end{align*}
\end{example}

An element in a cartesian product is sometimes called a \emph{tuple}. A tuple can be tought of as a set where the order of elements does matter, and thus elements can repeat. A tuple with $n$-elements is called an $n$-tuple.

\begin{example}{Tuples}{}
	\centering
	\begin{tabular}{lll}
		\toprule
		$n$ & Name & Example\\
		\midrule
		$0$  & empty tuple & $()$\\
		$1$  & monpule	 & $(-5)$\\
		$2$  & couple	 & $(-5,-3)$\\
		$3$  & triple	 & $(-7,-2,8)$\\
		$4$  & quadruple & $(1,3,3,7)$\\
		$5$  & quintuple & $(-8,8,-9,-5,8)$\\
		$6$  & sextuple  & $(-7,7,1,1,0,-5)$\\
		$7$  & septuple  & $(5,8,0,-8,-3,7,-7)$\\
		$8$  & octuple	 & $(-9,8,5,-5,2,-5,-1,2)$\\
		$9$  & nonuple	 & $(2,4,-3,-1,5,-1,-5,-4,-7)$\\
		$10$ & decuple	 & $(2,-3,7,2,3,-7,0,-1,-9,8)$\\
		$\vdots$ & $\vdots$ & $\vdots$\\
		\bottomrule
	\end{tabular}
\end{example}

A special case of Cartesian products are those products for which all the sets composing them are the same set. We denote these as the respective integer power, for example the Cartesian product $\mathbb{R}\times\mathbb{R}$ is denoted as $\Rs{2}$, the Cartesian product $\mathbb{R}\times\mathbb{R}\times\mathbb{R}$ is denoted as $\Rs{3}$, etc.

Specifically, the Cartesian product $\Rs{2}$ can be interpreted as the two-dimensional \emph{Euclidean space}, which is the space used to draw graphs in one-dimensional calculus and shapes in two-dimensional analytical geometry. We will explore this idea (and higher dimensional spaces) in more details in upcoming chapters.


\subsection{$u$ Substitution (Reversing the Chain Rule)}
The chain rule tells us that $\dd{}{x}f(g(x)) = f^\prime(g(x))g^\prime(x)$.
So, if we see that we are integrating a function that contains within it a function and its derivative, we might want to try running the chain rule in reverse.
\begin{example}
	Solve the following indefinite integral:
	\begin{equation*}
		\int{x^2\cdot 2x\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	Since we know that $\dd{x^2}{x} = 2x$, or in differential form $\d{x^2} = 2x\d{x}$, we can make a substitution.
	Letting $u = x^2$,
	\begin{align*}
		\int{x^2\cdot 2x\d{x}} &= \int{u\d{u}} \\
		&= \frac{u^2}{2} + C \\
		&= \frac{x^4}{2} + C.
	\end{align*}
\end{answer}

Sometimes the derivative itself won't be there, but a constant multiple of the derivative.
\begin{example}
	Solve the following indefinite integral:
	\begin{equation*}
		\int{\frac{x^2}{1-x^3}\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	A good way to spot that there might be the possibility of a $u$-substitution here is the fact that the denominator is a polynomial of one higher degree than the numerator.
	So, if we took the derivative of the denominator, we'd get a polynomial of the same degree as the numerator.
	Let $u = 1-x^3$.
	Then $\d{u} = -3x^2\d{x}$; $x^2\d{x} = -\d{u}/3$.
	\begin{align*}
		\int{\frac{x^2}{1-x^3}\d{x}} &= \int{\frac{-\d{u}}{u}} \\
		&= -\ln{\abs{u}} + C \\
		&= -\ln{\abs{1-x^3}} + C.
	\end{align*}
\end{answer}

We can also apply $u$-substitutions to definite integrals.
We'll need to change the bounds of integration to be in terms of $u$ before we evaluate.
\begin{example}
	Solve the following definite integral:
	\begin{equation*}
		\int_{0}^{1}{15x^2\sqrt{5x^3+4}\d{x}}.
	\end{equation*}
\end{example}
\begin{answer}
	Let $u = 5x^3 + 4$.
	Then $\d{u} = 15x^2\d{x}$.
	When $x=1$, $u=5(0)^3 + 4 = 4$; when $x=1$, $u = 5(1)^3  +4 = 9$.
	\begin{align*}
		\int_{0}^{1}{15x^2\sqrt{5x^3+4}\d{x}} &= \int_{4}^{9}{\sqrt{u}\d{u}} \\
		&= \frac{2u\sqrt{u}}{3} \biggr\rvert_{4}^{9} \\
		&= \frac{2(9)\sqrt{9}}{3} - \frac{2(4)\sqrt{4}}{3} \\
		&= \frac{54}{3} - \frac{16}{3} \\
		&= \frac{38}{3}.
	\end{align*}
\end{answer}

\subsubsection{Trig Functions}
We can get the integrals of some of the trig functions by simply applying the derivatives for trig functions in reverse.
\begin{table}[H]
	\begin{center}
		\begin{tabular}{ l l }
			$\begin{aligned}\int{\sin{x}\d{x}}=-\cos{x} + C\end{aligned}$ & $\begin{aligned}\int{\cos{x}\d{x}}=\sin{x}+C\end{aligned}$
		\end{tabular}
	\end{center}
\end{table}

We can get the integrals of $\tan{x}$ and $\cot{x}$ using basic $u$-substitutions.
\begin{align*}
	\int{\tan{x}\d{x}} &= \int{\frac{\sin{x}}{\cos{x}}\d{x}} \\
	u = \cos{x} &\text{ and } \d{u} = -\sin{x}\d{x} \\
	&= \int{\frac{-\d{u}}{u}} \\
	&= -\ln{\abs{u}} + C \\
	&= -\ln{\abs{\cos{x}}} + C.
\end{align*}
\begin{align*}
	\int{\cot{x}\d{x}} &= \int{\frac{\cos{x}}{\sin{x}}\d{x}} \\
	u = \sin{x} &\text{ and } \d{u} = \cos{x}\d{x} \\
	&= \int{\frac{\d{u}}{u}} \\
	&= \ln{\abs{u}} + C \\
	&= \ln{\abs{\sin{x}}} + C.
\end{align*}

We can also get the integrals of $\sec{x}$ and $\csc{x}$ using $u$-substitutions, but we have to be a bit more clever and multiply the integrand by a strange-looking fraction that equals 1.
\begin{align*}
	\int{\sec{x}\d{x}} &= \int{\sec{x}\frac{\sec{x} + \tan{x}}{\tan{x}+\sec{x}}\d{x}} \\
	&= \int{\frac{\sec^2{x}+\sec{x}\tan{x}}{\tan{x}+\sec{x}}} \\
	u = \tan{x} + \sec{x} &\text{ and } \d{u} = (\sec^2{x} + \sec{x}\tan{x})\d{x} \\
	&= \int{\frac{\d{u}}{u}} \\
	&= \ln{\abs{u}} + C \\
	&= \ln{\abs{\tan{x} + \sec{x}}} + C.
\end{align*}
\begin{align*}
	\int{\csc{x}\d{x}} &= \int{\csc{x}\frac{\csc{x}+\cot{x}}{\cot{x}+\csc{x}}\d{x}} \\
	&= \int{\frac{\csc^2{x}+\csc{x}\cot{x}}{\cot{x}+\csc{x}}\d{x}} \\
	u = \cot{x} + \csc{x} &\text{ and } \d{u} = -(\csc^2{x}+\csc{x}\cot{x})\d{x} \\
	&= \int{\frac{-\d{u}}{u}} \\
	&= -\ln{\abs{u}} + C \\
	&= -\ln{\abs{\cot{x}+\csc{x}}} + C.
\end{align*}


\section{Code listing}
The below code from my Text Simlarity assignment in the Data Science class \parencite{dsc-text-similarity}.\\

\begin{lstlisting}[language=Python]
# # Vectorisation of Text Data
# The process of converting or transforming a data set into a set of Vectors is called Vectorization.

# import libraries
import pandas as pd
import sklearn as sk
import math

# Calculate dot product of two vectors, divide it by the magnitudes to find the cos(angle between them)
# Use the result as a correlation coefficient
from collections import Counter

def cosine(vector1, vector2):
     # calculate nominator as a dot product
     intersect = set(vector1.keys()) & set(vector2.keys())
     numerator = sum([vector1[x] * vector2[x] for x in intersect])

     # calculate the denominator
     sum1 = sum([vector1[x] ** 2 for x in list(vector1.keys())])
     sum2 = sum([vector2[x] ** 2 for x in list(vector2.keys())])

     denominator = math.sqrt(sum1) * math.sqrt(sum2)
     if not denominator:
         return 0.0
     else:
         return float(numerator)/denominator

# # Assignment
with open('textfiles/A.txt', 'r', encoding='utf8', errors='ignore') as a:
    A = a.read().replace('\n', '')
with open('textfiles/B.txt', 'r', encoding='utf8', errors='ignore') as b:
    B = b.read().replace('\n', '')
with open('textfiles/C.txt', 'r', encoding='utf8', errors='ignore') as c:
    C = c.read().replace('\n', '')

A1 = A.split(" ")
B1 = B.split(" ")
C1 = C.split(" ")

# join the sets of words to remove duplications
all= set(A1).union(set(B1).union(C1))
#print(all)

def convertTextToVector(text):
    x = dict.fromkeys(all, 0)
    for word in text:
        x[word]+=1
    return x

def compareVectors():
    AVector = convertTextToVector(A1)
    BVector = convertTextToVector(B1)
    CVector = convertTextToVector(C1)

    corrAB = cosine(AVector, BVector)
    print("Similarity on A and B: ", corrAB)

    corrAC = cosine(AVector, CVector)
    print("Similarity on A and C: ", corrAC)

    corrBC = cosine(BVector, CVector)
    print("Similarity on B and C: ", corrBC)

    suggestion = ""

    highest = 0
    corrarray = [corrAB, corrAC, corrBC]
    for i in corrarray:
        if highest < i:
            highest = i

    if highest == corrAB:
        suggestion = "Text A = X, Text B = X, Text C = Y"
    elif highest == corrAC:
        suggestion = "Text A = X, Text B = Y, Text C = X"
    else:
        suggestion = "Text A = Y, Text B = X, Text C = X"
    return suggestion

suggestion = compareVectors()
print(suggestion)
\end{lstlisting}


\subsection{Least Absolute Shrinkage and Selection Operator (LASSO)}

\subsubsection{Introduction}

With LASSO we add a constraint to \(\hat \theta \).

\(\sum_i \hat \theta_i \le t\)

Regularisation of LLS. Sum of thetas are constrained to be below hyperparameter \(t\)

L1 regularisation

This is also known as sparce regression, because many weights are set to \(0\).

This now looks like:

\(w_{lasso} = \arg \min ||y-Xw||^2_2+\lambda ||w||_1\)

\subsubsection{Hyperparameter}

\(t\) is a hyperparameter.


\section{Piecewise linear functions on multilevel
  grids in 2D}\label{sec:functions}
An image can be viewed as a function on a grid.  Images with different
resolutions can then be viewed as functions on grids of different
sizes.  The use of such multiple-grids is a main technique used in the
standard multigrid method for solving discretized partial differential
equations, and it can also be interpreted as a main ingredient used in
convolutional neural networks (CNN) for image calssification.

An image can be viewed as a function on a grid \cite{krizhevsky2012imagenet} on
a rectangle  domain $\Omega\in \mathcal R^2$.  Without loss of generality,
 we assume that the grid, $\mathcal T$, is of size
$$
m=2^{s}+1~~~n=2^{t}+1
$$
for some integers $s, t\ge 1$.
Starting from $\mathcal T_1=\mathcal T$,  we consider a sequence of
coarse grids with $J=\min (s,t)$ (as depicted in Fig.~\ref{mgrid} with $J=4$):
\begin{equation}
\label{grids}
\mathcal T_1, \mathcal T_2, \ldots, \mathcal T_J
\end{equation}
such that ${\cal T}_\ell$ consist of $m_\ell\times n_\ell$ grid
points, with
\begin{equation}
\label{mn-ell}
 m_\ell=2^{s-\ell+1}+1,~~ n_\ell=2^{t-\ell+1}+1.
\end{equation}
\begin{figure}[!htbp]\label{mgrid}
	\begin{center}
		\includegraphics[width=0.15\textwidth]{grid2.png} \quad
		\includegraphics[width=0.15\textwidth]{grid1.png} \quad
		\includegraphics[width=0.15\textwidth]{grid0.png} \quad
		\includegraphics[width=0.15\textwidth]{grid.png}
	\end{center}
	$$
\hskip0.05 in \mathcal T_1\hskip 0.7in \mathcal T_2\hskip 0.7in  \mathcal T_3\hskip 0.7in \mathcal T_4
	$$
	\caption{multilevel grids for piecewise linear functions}
\end{figure}

The grid points of these grids can be given by
$$
x_i^{\ell}=i h_{\ell}, y_j^{\ell}=j h_{\ell},  i=0, \ldots, m_\ell-1,
j=0, \ldots, n_\ell-1.
$$
Here $h_{\ell} = 2^{-s + \ell -1}a$ for some $a >0$. The above geometric coordinates $(x_i^\ell, y_j^\ell)$
are usually not used in image precess literatures, but they are relevant
in the context of multigrid method for numerical solution of PDEs.
We now consider piecewise bilinear (or linear) functions on the sequence of grids
\eqref{grids} and we obtain a nested sequence of linear vector spaces
\begin{equation}
\label{Vk}
\mathcal V_1\supset\mathcal V_2\supset\ldots\supset \mathcal
V_J.
\end{equation}

497-6DL/8 Convolutional Multigrid Method/8.8-2DFEM.tex


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS671: Machine Learning
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

What is the Vapnik-Chervonenkis dimension of the class of rhombi defined above?

\subsection*{Solution}

Let $\mathcal{R}$ be the set of all rhombi with ratio $c/d$.
We claim that the Vapnik-Chervonenkis Dimension  (VCD) of $\mathcal{R}$ is two.
To support this claim, we first show that any set $ S = \{x, y\}$ where $x y \in \mathbb{R}^2$ can be shattered by rhombi of ratio $c/d$.
As well, we need to show that there is a set $S = \{x_1, x_2, x_3\}$ impossible to be shattered by $\mathcal{R}$.

\begin{enumerate}
\item VCD($\mathcal{R}$) is at least one, because for any point $P(x,y)$ where $x, y \in \mathbb{R}$, we can choose a rhombus with center $(0,0)$ and diagonals $2c$ and $2d$ where $min\{c,d\} > (x^2 + y^2)$ to enclose the point.
As well, we can be sure that any rhombus with $min\{c,d\} < (x^2 + y^2)$ does not contain $P(x,y)$.
Thus $\mathcal{R}$ shatters sets of size 1.

\item VCD($\mathcal{R}$) is at least two, because for any two points $P_1(x_1,y_1)$ and $P_2(x_2,y_2)$, there exists $\epsilon \in \mathbb{R}$ such that a rhombus with a center $P_s(x,y)$ and diagonals of length $x+\epsilon$ and $y+\epsilon$ can enclose $P_s$ but no other point.

As well, diagonals $2c_0$ and $2d_0$ can be found such that $min\{c,d\} > max\{x_1,x_2\}^2 + max\{y_1, y_2\}^2$ to make sure a rhombus with center $(0,0)$ and diagonals $2c > 2c_0$ and $2d > 2d_0$ will contain both $P_1(x_1,y_1)$ and $P_2(x_2,y_2)$.

\item VCD($\mathcal{R}$) is not three because three points $P_1(x_1,y_1)$, $P_2(x_2,y_2)$ and $P_3(x_3,y_3)$ can be given such that no rhombus with ratio $c/d$ can shatter them.
To show this point, take $P_1(x_1,y_1)$ and $P_2(x_2,y_2)$ at any two arbitrary points $P_1 \neq P_2$.

It is always possible to construct a rhombus with ratio $c/d$ such that $P_1$ and $P_2$ are on its edges.
If $P_3$ now be chosen inside this rhombus, no rhombus with fixed ratio of $c/d$ can shatter these points anymore.
Mathematically, any choice of $P_3(x_3,y_3)$ such that $x_1 < x_3 < x_2$ and $max\{y_3/y_1, y_3/y_2\} < c/d$ will prevent $\mathcal{R}$ to shatter the three points.
\end{enumerate}

Therefore Vapnik-Chervonenkis Dimension of $\mathcal{R}$ is only 2.


\include{config/config}

\begin{document}
% ========== Edit your name here
\title{MATH 2901 Basic Probability Lecture Notes 3}
\author{Instructor: Richard Kleeman}
\date{}
\maketitle

%\medskip

% ========== Contents begin here ==============
\section{Probability mass functions}
\begin{definition}
The \textbf{(probability) mass function} (p.m.f.) of a discrete random variable $X$ is the function $f: \R \to[0, 1]$ given by $f(x) = \Prob(X = x)$.
\end{definition}

Notice that $f(x)$ is not discontinuous. The distribution and mass functions are related by
\begin{equation*}
    F(x) = \sum_{i: x_i \leq x} f(x_i), \quad f(x) = F(x) - \lim_{y\uparrow x} F(y).
\end{equation*}

\begin{lemma}
The probability mass function $f: \R \to [0, 1]$ satisfies:
\begin{enumerate}[(a)]
    \item the set of $x$ such that $f(x) \neq 0$ is \textbf{countable},
    \item $\sum_{i} f(x_i) = 1$, where $x_1, x_2, \dots$ are the values of $x$ such that $f(x)\neq 0$.
\end{enumerate}
\end{lemma}

\begin{example}
\textbf{Binomial distribution.} A coin is tossed $n$ times, and a head turns up each time with probability $p (= 1 - q)$. Then $\Omega = \{H, T\}^n \}$. The total number $X$ of heads takes values in the set $\{0, 1, 2, \dots , n\}$ and is a discrete random variable. Its probability mass function $f(x) = \Prob(X = x)$ satisfies
\begin{equation*}
    f(k)= \binom{n}{k} p^{k}(1-p)^{n-k}, \quad k=0,1, \ldots, n.
\end{equation*}
The random variable $X$ is said to have the \textbf{binomial distribution} with parameters $n$ and $p$, It is the sum $X = Y_1 + Y_2 + \dots + Y_n$ of $n$ Bernoulli variables.
\end{example}

\begin{example}
\textbf{Poisson distribution.} If a random variable $X$ takes values in the set $\{O, 1, 2, \dots \}$ with mass function
\begin{equation*}
    f(k) = \frac{\lambda^k}{k!} e^{-\lambda}, \quad k = 0, 1, 2, \dots,
\end{equation*}
where $\lambda > 0$, then $X$ is said to have the \textbf{Poisson distribution} with parameter $\lambda$. Figure \ref{fig:poisson} shows how p.m.f varies with $k$.
\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.18]{plots/poisson.png}
    \caption{p.m.f of Poisson distribution}
    \label{fig:poisson}
\end{figure}

\begin{remark}
The interpretation of Poisson distribution can be found via this link: \href{https://towardsdatascience.com/the-poisson-distribution-and-poisson-process-explained-4e2cb17d459}{The Poisson Distribution and Poisson Process Explained}.
\end{remark}
\end{example}

\section{Independence of discrete random variables}
Recall that events $A$ and $B$ are called ``independent" if and only if $\Prob(A \cap B) = \Prob(A)\Prob(B)$.

\begin{definition}
Discrete variables $X$ and $Y$ are \textbf{independent} if the events $\{X = x\}$ and $\{Y = y\}$ are independent for \textbf{all} $x$ and $y$.
\end{definition}

Let $A: = \{ \omega \spacevert X(\omega) = x \}$ and $B:= \{ \omega \spacevert Y(\omega) = y \}$. Then we can write $P(A) = f_X(x), P(B) = f_Y(y)$ and $P(A \cap B) = f(x, y)$. Therefore, $A$ and $B$ are independent if and only if $f(x,y) = f_X(x) f_Y(y)$ for all $x$ and $y$.

\begin{remark}
The equality $f(x,y) = f_X(x) f_Y(y)$ can be used as the criterion to determine whether two discrete random variables $X$ and $Y$ are independent or not. But \textbf{we need to be careful with it when dealing with continuous random variables} as there will be additional assumptions to determine the independence of continuous random variables using this equality.
\end{remark}

\begin{theorem}
If $X$ and $Y$ are independent and $g, h: \R \to \R$, then $g(X)$ and $h(Y)$ are independent also.
\end{theorem}

More generally, we say that a family $\{ X_i \spacevert i \in I\}$ of (discrete) random variables is independent if the events $\{X_i = x_i \}, i \in I$, are independent for all possible choices of the set $\{x_i \spacevert i \in I\}$ of the values of the $X_i$ . That is to say, $\{X_i \spacevert i \in I \}$ is an independent family if and only if
\begin{equation*}
    \Prob(X_i = x_i \text{ for all }i\in J) = \prod_{i\in J} \Prob(X_i = x_i)
\end{equation*}
for all sets $\{x_i \spacevert i \in I\}$ and for all finite subsets $J$ of $I$.

\section{Probability density functions}
Recall that a random variable $X$ is continuous if its distribution function $F(x) = \Prob(X \leq x)$ can be written as\footnote{This is just a general integral, $f(u)$ may or may not be continuous.}
\begin{equation*}
    F(x) = \int_{\infty}^x f(u)du
\end{equation*}
for some integrable $f: \R \to [0, \infty)$.

\begin{definition}
The function $f$ is called the \textbf{(probability) density function} (p.d.f.) of the continuous random variable $X$.
\end{definition}

\begin{remark}
The function $f$ is \textbf{NOT} unique. We can add some separate points or a countable set of points which has zero measure to $f$. This doesn't change the value of the integral. However, if $F$ is differentiable at $u$ then we shall normally set $f(u) = F'(u)$.
\end{remark}

Next, we assume $f$ is continuous, then from the basic theorem of calculus, $F$ must be differentiable. Recall
\begin{equation*}
    P(a < X(\omega) \leq b) = F(b) - F(a) = \int_{a}^b f(u)du.
\end{equation*}
Then $P(X=x) = F(x) - \lim_{y \uparrow x} F(y)$. $F$ is \textbf{absolutely continuous} for continuous random variables. Thus we have
\begin{equation*}
    \lim_{y\uparrow x} F(y) = F(x) \quad \Rightarrow \quad P(X=x) = 0 \quad \forall x\in\R.
\end{equation*}
This means \textbf{the probability of a continuous random variable $X$ taking value at a certain point is 0}. Very roughly speaking, this lies in the observation that there are uncountably many possible values for $X$; this number is so large that the probability of $X$ taking any particular value cannot exceed zero.

The numerical value $f(x)$ is \textbf{NOT} a probability. Check the probability $\Prob(x < X \leq x + dx)$ for a very small $dx$:
\begin{equation*}
    \Prob(x < X \leq x + dx) = F(x + dx) - F(x) \approx f(x) dx.
\end{equation*}
Since $dx$ is a very small interval rather a number, we cannot say $f(x)$ is the probability of something.

\begin{lemma}
If $X$ has density function $f$, then
\begin{enumerate}[(a)]
    \item $\int_{-\infty}^\infty f(x)dx = 1$,
    \item $\Prob(X=x) = 0$ for all $x \in \R$,
    \item $\Prob(a < X \leq b) = \int_a^b f(x)dx$.
\end{enumerate}
\end{lemma}

\section{Independence of continuous random variables}
\subsection{Independence of general random variables}
\begin{definition}
Random variables $X$ and $Y$ are called \textbf{independent} if $\{ X \leq x\}$ and $\{Y \leq y\}$ are independent events for all $x, y \in \R$.
\end{definition}
Note that this definition is the \textbf{general} definition of the independence of any two variables $X$ and $Y$, \textbf{regardless of their types}. The independence of discrete random variables is included in this definition.

Recall the marginalization. If two random variables $X$ and $Y$ are independent, we have
\begin{equation*}
    \Prob(X\leq x) = \lim_{y\to\infty}F(x, y) \equiv F_X(x), \quad
    \Prob(Y \leq y) = \lim_{x\to\infty}F(x, y) \equiv F_Y(y).
\end{equation*}
Therefore,
\begin{equation}
    \label{eq:*}
    \tag{*}
    F(x,y) = \Prob(X \leq x, Y \leq y) = \Prob(X\leq x) \Prob(Y \leq y) = F_X(x) F_Y(y).
\end{equation}
Note that we are dealing with distribution function in \eqref{eq:*}. \eqref{eq:*} can be used as the general criterion to determine whether two random variables are independent or not.

\subsection{Independence of continuous variables}
If $X, Y$ are continuous, then we have
\begin{equation*}
    F(x, y) = \int_{-\infty}^x \int_{-\infty}^y f(x,y) dx dy.
\end{equation*}
\begin{equation*}
    F_X(x) = \int_{-\infty}^x \int_{-\infty}^\infty f(x, y)dx dy \quad \Rightarrow \quad f_X(x) = \int_{-\infty}^\infty f(x,y)dy,
\end{equation*}
where $f_X(x)$ is called the \textbf{marginal probability density function} of $X$. Similarly, we can define $F_Y(y)$ and $f_Y(y)$.

Assume $f(x,y)$ is continuous, then $F(x,y)$ must be twice differentiable w.r.t. $x$ and $y$. Therefore, from \eqref{eq:*}, we can derive a \emph{very practical criterion} to determine the independence of two continuous random variables:
\begin{equation}
    \label{eq:**}
    \tag{**}
    f(x,y) = \frac{\partial^2 F(x,y)}{\partial x \partial y} = \frac{\partial}{\partial x \partial y} F_X(x) F_Y(y) = f_X(x) f_Y(y).
\end{equation}

\begin{remark}
Note that the prerequisite of \eqref{eq:**} is that \textbf{$f(x,y)$ is continuous}.
\end{remark}

\begin{example}
\textbf{Uniform distribution.} The random variable $X$ is uniform on $[a, b]$ if it has distribution function
\begin{equation*}
    F(x) = \begin{cases}
        0 & x \leq x, \\ \frac{x-a}{b-a} & a < x \leq b, \\ 1 & x > b
    \end{cases}.
\end{equation*}
The density function is
\begin{equation*}
    f(x) = \begin{cases} \frac{1}{b-a} & a < x \leq b \\ 0 & o.w. \end{cases}
\end{equation*}
\end{example}

\begin{example}
\textbf{Exponential distribution}. The random variable $X$ is exponential with parameter $\lambda(> 0)$ if it has distribution function
\begin{equation*}
    F(x) = 1 - e^{-\lambda x}, \quad x \geq 0.
\end{equation*}
The density function is
\begin{equation*}
    f(x) = \begin{cases} \lambda e^{-\lambda x} & x > 0 \\ 0 & o.w.
    \end{cases}
\end{equation*}
Note that $F(x)$ is not differentiable at $x=0$. This means $f$ has discontinuity at $x=0$. Thus we need to choose some value for $f(0)$. It doesn't matter what value we choose as it doesn't affect the integral. Figure \ref{fig:exponential} shows the p.d.f. of Exponential distribution.
\begin{figure}[!htb]
    \centering
    \includegraphics[scale=0.25]{plots/exponential.png}
    \caption{p.d.f. of Exponential distribution}
    \label{fig:exponential}
\end{figure}

\begin{remark}
The interpretation of Exponential distribution can be found via \href{https://www.probabilitycourse.com/chapter4/4_2_2_exponential.php}{Exponential distribution 1} and \href{https://www.statlect.com/probability-distributions/exponential-distribution}{Exponential distribution 2}. Pay attention to the connection between Exponential distribution and Poisson distribution.
\end{remark}
\end{example}

\begin{example}
\textbf{Normal (or Gaussian) distribution.} The most important continuous distribution, which has two parameters $\mu$ and $\sigma^2$ and density function
\begin{equation*}
    f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp \left( -\frac{(x-\mu)^2}{2\sigma^2} \right), \quad x\in\R.
\end{equation*}
It is denoted by $N(\mu, \sigma^2)$. If $\mu=0$ and $\sigma^2=1$, then
\begin{equation*}
    f(x) = \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}x^2}, \quad x\in\R.
\end{equation*}
is the density of the standard normal distribution. It is easy to generalize 2D case to multivariable case. Suppose $\mathbf{x} \in \R^n$, $\mathbf{\mu} \in \R^n$ is the mean vector and $\mathbf{\sigma}^2 \in \R^{n\times n}$ is the covariance matrix. Then
\begin{equation*}
    f(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^n} \det(\mathbf{\sigma}^2)} \exp \left( -\frac{1}{2} (\mathbf{x}-\mathbf{\mu})^T (\mathbf{\sigma}^2)^{-1} (\mathbf{x}-\mathbf{\mu}) \right), \quad \mathbf{x} \in \R^n.
\end{equation*}

\begin{remark}
For 2D case, if $\sigma^2$ is diagonal, then $f(x,y) = f(x)f(y)$. Therefore, $\sigma^2$ is a measure of independence. For multivariable cases, if $\sigma^2$ is diagonal, then all random variables are independent with each other.
\end{remark}
\end{example}

\end{document}


\documentclass[12pt]{scrartcl}

\input{preamble}

\makeatletter
\title{Hack 5.0}\let\Title\@title
\subtitle{Computer Science I -- Honors\\
{\small
\vskip1cm
Department of Computer Science \& Engineering \\
University of Nebraska--Lincoln}
\vskip-1cm}
%\author{Dr.\ Chris Bourke}
\date{~}
\makeatother

\begin{document}

\maketitle

\hrule

\input{instructions.tex}

\section*{Problem Statement}

To get some practice designing and using functions, you will
create a small library of utility functions by implementing
the following functions with the given prototypes and specified
functionality.
\begin{enumerate}
  \item Write a method to convert degrees to radians using the formula
		$$\frac{d\cdot \pi}{180}$$
	Your method should have the following signature.

	\mintinline{java}{public static double degreesToRadians(double degree)}
  \item Write a function to compute the air distance between two locations
    identified by their latitude/longitude. Your method should have the
    following signature.

\begin{minted}{java}
public static double getAirDistance(double originLatitude,
                                    double originLongitude,
                                    double destinationLatitude,
                                    double destinationLongitude)
\end{minted}
The air distance between two latitude/longitude points can be calculated
using the Spherical Law of Cosines:
 $$d = \arccos{(\sin(\varphi_1) \sin(\varphi_2) + \cos(\varphi_1) \cos(\varphi_2) \cos(\Delta) )} \cdot R$$
where
\begin{itemize}
  \item $\varphi_1$ is the latitude of location $A$, $\varphi_2$ is the latitude of location $B$
  \item $\Delta$ is the difference between location $B$'s longitude and location $A$'s longitude
  \item $R$ is the (average) radius of the earth, 6,371 kilometers
\end{itemize}
Note: the formula above assumes that latitude and longitude are measured
in radians $r$, $-\pi \leq r \leq \pi$, but the function will expect
the latitude/longitude to be in degrees.  Latitude should be in the range
$[-90, 90]$ and longitude in the range $[-180, 180]$.  Negative values
correspond to the southern and western hemispheres.

  \item An object traveling at a velocity $v$ experiences time dilation
  relative to a stationary object which is quantified by the Lorentz equation:
  $$T = \frac{t}{\sqrt{(1-\frac{v^2}{c^2})}}$$
  where $t$ is the normal amount of lapsed time (stationary object)
  and $T$ is the dilated time experienced by the traveling object.
  For small velocities, the dilation is small, but for velocities
  approaching a \emph{percentage} of the speed of light, $c$, the
  dilation becomes significant.

  For example, at 25\% the speed of light, a year for the object
  traveling would correspond to 1.032796 years at the stationary object
  (or nearly 12 extra days).   A person traveling at high velocity
  would experience ``slowed'' time
  relative to the stationary \emph{frame}.  Implement a function to
  compute the dilated time given the normal time $t$ (units may vary)
  and the percentage of the speed of light.
\begin{minted}{java}
public static double lorentzTimeDilation(double t, double percentC);
\end{minted}

\end{enumerate}

\section*{Instructions}

\begin{itemize}
  \item You are encouraged to collaborate any number of students
  before, during, and after your scheduled hack session.
  \item Design at least 3 test cases for each function
  \emph{before} you begin
  designing or implementing your program.  Test cases are
  input-output pairs that are known to be correct using means
  other than your program.
  \item Include the name(s) of everyone who worked together on
  this activity in your source file's header.
  \item Place your methods with documentation in a source file named
    \mintinline{text}{Utils.java}.

  \item In addition, implement all of your test cases in a
    \emph{test driver} file named \mintinline{text}{UtilsTester.java}
    which should output the expected output, the actual
    output and a message on whether or not the test case passed. You
    must have at least 3 test cases for \emph{each} of your functions. You
    should \emph{not} prompt for input or use command line arguments. Your
    test cases should be hardcoded in your test driver's \mintinline{java}{main()} method.

  \item Turn in all of your files via webhandin, making sure that it runs
    and executes correctly in the webgrader. Each individual student
    will need to hand in their own copy and will receive their own
    individual grade.
\end{itemize}

\end{document}


\section{Parareal}

We would like to solve the autonomous ordinary differential equation:
\begin{equation*}
  \begin{cases}
    u'(t) = f(u), & t \in [t_0, t_f] \\
    u(t_0) = u_0
  \end{cases}
\end{equation*}
where $f : \mathbb{R}^d \to \mathbb{R}^d$ and $u: \mathbb{R} \to \mathbb{R}^d$.

Parareal is an iterative scheme to approximate $u$ for the above, which can be
derived from the concept of \textit{multiple shooting} methods. The idea behind
these methods are to turn the problem into a nonlinear optimization problem,
which then can be solved via Newton-Raphson. \cite{gandervandewalle}

Take our time domain $[t_0, t_f]$ and partition it into $N$ pieces $0 = t_0 <
\dots < t_N = t_f$. Looking specifically at the interval $[t_n, t_{n+1}]$, we
pose the new ODE:
\[
  \begin{cases}
    u'_n = f(u_n), & t \in [t_n, t_{n+1}] \\
    u(t_n) = U_n
  \end{cases}
\]
where $U_n$ is such that $U_n - u_{n-1}(t_n, U_{n-1}) = 0$, i.e. the initial
condition satisfies the solution to the previous time slice. The conditions on
$U_i$ form a nonlinear system $F(U) = 0$, which we can approximate using
Newton-Raphson, so we recieve the iteration $U^{k+1} = U^k -
J_F^{-1}(U^k)F(U^k)$. As it turns out, as seen in Gander and Vandewalle
\cite{gandervandewalle}, that we can actually change this into the form:
\[
  \begin{cases}
    U_0^{k+1} = u_0 \\
    U_{n+1}^{k+1} = u_n(t_{n+1}, U_n^k) + \frac{\partial u_n}{\partial
    U_n}(t_{n+1}, U_n^k)(U_n^{k+1} - U_n^k)
  \end{cases}
\]
If we were to then, call $u_n(t_{n+1}, U_n^k) = \fine(t_{n+1}, t_n, U_n^k)$
where $F$ is some near truth integrator and were to approximate the second
term with another integrator $\coarse(t_{n+1},t_n,U_n^{k+1}) -
\coarse(t_{n+1},t_n, U_n^k)$, then we would recieve the iteration:
\[
  \begin{cases}
    U_0^{k+1} = u_0 \\
    U_{n+1}^{k+1} = \fine(t_{n+1}, t_n, U_n^k) + \coarse(t_{n+1},t_n,U_n^{k+1})
    - \coarse(t_{n+1},t_n, U_n^k)
  \end{cases}
\]
This is precisely what we call parareal iteration. Alternatively, you can think
about this as the predictor-corrector scheme:
\[
  \begin{cases}
    U_0^{k+1} = u_0 \\
    U_{n+1}^{k+1} = \coarse(t_{n+1},t_n,U_n^{k+1}) + \left(\fine(t_{n+1}, t_n,
    U_n^k) - \coarse(t_{n+1},t_n, U_n^k)\right)
  \end{cases}
\]
Regardless, we notice that this iteration, which is an approximation to our
solution as $k \to \infty$, has a term which is decoupled from the current
iteration step, $\fine$. Furthermore, it satisfies a \textit{first same as last}
property with the coarse integrators $\coarse$. The idea behind the parareal
method is to take advantage of both of these properties.


\chapter{Orbits}

A satellite stays in orbit around the planet because the pull of the
planet's gravity causes it to accerate toward the center of the
planet. The satellite must be moving a very particular speed to keep a
constant distance from the planet -- to travel in a circular orbit.
If it is moving too slowly, it will get closer to the planet.  If it
is going too fast, it will get farther from the planet.

The radius of the earth is about 6.37 million meters. A satellite that
is in a low orbit is typically about 2 million meters above the
ground. At that distance, the acceleration due to gravity is more like
$6.8 m/s^2$, instead of the $9.8 m/s^2$ that we experience on the
surface of the planet.

How fast does the satellite need to be moving in a circle with a
radius of 8.37 million meters to have an acceleration of $6.8 m/s^2$? Real fast.

Recall that the acceleration vector is

$$a = \frac{v^2}{r}$$

Thus the velocity $v$ needs to be:

$$v = \sqrt{a r} = \sqrt{6.8(8.37 x 10^6} = 7,544 \text{ m/s}$$

(That's 16,875 miles per hour.)

When a satellite falls out of orbit, it enters the atmosphere at that
7,544 m/s.  The air rushing by generates so much friction that the
satellite gets very, very hot and usually disintegrates.

\section{Astronauts are \emph{not} weightless}

Some people see astronauts floating inside an orbiting spacecraft and
think there is no gravity: that the astronauts are so far away that
the gravity of the planet doesn't affect them. This is incorrect.  The
gravity might be slightly less (Maybe 6 newtons per kg instead of 9.8
newtons per kg), but the weightless they experience is because they
and the spacecraft are in free fall.  They are just moving so fast (in
a direction perpendicular to gravity) that they don't collide with the
planet.

\begin{Exercise}[title={Mars Orbit}, label=mars_orbit]

  The radius of Mars is 3.39 million meters. The atmosphere goes up
  another 11 km.  Let's say you want to put a satellite in a circular
  orbit around Mars with a radius of 3.4 million meters.

  The acceleration due to gravity on the surface of Mars is $3.721
  m/s^2$. We can safely assume that it is approximately the same 11 km
  above the surface.

  How fast does the satellite need to be traveling in its orbit?  How
  long will each orbit take?

\end{Exercise}
\begin{Answer}[ref=circular]
  $$v = \sqrt{3.721(3.4 \times 10^6)} = 3,557\text{ m/s}$$

  The circular orbit is $2\pi(3.4 \times 10^6) = 21.4 \times 10^6$ meters in circumference.

  The period of the orbit is $(21.4 \times 10^6)/3,557 \approx 6,000$ seconds.
\end{Answer}

\section{Geosynchronous Orbits}

The planet earth rotates once a day.  Satellites in low orbits circle
the earth many times in a day. Satellites in very high orbits circle
less than once per day. There is a radius at which a satellite orbits
exactly once per day.  Satellites at this radius are known as
``geosynchronous'' or ``geostationary'' because they are always
directly over a place on the planet.

The radius of a circular geosynchronous orbit is 42.164 million
meters. (About 36 km above the surface of the earth.)

A geosynchronous satellite travels at a speed of 3,070 m/s.

Geosynchronous satellites are used for the Global Positioning
Satellite system, weather monitoring system, and communications
system.


\chapter{Time Complexity}

\section{Basic Counts}
\rih{Double for-loops}
$$
\sum_{i=1}^N{\sum_{j=i}^N{1}} = {N \choose 2} \sim \frac{1}{2} N^2
$$
$$
\sum_{i=1}^N{\sum_{j=i}^N{1}} \sim \int_{x=1}^N \int_{y=x}^N  \mathrm{d}y\, \mathrm{d}x
$$
\rih{Triple for-loops}
$$
\sum_{i=1}^N{\sum_{j=i}^N{\sum_{k=1}^N{1}}} = {N \choose 3} \sim \frac{1}{6} N^3
$$
$$
\sum_{i=1}^N{\sum_{j=i}^N{\sum_{k=1}^N{1}}} \sim \int_{x=1}^N \int_{y=x}^N \int_{z=y}^N \mathrm{d}z\,
\mathrm{d}y\, \mathrm{d}x
$$

\section{Solving Recurrence Equations}
Basic recurrence equation solving techniques:
\begin{enumerate}
\item Guessing and validation
\item Telescoping
\item Recursion tree
\item Master Theorem
\end{enumerate}

\subsection{Master Theorem}
Recurrence relations:
$$T(n) = a \; T\!\left(\frac{n}{b}\right) + f(n)\mbox{, where }a \geq 1 \mbox{, } b > 1$$

Notice that $b>1$ rather than $b\geq1$.

\subsubsection*{Case 1}
If:
$$f(n) = o(n^{\log_b a})$$

, where in the condition it is $o$ rather than $O$. \\\\
Then:
$$T(n) = \Theta(n^{\log_b a})$$
\subsubsection*{Case 2}
If:
$$f(n) = \Theta(n^{\log_b a} \log^{k} n)$$

, for some constant $k \geq 0$\\\\
Then:
$$
T(n) = \Theta(n^{\log_b a} \log^{k+1} n)
$$

, typically $k=0$ in most cases.
\subsubsection*{Case 3}
If:
$$f(n) = \omega(n^{\log_b a})$$

, where in the condition it is $\omega$ rather than $\Omega$. \\\\
And with regularity condition:
$$f(\frac{n}{b}) \le k f(n)$$

, for some constant $k < 1$ and sufficiently large $n$\\\\
Then:
$$T\left(n \right) = \Theta\left(f(n) \right)$$
\section{Useful Math Equations}
Euler:
$$
\frac{1}{2}+\frac{1}{3}+\frac{1}{4} + ... + \frac{1}{n} = \ln{n}
$$
Logarithm power:
$$
a^{\log_b^n} &= n^{\log_b^a}
$$

proof:
\begin{align*}
a^{\log_b^n} &= n^{\log_b^a} \\
\Leftarrow \ln{a^{\log_b^n}} &= \ln{n^{\log_b^a}}\\
\Leftarrow  \frac{\ln n}{\ln b}\ln a &=\frac{\ln a}{\ln b}\ln n
\end{align*}


\chapter{Key Mathematical Results}

There are a number of key results that you are expected to know. These
will have been covered in previous courses. For rigorous statements
and proofs you should consult books on the reading list or basic
calculus or linear algebra texts.

\section{Linear Algebra}

A vector $\bv$ of length $N$ is a set of $N$ numbers ordered either as
a row or column. In either case the notation $v_i$ is used to denote
the coefficients.

The dot product of a row vector $\bx$ with a column vector $\by$ is
given by
%
\begin{equation}
  \label{eq:key_la1}
  \bx \cdot \by = \sum_{i=1}^n x_i \cdot y_i.
\end{equation}

A matrix $A$ of size $N$ is a set of $N^2$ numbers ordered into $N$
rows and $N$ columns. These numbers are called the coefficients,
denoted $a_{ij}$, $i$ indicating the row and $j$ the column.

Matrix-matrix multiplication written $A B = C$ defines the
coefficients of the matrix $C$ in terms of the (given) matrix
coefficients of $A$ and $B$ by
%
\begin{equation}
  \label{eq:key_la2}
  c_{ij} = \sum_{l = 1}^n a_{i l} b_{l j}.
\end{equation}
%
In other words, the $c_{ij}$ coefficient is given by the dot product
of the $i^{\text{th}}$ row of $A$ with the $j^{\text{th}}$ column of
$B$.

Matrix-vector multiplication $A \bx = \bb$ defines the coefficients of
the column vector $\bb$ in terms of the gives matrix $A$ and column
vector $\bx$; as expected this is given by
%
\begin{equation}
  \label{eq:key_la3}
  b_{i} = \sum_{l = 1}^n a_{i l} x_{l}.
\end{equation}

The system of linear equations $A \bx = \bb$ where $A, \bb$ are known
and $\bx$ is unknown, has a unique solution if, and only if, the
determinant of the matrix $A$ is non-zero.

Eigenvectors of a matrix $A$ are any vector $\bx$ such that $A \bx =
\lambda \bx$. The value $\lambda$ is called an eigenvalue. Eigenvalues
can be found from the characteristic polynomial $\det(A - \lambda I) =
0$, where $I$ is the $N \times N$ identity matrix (this is not a
practical way of computing eigenvalues when $N$ is
large). Eigenvectors are only defined up to a multiplicative constant
(i.e., if $\bx$ is an eigenvector then $\alpha \bx$ is an eigenvector
with the same eigenvalue, provided $\alpha \ne 0$).

A \emph{basis} is a set of $N$ independent vectors $\be_i$ such that
any vector $\bx$ can be expressed as a linear combination
%
\begin{equation}
  \label{eq:key_la4}
  \bx = \alpha_1 \be_1 + \dots \alpha_N \be_N.
\end{equation}
%
A particularly useful case is when the eigenvectors of a matrix $A$
form a basis.

\section{Taylor's theorem}

If $f$ is a continuous function on $x \in [a, b]$ and the
$(n+1)^{\text{th}}$ derivative exists on $(a, b)$, then for any point
$c \in [a, b]$, we can express $f(x)$ (for any point $x \in [a, b]$)
as
%
\begin{equation}
  \label{eq:key_taylor}
  f(x) = \left\{\sum_{k=0}^n \frac{1}{k!} f^{(k)}(c) \, (x - c)^k
  \right\} + E_n(x),
\end{equation}
%
where, for some point $\xi$ between $c$ and $x$, the \emph{error} or
\emph{remainder} $E_n$ can be expressed as
%
\begin{equation}
  \label{eq:key_taylor2}
  E_n(x) = \frac{1}{(n+1)!} f^{(n+1)}(\xi) \, (x - c)^{n+1}.
\end{equation}
%
Here the notation $f^{(n)}$ means the $n^{\text{th}}$ derivative of
$f$.

For our purposes, the most useful way of writing this will be in the
form of a Maclaurin series where $c=0$. We shall also often consider
cases where $x \rightarrow h$, $|h| \ll 1$, leading to
%
\begin{equation}
  \label{eq:key_taylor3}
  f(h) = f(0) + h f^{(1)}(0) + \frac{h^2}{2} f^{(2)}(0) +
  \frac{h^3}{3!} f^{(3)}(0) + \dots + \order{h^n}.
\end{equation}
%
The notation $\order{h^n}$ means that the remaining terms contain
powers of $h$ at least as big as $n$; when $h$ is small it is usually
reasonable to neglect these terms.

\section{Triangle inequality}

For any normed vector space $V$ with elements $\bx, \by \in V$, the
triangle inequality states that
%
\begin{equation}
  \label{eq:key_triangle}
  \| \bx + \by \| \leq \| \bx \| + \| \by \|.
\end{equation}

For our purposes we will mostly be concerned with real numbers where
$\| \bx \|$ has the simple interpretation as the absolute value of the
number $x$. In this case the triangle inequality simply says that the
sum of the lengths of any two sides of a triangle must be greater than
or equal to the length of the remaining side.

\section{Difference equations}

For linear constant coefficient ODEs
%
\begin{equation}
  \label{eq:key_DE}
  a_n y^{(n)}(x) + \dots + a_1 y^{(1)}(x) + a_0 y(x) = 0
\end{equation}
%
there is a simple solution method in terms of the \emph{characteristic
polynomial}
%
\begin{equation}
  \label{eq:key_DE2}
  a_n \lambda^n + \dots + a_1 \lambda + a_0 = 0.
\end{equation}
%
The characteristic polynomial follows from substituting the assumption
$y(x) = \exp[\lambda x]$ into equation~(\ref{eq:key_DE}); $\lambda$ is
then a root of the characteristic polynomial. The general solution is
the general linear combination of all solutions found from the
characteristic polynomial. Note that repeated roots complicate the
analysis slightly.

The same approach can be used for \emph{difference equations}. These
are equations that define sequences (of e.g.\ real numbers or
vectors), usually written in the form
%
\begin{equation}
  \label{eq:key_DE3}
  a_k y_n + \dots + a_1 y_{n-k+1} + a_0 y_{n-k} = 0.
\end{equation}
%
Substituting the assumption $y_n = \lambda^n$ gives the characteristic
polynomial
%
\begin{equation}
  \label{eq:key_DE4}
  a_n \lambda^n + \dots + a_1 \lambda + a_0 = 0.
\end{equation}
%
Assuming all roots are distinct, the solution is a linear combination
of the simple solutions $\lambda^n$. If $\mu$ is a root with
multiplicity $k$ then the sequences
%
\begin{equation}
  \label{eq:key_DE5}
  y^{\{l\}}_n = \dv[(l-1)]{}{\lambda} \left[
    \lambda^n \right], \quad l = 0, \dots, k
\end{equation}
%
are all solutions.

For our purposes the key is that for the solution of a difference
equation to be \emph{bounded} (i.e., $y_n$ is bounded as $n \rightarrow
\infty$) we need all roots $\lambda$ to satisfy $|\lambda| \le 1$ (and
if the root is repeated we need $|\lambda| < 1$).


\lab{Trust-Region Methods}{Trust-Region Methods}
\label{lab:trust_region}

\objective{Explore Trust-Region methods for optimization.}

When it comes to optimizing high-dimensional functions, a common strategy is to break
the problem up into a series of smaller, easier tasks, leading to a sequence of
successive approximations to the optimizer. This is the approach taken by Line-Search
algorithms such as Newton's method or conjugate gradient.
The class of algorithms known as trust-region methods are also based on this
strategy, although they differ from line-search methods in some important ways.

\section*{Overview of the Trust-Region Approach}
Suppose we wish to minimize a function $f$.
Given some particular point $x_k$ in the domain of $f$, how do we
select a new point $x_{k+1}$ that better minimizes the function? A line-search
algorithm solves this sub-problem by first choosing a search direction $d_k$
(often related to the gradient of $f$), and then a step length $\alpha_k$ so
as to minimize $f$ along the direction $d_k$. The next point, then, is simply
\[
x_{k+1} := x_k + \alpha_k d_k.
\]

A trust-region algorithm, on the other hand, does away with a search direction and
step length, and instead approximates the function $f$ with some simpler function
$m_k$ (called the \emph{model function}) in a neighborhood of $x_k$.
The model $m_k$ will likely not be close to $f$ over the entire
domain, and so we must restrict our attention to a ball of radius $r_k$ centered at
the point $x_k$, inside of which $m_k$ is reasonably close to $f$. We then minimize
$m_k$ over this ball, and set $x_{k+1}$ equal to this minimizer. That is, we compute $x_{k+1}$ by
solving the sub-problem
\[
x_{k+1} := \underset{x \in B(x_k, r_k)}{\text{argmin}} m_k(x).
\]
The ball $B(x_k, r_k)$ is called the \emph{trust region} because we trust that the
model function $m_k$ gives a reasonably accurate approximation of $f$ on this region.
Note that it is also possible to use other types of trust regions, such as
ellipsoidal or box-like regions.

\subsection*{The Model Function}
The model function is commonly taken to be a linear or quadratic approximation of
$f$ based on its Taylor Series expansion about the point $x_k$. In the linear case,
our model function has the form
\[
f(x_k) + (y-x_k)^T \nabla f(x_k).
\]
In the quadratic case, we simply add on a quadratic term to obtain
\[
f(x_k) + (y-x_k)^T \nabla f(x_k) + \frac{1}{2}(y - x_k)^T B_k (y-x_k),
\]
where $B_k$ is the Hessian matrix of $f$ at $x_k$, or some approximation thereof.
Given a trust region with radius $r_k$, note that our sub-problem can be
written in the following way:
\begin{align*}
x_{k+1} &= \underset{x \in B(x_k, r_k)}{\text{argmin}} m_k(x)\\
&= x_k + p_k,
\end{align*}
where
\begin{equation}
p_k = \underset{\|p\| < r_k}{\text{argmin}}\, \{f(x_k) + p^T \nabla f(x_k) + \frac{1}{2}p^T B_k p\}
\label{eq:step}
\end{equation}
($p_k$ is called a \emph{step}).
For the remainder of the lab, we define
\[
m_k(p) = f(x_k) + p^T \nabla f(x_k) + \frac{1}{2}p^T B_k p,
\]
and refer to \emph{this} function as the model function.

\subsection*{The Trust-Region Radius}
A crucial aspect of trust-region algorithms is the choice of radius $r_k$. If $r_k$ is too small, then the algorithm will
make slow progress toward the minimizer of $f$. If $r_k$ is too large, the model function will be a poor fit for the objective
function $f$, and the next iterate $x_{k+1}$ may fail to decrease $f$.
Of course, whether the radius is too small or large depends on the local behavior of $f$, which may change as the algorithm
converges. A reasonably robust trust-region algorithm must therefore be able to adaptively choose the trust-region radius.

Our strategy for choosing an appropriate radius $r_{k+1}$ for the $(k+1)$-th iterate involves evaluating the accuracy
of the model function at the $k$-th iterate. If the model was accurate and a large step was taken, we can optimistically choose $r_{k+1}$ to be larger
than $r_k$ in the hopes of achieving faster convergence.
To prevent the radius from growing too large, we set an overall bound $r_{max}$ on the trust-region radii.
If the model was very inaccurate, we make $r_{k+1}$ smaller than
$r_k$, since the model function can't be trusted over such a large region. If the model was neither particularly accurate
nor inaccurate, we simply choose $r_{k+1} = r_k$.

We measure the accuracy of the model by computing the following value:
\[
\rho_k = \frac{f(x_k)-f(x_k+p_k)}{m_k(0) - m_k(p_k)}.
\]
This value is the ratio of the \emph{actual reduction} to the \emph{predicted reduction} in the objective function. The closer
$\rho_k$ is to $1$, the more accurate the model.
Note that if $\rho_k$ is negative or below a certain positive threshold $\eta$,
then the point $x_k+p_k$ is a poor improvement over $x_k$ (and perhaps is worse).
In this case, we reject the new point and set $x_{k+1} = x_k$.

\subsection*{The Trust-Region Algorithm}
We now combine the two steps of minimizing the model function and choosing the trust-region radius to build the algorithm.
\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Trust-Region Algorithm}{}
    \State Choose initial point $x_0$, initial radius $r_0$, and threshold $\eta \in [0,0.25)$.
    \For{$k=0, 1, 2, \ldots$}
        \State Solve the sub-problem in Equation \ref{eq:step}.
        \State Compute $\rho_k$.
        \If{$\rho_k < 0.25$}
            \State $r_{k+1} = 0.25r_k$

        \Else
            \If{$\rho_k > 0.75$ and $\|p_k\| = r_k$}
                \State $r_{k+1} = \min(2r_k, r_{max})$
            \Else
                \State $r_{k+1} = r_k$
            \EndIf
        \EndIf
        \If{$\rho_k > \eta$}
            \State $x_{k+1} = x_k + p_k$
        \Else
            \State $x_{k+1} = x_k$
        \EndIf
    \EndFor
\EndProcedure
\end{algorithmic}
\caption{Trust-Region Algorithm}
\label{alg:trustregion}
\end{algorithm}
In practice, we halt the algorithm once $\|\nabla f(x_k)\|$ is less than some threshold value.
\begin{problem}
Implement the trust-region algorithm using the following function declaration.
\begin{lstlisting}
def trustRegion(f,grad,hess,subprob,x0,r0,rmax,eta,gtol=1e-5):
    """
    Minimize a function using the trust-region algorithm.

    Parameters
    ----------
    f : callable function object
        The objective function to minimize
    g : callable function object
        The gradient (or approximate gradient) of the objective function
    hess : callable function object
        The hessian (or approximate hessian) of the objective function
    subprob: callable function object
        Returns the step p_k
    x0 : numpy array of shape (n,)
        The initial point
    r0 : float
        The initial trust-region radius
    rmax : float
        The max value for trust-region radii
    eta : float in [0,0.25)
        Acceptance threshold
    gtol : float
        Convergence threshold

    Returns
    -------
    x : the minimizer of f

    Notes
    -----
    The functions f, g, and hess should all take a single parameter.
    The function subprob takes as parameters a gradient vector, hessian matrix, and radius.
    """
    pass
\end{lstlisting}
\end{problem}

\subsection*{Solving the Sub-problem: the Dogleg Method}
Our trust-region algorithm is as yet incomplete, since we do not have a viable means solving the subproblem
given by Equation \ref{eq:step}.
We may be tempted to search for the true minimizer of the model function over the trust region, but it turns out
that we can get away with just an approximate minimizer.
We will employ the ``dogleg" method when selecting an approximate minimizer of the model function.
This method works by minimizing the model function along a particular path extending from the origin out
to the boundary of the trust region.
This path is piecewise linear and has a shape vaguely reminiscent of a dog's leg, which explains the peculiar name of the method.

To calculate the dogleg minimizer of the model function, we first solve the unconstrained minimizer of the model function,
\[
p^B = -B_k^{-1}\nabla f(x_k).
\]
We then calculate the direction of steepest descent for the model function, given by
\[
p^U = -\frac{\nabla f(x_k)^T\nabla f(x_k)}{\nabla f(x_k)^TB_k\nabla f(x_k)}\nabla f(x_k).
\]
We define the dogleg path using these two points as follows:
\[
\gamma(\tau) =  \left\{
     \begin{array}{lr}
       \tau p^U, & 0\leq \tau \leq 1\\
       p^U+(\tau-1)(p^B-p^U), & 1\leq \tau\leq 2
     \end{array}
   \right.
\]

\begin{figure}
\centering
    \begin{subfigure}[b]{0.3\textwidth}
            \includegraphics[width=\textwidth]{dogleg3}
            \caption{Dogleg path completely within the trust region.}
            \label{fig:dl3}
    \end{subfigure}%
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.3\textwidth}
            \includegraphics[width=\textwidth]{dogleg1}
            \caption{Intersection in the first leg of the path.}
            \label{fig:dl1}
    \end{subfigure}
    ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
      %(or a blank line to force the subfigure onto a new line)
    \begin{subfigure}[b]{0.3\textwidth}
            \includegraphics[width=\textwidth]{dogleg2}
            \caption{Intersection in the second leg of the path.}
            \label{fig:dl2}
    \end{subfigure}
    \caption{Relationships between the dogleg path (black solid line),
    the trust region boundary (red dashed circle), and the dogleg minimizer (blue dot).}
    \label{fig:dogleg}
\end{figure}

It can be shown that the model function decreases along this path. Thus,
the dogleg minimizer is either the endpoint of the path if it lies completely within the trust region,
or the point of intersection between the path and the boundary of the trust region.
See Figure \ref{fig:dogleg} for an illustration of the three salient cases. We consider each case in turn.
\begin{itemize}
\item
When the path lies completely within the trust region, the dogleg minimizer is simply the endpoint, namely $p^B$.
This is the case when $\|p^B\| \leq r_k$. See Figure \ref{fig:dl3}.
\item
When the path intersects the boundary of the trust region in the first line segment, the dogleg minimizer is
$r_kp^U/\|p^U\|$.
This is the case when $\|p^U\| \geq r_k$. See Figure \ref{fig:dl1}.
\item
When the path intersects the boundary of the trust region in the second line segment, the dogleg minimizer is
given by $p^U + (\tau^*-1)(p^B-p^U)$, where $\tau^*$ satisfies the quadratic equation
\[
\|p^U + (\tau^*-1)(p^B-p^U)\|^2 = r_k^2.
\]
See Figure \ref{fig:dl2}.
\end{itemize}

\begin{problem}
Implement the dogleg method using the following function declaration.
Remember to avoid calculating the inverse of a matrix.
\begin{lstlisting}
def dogleg(gk,Bk,rk):
    """
    Calculate the dogleg minimizer of the quadratic model function.

    Parameters
    ----------
    gk : ndarray of shape (n,)
        The current gradient of the objective function
    Bk : ndarray of shape (n,n)
        The current (or approximate) hessian
    rk : float
        The current trust region radius

    Returns
    -------
    pk : ndarray of shape (n,)
        The dogleg minimizer of the model function.
    """
    pass
\end{lstlisting}

\end{problem}

Test your implementation of the trust-region algorithm on the Rosenbrock function contained the {\tt scipy.optimize} module.
Compare your answer with that obtained by SciPy's trust-region implementation. The code to accomplish this, together with the
correct results, is given below.
\begin{lstlisting}
>>> x = np.array([10.,10])
>>> rhat=2.
>>> r=.25
>>> eta=1./16
>>> tol=1e-5
>>> opts = {'initial_trust_radius':r, 'max_trust_radius':rhat, 'eta':eta, 'gtol':tol}
>>> sol1 = op.minimize(op.rosen, x, method='dogleg', jac=op.rosen_der, hess=op.rosen_hess, options=opts)
>>> sol2 = trustRegion(op.rosen, op.rosen_der, op.rosen_hess, dogleg, x, r, rhat, eta, gtol=tol)
>>> print np.allclose(sol1.x, sol2)
True
\end{lstlisting}

\subsection*{Solving Systems of Nonlinear Equations}
Trust-region methods can be used to find solutions of systems of nonlinear equations, which arise in applications across science and engineering.
Suppose we have a vector function $r : \mathbb{R}^n \rightarrow \mathbb{R}^n$, written as
\[
r(x) = \begin{bmatrix}
r_1(x)\\
r_2(x)\\
\vdots\\
r_n(x)
\end{bmatrix},
\]
where each $r_i$ is a nonlinear smooth function mapping from $\mathbb{R}^n$ into $\mathbb{R}$.
Our goal is to find $x \in \mathbb{R}^n$ that satisfies $r(x) = 0$; such an $x$ is called a solution
or root of the nonlinear system. In general, there may be several roots (even infinitely many), or there
may be none at all. Solving the equations by hand can range from arduous to impossible, so we turn
to trust-region methods for help.

In order to use our trust-region method to find the roots of a system
of equations, we need to come up with an objective function whose minima correspond to roots of the
system. As such, we consider the \emph{merit function}
\[
f(x) = \frac{1}{2}\|r(x)\|_2^2 = \frac{1}{2}\sum_{i=1}^nr_i(x)^2,
\]
which, roughly speaking, measures how close a point $x$ is to being a root of $r$. Note that $f(x) = 0$
if and only if $r(x) = 0$. Thus, if we can successfully find a global minimum of $f$, we will have
found a root to the nonlinear system.

Now that we have an objective function, we need to create a quadratic model function. If we let
$J_k$ be the Jacobian matrix of $r$ at the point $x_k$, i.e.
\[
J_k = \begin{bmatrix}
\nabla r_1(x_k)^T\\
\nabla r_2(x_k)^T\\
\vdots\\
\nabla r_n(x_k)^T
\end{bmatrix},
\]
then we can write the gradient $\nabla f(x_k) = J_k^Tr(x_k)$ and the Hessian $B_k = J_k^TJ_k$.
We can now use the same model function described earlier.

Let's work through an example. Consider the system
\[
r(x,y) = \begin{bmatrix}
-\sin x\cos y - 2\cos x\sin y\\
-\sin y\cos x - 2\cos y\sin x
\end{bmatrix}.
\]
Observe that the Jacobian takes the form
\[
J(x) = \begin{bmatrix}
-\cos x\cos y + 2\sin x\sin y & \sin x\sin y - 2\cos x\cos y\\
\sin y\sin x - 2\cos y\cos x & -\cos y\cos x + 2\sin y \sin x
\end{bmatrix}.
\]
In Python, we initialize all of the requisite functions and then find a root as follows:
\begin{lstlisting}
>>> # define the system of equations
>>> def r(x):
>>>     return np.array([-sin(x[0])*cos(x[1]) - 2*cos(x[0])*sin(x[1]),
>>>                      -sin(x[1])*cos(x[0]) - 2*cos(x[1])*sin(x[0])])
>>>
>>> # define the merit function
>>> def f(x):
>>>     return .5*(r(x)**2).sum()
>>>
>>> # define the jacobian function
>>> def J(x):
>>>     return np.array([[-cos(x[0])*cos(x[1]) + 2*sin(x[0])*sin(x[1]),
>>>                       sin(x[0])*sin(x[1]) - 2*cos(x[0])*cos(x[1])],
>>>                      [sin(x[1])*sin(x[0]) - 2*cos(x[1])*cos(x[0]),
>>>                       -cos(x[1])*cos(x[0]) + 2*sin(x[1])*sin(x[0])]])
>>>
>>> # define the gradient function
>>> def g(x):
>>>     return J(x).dot(r(x))
>>>
>>> # define the Hessian function
>>> def B(x):
>>>     return J(x).T.dot(J(x))
>>>
>>> # set trust-region parameters
>>> rhat=2.
>>> rr=.25
>>> eta=1./16
>>> tol=1e-5
>>> # set initial point
>>> x = np.array([3.5, -2.5])
>>> # find a minimizer of f
>>> xstar = trustRegion(f,g,B,dogleg,x,rr,rhat,eta,gtol=tol)
>>> print xstar
[ 3.14159265 -3.14159265]
>>> # verify that it is a root of r
>>> print r(xstar)
[ -7.75116117e-09   7.75147025e-09]
\end{lstlisting}

Of course, we are not guaranteed to always find a root, as convergence depends on the choice of initial point.
However, by running the algorithms with several randomly selected starting points, we are more likely to
be successful.

\begin{problem}
Some problem involving system of nonlinear equations. Consult http://www-sop.inria.fr/coprin/logiciels/ALIAS/Benches/
for ideas.
\end{problem}