\chapter{Sets and Measure Theory	}

\begin{multicols}{2}[\subsubsection*{Contents of this chapter}]
   \printcontents{}{1}{\setcounter{tocdepth}{2}}
\end{multicols}

Sets are a term to describe collections of things. The things could be countable objects, such as the integers between $1$ and $10$, or contain a continuum, for example all the real numbers between $1$ and $10$. Unless otherwise specified, the elements of a set are assumed to be distinct and as not having an internal order. That is, the set of letters in "Mississippi" is $\{M,i,s,p \} = \{i,p,M,s 	\} = ... $ and so on. A concise resource for notation is \citeasnoun{stanfordsetnotation}.

% representation
\section{Representation}

\begin{tabular}{ll}
Statement Form & $\{\mathrm{integers\ between\ 1\ and\ 5}\}$\\
Roster Form & $\{1,2,3,4,5\}$ \\
Setbuilder Form & $\{x|x\in\mathbb{N}, 1\leq x \leq 5 \} $
\end{tabular}

% Set Properties, Types of Sets
\section{Set Properties and Types of Sets}

\subsection{Cardinality}
The cardinality of a set is a measure of the size of a set. For finite sets, the cardinality is simply the number of elements. For example, the set  $A = \{a,b,c,d\}$ has size $|A|=4$.

For infinite sets, this intuition breaks down, though it is still possible to make meaningful comparisons.

\subsection{$\emptyset$ Empy Set, Null Set}
The empty set is the set with no elements. It is denoted $\{\}$ or $\emptyset$.

\subsection{Singleton Set}
A singleton set is a set with one element. I.e., $\{a\}, \{b\}$ are singleton subsets of $\{a,b\}$. A singleton has cardinality 1.

\subsection{Countable Sets}
A set is countable if it contains finitely many elements or if its elements can be brought into a one-to-one correspondence with the set of integers. The set of all even integers is countable. The set of real numbers between $0$ and $1$ is not countable.

\subsection{Infinite Sets}

\subsection{Multisets}
A multiset is a set that may contain an element more than once. I.e. it makes sense to write $A=\{a,a,a,b,b\}$. The number of times an element is included is the \textit{multiplicity}. The multiset $A$ may be defined in terms of a \textit{multiplicity function} $m_A(x)$, which describes the multiplicity of a type of element $x\in U$ where $U$ may be referred to as the \textit{universe} (en lieu of saying "univseral"!). The multiplicity function allows the extension of set characteristics and operations to multisets.

\subsubsection{Support}
The support of a multiset is:

\begin{equation}
Supp(A) = \{ x\in U | 	m_A(x)>0\}
\end{equation}

Which is the set of distinct elements in $A$. I.e., if $A = \{a,a,a,b,b,c\}$ then $Supp(A) = \{a,b,c\}$.

\subsubsection{Cardinality}

The cardinality is given by:

\begin{equation}
|A| = \sum_{x\in U} m_A(x)
\end{equation}

\subsubsection{$\subseteq$ Inclusion}
The concept of subsets can be extended to multisets as:

\begin{equation}
A \subseteq B
\end{equation}

if

\begin{equation}
\forall x\in U, \ \ m_A(x) \leq m_B(x)
\end{equation}

\subsubsection{$\bigcap$ Intersection}
The intersection of multisets is sometimes called the \textit{infimum} or \textit{greatest common divisor}. If $A\cap B = C$, then $C$ has multiplicity function:

\begin{equation}
m_C(x) = min\left(m_A(x),m_B(x)\right)
\end{equation}

That is, it is like an elementwise minimum.

\subsubsection{$\bigcup$ Union}
In the context of multisets, the term \textit{union} sometimes refers to multiset addition. Otherwise, it should refer to the overlap of two multisets, i.e.:

\begin{equation}
m_C(x) = max\left(m_A(x),m_B(x)\right)
\end{equation}

That is, it is like an elementwise maximum.

\subsubsection{$\uplus$ Multiset Addition}
In contrast to sets in which distinct elements are only contained once, multiset addition makes sense and uses a special symbol "$\uplus$". If $A\uplus B = C$, then $C$ has multiplicity function:

\begin{equation}
m_C(x) = m_A(x) + m_B(x)
\end{equation}

\subsubsection{Multiset Subtraction}
Multisets can be subtracted, with the condition that the multiplicity can not be less than zero. If $A- B = C$, then $C$ has multiplicity function:

\begin{equation}
m_C(x) = max\left(m_A(x) - m_B(x), 0\right)
\end{equation}

% Powerset
\subsection{Powersets}
The powerset $P(S)$ of a set $S$ is the set of all subsets of $S$, including the empty set and the set itself. That is:

\begin{equation}
P(S) = \{ A : A\subseteq S\}
\end{equation}

My guess is it's called powerset because the number of subsets of S, $|P(S)| = 2^{|S|}$. I have also seen the notation $2^S$ to refer to the space of subsets of $S$.

% Measurable Sets
\subsection{Measurable Sets}
Measurable sets have a way of measuring volume on them in a non-trivial way. That is, for some subset of a measurable set $A_i \subseteq A$, it is possible to define a metric $\mu$ so that $\mu(A_i) \neq0)$. Measurable sets are the elements of a $\sigma$-algebra.

% universal set
\subsection{Universal Set}
The \textit{universal set} is understood to refer to something that is not allowed in the context of Russell's Paradox. It is nevertheless useful to define a set $S$ so that $S^c = \emptyset$ and I've seen this type of set referred to as \textit{universal set} a couple of times. In the context of probability theory, $S$ may be the whole event space.

%% open sets, closed sets
\subsection{Open Sets, Closed Sets}
Open sets and closed sets are generalizations of open and closed intervals on the real line. Open sets a mentioned plenty in the context of measure theory. A set is open if and only if its complement is closed, and a set is closed if and only if its complement is open. Counterintuitively, a set can be both open and closed at the same time, or it can be neither open nor closed at the same time.

\subsubsection{Interior Points}
An interior point of a set $A$ is any point $X$ so that for some $\epsilon > 0$, the open interval $X-\epsilon,X+\epsilon \subseteq A$ is contained within $A$. One might imagine an interior point as a point that has a neighborhood (a "ball") that is included in $A$.

\subsubsection{Accumulation Points}
In contrast, accumulation points do not have a neighborhood contained in $A$. Formally, for some $\epsilon>0$, $(X-\epsilon,X+\epsilon)\cap (A\setminus \{X\}) \neq \emptyset$, which is only possible for a point that is on the very boundary of $A$.

\subsubsection{Open Sets}
Unsurprisingly, an open set is a set that only has interior points, i.e. $\{X: X\in A, \exists \epsilon > 0 \mathrm{\ s.\ th.\ } X-\epsilon,X+\epsilon \subseteq A  \}$.

\subsubsection{Closed Sets}
Closed sets contain all of their accumulation points. That is, the boundary is included. Points might also be standing alone, for example:

\begin{equation}
[2,4]\cup\{1\}
\end{equation}

Is a closed set.

\subsubsection{Both Open and Closed, Neither Open Nor Closed}
The set of real numbers $\mathbb{R}$ is both open and closed. Any point on the real line has a neighborhood in $\mathbb{R}$, so it is open. On the other hand, the accumulation points of $\mathbb{R}$ are $\mathbb{R}$, so it is closed. An interval that has some but not all accumulation points is neither open nor closed. For example $[1,2)$.

%% Image
\subsection{Image}
\label{sec:image}
The image are the elements of the codomain that a given subset of the domain is mapped to. For example, given a function $f:\mathbb{R} \rightarrow \mathbb{R}: f(x) = x^2$, the image of $\{-3,-2,2,3\}$ in the domain are the elements $\{4,9\}$ of the codomain.

%% Preimage
\subsection{Preimage, Inverse Image}
\label{sec:preimage}
The preimage are elements of the domain that are mapped to a given subset of the codomain. For example, given a function $f:\mathbb{R} \rightarrow \mathbb{R}: f(x) = x^2$, the preimage of the elements $\{4,9\}$ of the codomain are the elements $\{-3,-2,2,3\}$ of the domain..

%% Convex Sets
\subsection{Convex Sets}
Convex sets are subsets of vector spaces in which any point along the line connecting to points within the set is contained within the set. A disk in $\mathbb{R}^2$ is a convex set. A crescent in $\mathbb{R}^2$ is not a convex set. The boundary of a convex set is a convex function. For example, a parabola $f(x) = x^2$ with $x\in\mathbb{R}$ is a convex function, and the area above the parabola, called the epigraph, is a convex subset of $\mathbb{R}$. Convex optimization deals with the optimization of convex functions over convex sets. The intersection of convex sets is always convex, but the union of convex sets is only convex under certain conditions.

More generally, if $S$ is a convex set, then affine combinations of the elements $\mathbf{x}_i  \in S$ of the form:

\begin{equation}
\sum_i \mathbf{x}_i \lambda_i
\label{eq:convexsets}
\end{equation}

With $\sum_i \lambda_i = 1$ and $\lambda_i\geq 0$ are also contained in $S$. Rather than just the points along the line between two points, these are the essentially the weighted averages of multiple points in the set.

\subsubsection{Example: Discrete Probability Distributions}
The set of discrete probability distributions $P = \{ \mathbf{p}=(p_1,p_2,...): ||\mathbf{p}||_1 = 1, p_i \geq 0\}$ is a convex set. Property \ref{eq:convexsets} means that a weighted average over members of $P$ is also a member of $P$, so long as the weights satisfy $\sum_i \lambda_i = 1$. In general, this guarantees that marginalization results in a probability. I.e. if $p(X|Y) = \mathbf{p}(X|Y) \in P$ and $p(Y) = \mathbf{p}(Y) \in P$, then $\mathbf{p}(X) = \sum_i p_i(X|Y) p_i(Y) \in P$.

%% Choice sets, Transverse Sets, Cross-sections
\subsection{Choice Sets, Transversal Sets, Cross-Sections}
\label{sec:choicesets}
Choice sets, transversal sets or cross-sections are sets that are assembled by picking exactly one element from each member of a family of disjoint sets. The axiom of choice (cf. section \ref{sec:axiomofchoice}) states that such a set can always be formed.

\subsubsection{Example: Integers}
Consider the partition of the integers between $1$ and $40$ into a family of disjoint sets that each contain $10$ elements:

\begin{equation}
S = \{ \{1,...,10\}, \{11,...,20\}, \{21,...,30\},\{31,...,40\} \}
\end{equation}

Then a transversal set $A$ could for example be formed by picking the smallest number in each of the subsets.

\begin{equation}
A = \{1,11,21,31\}
\end{equation}

Though we might have picked any other choice function.

% Set operations
\section{Set Operations}

\subsection{$A^c$ Complement}
Given a subset $A \subseteq X$, the complement of the subset $A^c$ refers to everything that is not contained in $A$, i.e. $A^c = X\setminus A$.

\subsection{$\bigcup$ Union}
The union of sets is the set that contains all of their elements, counting all elements only once. The cardinality of the union is calculated using the important inclusion exclusion principle (\ref{sec:inclusionexclusion}). In terms of logic, think $A\cup B$ is "A or B (or both)".

\subsection{$\bigcap$ Intersection}
The intersection of sets is the elements shared by all sets. $A\cap B$ is "A and B".

% Disjoint Union
\subsection{$\bigsqcup$ Disjoint Union, Discriminated Union}
The disjoint union, or discriminated union, of sets is the union formed in a way in which the information about which subset an element belonged to is preserved. One way to write this is to include a subset index with each element, i.e.:

\begin{equation}
\bigsqcup_{i\in\{i\}}A_i = \bigcup_{i\in\{i\}} \{(x,i): x\in A_i\}
\end{equation}

For example, for the two sets $A=\{1,2,3,4\}$ and $B=\{1,2,3\}$, the disjoint union is:

\begin{equation}
A\sqcup B = \{(1,A),(2,A),(3,A),(4,A),(1,B),(2,B),(3,B)\}
\end{equation}

The cardinality of the disjoint union is simply the sum:

\begin{equation}
\left|\bigsqcup_{i\in\{i\}}A_i\right| = \sum_{i\in\{i\}}|A_i
\end{equation}

I have also seen the disjoint union be used in the context of forming the union of disjoint sets, perhaps to stress that the sets are disjoint and the cardinality can be calculated through simple summation as above. For example, in dividing up the interval: $[0,2]=[0,1)\sqcup[1,2]$.

% Bijection Principle
\section{Bijection Principle}
The bijection principle states that when a bijection exists between the elements of two sets, then those sets have the same size. This principle is very useful in combinatorics, and it allows for meaningful comparisons of infinite sets.

% DeMorgan
\section{DeMorgan's Rules}
\label{sec:demorgan}
DeMorgan's Rules relate the complement of the union to the intersection of the complements, and the complement of the intersection to the union of the complements.

\begin{equation}
\left(\bigcup_{i\in\{i\}}A_i\right)^c = \bigcap_{i\in\{i\}}A^c_i
\end{equation}

\begin{equation}
\left(\bigcap_{i\in\{i\}}A_i\right)^c = \bigcup_{i\in\{i\}}A^c_i
\end{equation}

%Inclusion Exclusion
\section{Inclusion - Exclusion Principle}
\label{sec:inclusionexclusion}

The inclusion-exclusion principle is used to calculate the size of the union of sets. This requires counting each region of some complicated overlapping Venn diagram exactly once, which, in turn requires accounting for overcounting wherever sets overlap. Let $\{A_i | i\in \{i\}_n \}$ be a collection of $n$ overlapping sets indexed by $i\in \{i\}_n$, then the inclusion-exclusion principle is given by:

\begin{equation}
\left|\bigcup_{i\in\{i\}_n} A_i\right| = \sum^n_{k=1} (-1)^{k-1} \sum_{\{j\}_k \subseteq \{i\}_n} \left|\bigcap_{j\in\{j\}_k} A_j\right|
\end{equation}

Where the sum over $\{j\}_k \subseteq \{i\}_n$ is over all $k$-element subsets of $\{i\}_n$.

\subsection{Example: n=2 Sets and n=3 Sets}

\subparagraph{n=2}
\begin{equation}
\begin{array}{rl}
\left|\bigcup_{i\in\{1,2\}}A_i\right| =&  \sum^2_{k=1} (-1)^{k-1} \sum_{\{j\}_k \subseteq \{1,2\}} \left|\bigcap_{j\in\{j\}_k} A_j\right|\\
=&(-1)^{0}\left(|A_1| + |A_2|\right) \\
&+ (-1)^{1}\left(|A_1 \cap A_2| \right)
\end{array}
\end{equation}

\subparagraph{n=3}
\begin{equation}
\begin{array}{rl}
\left|\bigcup_{i\in\{1,2,3\}}A_i\right| =&  \sum^3_{k=1} (-1)^{k-1} \sum_{\{j\}_k \subseteq \{1,2,3\}} \left|\bigcap_{j\in\{j\}_k} A_j\right|\\
=&(-1)^{0}\left(|A_1| + |A_2| + |A_3|\right) \\
&+ (-1)^{1}\left(|A_1 \cap A_2|  + |A_1 \cap A_3| + |A_2 \cap A_3|  \right) \\
&+ (-1)^{2}\left(|A_1 \cap A_2 \cap A_3|   \right)
\end{array}
\end{equation}

\subsubsection{Example: Counting Integers}

How many integers are there between 1 and 100 that are neither divisible by 3,5 nor 7?

Let $S$ be the set of all integers between 1 and 100. The size of the set is $|S| = 100$. The subset of $S$ that is numbers divisible by 3 is $A_3 \subseteq S$ with $|A_3| = 33$ because $100/3 = 33.\overline{333}$. Similarly, $|A_5| = 20$ and $|A_7| = 14$.  The set of integers that is not divisible by 3, 5 or 7 is:

\begin{equation}
S \setminus \bigcup_{i\in\{3,5,7\}} A_i
\end{equation}

So that the sought after quantity is :

\begin{equation}
\begin{array}{rl}
\left|S \setminus \bigcup_{i\in\{3,5,7\}} A_i\right| =& |S| - \left[ |A_3| + |A_5| + |A_7| \right.\\
& \left. - |A_3 \cap A_5| - |A_3 \cap A_7| - |A_5\cap A_7| + |A_3\cap A_5\cap A_7|\right]
\end{array}
\end{equation}

The size of the intersection $|A_3\cap A_5| = 6$ because 100 is 6 times divisible by $3\times 5 = 15$. Similarly, $|A_3 \cap A_7| =  4$, $|A_5 \cap A_7| =  2$ and $|A_3\cap A_5 \cap A_7| =  0|$. Hence:

\begin{equation}
\left|S \setminus \bigcup_{i\in\{3,5,7\}} A_i\right| = 100 - 33 - 20 - 14 + 6 + 4 + 2 - 0 = 45
\end{equation}

There are 45 integers between 1 and 100 that are not divisible by 3, 5 or 7.

\section{Axiom of Choice}
\label{sec:axiomofchoice}
The axiom of choice simply states that, given a collection of nonempty, mutually disjoint sets, it is possible to assemble a \textit{transversal} or \textit{choice} set that consists of exactly one element from each of the sets in the collection. For example, consider the students in 1st, 2nd, 3rd.. etc grades at a school to be a collection of nonempty, mutually disjoint sets. Then the axiom of choice says that it is possible to assemble a subset of students with exactly one student from each grade.

In terms of functions, one might think of defining a \textit{choice function} on the family of mutually disjoint sets, which selects members from the collection of sets and adds them to the \textit{choice} set. According to the axiom of choice, a choice function can be defined for any collection of nonempty, mutually disjoint subsets. This implies that all surjective functions have a right inverse. That is, for any surjective function $f:X\rightarrow Y$, there exists a function $g:Y\rightarrow X$ so that $f(g(y)) = y$.

The axiom of choice turns out to be associated with famous names and deep consequences \cite{stanfordaxiomofchoice}.

\subsection{Example: Pairs of Real Numbers, Right Inverse}
The collection of rank-2 sets of pairs of real numbers $A= \{ A_x = \{x,-x\} : x\in \mathbb{R}^{+}\}$ are a collection of mutually disjoint subsets. A transversal set $B$ may be assembled by choosing the largest element of the tuple: $B = \{y : y max(A_x), A_x \in A \}$. An equivalent surjection $f:A\rightarrow B$ is $f(A_x) = x$. There is a right-inverse $g:B\rightarrow A$ which is $g(x) = (x,-x)$ so that $f(g(x)) = x$.

\input{./chapters/sections/sets_sigma.tex}


%% document Notes

\documentclass[12pt]{article}
\usepackage{fullpage,graphicx,psfrag,amsmath,amsfonts,verbatim}
\usepackage[small,bf]{caption}

\bibliographystyle{alpha}

\title{Computational Intelligence         Bonus Problem }

\author{Mostafa Osama Ahmed Metwally Othman}

\begin{document}
\maketitle{Mini Max Problem}\\
This problem is a Mini max Quadratic problem where we want to solve using CVX solver or any other so we need to reformulate the problem to be in the form of one of the known forms and we are looking for the worst case scenario in order to achieve our goal.

\paragraph{Problem statement:}

%
\begin{equation}
\begin{aligned}
& \underset{\mathbf{X}}{\text{min   }}\underset{\mathbf{y}}{\text{max}}
& & || \mathbf{X} - \mathbf{ X^\star} ||, \\
& \text{subject to}
& & \begin{cases}

   (\mathbf{y}-\mathbf{a})^T \mathbf{D} (\mathbf{X}-\mathbf{b}) + \mathbf{s}^T \mathbf{y} + \mathbf{q}^T \mathbf{X} \leq h  \\
   ||\mathbf{H}\mathbf{y}+\mathbf{f}|| \leq p
    \end{cases}
\end{aligned}
\end{equation}

\paragraph{Solution:}

\begin{equation}
\begin{aligned}
\text{let us consider:}\\
& \mathbf{v} = ||\mathbf{H}\mathbf{y}+\mathbf{f}||\\
& \mathbf{y} = \mathbf{H}^{-1} (\mathbf{v} - \mathbf{f})\\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\text{substitute in main equation: }\\
& (\mathbf{y}-\mathbf{a})^T \mathbf{D} (\mathbf{X}-\mathbf{b}) + \mathbf{s}^T \mathbf{y} + \mathbf{q}^T \mathbf{X} \leq h  \\
\end{aligned}
\end{equation}
\begin{equation}
\begin{aligned}
& ( \mathbf{H}^{-1} (\mathbf{v} - \mathbf{f})-\mathbf{a})^T \mathbf{D} (\mathbf{X}-\mathbf{b})+ \mathbf{s}^T  \mathbf{H}^{-1} (\mathbf{v} - \mathbf{f}) + \mathbf{q}^T \mathbf{X} \leq h  \\
\text{let us expand the equation:}\\
\end{aligned}
\end{equation}
\begin{equation}
\begin{aligned}
& ( \mathbf{H}^{-1} \mathbf{v} - \mathbf{H}^{-1} \mathbf{f}-\mathbf{a})^T \mathbf{D} (\mathbf{X}-\mathbf{b})+ \mathbf{s}^T  \mathbf{H}^{-1} \mathbf{v} - \mathbf{s}^T\mathbf{H}^{-1} \mathbf{f} + \mathbf{q}^T \mathbf{X} \leq h  \\
\end{aligned}
\end{equation}
\begin{equation}
\begin{aligned}
& \mathbf{v}^T \mathbf{H}^{-T} \mathbf{D} (\mathbf{X}-\mathbf{b}) - (\mathbf{H}^{-1} \mathbf{f}-\mathbf{a})^T \mathbf{D} (\mathbf{X}-\mathbf{b})+ \mathbf{s}^T  \mathbf{H}^{-1} \mathbf{v} - \mathbf{s}^T\mathbf{H}^{-1} \mathbf{f} + \mathbf{q}^T \mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\text{Using vector operations:}\\
& \mathbf{s}^T  \mathbf{H}^{-1} \mathbf{v} = \mathbf{v}^T\mathbf{H}^{-T}\mathbf{s} \\
& \mathbf{s}^T\mathbf{H}^{-1} \mathbf{f} = \mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s} \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
& \mathbf{v}^T \mathbf{H}^{-T} \mathbf{D}\ (\mathbf{X}-\mathbf{b})-(\mathbf{H}^{-1}\mathbf{f}-\mathbf{a})^T\ \mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{v}^T\mathbf{H}^{-T}\mathbf{s}-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
& \mathbf{v}^T \mathbf{H}^{-T} \mathbf{D}\ (\mathbf{X}-\mathbf{b})-(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{v}^T\mathbf{H}^{-T}\mathbf{s}-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
& \mathbf{v}^T \mathbf{H}^{-T} (\mathbf{D}\ (\mathbf{X}-\mathbf{b})+\mathbf{s})-(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

Now by looking at this equation, the worst case scenario corresponds to the largest value of $\mathbf{V}^T  \mathbf{H}^{-T} (\mathbf{D} (\mathbf{X}-\mathbf{b})+\mathbf{s})$
where $\mathbf{V}^T$ should align with the other term in order to get the maximum possible value $\mathbf{p}$

\begin{equation}
\begin{aligned}
& \mathbf{v} = \mathbf{p} \frac{\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})}{||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})||} \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
&\mathbf{p} \frac{(\mathbf{H}^{-T} (\mathbf{D}\ (\mathbf{X}-\mathbf{b})+\mathbf{s}))^T\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s}) }
{||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})||}-(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
\mathbf{p} \frac{(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})^T \mathbf{H}^{-1}\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s}) } {||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})||}-(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}
\begin{equation}
\begin{aligned}
& \mathbf{p} \frac{||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})||^2}{||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})||}-(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
& \mathbf{p} ||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})|| -(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\ \\
\end{aligned}
\end{equation}

Then we can Re-Represent the problem into :

%
\begin{equation}
\begin{aligned}
& \underset{\mathbf{X}}{\text{min }}
& & || \mathbf{X} - \mathbf{ X^\star} ||, \\
& \text{subject to}
& & \mathbf{p} ||\mathbf{H}^{-T}(\mathbf{D}(\mathbf{X}-\mathbf{b})+\mathbf{s})|| -(\mathbf{H}\mathbf{a}+\mathbf{f})^T\ \mathbf{H}^{-T}\mathbf{D}(\mathbf{X}-\mathbf{b})-\mathbf{f}^T\ \mathbf{H}^{-T}\mathbf{s}+\mathbf{q}^T\mathbf{X} \leq h  \\
\end{aligned}
\end{equation}

Which is then considered as a SOCP Problem.

\end{document}


\subsection{Integers}\label{subsec:integers}

\begin{definition}\label{def:set_of_integers}
  The \hyperref[def:ring]{ring} \( \BbbZ \) of \term{integers} is defined as the \hyperref[thm:grothendieck_semiring_completion]{Grothendieck completion} of the \hyperref[def:semiring/commutative]{commutative semiring} \( \BbbN \) of zero-based natural numbers.
\end{definition}

\begin{lemma}\label{thm:integer_signum_lemma}
  Consider the canonical embedding \( \iota: \BbbN \to \BbbZ \). For every nonzero integer \( n \), there either exists a unique natural number \( a \) such that \( n = \iota(a) \), or a unique natural number \( a \) such that \( n = -\iota(a) \).
\end{lemma}
\begin{proof}
  \SubProof{Proof of existence} Due to the trichotomy on natural numbers shown in \fullref{def:natural_number_ordering}, we have the following mutually exclusive possibilities:
  \begin{itemize}
    \item If \( a = b \), then \( x = [(a, a)] = [(0_\BbbN, 0_\BbbN)] \), hence \( x \) is zero.
    \item If \( a < b \), by \fullref{def:natural_number_ordering}, there exists some positive natural number \( c \) such that \( a + c = 0_\BbbN + b \). Then
    \begin{equation*}
      x = [(a, b)] = [(c, 0_\BbbN)] = \iota(c).
    \end{equation*}

    \item If \( a > b \), there exists some natural number \( d \) such that \( 0_\BbbN + a = b + d \). Then
    \begin{equation*}
      x = [(a, b)] = [(0_\BbbN, d)] = -[(d, 0_\BbbN)] = -\iota(d).
    \end{equation*}
  \end{itemize}

  \SubProof{Proof of uniqueness} If \( \iota(a) = \iota(b) \), then there exists some natural number \( u \) such that
  \begin{equation*}
     a + 0_\BbbN + u = 0_\BbbN + b + u.
  \end{equation*}

  By \fullref{thm:natural_number_addition_properties}, \( \BbbN \) is cancellative, so \( a = b \).

  \SubProof{Proof of exclusive conditions} Finally, suppose that \( n = \iota(a) = -\iota(b) \). Then \( a + b = 0_\BbbN \). Since \( \BbbN \) is \hyperref[def:zerosumfree]{zerosumfree} by \fullref{thm:natural_number_addition_properties}, it follows that \( a = b = 0_\BbbN \), and hence \( n \) is zero.

  Therefore, if \( n \) is nonzero, either \( n = \iota(a) \) for some \( a \) or \( n = -\iota(b) \) for some \( b \).
\end{proof}

\begin{definition}\label{def:integer_signum}
  Consider the canonical embedding \( \iota: \BbbN \to \BbbZ \). Define the \term{signum} function
  \begin{equation*}
    \begin{aligned}
      &\sgn: \BbbZ \to \BbbZ \\
      &\sgn(n) \coloneqq \begin{cases}
        0,  &n = \iota(0_\BbbN), \\
        1,  &n = \iota(a) \T{for some nonzero natural number} a, \\
        -1, &n = -\iota(a) \T{for some nonzero natural number} a.
      \end{cases}
    \end{aligned}
  \end{equation*}

  This is a well-defined \hyperref[def:multi_valued_function/total]{total function} due to \fullref{thm:integer_signum_lemma}.

  We then classify integers based in their sign as follows
  \begin{center}
    \begin{tabular}{l | l || l | l}
      Positive    & \( \sgn(n) = 1 \)  & Nonpositive & \( \sgn(n) \neq 1 \) \\
      Negative    & \( \sgn(n) = -1 \) & Nonnegative & \( \sgn(n) \neq -1 \) \\
      Zero        & \( \sgn(n) = 0 \)  & Nonzero     & \( \sgn(n) \neq 0 \) \\
    \end{tabular}
  \end{center}
\end{definition}

\begin{proposition}\label{thm:def:integer_signum}
  \hyperref[def:integer_signum]{Integer signs} has the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:integer_signum/entire} The ring of integers is \hyperref[def:divisibility/zero]{entire}, i.e. an \hyperref[def:integral_domain]{integral domain}.
    \thmitem{thm:def:integer_signum/sum} If two integers have the same sign, their sum also has the same sign.
    \thmitem{thm:def:integer_signum/product} \( \sgn(nm) = \sgn(n) \cdot \sgn(m) \).
    \thmitem{thm:def:integer_signum/zero} \( n \) is zero if and only if \( -n \) is zero.
    \thmitem{thm:def:integer_signum/inverse} \( n \) is positive if and only if \( -n \) is negative.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:integer_signum/entire} Suppose that \( nm = 0 \).

  \begin{itemize}
    \item If \( n = \iota(a) \) and \( m = \iota(b) \) are nonnegative, since \( \BbbN \) is entire and since \( \iota \) is a homomorphism, then \( nm = \iota(a) \cdot \iota(b) = \iota(ab) = \iota(0_\BbbN) \) implies that at least one of \( a \) or \( b \) is zero.
    \item If \( n = \iota(a) \) is nonnegative and \( m = -\iota(b) \) is nonpositive, this reduces to the previous case since \( nm = 0 = n(-m) \).
    \item The other cases are similar.
  \end{itemize}

  Therefore, \( n = 0 \) or \( m = 0 \).

  \SubProofOf{thm:def:integer_signum/sum} Fix two integers \( n \) and \( m \).

  \begin{itemize}
    \item If \( n \) and \( m \) are both zero, their sum is also zero.
    \item If \( n = \iota(a) \) and \( m = \iota(b) \) are both positive, then \( n + m = \iota(a + b) \), so \( n + m \) is also positive.
    \item If \( n = -\iota(a) \) and \( m = -\iota(b) \) are both negative, then \( n + m = -\iota(a + b) \), so \( n + m \) is also negative.
  \end{itemize}

  \SubProofOf{thm:def:integer_signum/product} Fix two integers \( n \) and \( m \).

  \begin{itemize}
    \item If either \( n \) or \( m \) is zero, the product \( nm \) is again zero, so
    \begin{equation*}
      \sgn(nm) = \sgn(n) \cdot \sgn(m) = 0.
    \end{equation*}

    \item If both \( n = \iota(a) \) and \( m = \iota(b) \) are positive, then \( nm = \iota(ab) \) is nonnegative. Furthermore, \( nm \) cannot be zero because \( \BbbZ \) is an integral domain. Hence,
    \begin{equation*}
      \sgn(nm) = \sgn(n) \cdot \sgn(m) = 1.
    \end{equation*}

    \item If \( n = \iota(a) \) is positive and \( m = -\iota(b) \) is negative, then \( n(-m) = \iota(ab) \) is positive. Furthermore, \( nm = -n(-m) = -\iota(ab) \), and hence \( nm \) is negative. Then
    \begin{equation*}
      \sgn(nm) = \underbrace{\sgn(n)}_1 \cdot \underbrace{\sgn(m)}_{-1} = -1.
    \end{equation*}

    The case where \( n \) is negative and \( m \) is positive follows from this one due to commutativity.

    \item If both \( n = -\iota(a) \) and \( m = -\iota(b) \) are negative, then \( (-n)(-m) = \iota(ab) \) is positive. Furthermore, \( nm = (-n)(-m) = -\iota(ab) \), and hence \( nm \) is positive.
    \begin{equation*}
      \sgn(nm) = \underbrace{\sgn(n)}_{-1} \cdot \underbrace{\sgn(m)}_{-1} = 1.
    \end{equation*}
  \end{itemize}

  \SubProofOf{thm:def:integer_signum/zero} This is actually a statement about group inverses because \( 0 = -0 \).

  \SubProofOf{thm:def:integer_signum/inverse} From \fullref{thm:def:integer_signum/product} it follows that \( \sgn(-n) = \sgn(-1) \cdot \sgn(n) = -\sgn(n) \).
\end{proof}

\begin{definition}\label{def:integer_ordering}
  We extend the \hyperref[def:natural_number_ordering]{natural number ordering} \( \leq_\BbbN \) to the \hyperref[def:set_of_integers]{integers} \( \BbbZ \) via the following truth table:
  \begin{center}
    \begin{tabular}{c c c}
      \( n \) & \( m \) & \( n \leq m \) \\
      \hline
      nonnegative       & nonnegative & \( n \leq_\BbbN m \) \\
      nonnegative       & negative    & \( F \) \\
      negative          & nonnegative & \( T \) \\
      negative          & negative    & \( -m \leq_\BbbN -n \)
    \end{tabular}
  \end{center}
\end{definition}

\begin{proposition}\label{thm:def:integer_ordering}
  \hyperref[def:integer_ordering]{Integer ordering} has the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:integer_ordering/positive} \( 0 \leq n \) if and only if \( n \) is positive.
    \thmitem{thm:def:integer_ordering/negative} \( 0 \geq n \) if and only if \( n \) is negative.
    \thmitem{thm:def:integer_ordering/inverse} \( n \leq m \) if and only if \( -m \leq -n \).
    \thmitem{thm:def:integer_ordering/total} It is a \hyperref[def:totally_ordered_set]{total order}.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:integer_ordering/positive} Trivial.
  \SubProofOf{thm:def:integer_ordering/negative} Trivial.

  \SubProofOf{thm:def:integer_ordering/inverse} Let \( n \leq m \).
  \begin{itemize}
    \item If \( n \) and \( m \) are both either nonnegative or negative, then \( -m \leq -n \) by definition.
    \item If \( n \) is negative and \( m \) is not, then by \fullref{thm:def:integer_signum/inverse} \( -m \) is negative and \( -n \) is not. Hence \( -m \leq -n \).
  \end{itemize}

  The converse direction of the proof is identical.

  \SubProofOf{thm:def:integer_ordering/total} The order \( \leq \) is clearly defined for every pair of integers, so it remains to show that \( \leq \) is a partial order.

  \SubProofOf*[def:binary_relation/reflexive]{reflexivity} If \( n \) is positive, then \( n \leq n \) since \( n \leq_\BbbN n \). Otherwise, \( n \leq n \) since \( -n \leq_\BbbN -n \).

  \SubProofOf*[def:binary_relation/antisymmetric]{antisymmetry} Suppose that \( n \leq m \) and \( m \leq n \).
  \begin{itemize}
    \item If \( n \) and \( m \) are both nonnegative, then \( n = m \) from the antisymmetry of the natural number ordering.
    \item If \( n \) and \( m \) are both negative, then \( -n = -m \) from the antisymmetry of the natural number ordering.
    \item If \( n \) is nonnegative and \( m \) is negative, we have \( m \leq n \) but not \( n \leq m \). This contradicts our assumption.
    \item If \( n \) is negative and \( m \) is nonnegative, we have \( n \leq m \) but not \( m \leq n \). This contradicts our assumption.
  \end{itemize}

  \SubProofOf*[def:binary_relation/transitive]{transitivity} Suppose that \( n \leq m \) and \( m \leq k \).
  \begin{itemize}
    \item If \( n \), \( m \) and \( k \) are nonnegative, then \( n \leq k \) from the transitivity of the natural number ordering.
    \item If \( n \), \( m \) and \( k \) are negative, then \( -k \leq -n \) and, by \fullref{thm:def:integer_ordering/inverse}, \( n \leq k \).
    \item If \( n \) is negative and \( k \) is nonnegative, then \( n \leq k \) by definition.
  \end{itemize}
\end{proof}

\begin{definition}\label{def:integer_absolute_value}
  We define the function
  \begin{equation*}
    \begin{aligned}
      &\abs{\anon}: \BbbZ \to \BbbN \\
      &\abs{n} \coloneqq \begin{cases}
        n,  n \geq 0 \\
        -n, n < 0.
      \end{cases}
    \end{aligned}
  \end{equation*}

  We call \( \abs{\anon} \) the \term{absolute value function} in \( \BbbZ \). It does satisfy the absolute value axioms from \fullref{def:absolute_value}. A proof would be cyclic, however, since the real numbers rely on an extension of \( \abs{\anon} \).
\end{definition}

\begin{algorithm}[Integer division]\label{alg:integer_division}
  Fix two integers \( n \) and \( m \), and assume that \( m \) is nonzero. Define
  \begin{align*}
    q &\coloneqq \sgn(nm) \cdot \max\set{ y \geq 0 \colon y\abs{m} < \abs{n} } \\
    r &\coloneqq n - mq
  \end{align*}

  Then \( q \) and \( r \) and the unique integers such that \( \abs{r} < \abs{m} \) and
  \begin{equation*}
    n = mq + r.
  \end{equation*}

  We will use the notation \( \quot(n, m) \) and \( \rem(n, m) \) from \fullref{def:euclidean_domain}.
\end{algorithm}
\begin{proof}
  \SubProof{Proof of correctness} The case \( n = 0 \) is trivial so assume that \( n \neq 0 \). We have
  \begin{center}
    \begin{tabular}{l | l | l}
      \( n \)  & \( m \)  & \( q \) \\
      positive & positive & \( \phantom{-}\max\set{ y \geq 0 \colon (n - m \leq) \thickspace y(+m) < +n } \) \\
      positive & negative & \( -\max\set{ y \geq 0 \colon (n + m \leq) \thickspace y(-m) < +n } \) \\
      negative & positive & \( -\max\set{ y \geq 0 \colon (m - n \leq) \thickspace y(+m) < -n } \) \\
      negative & negative & \( \phantom{-}\max\set{ y \geq 0 \colon (n - m \leq) \thickspace y(-m) < -n } \)
    \end{tabular}
  \end{center}

  It follows that \( \abs{n - mq} < \abs{m} \).

  \SubProof{Proof of uniqueness} Suppose that \( n = mq + r = mq' + r' \), where \( \abs{r} < \abs{m} \) and \( \abs{r'} < \abs{m} \). Then
  \begin{equation*}
    m(q - q') = -(r - r').
  \end{equation*}

  Thus, \( m \) divides \( r - r' \). Then \( m \) divides \( r \) and \( r' \) contradicting the assumption that \( \abs{r} < \abs{m} \).
\end{proof}

\begin{proposition}\label{thm:integers_are_euclidean_domain}
  The \hyperref[def:set_of_integers]{ring of integers} is an \hyperref[def:euclidean_domain]{Euclidean domain} with division given by \fullref{alg:integer_division} and degree function \( \abs{\anon} \).
\end{proposition}
\begin{proof}
  By \fullref{thm:def:integer_signum/entire}, \( \BbbZ \) is an integral domain. The Euclidean domain structure is described by \fullref{alg:integer_division}.
\end{proof}

\begin{remark}\label{rem:integer_domain_chain}\hfill
  \begin{itemize}
    \item By \fullref{thm:integers_are_euclidean_domain}, \( \BbbZ \) is a \hyperref[def:euclidean_domain]{Euclidean domain}.
    \item By \fullref{thm:def:euclidean_domain/pid}, \( \BbbZ \) is a \hyperref[def:principal_ideal_domain]{principal ideal domain}.
    \item By \fullref{thm:def:principal_ideal_domain/ufd}, \( \BbbZ \) is a \hyperref[def:unique_factorization_domain]{unique factorization domain}.
    \item By \fullref{thm:def:unique_factorization_domain/gcd}, \( \BbbZ \) is a \hyperref[def:gcd_domain]{GCD domain}.
  \end{itemize}
\end{remark}

\begin{remark}\label{rem:integer_gcd}
  We discuss in \fullref{rem:choice_of_associates} and \fullref{rem:lattice_of_principal_ideals} how, in general \hyperref[def:gcd_domain]{GCD domain}, the \hyperref[def:gcd_and_lcm]{greatest common divisors} are not unique.

  The \hyperref[def:gcd_and_lcm]{greatest common divisor} of is, by convention, \hi{positive}. This leaves a canonical choice for both the greatest common divisor and the least common multiple. By \fullref{thm:natural_number_divisibility_lattice}, the \hyperref[thm:semiring_divisibility_order]{divisibility order} of positive integers is compatible with the usual \hyperref[def:integer_ordering]{integer ordering}.

  \Fullref{alg:euclidean_algorithm} allows us to explicitly compute both GCDs and LCMs.
\end{remark}

\begin{definition}\label{def:prime_number}
  A \term{prime number} is an integer greater than \( 1 \) whose only proper \hyperref[def:divisibility]{divisor} is \( 1 \). Non-prime integers greater than \( 1 \) are called \term{composite numbers}.
\end{definition}

\begin{remark}\label{rem:prime_numbers}
  The definition of a prime number given in \fullref{def:prime_number} is standard, however it seems quite inconsistent with \fullref{subsec:integral_domains}.

  First, \fullref{subsec:integral_domains} actually defines \hyperref[def:domain_divisibility/irreducible]{irreducible elements} rather than \hyperref[def:domain_divisibility/prime]{prime elements} of the domain \( \BbbZ \). Second, if \( p \) is a prime number, \( -p \) is also a prime number.

  Fortunately, prime and irreducible elements coincide in \hyperref[def:gcd_domain]{GCD domains} by \fullref{thm:def:gcd_domain/irreducible_is_prime} and \fullref{thm:def:gcd_domain/irreducible_is_prime}. Unfortunately, calling negative prime elements of \( \BbbZ \) \enquote{prime numbers} is not accepted.

  Coprime integers are, fortunately, defined as in general GCD domains via \fullref{def:coprime_elements}.
\end{remark}

\begin{lemma}[Euclid's lemma]\label{thm:euclids_lemma}
  If \( p \) is a \hyperref[def:prime_number]{prime number}, then \( p \mid nm \) implies \( p \mid n \) or \( p \mid m \).
\end{lemma}
\begin{proof}
  Since \( \BbbZ \) is a GCD domain, the lemma follows from \fullref{thm:def:gcd_domain/irreducible_is_prime}.
\end{proof}

\begin{theorem}[Fundamental theorem of arithmetic]\label{thm:fundamental_theorem_of_arithmetic}
  Every integer greater than \( 1 \) can be \hyperref[def:irreducible_factorization]{factored} into a product of \hyperref[def:prime_number]{prime} powers.
\end{theorem}
\begin{proof}
  We have discussed in \fullref{rem:integer_domain_chain} that \( \BbbZ \) is a unique factorization domain.
\end{proof}

\begin{definition}\label{def:eulers_totient_function}
  For any positive integer \( n \), denote by \( \varphi(n) \) the number of strictly smaller than \( n \) positive integers that are \hyperref[def:coprime_elements]{coprime} to \( n \). We call \( \varphi: \BbbZ_{>0} \to \BbbZ_{\geq 0} \) \term{Euler's totient function}.
\end{definition}

\begin{proposition}\label{thm:def:eulers_totient_function}
  \hyperref[def:eulers_totient_function]{Euler's totient function} \( \varphi \) has the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:eulers_totient_function/one} \( \varphi(1) = 0 \).
    \thmitem{thm:def:eulers_totient_function/prime} If \( p \) is \hyperref[def:prime_number]{prime}, then \( \varphi(p) = p - 1 \).
    \thmitem{thm:def:eulers_totient_function/zn} The \hyperref[def:semiring]{multiplicative group} \( \BbbZ_n^\times \) of the ring \hyperref[thm:ring_of_integers_modulo]{\( \BbbZ_n \)} of integers modulo \( n > 1 \) has order \( \varphi(n) \).
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:eulers_totient_function/one} There are no positive integers smaller than \( 1 \).

  \SubProofOf{thm:def:eulers_totient_function/prime} Every positive integer smaller than \( p \) is coprime to \( p \), and there are exactly \( p - 1 \) positive integers smaller than \( p \) --- \( 1, 2, \ldots, p - 1 \).

  \SubProofOf{thm:def:eulers_totient_function/zn} Follows from \fullref{thm:multiplicative_group_of_integers_modulo}.
\end{proof}

\begin{theorem}[Euler's totient theorem]\label{thm:eulers_totient_theorem}
  For positive coprime integers \( n \) and \( x \), we have
  \begin{equation*}
    x^{\varphi(n)} \cong 1 \pmod n,
  \end{equation*}
  where \( \varphi \) is \hyperref[def:eulers_totient_function]{Euler's totient function}.
\end{theorem}
\begin{proof}
  This is vacuous for \( n = 1 \) since all integers are equal modulo \( 1 \).

  Suppose that \( n > 1 \). First, use \fullref{alg:integer_division} to obtain integers \( q \) and \( y < n \) such that
  \begin{equation*}
    x = nq + y.
  \end{equation*}

  Since \( x \) is, by assumption, coprime with \( n \), then \( y \) is also coprime with \( n \). Indeed, every common divisor \( d \) of \( y \) and \( n \) is also a common divisor \( x \), and the largest such possible value is \( \gcd(n, x) = 1 \).

  Now consider the \hyperref[def:semiring]{multiplicative group} \( \BbbZ_n^\times \) of the ring \hyperref[thm:ring_of_integers_modulo]{\( \BbbZ_n \)} of integers modulo \( n \) and the \hyperref[def:cyclic_group]{cyclic subgroup} \( \set{ 1, y, y^2, \ldots } \) (modulo \( n \)). It is necessarily finite as a subgroup of \( \BbbZ_n^\times \). Furthermore, by \fullref{thm:lagranges_theorem_for_groups}, its order \( k \) divides the order of \( \BbbZ_n^\times \). By \fullref{thm:def:eulers_totient_function/zn}, the order of \( \BbbZ_n^\times \) is \( \varphi(n) \).

  We have \( y^k \cong 1 \pmod n \) since \( k \) is the order of a cyclic group. If \( \varphi(n) = km \), then
  \begin{equation*}
    y^{\varphi(n)}
    =
    y^{km}
    \reloset {\eqref{eq:thm:magma_exponentiation_properties/repeated}} =
    (y^k)^m
    \cong
    1^m
    \pmod n.
  \end{equation*}
\end{proof}

\begin{corollary}\label{thm:division_modulo}
  Given positive integers \( n \) and \( m \), we can apply \fullref{alg:integer_division} to obtain \( n = q \varphi(m) + r \), where \( \varphi \) is \hyperref[def:eulers_totient_function]{Euler's totient function}.

  Then, for a positive integer \( x \) coprime to \( m \), we have
  \begin{equation*}
    x^n \cong x^r \pmod m,
  \end{equation*}
\end{corollary}
\begin{proof}
  By \fullref{thm:eulers_totient_theorem}, \( x^{\varphi(m)} \cong 1 \pmod m \). Then
  \begin{equation*}
    x^n = (x^{\varphi(m)})^q x^r \cong x^r \pmod m.
  \end{equation*}
\end{proof}

\begin{example}\label{ex:division_modulo}
  The integers \( 9 \) and \( 10 \) are coprime. We have \( \varphi(9) = 6 \) and \( 1000 = 166 \cdot 6 + 4 \). By \fullref{thm:division_modulo},
  \begin{equation*}
    9^{1000} \cong 9^4 \cong 6561 \cong 1 \pmod {10}.
  \end{equation*}

  We can thus vastly simplify finding the last digit of the decimal representation of \( 9^{1000} \).
\end{example}

\begin{theorem}[Fermat's little theorem]\label{thm:fermats_little_theorem}
  For a \hyperref[def:prime_number]{prime number} \( p \) and for any positive integer \( x \), we have
  \begin{equation*}
    x^p \cong x \pmod p.
  \end{equation*}
\end{theorem}
\begin{proof}
  If \( p \mid x \), then both \( x^p \) and \( x \) and congruent to \( 0 \) modulo \( p \).

  Otherwise, by \fullref{thm:eulers_totient_theorem}, we have \( x^{\varphi(p) + 1} \cong x \pmod p \), and by \fullref{thm:def:eulers_totient_function/prime}, we have \( \varphi(p) + 1 = p \).
\end{proof}


\subsection{Exponential Decay}\label{ex:decay-set-sample}

To demonstrate the qualitative differences in the solutions provided by the set-- and sample--based methods for a nonlinear problem, we consider an exponential decay problem with uncertain decay rate and initial condition (which are paired to form the 2-D vector $\param$):
$$
\begin{cases}
  \frac{\partial u}{\partial t} & = \param_1 u(t), \\
  u(0) &= \param_2.
\end{cases}
$$

The solution is described by
\begin{equation}
  u(t;\param) = u_0\exp(\param_1 t), \; u_0 = \param_2 ,
\end{equation}

and a nominal value of $\param = 0.5$ is used to simulate the system.
We take a single observation at $t=0.5$s and assume a uniform density with interval length $0.2$ centered at $u(1,0.5)$ to represent the uncertainty in the measurement equipment.
We assume a uniform ansatz / initial density over the unit domain.
We use $N=50$ parameter samples to establish a coarse solution in Figure~\ref{fig:heatrod-sol-ex1}.

\begin{figure}
\begin{minipage}{.475\textwidth}
\includegraphics[width=\linewidth]{examples/fig_decay_q1/DecayModel--set_N50_em.png}
\end{minipage}
\begin{minipage}{.475\textwidth}
\includegraphics[width=\linewidth]{examples/fig_decay_q1/DecayModel--sample_N50_mc.png}
\end{minipage}
\caption{Observation taken at $t=1$s. The inverse image of the reference measure for set-based (left) and sample-based (right) solutions for $\nsamps=50$ parameter samples.}
\label{fig:heatrod-sol-ex1}
\end{figure}

The decay rate $\param_1$ shows little reduction in uncertainty overall.
If one were to look at marginals of the components of $\param$, it would not appear as if much was learned.
However, the relationship \emph{between} these two quantities has very certainly been elucidated by the solution of the inverse problem.
Where once $\pspace$ was a rectangular region, the set of possible parameters has been reduced to a diagonal band.
The sample-based approach, especially at this low sample size (density estimation in 2-D at $50$ samples is a stretch), has some visible downsides.
It does not capture the equivalence--class nature of the solution set the way the set--valued one does, which benefits from using $\ndiscs=1$ (aligning with the choice of uniform observed density).

We address what would occur had we been able to observe earlier in time at $t=0.5$ by showing the associated solutions under the same experimental conditions in \ref{fig:heatrod-sol-ex2}.
There is a marked reduction in uncertainty, as several regions of $\pspace$ have been ruled out from consideration.

\begin{figure}
\begin{minipage}{.475\textwidth}
\includegraphics[width=\linewidth]{examples/fig_decay_q2/DecayModel--set_N50_em.png}
\end{minipage}
\begin{minipage}{.475\textwidth}
\includegraphics[width=\linewidth]{examples/fig_decay_q2/DecayModel--sample_N50_mc.png}
\end{minipage}
\caption{Observation taken at $t=0.5$s. The inverse image of the reference measure for set-based (left) and sample-based (right) solutions for $\nsamps=50$ parameter samples.}
\label{fig:heatrod-sol-ex2}
\end{figure}

Observing earlier in time helps especially in reducing the (marginal) values for the initial condition $\param_2$, while the rate $\param_1$ is still to some degree able to take any values in its original domain.
Both solution types suffer from discretization error, as evidenced by the break in the contour structure.
By comparison to \ref{fig:heatrod-sol-ex1}, there is more more confidence in the solution (represented by the reduced support of the image).

However, at $\nsamps=50$, the sample--based approach struggles to assign uniform probability to different contour events.
This may suggest that in situations with very limited model evaluation budget and set-valued solutions involving uniform uncertainties in measurements, the set-valued approach may serve a useful purpose.


%\nomenclature[]{HMM}{Hidden Markov Model}
%\note{Add italics in places}
%\note{Could discuss:
%\\Background:
%What HMMs are,
%why they're useful in AI,
%the different commonly used HMMs.
%\\ Use in my research:
%How the problem can naturally be described by a HMM,
%Lead into discussion of how DBN is a more natural way to describe the problem and how it leads to efficient factorization of densities for state estimation}

HMMs appear frequently in AI literature as they provide an abstract framework to deal with stochastic processes, which themselves are pervasive in their use as a tool to model real-world phenomena. This section will outline what HMMs are and their use in literature describing target localisation algorithms. A general overview of HMMs and Markov Processes can be found in \cite{Murphy1994DynamicLearning}, \cite{Ghahramani2001AnNetworks}, \cite{Bhattacharya2009StochasticApplications} and \cite{papoulis02}, on which we base the following discussion.

\subsubsection{Markov Processes}\label{subsubsec:MarkovProcesses}
It is instructive to understand what is meant by a stochastic process for some of the concepts mentioned in this thesis. A random process can be described as a family of random variables indexed by a set $\tau$: $\{X_t\}_{t\in\tau}$ \cite{Bhattacharya2009StochasticApplications}. Commonly in AI, stochastic processes model the evolution of a random system through \textit{discrete} time steps: $\tau$=$\mathbb N$. Examples of phenomena that are frequently modelled by stochastic processes include the growth of a bacterial population and the movement of a gas molecule \cite{Bhattacharya2009StochasticApplications}.\par

A first-order discrete-time Markov process is a stochastic process that describes a discrete-time stochastic process for which the first-order \textit{Markov property} holds \cite{Ghahramani2001AnNetworks}. The first-order Markov property states that the probability distribution of the n$_{th}$ random variable in the process is conditionally independent of all previous probability distributions in the sequence but the $n-1_{st}$: $P(X_t = x_t | X_{t-1} = x_{t-1}, X_{t-2} = x_{t-2}, ... , X_{1} = x_{1}) = P(X_t = x_t | X_{t-1} = x_{t-1})$ \cite{Ghahramani2001AnNetworks}. This is often referred to as the memoryless property of Markov processes. In order to describe a Markov process, it is therefore necessary to describe what is known as the transition function between each pair of time-steps: $P(X_t = x_t | X_{t-1} = x_{t-1})$. A common assumption is that the rules that govern state transitions are time invariant, meaning that they can be specified generally for any given pair of time-steps. This assumption will be made for the subsequent discussion. If $X_t$ is a discrete random variable defined over $S$ states, the transition function can be described by a stochastic matrix T, where T$_{i,j}$ = $P(X_t = j | X_{t-1} = i)$:

\begin{center}
{$\displaystyle \left({\begin{matrix}T_{1,1}&T_{1,2}&\dots &T_{1,j}&\dots &T_{1,S}\\T_{2,1}&T_{2,2}&\dots &T_{2,j}&\dots &T_{2,S}\\\vdots &\vdots &\ddots &\vdots &\ddots &\vdots \\T_{i,1}&T_{i,2}&\dots &T_{i,j}&\dots &T_{i,S}\\\vdots &\vdots &\ddots &\vdots &\ddots &\vdots \\T_{S,1}&T_{S,2}&\dots &T_{S,j}&\dots &T_{S,S}\\\end{matrix}}\right)$}
\end{center}
\par

Some obvious results are worth pointing out; as for any stochastic matrix, by the axioms of probability theory, the sum of conditional probabilities across each column sum to one: {$\displaystyle \sum _{j=1}^{S}T_{i,j}=1$} and the transition probabilities over $k$ time-steps can be described by the $k_{th}$ power of the transition matrix: ${(T^k)}_{i,j}$ = $P(X_{t+k} = j | X_{t} = i)$.

It also is possible to calculate the probability of the process experiencing a sequence of states from time-steps 1 as far as $t$, using the chain rule of probability and the Markov property:
$P(X_{1:t}) = P(X_1, X_2, ..., X_t) = P(X_1)\times P(X_2 | X_1)\times P(X_3 | X_2, X_1) \times ... \times P(X_t | X_{t-1}, X_{t-2}, ... , X_1) = P(X_1) \times \prod_{i=2}^{t}{P(X_i | x_{i-1})}$. Marginalization over variables in this sequence allows the calculation of many useful quantities.
\par

Markov processes can described by graphical models. For example Figure \ref{fig:markov-processes} displays a graphical representation of first and second order Markov processes, where a directed arrow between nodes represents a directional dependence between those nodes.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\linewidth]{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/Figs/MarkovProcesses/MarkovProcesses.png}
    \caption{A graphical model describing first and second order Markov processes}
    \label{fig:markov-processes}
\end{figure}

\subsubsection{HMM Description}\label{subsubsec:HMMDesc}
Hidden Markov Models (HMMs) are models that build on the Markov Process model, which describes the evolution of a random system in the language of probability theory. HMMs assume that the system being modelled can be described by a Markov process, but that the states of this process are unobservable \cite{Ghahramani2001AnNetworks}. This means that it is not possible to determine the state of the system exactly at any given point in time. However, it is possible to make an observation of a random variable that is related to the hidden state which yields information about the hidden state. A graphical representation of a HMM is shown in Figure \ref{fig:hmm},
\begin{figure}[]
    \centering
    \includegraphics[width=0.8\linewidth]{Chapters/BackgroundKnowledgeAndRelatedWork/MultiAgentTargetDetectionBackground/Figs/HMMs/HMMGraphicalModel.png}
    \caption{A graphical model of a HMM}
    \label{fig:hmm}
\end{figure} where the Markov Process is shown by the variables $X_i$ and the observation variables are shown by variables $E_i$. A HMM can be specified by a triple, $\lambda$ = $(T, O, \pi)$, where $T$ is the stochastic transition matrix, $\pi$ is the initial distribution $P(X_1)$ and O describes the conditional probability of an observation given that the system is in a certain state: $O(E_i, X_i) = P(E_{i} | X_{i})$ \cite{Rabiner1989ARecognition}. Taken together, it is then possible to specify the joint distribution of the hidden state variables and the evidence variables, analogous to the Markov Process:
$
P(X_{1:t}, E_{1:t}) = P(X_1, X_2, ..., X_t, E_1, E_2, ..., E_t) = P(X_1) \times P(E_1 | X_1) \times
\prod_{i=2}^{t}{(P(X_i | x_{i-1}) \times P(E_t | X_t))}.
$ Given this representation, it is then possible to answer questions identified in \cite{Rabiner1989ARecognition} and \cite{Murphy1994DynamicLearning}:
\begin{itemize}

    \item Given the HMM, $\lambda$, what is the probability of occurrence of a particular observation sequence, $P(E_{1:t} | \lambda)$?

    \item Given a sequence of observations $(E_1, E_2, ..., E_n)$, what is the most likely sequence of hidden states that led to these observations? i.e. find \[\argmax_{X_{1:t}} P(X_{1:t} | E_{1:t})\]

    \item Determine the parameters of $T$ and $O$, given a training set of observations, i.e. find the solution to \[\argmax_{\lambda} P(E_{1:t} | \lambda)\]

    \item \textit{Filtering}: What is the current distribution of the hidden state given all previous evidence ("belief state") of the environment at time t: $P(X_t | E_{1:t})$?

    \item \textit{Prediction}: What is the distribution of the hidden state in the future, given all evidence to date: $P(X_{t+k} | E_{1:t})$, for some k$>$0?

    \item \textit{Smoothing}: What is the distribution of a past state given all observations up to the current point in time: $P(X_k | E_{1:t})$, for some 0 $\leq$ k $<$ t?

\end{itemize}
\par

%Might include a subsection on a taxonomy of commonly used HMMs.

%\subsubsection{Summary}\label{subsubsec:Summary}
%In summary, HMMs can be used to abstractly describe the evolution of a stochastic system. They have been used to achieve state of the art performance in problems such as speech recognition \cite{ChiuSTATE-OF-THE-ARTMODELS} and ... . An comprehensive overview of extensions to the vanilla HMM can be found at \cite{Murphy1994DynamicLearning}.


\section{Asymptotics and Recurrences}

% MATH REVIEW SUBSECTION |-----------------------------------------------------|

\subsection{Math Review}

Before we talk about asymptotics and recurrences, we quickly review the
logarithmic function $\log$.

\begin{definition*}
We define the \textbf{logarithm} to be the opposite of exponentiation, i.e. we
say that $\log_b(x) = y$ exactly if $b^y = x$. For example $\log_2(64) = 6$,
since $64 = 2^6$.
\end{definition*}

Here's a couple useful properties of the $\log$ that you will use in this
section extensively.

\begin{enumerate}[(1)]

\item Product $\to$ Summation: $\log_b(xy) = \log_b(x) + \log_b(y)$.

\begin{proof}

By definition, we know that $b^{\log_b(xy)} = xy$. However, also notice that $x
= b^{\log_b(x)}$ and $y = b^{\log_b(y)} \implies xy = b^{\log_b(x)}b^{\log_b(y)}
= b^{\log_b(x) + \log_b(y)} \implies b^{\log_b(xy)} = b^{\log_b(x) + \log_b(y)}$.

\end{proof}

\item Quotient $\to$ Subtraction: $\log_b \left( \frac{x}{y} \right) = \log_b(x)
- \log_b(y)$.

\begin{proof}
This can be seen using the previous identity, and just taking $y \mapsto
\frac{1}{y}$, and noticing that $\frac{1}{y} = \frac{1}{b^{\log_b(y)}} =
b^{-\log_b(y)}$.
\end{proof}

\item Power $\to$ Scalar Multiplication: $\log_b(x^p) = p\log_b(x)$.

\begin{proof}

By defintion:

$$
x = b^{\log_b(x)} \implies x^p = \left(b^{\log_b(x)}\right)^p = b^{p\log_b(x)}
$$

Therefore it must be true that $\log_b(x^p) = p\log_b(x)$.

\end{proof}

\item Change of Base: $\log_b(x) = \frac{\log_k(x)}{\log_k(b)}$.

\begin{proof}

Starting from $x = b^{\log_b(x)}$, we can take $\log_k$ of both sides to
recieve:

$$
\log_k(x) = \log_k b^{\log_b(x)} = \log_b(x) \log_k(b)
$$

Therefore, dividing through by $\log_k(b)$ gives us the desired identity.

\end{proof}

\end{enumerate}

% ASYMPTOTICS SUBSECTION |-----------------------------------------------------|

\subsection{Asymptotics Analysis}

In the analysis of algorithms, one of our goals is to classify growth of the
runtime and storage complexity of functions. To do so, it's in our interest to
examine the limiting behavior of functions, i.e. when our input becomes large
what does our function do. To this end we introduce Big O notation.

\subsubsection{Big O notation}

Big O notation exists to describe the limiting behavior of a function, and us
Computer Scientists use it to classify algorithms in terms of their input size
$n$. Due to this, from this point onwards all functions we define are
non-negative.

We give both the informal and formal definition of a couple different
useful characterizations. Let $f,g : \R \to \R$ be two functions.

\begin{itemize}

\item Upper bound: $\mathcal{O}$.

\begin{definition*}
Informally, we say that $f(x) = \mathcal{O}(g(x))$ if $ f(x) \leq cg(x)$, $c$
some constant, for sufficiently large $x$.

Formally, $f(x) = \mathcal{O}(g(x))$ iff:

$$
\limsup_{x \to \infty} \abs{\frac{f(x)}{g(x)}} < \infty
$$

For example, $n^2 = \mathcal{O}(n^2)$ and $n^2 = \mathcal{O}(n^5)$ but $n^2 \neq
\mathcal{O}(n)$. To highlight the sufficiently large $x$ portion, notice that it
is true that $n^2 = \mathcal{O}(n^3 - n^2)$ ($n^3 - n^2 > n^2, \forall n > 2$.)

\end{definition*}

\item Lower Bound: $\Omega$.

\begin{definition*}

Informally, we say that $f(x) = \Omega(g(x))$ if and only if $g(x) =
\mathcal{O}(f(x))$, or that $f(x) \geq cg(x)$.

Formally $f(x) = \Omega(g(x)) \iff \liminf_{n \to \infty}
\abs{\frac{f(n)}{g(n)}} > 0$.

\end{definition*}

\item Tight bound: $\theta$.

\begin{definition*}

We say that $f(x) = \theta(g(x))$ if $f(x) = \mathcal{O}(g(x))$ and $f(x) =
\Omega(g(x))$. Another way to say this is that $\exists c \in \R$ such that
$\frac{1}{c} g(x) \leq f(x) \leq cg(x)$, for sufficiently large enough $x$.

Formally we must have that:

$$
\limsup_{x \to \infty} \abs{\frac{f(x)}{g(x)}} \in \R_{> 0}
$$

To those that care, notice that having $f(x) = \mathcal{O}(g(x))$ and
$\Omega(g(x)) \centernot \implies \lim_{x \to \infty} \abs{\frac{f(x)}{g(x)}}$ exists.
Instead $\liminf$ and $\limsup$ may converge to different values.

So for example, $n^2 = \theta(n^2)$ but $n^2 \neq \theta(n)$ and $\neq
\theta(n^3)$.

\end{definition*}

\end{itemize}

\subsubsection{Properties of Big O}

There's a couple of useful properties of Big O that are good to know. Let $f_1 =
\theta(g_1)$ and $f_2 = \theta(g_2)$.

\begin{enumerate}[(1)]

\item Product: $f_1f_2 = \theta(g_1g_2)$. In addition $f \theta(g) =
\theta(fg)$.

\item Sum: $f_1 + f_2 = \theta(g_1 + g_2)$. In particular if $f_1 =
\mathcal{O}(g)$ and $f_2 = \mathcal{O}(g)$ then $f_1+f_2 = \mathcal{O}(g)$.

\item Scalar multiplication: $\theta(kg) = \theta(g)$ supposing $k \neq 0$. In
addition $kf_1 = \theta(g_1)$.

\item Summation $\to$ maximization.: $\theta(g_1 + g_2) = \theta(\max(g_1,
g_2))$.

\end{enumerate}

\subsubsection{Time Complexity Comparison}

Typically for problems where we want to compare which function grows faster
asymptotically, we really want to see at large $x$, which one dominates. Our
main strategy to do this, is to reduce these functions down to functions which
are more easily compared, which I call the time complexity classes. See the time
complexity chart in the figures section for a good table of a commmon few which
you should remember.

Take for examples:

\begin{enumerate}[(1)]

\item $2^n$ versus $n^2$.

\begin{proof}[Solution]

It's clear that $2^n$ is exponential, which grows faster than $n^2$ which is
polynomial.

\end{proof}

\item $n^{\log(n)}$ versus $n^{100}$.

\begin{proof}[Solution]

The left hand function is somewhere above polynomial due to the power being an
increasing function of $n$. Therefore, it grows faster than $n^{100}$.

\end{proof}

\item $n^{1/\log_2(n)}$ versus $2$.

These are strange to compare, since we aren't sure what class the left hand
function falls in. To try to fit it to something we understand, we remember that
the logarithm takes powers to scalar multiplcation, so then we notice:

$$
\log_2(n^{1/\log_2(n)}) = \frac{1}{\log_2(n)} \log_2(n) = 1
$$

But also $\log_2(2) = 1$. Therefore we find that these functions are actually
equal in terms of asymptotic growth.

\end{enumerate}

% RANDOMIZED FUNCTION SUBSECTION |---------------------------------------------|

\subsection{Algorithm $\to$ Recurrence}

Here we want to examine how to turn recursive code into reucurrences
characterizing their properties. We do this by example.

\subsubsection{Fibbonacci Sequence}

\begin{algorithmic}[1]
\Procedure{F}{$n$}\Comment{$a_1 = 1, a_2 = 1, a_n = a_{n-1}+a_{n-2}$}
	\If{$n \leq 2$}
		\State Return $1$
	\Else
		\State Return $F(n-1)+F(n-2)$
	\EndIf
\EndProcedure
\end{algorithmic}

Suppose we want to characterize the runtime of the above algorithm using a
recurrence equation. The idea is that we count the work per level, and add it.
Letting $G(n)$ characterize the runtime of $F(n)$.

\begin{algorithmic}[1]
\Procedure{F}{$n$}
	\If{$n \leq 2$} \Comment{$\theta(1)$}
		\State Return $1$ \Comment{$\theta(1)$}
	\Else \Comment{$\theta(1)$}
		\State Return $F(n-1)+F(n-2)$ \Comment{$G(n-1)+G(n-2) + \theta(1)$}
	\EndIf \Comment{$\theta(1)$}
\EndProcedure
\end{algorithmic}

Notice that $\theta(1)$ summed a constant number of times is still $\theta(1)$,
therefore we have:

$$
G(n) = \begin{cases}
\theta(1) & n \leq 2 \\
\theta(1) + G(n-1) + G(n-2) & \text{otherwise}
\end{cases}
$$

For characterizing things that aren't work, i.e. how many times the $-1$
operation happens in the above code, we can do a similar process. Let $H(n)$
characterize the number of $-1$ operations in $F(n)$. Then:

\begin{algorithmic}[1]
\Procedure{F}{$n$}
	\If{$n \leq 2$} \Comment{$0$}
		\State Return $1$ \Comment{$0$}
	\Else \Comment{$0$}
		\State Return $F(n-1)+F(n-2)$ \Comment{$H(n-1)+H(n-2) + 1$}
	\EndIf \Comment{$0$}
\EndProcedure
\end{algorithmic}

Therefore:

$$
H(n) = \begin{cases}
0 & n \leq 2 \\
1 + H(n-1) + H(n-2) & \text{otherwise}
\end{cases}
$$

\subsubsection{Merge Sort}

Let $F(n)$ characterize the work of Merge Sort on an array of length $n$. Then:

\begin{algorithmic}[1]
\Procedure{Mergesort}{$A$ : Array}
	\State Let $n = len(A)$.
	\If{$n = 1$} \Comment{$\theta(1)$}
		\State Return $A$ \Comment{$\theta(1)$}
	\EndIf \Comment{$\theta(1)$}
	\State $B \gets A[1... \lfloor n/2 \rfloor]$ \Comment{$\theta(1)$}
	\State $C \gets A[\lceil n/2 \rceil ... n]$ \Comment{$\theta(1)$}
	\State $Mergesort(B), Mergesort(C)$ \Comment{$F(\lfloor n/2 \rfloor) +
	F(\lceil n/2 \rceil)$}
	\State Return $Merge(B,C)$ \Comment{$\theta(n)$}
\EndProcedure
\end{algorithmic}

Therefore we have:

$$
F(n) = \begin{cases}
\theta(1) & n = 1 \\
\theta(n) + F(\lfloor n/2 \rfloor) + F(\lceil n/2 \rceil) & \text{otherwise}
\end{cases}
$$

Similarily we can do this for $H(n)$ characterizing storage constraints, but
I'll leave that as exercise.

\subsubsection{Randomized Algorithms}

A classic Siegel question is to characterize various facets of the randomized
algorithm:

\begin{algorithmic}[1]
\Procedure{Rand}{$n$}
	\If{$n \leq 1$}
		\State Return 0
	\EndIf
	\State Let $x \gets 1, 2$ or $3$ with probabilities $1/2, 1/3, 1/6$
	respectively.
	\If{$x = 1$}
		\State Return $2Rand(n)$
	\ElsIf{$x = 2$}
		\State Return $7Rand(n-1) + 12Rand(n-2)$
	\Else
		\State Return $nRand(n-1)$
	\EndIf
\EndProcedure
\end{algorithmic}

Define function $F, G, H$ which characterize runtime, exact value returned, and
number of times $x \gets 3$ respectively. The trick to characterizing the above,
is simply to take whatever happens inside of an if statement and multiply it by
the probability of the event occuring. So:

\begin{align*}
F(n) & = \begin{cases}
\theta(1) & n \leq 1 \\
\theta(1) + \frac{1}{2} (\theta(1) + F(n)) + \frac{1}{3} (\theta(1) + F(n-1) +
F(n-2)) + \frac{1}{6}( \theta(1) + F(n-1) ) & \text{otherwise}
\end{cases} \\
G(n) & = \begin{cases}
0 & n \leq 1 \\
\frac{1}{2} (2H(n)) + \frac{1}{3} (7H(n-1) + 12H(n-2)) + \frac{1}{6}( nH(n-1) )
& \text{otherwise}
\end{cases} \\
H(n) & = \begin{cases}
0 & n \leq 1 \\
\frac{1}{2} (0 + H(n)) + \frac{1}{3} (0 + H(n-1) + H(n-2)) + \frac{1}{6}(1 +
H(n-1) ) & \text{otherwise}
\end{cases} \\
\end{align*}

This always appears on the exam.

% Solving RECURRENCES SUBSECTION |---------------------------------------------|

\subsection{Solving Recurrences}

Ok, so now that we know how to characterize certain aspects of recursive code
using recurrences, one might ask, how do we solve recurrences? The idea stems
behind something called the tree method. For example, consider the recurrence
equation:

$$
f(n) = \begin{cases}
1 & n = 1 \\
n + 2f(n/3) & \text{otherwise}
\end{cases}
$$

Here we call $n$ the work function, $3$ the decay rate, and $2$ the branching
factor.

What we do to solve this is \textit{unroll} the recursive term over and over
again. We represent each unrolling as a new level of the tree, so it may look
like:

\Tree
[.$n$
	[.$n/3$
		[.$n/9$ $\vdots$ $\vdots$ ]
		[.$n/9$ $\vdots$ $\vdots$ ]
	]
	[.$n/3$
		[.$n/9$ $\vdots$ $\vdots$ ]
		[.$n/9$ $\vdots$ $\vdots$ ]
	]
]

The top level looks like $n + 2f(n/3)$, but then we unroll $f(n/3)$ to $n/3 +
2(n/9)$, which is our second level. To then figure out what our summation totals
to, we sum accross the level, and then downward. In the above example, the first
level totals to $n$, the second to $(2/3)n$, the third to $(4/9)n$, etc. In
fact, we will find that in general the $j$th level will sum to
$\frac{2^j}{3^j}n$; this is guarenteed by the unrolling. But what about the leaf
level?

In general, we have to examine the leaf level seperately, because the recurrence
defines something different for it. Here, our recurrence says that each leaf has
cost $1$. Looking at our tree, we see that at each level the number of vertices
doubles (powers of branching factor), therefore at the leaf level $k$ we must
have $2^k$ leaves. Each costs 1, therefore the contribution from the leaf level
is $2^k$.

Therefore our final
total actually looks something like $n + \frac{2}{3} n + \frac{2^2}{3^n} n +
\dots + \frac{2^{k-1}}{3^{k-1}}n + 2^k$, for some stopping point $k$. Let's find
$k$.

$k$ is determined by the point we can \textit{unroll} no longer, i.e. the decay
rate has been applied sufficiently enough times such that we get down to our
base case, $1$. In other words, we want to solve for $k$ where:

$$
\frac{n}{3^k} = 1 \implies n = 3^k \implies k = \log_3(n)
$$

Since we choose $n$ a power of $3$, we find that $k$ is an integer like we
wanted it to be. Then, we can write our final answer as:

$$
f(n) = n \left(1 + \frac{2}{3} + \frac{2^2}{3} + \dots +
\left(\frac{2}{3}\right)^{\log_3(n)-1} \right) + 2^{\log_3(n)}
$$

We make one note here about the leaf level. Notice that we can say:

$$
2^{\log_3(n)} = \frac{n}{n} 2^{\log_3(n)} = n
\frac{2^{\log_3(n)}}{3^{\log_3(n)}} = n \left( \frac{2}{3} \right)^{\log_3(n)}
$$

This looks exactly like our pattern! What's happening? What we've discovered is
that our leaf level \textit{obeys} our recurrence, and I claim that this is in
general true if the work at the level matches the work function. For example,
the work function $w(n)$ is $n$ here. The base case is at $n = 1$, therefore
looking at $w(1)$ we see it's $1$. \textit{This matches our base case}, i.e.
that at $n = 1, f(n) = 1$. Whenever this happens, it's unecessary to compute the
leaf level seperately, and we can pull it into our summation. Rough reasoning
is that it's modeled perfectly by the recurrence. If we had it that when $n =
1, f(n) = 5$, we could not do this. Therefore we can write:

$$
f(n) = n \left(1 + \frac{2}{3} + \frac{2^2}{3} + \dots +
\left(\frac{2}{3}\right)^{\log_3(n)} \right)
$$

Let's solve a couple different types of common recurrences you might see.

\subsubsection{Basic Recurrences}

\begin{enumerate}[(1)]

\item Solve the recurrence:

$$
A(n) = \begin{cases}
1 & n = 1 \\
n + 2A(n/2) & \text{otherwise}
\end{cases}
$$

\begin{proof}[Solution]

For this we have work function $f(n) = n$, decay rate $2$ and branching factor
$2$. The tree we draw looks like:

\Tree
[.$n$
	[.$n/2$
		[.$n/4$ $\vdots$ $\vdots$ ]
		[.$n/4$ $\vdots$ $\vdots$ ]
	]
	[.$n/2$
		[.$n/4$ $\vdots$ $\vdots$ ]
		[.$n/4$ $\vdots$ $\vdots$ ]
	]
]

Summing across, we come across the surprising fact that each level sums to $n$.
This is roughly because the decay rate and the branching factor cancel each
other out. In general, we find this to be true when $f(\delta(n)) = \beta$ where
$\beta$ is the branching factor and $\delta$ is the decay rate as a function of
$n$ (can you see why?). Ok, so now considering the tree, we can ask how many
terms are there? Recall this is solving for $\frac{n}{2^k} = 1 \implies k =
\log_2(n)$. This tells us that there must be $\log_2(n) + 1$ levels ($k$ tells
us $k$th level, but count starting from $0$). Finally we notice that $f(1) =
A(1)$, which implies that we don't have to do the leaf level seperately.
Therefore our solution is:

$$
A(n) = n + n + \dots + n = n( \text{Number of levels}) = n(log_2(n)+1)
$$

\end{proof}

\item Solve the reucrrence:

$$
B(n) = \begin{cases}
5, & n = 1 \\
n + 2B(n/2), & \text{otherwise}
\end{cases}
$$

\begin{proof}[Solution]
The difference between this one at the last one is the leaf level. Here we have
that $B(1) \neq f(1)$, which implies we need to compute the leaf level
seperately. There are $2^{\log_2(n)} = n$ leaves, and each has work $5$, which
implies that the leaf level cost is $5n$. Therefore we can write our answer as:
$$
B(n) = nlog_2(n) + 5n
$$
\end{proof}

\subsubsection{Changing the Leaf Level}

What if we have a recurrence like:

$$
C(n) = \begin{cases}
8, & n = 8 \\
n + 3C(n/2), \text{otherwise}
\end{cases}
$$

In this case what's strange is that we've changed when we enter the leaf level.
The tree method, however, handles this perfectly. Before we asked when
$\frac{n}{b^k} = 1$ where $b$ was the decay rate. Here the base case is when $n
= 8$, so we instead ask $\frac{n}{2^k} = 8 = 2^3 \implies k = \log_2(n) - 3$.
Then the problem is solved in the same exact manner as in the basic recurrences.

\subsubsection{Multiple Decay Rates}

What happens if we have a recurrence like:

$$
D(n) = \begin{cases}
1, & n = 1 \\
n + D(n/2) + D(n/3), & \text{otherwise}
\end{cases}
$$

What we lose is the guarentee that all leaves live on the same level. In fact,
the end of the \textit{unrolling} will come sooner for the harsher decay rate,
and later for the kinder decay rate. In $D(n)$, we see that $\frac{n}{2^k} \to_k
1$ slower than $\frac{n}{2^k} \to_k 1$. To resolve this problem, what we do is
approximate; we say that our leaf level $k$ lives somewhere between the leaf
level defined by $2$ and $3$.

If we follow the decay rate $2$ all of the way down, we know that our leaf level
is $k_2 = \log_2(n)$. If we follow $3$, similarily we will recieve $k_3 =
\log_3(n)$. We say that our "leaf level" lives between these two, or $\log_3(n)
\leq k \leq \log_2(n)$.

Then to solve our problem we have the tree:

\Tree
[.$n$
	[.$n/2$
		[.$n/4$ $\vdots$ $\vdots$ ]
		[.$n/6$ $\vdots$ $\vdots$ ]
	]
	[.$n/3$
		[.$n/6$ $\vdots$ $\vdots$ ]
		[.$n/9$ $\vdots$ $\vdots$ ]
	]
]

We find that our summation looks like:

$$
D(n) \approx n\left[ 1 + \frac{5}{6} + \frac{5^2}{6^2} + \dots +
\left(\frac{5}{6}\right)^k \right],\quad  \log_3(n) \leq k \leq \log_2(n)
$$

\subsubsection{Strange Decay Rates}

What happens if we have a recurrence like:

$$
E(n) = \begin{cases}
1, & n = 1 \\
1 + 2E(\delta(n)), & \text{otherwise}
\end{cases}
$$

Lots of people get stuck on this, but I claim that our tree method handles this
perfectly. The tree itself looks like:

\Tree
[.$1$
	[.$1$
		[.$1$ $\vdots$ $\vdots$ ]
		[.$1$ $\vdots$ $\vdots$ ]
	]
	[.$1$
		[.$1$ $\vdots$ $\vdots$ ]
		[.$1$ $\vdots$ $\vdots$ ]
	]
]

The real question is, where is the leaf level? Before we asked for decay rate
$\delta(n) = \frac{n}{b}$, how many times do we have to apply it to get down to
the base case. This formulated as $\frac{n}{b^k} = 1 \implies k = \log_b(n)$.
Our question in general is for what $k$ does $\delta^k(n) = 1$, where $k$ isn't
a power but repeated function compositions. For example if $\delta(n) = n-1$,
then we would be asking $n - k(-1) = 1 \implies k = n-1$.

\end{enumerate}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 5.1 Eigenvalues and Eigenvectors of a Symmetric Matrix

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Compute the N eigenvalues and eigenvectors of an N $\times $ N symmetric matrix $A$.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[REAL]  \ {\bf A}(LDA,$\geq $N) [LDA $\geq $ N]{\bf , EVAL}($\geq N)$%
{\bf ,\newline
WORK}($\geq N)$

\item[INTEGER]  \ {\bf LDA, N, IERR}
\end{description}

Assign values to A(,), LDA, and N.

\begin{center}
\fbox{\begin{tabular}{@{\bf }c}
CALL SSYMQL(A, LDA, N, EVAL,\\
WORK, IERR)\\
\end{tabular}}
\end{center}

Results are returned in A(,) and EVAL().

\subsubsection{Argument Definitions}

\begin{description}
\item[A(,)]  \ [inout] On entry the locations on and below the diagonal of
this array must contain the lower-triangular elements of the N $\times $ N
symmetric matrix $A$. On return the eigenvectors of $A$ will be stored as
column vectors in the array A(,). These N eigenvectors will be mutually
orthogonal and of unit Euclidean length. The eigenvector stored in column
J will be associated with the eigenvalue stored in EVAL(J).

\item[LDA]  \ [in] Dimension of the first subscript of the array A(,). Require
LDA $\geq $ N.

\item[N]  \ [in] Order of the symmetric matrix $A$. N $\geq 1.$

\item[EVAL()]  \ [out] Array in which the N eigenvalues of $A$ will be stored
by the subroutine. The eigenvalues will be sorted with the algebraically
smallest eigenvalue first.

\item[WORK()]  \ [scratch] An array of at least N locations used as
temporary space.

\item[IERR]  \ [out] On exit this is set to~0 if the QL algorithm converges,
otherwise see Section E.
\end{description}

\subsubsection{Modifications for Double Precision}

Change SSYMQL to DSYMQL, and the REAL type statement to DOUBLE PRECISION.

\subsection{Examples and Remarks}

The following symmetric matrix $A$ is
given on page~55 of~\cite{Gregory:1969:ACM}.
\begin{equation*}
A=\left[
\begin{array}{rrrr}
5 & 4 & 1 & 1 \\
4 & 5 & 1 & 1 \\
1 & 1 & 4 & 2 \\
1 & 1 & 2 & 4
\end{array}
\right]
\end{equation*}
The eigenvalues of this matrix are 1, 2, 5, and 10. Unnormalized
eigenvectors associated with these eigenvalues are (1, -1, 0, 0),
(0, 0, -1, 1), (-1, -1, 2, 2), and (2, 2, 1, 1), respectively.

The code in DRSSYMQL, given below, computes the eigenvalues and eigenvectors
of this matrix. Output from this program is given in the file ODSSYMQL.

Before the call to SSYMQL, the matrix is saved in an array ASAV() in order
to compute the relative residual matrix $D$ defined as%
\begin{equation*}
D=\left( AW-W\Lambda \right) /\gamma
\end{equation*}
where $W$ is the matrix whose columns are the computed eigenvectors
of $A$, $\Lambda $ is the diagonal matrix of eigenvalues,
and $\gamma $ is the maximum-row-sum norm of $A$.

Recall that if ${\bf v}$ is an eigenvector, then so is $\alpha {\bf
v}$ for any nonzero scalar $\alpha$.  More generally, if an
eigenvalue, $\lambda$, of a symmetric matrix occurs with multiplicity
$k$, there will be an associated  $k$-dimensional subspace in which
every vector is an eigenvector for $\lambda$.  This subroutine will
return eigenvectors constituting an orthogonal basis for such an
eigenspace.

\subsection{Functional Description}

The implicit-shift QL algorithm implemented in this subroutine is based on
the Algol procedure given in \cite{Dubrelle:1968:IQR}, pp.~337--383.  The
code combines slightly modified EISPACK routines TRED2, and IMTQL2, see
\cite{Smith:1974:MER}.  Modifications made are minor changes to convert
the code to take advantage of Fortran~77; they should not affect results.
TRED2 uses Householder orthogonal similarity transformations to transform
the matrix $A$ to tridiagonal form.  IMTQL2 uses the QL algorithm with
implicit shifts to reduce the off-diagonal elements of the tridiagonal
matrix to a magnitude of approximately the last bit of the largest element
of $A$.

The resulting diagonal elements are the eigenvalues of $A$. The matrix of
eigenvectors is computed as the product of the orthogonal transformation
matrices used in transforming $A$ first to tridiagonal form and then to
(almost) diagonal form.

The eigenvalues are sorted in nondecreasing algebraic order and the
eigenvectors are permuted as necessary to correspond to the ordered
eigenvalues.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

If the QL algorithm fails to converge in 30~iterations on the $J^{th}$
eigenvalue the subroutine sets IERR $= J.$ In this case $J-1$ eigenvalues
and eigenvectors are computed correctly but the eigenvalues are not ordered.
If N $\leq $ 0 on entry, IERR is set to $-$1.  In either case an error
message is printed using IERM1 of Chapter 19.2 with an error level of 0,
before the return.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DSYMQL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
DIMQL, DSYMQL, ERFIN, ERMSG, IERM1, IERV1\rule[-5pt]{0pt}{8pt}}\\
SSYMQL & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
ERFIN, ERMSG, IERM1, IERV1, SIMQL, SSYMQL}\\
\end{tabular}

Converted by: F. T. Krogh, JPL, October~1991.

\begcode

\medskip\
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSSYMQL}\vspace{10pt}
\lstinputlisting{\codeloc{ssymql}}

\vspace{30pt}\centerline{\bf \large ODSSYMQL}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{ssymql}}

\end{document}


\chapter{Numeric algorithms}
\Label{cha:numeric}

The algorithms that we considered so far only \emph{compared}, \emph{read} or
\emph{copied} values in sequences.
In this chapter, we consider so-called \emph{numeric} algorithms of the
\cxx Standard Library \cite[\S 29.8]{cxx-17-draft} that use arithmetic
operations on \valuetype to
combine the elements of sequences.

\begin{listing}[hbt]
\begin{center}
\begin{minipage}{0.5\textwidth}
\begin{lstlisting}[style=acsl-block]
    #define VALUE_TYPE_MAX  INT_MAX
    #define VALUE_TYPE_MIN  INT_MIN
\end{lstlisting}
\end{minipage}
\end{center}
\vspace*{-0.5cm}
\caption{\Label{lst:value-type-limits}Limits of \valuetype}
\end{listing}

In order to refer to potential arithmetic overflows we introduce the
two constants shown in Listing~\ref{lst:value-type-limits}
which refer to the numeric limits of \valuetype
(see also \S\ref{sec:types}).

We consider the following algorithms.

\begin{itemize}

\item \iotai
writes sequentially increasing values into a range
(\S\ref{sec:iotai})

\item \accumulate
computes the sum of the elements in a range
(\S\ref{sec:accumulate})

\item \innerproduct
computes the inner product of two ranges
(\S\ref{sec:innerproduct})

\item \partialsum
computes the sequence of partial sums of a range
(\S\ref{sec:partialsum})

\item \adjacentdifference
computes the differences of adjacent elements in a range
(\S\ref{sec:adjacentdifference})

\item
Finally, in \S\ref{sec:partialsuminv} we show that under
appropriate preconditions the algorithms \partialsum and
\adjacentdifference are inverse to each other.

\end{itemize}

The formal specifications of these algorithms raise new questions.
In particular, we now have to deal with arithmetic overflows in \valuetype.

\clearpage

\input{numeric/iota}
\input{numeric/accumulate}
\input{numeric/inner_product}
\input{numeric/partial_sum}
\input{numeric/adjacent_difference}
\input{numeric/numeric_inverse}


%\newpage
\section{Nonlinear classifiable sets}
In the section, we will extend the linearly separable sets to the nonlinear case.
A natural extension is like what the kernel method does in SVM for binary case. We will introduce the so-called
feature mapping.

%Thus, we have the following natural extension for linearly separable by using feature mapping and original definition of linearly separable.

\begin{definition}[nonlinearly separable sets]
	These data sets $A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$ are called nonlinearly separable, if there exist
	a feature space $\mathbb{R}^{\tilde d}$ and a smooth (if it has derivatives of all orders) feature mapping
	\begin{equation}\label{key}
	\varphi: \mathbb{R}^d \mapsto \mathbb{R}^{\tilde d}
	\end{equation}
	such that
	\begin{equation}\label{key}
	\tilde A_i := \varphi(A_i) = \{ \tilde x ~|~ \tilde x = \varphi(x), x \in A_i \}, \quad i = 1, 2, \dots, k,
	\end{equation}
	are linearly separable.
\end{definition}

\begin{remark}$ $\\
	\begin{enumerate}
		\item This definition is  consistent with the definition of linearly separable as we can just take $\tilde d = d$ and $\varphi = {\rm id}$ if $A_1, A_2, \cdots, A_k$ are already linearly separable.
		\item The kernel method in SVM is mainly based on this idea for binary case (k=2) where they use kernel functions to approximate $\varphi(x)$.
		\item Most commonly used deep learning models are related to softmax mappings which means we can interpret these deep learning models as the approximation for feature mapping $\varphi$.
	\end{enumerate}
\end{remark}

\begin{theorem}
	$A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$ are nonlinearly separable  is equivalent to that there
	 exists a smooth classification function
	\begin{equation}\label{key}
	\psi: \mathbb{R}^d \mapsto \mathbb{R}^{k}
	\end{equation}
	such that for all $1\leq i \leq k$ and $ j \neq i$
	\begin{equation}\label{key}
	\psi_i (x) > \psi_j (x), \quad \forall x \in A_i.
	\end{equation}
\end{theorem}

\begin{proof}
	On the one hand, it is easy to see that if $A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$ are nonlinearly separable, we can take
	\begin{equation}\label{key}
	\psi(x) = \bm p(\varphi(x); \theta),
	\end{equation}
	where $\bm p(\varphi(x); \theta)$ is the softmax function for linearly separable sets $\varphi(A_i)$ for $i=1,2,\cdots,k$.

	On the other hand, let assume that $\psi$ is the smooth classification functions for $A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$.
	We can take $\varphi(x) = \psi(x)$ and then
	\begin{equation}\label{key}
	\varphi(A_1), \varphi(A_2), \cdots, \varphi(A_k) \subset \mathbb{R}^k \quad ( \tilde d = k),
	\end{equation}
	will be linearly separable. Recall the definition of softmax mapping in Definition \ref{softmax}, if we take $\theta = (I, 0)$ in softmax mapping $\bm p(x;\theta)$,
	then the monotonicity of $e^x$ shows that for all $i = 1:k$ and $ j \neq i$
	\begin{equation}\label{key}
	\bm p_i ( \varphi(x);\theta) = \frac{e^{\psi_i(x)}}{\sum_{i=1}^k e^{\psi_i(x)}} > \frac{e^{\psi_j(x)}}{\sum_{i=1}^k e^{\psi_i(x)}} = \bm p_j ( \varphi(x);\theta) , \quad \forall x \in A_i.
	\end{equation}

\end{proof}

Similar to linearly separable sets, we have the next lemma for $k=2$.
\begin{lemma}$A_1$ and $A_2 \subset \mathbb{R}^d$ are nonlinearly separable  is equivalent that there
	exists a function $\varphi: \mathbb{R}^d \mapsto \mathbb{R}$ such that
	\begin{equation}\label{NonlinearBinary}
	\varphi(x) > 0 \quad \forall x \in A_1 \quad \text{and} \quad 	\varphi(x) < 0 \quad \forall x \in A_2.
	\end{equation}
\end{lemma}

\begin{proof}
On the one hand, based the equivalence of nonlinearly separable sets, there exists $\psi_1(x)$ and $\psi_2(x)$ such that
for all $i = 1:2$ and $ j \neq i$
\begin{equation}\label{key}
\psi_i (x) > \psi_j (x), \quad \forall x \in A_i.
\end{equation}
Then, we can just take
\begin{equation}\label{key}
\varphi(x) = \psi_1 (x) - \psi_2 (x).
\end{equation}

On the other hand, if there exist $\varphi(x)$ satisfies (\ref{NonlinearBinary}), then we can construct $\psi_1(x)$ and $\psi_2(x)$ as
\begin{equation}\label{key}
\psi_1(x) =  \frac{1}{2}\varphi(x) \quad \text{and}\quad
\psi_2(x) = -\frac{1}{2}\varphi(x).
\end{equation}
\end{proof}

%\begin{definition}[nonlinearly separable]
%	These data sets $A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$ are called nonlinearly separable, if there exist
%	a infinitely differentiable classification function
%	\begin{equation}\label{key}
%	\psi: \mathbb{R}^d \mapsto \mathbb{R}^{k}
%	\end{equation}
%	such that for all $i = 1:k$ and $ j \neq i$
%	\begin{equation}\label{key}
%	\psi_i (x) > \psi_j (x), \quad \forall x \in A_i.
%	\end{equation}
%\end{definition}

\begin{remark}
%This definition is consistent with the original definition of linearly separable.
%If $A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$ are linearly separable, then there exists a $\theta$
%and softmax mapping $\bm p(x; \theta)$ such that for $i = 1:k$ and $ j \neq i$
%\begin{equation}\label{key}
%\bm p_i (x;\theta) > \bm p_j (x;\theta), \quad \forall x \in A_i.
%\end{equation}
%Thus, $A_1, A_2, \cdots, A_k$ are also nonlinearly separable as we can take $\psi(x) = \bm p(x;\theta)$.
Here the only assumption is that for all $i = 1:k$ and $ j \neq i$, we have $\psi_i (x) > \psi_j (x)$, $\forall x \in A_i$ for nonlinearly separable. We do not assume that $\psi_i (x) \ge 0$ or $\displaystyle\sum_{i=1}^k \psi_i (x) = 1$, which means that
$$\psi(x) = \begin{pmatrix}
\psi_1 (x), &\psi_2 (x), &\cdots, &\psi_k (x)
\end{pmatrix}^T$$
is not a discrete probability distribution over all k classes.
\end{remark}

The previous theorem shows that the softmax function is not so crucial in nonlinearly separable case. Combined with deep learning models, we have the
following understanding about what deep learning models.
\begin{enumerate}
	\item If  the classification model is followed with a softmax, then it is approximating the feature mapping $\varphi: \mathbb{R}^d \mapsto \mathbb{R}^{\tilde d}$.
	\item If the classification model dose not followed by a softmax, then it is approximating $\psi: \mathbb{R}^d \mapsto \mathbb{R}^{k}$ directly.
\end{enumerate}

 \begin{example}
 	Consider $k=2$ and
	$$
	A_1 \subset \{ (x_1, x_2) | x_1^2 + x_2^2 < 1\}, \quad A_2 \subset  \{(x_1, x_2) | x_1^2 + x_2^2 > 1\},
	$$
	then we have the following nonlinear
 	 feature mapping:
 	 \begin{equation*}
 	 \adjustbox{valign=c}{\includegraphics[width=.25\textheight,height =0.4\textheight]{figures/nonlinear2sets.png}} \xrightarrow[\text{feature map}]{\varphi(x,y) = x_1^2 + x_2^2}
 	 \adjustbox{valign=c}{\includegraphics[width=.25\textheight]{figures/nonlinear2sets1D.png}}
 	 \end{equation*}
 \end{example}

Here we have the following comparison between linear and nonlinear models from the viewpoint of
loss functions:

\begin{description}
\item[Linear case (Logistic regression):]
$$
L_{\lambda }(\theta) = \sum_{j=1}^N \ell (y_j, p(x_j; \theta)) + \lambda R(\|\theta\|),
$$
as defined in \eqref{eq:logisticlambda}.
\item[Nonlinear case: ]
$$
L_{\lambda }(\theta) = \sum_{j=1}^N \ell (y_j, p( \varphi(x_j; \theta_1); \theta_2)) + \lambda R(\|\theta\|).
$$
\end{description}
Here $p(x; \theta) = {\rm softmax}(Wx + b)$ where $\theta = (W,b)$,
and $\theta = (\theta_1, \theta_2)$ for the nonlinear case.
For both cases, $\displaystyle \ell(q, p) = \sum_{i=1}^k - q_i \log p_i$  represents the cross-entropy, and $\lambda R(\|\theta\|)$ is the  regularization term.
%\begin{remark}We have the following remarks.
%\begin{enumerate}
%	\item $\ell(q, p) = \sum_{i=1}^k - q_i \log p_i \leftrightarrow$  cross-entropy
%	\item $p(x; \theta) = {\rm softmax}(Wx + b)$ where $\theta = (W,b)$
%	\item $\theta = (\theta_1, \theta_2)$ for nonlinear case
%	\item $\lambda R(\|\theta\|)$ $\leftrightarrow$  regularization term
%\end{enumerate}
%\end{remark}

In general, we have the following popular nonlinear models for $\varphi(x;\theta)$:
\begin{enumerate}
	\item Polynomials.
	\item Piecewise polynomials (finite element method).
	\item Kernel functions in SVM, see in Section \ref{sec:SVM}.
	\item Deep neural networks.
\end{enumerate}

\endinput

Based on the theorem of partition of unit, we may also have the next definition for
nonlinear classifiable.
\begin{definition}[nonlinearly separable via partition of unit]
	These data sets $A_1, A_2, \cdots, A_k \subset \mathbb{R}^d$ are called nonlinearly separable,
	if there exist smooth
	\begin{equation}\label{key}
	\varphi_i: \mathbb{R}^d \mapsto [0, 1], \quad i = 1, 2, \cdots, k,
	\end{equation}
	such that
	\begin{equation}\label{key}
	\varphi_i(x) = 1, x \in A_i  \text{ and } \varphi_i(0) = 1, x \in A_j,    \text{ for } i, j= 1, 2, \cdots, k \text{ but } j \neq i,
	\end{equation}
	and
	\begin{equation}\label{key}
	\sum_{i=1}^k \varphi_i(x) = 1, \quad \forall x \in \mathbb{R}^d.
	\end{equation}
	are linearly separable.
\end{definition}

\begin{remark}
	For these next observations for definition of nonlinear separable via partition of unit.
	\begin{enumerate}
		\item Nonlinearly separable via partition of unit is a special case of nonlinear separable via feature mapping as
		we can always choose $\tilde d = k$ and
		\begin{equation}\label{key}
		\varphi(x) = \begin{pmatrix}
		\varphi_1(x) \\
		\varphi_2(x) \\
		\vdots \\
		\varphi_k(x)
		\end{pmatrix}.
		\end{equation}
		Then it is easy to verify that $\{\varphi(A_i)\}_{i=1}^k$ are linearly separable.
		\item Based on the properties of partition of unit, we can prove that if $A_i$ are
		finite sets, then there must exist the partition of unit which makes them must be nonlinearly separable.
%		\item
	\end{enumerate}
\end{remark}

\subsection{Decision boundary}


\section{Skeletonization}

\noindent
\textbf{Volume Downsampling}
We downsample the segmentation data to a resolution of $30$ nm in each direction.
The skeletonization algorithm that we use takes as a parameter the minimum distance between joints in the skeleton~\cite{zhao2014automatic}.
When we downsample, we also reduce this minimum distance by the same rate.
We find that this produces expressive skeletons at a fraction of the computational cost.
Table~\ref{table:skeleton} shows statistics comparing the skeletonization algorithm on 200 randomly chosen labels.

\begin{table}
	\scriptsize
	\begin{center}
		\begin{tabular}{c c c c c} \hline
			& \textbf{Average Time} &  \textbf{Skeleton Length} & \textbf{No. Branches} & \textbf{No. Endpoints} \\ \hline
			Full Resolution & 16.35s & 1,352,157nm & 9,858 & 1,068 \\
			Downsampled & 0.56s & 1,671,138nm & 9,359  & 1,824 \\ \hline
		\end{tabular}
	\caption{The results of the skeletonization algorithm on both the full resolution and the downsampled data. Downsampling the data provides expressive skeletons at a fraction of the computational cost.}
	\label{table:skeleton}
	\end{center}
\end{table}

\begin{figure}
	\begin{center}
		\includegraphics[width=0.45\linewidth]{./figures/node-threshold.png}
		\caption{The number of remaining nodes after increasing the threshold (blue) and the number of voxels excluded as a percent of the total volume (green).}
		\label{fig:node-pruning}
	\end{center}
\end{figure}

\noindent
\textbf{Node Generation}
We remove all nodes from the graph corresponding to labels with fewer than $t_{seg}$ voxels.
Figure \ref{fig:node-pruning} shows the results of varying $t_{seg}$ on two different quantities for the Kasthuri training volume.
The blue line indicates the number of nodes remaining in the graph.
The rate of node reduction decreases for larger thresholds since there are fewer labels of larger size.
The green line shows the percent of voxels with a label pruned from the graph.
Ideally this number is low since we want to remove small segments which do not contribute much to the overall volume.
This ``volume lost" grows at an increasing rate as the larger segments are removed.
Based on these curves we set $t_{seg} = 20,000$ voxels.
With this threshold, we prune over half of the labels and only lose $1.5\%$ of the total volume.