\chapter{Minimum Search on Rotated Array}
The objective of this challenge is to find the smallest element in an array
that was rotated at a random index after being sorted. Duplicate elements are
not allowed for this challenge.

\section{Solution}
\imb[3][11]{\alg/rotatedarray.cpp}
\imb[34][39]{\alg/rotatedarray.cpp}
\imb[42][43]{\alg/rotatedarray.cpp}

By using a binary-search method with three nodes, the code above is successful
in finding and returning the smallest element. This algorithm uses recursive
functions, and shows the time complexity of the following:
\[ T(n)=2T\left(\frac{n}{2}\right)+\alpha=4T\left(\frac{n}{4}\right)+3\alpha=
\cdots=2^kT\left(\frac{n}{2^k}\right)+\frac{k(k+1)}{2}\alpha,~\therefore T(n)=
O(\log n) \]

\iffalse
\section{With Duplicate Elements}
With duplicate elements, the time complexity cannot be kept at $\log n$. Follow-
up explanations for rotated arrays with duplicates will soon be committed.
\fi


\problem{Induction 1}

Let $n \in \mathcal{N}$. Conjecture a formula for:

$$a_n = \frac{1}{(1)(2)} + \frac{1}{(2)(3)} + ... + \frac{1}{(n)(n + 1)}$$

\solution

\part

\begin{theorem}
The formula $a_n = \frac{n}{n + 1}$ describes the summation.
\end{theorem}
\begin{proof}
Proceeding by induction, it is first established that the formula works for $a_1$, because $a_1 = \frac{1}{(1) * (2)} = \frac{1}{2}$.
Then, suppose the formula is true for an arbitrary $n \geq 1$, i.e. that $a_n = \frac{n}{n + 1}$.
From this, $a_{n + 1} = \frac{n}{n + 1} + \frac{1}{(n + 1)(n + 2)} = \frac{n^2 + 2n + 1}{(n + 1)(n + 2)} = \frac{(n + 1)(n + 1)}{(n + 1)(n + 2)} = \frac{n + 1}{n + 2}$.
By induction, this formula describes the summation $a_n$.

\end{proof}

\subsection{Scratch work}

$a_1 = (1/2)$ \\
$a_2 = (1/2) + (1/6) = 4/6 = 2/3$ \\
$a_3 = (1/2) + (1/6) + (1/12) = 9/12 = 3/4$ \\

A likely formula is $a_n = (n/n+1)$


\section{Math Extra}
\subsection{Combinatorial formulas}

 $\sum_{k=0}^{n}k^{2}=n(n+1)(2n+1)/6$\\
 $\sum_{k=0}^{n}k^{3}=n^{2}(n+1)^{2}/4$\\
 $\sum_{k=0}^{n}k^{4}=(6n^{5}+15n^{4}+10n^{3}-n)/30$\\
 $\sum_{k=0}^{n}k^{5}=(2n^{6}+6n^{5}+5n^{4}-n^{2})/12$\\
 $\sum_{k=0}^{n}x^{k}=(x^{n+1}-1)/(x-1)$\\
 $\sum_{k=0}^{n}kx^{k}=(x-(n+1)x^{n+1}+nx^{n+2})/(x-1)^{2}$\\
 ${n \choose k}=\frac{n!}{(n-k)!k!}$\\
 ${n \choose k}={n-1 \choose k}+{n-1 \choose k-1}$\\
 ${n \choose k}=\frac{n}{n-k}{n-1 \choose k}$\\
 ${n \choose k}=\frac{n-k+1}{k}{n \choose k-1}$\\
 ${n+1 \choose k}=\frac{n+1}{n-k+1}{n \choose k}$\\
 ${n \choose k+1}=\frac{n-k}{k+1}{n \choose k}$\\
 $\sum_{k=1}^{n}k\tbinom{n}{k}=n2^{n-1}$\\
 $\sum_{k=1}^{n}k^{2}\tbinom{n}{k}=(n+n^{2})2^{n-2}$\\
 ${m+n \choose r}=\sum_{k=0}^{r}{m \choose k}{n \choose r-k}$\\
 ${n \choose k}=\prod_{i=1}^{k}\frac{n-k+i}{i}$\\

\subsection{Number theory identities}
\textbf{Lucas' Theorem:} For non-negative integers $m$ and $n$ and a prime $p$,

$$\binom{m}{n}\equiv\prod_{i=0}^k\binom{m_i}{n_i}\pmod p,$$
where
$$m=m_kp^k+m_{k-1}p^{k-1}+\cdots +m_1p+m_0$$
is the base $p$ representation of $m$, and similarly for $n$.

\subsection{Stirling Numbers of the second kind}
Number of ways to partition a set of $n$ numbers into $k$ non-empty subsets.

$${n \brace k}=\frac{1}{k!}\sum_{j=0}^{k}(-1)^{(k-j)}{k \choose j}j^n$$

Recurrence relation:

  $${0 \brace 0}=1$$
  $${n \brace 0}={0 \brace n}=1$$
  $${n+1 \brace k}=k{n \brace k}+{n \brace k-1}$$

\subsection{Burnside's Lemma}
Let $G$ be a finite group that acts on a set $X$. For each $g$ in $G$ let $X^g$ denote the set of elements in $X$ that are fixed by $g$, which means $X^g=\{x\in X| g(x)=x\}$. Burnside's lemma assers the following formula for the number of orbits, denoted $|X/G|$:
\begin{align*}
|X/G|=\frac{1}{|G|} \sum_{g\in G} |X^g|
\end{align*}

\subsection{Numerical integration}
RK4: to integrate $\dot{y} = f(t, y)$ with $y_0 = y(t_0)$, compute
\begin{align*}
  k_1 &= f(t_n, y_n) \\
  k_2 &= f(t_n + \frac h 2, y_n + \frac h 2 k_1) \\
  k_3 &= f(t_n + \frac h 2, y_n + \frac h 2 k_2) \\
  k_4 &= f(t_n + h, y_n + h k_3) \\
  y_{n+1} &= y_n + \frac h 6 (k_1 + 2k_2 + 2k_3 + k_4)
\end{align*}


% !TEX root = report.tex
\section{Introduction}

\subsection{Basic definition of database system}

In database systems, a \textbf{database} is a collection of tables, and each \textbf{table} stores one coherent dataset. Each row of a table represents a single data point and each column represents an attribute of the data. Theoretically speaking, a table is merely a mathematical \textbf{relation} over one or more sets of scalar values (numbers, strings, etc.) and each row of the table is a \textbf{tuple} (or an element) in such relation.

For the sake of keeping our explanation simple, the relationship of data across different tables can be expressed by declaring \textbf{keys} (usually referred as IDs) as explicit columns in the table. Each row of a table may have a \textbf{primary key} declared as an anchor point to which other tables can refer to with \textbf{foreign keys}. In practice, since integers can be used as keys, it is sufficient to portray keys no different than other scalar values. Therefore, we omit the keys from our discussion for the rest of this project.

\smallskip
\marginhead{First example of database}
For example, let us say that we would like to construct a database to store information of four students: Alice, Bob, Carol, and David. The database must contain their personal data (e.g. year of birth) as well as a network graph representing their friendship relation. One possibility is that we create two tables, described below.

\newrobustcmd\nameA{\field{Name\hrsp{}A}}
\newrobustcmd\nameB{\field{Name\hrsp{}B}}
\newrobustcmd\personaldata{\rel{PersonalData}(\field{Name},\field{BirthYear})}
\newrobustcmd\friendship{\rel{Friendship}(\nameA,\nameB)}

\begin{itemize}[topsep=0.5pc,itemsep=0.25pc]
    \item  $\personaldata$: a two-column table containing each student's name and year of birth;
    \item  $\friendship$: a (possibly non-symmetric) two-column table where each row contains names and only names of two students who are friends of each other.
\end{itemize}
An example of an instance of such database are shown in \autoref{tab:age} and \autoref{tab:friend}, and the visualization is shown in \autoref{fig:alicebob}.

\smallskip
\begin{table}[!h]
    \small\centering
    \begin{minipage}{0.45\linewidth}
        \centering
        \begin{tabular}{ll}
            \toprule
            \field{Name} & \field{BirthYear} \\
            \midrule
            \str{Alice} & \val{1994} \\
            \str{Bob} & \val{1995} \\
            \str{Carol} & \val{1994} \\
            \str{David} & \val{1993} \\
            \bottomrule
        \end{tabular}
        \caption{Table \rel{PersonalData} with four students: Alice, Bob, Carol, and David, and their corresponding year of birth.}
        \label{tab:age}
    \end{minipage}
    \begin{minipage}{0.45\linewidth}
        \centering
        \begin{tabular}{ll}
            \toprule
            \nameA & \nameB \\
            \midrule
            \str{Alice} & \str{Bob} \\
            \str{Bob} & \str{Carol} \\
            \str{Alice} & \str{Carol} \\
            \str{Carol} & \str{David} \\
            \bottomrule
        \end{tabular}
        \caption{Table \rel{Friendship} showing that Alice, Bob, and Carol know each other, but David knows only Carol.}
        \label{tab:friend}
    \end{minipage}
\end{table}
\begin{figure}[!h]
    \small\centering
    \begin{minipage}[b]{0.35\linewidth}
        \caption{The visualization of data in the table \rel{PersonalData} and \rel{Friendship}.}
        \label{fig:alicebob}
    \end{minipage}
    \begin{minipage}[b]{0.55\linewidth}
        \includegraphics[width=0.8\linewidth]{figures/alicebob.eps}
    \end{minipage}
\end{figure}

\newpage
\subsection{Database query model}

One of the most basic database function is the \textbf{query} function, i.e., the process of fetching the stored data from the database. SQL model is an example of one of the most practical query model adopted in industry. Nonetheless, it is also useful to discuss theoretical query models (such as \textbf{relational calculus} and \textbf{relational algebra}) in order to gain a better understanding of database query models.

\marginhead{Definition of Domain Relational Calculus}In this report, we mainly discuss one particular variant of relational calculus query model, namely the \textbf{domain relational calculus} (\textsc{drc}). In this model,
\begin{itemize}[topsep=0.5pc,itemsep=0.25pc]
    \item  We use set comprehension notation (i.e., set defined by a predicate) to construct a query result. Such predicate must be in first-order logic.
    \item  Each identifier in the set comprehension must represent a scalar value in the database (as opposed to representing a tuple in tuple relational calculus).
\end{itemize}

\phantomsection
\label{psec:table-name-pred}
\marginhead{Table names as predicates}
As part of the predicate in set comprehension, the following notation regarding the existence of a tuple in a database table is allowed.

\begin{notation}
    Suppose that $R$ is a database table with $m$ columns. Let $\boldsymbol{x} = \langle x_1, x_2, \ldots, x_m \rangle$ be an $m$-tuple, where each $x_i$ represents a scalar value. Then $R(\boldsymbol{x})$ denotes a predicate whose value is true \emph{if and only if} the table $R$ contains the tuple $\boldsymbol{x}$. Sometimes we also write $R(x_1, x_2, \ldots, x_m)$ directly instead.
\end{notation}

For example, based on the data from \autoref{tab:age}, we could say that $\rel{PersonalData}(\str{Alice}, \val{1994})$ is true whereas $\rel{PersonalData}(\str{Bob}, \val{1993})$ is false.

\marginhead{Examples of \textnormal{\textsc{drc}} queries}
Here are some examples of \textsc{drc} queries.

\smallskip
\begin{example}
    Suppose that we want to obtain all students and their year of birth who were born \emph{strictly} before \val{1995} (using the database table $\personaldata$ as described in \autoref{tab:age}). We use the following \textsc{drc} query.
    \[
        Q_\text{\,before\,\val{1995}} = \{\var{name}, \var{year} \mid
            \rel{PersonalData}(\var{name}, \var{year}) \wedge (\var{year} < \val{1995})\}
    \]
    This query returns a set of three tuples: $\{(\str{Alice},\val{1994}),(\str{Carol},\val{1994}),(\str{David},\val{1993})\}$.
\end{example}

\begin{example}
    Suppose that we want to obtain all friends of Bob (based on the database table $\friendship$ from \autoref{tab:friend}). We use the following \textsc{drc} query.
    \[
        Q_\text{\,Bob's\,friend} = \{\var{name} \mid
            \rel{Friendship}(\var{name}, \str{Bob}) \vee
            \rel{Friendship}(\str{Bob}, \var{name})\}
    \]
    This query returns a set of two elements: $\{\str{Alice},\str{Carol}\}$. \emph{Note that we sometimes omit the tuple notation when it contains only a single column.}
\end{example}

\newpage
\begin{example}
    Suppose that we want to obtain all pairs of students who share a common friend using the same data as above. We use the following query.
    \begin{align*}
        Q_\text{\,friend\,of\,friend} = \{ x, y \mid (x < y) \wedge \exists z[
                &(\rel{Friendship}(x, z) \vee \rel{Friendship}(z, x)) \\
                &\quad\wedge (\rel{Friendship}(y, z) \vee \rel{Friendship}(z, y))
            ]\}
    \end{align*}
    This query would return $\{(\str{Alice},\str{Bob}),(\str{Alice},\str{Carol}),(\str{Bob},\str{Carol}),(\str{Alice},\str{David}),$ $(\str{Bob},\str{David})\}$. \emph{This query utilizes the existential quantification in first-order logic to represent a friend in common between two students. Also assume that $<$ does a lexicographical comparison of two strings.}
\end{example}

\medskip
\phantomsection
\label{psec:explicit-domain}
Notice how the domain for variables (such as \var{name} and \var{year}) are not explicit in the query. This is okay because predicates \rel{PersonalData} and \rel{Friendship} provides the \emph{finite bound} of what could be in the result of the query. We discuss this in more depth in the next subsection.

\subsection{Safety in \textsc{drc} queries}

\marginhead{Motivated example of unsafe queries}
\noindent
Let us consider the query $Q_\text{\,before\,\val{1995}}$ from above once again. You may have noticed that, at least in the paradigm of database systems, it would \emph{not} make practical sense to make such query but \emph{without} the predicate $\rel{PersonalData}(\var{name}, \var{year})$. If it was the case, then the result of the query would also have included tuples that are \emph{not} from the table \rel{PersonalData} in order to be consistent with the mathematical definition of set notation.

More concretely, the result of
\[
    Q^*_\text{\,before\,\val{1995}} = \{\var{name}, \var{year} \mid
        \var{year} < \val{1995}\}
\]
would change depending on what is defined as the scope or the domain of the scalar values. For example, if all nonempty strings are allowed for student names, and all integers are allowed for year of birth, then $(\str{Eve}, \val{-80})$ would be part of the query result. However, if the domain only allows positive integers for year of birth, then the same tuple would \emph{not} appear in the result.

\marginhead{Definition of safe vs. unsafe query}
We call queries like $Q^*_\text{\,before\,\val{1995}}$ \textbf{unsafe} as their result changes when the domain changes (i.e., results are \textbf{domain-dependent}). On the other hand, queries such as $Q_\text{\,before\,\val{1995}}$ and $Q_\text{\,Bob's\,friend}$ are examples of \textbf{safe} queries because the result of the query is always the same no matter what the scope of the domain is. In other words, their results rely only on data in the database. See Chapter 5 of \emph{Foundations of Databases} \cite[p.~75]{Abiteboul:1995:FDL:551350} for more information.

The concept of domain-dependency was summarized by Fagin in his paper \cite{Fagin:1982:HCD:322344.322347} and it was historically different from the \emph{original} meaning of safety of query formulae as introduced by Ullman \cite{Ullman:1983:PDS:538906}. For this project, we adopt a certain assumption \cite{Abiteboul:1988:inria-00075707} so we refer to a query as \textbf{safe} \emph{interchangeably} with the term \textbf{domain-independent}.

\newpage
\marginhead{More examples of unsafe queries}
Now let us consider a few more examples of unsafe queries. Assume that we have a database table $\rel{Follows}(\field{fan}, \field{idol})$ representing the fact that \field{fan} is following \field{idol} on a social network. Consider the following queries in \textsc{drc}.

\smallskip
\begin{example}
    \label{ex:not-follow}
    \[
        Q_\text{\,not\,following\,Alice} =
            \{ x \mid \neg\,\rel{Follows}(x, \str{Alice}) \}
    \]
    The first query, $Q_\text{\,not\,following\,Alice}$, returns a set of all people who did not follow Alice. It is unsafe because, for some database instance (such as when no one follows Alice), the result of the query will be every person within the domain. Hence, the result would be domain-dependent.
\end{example}

\begin{example}
    \[
        Q_\text{\,weird\,pairing} =
            \{ x, y \mid \rel{Follows}(x, \str{Alice}) \vee \rel{Follows}(y, \str{Bob}) \}
    \]
    The second query, $Q_\text{\,weird\,pairing}$, returns a set of pairs of people such that the first person follows Alice, or that the second person follows Bob. This query is also not safe. One counterexample is that no one follows Alice but Alice is the only person who follows Bob. Then $(x, \str{Alice})$ will appear in result for every $x$ in the domain of people, and thus is domain-dependent.
\end{example}

\begin{example}
    \label{ex:follow-all}
    \[
        Q_\text{\,follows\,all} =
            \{ x \mid \forall y[\rel{Follows}(x, y)] \}
    \]
    The third query $Q_\text{\,follows\,all}$ returns a set of people who follows everyone in the domain. Notice how the result for this query is guaranteed to be bounded even if the domain was infinite. However, suppose that Alice follows everyone in the domain $D_1$. If the domain $D_2$ is defined as $D_2 = D_1 \cup \{\str{Bob}\}$ where $\str{Bob} \not\in D_1$, then Alice would appear in the result when the domain $D_1$ is used, but would \emph{not} if $D_2$ is used. Therefore, query $Q_\text{\,follows\,all}$ is domain-dependent.
\end{example}

\subsection{Project goal and previous work}
\label{sub:goal}

The goal for this project is to study and analyze the safety of a query in \textsc{drc} model for a specified database schema using Alloy Analyzer (introduced in the next section). Specifically,

\begin{quote}
    \makebox[0pt][r]{``}Given a database schema $S$ and a query $Q$, does there exist a database instance under two \emph{distinct} domains $D_1$ and $D_2$ such that the respective results $R_1$ and $R_2$ differ. If so, then the query $Q$ is considered unsafe; otherwise it is safe.''
\end{quote}

The verification of data models using Alloy Analyzer has been investigated in prior work, including the data models whose specifications are in relational database schemata \cite{Cunha:2009:MAS:1685167.1685438} and other specification language such as \textsc{ora-ss} \cite{Wang:2006:VOD:1129030.1129454}.
The verification of data models in the context of web applications has also been studied \cite{Nijjar:2011:BVR:2001420.2001429,Nijjar:2015:DMP:2820114.2699691}. However, none of them involved the modeling of database relational queries.
Also note that we traditionally use Codd's Theorem \cite{codd1972relational} to verify whether a \textsc{drc} query is safe by providing an equivalent relational algebra query statement, which is outside the scope of this project.

Although relational calculus is not a practical way to make a query to the database, this project should provide an insight into the fundamental concepts of database theory as well as provide us a framework to help us on determining whether a \textsc{drc} query is safe, a task usually tediously done by humans due to its complexity.


\chapter{Logistic Regression}
\label{ch:logistic_regression}

Logistic regression is one of the best-known classifiers. The model returns the probability of a class variable, based on input features. First, it computes probabilities with a one-versus-all approach, meaning that for a multiclass problem, it will take one target value and treat all the rest as "other", effectively transforming the problem to binary classification.

Second, it tries to find an optimal plane that separates instances with the target value from the rest. Then it uses logistic function to transform the distance to the plane into probabilities. The further away from the plane an instance will be, the higher the probability it belongs to the class on that side of the plane. The closer it is to the decision boundary (the plane), the more uncertain the prediction becomes (i.e. it gets close to 0.5).

\begin{marginfigure}
    \includegraphics[width=\textwidth]{logistic-regression.png}%
    \caption{Can you guess what would the probability for belonging to the blue class be for A, B, and C?}
    \label{fig:logreg}
  \end{marginfigure}

Logistic regression tries to find such a plane that all points from one class are as far away from the boundary (in the correct direction) as possible.

\begin{marginfigure}
    \includegraphics[width=\textwidth]{workflow.png}%
    \label{fig:workflow}
  \end{marginfigure}

A great thing about \widget{Logistic Regression} is that we can interpret it with a \widget{Nomogram}. Nomogram shows the importance of variables for the model. The higher the variable is in the list, the greater its importance. Also, the longer the line, the greater the importance. The line corresponds to the coefficient of the variable, which is then mapped to the probability. You can drag the blue point on the line left or right, decreasing or increasing the probability of the target class. This will show you how different values affect the outcome of the model.

\begin{figure}[h]
    \centering
    \includegraphics[scale=0.4]{nomogram.png}
    \caption{$\;$}
\end{figure}

Another characteristic of logistic regression is that it observes all variables at once and takes the correlation into account. If some variables are correlated, their importance will be spread among them.

A not so great thing about logistic regression is that it operates with planes, meaning that the model won't work when the data cannot be separated in such a way. Can you think of such a data set?


\section*{Appendix}

In this part, several interesting constructions are listed,
while others are left in \mintinline{bash}|./libs/Init/Prelude.v|.

\subsection*{Boolean}
\begin{center}
\begin{minted}{coq}
Inductive True : Type :=
| I : True.

Inductive False : Type :=.

Inductive and (A : Type) (B : Type) : Type :=
| conj : A -> B -> and A B.

Inductive or (A : Type) (B : Type) : Type :=
| or_introl : A -> or A B
| or_intror : B -> or A B.
\end{minted}
\end{center}

\subsection*{Logic}
\begin{center}
\begin{minted}{coq}
Definition If_then_else (P : Type) (Q : Type) (R : Type) : Type :=
    or (and P Q) (and (not P) Q).

Definition not (A : Type) : Type :=
    A -> False.

Definition iff (A : Type) (B : Type) : Type :=
    and (A -> B) (B -> A).

Inductive ex (A : Type) (P : A -> Type) : Type :=
| ex_intro : forall (x : A), P x -> ex A P.

Inductive eq (A : Type) (x : A) : A -> Type :=
| eq_refl : eq A x x.
\end{minted}
\end{center}

\subsection*{Natural Number}
\begin{center}
\begin{minted}{coq}
Inductive nat : Type :=
| O : nat
| S : nat -> nat.

Fixpoint plus (n : nat) (m : nat) : nat :=
    match n as n0 in nat return nat with
    | O => m
    | S n1 => S (plus n1 m)
    end.

Fixpoint multiply (n : nat) (m : nat) : nat :=
    match n as n0 in nat return nat with
    | O => O
    | S n1 => plus m (multiply n1 m)
    end.

Fixpoint power (n : nat) (m : nat) : nat :=
    match m as m0 in nat return nat with
    | O => S O
    | S m1 => multiply n (power n m1)
    end.
\end{minted}
\end{center}

\subsection*{List}
\begin{center}
\begin{minted}{coq}
Inductive list (T : Type) : Type :=
| nil : list T
| cons : T -> list T -> list T.

Inductive iliist (T : Type) : nat -> Type :=
| inil : ilist T O
| icons : forall (n : nat), T -> ilist T n -> ilist T (S n).
\end{minted}
\end{center}


\section*{Binary Classification}
A set of $N_{\text{act}}$ data instances undergo binary classification
prediction.

\subsection*{Count}

\begin{itemize}
\item [true (positive, negative):] $N_{\text{tp}}, N_{\text{tn}}$
%\item [true negative:] $N_{\text{tn}}$
\item [false (positive, negative):] $N_{\text{fp}}, N_{\text{fn}}$
%\item [false negative:] $N_{\text{fn}}$
\item [actual positive:] $N_{\text{act-p}} = N_{\text{tp}} + N_{\text{fn}}$
\item [actual negative:] $N_{\text{act-n}} = N_{\text{tn}} + N_{\text{fp}}$
\item [predicted positive:] $N_{\text{prd-p}} = N_{\text{tp}} + N_{\text{fp}}$
\item [predicted negative:] $N_{\text{prd-n}} = N_{\text{tn}} + N_{\text{fn}}$
\item [predicted true:] $N_{\text{prd-t}} = N_{\text{tp}} + N_{\text{tn}}$
\item [predicted false:] $N_{\text{prd-f}} = N_{\text{fp}} + N_{\text{fn}}$
\item [predicted:] $N_{\text{prd}} = N_{\text{prd-p}} + N_{\text{prd-n}} =
  N_{\text{prd-t}} + N_{\text{prd-f}}$
\item [actual:] $N_{\text{act}} = N_{\text{prd}}$
\end{itemize}

\subsection*{Classify}
\begin{itemize}
\item [true positive rate:] $R_{\text{tp}} = N_{\text{tp}} / N_{\text{act-p}}$
\item [true negative rate:] $R_{\text{tn}} = N_{\text{tn}} /
  N_{\text{act-n}}$
\item [accuracy:] $R_{\text{accur}} = N_{\text{prd-t}} / N_{\text{act}}$
\item [error:] $R_{\text{error}} = N_{\text{prd-f}} / N_{\text{act}} = 1 - R_{\text{accur}}$
\item [precision:] $R_{\text{precis}} = N_{\text{tp}} / N_{\text{prd-p}}$
\item [recall:] $R_{\text{recall}} = N_{\text{tp}} / N_{\text{act-p}} = R_{\text{tp}}$
\item [sensitivity:] $R_{\text{sensit}} = N_{\text{tp}} / N_{\text{act-p}}
  = R_{\text{tp}} = R_{\text{recall}}$
\item [specificity:] $R_{\text{specif}} = N_{\text{tn}} / N_{\text{act-n}}
  = R_{\text{tn}}$
\item [positive likelihood:] $R_{\text{lkl-pos}} = R_{\text{sensit}} / (1 - R_{\text{specif}})$
\item [negative likelihood:] $R_{\text{lkl-neg}} = (1 -
  R_{\text{sensit}}) / R_{\text{specif}}$
\item [normalized likelihood:] \mbox{}\\%
  $R_{\text{lkl-norm}} = (\sqrt{3}/\pi)
  [\log(R_{\text{lkl-pos}}) + \log(R_{\text{lkl-neg}})]$
\item [Youden's index:] $R_{\text{Youden}} = R_{\text{sensit}} - (1 -
  R_{\text{specif}})$
\end{itemize}

\subsection*{Combine}
\begin{itemize}
\item [balanced classification:] $R_{\text{bcr}} = (R_{\text{tp}} +
  R_{\text{tn}})/2 = (R_{\text{sensit}} + R_{\text{specif}}) / 2$
\item [balanced error:] $R_{\text{ber}} = 1 - R_{\text{bcr}}$
\item [$F_1$-measure:] $F_1 = 2(R_{\text{precis}} R_{\text{recall}}) / (R_{\text{precis}} + R_{\text{recall}})$
\item [$F_{\beta}$-measure:] \mbox{}\\ $F_{\beta} = (1+\beta^2) (R_{\text{precis}}
  R_{\text{recall}}) / (\beta^2R_{\text{precis}} + R_{\text{recall}})$
\item [Matthew's correlation coefficient:] \mbox{}\\ $C_{\text{Matthews}} =
  (N_{\text{tp}}N_{\text{tn}} - N_{\text{fp}}N_{\text{fn}}) / %
  \sqrt{N_{\text{act-p}} N_{\text{prd-p}} N_{\text{act-n}} N_{\text{prd-n}}}$
\end{itemize}


\lab{Nearest Neighbor Search}{Nearest Neighbor Search}

\objective{The nearest neighbor problem is an optimization problem that arises in applications such as computer vision, internet marketing, and data compression.
The problem can be solved efficiently with a $k$-d tree, a generalization of the binary search tree.
In this lab we implement a $k$-d tree, use it to solve the nearest neighbor problem, then use that solution as the basis of an elementary machine learning algorithm.}

\section*{The Nearest Neighbor Problem} % =====================================

Let $X \subset \mathbb{R}^k$ be a collection of data, called the \emph{training set}, and let $\z \in \mathbb{R}^k$, called the \emph{target}.
The \emph{nearest neighbor search problem} is determining the point $\x^* \in X$ that is ``closest'' to $\z$.

For example, suppose you move into a new city with several post offices.
Since your time is valuable, you wish to know which post office is closest to your home.
The set $X$ could be addresses or latitude and longitude data for each post office in the city; $\z$ would be the data that represents your new home.
The task is to find the closest post office in $\x \in X$ to your home $\z$.

\subsection*{Metrics and Distance} % ------------------------------------------

Solving the nearest neighbor problem requires a definition for distance between $\z$ and elements of $X$.
In $\mathbb{R}^k$, distance is typically defined by the \emph{Euclidean metric}.
\begin{align}
\label{eq:nearest-neighbor-euclidean-metric}
d(\x,\z) = \|\x - \z\| = \sqrt{\sum_{i=1}^k(x_i-z_i)^2}
\end{align}
Here $\|\cdot\|$ is the standard \emph{Euclidean norm}, which computes vector length.
In other words, $d(\x,\z)$ is the length of the straight line from $\x$ to $\z$.
With this notation, the nearest neighbor search problem can be written as follows.
\begin{align}
\label{eq:nearest-neighbor-def}
\x^* = \underset{\x\in X}{\text{argmin}}\:d(\x,\z)
\qquad\qquad
d^* = \min_{\x\in X}d(\x,\z)
\end{align}

NumPy and SciPy implement the Euclidean norm (and other norms) in \li{linalg.norm()}.
This function accepts vectors or matrices.
Use the \li{axis} argument to compute the norm along the rows or columns of a matrix: \li{axis=0} computes the norm of each column, and \li{axis=1} computes the norm of each row (see the NumPy Visual Guide).

\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la

>>> x0 = np.array([1, 2, 3])
>>> x1 = np.array([6, 5, 4])

# Calculate the length of the vectors x0 and x1 using the Euclidean norm.
>>> la.norm(x0)
3.7416573867739413
>>> la.norm(x1)
8.7749643873921226

# Calculate the distance between x0 and x1 using the Euclidean metric.
>>> la.norm(x0 - x1)
5.9160797830996161

>>> A = np.array([[1, 2, 3],        # or A = np.vstack((x0,x1)).
...               [6, 5, 4]])
>>> la.norm(A, axis=0)              # Calculate the norm of each column of A.
array([ 6.08276253,  5.38516481,  5.        ])
>>> la.norm(A, axis=1)              # Calculate the norm of each row of A.
array([ 3.74165739,  8.77496439])   # This is ||x0|| and ||x1||.
\end{lstlisting}

\subsection*{Exhaustive Search} % ---------------------------------------------

Consider again the post office example.
One way to find out which post office is closest is to drive from home to each post office, measuring the distance travelled in each trip.
That is, we solve \eqref{eq:nearest-neighbor-def} by computing $\|\x - \z\|$ for every point $\x\in X$.
This strategy is called a \emph{brute force} or \emph{exhaustive search}.

\begin{problem} % Exhaustive search method.
Write a function that accepts a $m\times k$ NumPy array $X$ (the training set) and a 1-dimensional NumPy array $\z$ with $k$ entries (the target).
Each of the $m$ rows of $X$ represents a point in $\mathbb{R}^k$ that is an element of the training set.

Solve \eqref{eq:nearest-neighbor-def} with an exhaustive search.
Return the nearest neighbor $\x^*$ and its distance $d^*$ from the target $\z$.
\\(Hint: use array broadcasting and the \li{axis} argument to avoid using a loop.)
\label{prob:nearest-neighbor-exhaustive-search}
\end{problem}

The complexity of an exhaustive search for $X\subset\mathbb{R}^k$ with $m$ points is $O(km)$, since \eqref{eq:nearest-neighbor-euclidean-metric} is $O(k)$ and there are $m$ norms to compute.
This method works, but it is only feasible for relatively small training sets.
Solving the problem with greater efficiency requires the use of a specialized data structure.

\section*{K-D Trees} % ========================================================

% \footnote{This lab is a sequel to the Data Structures II lab, and should not be attempted until the reader has successfully implemented a binary search tree in Python.}
A \href{https://en.wikipedia.org/wiki/K-d_tree}{\emph{$k$-d tree}} is a generalized binary search tree where each node in the tree contains $k$-dimensional data.
Just as a BST makes searching easy in $\mathbb{R}$, a $k$-d tree provides a way to efficiently search $\mathbb{R}^k$.

A BST creates a partition of $\mathbb{R}$: if a node contains the value $x$, all of the nodes in its left subtree contain values that are less than $x$, and the nodes of its right subtree have values that are greater than $x$.
Similarly, a $k$-d tree partitions $\mathbb{R}^k$.
Each node is assigned a \emph{pivot} value $i \in \{0,1,\ldots,k-1\}$ corresponding to the depth of the node: the root has $i=0$, its children have $i=1$, their children have $i=2$, and so on.
If a node has $i=k-1$, its children have $i=0$, their children have $i=1$, and so on.
% In general, if a node has \li{pivot} $i$, its children have an \li{pivot} of $(i + 1) \mod k$.
The tree is constructed such that for a node containing $\x = [x_0,x_1,\ldots,x_{k-1}]\trp \in\mathbb{R}^k$, if a node in the left subtree contains $\y$, then $y_i < x_i$.
Conversely, if a node in the right subtree contains $\z$, then $x_i \le z_i$.
See Figure \ref{fig:nearest-neighbor-pivot-tree} for an example where $k = 3$.

% The case where $k=1$ results in a usual BST, since each value in the tree is a member of $\mathbb{R}$.

\begin{figure}[H] % 3-d tree with pivot entries marked.
\begin{tikzpicture}[
    level 1/.style={sibling distance=7cm},
    level 2/.style={sibling distance=4cm},
    level 3/.style={sibling distance=3cm},
    level 4/.style={sibling distance=3cm},
    edge from parent/.style={draw, ->,thick, >=stealth'}]

    % Place the nodes and labels
    \node [thick,circle,draw] (a) {$[\textcolor{red}{3},1,4]$}
        child {node[thick,circle,draw] (b) {$[1,\textcolor{red}{2},7]$}
            child {node[thick,circle,draw] (c) {$[2,0,\textcolor{red}{3}]$}}
            child {node[thick,circle,draw] (d) {$[2,4,\textcolor{red}{5}]$}
                child {node[thick,circle,draw] (e) {$[\textcolor{red}{1},4,3]$}}
                child {node[thick,circle,draw] (f) {$[\textcolor{red}{0},5,7]$}}}}
        child {node[thick,circle,draw] (g) {$[4,\textcolor{red}{3},5]$}
            child {node[thick,circle,draw] (h) {$[6,1,\textcolor{red}{4}]$}
                child {node[draw = none] (invisble) {} edge from parent[draw=none]}
                child {node[thick,circle,draw] (i) {$[\textcolor{red}{5},2,5]$}}}
            child {node[draw = none] (invisble1) {} edge from parent[draw=none]}};

    % pivot labels
    \node [draw=none,red,node distance=5.7cm]  (1) [right of=a] {\li{pivot=0}};
    \node [draw=none,red,node distance=2.18cm] (2) [right of=g] {\li{pivot=1}};
    \node [draw=none,red,node distance=4.19cm] (3) [right of=h] {\li{pivot=2}};
    \node [draw=none,red,node distance=2.69cm] (4) [right of=i] {\li{pivot=0}};
\end{tikzpicture}

\caption{A $k$-d tree with $k = 3$.
The root $[\textcolor{red}{3},1,4]$ has an \li{pivot} of $0$, so $[1,2,7]$ is to the left of the root because $1<\textcolor{red}{3}$, and $[4,3,5]$ is to the right since $\textcolor{red}{3} \le 4$.
Similarly, the node $[2,4,\textcolor{red}{5}]$ has an \li{pivot} of $2$, so $[1,4,3]$ is to its left since $4<\textcolor{red}{5}$ and $[0,5,7]$ is to its right because $\textcolor{red}{5}\le 7$.
The nodes that are furthest from the root have an \li{pivot} of $0$ because their parents have an \li{pivot} of $2 = k - 1$.}
\label{fig:nearest-neighbor-pivot-tree}
\end{figure}

\begin{problem} % Write a KDTNode class from scratch.
Write a \li{KDTNode} class whose constructor accepts a single parameter $\x \in \mathbb{R}^k$.
If $\x$ is not a NumPy array (of type \li{np.ndarray}), raise a \li{TypeError}.
Save $\x$ as an attribute called \li{value}, and initialize attributes \li{left}, \li{right}, and \li{pivot} as \li{None}.
The \li{pivot} will be assigned when the node is inserted into the tree, and \li{left} and \li{right} will refer to child nodes.
\end{problem}

% Now we construct the $k$-d tree class.
% For an optimal $k$-d tree, the data needs to be inserted in a very particular order.
% However, inserting at random still usually produces a good tree.
% Here we simply insert the data in the order that it is given.
% TODO: put this elsewhere (in the problem statement?).

\subsection*{Constructing the Tree} % -----------------------------------------

\subsubsection*{Locating Nodes} % - - - - - - - - - - - - - - - - - - - - - - -

The \li{find()} methods for $k$-d trees and binary search trees are very similar.
Both recursively compare the values of a target and nodes in the tree, but in a $k$-d tree, these values must be compared according to their \li{pivot} attribute.
% Solving the nearest neighbor problem does not require a \li{find()} method, the $k$-d tree version of \li{find()} is provided as an instructive example.
Every comparison in the recursive \li{_step()} function, implemented below, compares the data of \li{target} and \li{current} based on the \li{pivot} attribute of \li{current}.
See Figure \ref{fig:nearest-neighbor-kdfind}.

\begin{lstlisting}
class KDT:
    """A k-dimensional tree for solving the nearest neighbor problem.

    Attributes:
        root (KDTNode): the root node of the tree. Like all other nodes in
            the tree, the root has a NumPy array of shape (k,) as its value.
        k (int): the dimension of the data in the tree.
    """
    def __init__(self):
        """Initialize the root and k attributes."""
        self.root = None
        self.k = None

    def find(self, data):
        """Return the node containing the data. If there is no such node in
        the tree, or if the tree is empty, raise a ValueError.
        """
        def _step(current):
            """Recursively step through the tree until finding the node
            containing the data. If there is no such node, raise a ValueError.
            """
            if current is None:                     # Base case 1: dead end.
                raise ValueError(str(data) + " is not in the tree")
            elif np.allclose(data, current.value):
                return current                      # Base case 2: data found!
            elif data[current.pivot] < current.value[current.pivot]:
                return _step(current.left)          # Recursively search left.
            else:
                return _step(current.right)         # Recursively search right.

        # Start the recursive search at the root of the tree.
        return _step(self.root)
\end{lstlisting}

\begin{figure}[H]
\begin{tikzpicture}[
    level 1/.style={sibling distance=7cm},
    level 2/.style={sibling distance=4cm},
    level 3/.style={sibling distance=3cm},
    level 4/.style={sibling distance=3cm},
    edge from parent/.style={draw, ->,thick, >=stealth'}]

    % Place the nodes and labels
    \node [draw=blue, circle,fill=blue!10!,thick] (a) {$[\textcolor{red}{3},1,4]$}
        child {node[draw=blue, circle, fill=blue!10!,thick] (b) {$[1,\textcolor{red}{2},7]$}
            child {node[thick,circle,draw] (c) {$[2,0,\textcolor{red}{3}]$}}
            child {node[draw=blue, circle,thick,fill=blue!10!] (d) {$[2,4,\textcolor{red}{5}]$}
                child {node[draw=blue, circle,thick,fill=blue!10!] (e) {$[\textcolor{red}{1},4,3]$}}
                child {node[thick,circle,draw] (f) {$[\textcolor{red}{0},5,7]$}}}}
        child {node[thick,circle,draw] (g) {$[4,\textcolor{red}{3},5]$}
            child {node[thick,circle,draw] (h) {$[6,1,\textcolor{red}{4}]$}
                child {node[draw = none] (invisble) {} edge from parent[draw=none]}
                child {node[thick,circle,draw] (i) {$[\textcolor{red}{5},2,5]$}}}
            child {node[draw = none] (invisble1) {} edge from parent[draw=none]}};
    \draw [blue!100!,->,thick,>=stealth'] (a)--(b);
    \draw [blue!100!,->,thick,>=stealth'] (b)--(d);
    \draw [blue!100!,->,thick,>=stealth'] (d)--(e);

    % pivot labels
    \node [draw=none,red,node distance=5.7cm]  (1) [right of=a] {\li{pivot=0}};
    \node [draw=none,red,node distance=2.18cm] (2) [right of=g] {\li{pivot=1}};
    \node [draw=none,red,node distance=4.19cm] (3) [right of=h] {\li{pivot=2}};
    \node [draw=none,red,node distance=2.69cm] (4) [right of=i] {\li{pivot=0}};
\end{tikzpicture}
\caption{To locate the node containing $[1,4,3]$, start by comparing $[1,4,3]$ to the root $[\textcolor{red}{3},1,4]$.
The root has an \li{pivot} of $0$, so compare the first component of the data to the first component of the root: since $1<\textcolor{red}{3}$, step left.
Next, $[1,4,3]$ must be to the right of $[1,\textcolor{red}{2},7]$ because $\textcolor{red}{2}\le 4$.
Similarly, $[1,4,3]$ must be to the left of $[2,4,\textcolor{red}{5}]$ as $3<\textcolor{red}{5}$.} %, where the node containing $[1,4,3]$ is found.}
\label{fig:nearest-neighbor-kdfind}
\end{figure}

\begin{figure}[H] % KDT.insert()
\captionsetup[subfigure]{justification=justified}
\begin{center}
\begin{subfigure}{\textwidth}
    \centering
    \begin{tabular}{lcr}
    \begin{tikzpicture}[
        baseline=(current bounding box.south),
        level 1/.style={sibling distance=4cm, level distance=1cm},
        level 2/.style={sibling distance=2.5cm, level distance=1cm},
        level 3/.style={sibling distance=1.5cm, level distance=1cm},
        edge from parent/.style={draw, ->, thick, >=stealth'}]
        \node [thick,circle,draw] (a) {$[\textcolor{red}{5},5]$}
          child {node[draw=none] (b) {\textcolor{gray}{$A$}} edge from parent[draw=gray]
            child {node[draw=none] (d) {} edge from parent[draw=none]}
            child {node[draw=none] (e) {} edge from parent[draw=none]
                child {node[draw=none] (i) {} edge from parent[draw=none]}
                child {node[draw=none] (j) {} edge from parent[draw=none]}}}
          child {node[draw=none] (b) {\textcolor{gray}{$B$}} edge from parent[draw=gray]
            child {node[draw=none] (f) {} edge from parent[draw=none]}
            child {node[draw=none] (g) {} edge from parent[draw=none]}};
    \end{tikzpicture}
    &\textcolor{white}{----------}&\textcolor{white}{[]}
    \includegraphics[width=.4\textwidth]{figures/partition1.pdf}
    \end{tabular}
    \caption{Insert $[5,5]$ as the root.
    The root always has an \li{pivot} of $0$, so nodes to the left of the root contain points from $A = \{(x,y)\in\mathbb{R}^2 : x < \textcolor{red}{5}\}$, and nodes on the right branch have points in $B = \{(x,y)\in\mathbb{R}^2 : \textcolor{red}{5}\le x\}$.}
\end{subfigure}
\begin{subfigure}{\textwidth}
    \centering
    \begin{tabular}{lcr}
    \begin{tikzpicture}[
        baseline=(current bounding box.south),
        level 1/.style={sibling distance=4cm, level distance=1cm},
        level 2/.style={sibling distance=2.5cm, level distance=1cm},
        level 3/.style={sibling distance=1.5cm, level distance=1cm},
        edge from parent/.style={draw, ->, thick, >=stealth'}]
        \node [thick,circle,draw] (a) {$[\textcolor{red}{5},5]$}
          child {node[thick,circle,draw] (b) {$[3,\textcolor{blue}{2}]$}
            child {node[draw=none] (d) {\textcolor{gray}{$A$}} edge from parent[draw=gray]}
            child {node[draw=none] (e) {\textcolor{gray}{$B$}} edge from parent[draw=gray]
                child {node[draw=none] (i) {} edge from parent[draw=none]}
                child {node[draw=none] (j) {} edge from parent[draw=none]}}}
          child {node[thick,circle,draw] (b) {$[8,\textcolor{blue}{4}]$}
            child {node[draw=none] (f) {\textcolor{gray}{$C$}} edge from parent[draw=gray]}
            child {node[draw=none] (g) {\textcolor{gray}{$D$}} edge from parent[draw=gray]}};
    \end{tikzpicture}
    &\textcolor{white}{----------}&
    \includegraphics[width=.4\textwidth]{figures/partition2.pdf}
    \end{tabular}
    \caption{Insert $[3,2]$, then $[8,4]$.
    Since $3 < \textcolor{red}{5}$, $[3,2]$ becomes the left child of $[\textcolor{red}{5},5]$.
    Likewise, as $\textcolor{red}{5}\le 8$, $[8,4]$ becomes the right child of $[\textcolor{red}{5},5]$.
    These new nodes have an \li{pivot} of $1$, so they partition the space vertically:
    nodes to the right of $[3,\textcolor{blue}{2}]$ contain points from $B = \{(x,y)\in\mathbb{R}^2 : x < \textcolor{red}{5},\ \textcolor{blue}{2} \le y\}$;
    nodes to the left of $[8,\textcolor{blue}{4}]$ hold points from $C = \{(x,y)\in\mathbb{R}^2 : \textcolor{red}{5} \le x,\ y < \textcolor{blue}{8}\}$.}
\end{subfigure}
\begin{subfigure}{\textwidth}
    \centering
    \begin{tabular}{lcr}
    \begin{tikzpicture}[
        baseline=(current bounding box.south),
        level 1/.style={sibling distance=4cm, level distance=1cm},
        level 2/.style={sibling distance=2.5cm, level distance=1cm},
        level 3/.style={sibling distance=1.5cm, level distance=1cm},
        edge from parent/.style={draw, ->, thick, >=stealth'}]
        \node [thick,circle,draw] (a) {$[\textcolor{red}{5},5]$}
            child {node[thick,circle,draw] (b) {$[3,\textcolor{blue}{2}]$}
                child {node[draw=none] (d) {\textcolor{gray}{$A$}} edge from parent[draw=gray]}
                child {node[thick,circle,draw] (e) {$[\textcolor{red}{2},6]$}
                  child {node[draw=none] (i) {\textcolor{gray}{$B$}} edge from parent[draw=gray]}
                  child {node[draw=none] (j) {\textcolor{gray}{$C$}} edge from parent[draw=gray]}}}
            child {node[thick,circle,draw] (c) {$[8,\textcolor{blue}{4}]$}
                child {node[draw=none] (f) {\textcolor{gray}{$D$}} edge from parent[draw=gray]}
                child {node[draw=none] (g) {\textcolor{gray}{$E$}} edge from parent[draw=gray]}};
    \end{tikzpicture}
    &\textcolor{white}{----------}&
    \includegraphics[width=.4\textwidth]{figures/partition3.pdf}
    \end{tabular}
    \caption{Insert $[2,6]$.
    The \li{pivot} cycles back to $0$ since $k = 2$, so nodes to the left of $[\textcolor{red}{2},6]$ have points that lie in $B = \{(x,y)\in\mathbb{R}^2 : x < \textcolor{red}{2},\ \textcolor{blue}{2} \le y\}$
    and nodes to the right store points in $C = \{(x,y)\in\mathbb{R}^2 : \textcolor{red}{2} \le x < \textcolor{red}{5},\ \textcolor{blue}{2} \le y\}$.}
\end{subfigure}
\end{center}
\caption{As a $k$-d tree is constructed (left), it creates a partition of $\mathbb{R}^k$ (right) by defining separating hyperplanes that pass through the points.
The more points, the finer the partition.}
\label{fig:nearest-neighbor-KDT_insert}
\end{figure}

\subsubsection*{Inserting Nodes} % - - - - - - - - - - - - - - - - - - - - - -

% New nodes in a $k$-d tree are always added as leaf nodes.
To add a new node to a $k$-d tree, determine which existing node should be the parent of the new node by recursively stepping down the tree as in the \li{find()} method.
Next, assign the new node as the \li{left} or \li{right} child of the parent, and set its \li{pivot} based on its parent's \li{pivot}: if the parent's \li{pivot} is $i$, the new node's \li{pivot} should be $i+1$, or $0$ if $i = k-1$.

Consider again the $k$-d tree in Figure \ref{fig:nearest-neighbor-kdfind}.
To insert $[2,3,4]$, search the tree for $[2,3,4]$ until hitting an empty slot.
In this case, the search steps from the root down to $[\textcolor{red}{1},4,3]$, which has an \li{pivot} of $0$.
Then since $\textcolor{red}{1} \le 2$, the new node should be to the right of $[1,4,3]$.
However, $[\textcolor{red}{1},4,3]$ has no right child, so it becomes the parent of $[2,3,4]$.
The \li{pivot} of the new node should therefore be $1$.
See Figure \ref{fig:nearest-neighbor-KDT_insert} for another example.
% This method partitions $\mathbb{R}^k$ into smaller subsections, allowing for efficient searching.

\begin{problem} % KDT.insert()
\label{prob:kdt-insert}
Write an \li{insert()} method for the \li{KDT} class that accepts a point $\x\in\mathbb{R}^k$.
\begin{enumerate}
\item If the tree is empty, create a new \li{KDTNode} containing $\x$ and set its \li{pivot} to $0$.
Assign the \li{root} attribute to the new node and set the \li{k} attribute as the length of $\x$.
Thereafter, raise a \li{ValueError} if data to be inserted is not in $\mathbb{R}^k$.
\item If the tree is nonempty, create a new \li{KDTNode} containing $\x$ and find the existing node that should become its parent.
Determine whether the new node will be the parent's \li{left} or \li{right} child, then link the parent to the new node accordingly.
Set the \li{pivot} of the new node based on its parent's \li{pivot}.
\\(Hint: write a recursive function like \li{_step()} to find and link the parent.)
\item Do not allow duplicates in the tree: if there is already a node in the tree containing $\x$, raise a \li{ValueError}.
% Be careful to check the entire array, not just the \li{pivot} entry.
\end{enumerate}
To test your method, use the \li{__str__()} method provided in the Additional Materials section.
Try constructing the trees in Figures \ref{fig:nearest-neighbor-pivot-tree} and \ref{fig:nearest-neighbor-KDT_insert}.
Also check that the provided \li{find()} method works as expected.
\end{problem}

\subsection*{Nearest Neighbor Search with K-D Trees} % ------------------------

Given a target $\z\in\mathbb{R}^k$ and a $k$-d tree containing a set $X\subset\mathbb{R}^k$ of $m$ points, the nearest neighbor problem can be solved by traversing the tree in a manner that is similar to the \li{find()} or \li{insert()} methods from the previous section.
The advantage of this strategy over an exhaustive search is that not every $\x\in X$ has to be compared to $\z$ via \eqref{eq:nearest-neighbor-euclidean-metric}; the tree structure makes it possible to rule out some elements of $X$ without actually computing their distances to $\z$.
The complexity is $O(k\log(m))$, a significant improvement over the $O(km)$ complexity of an exhaustive search.

To begin, set $\x^*$ as the value of the root and compute $d^* = d(\x^*,\z)$.
Starting at the root, step down through the tree as if searching for the target $\z$.
At each step, determine if the value $\x$ of the current node is closer to $\z$ than $\x^*$.
If it is, assign $\x^* = \x$ and recompute $d^* = d(\x^*, \z)$.
Continue this process until reaching a leaf node.

Next, backtrack along the search path and determine if the non-explored branch needs to be searched.
To do this, check that the sphere of radius $d^*$ centered at $\z$ does not intersect with the separating hyperplane defined by the current node.
That is, if the separating hyperplane is further than $d^*$ from $\z$, then no points on the other side of the hyperplane can possibly be the nearest neighbor. %, so the branch that hasn't been searched can be safely ignored.
See Figure \ref{fig:nearest-neighbor-query} for an example and Algorithm \ref{alg:nearest-neighbor-search} for the details of the procedure.

\begin{figure}[H] % KDT.query()
\captionsetup[subfigure]{justification=justified}
\begin{center}
\begin{subfigure}{\textwidth}
    \centering
    \begin{tabular}{lcr}
    \begin{tikzpicture}[
        baseline=(current bounding box.south),
        level 1/.style={sibling distance=4cm, level distance=1cm},
        level 2/.style={sibling distance=2.5cm, level distance=1cm},
        level 3/.style={sibling distance=1.5cm, level distance=1.5cm},
        edge from parent/.style={draw, ->,thick, >=stealth'}]
        \node [draw=purple,thick,circle,fill=green!20!] (a) {$[\textcolor{red}{5},5]$}
          child {node[thick,circle,draw] (b) {$[3,\textcolor{blue}{2}]$}
            child {node[draw = none] (invisble) {} edge from parent[draw=none]}
            child {node[thick,circle,draw] (c) {$[\textcolor{red}{2},6]$}
              child {node[draw=none] (i) {} edge from parent[draw=none]}}}
          child {node[thick,circle,draw] (g) {$[8,\textcolor{blue}{4}]$}
            child {node[draw = none] (invisible1) {} edge from parent[draw=none]}
            child {node[draw,circle] (e) {$[\textcolor{red}{7},7]$}}};
    \end{tikzpicture}
    &\textcolor{white}{----------}&
    \includegraphics[width=.4\textwidth]{figures/kdtnn_search1.pdf}
    \end{tabular}
    \caption{Start at the root, setting $\x^* = [\textcolor{red}{5},5]$.
    The sphere of radius $d^* = d(\x^*,\z)$ centered at $\z$ intersects the hyperplane $x = 5$, so (at this point) it is possible that a nearer neighbor lies to the right of the root.}
\end{subfigure}
\begin{subfigure}{\textwidth}
    \centering
    \begin{tabular}{lcr}
    \begin{tikzpicture}[
        baseline=(current bounding box.south),
        level 1/.style={sibling distance=4cm, level distance=1cm},
        level 2/.style={sibling distance=2.5cm, level distance=1cm},
        level 3/.style={sibling distance=1.5cm, level distance=1.5cm},
        edge from parent/.style={draw, ->,thick, >=stealth'}]
        \node [thick,circle,draw] (a) {$[\textcolor{red}{5},5]$}
          child {node[draw=purple,thick,circle,fill=green!20!] (b) {$[3,\textcolor{blue}{2}]$} edge from parent[draw=red]
            child {node[draw = none] (invisble) {} edge from parent[draw=none]}
            child {node[thick,circle,draw=black] (c) {$[\textcolor{red}{2},6]$} edge from parent[draw=black]
              child {node[draw=none] (i) {} edge from parent[draw=none]}}}
          child {node[thick,circle,draw] (g) {$[8,\textcolor{blue}{4}]$}
            child {node[draw = none] (invisible1) {} edge from parent[draw=none]}
            child {node[draw,circle] (e) {$[\textcolor{red}{7},7]$}}};
    \end{tikzpicture}
    &\textcolor{white}{----------}&
    \includegraphics[width=.4\textwidth]{figures/kdtnn_search2.pdf}
    \end{tabular}
    \caption{If the target $\z = [3,2.75]$ were in the tree, it would be to the left of the root, so step left and examine $\x = [3,\textcolor{blue}{2}]$.
    Since $d(\x,\z) < d(\x^*,\z)$, reassign $\x^* = \x$ and recompute $d^*$.
    Now the sphere of radius $d^*$ centered at $\z$ no longer intersects the root's hyperplane, so the nearest neighbor cannot be in the root's right subtree.}
\end{subfigure}
\begin{subfigure}{\textwidth}
    \begin{tabular}{lcr}
    \begin{tikzpicture}[
        baseline=(current bounding box.south),
        level 1/.style={sibling distance=4cm, level distance=1cm},
        level 2/.style={sibling distance=2.5cm, level distance=1cm},
        level 3/.style={sibling distance=1.5cm, level distance=1.5cm},
        edge from parent/.style={draw, ->,thick, >=stealth'}]
        \node [thick,circle,draw] (a) {$[\textcolor{red}{5},5]$}
          child {node[thick,circle,draw=purple] (b) {$[3,\textcolor{blue}{2}]$}
            child {node[draw=none] (invisble) {} edge from parent[draw=none]}
            child {node[draw,thick,circle,fill=green!20!] (c) {$[\textcolor{red}{2},6]$} edge from parent[draw=blue]
              child {node[draw=none] (i) {} edge from parent[draw=none]}}}
          child {node[thick,circle,draw] (g) {$[8,\textcolor{blue}{4}]$}
            child {node[draw=none] (invisible1) {} edge from parent[draw=none]}
            child {node[draw,circle] (e) {$[\textcolor{red}{7},7]$}}};
    \end{tikzpicture}
    &\textcolor{white}{----------}&
    \includegraphics[width=.4\textwidth]{figures/kdtnn_search3.pdf}
    \end{tabular}
    \caption{Continuing the search, step right to check the point $\x = [\textcolor{red}{2},6]$.
    In this case $d(\x,\z) > d(\x^*,\z)$, meaning $\x$ is \textbf{not} nearer to $\z$ than $\x^*$.
    Since $[\textcolor{red}{2},6]$ is a leaf node, retrace the search steps up the tree to check the non-searched branches.
    However, the sphere around $\z$ does not intersect any splitting hyperplanes defined by the tree, so $\x^*$ is guaranteed to be the nearest neighbor.}
\end{subfigure}
\end{center}
\caption{Nearest neighbor search of a $k$-d tree with $k= 2$.
The target is $\z = [3,2.75]$ and the nearest neighbor is $\x^* = [3,\textcolor{blue}{2}]$ with minimal distance $d^* = 0.75$.
The tree structure allows the algorithm to eliminate $[8,\textcolor{blue}{4}]$ and $[\textcolor{red}{7},7]$ from consideration without computing their distance from $\z$.}
\label{fig:nearest-neighbor-query}
\end{figure}

\begin{algorithm}[h] % K-d search algorithm.
\begin{algorithmic}[1]
% \State Let $\z \in \mathbb{R}^k$ be a search target, $d(\cdot,\cdot)$ be a metric, and \texttt{root} be the top of a $k$-d tree. %; \texttt{current} and \texttt{nearest} are \li{KDTNode}s.
\Procedure{Nearest Neighbor Search}{$\z$, \texttt{root}}
\Procedure{KDSearch}{\texttt{current}, \texttt{nearest}, $d^*$}
\If {\texttt{current} \textcolor{blue}{\texttt{is None}}}
    \Comment{Base case: dead end.}
    \State \pseudoli{return} \texttt{nearest}, $d^*$
\EndIf
\State $\x \gets$ \texttt{current.value}
\State $i \gets$ \texttt{current.pivot}
\If {$d(\x,\z) < d^*$}
    \Comment{Check if \texttt{current} is closer to $\z$ than \li{nearest}.}
    \State \texttt{nearest} $\gets$ \texttt{current}
    \State $d^* \gets d(\x,\z)$
\EndIf
\If {$z_i < x_i$}
    \Comment{Search to the left.}
    \State \texttt{nearest}, $d^* \gets$ \texttt{KDSearch}(\texttt{current.left}, \texttt{nearest}, $d^*$)
    \If {$z_i + d^* \ge x_i$}
    \Comment{Search to the right if needed.}
        \State \texttt{nearest}, $d^* \gets$ \texttt{KDSearch}(\texttt{current.right}, \texttt{nearest}, $d^*$)
    \EndIf
\Else
    \Comment{Search to the right.}
    \State \texttt{nearest}, $d^* \gets$ \texttt{KDSearch}(\texttt{current.right}, \texttt{nearest}, $d^*$)

    \If {$z_i - d^* \le x_i$}
    \Comment{Search to the left if needed.}
        \State \texttt{nearest}, $d^* \gets$ \texttt{KDSearch}(\texttt{current.left}, \texttt{nearest}, $d^*$)
    \EndIf
\EndIf
\State \pseudoli{return} \texttt{nearest}, $d^*$
\EndProcedure
\State \texttt{node}, $d^* \gets \texttt{KDSearch}(\texttt{root}, \texttt{root}, d(\texttt{root.value},\z))$
\State \pseudoli{return} \texttt{node.value}, $d^*$
\EndProcedure
\end{algorithmic}
\caption{$k$-d tree nearest neighbor search}
\label{alg:nearest-neighbor-search}
\end{algorithm}

\begin{problem} % K-d nearest neighbor search algorithm.
\label{prob:kdtree-nearest-neighbor-search}
Write a method for the \li{KDT} class that accepts a target point $\z\in\mathbb{R}^k$.
Use Algorithm \ref{alg:nearest-neighbor-search} to solve \eqref{eq:nearest-neighbor-def}.
Return the nearest neighbor $\x^*$ (the actual NumPy array, not the \li{KDTNode}) and its distance $d^*$ from $\z$.

Compare your method to the exhaustive search in Problem \ref{prob:nearest-neighbor-exhaustive-search} and to SciPy's built-in \li{KDTree} class.
This structure is essentially a heavily optimized version of the \li{KDT} class.
To solve the nearest neighbor problem, initialize the tree with data, then ``query'' the tree with the target point.
The \li{query()} method returns a tuple of the minimum distance and the index of the nearest neighbor in the data.

\begin{lstlisting}
>>> from scipy.spatial import KDTree

# Initialize the tree with data (in this example, use random data).
>>> data = np.random.random((100,5))    # 100 5-dimensional points.
>>> target = np.random.random(5)
>>> tree = KDTree(data)

# Query the tree for the nearest neighbor and its distance from 'target'.
>>> min_distance, index = tree.query(target)
>>> print(min_distance)
0.24929868807
>>> tree.data[index]                    # Get the actual nearest neighbor.
array([ 0.26927057,  0.03160271,  0.46830759,  0.26766863,  0.63073275])
\end{lstlisting}
\end{problem}

\begin{warn}
There are a few caveats to using a $k$-d tree for the nearest neighbor search problem.
\begin{itemize}
\item Constructing the tree takes time.
For small enough data sets, an exhaustive search may be faster than the combined time of constructing and searching a tree.
On the other hand, once the tree is constructed, it can be used for multiple nearest-neighbor queries.
\item In the worst case---when the tree is completely unbalanced---the search complexity is $O(km)$ instead of $O(k\log(m))$.
Fortunately, there are algorithms for constructing the tree intelligently so that it is mostly balanced, and a random insertion order usually results in a somewhat balanced tree.
\end{itemize}
% In general, $k$-d trees may be unnecessary or even a hinderance if $X$ is small, but it certainly pays to have a well-constructed $k$-d tree if $X$ is large.
\end{warn}

\section*{K-Nearest Neighbors} % ==============================================

The nearest neighbor algorithm provides one way to solve a common machine learning problem.
In \emph{supervised learning},
% \footnote{In \emph{unsupervised learning} there are no labels provided in the first place.}
a \emph{training set} $X\subset D$ has a corresponding set of \emph{labels} $Y$ that specifies a category for each element of $X$.
For instance, $X$ could contain financial data on $m$ individuals, and $Y$ could be a set of $m$ booleans indicating which individuals have filed for bankruptcy.
Supervised learning algorithms use the training data to construct a function $f:D\rightarrow Y$ that maps points to their corresponding label.
In other words, the algorithm ``learns'' enough about the relationship between $X$ and $Y$ to intelligently label arbitrary elements of $D$.
In the bankruptcy example, a person could then use their own financial data to learn whether or not they look more like someone who files for bankruptcy or someone who does not.

A \emph{$k$-nearest neighbors} classifier uses a simple strategy to label an arbitrary $\z\in D$: find the $k$ elements of $X$ that are nearest to $\z$ (usually in terms of the Euclidean metric) and choose the most common label from those $k$ elements as the label of $\z$.
That is, the points in the $k$ labeled points that are most like $\z$ are allowed to ``vote'' on how $\z$ should be labeled.
See Figure \ref{fig:k-nearest-neighbors}.

\begin{figure}[H]
\begin{tabular}{ccccc}
\begin{tikzpicture}[circ/.style={circle,align=center,minimum size=.5cm}]
    \node[circ,draw=black,thick] (R) {};
    \node[cyan!70,fill=cyan!70,circ,above left=.3cm and .3cm of R] (B1) {};
    \node[cyan!70,fill=cyan!70,circ,below left=-.3cm and .6cm of R] (B3) {};
    \node[yellow!70,fill=yellow!70,circ,below left=.5cm and .6cm of R] (Y1){};
    \node[yellow!60,fill=yellow!60,circ,above right=.3cm and .3cm of R](Y2){};
    \node[yellow!60,fill=yellow!60,circ,below right=0cm and 1cm of R] (Y3) {};
\end{tikzpicture}
%
\quad\qquad\qquad&\qquad\qquad&
%
\begin{tikzpicture}[circ/.style={circle,align=center,minimum size=.5cm}]
    \node[circ,cyan!70,fill=cyan!70,draw=black,thick] (R) {};
    \node[cyan!70,fill=cyan!70,circ,above left=.3cm and .3cm of R] (B1) {};
    \node[cyan!70,fill=cyan!70,circ,below left=-.3cm and .6cm of R] (B3) {};
    \node[yellow!70,fill=yellow!70,circ,below left=.5cm and .6cm of R] (Y1) {};
    \node[yellow!60,fill=yellow!60,circ,above right=.3cm and .3cm of R] (Y2){};
    \node[yellow!60,fill=yellow!60,circ,below right=0cm and 1cm of R] (Y3) {};
    \foreach \a/\b in {R/B1,R/B3,R/Y2} \draw[line width=.05cm] (\a) -- (\b);
\end{tikzpicture}
%
&\qquad\qquad&\qquad\qquad\quad
%
\begin{tikzpicture}[circ/.style={circle,align=center,minimum size=.5cm}]
    \node[circ,yellow!60,fill=yellow!60,draw=black,thick] (R) {};
    \node[cyan!70,fill=cyan!70,circ,above left=.3cm and .3cm of R] (B1) {};
    \node[cyan!70,fill=cyan!70,circ,below left=-.3cm and .6cm of R] (B3) {};
    \node[yellow!70,fill=yellow!70,circ,below left=.5cm and .6cm of R] (Y1) {};
    \node[yellow!60,fill=yellow!60,circ,above right=.3cm and .3cm of R] (Y2){};
    \node[yellow!60,fill=yellow!60,circ,below right=0cm and 1cm of R] (Y3) {};
    \foreach \a/\b in {R/B1,R/B3,R/Y2,R/Y3,R/Y1} \draw[line width=.05cm] (\a) -- (\b);
\end{tikzpicture}
\end{tabular}
\caption{To classify the center node, determine its $k$-nearest neighbors and pick the most common label of the neighbors.
If $k=3$, the $k$ nearest points are two blues and a yellow, so the center node is labeled blue.
For $k=5$, the $k$ nearest points consists of two blues and three yellows, so the center node is labeled yellow.}
\label{fig:k-nearest-neighbors}
\end{figure}

\begin{warn}
The $k$ in $k$-d tree refers to the \textbf{dimension} of the data housed in the tree, but the $k$ in $k$-nearest neighbors refers to the \textbf{number of neighbors} to use in the voting scheme.
Unfortunately, both names are standard.
% The $k$ in the \emph{$k$-means algorithm} refers to the number of clusters to separate the data into.
\end{warn}

\begin{problem} % Write a KNeighborsClassifier class.
\label{prob:kneighbors-classifier}
Write a \li{KNeighborsClassifier} class with the following methods.
\begin{enumerate}
\item The constructor should accept an integer \li{n_neighbors}, the number of neighbors to include in the vote (the $k$ in $k$-nearest neighbors).
Save this value as an attribute.

\item \li{fit()}: accept an $m\times k$ NumPy array $X$ (the training set) and a 1-dimensional NumPy array $\y$ with $m$ entries (the training labels).
As in Problems \ref{prob:nearest-neighbor-exhaustive-search} and \ref{prob:kdtree-nearest-neighbor-search}, each of the $m$ rows of $X$ represents a point in $\mathbb{R}^k$.
Here $y_i$ is the label corresponding to row $i$ of $X$.

Load a SciPy \li{KDTree} with the data in $X$.
Save the tree and the labels as attributes.

\item \li{predict()}: accept a 1-dimensional NumPy array $\z$ with $k$ entries.
Query the \li{KDTree} for the \li{n_neighbors} elements of $X$ that are nearest to $\z$ and return the most common label of those neighbors.
If there is a tie for the most common label (such as if $k=2$ in Figure \ref{fig:k-nearest-neighbors}), choose the alphanumerically smallest label.
\\(Hint: use \li{scipy.stats.mode()}. The default behavior splits ties correctly.)

To get several nearest neighbors from the tree, specify \li{k} in \li{KDTree.query()}.
\begin{lstlisting}
>>> data = np.random.random((100,5))    # 100 5-dimensional points.
>>> target = np.random.random(5)
>>> tree = KDTree(data)

# Query the tree for the 3 nearest neighbors.
>>> distances, indices = tree.query(target, k=3)
>>> print(indices)
[26 30 32]
\end{lstlisting}
\end{enumerate}
\end{problem}

\begin{info}
The format of the \li{KNeighborsClassifier} in Problem \ref{prob:kneighbors-classifier} conforms to the style of \emph{scikit-learn} (\li{sklearn}), a large machine learning library in Python.
In fact, scikit-learn has a class called \li{sklearn.neighbors.KNeighborsClassifier} that is a more robust version of the class from Problem \ref{prob:kneighbors-classifier}.
See \url{http://scikit-learn.org/stable/modules/neighbors.html} for more tools from scikit-learn for solving the nearest neighbor problem in the context of machine learning.
\end{info}

\begin{comment}
To determine the accuracy of a machine learning model, first divide the set of labeled data into a \emph{training set} and a \emph{test set}.
The training set is used to find potentially predictive relationships between the \emph{features}, or individual properties, of a data point.
In the bankruptcy example above, the incomes and debt levels were features of the data.
Relationships between features are discovered in the \emph{training phase} of machine learning.
In the nearest neighbor algorithm, the training phase consists of simply storing the training set.
The test set is a small subset (usually around 20\%) of the original data set.
During the \emph{test phase}, the accuracy of the model is calculated using the test set.
The test set is treated as a set of novel data, and is run through the model.
The model's classifications of the test data are compared to the actual labels of the test data.
The ratio of correct classifications to total number of instances in the test set is the accuracy of the model.

\subsection*{Breaking Ties}
Consider the illustration in Figure \ref{fig:k-nearest-neighbors}.
Suppose that the three neighbors of a new data point were found, but all of these neighbors had different classifications: one neighbor was of class A, one was B, and one was C.
How would the new instance be classified?
A common approach to dealing with ties is to remove the farthest neighbor's vote until a majority vote is found.
In some cases, this results in only the closest neighbor determining the output of a new instance.
\end{comment}

\subsection*{Handwriting Recognition} % ---------------------------------------

\emph{Computer vision} is a challenging area of artificial intelligence that focuses on autonomously interpreting images.
Perhaps the simplest computer vision problem is that of translating images into text.
Roughly speaking, computers store grayscale images as $M \times N$ arrays of pixel brightness values: 0 corresponds to black, and 255 to white.
Flattening out such an array yields a vector in $\mathbb{R}^{MN}$.
Given some images of characters with labels (assigned by humans), a $k$-nearest neighbor classifier can intelligently decide what character the image represents.

\begin{problem}
The file \texttt{mnist\_subset.npz} contains part of the MNIST dataset,\footnote{See \url{http://yann.lecun.com/exdb/mnist/}.} a collection of $28\times 28$ images of handwritten digits and their labels.
The data is split into four parts.
\begin{itemize}
\item \li{X_train}: A $3000\times 728$ matrix, the training set.
Each of the $3000$ rows is a flattened $28\times 28$ image to be used in training the classifier.

\item \li{y_train}: A 1-dimensional NumPy array with $3000$ entries.
The entries are integers from 0 to 9, the labels corresponding to the images in \li{X_train}.

\item \li{X_test}: A $500\times 728$ matrix of $500$ images to classify.

\item \li{y_test}: A 1-dimensional NumPy array with $500$ entries.
These are the labels corresponding to \li{X_test}, the ``right answers'' that the classifier will try to guess.
\end{itemize}
The following code uses \li{np.load()} to extract the data.
% Use \li{np.load()} to extract the data.
% An \texttt{npz} file holds several arrays, which can be accessed like the values of a dictionary.
\begin{lstlisting}
>>> data = np.load("mnist_subset.npz")
>>> X_train = data["X_train"].astype(np.<<float>>)          # Training data
>>> y_train = data["y_train"]                           # Training labels
>>> X_test = data["X_test"].astype(np.<<float>>)            # Test data
>>> y_test = data["y_test"]                             # Test labels
\end{lstlisting}
To visualize one of the images, reshape it as a $28\times 28$ array and use \li{plt.imshow()}.
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> plt.imshow(X_test[0].reshape((28,28)), cmap="gray")
>>> plt.show()
\end{lstlisting}
\begin{center}
\includegraphics[width=.15\textwidth]{figures/digit.pdf}
\end{center}

% \begin{figure}[H]
% \centering
% \includegraphics[width=.4\textwidth]{figures/digit.pdf}
% \caption{The number 6 taken from the data set.}
% \label{fig:digit}
% \end{figure}

Write a function than accepts an integer \li{n_neighbors}.
Load a classifier from Problem \ref{prob:kneighbors-classifier} %or a \li{sklearn.neighbors.KNeighborsClassifier}
with the data \li{X_train} and the corresponding labels \li{y_train}.
Use the classifier to predict the labels of each image in \li{X_test}.
Return the classification accuracy, the percentage of predictions that match \li{y_test}.
The accuracy should be at least $90\%$ using $4$ nearest neighbors.
\end{problem}

\begin{info}
The $k$-nearest neighbors algorithm is \textbf{not} the best machine learning algorithm for this problem, but it is a good starting point because of its simplicity.
In fact, $k$-nearest neighbors is often used as a baseline to compare against more complicated machine learning techniques.
\end{info}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Ball Trees} % ----------------------------------------------------

The nearest neighbor problem can also be solved efficiently with a \emph{ball tree}, another space-partitioning data structure.
Instead of separating $\mathbb{R}^k$ by hyperplanes, a ball tree uses nested hyperspheres to split up the space.
Since the partitioning scheme is different, a nearest neighbor search through a ball tree is more efficient than the $k$-d tree search for some data sets.
See \url{https://en.wikipedia.org/wiki/Ball_tree} for more details.

\subsection*{The Curse of Dimensionality} % -----------------------------------

The \emph{curse of dimensionality} refers to a phenomena that occurs when dealing with high-dimensional data: the computational cost of an algorithm increases much more rapidly as the dimension increases than it does when the number of points increases.
This problem occurs in many other areas involving multi-dimensional data, but it is quite apparent in a nearest neighbor search.

\begin{figure}[H] % Curse of Dimensionality
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/curse_m.pdf}
    \caption{Fixing $k$ and increasing $m$ leads to consistent growth in execution time.}
\end{subfigure}
%
\begin{subfigure}{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/curse_k.pdf}
    \caption{For fixed $m$, the times takes a sharp upturn around $k = 2^9$ relative to previous growth rates.}
\end{subfigure}
\caption{Construction and nearest neighbor search times for a $k$-d tree with a $m\times k$ training set.}
\end{figure}

See \url{https://en.wikipedia.org/wiki/Curse_of_dimensionality} for more examples.
One way to avoid the curse of dimensionality is via \emph{dimension reduction}, a process usually based on the singular value decomposition (SVD) that projects data into a lower-dimensional space.

\subsection*{Tiebreaker Strategies} % -----------------------------------------

As mentioned in Problem \ref{prob:kneighbors-classifier}, the majority voting scheme in the $k$-nearest neighbor algorithm can often result in a tie.
Breaking the tie intelligently is a science unto itself, but here are a few common strategies.
\begin{enumerate}
\item For binary classification (meaning there are only two labels), choose an odd $k$ to avoid a tie in the first place.
\item Redo the search with $k-1$ neighbors, repeating as needed until $k=1$.
\item Choose the label that appears more frequently in the test set.
\item Choose randomly among the labels that are tied for most common.
\end{enumerate}

\subsection*{Additional Code} % -----------------------------------------------

The following code creates a string representation for the \li{KDT} class.
Use this to test Problem \ref{prob:kdt-insert}.

\begin{lstlisting}
class KDT:
    # ...
    def __str__(self):
        """String representation: a hierarchical list of nodes and their axes.

        Example:                           'KDT(k=2)
                    [5,5]                   [5 5]   pivot = 0
                    /   \                   [3 2]   pivot = 1
                [3,2]   [8,4]               [8 4]   pivot = 1
                    \       \               [2 6]   pivot = 0
                    [2,6]   [7,5]           [7 5]   pivot = 0'
        """
        if self.root is None:
            return "Empty KDT"
        nodes, strs = [self.root], []
        while nodes:
            current = nodes.pop(0)
            strs.append("{}\tpivot = {}".<<format>>(current.value, current.pivot))
            for child in [current.left, current.right]:
                if child:
                    nodes.append(child)
        return "KDT(k={})\n".<<format>>(self.k) + "\n".join(strs)
\end{lstlisting}


\subsection{Convolution with stride by introducing stride operator}
As discussed above, the restriction mapping is consistent with the
co-called convolution with stride $s=2$, which is defined as:
\begin{equation}\label{stride_2}
[K \ast_2 f]_{i,j} = \sum_{p,q=-k}^k K_{p,q} f_{2i + p, 2j + q},
\quad i = 1: \frac{m+1}{2} , j = 1: \frac{n+1}{2}.
\end{equation}

\begin{remark}
More generally in CNN, given an integer $s\ge1$, a convolution with stride $s$ for $f \in \mathbb{R}^{(m)\times
		(2^n+1)}$ can also be defined as:
	\begin{equation}\label{stride}
	[K \ast_s f]_{i,j} = \sum_{p,q=-k}^k K_{p,q} f_{si + p, sj + q},
	\quad i = 1: \lceil  \frac{m}{s}\rceil , j = 1: \lceil  \frac{n}{s}\rceil.
	\end{equation}

	Here $ \lceil  \frac{m}{s}\rceil$ denotes the smallest integer that greater than $\frac{m}{s}$.
\end{remark}

As general in CNN and multigrid method, convolution with stride $s=2$ is the most important case.
To make this operator more clear, now we are going to introduce the stride operator $\mathcal S$ as:
\begin{equation}\label{eq:strideopdim}
\mathcal S: \mathbb{R}^{m \times n} \mapsto \mathbb{R}^{\frac{m+1}{2} \times \frac{n+1}{2}},
\end{equation}
with
\begin{equation}\label{eq:strideop}
[\mathcal S(f)]_{i,j} = f_{2i, 2j}, \quad i  = 1:\frac{m+1}{2}, j = 1:\frac{n+1}{2}.
\end{equation}

Then we have the rewrite form of convolution with stride $2$ as:
\begin{equation}\label{eq:convstride_2_1}
K \ast_2 f = \mathcal S( K\ast f).
\end{equation}

Then the deconvolution in defined by the transposed of convolution
with stride with respect to the F-inner product defined as:
$$
(u, v)_F := \sum_{i,j} u_{i,j}v_{i,j}.
$$
Then we have the definition of deconvolution as
\begin{equation}\label{eq:def_deconv}
(K \ast_2 u, v)_F = (u, K \ast_2^\top v),
\end{equation}
with
\begin{equation}
u \in \mathbb{R}^{m \times n} \quad \text{and} \quad v \in \mathbb{R}^{\frac{m+1}{2} \times \frac{m+1}{2}}.
\end{equation}

For a more comprehensive notation, let us also define
$K\ast$ as an operator from $\mathbb{R}^{m\times n} \quad \text{and}$
to $\mathbb{R}^{m\times n}$ as $\mathcal C_K$.
Then we have the ``transpose'' of convolution without stride as:
\begin{equation}\label{eq:def_tran_conv}
(K \ast u, v)_F = (\mathcal C_K (u), v) =  (u,  \mathcal C_K^\top v).
\end{equation}

\begin{lemma}\label{lemm:tilde-K}
For the transposed convolution without stride, we have the next result
\begin{equation}\label{eq:}
\mathcal C_K^\top = \mathcal C_{\tilde K},
\end{equation}
where $\tilde K$ is defined as
\begin{equation}\label{eq:def_tildeK}
\tilde K_{p,q} = K_{-p, -q}, \quad p,q = -k:k.
\end{equation}
Intuitively, if we take $K_{0,0}$ as the center for the convolutional kernel $K$,
then $\tilde K$ is the central symmetry of $K$.
In 2D case, it can also be understood as the rotation of $\pi$ with respect to
the center $K_{0,0}$.
\end{lemma}

Then recall the definition of deconvolution in \eqref{eq:def_deconv} we have
\begin{equation}\label{eq:op_deconv}
\begin{aligned}
(u,  K \ast_2^\top v) &= (K \ast_2 u, v)_F = (\mathcal S \mathcal C_K u, v) \\
&= (u,  \mathcal C^\top_K \mathcal S^\top v),
\end{aligned}
\end{equation}
with definition
\begin{equation}\label{eq:de_stride_dim}
\mathcal S^\top:   \mathbb{R}^{\frac{m+1}{2} \times\frac{n+1}{2}} \mapsto \mathbb{R}^{m\times n},
\end{equation}
and
\begin{equation}\label{eq:de_stride}
[\mathcal S^\top (f)]_{i,j} =
\begin{cases}
0 \quad &\text{if i or j is even}, \\
f_{i/2, j/2}, \quad &\text{else}.
\end{cases}
\end{equation}

Thus to say, we have the simple version of the deconvolution for $K \ast $ as
\begin{equation}\label{eq:simple_deconv}
K \ast_2^\top v = \mathcal C_K\top \circ \mathcal S^\top (v) = \mathcal C_{\tilde K} \circ \mathcal S^\top (v) = \tilde K \ast \mathcal S^\top (v),
\end{equation}
thus to say
\begin{equation}\label{eq:final}
K \ast_2^\top  = \tilde K \ast \mathcal S^\top.
\end{equation}

In short, we have the next decomposition
\begin{itemize}
	\item convolution with stride = stride $ \circ$ convolution,
	\item deconvolution with stride  = transposed convolution $\circ$ transposed stride = convolution with the central symmetry of original kernel $\circ$ transposed stride.
\end{itemize}

\begin{theorem}\label{thm:deconv_op}
Here let us show a simple example about some details for computing deconvolution.
Let us consider
\begin{equation}
K_{p,q} \quad p,q = -1, 0, 1.
\end{equation}
Then we have
$$
K \ast_2^\top v = \tilde K \ast \mathcal S^\top (v).
$$
As in \eqref{eq:de_stride} and the Lemma \ref{lemm:tilde-K}, we have the
final version is
\begin{equation}
\label{eq:7}
[K \ast_2^\top v ]_{2i,2j}=  K_{0,0}v_{i,j},
\end{equation}
with
\begin{equation}
\label{eq:9}
[K \ast_2^\top v ]_{2i-1, 2j} = K_{0,1}v_{i-1,j} + K_{0,-1}v_{i,j}, \quad
[K \ast_2^\top v ]_{2i, 2j-1} = K_{1,0}v_{i,j} + K_{-1,0}v_{i,j-1},
\end{equation}
and
\begin{equation}
%\begin{tiny}
%{\scriptsize
[K \ast_2^\top v ]_{2i-1, 2j-1}  =
K_{1,1}v_{i,j} + K_{-1,1}v_{i-1,j} + K_{1,-1}v_{i,j-1} + K_{-1,-1}v_{i-1,j-1}.
%\end{tiny}
%}
\end{equation}

\end{theorem}


\section{Hypothesis testing}
Is the effect you see in a sample likely to appear in a larger population?
There are several ways we could formulate this question, including Fisher null hypothesis testing, Neyman-Pearson decision theory, and Bayesian inference.
The logic of this process is similar to a proof by contradiction.

Steps:
\begin{enumerate}
	\item choose a test statistic, e.g. the difference between means of two groups.
	\item define a null hypothesis
	\item compute a p-value, which is the probability of seeing the apparent effect if the null hypothesis is true.
	\item interpret the result. If the p-value is low, the effect is said to be statistically significant,
		which means that it is unlikely to have occurred by chance.
		In that case we infer that the effect is more likely to appear in the larger population.
\end{enumerate}

Interpret p-values according to their order of magnitude.  If the p-value is
\begin{itemize}
	\item  $<$ 1\%, the effect is unlikely to be due to chance;
	\item $>$ 10\%, the effect can plausibly be explained by chance.
	\item P-values between 1\% and 10\% should be considered borderline.
\end{itemize}

In general the p-value for a one-sided test is about half the p-value for a two-sided test, depending on the shape of the distribution.

\subsubsection{testing a correlation}
If x and y appear correlated, mix up the x, y pairs and look at the correlation of this permuted data.
Repeat 100x, and look at the distribution of correlations seen;
	this gives perspective on whether the correlation is real.

\subsubsection{testing proportions}
E.g. testing whether a die is fair.
Define a test statistic to be the $\sum$ abs((actual counts) - (expected counts)).
Simulate what this test statistic ends up being for a fair die.
The p-value is the fraction of the time this statistic is at your threshold or higher.

If instead you use the square, $\sum$ ((actual counts) - (expected counts))$^2$, large outliers are weighted more strongly, and the probability appears more extreme.
You are now in the realm of Chi-squared tests.

\subsubsection{Chi-squared tests}
$\displaystyle \chi^2 = \sum_i \frac{(O_i - E_i)^2}{E_i}$
$O_i$ are observed frequencies, $E_i$ are expected frequencies.


\chapter{{\tt SemiImplMtx}: Semi-Implicit Factorization}
\label{chapter:SemiImplMtx}
\par
The {\tt SemiImplMtx} object contains a semi-implicit
representation of a sparse matrix factorization.
Assume that the matrix $A$ has been factored as
$PAQ = LDU$, where $L$ is unit lower triangular
and $U$ is unit upper triangular.
Now consider $PAQ$ (and so $L$, $D$ and $U$) partitioned as
follows.
$$
{\widehat A} = PAQ =
\begin{bmatrix}
{\widehat A}_{1,1} & {\widehat A}_{1,2} \\
{\widehat A}_{2,1} & {\widehat A}_{2,2}
\end{bmatrix}
=
\begin{bmatrix}
L_{1,1} &   0     \\
L_{2,1} & L_{2,2}
\end{bmatrix}
\begin{bmatrix}
D_{1,1} &    0    \\
   0    & D_{2,2}
\end{bmatrix}
\begin{bmatrix}
U_{1,1} & U_{1,2} \\
   0    & U_{2,2}
\end{bmatrix}
$$
After some algebra we can arrive at the following identities.
$$
L_{2,1} = {\widehat A}_{2,1} D_{1,1}^{-1}
\text{\qquad and \qquad}
U_{1,2} = D_{1,1}^{-1} {\widehat A}_{1,2}
$$
The straightforward solution of $AX = B$ can be done as follows,
as we solve the permuted linear system
${\widehat A} {\widehat X} = {\widehat B}$,
where ${\widehat X} = Q^T X$ and ${\widehat B} = P B$.
\begin{itemize}
\item solve $L_{1,1} Y_1 = {\widehat B}_1$.
\item solve $L_{2,2} Y_2 = {\widehat B}_2 - L_{2,1} Y_1$.
\item solve $D_{1,1} Z_1 = Y_1$.
\item solve $D_{2,2} Z_2 = Y_2$.
\item solve $U_{2,2} {\widehat X}_2 = Z_2$.
\item solve $U_{1,1} {\widehat X}_1 = Z_1 - U_{1,2} Z_2$.
\end{itemize}
An equivalent process does not requires $L_{2,1}$ and $U_{1,2}$,
but instead uses the ${\widehat A}_{1,2}$ and ${\widehat A}_{2,1}$
matrices.
\begin{itemize}
\item
solve $L_{1,1} D_{1,1} U_{1,1} T_1 = {\widehat B}_1$.
\item
solve $L_{2,2} D_{2,2} U_{2,2} {\widehat X}_2 = {\widehat B}_2 - A_{2,1} T_1$.
\item
solve $L_{1,1} D_{1,1} U_{1,1} {\widehat X}_1
       = {\widehat B}_1 - A_{1,2} {\widehat X}_2$.
\end{itemize}
In effect, we have traded multiplies with $L_{2,1}$ and $U_{1,2}$
for multiplies with $A_{1,2}$ and $A_{2,1}$ and
two extra solves with $D_{1,1}$.
In some cases this {\it semi-implicit} procedure
(so named because $L_{2,1}$ and $U_{1,2}$ are stored in a
semi-implicit form) can pay off ---
storage can be saved when the number of entries in
$L_{2,1}$ and $U_{1,2}$ are larger than the number of entries in
$A_{2,1}$ and $A_{1,2}$.
The number of solve operations is reduced by
$|L_{2,1}| + |U_{1,2}| - 2|D_{1,1}| -|A_{2,1}| - |A_{1,2}|$,
where $|\cdot|$ denotes the number of nonzeroes in a matrix.


\subsection{Absolute risk aversion}

Given a utility function we can calculate the risk aversion.

\(A(x)=-\dfrac{u''(x)}{u'(x)}\)

Constant Absolute Risk Aversion (CARA) is:

\(A(x)=c\)

\(u(x)=1-e^{\alpha x}\)

Hyperbolic Absolute Risk Aversion (HARA) is:

\(A(x)=\dfrac{1}{ax+b}\)

Increasing and Decreasing Absolute Risk Aversion (IARA and DARA):

Risk aversion increase or decreases in \(x\).


\chapter{Stream}
\section{Sliding Window}
\runinhead{Sliding Window Maximum.} Given an array $nums$, Find the list of maximum in the sliding window of size $k$ which is moving from the very left of the array to the very right. $\rightarrow$ double-ended queue.

Invariant: the queue is storing the non-decreasing-ordered elements of current window.

\runinhead{Sliding Window Median.} Find the list of median in the sliding window. $\rightarrow$ Dual heap with lazy deletion - section \ref{dh_lazy_del}.


% Time-stamp: <2004/04/06, 16:46:43 (EST), maverick, test.tex>
\subsection{Strict diagonal-dominance}
Suppose we are given a matrix $A=L+D$, where $L$ is a Laplacian and
$D$ is a nonnegative diagonal matrix, for which we seek to construct a
preconditioner.

We may construct a Support Tree Preconditioner, $B =
\begin{pmatrix} T & U\\U\TT & W\end{pmatrix}$ for $L$ and to use $B'
=\begin{pmatrix} T & U \\U\TT & W+D\end{pmatrix}$ as a preconditioner
for $A$.  If we let $Q = W - U\TT T\IV U$, by Lemma~\ref{lem:stcg} it
suffices to bound $\sigma(A/Q+D)$ and $\sigma(Q+D/A)$.

\begin{proposition}\label{prop:XZ-YZ}
If $X$, $Y$, and $Z$ are spsd matrices of the same size then
$\sigma(X+Z/Y+Z) \leq \max\{\sigma(X/Y),\, 1\}$.
\end{proposition}

\Proof We have $\sigma(X+Z/Y+Z) =
\min\{\tau \mid \forall\vv{x},\, \tau\cdot \vv{x}\TT (Y+Z)\vv{x} \geq
       \vv{x}\TT(X+Z)\vv{x}\} =
\min\{\tau \mid \forall\vv{x},\, (\tau-1)\cdot \vv{x}\TT Z\vv{x} +
      \tau \cdot\vv{x}\TT Y\vv{x} \geq \vv{x}\TT X\vv{x}\} \leq
\max\{1,\,\sigma(X/Y)\}$.\QED


\section{Bayesian linear regression}

\subsection{Linear regression revisited}
\begin{itemize}
\item Suppose that we have data $D=\{(\xx_1, t_1), \ldots, (\xx_N, t_N) \}$
\item We assume that the data can be modelled by some function $t_n \approx y(x, \omega)+ \epsilon$, where $\epsilon$ models additive noise.
\item  We assume that noise is independent, identically distributed and Gaussian:
\begin{align}
	\epsilon &\sim \mathcal{N}(0,\beta^{-1})\\
	t|\xx, \omega, \beta &\sim \mathcal{N}(y(\xx,\omega), \beta^{-1})
\end{align}
\end{itemize}

\begin{figure}
	\centering
	\includegraphics[width=.5\textwidth]{./lecture5/Figure128.pdf}
	\caption{Linear regression. Bishop PRML Figure 1.28}
\end{figure}

\subsection{Maximum likelihood estimation for linear regression}
\begin{itemize}
\item We consider a linear model $y(\xx,\omega)= \omega^\top \xx$
\item ~[On board: Likelihood, Maximum Likelihood Solution, Predictive distribution from MLE]\\
\item  For a Bayesian treatment, we need priors.
\end{itemize}

\begin{bbbox}{Maximum likelihood estimation for linear regression}
	\begin{align*}
			L(t | x,\omega) &= \log \left( \prod_{n=1}^N p(t_n |  x_n, \omega ) \right) \\
			        &= \sum_{n=1}^N \log \left( \frac{1}{\sqrt(2\pi)} \exp \left( - \frac{\beta}{2} \left( t_n - 	\omega^{\top} x_n \right)^2 \right) \right) \\
			        &= const. + \sum_{n=1}^N - \frac{\beta}{2} \left( t_n - \omega^{\top} x_n \right)^2 \\
			        &= const - \underbrace{\frac{\beta}{2} \sum_{n=1}^N \left( t_n - \omega^{\top} x_n \right)^2}_{AMSE - average mean squared error} \\
	   \hat{\omega} &= \left( \sum_{n=1}^N x_n x_n^{\top} \right)^{-1} \left( \sum_{n=1} x_n t_n \right) \\			 	    \end{align*}
	Predictive distribution \\
	$P(t^{*} | x^{*},0) = P(t^{*} | x^{*},\hat{\omega}) = \mathcal{N}(\omega^{\top} x^{*}, \frac{1}{\beta})$
\end{bbbox}

\subsection{Maximum a posteriori estimation for the linear regression.}
\begin{align}
\omega_i &\sim \mathcal{N}(0,\alpha^{-1})\\
p(\omega| \alpha) &= \prod_{i=1}^M  \sqrt{\frac{\alpha}{{2\pi}}}\exp\left( -\frac{\alpha}{2}  {\omega_i^2}\right)\\
&%
 =\left(\frac{\alpha}{2\pi}\right)^{M/2} \exp \left(-\frac{\alpha}{2} \omega^\top \omega  \right) \\
 \text{where } \forall i,j: \alpha_i &= \alpha_j
\end{align}

\begin{bbbox}{Finding the maximum-a-posteriori of $\omega$}
	\begin{align*}
		L_{post}(t | x,\omega) &= \log \left( \prod_{n=1}^N p(t_n |  x_n, \omega ) p(\omega|\alpha)\right) \\
		 &= \sum_{n=1}^N \log \left( \frac{1}{\sqrt(2\pi)} \exp \left( - \frac{\beta}{2} \left( t_n - 	\omega^{\top} x_n \right)^2 \right) \frac{\alpha}{2\pi}^{\frac{M}{2}} \exp \left( -\frac{\alpha}{2} \omega^{\top}\omega \right) \right) \\
		 &= const. - \frac{\beta}{2} \sum_{n=1}^N \left( t_n - \omega^{\top} x_n \right)^2 -\frac{\alpha}{2} \omega^{\top}\omega \\
	\hat{\omega} &= \left( \sum_{n=1}^N x_n x_n^{\top} + \frac{\alpha}{\beta} \mathbf{I}_M \right)^{-1} \left( \sum_{n=1} x_n t_n \right) \\
	\end{align*}
\end{bbbox}

\subsection{Basis functions in linear regression}
\textbf{If you are smart about choosing good basis functions ('features'), linear regression can get you pretty far.}
\begin{itemize}
\item If we use nonlinear basis functions $\phi(x)$, can model nonlinear relationships with $y(\omega, \xx)= \omega^\top \phi(x)$.
\item Polynomial regression: $\phi(x)=(1,x,x^2,x^3)$ ; (cubic feature space)
\item 'Gaussian bumps': $\phi_i(x)= \exp\left((x-s_i)^2/\sigma_i^2 \right)$
\item Sigmoids $\phi_i(x)=1/(1+\exp(-x-s_i))$
\item  'Kernel methods' are essentially linear algorithms which take one basis function per data-point.
\item Predictive Mean [on board]
\end{itemize}

\begin{figure}
\centering
	\begin{subfigure}[b]{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{./lecture5/Figure31a.pdf}
                \caption{Polynomial basis.}
    \end{subfigure}%
	~
	\begin{subfigure}[b]{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{./lecture5/Figure31b.pdf}
                \caption{Gaussian bumps.}
    \end{subfigure}%
	~
	\begin{subfigure}[b]{0.3\textwidth}
                \centering
                \includegraphics[width=\textwidth]{./lecture5/Figure31c.pdf}
                \caption{Logistic sigmoids.}
    \end{subfigure}%
    \caption{Non-linear basis functions, taken from Bishop PRML}
\end{figure}

\subsection{Bayesian linear regression}

\textbf{Bayesian linear regression takes into account our uncertainty about parameters.}

\begin{itemize}
\item Posterior distribution is Gaussian $\rightarrow$ Posterior mean and MAP coincide!
\item  However, neither the MLE nor the MAP solution take into account that we have (posterior) uncertainty about the parameters
\end{itemize}

\begin{figure}
\centering
	\begin{subfigure}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=\textwidth]{./lecture5/LinReg.pdf}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=\textwidth]{./lecture5/LinRegBayes.pdf}
	\end{subfigure}
\end{figure}

\subsection{Gaussian processes}
\textbf{Illustration: Climate prediction [by Carl Rasmussen, University of Cambridge, using Gaussian Processes]}

\begin{figure}
	\centering
	\includegraphics[width=0.5\textwidth]{./lecture5/Rasmussen2}
	\caption{Gaussian process regression.}
\end{figure}

\subsection{Sequential update of the posterior distribution}
\textbf{Bayesian regression illustrated: The more data we observe, the more constrained the parameters are.}

\begin{figure}
	\includegraphics[width=\textwidth]{./lecture5/Figure37.pdf}
	\caption{Sequential update of the Gaussian posterior, figures taken from Bishop PRML}
\end{figure}

\textbf{Calculating the predictive mean and variance for Bayesian regression}

Assume $\alpha$ and $\beta$ as given.\\ Posterior distribution: [derivation on board]

\begin{bbbox}{Bayesian linear regression: calculating posterior mean and variance}
	Recall the quadratic form for the Gaussian: \\
	\begin{align*}
		&\frac{1}{Z} \exp \left( -\frac{1}{2} \left( x-\mu \right)^{\top} \Sigma^{-1} \left( x-\mu \right) \right) \\
		           &= \frac{1}{Z} \exp \left( -\frac{1}{2} \left( x^{\top}\Sigma^{-1}x + \mu^{\top}\Sigma^{-1}\mu
		             - \mu^{\top}\Sigma^{-1}x - x^{\top}\Sigma^{-1}\mu \right) \right)\\
		           &= \frac{1}{Z} \exp \left( -\frac{1}{2} x^{\top}\Sigma^{-1}x + x^{\top}\Sigma^{-1}\mu + const. \right) \\
	\end{align*}
	Here again we can easily read out the posterior mean and covariance from the two terms. \\
	We say that $\omega | D \sim \mathcal{N}\left( \mu_{post}, \Sigma_{post}\right)$. \\
	Since the mode and mean of a Gaussian are the same thing we already know from our MAP estimation that:
	$ \mu_{post} = \left( \sum_{n=1}^N x_n x_n^{\top} + \frac{\alpha}{\beta} \mathbf{I}_M \right)^{-1} \left( \sum_{n=1} x_n t_n \right)$ \\
	The next thing we have to do is to bring the posterior  distribution into the quadratic form from above and to read out the posterior covariance. Note that all terms that are independents from $\omega$ will sucked into the constant term: \\
	\begin{align*}
		P(\omega|D) &= \frac{1}{Z_1} \prod_{n=1}^N P(t_n|x_n,\omega) P(\omega) \\
		&= \frac{1}{Z_2} \exp \left( \sum_{n=1}^N -\frac{\beta}{2} \left( \omega^{\top} x_n - t_n \right) ^2 - \frac{\alpha}{2} \omega^{\top} \omega \right) \\
		&= \frac{1}{Z_2} \exp \left( -\frac{\beta}{2} \sum_{n=1}^N \left( \omega^{\top} x_n - t_n \right) \left( \omega^{\top} x_n - t_n \right)^{\top} - \frac{\alpha}{2} \omega^{\top} \omega \right) \\
		&= \frac{1}{Z_3} \exp \left( -\frac{1}{2} \left( \beta \sum_{n=1}^N \omega^{\top} x_n x_n^{\top} \omega + \alpha \omega^{\top} \omega \right) \right) \\
		&= \frac{1}{Z_3} \exp \left( -\frac{1}{2} \left( \omega^{\top} \left( \beta \sum_{n=1}^N x_n x_n^{\top} + \alpha \mathbf{I}_M \right) \omega \right) \right) \\
		\Sigma_{post}^{-1}&=\alpha \mathbf{I}_M + \beta \sum_{n=1}^N x_n x_n^\top\\
	\end{align*}
\end{bbbox}

\begin{align}
\Sigma_{post}^{-1}&=\alpha \mathbf{I}+\beta \sum_i x_i x_i^\top\\
\mu_{post}&=\Sigma_{post} \beta \sum_i x_i t_i
\end{align}

Predictive distribution: [derivation on board]

\begin{bbbox}{Predictive distribution}
	We note that in the case of a regression the predictive error arises from our uncertainty about the parameters as well as the variance which is inherent to our predicted variable. We want to calculate the predictive distribution starting from the observation that making a prediction is basically a concatenation of two random variables.
	\begin{align*}
		t^* &| D, x_*: \;
		t^* = \underbrace{\omega^{\top}x_*}_{y^*} + \epsilon \\
%		\mu &\sim \mathcal{N}(\mu_0, \frac{1}{\tau}); \;
%		x|\mu \sim \mathcal{N}(\mu,\frac{1}{\beta}); \;
%		\rightarrow x \sim \mathcal{N}(\mu_0, \frac{1}{\tau} + \frac{1}{\beta}) \\
		y^* &\sim \mathcal{N} (\mu_*,\frac{1}{\tau_{*}}); \;
		t^*|y^* \sim \mathcal{N}(y^*,\frac{1}{\beta}); \;
		\rightarrow \underbrace{t^* \sim \mathcal{N}(\mu_*,\frac{1}{\tau_{*}} + \frac{1}{\beta})}_{\text{Predictive distribution}} \\
	\end{align*}

We see that the variance of our predictive distribution is calculated from the variance in our parameter estimate and from the variance in $t$. We can further express the posterior distribution based on:
	\begin{align*}
		\mu_* &= E(y^* | D) = E(\omega^{\top} x_* | D) \\
		      &= E(\omega^{\top} | D) x_* = E(\omega | D)^{\top} x_* \\
		      &= \mu_{post}^{\top} x_* \\
		\frac{1}{\tau_{*}} &= \mbox{Var}(y^* | D) \\
		     &= \mbox{Var}(\omega^{\top} x_* | D) = x_*^{\top} \mbox{Var}(\omega^{\top} | D) x_* \\
   		     &= x_*^{\top} \Sigma_{post} x_* \\
	\end{align*}
\end{bbbox}

\begin{align}
E(t^*|D,x^*)&= \mu_{post}^\top x^* \\
\mbox{Var}(t^*|D,x^*)&=  1/\beta+ x^{*\top} \Sigma_{post} x^*
\end{align}
What if there are basis functions? [on board]

\subsection{Fully Bayesian linear regression}

\textbf{But, where do we get $\alpha$ and $\beta$ from?}

\begin{itemize}
\item Bad news: Getting these makes things more complicated.
\item  Good news: This will not be on the exam (unless I take that back explicitly...).
\item  'Full' Bayesian inference: Integrate out $\alpha$, $\beta$. No closed form solution. Use (e.g.) variational inference.
\item  Practical solution: optimize $\alpha$ and $\beta$ by \emph{maximizing the evidence}, also known as \emph{marginal likelihood} or \emph{likelihood type 2}
\begin{align}
E&=\log P(\alpha, \beta|D)\\
&= \log \int_\omega p(D|\omega,\beta) p(\omega|\alpha) p(\alpha,\beta)    d \omega\\
&= \frac{M}{2}\log(\alpha)+\frac{N}{2}\log \beta-\mbox{M}(\mu_{post})+\frac{1}{2}|\Sigma_{post}|-\frac{N}{2} \log(2\pi)
\end{align}
\item $\mu_{post}$ and $\Sigma_{post}$ are the posterior mean and covariance, and $\mbox{M}(\mu_{post})=\frac{\beta}{2}\sum_n (t_n-y(\mu_{post},\xx_n))^2+\frac{\alpha}{2} \mu_{post}\mu_{post}^\top$ is the quadratic cost function evalulated at the posterior mean (see Bishop 3.5 for details).
\end{itemize}

\textbf{Optimization of the marginal likelihood is a Bayesian alternative to parameter-setting by cross-validation.}

\textbf{What we have not had time to cover:}
\begin{itemize}
\item \emph{Non-Gaussian priors:} The most important non-Gaussian prior is the `Laplace prior' ('L1 regularization'), which leads to sparse MAP-solutions.
\item \emph{Non-Gaussian noise models:} If you know that your noise is not Gaussian but, say, Poisson, use 'generalized linear regression'. Some choices of noise models are more robust to outliers than the Gaussian. We will do one specific example of generalized linear regression in the last lecture.
\item \emph{Nonlinear regression models:} The most important Bayesian nonlinear regression technique is \emph{Gaussian process regression}. In a nutshell, GP regression is like linear regressions but the algorithm puts ne basis function at each data-point. To understand GP regression, you need to understand Gaussians.
\end{itemize}