\chapter{Medians and Order Statistics}

\section{Minimum and maximum}

\begin{enumerate}

\item[9.1{-}1]{Show that the second smallest of $n$ elements can be found with
$n + \ceil{\lg n} - 2$ comparisons in the worst case. (\emph{Hint:} Also find
the smallest element.)}

\begin{framed}
Lets find first the smallest element. Compare the elements in pairs and discard
the largest element of each pair. The number of elements is now $\ceil{n/2}$.
Repeat this operation recursively to the remaining elements until the smallest
element is found. Since we discard one element in each comparison, the number of
comparisons is the number of elements that is not the smaller.  Thus, $n - 1$
comparisons. Note that the second smallest element can only be greater than the
smallest element. Thus, the second smallest element is among these
$\ceil{\lg n}$ elements that were discarded when compared to the smallest
element. Use the same recursive approach on these $\ceil{\lg n}$ elements to
find the second smallest with $\ceil{\lg n} - 1$ comparisons. The total number
of comparisons in the worst-case is then $n - 1 + \ceil{\lg n}
- 1 = n + \ceil{\lg n} - 2$.
\end{framed}

\item[9.1{-}2]{($\star$) Prove the lower bound of $\ceil{3n/2} - 2$ comparisons
in the worst case to find both the maximum and minimum of $n$ numbers.
(\emph{Hint:} Consider how many numbers are potentially either the maximum or
minimum, and investigate how a comparison affects these counts.)}

\begin{framed}
At the start, any of the $n$ the elements can be both the minimum and the
maximum. After the first comparison, we can discard the largest as not being the
minimum and the smallest as not being the maximum. From now on we have two
options: compare two different elements or compare one of the elements
previously compared with a different element. The first option will decrease by
one both the number of potential minimums and potential maximums, while the
second option will only decrease one of these totals. Thus, the best way to
start is to group the elements in pairs and compare them, which requires
$\floor{n/2}$ comparisons. After comparing all the pairs, we will have
$\ceil{n/2}$ potential maximums and $\ceil{n/2}$ potential minimums. In the
worst-case, those sets are disjoint and must be treated independently. We know
from the previous question that the minimum number of comparisons needed to find
the minimum or the maximum among $\ceil{n/2}$ elements is $\ceil{n/2} - 1$.
Thus, the lower bound to find both the maximum and the minimum of $n$ numbers in
the worst-case is
\[
  \Bigl\lfloor \frac{n}{2} \Bigr\rfloor + 2 \left( \Bigl\lceil \frac{n}{2} \Bigr\rceil - 1 \right).
\]

If $n$ is even, we have
\[
  \Bigl\lfloor \frac{n}{2} \Bigr\rfloor + 2 \left( \Bigl\lceil \frac{n}{2} \Bigr\rceil - 1 \right)
  = \frac{n}{2} + n - 2
  = \frac{3n}{2} - 2
  = \Bigl\lceil \frac{3n}{2} \Bigr\rceil - 2.
\]

If $n$ is odd, we have
\[
  \Bigl\lfloor \frac{n}{2} \Bigr\rfloor + 2 \left( \Bigl\lceil \frac{n}{2} \Bigr\rceil - 1 \right)
  = \frac{n - 1}{2} + (n + 1) - 2
  = \frac{3n - 3}{2}
  = \Bigl\lceil \frac{3n}{2} \Bigr\rceil - 2.
\]
\end{framed}

\end{enumerate}

\newpage

\section{Selection in worst-case linear time}

\begin{enumerate}

\item[9.2-1]{Show that \textsc{Randomized-Select} never makes a recursive call
to a 0-length array.}

\begin{framed}
At the start of each recursive call, a random pivot is chosen. If it happens to
be the $i$th element, the element being searched has been found and is returned
without any additional recursion call. Otherwise, the $i$th element is either
before or after the pivot and a recursive call is made on the side of the
subarray that includes the $i$th element.
\end{framed}

\item[9.2-2]{Argue that the indicator random vartiable $X_k$ and the value
$T(\max(k - 1, n - k))$ are independent.}

\begin{framed}
Both $X_k$ and $T(\max(k - 1, n - k))$ depends on the value of $k$. However, no
matter if $X_k$ is 0 or 1, the value of $T(\max(k - 1, n - k))$ is the same.
\end{framed}

\item[9.2-3]{Write an iterative version of \textsc{Randomized-Select}.}

\begin{framed}
The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Randomized-Select-Iterative}
\SetKwProg{myalg}{}{}{}
\nonl\myalg{\algo{A, p, r, i}}{%
  \If{$p == r$}{%
    \Return{$A[p]$}\;
  }
  \While{\texttt{\upshape{True}}}{%
    $q = \texttt{Randomized-Partition}(A, p, r)$\;
    $k = q - p + 1$\;
    \If{$i == k$}{%
      \Return{$A[q]$}\;
    }
    \ElseIf{$i < k$}{%
      $r = q - 1$\;
    }
    \Else{%
      $p = q + 1$\;
      $i = i - k$\;
    }
  }
}
\end{algorithm}
\end{framed}

\item[9.2-4]{Suppose we use \textsc{Randomized-Select} to select the minimum
element of the array $A = \langle 3, 2, 9, 0, 7, 5, 4, 8, 6, 1 \rangle$.
Describe a sequence of partitions that results in a worst-case performance of
\textsc{Randomized-Select}.}

\begin{framed}
The worst-case occurs when the pivot is always the greatest element. The number
of calls to partition in this case is $n - 1$.
\end{framed}

\end{enumerate}

\newpage

\section{Selection in worst-case linear time}

\begin{enumerate}

\item[9.3-1]{In the algorithm \textsc{Select}, the input elements are divided
into groups of 5. Will the algorithm work in linear time if they are divided
into groups of 7? Argue that \textsc{Select} does not run in linear time if
groups of 3 are used.}

\begin{framed}
If the elements are divided into groups of 7, the number of elements
greater/smaller than the median-of-medians is at least
\[
  7 \left( \Bigl\lceil \frac{1}{2} \Bigl\lceil \frac{n}{7} \Bigr\rceil \Bigr\rceil - 2 \right)
  \ge \frac{4n}{14} - 8 = \frac{2n}{7} - 8,
\]
which implies that, in the worst-case, step 5 calls \textsc{Select} recursively
on at most
\[
  n - \left( \frac{2n}{7} - 8 \right) = \frac{5n}{7} + 8
\]
elements. We then have the recurrence
\[
  T(n) = T\left( \Bigl\lceil \frac{n}{7} \Bigr\rceil \right) + T\left( \frac{5n}{7} + 8 \right) + O(n).
\]

We shall prove that its running time is linear by substitution. More
specifically, we will show that
\[
    T(n) \le cn \; \Forall n \ge n_0,
\]
where $c$ and $n_0$ are positive constants. Substituting into the recurrence,
yields
\begin{equation*}
\begin{aligned}
  T(n) &\le c \Bigl\lceil \frac{n}{7} \Bigr\rceil + c \left( \frac{5n}{7} + 8 \right) + an\\
       &\le c \frac{n}{7} + c + c \frac{5n}{7} + 8c + an & \text{($c \ge 1$)}\\
       &=   \frac{6}{7} cn + 9c + an\\
       &=   cn + \left( - \frac{1}{7} cn + 9c + an \right)\\
       &\le cn,
\end{aligned}
\end{equation*}
where the last step holds for
\[
  -\frac{1}{7} cn + 9c + an \le 0 \rightarrow c \ge 7a \left(\frac{n}{n - 63}\right),
\]
and picking $n_0 = 126$, it holds for $c \ge 14a$.

Similarly, with groups of 3, the number of elements greater/smaller than the
median-of-medians is at least
\[
  2 \left( \Bigl\lceil \frac{1}{2} \Bigl\lceil \frac{n}{3} \Bigr\rceil \Bigr\rceil - 2 \right)
  \ge \frac{n}{3} - 4.
\]
which implies that, in the worst-case, step 5 calls \textsc{Select} recursively
on at most
\[
  n - \left( \frac{n}{3} - 4 \right) = \frac{2n}{3} + 4
\]
elements. We then have the recurrence
\[
  T(n) = T\left( \Bigl\lceil \frac{n}{3} \Bigr\rceil \right) + T\left( \frac{2n}{3} + 4 \right) + O(n).
\]

We shall prove that its running time is $\omega(n)$ by substitution. More
specifically, we will show that
\[
  T(n) > cn + d \; \Forall n \ge n_0,
\]
where $c$, $d$, and $n_0$ are positive constants. Substituting into the
recurrence, yields
\begin{equation*}
\begin{aligned}
  T(n) &> c \Bigl\lceil \frac{n}{3} \Bigr\rceil + d + c \left( \frac{2n}{3} + 4 \right) + d + an\\
       &> c \frac{n}{3} + c + d + c \frac{2n}{3} + 4c + d + an & \text{($c \ge 1$)}\\
       &= cn + 5c + 2d + an\\
       &> cn,\\
\end{aligned}
\end{equation*}
where the last step holds for $5c + 2d + an > 0$.
\end{framed}

\item[9.3-2]{Analyze \textsc{Select} to show that if $n \ge 140$, then at least
$\ceil{n/4}$ elements are greater than the median-of-medians $x$ and at least
$\ceil{n/4}$ elements are less than $x$.}

\begin{framed}
We have that at least
\[
  \frac{3n}{10} - 6
\]
elements are greater/smaller than $x$. To this number be equal to or greater
than $\ceil{n/4}$, we find $n$ such that
\begin{equation*}
\begin{aligned}
  \frac{3n}{10} - 6 \ge \Bigl\lceil \frac{n}{4} \Bigr\rceil
  &\rightarrow \frac{3n}{10} - 6 \ge \frac{n}{4} + 1\\
  &\rightarrow \frac{6n - 5n}{20} \ge 7\\
  &\rightarrow \frac{n}{20} \ge 7\\
  &\rightarrow n \ge 140.
\end{aligned}
\end{equation*}
\end{framed}

\item[9.3-3]{Show how quicksort can be made to run in $O(n \lg n)$ time in the
worst-case, assuming that all elements are distinct.}

\begin{framed}
Update the partition procedure to use the median as the pivot. It will take an
additional $O(n)$-time to find the median with the \textsc{Select} procedure,
but the running time of partition will still be linear. We will then have the
recurrence
\[
  T(n) = 2T \left( \frac{n}{2} \right) + O(n),
\]
which takes
\[
  \sum_{i = 0}^{\lg n} 2^i \cdot \frac{n}{2^i} = \sum_{i = 0}^{\lg n} n = O(n \lg n).
\]
\end{framed}

\item[9.3-4]{($\star$) Suppose that an algorithm uses only comparisons to find
the $i$th smallest element in a set of $n$ elements. Show that it can also find
the $i - 1$ smaller elements and the $n - i$ larger elements without performing
any additional comparisons.}

\begin{framed}
Assume without loss of generality that the elements of the array are distinct.
Let $x$ denote the $i$th order statistic that was found through comparisons.
First note that if there exists an element $y$ that was never compared to any
other element, its value was not taken into account to determine $x$, which
implies that there are at least two possible order statistics for $x$ {--} one
for $y < x$ and another for $y > x$. The same occurs if $y$ is only compared to
elements that are not between $x$ and $y$ in the sorted order. Note that these
comparisons are insufficient to determine if $y$ is smaller or greater than $x$,
and there will also be at least two possible order statistics for $x$.
Therefore, to find $x$, the algorithm must compare $y$ to $x$ directly or by
transitivity. These comparisons are sufficient to determine the relative order
of every element with respect to $x$, and therefore to also determine the
$i - 1$ smaller and the $n - i$ greater elements of the array.
\end{framed}

\newpage

\item[9.3-5]{Suppose that you have a ``black-box'' worst-case linear-time median
subroutine. Give a simple, linear-time algorithm that solves the selection
problem for an arbitrary order statistic.}

\begin{framed}
A simple algorithm works as follows:
\begin{enumerate}
\item Find the lower median $m$ using the ``black-box'' median subroutine.
\item If $i = \ceil{n/2}$, just return $m$. Otherwise, partition the array using
$m$ as the pivot and recursively find the $i$th element on the first
$\ceil{n / 2} - 1$ elements if $i < \ceil{n / 2}$, or the $(i - \ceil{n / 2})$th
element on the last $\floor{n/2}$ elements if $i > \ceil{n/2}$.
\end{enumerate}

This algorithm has the recurrence
\[
  T(n) = T(n/2) + O(n),
\]
which can be solved using case 3 of the master method, since $n^{\lg 1}$ is
polynomially smaller than $f(n)$. Thus, $T(n) = \Theta(n)$.

The pseudocode of this algorithm is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{Select'}
\SetKwProg{myalg}{}{}{}
\nonl\myalg{\algo{A, p, r, i}}{%
  $m = \texttt{Median}(A, p, r)$\;
  $k = \ceil{n/2}$\;
  \If{$i == k$}{%
    \Return{$m$}\;
  }
  \Else{%
    $q = \texttt{Partition}(A, p, r, m)$\;
    \If{$i < k$}{%
      // recurve over the first $\ceil{n/2} - 1$ elements\;
      $\texttt{Select'}(A, p, p + k - 2, i)$\;
    }
    \Else{%
      // recurve over the last $\floor{n/2}$ elements\;
      $\texttt{Select'}(A, p + k, r, i - k)$\;
    }
  }
}
\end{algorithm}

\end{framed}

\newpage

\item[9.3-6]{The $k$th \textbf{\emph{quantiles}} of an $n$-element set are the
$k - 1$ order statistics that divide the sorted set into $k$ equal-sized sets
(to within 1). Give an $O(n \lg k)$-time algorithm to list the $k$th quantiles
of a set.}

\begin{framed}
Let $S$ be an $n$-set and $S_{(i)}$ denote the $i$th order statistic of $S$. The
$k$th quantiles of $S$ are the elements
\[
  S_{(1 (n / k))}, S_{(2 (n / k))}, \dots, S_{((k - 1) (n / k))}.
\]
An efficient algorithm to find the above elements work as follows:
\begin{enumerate}
\item If $k = 1$, then return $\emptyset$.
\item Otherwise, do the following:
\begin{enumerate}
\item Partition $S$ around the element $S_{(\floor{k / 2} (n / k))}$. Let $q$
denote the position of the pivot after partition and let $S_1$ and $S_2$ denote
the subsets $S[1, \dots, q]$ and $S[q + 1, \dots, n]$, respectively.
\item Recursively solve the $(\floor{k/2})$th quantiles of $S_1$ and the
$(\ceil{k/2})$ quantiles of $S_2$. Let $Q_1$ and $Q_2$ denote the solutions of
$S_1$ and $S_2$, respectively.
\item Return $Q_1 \cup \{S[q]\} \cup Q_2$.
\end{enumerate}
\end{enumerate}

We shall now prove that this algorithm runs in $O(n \lg k)$. First note that
since
\[
  n \text{ mod } k = 0,
\]
$k$ is even implies that $n$ is also even. Thus, for even $k$, we have
\begin{equation*}
\begin{aligned}
  \Bigl\lfloor \frac{k}{2} \Bigr\rfloor \cdot \frac{n}{k}
  &= \frac{k}{2} \cdot \frac{n}{k}\\
  &= \frac{n}{2}\\
  &= \Bigl\lfloor \frac{n}{2} \Bigr\rfloor,
\end{aligned}
\end{equation*}
which implies that $q$ is the lower median. When $k$ is odd, we have
\begin{equation*}
\begin{aligned}
  \Bigl\lfloor \frac{k}{2} \Bigr\rfloor \cdot \frac{n}{k}
  &=   \frac{k - 1}{2} \cdot \frac{n}{k}\\
  &=   \left( \frac{k}{2} - \frac{1}{2} \right) \frac{n}{k}\\
  &=   \frac{n}{2} - \frac{n}{2k}.
\end{aligned}
\end{equation*}
Step (a) takes $O(1)$. Step (b) has the recurrence
\begin{equation*}
T(n, k) =
\begin{cases}
  O(1), & k = 1\\
  T\left( \Bigl\lfloor \frac{n}{2} \Bigr\rfloor, \Bigl\lfloor \frac{k}{2} \Bigr\rfloor \right) +
  T\left( \Bigl\lceil \frac{n}{2} \Bigr\rceil, \Bigl\lceil \frac{k}{2} \Bigr\rceil \right) +
  O(n), & \text{$k > 1$ and $k$ is even}\\
  T\left( \frac{n}{2} - \frac{n}{2k}, \Bigl\lfloor \frac{k}{2} \Bigr\rfloor \right) +
  T\left( \frac{n}{2} + \frac{n}{2k}, \Bigl\lceil \frac{k}{2} \Bigr\rceil \right) +
  O(n), & \text{$k > 1$ and $k$ is odd}
\end{cases}
\end{equation*}

We shall solve this recurrence through the analysis of its recursion-tree. Since
the problem is always divided into two subproblems, without overlap, the total
cost over all nodes at depth $i$ is $cn$. The bottom level at depth $\lg k$
has $2^{\lg k} = k$ nodes, each contributing cost $O(1)$, for a total cost of
$O(k)$. Thus, the cost of the entire tree is \begin{equation*}
\begin{aligned}
  T(n, k) &= \sum_{i = 0}^{\lg k - 1} cn + O(k)\\
          &= cn \lg k + O(k)\\
          &= O(n \lg k).
\end{aligned}
\end{equation*}

\end{framed}

\newpage

\item[9.3-7]{Describe an $O(n)$-time algorithm that, given a set $S$ of $n$
distinct numbers and a positive integer $k \le n$, determines the $k$ numbers in
$S$ that are closest to the median of $S$.}

\begin{framed}
Let $A$ be an array of size $n$. The following algorithm finds $k$ elements of
$A$ such that every element
\begin{itemize}
\item is greater than or equal to the $(\floor{n/2} - \floor{(k - 1)/2})$th
  order statistic of $A$, and
\item is lower than or equal to the $(\floor{n/2} + \ceil{(k - 1)/2})$th order
  statistic of $A$.
\end{itemize}

Do the following steps:
\begin{enumerate}
\item Find the $q$th order statistic of $A$, such that
$q = \floor{n/2} - \floor{(k - 1)/2}$, and partition $A$ around this element.
\item If $k = 1$, return $A[q]$.
\item Otherwise, do the following:
\begin{enumerate}
\item Let $A'$ denote subarray $A[q, \dots, n]$.
\item Find the $k$th order statistic of $A'$ and partition $A'$ around this
element.
\item Return the subarray $A'[q, \dots, q + k - 1]$.
\end{enumerate}
\end{enumerate}

The algorithm do at most two selections and two partitions. Thus, its running
time is $4 \cdot O(n) + O(1) = O(n)$.
\end{framed}

\item[9.3-8]{Let $X[1 \dots n]$ and $Y[1 \dots n]$ be two arrays, each
containing $n$ numbers already in sorted order. Give an $O(\lg n)$-time algorithm
to find the median of all $2n$ elements in arrays $X$ and $Y$.}

\begin{framed}
Note that, since both arrays are sorted, the order statistic of $X[i]$ is
$\floor{(n + 1)/2}$ (the median) if, and only if,
\[
  X[i] \ge Y\left[ \Bigl\lfloor \frac{n + 1}{2} \Bigr\rfloor - i \right],
\]
and
\[
  X[i] \le Y\left[ \Bigl\lfloor \frac{n + 1}{2} \Bigr\rfloor - i + 1 \right].
\]

Start testing the median of $X$. If the first comparison fails, recurse over the
right half. If the second comparison fails, recurse over the left half.
Otherwise, return $X[i]$. If a recursion is performed on an empty array, the
median is not within $X$. Repeat a similar procedure on $Y$ to find the median.
The complexity of this algorithm is $O(\lg n) + O(\lg n) = O(\lg n)$.
\end{framed}

\item[9.3-9]{Professor Olay is consulting for an oil company, which is planning
a large pipeline running east to west through an oil field of $n$ wells. The
company wants to connect a spur pipeline from each well directly to the main
pipeline along a shortest route (either north or south), as shown in Figure 9.2.
Given the $x$- and $y$-coordinates of the wells, how should the professor pick
the optimal location of the main pipeline, which would be the one that minimizes
the total length of the spurs? Show how to determine the optimal location in
linear time.}

\begin{framed}
The optimal locations are the lower and upper medians of the $y$ values. Find
one of them with the \textsc{Select} algorithm.
\end{framed}

\end{enumerate}

\newpage

\section*{Problems}
\addcontentsline{toc}{section}{\protect\numberline{}Problems}%

\begin{enumerate}

\item[9-1]{\textbf{\emph{Largest i numbers in sorted order}}\\
Given a set of $n$ numbers, we wish to find the $i$ largest in sorted order
using a comparison-based algorithm. Find the algorithm that implements each of
the following methods with the best asymptotic worst-case running time, and
analyze the running times of the algorithms in terms of $n$ and $i$.

\begin{enumerate}
  \item[\textbf{a.}] Sort the numbers, and list the $i$ largest.
  \item[\textbf{b.}] Build a max-priority queue from the numbers, and call \textsc{Extract-Max} $i$ times.
  \item[\textbf{c.}] Use an order-statistic algorithm to find the $i$th largest number,
    partition around that number, and sort the $i$ largest numbers.
\end{enumerate}
}

\begin{framed}
\begin{enumerate}
\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{LargestNumbersSort}
\SetKwProg{myalg}{}{}{}
\nonl\myalg{\algo{A, i}}{%
  Let $B$ be an integer array of size $i$\;
  \texttt{Heapsort}($A, 1, A.length$)\;
  \For{$j = 1$ \KwTo $i$}{%
    $B[j] = A[j]$\;
  }
  \Return{$B$}\;
}
\end{algorithm}

    This algorithm runs in $\Theta(n \lg n + i) = \Theta(n \lg n)$.

\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{LargestNumbersPriorityQueue}
\SetKwProg{myalg}{}{}{}
\nonl\myalg{\algo{A, i}}{%
  Let $B$ be an integer array of size $i$\;
  \texttt{Build-Max-Heap}($A$)\;
  \For{$j = 1$ \KwTo $i$}{%
    $\text{\emph{element}}$ = \texttt{Heap-Extract-Max}($A$)\;
    $B[i - j + 1] = element$\;
  }
  \Return{$B$}\;
}
\end{algorithm}

\textsc{Build-Max-Heap} call takes $O(n)$, \textsc{Extract-Max} call takes
$O(\lg n)$. This algorithm runs in $O(n + i \lg n)$.

\item The pseudocode is stated below.

\begin{algorithm}[H]
\SetAlgoNoEnd\DontPrintSemicolon
\BlankLine
\SetKwFunction{algo}{LargestNumbersOrderStatistic}
\SetKwProg{myalg}{}{}{}
\nonl\myalg{\algo{A, i}}{%
  Let $B$ be an integer array of size $i$\;
  $q = \texttt{Select}(A, i)$\;
  \texttt{Partition}($A, 1, A.length, q$)\;
  \texttt{Heapsort}($A, 1, i$)\;
  \For{$j = 1$ \KwTo $i$}{%
    $B[j] = A[j]$\;
  }
  \Return{$B$}\;
}
\end{algorithm}

\textsc{Select} call takes $O(n)$, \textsc{Partition} call takes $O(n)$,
\textsc{Quicksort} call takes $O(i \lg i)$. This algorithm runs in $O(n + i \lg i)$.
\end{enumerate}
\end{framed}

\newpage

\item[9-2]{\textbf{\emph{Weighted median}}\\
For $n$ distinct elements $x_1, x_2, \dots, x_n$ with positive weights
$w_1, w_2, \dots, w_n$ such that $\sum_{i = 1}^{n} w_i = 1$, the
\textbf{\emph{weighted (lower) median}} is the element $x_k$ satisfying
\[
  \sum_{x_i < x_k} w_i < \frac{1}{2},
\]
and
\[
  \sum_{x_i > x_k} w_i \le \frac{1}{2}.
\]

For example, if the elements are 0.1, 0.35, 0.05, 0.1, 0.15, 0.05, 0.2 and each
element equals its weight (that is, $w_i = x$ for $i = 1, 2, \dots, 7$), the
median is 0.1, but the weighted median is 0.2.

\begin{enumerate}
\item[\textbf{a.}] Argue that the median of $x_1, x_2, \dots, x_n$ is the
weighted median of the $x_i$ with weights $w_i = 1/n$ for $1, 2, \dots, n$.
\item[\textbf{b.}] Show how to compute the weighted median of $n$ elements in
$O(n \lg n)$ worst-case time using sorting.
\item[\textbf{c.}] Show how to compute the weighted median in $\Theta(n)$
worst-case time using a linear-time median algorithm such as \textsc{Select}
from Section 9.3.
\end{enumerate}

The \textbf{\emph{post-office location problem}} is defined as follows. We are
given $n$ points $p_1, p_2, \dots, p_n$ with associated weights
$w_1, w_2, \dots, w_n$. We wish to find a point $p$ (not necessarily one of the
input points) that minimizes the sum $\sum_{i = 1}^n w_i d(p, p_i)$, where
$d(a, b)$ is the distance between points $a$ and $b$.

\begin{enumerate}
\item[\textbf{d.}] Argue that the weighted median is a best solution for the
1-dimensional post-office location problem, in which points are simply real
numbers and the distance between points $a$ and $b$ is $d(a, b) = |a - b|$.

\item[\textbf{e.}] Find the best solution for the 2-dimensional post-office
location problem, in which the points are $(x, y)$ coordinate pairs and the
distance between points $a = (x_1, y_1)$ and $b = (x_2, y_2)$ is the
\textbf{\emph{Manhattan distance}} given by
$d(a, b) = |x_1 - x_2| + |y_1 - y_2|$.
\end{enumerate}
}

\begin{framed}
\begin{enumerate}
\item Note that there are at most $\floor{(n - 1)/2}$ elements that are smaller
than the median and at most $\ceil{(n - 1)/2}$ elements that are greater than
the median. Since the weight of each element is $1/n$, we have
\[
  \sum_{x_i < x_k} w_i = \sum_{x_i < x_k} \frac{1}{n}
    = \frac{1}{n} \sum_{x_i < x_k} 1
    \le \frac{1}{n} \cdot \Bigl\lfloor \frac{n - 1}{2} \Bigr\rfloor
    < \frac{1}{n} \cdot \frac{n}{2} = \frac{1}{2},
\]
and
\[
  \sum_{x_i < x_k} w_i = \sum_{x_i < x_k} \frac{1}{n}
  = \frac{1}{n} \sum_{x_i < x_k} 1
  \le \frac{1}{n} \cdot \Bigl\lceil \frac{n - 1}{2} \Bigr\rceil
  \le \frac{1}{n} \cdot \frac{n}{2} = \frac{1}{2},
\]
which implies that the median is also the weighted median.

\item Sort the array with \textsc{Heapsort}. Iterate over the elements of the
array, accumulating the sum of their weights until the sum achieves a value that
is greater than or equal to $1/2$. Let $x_k$ denote the last element that made
the sum accumulate a value greater than or equal to $1/2$. Note that at that
point
\[
  \sum_{x_i < x_k} w_i < \frac{1}{2}
\]
holds since the sum of the weights until the element right before $x_k$ is
smaller than $1/2$ and
\[
  \sum_{x_i > x_k} w_i \le \frac{1}{2}
\]
holds since the sum of the weights until $x_k$ is greater than or equal to $1/2$
and $\sum_{i = 1}^{n} w_i = 1$. Thus, $x_k$ is the weighted median. This
algorithm takes $\Theta(n \lg n)$ to sort the array with \textsc{Heapsort} and
$O(n)$ to accumulate the weights and find the weighted median.

\item Do the following steps:
\begin{enumerate}
\item Find the median with the \textsc{Select} algorithm.
\item Partition the array around the median.
\item Let $x_m$ denote the position of the median after partitioning. Let
$W_L = \sum_{X_i < X_m} w_i$ and $W_R = \sum_{x_i > x_m} w_i$.
\item If $W_L < 1/2$ and $W_R \le 1/2$, $x_m$ is the weighted median. Otherwise,
do the following:
\begin{enumerate}
\item If $W_L \ge 1/2$, the weighted median is before $x_m$.
Set $w_m = w_m + W_R$ and recurse on the left half of the array, including
$x_m$.
\item If $W_R > 1/2$, the weighted median is after $x_m$. Set $w_m = w_m + W_L$
and recurse on the right half of the array, including $x_m$.
\end{enumerate}
\end{enumerate}

This algorithm has the recurrence:
\[
  T(n) = T\left(\frac{n}{n} + 1\right) + \Theta(n)
       = \sum_{i = 0}^{\lg n} \left(\frac{n}{2^i} + 1\right)
       = n \sum_{i = 0}^{\lg n} \frac{1}{2^i} + \sum_{i = 0}^{\lg n} 1
       \le 2n + \lg n + 1
       = \Theta(n).
\]

\item Skipped.

\item Skipped.

\end{enumerate}
\end{framed}

\newpage

\item[9-3]{\textbf{\emph{Small order statistics}}\\
We showed that the worst-case number $T(n)$ of comparisons used by
\textsc{Select} to select the ith order statistic from $n$ numbers satisfies
$T(n) = \Theta(n)$, but the constant hidden by the $\Theta$-notation is rather
large. When $i$ is small relative to $n$, we can implement a different procedure
that uses \textsc{Select} as a subroutine but makes fewer comparisons in the
worst case.

\begin{enumerate}
\item[\textbf{a.}] Describe an algorithm that uses $U_i(n)$ comparisons to
find the $i$th smallest of $n$ elements, where
\[
U_i(n) =
\begin{cases}
  T(n) & \text{if } i \ge n/2,\\
  \floor{n/2} + U_i(\ceil{n/2}) + T(2i) & \text{otherwise}.
\end{cases}
\]

(\emph{Hint:} Begin with $\floor{n/2}$ disjoint pairwise comparisons, and
recurse on the set containing the smaller element from each pair.)

\item[\textbf{b.}] Show that, if $i < n/2$, then $U_i(n) = n + O(T(2i) \lg (n/i))$.

\item[\textbf{c.}] Show that if $i$ is a constant less than $n/2$, then $U_i(n) = n + O(\lg n)$.

\item[\textbf{d.}] Show that if $i = n/k$ for $k \ge 2$, then $U_i(n) = n + O(T(2n/k) \lg k)$.
\end{enumerate}
}

\begin{framed}
\begin{enumerate}
\item First, note that the \textsc{Select} algorithm find the $i$th element by
partitioning the array. That is, when the $i$th element is found, the first $i$
elements are the $i$ smallest. However, when $n$ is too large with respect to
$i$, it perform more comparisons than necessary. Taking the hint that the
question gave us, we can reduce the number of comparisons when $n$ is too large
by running \textsc{Select} only when $n$ is smaller than or equal to $2i$.

The key insight to solve the question is to observe that if we first make
disjoint pairwise comparisons and then run \textsc{Select} only among the
smallest element of each pair, the $i$th order statistic of the whole array is
among the $i$ smallest elements that were found by \textsc{Select} and their
large counterparts on the right half of the array. This occurs because the
remaining elements on the left half are larger than at least $i$ elements and
their larger counterparts on the right half are even larger.

We can then use this notion to build a recursive algorithm that solves the
selection problem with fewer comparisons, using the \textsc{Select} algorithm
only when $n$ is small enough:
\begin{enumerate}
  \item If $i \ge n/2$, run \textsc{Select} and return its result.
  \item Otherwise, do the following:
  \begin{enumerate}
    \item Perform disjoint pairwise comparisons and rearrange the array such
      that the smaller element of each pair appears on the left half of the
      array, in the same order of its larger counterparts.
    \item Recursively find the $i$th element among the elements on the left half
      of the array.
    \item The $i$th order statistic is among the first $i$ elements of the array
      and their larger counterparts. Run \textsc{Select} on these $2i$ elements
      and return the result.
  \end{enumerate}
\end{enumerate}

\item Can be proved by substitution.

\item From the previous item, we have
\[
  U_i(n) = n + O(T(2i) \lg(n/i)),
\]

which implies that, when $i$ is a constant less than $n/2$, we have
\begin{equation*}
\begin{aligned}
  U_i(n) &= n + O(T(2i) \lg(n/i))\\
         &= n + O(O(1) O(\lg n))\\
         &= n + O(\lg n).
\end{aligned}
\end{equation*}

\item If $k > 2$, then $i < n/2$ and we can use the result of item (b):
\begin{equation*}
\begin{aligned}
  U_i(n) &= n + O(T(2i)\lg(n/i))\\
         &= n = O(T(2n/k) \lg(n/(n/k)))\\
         &= n = O(T(2n/k) \lg(k)).
\end{aligned}
\end{equation*}

If $k = 2$, then $i = n/2$ and $\lg k = 1$. Thus, we have
\begin{equation*}
\begin{aligned}
  U_i(n) &=   T(n)\\
         &\le n + T(n) + \lg k\\
         &=   n + O(T(n) + \lg k)\\
         &=   n + O(T(2n/k) + \lg k).
\end{aligned}
\end{equation*}

\end{enumerate}
\end{framed}

\newpage

\item[9-4]{\textbf{\emph{Alternative analysis of randomized selection}}\\
In this problem, we use indicator random variables to analyze the
\textsc{Randomized-Select} procedure in a manner akin to our analysis of
\textsc{Randomized-Quicksort} in Section 7.4.2.

As in the quicksort analysis, we assume that all elements are distinct, and we
rename the elements of the input array $A$ as $z_1, z_2, \dots, z_n$, where
$z_i$ is the $i$th smallest element. Thus, the call
\textsc{Randomized-Select}($A, 1, n, k$) returns $z_k$.

For $1 \le i < j \le n$, let $X_{ijk}$ = I\{$z_i$ is compared with $z_j$
sometime during the execution of the algorithm to find $z_k$\}.

\begin{enumerate}
\item[\textbf{a.}] Give an exact expression for $\text{E}[X_{ijk}]$.
(\emph{Hint:} Your expression may have different values, depending on the values
of $i, j,$ and $k$.)

\item[\textbf{b.}] Let $X_k$ denote the total number of comparisons between
elements of array $A$ when finding $z_k$. Show that
\[
  \text{E}[X_k] \le 2 \left( \sum_{i = 1}^{k} \sum_{j = k}^{n} \frac{1}{j - i + 1} +
                             \sum_{j = k + 1}^{n} \frac{j - k - 1}{j - k + 1} +
                             \sum_{i = 1}^{k - 2} \frac{k - i - 1}{k - i + 1} \right).
\]

\item[\textbf{c.}] Show that $\text{E}[X_k] \le 4n$.

\item[\textbf{d.}] Conclude that, assuming all elements of array $A$ are
distinct, \textsc{Randomized-Select} runs in expected time $O(n)$.
\end{enumerate}
}

\begin{framed}
Skipped.
\end{framed}

\end{enumerate}


\subsection{Order statistics}

\subsubsection{Defining order statistics}

The \(k\)th order statistic is the \(k\)th smallest value in a sample.

\(x_{(1)}\) is the smallest value in a sample, the minimum.

\(x_{(n)}\) is the largest value in a sample, the maximum.

\subsubsection{Probability distributions of order statistics}

The probability distribution of order statistics depends on the underlying probability distribution.

\subsubsection{Probability distribution of sample maximum}

If we have:

\(Y=\max \mathbf X\)

The probability distribution is:

\(P(Y\le y)=P(X_1\le y, X_2\le y,...,X_n\le y)\)

If these are iid we have:

\(P(Y\le y)=\prod_i P(X_i\le y)\)

\(F_y(y)=F_X(y)^n\)

The density function is:

\(f_y(y)=nF_X(y)^{n-1}f_x(y)\)

\subsubsection{Probability distribution of the sample minimum}

If we have:

\(Y=\min \mathbf X\)

The probability distribution is:

\(P(Y\le y)=P(X_1\ge y, X_2\ge y,...,X_n\ge y)\)

If these are iid we have:

\(P(Y\le y)=\prod_i P(X_i\ge y)\)

\(F_y(y)=[1-F_X(y)]^n\)

The density function is:

\(f_y(y)=-n[1-F_X(y)]^{n-1}f_x(y)\)


\graphicspath{{img/algor/}}
\chapter{Códigos}

\section{Búsqueda de raíces}

\subsection{Detección de las raíces de una función en un intervalo dado}
\begin{minted}[mathescape,
           gobble=4,
           frame=lines,
           framesep=2mm]{python}
    from numpy import zeros

    def bracketing(fun, a, b, N):
        msg = "Maximum number of iterations reached."
        dx = (b - a)/(N - 1)
        iroot = 0
        x2 = a
        xR = zeros(N, float)
        for i in range(0, N):
            x1 = x2
            x2 = x1 + dx
            if (fun(x1) * fun(x2)) < 0:
                msg = "A change of sign was found."
                iroot = iroot + 1
                xR[i] = x1
        return xR, msg

    # Function call
    a = -10.0
    b = 10.0
    N = 21
    fun = lambda x: x**3 + 4*x**2 - 10
    xR, msg = bracketing(fun, a, b, N)
    print(msg)
    print(xR)
\end{minted}

\subsection{Método de bisección para la localización de raíces en un intervalo
dado}

\begin{minted}[mathescape,
gobble=4,
frame=lines,
framesep=2mm]{python}
def bisection(fun, a, b, xtol=1e-6, ftol=1e-12, verbose=False):
    """
    Use bisection method to estimate the root of a real function
    """
    if fun(a) * fun(b) > 0:
        c = None
        msg = "The function should change sign in the interval."
    else:
        nmax = int(ceil(log2((b - a)/xtol)))
        for cont in range(nmax):
            c = 0.5*(a + b)
            if verbose:
                print("n: {}, x: {}".format(cont, c))
            if abs(fun(c)) < ftol:
                msg = "Root found with desired accuracy."
                break
            elif fun(a) * fun(c) < 0:
                b = c
            elif fun(b) * fun(c) < 0:
                a = c
            msg = "Maximum number of iterations reached."
    return c, msg

x, msg = bisection(lambda x: x**3 + 4*x**2 -10, -2, 2, xtol=1e-4,
                    verbose=True)
print(msg)
print(x)
\end{minted}

\subsection{Método de Newton-Raphson para la localización de raíces en un
intervalo dado}

\begin{minted}[mathescape,
           gobble=4,
           frame=lines,
           framesep=2mm]{python}
def newton(fun, grad, x, niter=50, ftol=1e-8, verbose=False):
    """
    Use Newton method to estimate the root of a real function
    """
    msg = "Maximum number of iterations reached."
    for cont in range(niter):
        if abs(grad(x)) < ftol:
            x = None
            msg = "Derivative near to zero."
            break
        if verbose:
            print("n: {}, x: {}".format(cont, x))
        x = x - fun(x)/grad(x)
        if abs(fun(x)) < ftol:
            msg = "Root found with desired accuracy."
            break
    return x, msg

func = lambda x:x**3 + 4*x**2 - 10
deriv = lambda x: 3*x**2 + 8*x
result = newton(func, deriv, 2, verbose=True)
print(result)
\end{minted}

\section{Integración numérica}

\subsection{Regla del trapecio}

\begin{minted}[mathescape,
    gobble=4,
    frame=lines,
    framesep=2mm]{python}
    import numpy as np
    from sympy import symbols, integrate

    def trapz(fun, x0, x1, n):
        """Trapezoidal rule for integration

        Parameters
        ----------
        fun : callable
             Function to integrate.
        x0 : float
             Initial point for the integration interval.
        x1 : float
             End point for the integration interval.
        n : int
             Number of points to take in the interval.

        Returns
        -------
        inte : float
             Approximation of the integral

        """
        x = np.linspace(x0, x1, n)
        y = fun(x)
        dx = x[1] - x[0]
        inte = 0.5*dx*(y[0] + y[-1])
        for cont in range(1, n - 1):
        inte = inte + dx*y[cont]
        return inte

    fun = lambda x: x**3 + 4*x**2 - 10
    for cont in range(2, 11):
        numeric_int = trapz(fun, -1, 1, cont)
        print("Approximation for {} subdivisions: {:.6f}".format(cont - 1,
               numeric_int))

    x  = symbols('x')
    analytic_int = integrate(fun(x) , (x , -1 , 1))
    print("Analytic integral: {:.6f}".format(float(analytic_int)))
\end{minted}

\subsection{Cuadratura Gaussiana}

\begin{minted}[mathescape,
	gobble=4,
	frame=lines,
	framesep=2mm]{python}
    import numpy as np
    from scipy.special import roots_legendre
    from sympy import symbols, integrate

    def gauss1d(fun, x0, x1, n):
        """Gauss quadrature in 1D

        Parameters
        ----------
        fun : callable
            Function to integrate.
        x0 : float
            Initial point for the integration interval.
        x1 : float
            End point for the integration interval.
        n : int
            Number of points to take in the interval.

        Returns
        -------
        inte : float
             Approximation of the integral

        """
        xi, wi = roots_legendre(n)
        inte = 0
        h = 0.5 * (x1 - x0)
        xm = 0.5 * (x0 + x1)
        for cont in range(n):
            inte = inte + h * fun(h * xi[cont] + xm) * wi[cont]
        return inte

    fun = lambda x: x**3 + 4*x**2 - 10
    gauss_inte = gauss1d(fun, -1, 1, 4)
    x  = symbols('x')
    analytic_inte = integrate(fun(x) , (x , -1 , 1))
    print("Analytic integral: {:.6f}".format(float(analytic_inte)))
    print("Gauss quadrature: {:.6f}".format(gauss_inte))
\end{minted}

\subsection{Integración en 2 dimensiones}

\begin{minted}[mathescape,
    gobble=4,
    frame=lines,
    framesep=2mm]{python}
    import numpy as np
    from scipy.special import roots_legendre

    def gauss2d(fun, x0, x1, y0, y1, nx, ny):
        """Gauss quadrature for a rectangle in 2D

        Parameters
        ----------
        fun : callable
        Function to integrate.
        x0 : float
        Initial point for the integration interval in x.
        x1 : float
        End point for the integration interval in x.
        y0 : float
        Initial point for the integration interval in y.
        y1 : float
        End point for the integration interval in y.
        nx : int
        Number of points to take in the interval in x.
        ny : int
        Number of points to take in the interval in y.

        Returns
        -------
        inte : float
        Approximation of the integral

        """
        xi, wi = roots_legendre(nx)
        yj, wj = roots_legendre(ny)
        inte = 0
        hx = 0.5 * (x1 - x0)
        hy = 0.5 * (y1 - y0)
        xm = 0.5 * (x0 + x1)
        ym = 0.5 * (y0 + y1)
        for cont_x in range(nx):
        for cont_y in range(ny):
        f = fun(hx * xi[cont_x] + xm, hy * yj[cont_y] + ym)
        inte = inte + hx* hy * f * wi[cont_x] * wj[cont_y]
        return inte

    fun = lambda x, y: 3*x*y**2 - x**3
    gauss_inte = gauss2d(fun, 0, 2, 0, 2, 2, 2)
    print("Gauss quadrature: {:.6f}".format(gauss_inte))
\end{minted}


\subsection{Quantum harmonic oscillator}
For total energy $E$, kinetic energy $K$ and potential energy $V$ we have
$$E=K+V$$
The corresponding formula for a quantum harmonic oscillator is
$$(2n+1)\psi=-{d^2\psi\over dx^2}+x^2\psi$$
where $n$ is an integer and represents the quantization of energy values.
The solution to the above equation is
$$\psi_n(x)=\exp(-x^2/2)H_n(x)$$
where $H_n(x)$ is the $n$th Hermite polynomial in $x$.
The following Eigenmath code checks $E=K+V$ for $n=7$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
n = 7
psi = exp(-x^2/2)*hermite(x,n)
E = (2*n+1)*psi
K = -d(psi,x,x)
V = x^2*psi
E-K-V
\end{Verbatim}

$\displaystyle 0$


%% SECTION HEADER /////////////////////////////////////////////////////////////////////////////////////
\section{The Spectral Element Method}
\label{sec:sem}

%% SECTION CONTENT ////////////////////////////////////////////////////////////////////////////////////

The general concept of the \ac{sem} is based on the idea of the \ac{fem}.
The similarity of both methods lies in the fact that the modeled domain is divided into non-overlapping finite elements, and external forces and arbitrary boundary conditions are imposed in the particular nodes.
The main difference between those methods is a choice of the shape function \( N=N(\xi )\), which is interpolated by a Lagrange polynomial that passes through the element nodes. The nodes are localized on the endpoint of an interval, \(\xi\in[-1,1]\), and the roots of the first derivative of Legendre polynomial P of degree \(p-1\):
\begin{eqnarray}
	(1-\xi^2)P'_{p-1}(\xi)=0.
	\label{eq:nodes}
\end{eqnarray}

The approximation of an integral over the elements is achieved according to \ac{gll} rule at points coinciding with the element nodes,
and the weights \(w=w(\xi)\) calculated as:
\begin{eqnarray}
	{w(\xi)} = \frac{2}{p(p-1)(P_{p-1}(\xi))^2}.
	\label{eq:weights}
\end{eqnarray}

This approach guarantees a diagonal mass matrix.
The shape functions and the weights for \ac{2d} or \ac{3d} elements are obtained by the Kronecker product of vectors of individual axes, denoted by \(\otimes\) as follows:
\begin{eqnarray}
	N(\xi,\eta) = N(\xi)\otimes N(\eta), & N(\xi,\eta,\zeta) = N(\xi)\otimes N(\eta)\otimes N(\zeta), \nonumber\\
	w(\xi,\eta) = w(\xi)\otimes w(\eta), & w(\xi,\eta,\zeta) = w(\xi)\otimes w(\eta)\otimes w(\zeta).
	\label{eq:3Dshape_weights}
\end{eqnarray}

The elementary equations of motion is defined as:
\begin{eqnarray}
	\label{eq:motion}
	\textbf{M} \ddot{\textbf{d}} + \textbf{D} \dot{\textbf{d}} + \textbf{K} \textbf{d} = \textbf{F}_{ext}
\end{eqnarray}
where \textbf{d} is the displacement vector; \textbf{M}, \textbf{D}, \textbf{K} are structural mass, damping and stiffness matrices, respectively; \textbf{F}$_{ext}$ is the external forces vector; \((\dot{\ })=\frac{\partial}{\partial t}\). Construction of the \textbf{M}, \textbf{D}, \textbf{K} matrices is similar to the classical approach in \ac{fem}.

The convergence of the equation~(\ref{eq:motion}) in the \ac{sem} is already achieved for six nodes per wavelength, while at least fifteen nodes are needed in case of linear elements in classic \ac{fem}~\cite{wee2017simulating}. Moreover, the mass matrix is diagonal when the \ac{gll} approach is used.


\lab{K-Means Clustering}{K-Means Clustering}
\objective{Understand the basics of \emph{k-means} clustering, and apply to the problem of clustering earthquake epicenters.}

\subsection*{Clustering}
In Lab \ref{lab:pca}, we analyzed the iris dataset using PCA; we have reproduced the first two principal components of the iris data in Figure \ref{fig:iris_data}.
Upon inspection of the first two principal components, a human can easily see that there are two very distinct groups of irises.
Can we create an algorithm to identify these groups without human supervision?
This task is called \emph{clustering}, an instance of \emph{unsupervised learning}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{iris_pca.pdf}
\caption{The first two principal components of the iris dataset.}
\label{fig:iris_data}
\end{figure}

The objective of clustering is to find a partition of the data such that points in the same subset will be ``close'' according to some metric.
The metric used will likely depend on the data, but some obvious choices include Euclidean distance and angular distance.
Throughout this lab we will use the metric $d(x,y) = \|x-y\|_2$, the Euclidean distance between $x$ and $y$.

More formally, suppose we have a collection of $\mathbb{R}^K$-valued observations $X = \{x_1,x_2,\ldots,x_n\}$.
Let $N \in \mathbb{N}$ and let $\mathcal{S}$ be the set of all $N$-partitions of $X$, where an $N$-partition is a partition with exactly $N$ nonempty elements.
We can represent a typical partition in $\mathcal{S}$ as $S = \{S_1,S_2,\ldots,S_N\}$, where
\[
X = \bigcup_{i=1}^N S_i
\]
and
\[
|S_i| > 0, \qquad i=1,2,\ldots,N.
\]
We seek the $N$-partition $S^*$ that minimizes the within-cluster sum of squares, i.e.
\[
S^* = \underset{S\in\mathcal{S}}{\arg\min} \sum_{i=1}^N\sum_{x_j\in S_i}\|x_j-\mu_i\|_2^2,
\]
where $\mu_i$ is the mean of the elements in $S_i$, i.e.
\[
\mu_i = \frac{1}{|S_i|}\sum_{x_j\in S_i}x_j.
\]

\subsection*{The \emph{K-Means} Method}
Finding the global minimizing partition $S^*$ is generally intractable since the set of partitions can be very large indeed,
but the \emph{k-means} algorithm is a heuristic approach that can often provide good results.

We begin by specifying an initial cluster mean $\mu_i^{(1)}$ for each $i = 1, \cdots, N$ (this can be done by random initialization, or according to some heuristic).
For each iteration, we adopt the following procedure.
Given a current set of cluster means $\mu^{(t)}$, we find a partition $S^{(t)}$ of the observations such that
\begin{equation*}
S_{i}^{(t)} = \{x_j \; : \; \|x_j - \mu_{i}^{(t)}\|_2^2 \leq \|x_j - \mu_{l}^{(t)}\|_2^2,\,\,\,  l = 1, \cdots, N\}.
\end{equation*}
We then update our cluster means by computing for each $i = 1, \cdots, N$.
We continue to iterate in this manner until the partition ceases to change.

Examine Figure \ref{fig:iris_clusterings}, which shows two different clusterings of the iris data produced by the \emph{k-means} algorithm.
Note that the quality of the clustering can depend heavily on the initial cluster means.
We can use the within-cluster sum of squares as a measure of the quality of a clustering (a lower sum of squares is better).
Where possible, it is advisable to run the clustering algorithm several times, each with a different initialization of the means,
and keep the best clustering.
Note also that it is possible to have very slow convergence.
Thus, when implementing the algorithm, it is a good idea to terminate after some specified maximum number of iterations.

\begin{figure}[h]
	\centering
	\begin{tabular}{cc}
	\includegraphics[width=.49\textwidth]{iris_means_1.pdf} &
	\includegraphics[width=.49\textwidth]{iris_means_2.pdf}
	\end{tabular}
	\caption{Two different K-Means clusterings for the iris dataset.
            Notice that the clustering on the left predicts the flower species to a high degree of accuracy,
            while the clustering on the right is less effective.}
    \label{fig:iris_clusterings}
\end{figure}

\begin{problem}
Implement the \emph{k-means} algorithm using the following function declaration.

\begin{lstlisting}
def kmeans(data,n_clusters,init='random',max_iter=300):
    """
    Cluster a dataset using the k-means algorithm.

    Parameters
    ----------
    data : ndarray of shape (n,k)
        Each row is an observation.
    n_clusters : int
        The number of clusters.
    init : string or ndarray of shape (n_clusters,k)
        If init is the string 'random', then randomly initialize the cluster means.
        Else, the initial cluster means are given by the rows of init.
    max_iter : int
        The maximum allowable number of iterations.

    Returns
    -------
    means : ndarray of shape (n_cluster,k)
        The final cluster means, given as the rows.
    labels : ndarray of shape (n,)
        The i-th entry is an integer in [0,n_clusters-1] indicating
        which cluster the i-th row of data belongs to relative to
        the rows of means.
    measure : float
        The within-cluster sum of squares quality measure.
    """
    pass
\end{lstlisting}

Test your function on the first two principal components of the iris dataset.
Run it 10 times, using a different random initialization of the means each time.
Retain the clustering with the smallest within-cluster sum of squares.
Your clustering should be similar to the first clustering in Figure \ref{fig:iris_clusterings}.
\end{problem}

\subsection*{Detecting Active Earthquake Regions}
Suppose we are interested in learning about which regions are prone to experience frequent earthquake activity.
We could make a map of all earthquakes over a given period of time and examine it ourselves, but we recognize this as an unsupervised learning problem and are eager to apply our new k-means clustering tool.

Our data is contained in 6 text files, each with a months worth of earthquake data throughout the world, from January 2010 through June 2010.
These files contain a lot of information which we aren't interested in at the time; all we would like to extract from them is the location of each earthquake, which appears in characters $21$ through $33$ of each line.
Characters $21$ through $26$ contain the latitude of each epicenter, character $26$ denoting North or South, and characters $27$ through $33$ contain the longitude of each epicenter, character $33$ denoting East or West.
We need to divide each value by $1,000$ to represent these as degrees and decimals.
\begin{figure}
	\centering
	\includegraphics[width=\textwidth]{earthquakes.png}
	\caption{Earthquake epicenters over a 6 month period.}
	\label{fig:earthquakes}
\end{figure}
\begin{problem}
Load the earthquake data into a $n\times 2$ array, where each row gives the longitude and latitude of an earthquake in degrees.
Multiply South latitudes and West longitudes by $-1$.
Create a scatter plot of the resulting data. You should be able to see the outlines of some of the continents and tectonic plates (since these are often areas of greater seismic activity). See also Figure \ref{fig:earthquakes}.
\end{problem}

We want to cluster this data into active earthquake regions.
For this task, we might think that we can regard any epicenter as a point in $\mathbb{R}^{2}$ with coordinates being their latitude and longitude.
This, however, would be incorrect, because the earth is not flat. We must recognize that latitude and longitude are best viewed as a variation of spherical coordinates in $\mathbb{R}^{3}$, and we should interpret them as such.
Since our \emph{k-means} algorithm is based on Euclidean distance, we need to transform our data into 3-dimensional Euclidean coordinates.

A simple way to accomplish this transformation is to first transform the latitude and longitude values to spherical coordinates, and then to Euclidean coordinates.
Recall that a spherical coordinate in $\mathbb{R}^3$ is a triple $(r,\theta,\varphi)$, where $r$ is the distance from the origin, $\theta$ is the radial angle in the $xy$ plane from the $x$-axis,
and $\varphi$ is the angle from the $z$-axis. In our earthquake data, the longitude is already the appropriate $\theta$ value, and the $\varphi$ value (in degrees) is simply $90^\circ$ minus the latitude.
For simplicity, we can take $r=1$, since the earth is roughly a sphere.
We can then transform to Euclidean coordinates using the following relationships:
\begin{align*}
r & = \sqrt{x^{2} + y^{2} + z^{2}} & x & = r \sin \varphi \cos \theta \\
\varphi & = \arccos \frac{z}{r} & y & = r \sin \varphi \sin \theta \\
\theta & = \arctan \frac{y}{x} & z & = r \cos \varphi
\end{align*}

\begin{problem}
Transform your earthquake data into Euclidean coordinates.
Be sure to consider if and when you need to transform your data from degrees to radians!
\end{problem}

We are now ready to cluster the earthquake data using the Euclidean coordinates.
We need to address one further issue, however.
Notice that each earthquake data point has norm 1 in Euclidean coordinates, since it lies on the surface of a sphere of radius 1.
We also need to ensure that our cluster means have norm 1.
Otherwise, the means can't be interpreted as locations on the surface of the earth.
Furthermore, the \emph{k-means} algorithm will struggle to find good clusters.
A solution to this problem is to normalize the mean vectors at each iteration, so that they are always unit vectors.
Thus, we need to add optional functionality to our \li{kmeans} function.
\begin{figure}
	\centering
	\includegraphics[width=\textwidth]{earthquake_clusters.png}
	\caption{Earthquake epicenter clusters with $N = 15$.}
	\label{fig:earthquakeclusters}
\end{figure}

\begin{problem}
Add a keyword argument \li{normalize=False} to you \li{kmeans} function, and add code to normalize the means at each iteration, should this argument be set to \li{True}.
Use your function to cluster the earthquake data into 15 clusters. Run this 10 times, keeping the best clustering.
Transform the cluster means back to latitude and longitude coordinates (when calculating $\theta$ using the inverse tangent, use \li{numpy.arctan2} or \li{math.arctan2},
so that that correct quadrant is chosen).
Create a scatter plot showing each cluster mean, along with the earthquake epicenters color-coated according to their cluster. You plot should resemble that in Figure \ref{fig:earthquakeclusters}.
\end{problem}


\section{Matrices and Linear Maps}
\subsection{Definitions}
\begin{definition}
    A linear map is a function that preserves linear combination.
    That is, for vector spaces $V,W$ over the same field $k$, a linear map $T:V\to W$ satisfies
    $$\forall\lambda,\mu\in k, \underline{v},\underline{v'}\in V, T(\lambda\underline{v}+\mu\underline{v'})=\lambda T(\underline{v})+\mu T(\underline{v'})$$
\end{definition}
\begin{definition}
    The image of the entire vector space $V$ under $T$, $\operatorname{Im}T$, is the collection of all images, that is, $\{\underline{w}\in W: \exists \underline{v}\in V,T(\underline{v})=\underline{w}\}$.
    Also, the kernel $\ker T$ is the set $\{\underline{v}\in V: T(\underline{v}=\underline{0})\}$
\end{definition}
\begin{proposition}
    The kernel is a subspace of $V$ and the image a subspace of $W$.
\end{proposition}
\begin{definition}
    The dimension of the image is called the rank of $T$, $\operatorname{rank}T$, and the dimension of the kernel is the nullity of $T$, $\operatorname{null}T$.
\end{definition}
\begin{example}
    1. The zero linear map $T$ mapping each vector to the zero vector is a linear.
    Its rank is $0$ and the nullity of $T$ is the dimension of $V$.\\
    2. The identity map on $V$ is linear with kernel $\{0\}$ and the image $V$.\\
    3. Suppose $V=W=\mathbb R^3$ and the map $T$ given by
    $$
    T\underline{x}=
    \begin{pmatrix}
        3&1&5\\
        -1&0&-2\\
        2&1&3
    \end{pmatrix}
    \underline{x}
    $$
    is linear.
    Note that this matrix is singular, so its kernel is nontrivial since it contains at least one vector, say $(2,-1,-1)$.
    One can show that the nullity is $1$ (that is, it is entirely generated by this vector) and the rank is $2$.
\end{example}
\begin{definition}
    If $T,S:V\to W$ are both linear maps, then $\alpha T+\beta S$ for any $\alpha,\beta\in F$ is obviously also a linear map.
    We say it is the linear combination of the linear maps.
\end{definition}
\begin{definition}
    If $T:V\to W, S:U\to V$ are both linear, then easily $T\circ S:U\to W$ is linear and is called the composition of linear maps.
\end{definition}
The third example shown above triggers the following theorem:
\begin{theorem}[Rank-Nullity Theorem]
    Suppose $T:V\to W$ is linear, then $\operatorname{rank}T+\operatorname{null}T=\dim V$.
\end{theorem}
\begin{proof}
    Let $\underline{e_1},\ldots,\underline{e_k}$ be a basis for $\ker T$.
    We can extend it to a basis $\underline{e_1},\ldots,\underline{e_n}$ of $V$.
    Now $T(\underline{e_{k+1}}),\ldots,T(\underline{e_n})$ is a basis for $\operatorname{Im}T$, following which the theorem is proved.
    Indeed,
    $$T\left(\sum_{i=1}^na_i\underline{e_i}\right)=\sum_{i=1}^na_iT(\underline{e_i})=\sum_{i=k+1}^na_iT(\underline{e_i})$$
    so this set indeed spans $\operatorname{Im}T$.
    It is also linearly independent since
    $$\sum_{i=k+1}^na_iT(\underline{e_i})=0\implies T\left(\sum_{i=k+1}^na_i\underline{e_i}\right)=0\implies \sum_{i=k+1}^na_i\underline{e_i}=\sum_{i=1}^ka_i\underline{e_i}$$
    But that would imply that we have found a nontrivial relation between our $\underline{e_i}$ which are independent, which is false.
    So it is independent, hence the proof is done.
\end{proof}
\subsection{Matrices as Linear Maps in Real Vector Spaces}
Let $M$ be an $n\times n$ array with entries $M_{ij}$ where $i$ labels the rows and $j$ labels the columns.
For example, if $n=3$,
$$
M=
\begin{pmatrix}
    M_{11}&M_{12}&M_{13}\\
    M_{21}&M_{22}&M_{23}\\
    M_{31}&M_{32}&M_{33}
\end{pmatrix}
$$
We define the map $T:\mathbb R^n\to\mathbb R^n$ by $T(\underline{x})_i=M_{ij}x_j$.
It is obviously a linear map.
Note that if $\underline{x}=x_i\underline{e_i}$, then $T(\underline{x})=x_iT(\underline{e_i})=x_i\underline{c_i}$ where $\underline{c_i}$ is the column $i$ of the matrix.
Therefore the image of $T$ is the span of the columns of $M$.\\
Given that, it is useful to consider the rows $\underline{r_i}$ and $\underline{c_i}$ that are the rows and columns of $M$.
We can write $(\underline{R_i})_j=M_{ij}=(\underline{C_j})_i$.
So $(M\underline{x})_i=\underline{R_i}\cdot\underline{x}$, the kernel of $M$ is the set of all $\underline{x}$ that vanishes under the linear map.
\begin{example}
    1. $V=W=\mathbb R^n$, the zero map corresponds to the zero matrix.\\
    2. The identity mao corresponds to the identity matrix $I_{ij}=\delta_{ij}$.\\
    3. $T:\mathbb R^3\to\mathbb R^3$ corresponding to the matrix
    $$M=\begin{pmatrix}
        3&1&5\\
        -1&0&-2\\
        2&1&3
    \end{pmatrix}$$
    The image of it is the span of columns, that is, the $2$-dimensional subspace spanned by $\underline{C_1},\underline{C_2}$ (since $\underline{C_3}$ is in this space).
    The kernel then is the $1$-dimensional subspace spanned by the vector $(2,-1,-1)$.
\end{example}
\subsection{Geometrical Examples}
We first think about rotations $\mathbb R^2\to\mathbb R^2$ by an angle $\theta$ can be described as
$$
\begin{pmatrix}
    \cos\theta&-\sin\theta\\
    \sin\theta&\cos\theta
\end{pmatrix}
$$
which one can eariy derive from either polar coordinate or the its behaviour on basis vectors.\\
Things go more interesting when we get to dimension $3$.
We consider $\underline{x}=\underline{x}_\parallel+\underline{x}_\perp$ as the decomposition of $\underline{x}$ along the $\underline{n}$ direction, i.e. such that $\underline{x}_\parallel\parallel\underline{n}$ and $\underline{x}_\perp\perp\underline{n}$.
Then,
$$|\underline{x}_\parallel|=|\underline{x}|\cos\phi,|\underline{x}_\perp|=|\underline{x}|\sin\phi$$
So under the rotation along the axis $\underline{n}$, $\underline{x}_\parallel$ stays the same while the $\underline{x}_\perp$ changes.
Assuming the angle is $\theta$, we can reassemble things to obtain
$$\underline{x}\mapsto \underline{x}_\parallel+\cos\theta\underline{x}_\perp+\sin\theta\underline{n}\times\underline{x}$$
So in components
\begin{align*}
    (M\underline{x})_i=M_{ij}x_j&=x_i\cos\theta+(1-\cos\theta)n_jx_jn_i+\sin\theta\epsilon_{ijk}n_jx_k\\
    &=(\delta_{ij}\cos\theta+(1-\cos\theta)n_in_j+\sin\theta\epsilon_{ijk}n_k)x_j
\end{align*}
We can also have reflection across the plane with normal $\underline{n}$ which, in matrix form, would be $M_{ij}=\delta_{ij}-2n_in_j$.\\
Dilation and scaling are linear maps as well.
If we set the scaling factors to be $\alpha,\beta,\gamma$ along $\underline{e_i},\underline{e_2},\underline{e_3}$, then the corresponding matrix would be
$$
\begin{pmatrix}
    \alpha&0&0\\
    0&\beta&0\\
    0&0&\gamma
\end{pmatrix}
$$
There is another kind of linear transformation called shears.
Given unit vectors $\underline{a},\underline{b}$ perpendicular, then a shear with parameter $\lambda$ is defined by $\underline{x}\mapsto \underline{x}+\lambda\underline{a}(\underline{b}\cdot\underline{x})$.
So $\underline{a}\mapsto\underline{a}$ (in general $\underline{u}\perp\underline{b}\implies \underline{u}\mapsto\underline{u}$), $\underline{b}\mapsto\underline{b}+\lambda\underline{a}$.
In component, $T(\underline{x})_i=(\delta_{ij}+\lambda a_ib_j)x_j$.
\subsection{Matrices in General; Matrix Algebra}
\begin{definition}
    Consider a linear map $T:V\to W$ where $V,W$ are real or complex vector spaces of dimensions $n,m$ respectively.
    Assume that we have obtained a basis $\{\underline{e_i}\}$ for $V$ and a basis $\{\underline{f_a}\}$ for $W$.
    So a matrix representation of $T$ with respect to these bases is an array $M_{ai}$ with entries in $\mathbb R,\mathbb C$ as appropriate, with $a\in\{1,2,\ldots,m\}$ (`rows'), $i\in\{1,2,\ldots,n\}$ (`columns') with
    $$T(\underline{e_i})=\sum_{a}M_{ai}\underline{f_a}$$
    which extends to any vectors in $V$ by linearity.
\end{definition}
So we have $M\underline{x}=T(\underline{x})=M_{ai}x_i\underline{f_a}$ where the summation convention is being used.
The moral of the story is that by choice with basis we can identify the vector spaces as $\mathbb R^n,\mathbb R^m$ or $\mathbb C^n,\mathbb C^m$ and $T$ as a matrix.
\begin{definition}
    Suppose $T:V\to W, S:V\to W, R:W\to Z$, then given choices of bases on $V,W,Z$, and hence the matrices $M$ of $T$, $N$ of $S$, $L$ of $R$, then $(M+N)_{ij}=M_{ij}+N_{ij}$ and $LM$ the matrix of $R\circ T$, or (as one can check) equivalently $(LM)_{ij}=L_{ia}M_{aj}$.
\end{definition}
One can also observe that the products of two matrices consists of exactly the dot products of the rows of one and the columns of the other.
This follows immediately from definitions.
Note that the matrix multiplication cannot be defined in two arbitrary matrices.
To multiply two matrices $A,B$ to get $AB$, we must have $\operatorname{dom}A=\operatorname{cod}B$.
\begin{proposition}
    For matrices where matrix multiplication is defined, it is associative and distributive over matrix addition.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{definition}
    For $m\times n$ matrices $A$ and $n\times m$ matrices $B,C$, $B$ is a left inverse of $A$ if $BA=I$ and $C$ a right inverse of $A$ if $AC=I$ where $I$ is the identity.\\
    Note that not every matrices has inverse(s), but if it does, we say that it is invertible, or non-singular.
\end{definition}
\begin{proposition}
    If $n=m$, then if $A$ is invertible, then it has both an unique left inverse and an unique right inverse, and they are the same.
\end{proposition}
\begin{proof}
    Obvious enough.
\end{proof}
\begin{example}
    1. The rotation matrix $R(\theta)$ with respect to some (hyper-)axis has an inverse, since $R(\theta)\circ R(-\theta)=I$.\\
    2. Consider $n\times n$ matrix $M$.
    If $\underline{x'}=M\underline{x}$, then $M^{-1}\underline{x'}=\underline{x}$.
    This shows the uniqueness criterion of a system of linear equations.
\end{example}
One can check that a $2\times 2$ matrix $M$ is uniquely solvable if and only if its determinant $\det M=[M\underline{e_1},M\underline{e_2}]$ is nonzero.
One realize that the determinant is the area of the parallelogram spanned by the images of the bases.
\subsubsection{Transpose and Hermitian Conjugate}
If $M$ is an $m\times n$ matrix, then transpose, written $M^\top$ is the $n\times m$ defined by $(M^\top)_{ij}=M_{ji}$.
Note that for any two $n\times m$ matrices $M,N$ and scalars $\lambda,\mu$, then $(\lambda A+\mu B)^\top=\lambda A^\top+\mu B^\top$.
If $A$ is a $m\times n$ and $B$ is $n\times p$, then $(AB)^\top=B^\top A^\top$.
For square matrix $A$, we say $A$ is symmetric if $A^\top=A$, antisymmetric if $A^\top=-A$.\\
For complex matrices, the Hermitian conjugate of an $m\times n$ matrix $M$ is defined by $(M^\dagger)_{ij}=\overline{M_{ji}}$.
We can define Hermitian and anti-Hermitian matrices in the same way in square complex matrices.
\subsubsection{Trace}
Consider a complex $n\times n$ matrix $M$, the trace of $M$, $\operatorname{tr}(M)$, is $M_{ii}$ (where the summation convention is being used).
Immediately, $\operatorname{tr}(\alpha M+\beta N)=\alpha\operatorname{tr}(M)+\beta\operatorname{tr}(N)$, and $\operatorname{tr}(MN)=\operatorname{tr}(NM)$ and $\operatorname{tr}(M)=\operatorname{tr}(M^\top)$, and $\operatorname{tr}(I)=n$.
\begin{example}
    The reflection across the plane with normal $\underline{n}$ can be represented by the matrix $H$ defined by $H_{ij}=\delta_{ij}-2n_in_j$.
    Now $\operatorname{tr}(H)=H_{ii}=\delta_{ii}-2n_in_i=3-2=1$.
\end{example}
Note as well that antisymmetric matrices always have zero trace.
\subsubsection{Decomposition}
An $n\times n$ matrix $M$ can be written as $M=S+A$ where $S=(M+M^\top)/2,A=(M-M^\top)/2$.
Note that $S$ and $A$ are symmetric and antisymmetric respectively.
Consider the matrix $T$ defined by $T_{ij}=\delta_{ij}-\operatorname{tr}(S)\delta_{ij}/n$, so $T$ is traceless as well.
But $\operatorname{tr}S=\operatorname{tr}M$ since $A$ is antisymmetric hence traceless.
Thus $M_{ij}=T_{ij}+A_{ij}+\operatorname{tr}(M)\delta_{ij}/n$ decompose the matrix $M$ into symmetric traceless, antisymmetric and pure trace part.
\begin{example}
    For $n=3$, suppose $T_{ij}=0$, set $A_{ij}=\epsilon_{ijk}a_k$ and $\operatorname{tr}M=3\lambda$, so $M\underline{x}=\underline{x}\times\underline{a}+\lambda\underline{x}$.
\end{example}
\subsubsection{Orthogonal and Unitary Matrices}
A sqaure matrix $U$ is orthogonal if $UU^\top=U^\top U=I$.
So the column vectors of $U$ actually are orthonormal, same for the rows.
For example, rotation matrices are orthogonal.
\begin{proposition}
    $U$ is orthogonal if and only if it preserves inner products.
\end{proposition}
\begin{proof}
    For any square matrix $U$, we have
    $$(U\underline{x})\cdot(U\underline{y})=(U\underline{x})^\top U\underline{y}=\underline{x}^\top U^\top U\underline{y}$$
    So if $U$ is orthogonal, the last expression equals $\underline{x}\cdot\underline{y}$, hence $U$ preserves dot product.
    Conversely, for any square matrix $A$, $\underline{e_j}^\top A\underline{e_i}=A_{ji}$, so if $U$ preserves inner product, then by taking $A=U^\top U$ we have $U^\top U=I$
\end{proof}
Note that in $\mathbb R^n$, if $\{\underline{e_i}\}$ is an orthonormal basis, then $\{U\underline{e_i}\}$ would also be an orthonormal basis.
\begin{example}
    The general $2\times 2$ orthogonal matrices are all rotational or reflectional (from an axis through the origin) matrices.
    This can be trivially checked.
    In particular, reflections have determinant $-1$ while rotations always have determinant $1$.
\end{example}
\begin{definition}
    A complex $n\times n$ matrix $U$ is unitary if $U^\dagger U=UU^\dagger=I$.
    Equivalently, $U^\dagger=U^{-1}$.
\end{definition}
\begin{proposition}
    $U$ is unitary if and only if it preserves complex inner product.
\end{proposition}
\begin{proof}
    $(U\underline{x},U\underline{y})=\underline{x}^\dagger U^\dagger U\underline{y}$.
    Necessity is implied, and sufficiency is by $\underline{x}=\underline{e_i},\underline{y}=\underline{e_j}$.
\end{proof}


\section{Classification}

Solve $w^* = \arg\min\limits_w \hat R(w)$, \highlight{$y=\operatorname{sign}(w^\top x)$}

$\hat R(w) = \frac{1}{n} \sum_{i=1}^n \ell(w;x_i,y_i)$, \enskip $\nabla\!_w \hat R = \frac{1}{n} \sum_n \nabla\!_w \ell$%\enskip$\ell$: loss function

% ===
\emph{0/1 loss:}
$\normalcolor\to$ intractable

$\ell_{0/1} (w;x_i,y_i) = [y_i \neq \operatorname{sign}(w^Tx_i)] \color{gray}\in \{0,1\}$

% ===
\emph{Perceptron algorithm:}
$\normalcolor\to$ use $\normalcolor\ell_P$ and SGD

$\ell_P (w;x_i,y_i) = \max(0, -y_i w^\top x_i)$

%$\nabla\!_w \hat R(w) = \frac{1}{n}\sum_{i=1}^n \nabla\!_w \ell_P(w;x_i,y_i)$

$\nabla\!_w \ell_P(w;x_i,y_i) =
\begin{cases}
    0 &\text{if } y_i w^\top x_i \geq 0\\
    -y_i x_i &\text{otw. \color{gray}(incorrect)}
\end{cases}$

Data lin. separable $\Rightarrow$ obtains a lin. separator

% ===
\emph{Support Vector Machine (SVM):}
$\normalcolor\to$ \textbf{Hinge}

$\ell_H(w;x_i,y_i) = \max(0,{\color{red}1}-y_i w^T x_i)$

$\hat R(w) = \frac{1}{n}\sum_n \ell_H + \lambda\norm{w}_2^2$, \enskip $\nabla\!_w \hat R = \ldots + 2\lambda w$

%$\nabla\!_w \hat R(w) = \frac{1}{n}\sum_{i=1}^n \nabla\!_w \ell_P(w;x_i,y_i) + 2\lambda w$

$\nabla_w \ell_H(w;x_i,y_i) =
\begin{cases}
    0 &\text{if } y_iw^\top x_i \geq {\color{red}1}\\
    -y_i x_i &\text{otw.}
\end{cases}$

$w_{t+1} \leftarrow w_t(1-2\eta_t\lambda) + y_ix_i\eta_t\,[y_iw^\top x_i < 1]$

For \textbf{L1-SVM} (feature selection) use $\norm{w}_1$


\chapter{Key exchange protocol}\label{chap:exchange}
The private key system we developed in Chapter~\ref{chap:encryption} and \ref{chap:security} required the two parties to possess the same shared key. One of the major development in modern cryptography is the public key system that allows parties to securely exchange a shared key to use in a symmetric system. In this chapter, we build a Rubik's Cube key exchange protocol using a structure similar to Diffie-Hellman key exchange.

\section{Diffie-Hellman key exchange}
\par As we mentioned before, communication between two parties using private-key encryption requires that they first exchange keys by some secure channel. Diffie-Hellman key exchange, shorten as DHKE, is a method of securely exchanging cryptographic keys over a public channel, and our Rubik's Cube key exchange protocol builds upon it.
\begin{enumerate}
    \item Alice and Bob agree on a arbitrary finite Abelian group $G$ of order $n$ and an element $g \in G$. Both group $G$ and the element $g$ will be made public to everyone. (The group $G$ is written multiplicatively.)
    \item Alice picks a random natural number $a$, where $1 \leq a < n$, and sends $A = g^a$ to Bob. ($a$ remains secret to everyone besides Alice.)
    \item Bob picks a random natural number $b$, where $1 \leq b < n$, and sends $B = g^b$ to Alice. ($b$ remains secret to everyone besides Bob.)
    \item Alice computes $S_a = A^b$ and Bob computes $S_b = B^a$.
\end{enumerate}
We claim $S = S_a = S_b$ because $S_a = A^b = (g^a)^b = g^{ab} = (g^b)^a = B^a = S_b$. This ensures that Alice and Bob end up with the same shared value. In Table~\ref{tab:diffie-hellman-value}, we display each party's knowledge on various values assuming the existence of an eavesdropper.
\begin{table}[ht]
    \centering
    \begin{tabular}{|c|c|c|c|}
        \hline Value & Alice & Bob & Eavesdropper \\ \hline \hline
        G & \cmark & \cmark & \cmark \\ \hline
        g & \cmark & \cmark & \cmark \\ \hline
        a & \cmark & \xmark & \xmark \\ \hline
        b & \xmark & \cmark & \xmark \\ \hline
        A & \cmark & \cmark & \cmark \\ \hline
        B & \cmark & \cmark & \cmark \\ \hline
        S & \cmark & \cmark & \xmark \\ \hline
    \end{tabular}
    \caption{Diffie-Hellman secrecy table}
    \label{tab:diffie-hellman-value}
\end{table}
We see that the security of the DHKE protocol is based on that given $g, g^a, g^b$, finding $a$ or $b$ is difficult and thus finding the shared value $g^{ab}$ is hard as well. This problem is known as the \textit{discrete logarithm problem}. Though discrete logarithms are quickly computable in a few special cases, no efficient method is known for computing them in general.
\par We want to build a minimal example with the multiplicative group to walk through the details of DHKE. Suppose Alice and Bob agree to use a modulus $p = 11$ ($G = \mathbb{U}_{11}$), and a base $g = 2$ (which generates $G$).
\begin{enumerate}
    \item Alice picks a random natural number $a = 3$, and sends $A = g^a$ to Bob. \\
    $A = 2^3 \mod 11 = 8 \mod 11 = 8$
    \item Bob picks a random natural number $b = 7$, and sends $B = g^b$ to Alice. \\
    $B = 2^7 \mod 11 = 128 \mod 11 = 7$
    \item Alice computes $S = B^a = 7^3 \mod 11 = 343 \mod 11 = 2$
    \item Bob computes $S = A^b = 8^7 \mod 11 = 2097152 \mod 11 = 2$
    \item Alice and Bob now share a secret, the number 2.
\end{enumerate}
Of course, much larger values of $a$, $b$, and $p$ would be needed to make the DHKE secure. In real life applications, $p$ should have a length of 1024 bits or even longer and $g$ should have large prime order. The most common used group by the DHKE protocol is the multiplicative group of integers modulo $p$, with the elliptic curves being a significant variant.

\section{Rubik's Cube key exchange}
\par Notably, the original implementation of the DHKE requires the selected group $G$ to be Abelian but the Rubik's Cube group is not since the fundamental moves do not all commute with each other. By extracting the ideas presented in the paper\cite{exchange}, we can design a DHKE-like protocol that helps us securely exchange information between two parties but does not require the group $G$ to be Abelian.
\par Assume that we have two parties, Alice and Bob, who want to share the secret key. Let them agree on using the Rubik's cube group $G_3$. An element (one state) $g \in G_3$ is chosen and made public as well as an arbitrary automorphism $\phi$ of $G_3$. Alice chooses a private number $a \in \mathbb{N}$ and Bob chooses a private number $b \in \mathbb{N}$.
\begin{enumerate}
    \item Alice picks a natural number $a > 1$ and computes $A = \phi^{a - 1}(g) \cdots \phi^2(g) \cdot \phi(g) \cdot g$ and sends this value to Bob.
    \item Bob picks a natural number $b > 1$ and computes $B = \phi^{b - 1}(g) \cdots \phi^2(g) \cdot \phi(g) \cdot g$ and sends this value to Alice.
    \item Alice computes her key $K_A = \phi^a(B) \cdot A$.
    \item Bob computes his key $K_B = \phi^b(A) \cdot B$.
\end{enumerate}
We claim that Alice and Bob share the same secret key. Alice's shared key is:
\begin{align}
    K_A = \phi^a(B) \cdot A & = \phi^a(\phi^{b - 1}(g) \cdots \phi(g) \cdot g) \cdot ( \phi^{a - 1}(g) \cdots \phi(g) \cdot g) \\
    & = (\phi^a(\phi^{b - 1}(g)) \cdots \phi^{a}(\phi(g)) \cdot \phi^a(g)) \cdot ( \phi^{a - 1}(g) \cdots \phi(g) \cdot g) \\
    & = (\phi^{a + b - 1}(g) \cdots \phi^{a + 1}(g) \cdot \phi^a(g)) \cdot ( \phi^{a - 1}(g) \cdots \phi(g) \cdot g) \\
    & = \phi^{a + b - 1}(g) \cdots \phi(g) \cdot g
\end{align}
We could go from step $(\ref{chap:exchange}.1)$ to step $(\ref{chap:exchange}.2)$ because of the operation preserving feature of automorphism, which is $\phi(g_1g_2) = \phi(g_1)\phi(g_2)$. The shared key Bob holds can be calculated in a similar fashion:
\begin{align*}
    K_B = \phi^b(A) \cdot B & = \phi^b(\phi^{a - 1}(g) \cdots \phi(g) \cdot g) \cdot ( \phi^{b - 1}(g) \cdots \phi(g) \cdot g) \\
    & = (\phi^{b + a - 1}(g) \cdots \phi^{b + 1}(g) \cdot \phi^b(g)) \cdot ( \phi^{b - 1}(g) \cdots \phi(g) \cdot g) \\
    & = \phi^{b + a - 1}(g) \cdots \phi(g) \cdot g = K_A
\end{align*}
Notice that the Diffie-Hellman secrecy table displayed in Table~\ref{tab:diffie-hellman-value} still holds true for above settings. If we pick $\phi$ of $G_3$ as $\phi(g) = g$, we can discover that, different from the standard DHKE, correctness of the Rubik's Cube key exchange is based on the equality $g^a \cdot g^b = g^b \cdot g^a = g^{a+b}$.
\par Let us show an example of how to use Rubik's Cube key exchange protocol. The group we are using is $G_3$ and this information is known by public. We also pick a public automorphism $\phi$ to be a right conjugation by $R$. That is $\phi(g) = RgR^{-1}$. Finally we pick a public group element $g = U$ (the state we reach to by applying move $U$ to a solved cube.)
\begin{enumerate}
    \item Alice picks a random natural number $a = 4$, and sends $A$ to Bob.
    \begin{align*}
        A & = \phi^3(g) \cdot \phi^2(g) \cdot \phi(g) \cdot g \\
        & = R^3(g)R^{-3}R^2(g)R^{-2}R(g)R^{-1}g \\
        & = R^3(g)(R^{-1}g)(R^{-1}g)(R^{-1}g) \\
        & = R^3(g)(R^{-1}g)^3
    \end{align*}
    \item Bob picks a random natural number $b = 8$, and sends $B$ to Alice.
    \begin{align*}
        B & = \phi^7(g) \cdot \phi^6(g) \cdots \phi(g) \cdot g \\
        & = R^7(g)R^{-7}R^6(g)R^{-6} \cdots R(g)R^{-1}g \\
        & = R^7(g)(R^{-1}g)(R^{-1}g) \cdots (R^{-1}g) \\
        & = R^7(g)(R^{-1}g)^7
    \end{align*}
    \item Alice computes $K_A = \phi^4(B) \cdot A$ and finds the following result.
    \begin{align*}
        \phi^4(B) \cdot A & = \phi^4(R^7(g)(R^{-1}g)^7)\;R^3(g)\;(R^{-1}g)^3 \\
        & = R^4R^7(g)(R^{-1}g)^7R^{-4}R^3(g)(R^{-1}g)^3 \\
        & = R^{11}(g)(R^{-1}g)^7(R^{-1}g)(R^{-1}g)^3 \\
        & = R^{11}(g)(R^{-1}g)^{11}
    \end{align*}
    \item Bob computes $K_B = \phi^8(A) \cdot B$ and finds the following result.
    \begin{align*}
        \phi^8(A) \cdot B & = \phi^8(R^3(g)(R^{-1}g)^3)\;R^7(g)\;(R^{-1}g)^7 \\
        & = R^8R^3(g)(R^{-1}g)^3R^{-8}R^7(g)(R^{-1}g)^7 \\
        & = R^{11}(g)(R^{-1}g)^3(R^{-1}g)(R^{-1}g)^7 \\
        & = R^{11}(g)(R^{-1}g)^{11}
    \end{align*}
    \item Alice and Bob now share a secret series of fundamental moves of $C_3$.
    $$K_A = K_B = R^{11}(U)(R^{-1}U)^{11}$$
\end{enumerate}
Since $R^{-1}$ is the move that reverses $R$, we can simply rewrite it as $R'$. Finally the move Alice and Bob shares is $\underbrace{R...R}_{11}U\underbrace{R'U...R'U}_{11}$. To get more complex result in real applications, we could explore the group structure of $G_3 \cong (\mathbb{Z}_3^7 \times \mathbb{Z}_2^{11}) \rtimes (S_8 \times A_{12})$ to select more complicated $g$ and $\phi$ at the beginning.
\par Most modern key exchange protocols help communicating parties to establish a commonly shared number that is used for the key with a symmetric system like AES. We can add one more step in the key exchange protocol so that Alice and Bob can use their secretly shared moves to retrieve a number. Alice and Bob need to agree on a particular setting of $C_3$. We will refer this cube as \textit{number exchange cube}. They both fill cubie on up face, down face, right face by ones and fill cubies on front face, back face, left face by zeros as shown in Table~\ref{tab:exchange-start-bit}. After they retrieve the shared fundamental moves $K_A = K_B$, they apply that series of moves to the number exchange cube and then read out the bits as the binary number. Notice that here each cubie only holds one bit and we do not want the center cubies to hold any information since they are fixed. We also do not want to use all bits from the cube because then the enemy knows that we are generating a binary of length 48 where half bits are ones and half bits are zeros. We instead only take bits from three faces, the up face, the front face and the right face. So for an arbitrary large cube $C_n$ we can use it to share a binary value with length of $3 \cdot (n^2 - (n \mod 2))$ bits.
\par To illustrate a detailed example of above procedure, let us first take a look at Table~\ref{tab:exchange-start-bit}, which describes the starting position of the number exchange cube $C_3$ that Alice and Bob uses to retrieve the shared number.
\begin{table}[ht]
    \centering
    \begin{tabular}{|c|c|}
        \hline Up face & 11111111  \\
        \hline Front face & 00000000  \\
        \hline Right face & 11111111  \\
        \hline Down face & 11111111  \\
        \hline Back face & 00000000  \\
        \hline Left face & 00000000  \\ \hline
    \end{tabular}
    \caption{Starting position of the number exchange cube $C_3$}
    \label{tab:exchange-start-bit}
\end{table}
Both Alice and Bob apply the shared moves $\underbrace{R...R}_{11}U\underbrace{R'U...R'U}_{11}$ to the number exchange cube, and the bits on each face is displayed in Table~\ref{tab:exchange-final-bit}.
\begin{table}[ht]
    \centering
    \begin{tabular}{|c|c|}
        \hline Up face & \textcolor{blue}{00011010}  \\
        \hline Front face & \textcolor{blue}{00100001}  \\
        \hline Right face & \textcolor{blue}{11110001}  \\
        \hline Down face & 11110100  \\
        \hline Back face & 01100000  \\
        \hline Left face & 11111110  \\ \hline
    \end{tabular}
    \caption{Final position of the number exchange cube $C_3$}
    \label{tab:exchange-final-bit}
\end{table}
Remember that we only use bits from three faces, and those bits we want are colored in blue. Both Alice and Bob read them out in order of: up face $\rightarrow$ front face $\rightarrow$ right face to retrieve the shared binary number $000110100010000111110001_2$.
\par In practice, symmetric keys are usually much longer. For example, the software FireVault on Mac OS uses AES with a 256-bit key to encrypt user's hard drive. We could use $C_{10}$ to exchange the key since $C_{10}$ are capable of holding $3 \cdot 10^2 = 300$ bits.
\par Even though we do not directly use numbers as the key to our Rubik's Cube encryption $\Pi_{RC}$, given a binary number we can still convert it to a series of fundamental moves. Suppose we have a set $S$ which contains all base 18 numbers with only one digit. Thus $S = \{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f, g, h\}$. In Table~\ref{tab:map-move} find a one-to-one mapping between elements of $S$ and the 18 fundamental moves.
\begin{table}[ht]
    \centering
    \begin{tabular}{|c|c|c|c|c|c|}
        \hline 0 - $U$ & 1 - $F$ & 2 - $R$ & 3 - $D$ & 4 - $B$ & 5 - $L$ \\
        \hline 6 - $U2$ & 7 - $F2$ & 8 - $R2$ & 9 - $D2$ & a - $B2$ & b - $L2$ \\
        \hline c - $U'$ & d - $F'$ & e - $R'$ & f - $D'$ & g - $B'$ & h - $L'$ \\ \hline
    \end{tabular}
    \caption{Mapping between set $S$ and fundamental moves}
    \label{tab:map-move}
\end{table}
Alice and Bob can convert the shared binary number to a base 18 number, $g5bff_{18}$. Then their commonly shared key will be ``$B'$ $L$ $R2$ $D'$ $D'$ ''. Notice that the biggest possible binary we can share using $C_3$ is $2^{24} - 1$. By converting that number to base 18, we get number $8fed99$. Since the maximum value on the first digits is 8, if we always use all 6 digits, the first movement will always be restricted. We thus simply ignore the first digit and use last 5 digits to represent fundamental moves when the number happen to have 6 digits. When the number has less than 5 digits, we treat leading digits as zeros. In Table~\ref{tab:share-length}, we show the length of the key different $C_n$ can share.
\begin{table}[ht]
    \centering
    \begin{tabular}{|c|c|c|c|c|}
        \hline $C_3 - 5$ & $C_4 - 11$ & $C_5 - 17$ & $C_6 - 25$ & $C_7 - 34$ \\
        \hline $C_8 - 46$ & $C_9 - 57$ & $C_{10} - 71$ & $C_{15} - 161$ & $C_{19} - 258$ \\ \hline
    \end{tabular}
    \caption{Maximum length of the $\Pi_{RC}$ key $C_n$ can share}
    \label{tab:share-length}
\end{table}
\par In Chapter~\ref{chap:security}, we suggested to use a key with length 30. Hence choosing $C_7$ here might be most appropriate option. We now have completed the setting of the key-exchange encryption protocol and shown how to apply this to the actual Rubik's Cube encryption $\Pi_{RC}$.

\section{Complete example}
\par Let us put everything together and demonstrate a minimal example on how Alice and Bob can securely communicate with each other by using the Rubik's Cube encryption and the Rubik's Cube key exchange protocol on $C_3$. Suppose Alice now wants to send out ``Hello Bob!''
\par Before the secure channel is built, Alice and Bob first exchange the shared key. They pick the public element $g = U\,R\,B\,D\,L'$ and let the public automorphism $\phi \in Aut(G_3)$ to be an inner-automorphism $\phi(g) = U2\,L2\,D'\,g\,D\,L2\,U2$, and agree to use the bit shift as the permutation $p$ during encryption.
\begin{itemize}
    \item Alice picks a random natural number $a = 23$, and sends $A$ to Bob.
    \begin{align*}
        A & = \phi^{22}(g) \cdots \phi(g) \cdot g \\
        & = (U2\,L2\,D')^{22}U\,R\,B\,D\,L'(D\,L2\,U2\,U\,R\,B\,D\,L')^{22}
    \end{align*}
    \item Bob picks a random natural number $b = 37$, and sends $B$ to Alice.
    \begin{align*}
        B & = \phi^{36}(g) \cdots \phi(g) \cdot g \\
        & = (U2\,L2\,D')^{36}U\,R\,B\,D\,L'(D\,L2\,U2\,U\,R\,B\,D\,L')^{36}
    \end{align*}
    \item Alice computes $K_A = \phi^{23}(B) \cdot A$ and finds the following result.
    $$\phi^4(B) \cdot A = (U2\,L2\,D')^{59}U\,R\,B\,D\,L'(D\,L2\,U2\,U\,R\,B\,D\,L')^{59}$$
    \item Bob computes $K_B = \phi^{37}(A) \cdot B$ and finds the following result.
    $$\phi^8(A) \cdot B = (U2\,L2\,D')^{59}U\,R\,B\,D\,L'(D\,L2\,U2\,U\,R\,B\,D\,L')^{59}$$
    \item Alice and Bob now share a secret series of fundamental moves of $C_3$.
    $$K = K_A = K_B = (U2\,L2\,D')^{59}U\,R\,B\,D\,L'(D\,L2\,U2\,U\,R\,B\,D\,L')^{59}$$
\end{itemize}
\par Both Alice and Bob apply the shared key $K$ to the number exchange cube $C_3$.
\begin{table}[ht]
    \centering
    \begin{minipage}{0.45\textwidth}
        \centering
        \begin{tabular}{|c|c|}
            \hline Up face & 11111111  \\
            \hline Front face & 00000000  \\
            \hline Right face & 11111111  \\
            \hline Down face & 11111111  \\
            \hline Back face & 00000000  \\
            \hline Left face & 00000000  \\ \hline
        \end{tabular}
        \caption{Starting position of the \\\hspace{\textwidth} number exchange cube $C_3$}
        \label{tab:example-exchange-start-bit}
    \end{minipage}
    \begin{minipage}{0.45\textwidth}
        \centering
        \begin{tabular}{|c|c|}
            \hline Up face & \textcolor{blue}{11101101}  \\
            \hline Front face & \textcolor{blue}{01000010}  \\
            \hline Right face & \textcolor{blue}{10011011}  \\
            \hline Down face & 00110000  \\
            \hline Back face & 00010110  \\
            \hline Left face & 10110111  \\ \hline
        \end{tabular}
        \caption{Final position of the \\\hspace{\textwidth} number exchange cube $C_3$}
        \label{tab:example-exchange-final-bit}
    \end{minipage}
\end{table}
The initial bits and final bits on faces of $C_3$ are displayed in Table~\ref{tab:example-exchange-start-bit} and Table~\ref{tab:example-exchange-final-bit} separately. The binary number Alice and Bob both retrieve is $111011010100001010011011_2$. Then, they convert this binary value to a base 18 number, $8422hh_{18}$. Alice and Bob will omit the first digit since they agree to use only the last 5 digits. By looking at the mapping described in Table~\ref{tab:map-move}, they find the shared key is ``$B\,R\,R\,L'\,L'$.'' Now Alice can convert her message to its binary representation.
\begin{center}
    0100100001100101011011000110110001101111 \\
    0010000001000010011011110110001000100001
\end{center}
Alice no longer has to remove punctuations or capitalize all letters because the message will get changed during the encryption. Since this message contains merely 80 bits, and $C_3$ needs an input of 180 bits to encrypt. Alice needs to pad it first. The padded string is shown as below:
\begin{center}
    010010000110010101101100011011000110111100100000010000100110
    111101100010001000011000000000000000000000000000000000000000
    000000000000000000000000000000000000000000000000000000000000
\end{center}
After inputting the string, 36 random bits will be generated by the encryption protocol ``111100001101010110101000010000011110,'' and those bits will be added to the end of the padded string. During the encryption process, before applying each fundamental move in the key, the encryption protocol first does a XOR between faces, and then shifts bits to the right by one space. After all moves are executed, the ciphertext is:
\newpage
\begin{center}
    000111011001000100101100011101101110001010000000000100100100101010100001
    010001000111011010000000100000010001101111111011000010000100110010011101
    010101110000101000100101010000100110001110000011010001101011000110101010
\end{center}
Alice can convert this binary value back to its ASCII representation, which looks like ``\;\'{E},v$\Upgamma$\c{C}\;J\'{\i}Dv\c{C}\"{u}\;$\surd{}$\;L\textyen{}W\;\%Bc\^{a}F|$\neg$\,,'' and sends it to Bob.
\par To decrypt Alice's message, Bob has to first find the inverse of the key ``$B\,R\,R\,L'\,L'$'', which will be ``$L$\,$L$\,$R'$\,$R'$\,$B'$.'' Then he converts the received message back to its binary representation to retrieve the 216 bits and fills an empty $C_3$ with the bits. For each fundamental move in ``$B\,R\,R\,L'\,L'$'', Bob first applies the move to the cube, then shift bits back to the left by one space and finally performs the XOR operation between the left face and the rest of faces.
\par After all moves in the inverse of the key are applied to the cube, Bob will get the padded string. He removes all tailing zeros and one extra 1 at the end to get the plaintext in its binary representation.
\begin{center}
    0100100001100101011011000110110001101111 \\
    0010000001000010011011110110001000100001
\end{center}
By looking up the ASCII table, he gets Alice's message ``Hello Bob!'' To reply Alice's message, he can follow the same encryption process used by Alice, and he will keep using the shared key without exchanging new ones.
\par The example above illustrates how everything we developed ties together and serves as a complete encryption system that helps two parties to communicate with some protection.