\chapter{Elliptic Curve Commitments}
\label{chpr:ec-commitments}
Elliptic curve cryptography is used in Bitcoin and in similar systems to secure the transactions.
We will give a brief overview of this cryptosystem, then we show how an elliptic curve point can be a commitment, finally we describe the consequent practical timestamping applications with Bitcoin.

\section{Elliptic Curve Public Key Cryptosystem}
We start with a general definition taken from \cite{Koblitz1987}\footnote{See Appendix \ref{app:A} for a more basic approach.},
\begin{mydef}
	An elliptic curve $E_K$ defined over a field $K$ of characteristic $\neq 2, 3$ is the set of solutions $(x,y)\in K^2$ to the equation
	\begin{equation}
	\label{ec-eq}
	y^2 = x^3 + ax + b, \quad a,b \in K
	\end{equation}
	together with a \textquotedblleft point at infinity\textquotedblright $\mathcal{O}$.
\end{mydef}
$\mathcal{O}$ is the projective closure of (\ref{ec-eq}) and may not be described in terms of two coordinates in $K$.
The points on $E_K$ form a group with identity element the point at infinity. The negative point $P \in E_K$ is the second point on $E_K$ having the same $x$-coordinate as $P$. Let $P_1=(x_1,y_1)$ and $P_2=(x_2,y_2)$ be two points on the curve, their sum $P_3=(x_3,y_3) = P_1 + P_2$ is given by:
\begin{equation}
x_3 = -x_1 -x_2 + \alpha^2, \quad
y_3 = -y_1 + \alpha(x_1 - x_3),
\end{equation}
where
\begin{equation}
\alpha = \begin{cases}
			(y_2 - y_1)/(x_2 - x_1) & \textrm{if } P_1 \neq P_2, \\
			(3x_1^2 + a)/(2y_1) & \textrm{if } P_1 = P_2.
\end{cases}
\end{equation}
This \textit{addition} operation for elliptic curve points has a geometric interpretation for $K = \mathbb{R}$, from which the above more general algebraic formulae can be derived.

Using these formulae, one can compute a multiple $mP$ of a given point $P$ in polynomial time by means of $O(\log m)$ doubling and additions, e.g. $11P = P + 2(P + 2(2P))$. This operation is called \textit{scalar multiplication}.

In cryptography most applications use finite fields, in particular a finite field contains $p^m$ elements with $p$ prime and $m \geq 1$. We will confine ourselves to the case $m=1$. So let $K=GF(p)=\mathbb{F}_p =\mathbb{Z}_p  = \{0, 1, ..., p-1\}$, $K$ is a finite field, the points of $E_K$, together with the addition operation defined above, form a finite Abelian group. The elliptic curve becomes:
%cyclic or product of two cyclic groups
\begin{equation}
E_{\mathbb{F}_p}=\{(x,y)\in \mathbb{F}_p^2 \quad \textmd{s.t. }
y^2 = x^3 + ax + b \mod p, \quad a,b \in \mathbb{F}_p \} \cup \mathcal{O}
\end{equation}
Let $G \in E_{\mathbb{F}_p}$ be a conventional element of order $n$, called $generator$. The subgroup generated by $G$ is:
\begin{equation}
	\langle G \rangle = \{xG|x \in \mathbb{Z}_n\} \subseteq E_{\mathbb{F}_p}
\end{equation}
Which is a cyclic group isomorphic to $\mathbb{Z}_n$; in particular, if $n$ is prime, then $\langle G \rangle = E_{\mathbb{F}_p}$.

Computing the isomorphism from $\mathbb{Z}_n$ to $\langle G \rangle$ is efficient, it takes $O(\log n)$ group operations; while the opposite isomorphism is much harder to compute, at moment, the best algorithm known takes approximately $\sqrt{n}$ operations. The latter procedure is called discrete logarithm and it stands at the base of the cryptosystem, more precisely:
\begin{mydef}
	Elliptic Curve Discrete Logarithm Problem (ECDLP). Given an elliptic curve $E$ defined over $GF(q)$ and two points $P, Q \in E$, find an integer $x$ such that $Q = xP$ if such $x$ exists.
\end{mydef}
Elliptic curve cryptography is based on the premise that ECDLP is hard, actually it appears to be more intractable than DLP in finite fields.
The ECDLP difficulty enables a Diffie-Hellmann key exchange which precedes the ElGamal signature scheme, these techniques are at the foundations of the public key cryptosystem.
Fixed a point $P \in E_{\mathbb{F}_p}$, a \textit{public key} is a point $Q \in E_{\mathbb{F}_p}$ while its discrete logarithm $x$ w.r.t. $P$ ($xP=Q$) is the \textit{private key}.
Given $x$ is easy and fast to compute $Q$, while given $Q$ is infeasible to find $x$.

To classify different curves the Standard for Efficient Cryptography (SEC) proposed a set of parameter for elliptic curves over $\mathbb{F}_p$:
\begin{equation}
(p, a, b, G, n, h)
\end{equation}
\begin{itemize}
	\item $p$ prime defines the finite field $\mathbb{F}_p$
	\item $a, b \in \mathbb{F}_p$ define the curve $E_{\mathbb{F}_p}$
	\item $G\in E_{\mathbb{F}_p}\backslash \{\mathcal{O}\}$ is a generator of the group
	\item $n = |\langle G \rangle|$ is the order of the group (smallest $n>0$ s.t. $nG = \mathcal{O}$)
	\item $h = |E_{\mathbb{F}_p}| / n$ is the cofactor
\end{itemize}
Note that if $n$ prime, then $\langle G \rangle = E_{\mathbb{F}_p}$, thus $n = |E_{\mathbb{F}_p}|$, $h=1$.
Bitcoin uses the curve named $secp256k1$ with parameters
\begin{verbatim}
p = 0x FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
       FFFFFFFE FFFFFC2F
a = 0
b = 7
G = (0x 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9
        59F2815B 16F81798,
     0x 483ADA77 26A3C465 5DA4FBFC 0E1108A8 FD17B448 A6855419
        9C47D08F FB10D4B8)
n = 0x FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFE BAAEDCE6 AF48A03B
       BFD25E8C D0364141
h = 1
\end{verbatim}
With this curve some calculation becomes more efficient, like computing the modular square root ($\sqrt{x}=x^{\lfloor\frac{p+1}{4}\rfloor}\textmd{ mod }p$, since $p= 3 \textmd{ mod }4$) and the modular inverse ($x^{-1}=x^{p-2}\textmd{ mod }p$, since $p$ prime).

Looking at the generator above one may think that to store an elliptic curve point is necessary to use 32 bytes for the $x$-coordinate and 32 bytes for the $y$-coordinate.
However it is not necessary to use all that space, infact it is possible to take advantage of the elliptic curve equation. Suppose $x \in \mathbb{F}_p$ is the $x$-coordinate of a point, then $y$ is given by $y^2 = x^3 + ax + b \quad \text{mod }p$, which, for $p$ prime, has exactly two solutions in $\mathbb{F}_p$, $y$ and $p-y$, one is odd and the other is even.
The solutions are easily computable thanks to the above formula.
Having consider this, to store an elliptic curve point $P$ one can store $P_x$ and the parity of $P_y$.
In Bitcoin the compressed encoding of a point $P$ is given by a byte for the parity of $P_y$ ($02$ if even, $03$ if odd) followed by the bytes representing $P_x$, for instance the generator $G$ is encoded as follows:
\begin{verbatim}
G = 02 79BE667E F9DCBBAC 55A06295 CE870B07 029BFCDB 2DCE28D9
       59F2815B 16F81798
\end{verbatim}

\section{A New Commitment Operation}

Combining conveniently elliptic curve points and hash functions, it is possible to create new commitment operations.
A similar technique was used for the first time for deriving public keys in deterministic wallets \cite{Max11},
then the concept of embedding a value in an elliptic curve point was exploited \cite{journals/corr/abs-1212-3257, Sidechain},
later it was reformulated in a more refined way \cite{PoePR, PoeIs}
suitable for the integration in OpenTimestamps.
Let's analyse how this new commitment operations are structured.

\begin{myprop}
	Let $E_{\mathbb{F}_p}$ be an elliptic curve defined on the finite field $\mathbb{F}_p$ with $p$ prime, $G \in E_{F_p}$ be the generator of the curve with order $n$ large prime. Let $h$ be a random oracle hash function, its input are somehow serialized in bits, its output are in $\{0,1\}^k$ and are interpreted as integers.
	Let $C$ be defined as follows:
	\begin{equation}
	\begin{split}
	C : \{0,1\}^* \times E_{\mathbb{F}_p}  & \rightarrow E_{\mathbb{F}_p} \\
	    m, P & \mapsto h(P||m)G + P
	\end{split}
	\end{equation}
	If $n$ is close to $2^k$, then $C$ is a commitment operation.
\end{myprop}
\begin{proof}
	First we show a direct proof that works when $P$ (or $P'$) is fixed, then we show the general proof. Actually, in the former case, it is enough requiring $h$ to be second-preimage resistant, while, in the latter, a stronger assumption is needed.
	\\
	Let $m,P$ and $C(m,P)$ be fixed.
	%Let $C(m,P)$ be a commitment to $m, P$.
	$\forall P' \in E_{\mathbb{F}_p}$ fixed in advance, the problem is to find $m' \in \{0,1\}^*$ s.t. $C(m,P)=C(m',P')$ and $(m', P') \neq (m, P)$.
	We want to show that such problem is infeasible.
	$m'$ satisfies $h(m'||P')G + P'=C(m,P)$.
	%The substitute value $m'$ needs to satisfy $h(m'||P')G + P'=C(m,P)$.
	Let $x$ be such that $xG = C(m,P) - P'$, with $0\leq x<n$.
	%so $x=h(P'||m')\textmd{ mod }n$. The problem to find a second preimage $m'$ of $x$.
	Let $h_{P'}$ be defined as follows:
	\begin{equation*}
	\begin{split}
	h_{P'}:\{0,1\}^* & \rightarrow \{0,1\}^k\\
	m' & \mapsto h(m'||P')
	\end{split}
	\end{equation*}
	Consider the elements in $\{0,1\}^{k}$ equivalent to $x$ modulo $n$, namely $I_x=\{h \in \{0,1\}^k \approx \mathbb{Z}_{2^k} | h=x\textmd{ mod }n\}$.
	%Let $I_x=\{h \in \{0,1\}^k \approx \mathbb{Z}_{2^k} | h=x\textmd{ mod }n\}$ be the set of images of $h_{P'}$ suitable to find a second-preimage.
	Note that:
	\begin{equation*}
		|I_x|=\begin{cases}
				\lceil\frac{2^k}{n}\rceil & \textmd{if } x\geq n \\
				\lfloor\frac{2^k}{n}\rfloor & \textmd{if } x< n
		      \end{cases}
	\end{equation*}
	The problem is finding $h_{p'}(m')\in I_x$.
	Note that $h_{p'}$ is second-preimage resistant, since it is the composition of two functions with that property, $prepend(P')$ and $h$.
	Finally, if $n \approx 2^k$, the elements in $I_x$ are few (eventually a single one), thus finding $m'$ is infeasible because $h_{P'}$ is second-preimage resistant.
	%$I_x$ contains  $\lceil\frac{2^k}{n}\rceil$ or $\lfloor\frac{2^k}{n}\rfloor$ elements.
	%Hence to find a second-preimage one needs to find $m'$ s.t. $h_{P'}(m') \in I_x$. Note that $h_{P'}$ is second-preimage resistant, since it is the composition of two function with that property, $prepend(P')$ and $h$. Finally, if $n \approx 2^k$, the elements in $I_x$ are few (eventually a single one), thus finding $m'$ is unfeasible because $h_{P'}$ is second-preimage resistant.
	\\
	Now consider the case in which $P$ is not fixed.
	We model the hash function $h$ as a random oracle $H$ with range $\{0,1\}^k\approx \mathbb{Z}_{2^k}$.
	$H$ initially has an empty table.
	When someone queries the oracle for a value $x \in \{0,1\}^*$,
	the oracle samples uniformly a random value $H(x)$ in $\mathbb{Z}_{2^k}$ and associate it with $x$.
	The oracle outputs $H(x)$ and annotate it on his table.
	If someone calls again the oracle for $x$ he will answer with $H(x)$.
	The oracle is queried by different entities for possibly different values.
	Queries for new values result in a new line in the table.
	Thus before querying $H$ for $x \in \{0,1\}^*$, we expect $H$ to output a uniform random value in $\mathbb{Z}_{2^k}$.
	\\
	Now suppose the  discrete logarithm is broken, so anyone can run the map $x \mapsto xG$ both ways.
	The map:
	\begin{equation*}
	\begin{split}
	\tilde{C} : \{0,1\}^* \times E_{\mathbb{F}_p}  & \rightarrow E_{\mathbb{F}_p} \\
	m, P & \mapsto H(P||m)G + P
	\end{split}
	\end{equation*}
	where $h$ is substituted by $H$.
	If $P=xG$, it becomes:
	\begin{equation*}
	\begin{split}
	\hat{C} : \{0,1\}^* \times \mathbb{Z}_n  & \rightarrow \mathbb{Z}_n \\
	m, x & \mapsto (H(xG||m) + x)\textmd{ mod }n
	\end{split}
	\end{equation*}
	We show that $\hat{C}$ can be used as a random oracle.
	$H$ is uniformly random and independent from its inputs,
	moreover the offset $x$ cannot affect that.
	Applying $\textmd{mod }n$ shrinks the range to $\mathbb{Z}_n$,
	still $\hat{C}$ can be used as a random oracle.
	Finally the output of $\hat{C}$, which is equivalent to $C$, is indistinguishable from the output of a random uniform distribution in $\mathbb{Z}_n$.
	Since $n$ is close to $2^k$ finding a second-preimage to $C(m,P)$ is infeasible.
	%Now consider the case in which $P$ is not fixed.
	%Suppose we are able to find $x$ s.t. $P=xG$, the map $C$ can be seen as $m,x \mapsto x + h(xG||m)$ which is basically $m,x \mapsto x + \tilde{h}(x,m)$ for an opportune $\tilde{h}$.
	%If such map is a random oracle then we are done, since random oracles are second preimage resistant.
	%We assume $h$ is a random oracle, then also $\tilde{h}$ has the same feature, since it is independent from its input, moreover the offset $x$ does not affect the property.
	%Hence we conclude that the map is a random oracle which gives second preimage resistance.
\end{proof}

The order $n$ has to be large so that $E_{\mathbb{F}_p}$ is rich enough to make the ECDLP intractable and to avoid shrinking the hash function codomain. Fixed $n$ the choice of $k$ should be made properly: if it is too low $h$ by itself will be too weak, if it is too high, $|I_x|$ will increase, weakening $C$. Hence a good compromise is choosing $n$ the closest possible to $2^k$.

The security of the commitment is given by the security of the hash function, and does not rely on the intractability of the ECDLP. Being able to compute the private key from the commitment is not enough to compute a second input committing to the same point, to do so it is necessary to able to find a second-preimage to an hash value.

With this commitment an elliptic curve point used for one purpose can be tweaked with $h(P||m)$ and, while still serving for the previous purpose, it can be a commitment to a value $m$. In fact suppose $x$ is the private key of $P$, $P=xG$, then $C(m,P)=(h(P||m)+x)G$. So the new private key is $(h(P||m)+x) \textmd{mod } n$ and, since $x$ is secret and $h(P||m)$ is a constant value, the resulting key is still secret.
So each time an elliptic curve point is written, we can encapsulate it in a commitment to an arbitrary value.
This technique may be extended to more general cases, but we will treat only the case of elliptic curves.

Several hash functions and elliptic curves can be used, considering we want to use Bitcoin as notary, we will focus an a particular case with $h=$ SHA256 and $E_{\mathbb{F}_p}=secp256k1$ since Bitcoin itself relies on the assumption that this hash function and this elliptic curve are not broken.
Moreover, the order of the curve $n$ is extremely close\footnote{Assuming that a generic output $x$ of SHA256 is indistinguishable from a sampling from an uniform distribution in $\mathbb{Z}_{2^{256}}$, then $\mathbb{P}(x\geq n)=1-\frac{n}{2^{256}}\approx 10^{-33}$. So the chance to choose $(P||m)$ leading to $|I_x|=2$ is almost zero.}
to the cardinality of the hash function codomain $2^{256}$. This commitment operation is called \verb|OpSecp256k1Commitment|.

Furthermore, to use this operation in as OpenTimestamp receipt, it has to be an unary operation, hence it will take as input $P||m$ and will return the $x$-coordinate of $C(m P)$ as output, both in bytes. More precisely it will operate as described in Algorithm \ref{alg:opsecp256k1}.
\begin{algorithm}
	\caption{Commitment to a $secp256k1$ point using SHA256}
	\label{alg:opsecp256k1}
	\begin{algorithmic}[1]
	\Procedure{OpSecp256k1Commitment}{$c$}\Comment{$c$ is $P||m$}
	\State $P,m \gets$ \Call{decode}{$c$}\Comment{for bad $c$  return error}
	\State $tweak \gets h(P||m)$\Comment{interpreted as an int}
	\State $Q \gets tweak G + P$
	\State \textbf{return} \Call{encode}{$Q_x$}\Comment{output in bytes}
	\EndProcedure
	\end{algorithmic}
\end{algorithm}
It can be used for several purposes, however, for this work, we focus only on timestamping.

\section{Timestamping Applications}
On the Bitcoin chain, elliptic curve points are used as public keys locking bitcoins or as part of the signature. The first case lead to the \textit{pay-to-contract} technique, the second to \textit{sign-to-contract}. We will analyse both uses and we will explain which one should be preferred.

These names owe their origin to the first application for which they were though: associate a \textit{contract} to an elliptic curve point.
The term contract may seem misleading, but it turns out to be useful dealing with \textit{sign-to-contract}, since it let us distinguish between the message signed and the contract committed.

\subsection{\textit{pay-to-contract}}
Public keys are elliptic curve points, here we show how they could commit to a value while still maintaining the secrecy of the private key.

We illustrate it through an example.
Alice needs to send Bob some bitcoins.
Bob has public key $P=xG$.
Bob wants to timestamp the message $m$.
Bob computes $Q=h(P||m)G+P$.
Bob tells Alice that his public key is $Q$.
Alice broadcasts a transaction sending bitcoins to $Q$, for instance she uses a P2PK publishing on the chain:
\begin{verbatim}
script pubkey:
<Q> OP_CHECKSIG
\end{verbatim}
Bob can spend the bitcoin locked in this script because he knows $P$, $m$ and $x$, so he can compute the private key corresponding to $Q$, that is $(h(P||m)+x) \textmd{ mod } n$.
To create the timestamp proof he has to decompose the transaction including $Q$ and create a proof that will look like:
\begin{verbatim}
prepend P
secp256k1commitment
prepend TX_p
append TX_a
sha256
...
sha256
verify BitcoinBlockHeaderAttestation(block)
\end{verbatim}
In the case of P2PKH, the committed public key is hashed as shown in the following real example, where \verb|b'Pay to contract!\n'| is timestamped:
\begin{Verbatim}[frame=single]
File sha256: 47257ff8c07f55a2e697ab9d89e47b471f60ab3f6883ed05
             44561b2a39a26140
Timestamp:
prepend 02a1e5aafa5082d035c659143660b2526a4ba60d4ab5b2e603905
        0eae9444d56ee
secp256k1commitment
prepend 02
sha256
ripemd160
prepend 01000000018fccf63afda6cf748acfe946a344f417bd4a8994bc1
        bf933501a87986363464c000000006a47304402207724a6a96e91
        a10821ee0c6db30a2f764ba8bd1dcdc82812fb958f6b91e97a4a0
        22062db4df7205e6c97d9833f7ab0d597b7685e8320bb2031f57a
        a6981cd2f626a40121030eb7a6c01ab07d3bfe598c295e9edfbeb
        38e5d2df7320f16b4349fb89a975ab7fdffffff02e22000000000
        00001976a914d8da7633fe644eb12617b2b1f0ba3f0461a2bc5e8
        8ac10270000000000001976a914
append 88ac4ed90700
# Bitcoin transaction id
1b07d87e0f4e32d545932bf03e306d1532bc7d91f56e81dee81b7cd0b707a9
9d
sha256
sha256
prepend 9d12daa914a3d39cd25b36516383683aae3ac6f873b952bbab11cc
        417650bb49
sha256
sha256
prepend 0123f36690131b2416d32a7e6c3c63110d9d77873911f71ad22740
        b398a13874
sha256
sha256
append 9f45bbc92ac4ef65b5e5bfad479da46c400f6e7ab96217a20b4e08d
       bfab47a45
sha256
sha256
prepend 949c83f6b502ec75c4647da6ed4e26d181de07b325eac75881de7b
        b715a44c50
sha256
sha256
prepend 2ef8ff1aad05e891215698fe237546c73347967419f33d08baf0d9
        71ab00004d
sha256
sha256
append 9bf69359a440f6a15a2a11adb1f257c96089ad27ef66e57a19056e3
       3ecc1795b
sha256
sha256
prepend c1c9bb36792745df3967704dd5d15899bab47b0a9de35486eb339a
        b4d00ff340
sha256
sha256
append c0b13e8aa4dc85a8256efb03272dea659e41fc67bc2f44add154111
       c68f700be
sha256
sha256
append 1bf843d12afce2a7b02ebe1f083eb2c39a101a63474a622724f609a
       3f08f8c7f
sha256
sha256
append a29eece3554c358e5df3901478c8670c71dbafe4e435cf660a18bb6
       09d6d025d
sha256
sha256
append aa2c4696c3b75f73713345a7e4279805a9519fd067835769b41dcf7
       88d6a7c96
sha256
sha256
prepend f7290a75923a0c54e87a50bbeff75f614437ed799347a46053a633
        8dba42b5c7
sha256
sha256
verify BitcoinBlockHeaderAttestation(514394)
# Bitcoin block merkle root
be6859c5093de84a06e495b6621054616ce5bf7a38f24374a225d0da0c0de8
88
\end{Verbatim}
The raw transaction with the above TXID is
\begin{Verbatim}[commandchars=+\[\], frame=single]
01000000018fccf63afda6cf748acfe946a344f417bd4a8994bc1bf933501a
87986363464c000000006a47304402207724a6a96e91a10821ee0c6db30a2f
764ba8bd1dcdc82812fb958f6b91e97a4a022062db4df7205e6c97d9833f7a
b0d597b7685e8320bb2031f57aa6981cd2f626a40121030eb7a6c01ab07d3b
fe598c295e9edfbeb38e5d2df7320f16b4349fb89a975ab7fdffffff02e220
0000000000001976a914d8da7633fe644eb12617b2b1f0ba3f0461a2bc5e88
ac10270000000000001976a914+underline[57529515dc2e14701374eb65f0191b61ecfd]
+underline[d0e3]88ac4ed90700
\end{Verbatim}
Where the commitment to the data is underlined.

This technique is completely viable, but it has a relevant issue.
Almost all bitcoin wallets (software to manage private keys) use a deterministic derivation for creating new keys \cite{BIP32}. An initial value is generated at random using a cryptographically secure procedure, this value is called \textit{seed} and sometimes encoded as a list of words from a given dictionary. The keys are obtained from the seed using the specifications given by BIP32 and they are something like $h(seed||number)$. This procedure make possible to completely recover a wallet from the seed only, so if a user wants to use his wallet from another device he just need to remember the seed and all his private keys will be reconstructed.
Using pay-to-contract actually Bob goes outside of the BIP32 derivation. So if he looses $m$ or $P$ he won't be able to spend the bitcoin locked by $Q$. For this reason the use of \textit{pay-to-contract} for mere timestamping purposes should be limited.

In the case of a P2PKH, a timestamp made with \textit{pay-to-contract} reveals the public key which is the preimage of the receiving address.
The public key will be revealed anyway when the corresponding output will be spent, but, if it is still unspent, such disclosure may be an undesired feature.
In addition, when spending that UTXO, the public key is actually written in the chain, giving another anchoring point to create a different timestamp; this timestamp is just another path from the data to the chain, but it is not really useful since it cannot precede the other timestamp.

\subsection{\textit{sign-to-contract}}
The other place where elliptic curve points are published in the chain is the signature. Bitcoin uses the elliptic curve digital signature algorithm, ECDSA\footnote{\textit{sign-to-contract} works also with other signature schemes involving elliptic curves, like Schnorr signature.}, that works as detailed in Algorithm \ref{alg:ecdsa-sign}.
\begin{algorithm}
	\caption{ECDSA signature}
	\label{alg:ecdsa-sign}
	\begin{algorithmic}[1]
		\Procedure{ECDSAsig}{$x, m$}\Comment{$x$ private key signing}
		\Statex \Comment{$m$ 32 bytes message to be signed}
		\State $k \in_R \mathbb{Z}_n \backslash \{0\}$ \Comment{select $k$ at random in $\{1,..., n-1\}$}
		\State $R \gets k G$
		\State $r \gets R_x \textmd{ mod }n$\Comment{if $r=0$, fail}
		\State $s \gets k^{-1}(m + rx) \textmd{ mod }n$\Comment{if $s=0$, fail}
		\State \textbf{return} $(r,s)$
		\EndProcedure
	\end{algorithmic}
\end{algorithm}
The value $k$ is called nonce or \textit{ephemeral private key}, $R$ is called \textit{ephemeral public key}.

A signature is a couple of integers in $\{1,...,n-1\}$, the first one is the $x$-coordinate (mod $n$) of the ephemeral public key.
The idea of \textit{sign-to-contract}, exploited in Algorithm \ref{alg:ecdsa-s2c}, is to tweak $R$, so that the first part of the signature will be a commitment to (also) another message, the contract $c$.
\begin{algorithm}
	\caption{ECDSA \textit{sign-to-contract} (s2c)}
	\label{alg:ecdsa-s2c}
	\begin{algorithmic}[1]
		\Procedure{ECDSAs2c}{$x, m, c$}\Comment{$x$ private key signing}
		\Statex \Comment{$m$ 32 bytes message to be signed}
		\Statex \Comment{$c$ contract to commit}
		\State $k \in_R \mathbb{Z}_n \backslash \{0\}$ \Comment{select $k$ at random in $\{1,..., n-1\}$}
		\State $R \gets k G$
		\State $tweak \gets h(R||c)$\Comment{interpreted as an int}
		\State $e \gets (k + tweak) \textmd{ mod }n$ \Comment{if $e=0$, fail}
		\State $Q \gets tweak G + R$
		\State $q \gets Q_x \textmd{ mod }n$\Comment{if $q=0$, fail}
		\Statex \Comment{if $q \neq Q_x$, commitment fail}
		\State $s \gets e^{-1}(m + qx) \textmd{ mod }n$\Comment{if $s=0$, fail}
		\State \textbf{return} $(q,s), R$\Comment{$R$ is needed to prove the commitment}
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

Let's examine a real timestamp for the data \verb|b'Sign to contract\n'| made with the described technique.
\begin{Verbatim}[frame=single]
File sha256: dd60bcfecd023823efdcb8d8a5b04939111ef82dc1d674320
             7e164e5aab08844
Timestamp:
append eb7e45e783d98504b2e64342b0bea3f5
sha256
prepend 0372a1fb359a24eab552e8c588f84b7e08144bbb10e87bfa6db649
        8c7df730e867
secp256k1commitment
prepend 01000000018fccf63afda6cf748acfe946a344f417bd4a8994bc1b
        f933501a87986363464c010000006a4730440220
append 022057db028ba602b467d09f67b6a6327d3219f2d9a264aae935873
       146247a18008a0121027f4b59c84fbad07dec6cff8555214b1d3740
       43bcdf47a35fbe08cf5a816b2a9ffdffffff02a6220000000000001
       976a914c7a270de581a188f1decef735602cfd65a70607c88ac1027
       0000000000001976a914ebc32f6f0a4d63da2d1a2f1f5cb762d0d89
       824d488acf3d90700
# Bitcoin transaction id
3b6b0f10729cd0d90087e8c8c9261a2b41afa4e26508591700ddd1790b5087
05
sha256
sha256
append 8c6c3e7341ac6b64c17e4558a5279da1ccf5a0346abbcb1eed412db
       103ff7cb5
sha256
sha256
prepend 8446d10571b0a6c63a0fb9538531d846148c9465eff456cccbaf91
        1a967bc74c
sha256
sha256
prepend 66cdb8cf28763b45e195028566a0bd976afcf7ad072188e711e515
        41f6867e43
sha256
sha256
prepend 465585e6b3ff7dfc8d753acce6e60c1ccd246975641dd4ad8b95c3
        803244aca7
sha256
sha256
append 93bcaa3a00534081d6f2230412cde7e59a73acb0d98c809174f630d
       e3b07b89d
sha256
sha256
append 0f1b327e68d8700e9c4074d8b4b82b0e28a5a7933f29f643cb27bd5
       6bf668ec6
sha256
sha256
append d8738b3726def527296f47a70e0ba6841e35932e2ac8a0832c25393
       ee320d4fc
sha256
sha256
append 73694be809a1f8d8a81f55e812c47d388747e1d99003d5b5427ca41
       1a5fd4408
sha256
sha256
append 816fb904a9d0678198e84f060aecca9383320bebfec2d23783c922b
       cdcb58af2
sha256
sha256
prepend f42214bc9a9c8e4b61a53e51c94ef9bbb2956202356054cd7a7677
        858caae2de
sha256
sha256
prepend 1d71d75ef769c40aec08c7ccda1a64a82ee6e858efb5f8598f5199
        a093b512d1
sha256
sha256
append 00b4475e869c96c8c297fce9ea8494f0b8f5c74d7b4e6208ba8fc84
       d103f61d2
sha256
sha256
verify BitcoinBlockHeaderAttestation(514550)
# Bitcoin block merkle root
1d978e90baecf86c9b59ecad7d8e635da27aad41b39a1d4452c7654f9d5cd3
dd
\end{Verbatim}
The raw transaction with the above TXID is
\begin{Verbatim}[commandchars=+\[\], frame=single]
01000000018fccf63afda6cf748acfe946a344f417bd4a8994bc1bf933501a
87986363464c010000006a4730440220+underline[280686720849bfd72a3c7793a45610]
+underline[db2f0152422183bb1f7181ca003674aea5]022057db028ba602b467d09f67b6
a6327d3219f2d9a264aae935873146247a18008a0121027f4b59c84fbad07d
ec6cff8555214b1d374043bcdf47a35fbe08cf5a816b2a9ffdffffff02a622
0000000000001976a914c7a270de581a188f1decef735602cfd65a70607c88
ac10270000000000001976a914ebc32f6f0a4d63da2d1a2f1f5cb762d0d898
24d488acf3d90700
\end{Verbatim}
Where the commitment value is underlined.

Using \textit{sign-to-contract} every signature can include a commitment to a certain value. Contrary to \textit{pay-to-contract}, the loss of $c$ or $R$ leads to the impossibility of proving the commitment but not to a loss of funds. Indeed the signature published on the chain has already provided to its original purpose moving the coins to another owner. This makes \textit{sign-to-contract} a preferable commitment scheme.

In Algorithm \ref{alg:ecdsa-sign} and \ref{alg:ecdsa-s2c} the nonce $k$ is generated at random, however often computers are poor sources of randomness which may lead to security issues, namely exposing the nonce $k$ actually reveals the private key $x$.
To reduce this problem, it is a common practice to avoid the random generation, instead, it is performed a deterministic derivation of the type $k \gets h(x||m)$, the precise specification of $h$ is given by the RFC6979 standard \cite{rfc6979}. With this technique,  in a signature, the private key is unique source of entropy used for security, indeed the message $m$ is public and the nonce $k$ is as secret as $x$, since, without $x$, it is not possible to guess $h(x||m)$.
Deterministic nonce and \textit{sign-to-contract} are completely independent techniques that can improve a signature scheme, it is possible to implement one without the other.

In addition, exposing two signature for the same message $m$, generated with the same private key $x$, but made using two different nonces $k_1, k_2$ reveals the private key.
This is a relevant issue when managing bitcoin, however signing with a deterministic nonce solves this problem.
With \textit{sign-to-contract}, considering a message $m$, private $x$, a deterministic nonce, but two contracts $c_1, c_2$, such issue, despite the deterministic derivation, arises again.
Thus is important take care of this chance during the implementation.

Using a deterministic nonce also, in some sense\footnote{To verify that the nonce is deterministic, it is necessary the knowledge of the private key, thus the verification cannot be performed by whoever. Moreover, in some cases, like bitcoin hardware wallets, extracting the secret key may be hard or unsafe.}, closes the subliminal channel in ECDSA signature.
The signer has some arbitrariness in choosing the nonce and could use it to transmit a certain message, as exploited in \cite{DBLP:journals/jsac/Simmons98}.
With \textit{sign-to-contract} the nonce is still deterministic, but is a function of $x,m$ and, in addition, $c$.
The technique could be seen as a particular use of the subliminal channel of ECDSA.
Consider the case where Alice wants to secretly communicate a simple message $c$ to Bob. Alice produces a signature which is a commitment via \textit{sign-to-contract} to a simple message $c$. Alice declares $P$ publicly. Bob knows that $c$ comes from a brute forceable set $S$. Bob sees the signature and tries all $c \in S$ until he finds the one which generated the commitment. As a result, Alice sent to Bob a message without anyone noticing the communication using the subliminal channel that ECDSA leaves open.


\lab{Givens rotations and Least squares}{Givens rotations and Least squares}
\objective{Use Givens rotations to find the QR decomposition and use least squares to fit curves to data.}
\label{lab:givens}

In Lab \ref{lab:QRdecomp}, we found the QR decomposition of a matrix using Householder transformations, applying a series of these transformations to a matrix until it was in upper triangular form.
We can use the same strategy to compute the QR decomposition with rotations instead of reflections.

\section*{Givens rotations}

Let us begin with Givens rotations in $\mathbb{R}^2$.
An arbitrary vector $\x = (a, b)\trp$ can be rotated into the span of $e_1$ via an orthogonal transformation.
In fact, the matrix $T_{\theta} = \begin{pmatrix}\cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{pmatrix}$ rotates a vector counterclockwise by $\theta$.
Thus, if $\theta$ is the clockwise-angle between $\x$ and $e_1$, the vector $T_{-\theta}\x$ will be in the span of $e_1$.
We can find $\sin \theta$ and $\cos \theta$ with the formulas $\sin = \frac{\text{opp}}{\text{hyp}}$ and $\cos = \frac{\text{adj}}{\text{hyp}}$, so $\sin \theta = \frac{b}{\sqrt{a^2+b^2}}$ and $\cos \theta =  \frac{a}{\sqrt{a^2+b^2}}$ (see Figure\ref{fig:angle}).
Then
\[
T_{-\theta}\x = \begin{pmatrix}\cos \theta &  \sin \theta \\ -\sin \theta & \cos \theta \end{pmatrix} \begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix}\frac{a}{\sqrt{a^2+b^2}} & \frac{b}{\sqrt{a^2+b^2}} \\ -\frac{b}{\sqrt{a^2+b^2}} & \frac{a}{\sqrt{a^2+b^2}} \end{pmatrix}\begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix} \sqrt{a^2+b^2} \\ 0 \end{pmatrix}.
\]

\begin{figure}
\begin{center}
\begin{tikzpicture}
\draw[-, thick](-.5,0)--(3,0);
\draw[-, thick](0,-.5)--(0,1.5);
\draw[->, thick, >=stealth'](0,0)--(2.5,1);
\draw[-,thick](2.5,-.2)--(2.5,.2);
\draw[-, thick](-.2,1)--(.2,1);
\node[draw=none](point_b)at(2.5,-.4){$a$};
\node[draw=none](point_a)at(-.4,1){$b$};
\draw[-, thick] (.5,.2)arc [start angle=60,
	end angle=-20, radius=4.5pt];
\node[draw=none](theta)at(.8, .15){$\theta$};
\end{tikzpicture}
\caption{Rotating clockwise by $\theta$ will send the vector $(a,b)\trp$ to the span of $e_1$.}
\label{fig:angle}
\end{center}
\end{figure}

The matrix $T_{\theta}$ above is an example of a $2 \times 2$ Givens rotation matrix.
In general, the Givens matrix $G(i,j,\theta)$ represents the orthonormal transformation that rotates the 2-dimensional span of $e_i$ and $e_j$ by $\theta$ radians.
The matrix for this transformation is
\begin{equation*}
G(i,j,\theta) = \begin{pmatrix}
I & 0 & 0 & 0 & 0 \\
0 & c & 0 & -s & 0 \\
0 & 0 & I & 0 & 0 \\
0 & s & 0 & c & 0 \\
0 & 0 & 0 & 0 & I
\end{pmatrix}.
\end{equation*}
This matrix is in block form with $I$ representing the identity matrix, $c=\cos \theta$, and $s=\sin \theta$.
The $c$'s appear on the $i^{th}$ and $j^{th}$ diagonal entries.

As before, we can choose $\theta$ so that $G(i,j,\theta)$ rotates a given vector so that its $e_j$-component is 0.
Such a transformation will only affect the $i^{th}$ and $j^{th}$ entries of any vector it acts on (and thus the $i^{th}$ and $j^{th}$ rows of any matrix it acts on).

This flexibility makes Givens rotations ideal for some problems.
For example, Givens rotations can be used to solve linear systems defined by sparse matrices by modifying only small parts of the array.
Also, Givens rotations can be used to solve systems of equations in parallel.

The advantages of Givens rotations are that they orthonormal and hence numerically stable (like Householder reflections), and they affect only a small part of the array (like Gaussian elimination).
The disadvantage is that they require a greater number of floating point operations than Householder reflections.
% Accuracy and Stability of Numerical Algorithms, Nicholas J. Higham
In practice, the Givens algorithm is slower than the Householder algorithm, even when it is modified to decrease the number of floating point operations.
% Fast Plane Rotations With Dynamic Scaling, Anda and Park, SIAM, 1994
However, since Givens rotations can be parallelized, they can be much faster than the Householder algorithm when multiple processors are used.
% Givens and Householder Reductions for Linear Least Squares on a Cluster of Workstations, Omer Egecioglu and Ashok Srinivasan

\subsection*{Givens triangularization}
We can apply Givens rotations to a matrix until it is in upper triangular form, producing a factorization $A = QR$ where $Q$ is a composition of Givens rotations and $R$ is upper triangular.
This is exactly the QR decomposition of $A$.

The idea is to iterate through the subdiagonal entries of $A$ in the order depicted by Figure \ref{fig:givens}.
We zero out the $ij^{th}$ entry with a rotation in the plane spanned by $e_{i-1}$ and $e_i$.
This rotation is just multiplication by the Givens matrix $G(i-1,i,\theta)$, which can be computed as in the example at the start of the previous section.
We just set $a=a_{i-1,j}$ and $b=a_{i,j}$, so $c = \cos \theta = a/\sqrt{a^2+b^2}$ and $s = -b/\sqrt{a^2+b^2}$.

\begin{figure}
\begin{center}
\begin{tikzpicture}[xscale=.7, yscale=.7]
\draw[->, gray, thick](0,0)--(0,5.5);
\draw[->, gray, thick](1,0)--(1,4.5);
\draw[->, gray, thick](2,0)--(2,3.5);
\draw[->, gray, thick](3,0)--(3,2.5);
\draw[->, gray, thick](4,0)--(4,1.5);
\node[draw=none] at(0,-1){\large 1};
\node[draw=none] at(1,-1){\large 2};
\node[draw=none] at(2,-1){\large 3};
\node[draw=none] at(3,-1){\large 4};
\node[draw=none] at(4,-1){\large 5};

\draw[-, ultra thick] (0,6)--(4,2);

\draw[-, thick](-1,-.5)--(-1,6.5);
\draw[-, thick](-1, -.5)--(-.5,-.5);
\draw[-, thick](-1,6.5)--(-.5,6.5);

\draw[-, thick](5,-.5)--(5,6.5);
\draw[-, thick](5,-.5)--(4.5,-.5);
\draw[-, thick](5,6.5)--(4.5,6.5);

%\draw[-, thick](-.5,0)--(3,0);
%\draw[-, thick](0,-.5)--(0,1.5);
%\draw[->, thick, >=stealth'](0,0)--(2.5,1);
%\draw[-,thick](2.5,-.2)--(2.5,.2);
%\draw[-, thick](-.2,1)--(.2,1);
%\node[draw=none](point_b)at(2.5,-.4){$a$};
%\node[draw=none](point_a)at(-.4,1){$b$};
%\draw[-, thick] (.5,.2)arc [start angle=60,
%	end angle=-20, radius=4.5pt];
%\node[draw=none](theta)at(.8, .15){$\theta$};
\end{tikzpicture}
\caption{This figure illustrates the order in which to zero out subdiagonal entries in the Givens triangularization algorithm.
The heavy black line is the main diagonal of the matrix.
Entries should be zeroed out from bottom to top in each column, beginning with the leftmost column.}
\label{fig:givens}
\end{center}
\end{figure}

For example, on a $2 \times 3$ matrix we may perform the following operations:

\def\mc#1{\multicolumn{1}{c|}{#1}}
\def\lc#1{\multicolumn{1}{|c}{#1}}
\[
\begin{array}{ccccccc}
\begin{pmatrix}
*&*\\
*&*\\
*&*
\end{pmatrix}
&
\underrightarrow{G(2,3,\theta_1)}
&\begin{pmatrix}
&*&*&\\ \cline{2-3}
&\lc{*}&\mc{*}&\\
&\lc{0}&\mc{*}& \\ \cline{2-3}
\end{pmatrix}
&
\underrightarrow{G(1,2,\theta_2)}
& \begin{pmatrix} \cline{2-3}
&\lc{*}&\mc{*}&\\
&\lc{0}&\mc{*}&\\ \cline{2-3}
&0&*&
\end{pmatrix}
&
\underrightarrow{G(2,3,\theta_3)}
&\begin{pmatrix}
*&*&\\ \cline{2-2}
\mc{0}&\mc{*}&\\
\mc{0}&\mc{0}&\\ \cline{2-2}
\end{pmatrix}
\end{array}
\]
At each stage, the boxed entries are those modified by the previous transformation.
The final transformation $G(2,3,\theta_3)$ operates on the bottom two rows, but since the first two entries are zero, they are unaffected.
Assuming that at the $ij^{th}$ stage of the algorithm, $a_{ij}$ is nonzero, Algorithm \ref{Alg:givens} computes the Givens triangularization of a matrix..

\begin{algorithm}
\begin{algorithmic}[1]
\caption{Givens triangularization. Return an orthogonal matrix $Q$ and an upper triangular matrix $R$ satisfying $A = QR$.}
\label{Alg:givens}
\Procedure{Givens Triangularization}{$A$}
\State $m, n \gets \shape{A}$
\State $R \gets \makecopy{A}$
\State $Q \gets \Id{m}$
\State $G \gets \allocate{2}{2}$
\For{$j=0\ldots n-1$}
    \For{$i=m\ldots j+1$}
      \State $a, b \gets R[i-1,j], R[i,j]$
      \State $G \gets [[a, b],[-b,a]]/\sqrt{a^2+b^2}$
      \State $R[i-1:i+1, j:] \gets GR[i-1:i+1, j:]$
      \State $Q[i-1:i+1,:] \gets GQ[i-1:i+1,:]$
    \EndFor
\EndFor
\State \pseudoli{return} $Q\trp , R$
\EndProcedure
\end{algorithmic}
\end{algorithm}

Notice that in Algorithm \ref{Alg:givens}, we \emph{do not} actually create the matrices $G(i,j,\theta)$ and multiply them by the original matrix.
Instead we modify only those entries of the matrix that are affected by the transformation. As an additional way to save memory, it is possible to modify this algorithm so that $Q$ and $R$ are stored in the original matrix $A$.
\begin{comment}
An interesting side-note is that each Givens rotation can be represented as a single floating point number, so, when operating in place, $Q$ can be stored entirely in the lower triangular portion of the array on which we are operating by storing each rotation in the entry that it zeroes out.
A similar approach would to store Householder reflectors in the columns they zero out.
In either case, we can represent the QR decomposition of an array using only the memory that was originally used to store the array itself.
This is similar to the approach  for computing the LU decomposition entirely in place.
These representations of $Q$ and $R$ can be used in various ways to perform matrix multiplication by $Q$, $Q\trp$ and $R$ as needed.
\end{comment}

\begin{comment}
\begin{itemize}[$\bullet$]

\item Make $R$ a copy of $A$ and $Q$ an identity array of the appropriate size.

\item Make an empty $2 \times 2$ array $G$ that will be used to apply the Givens rotations.

\item For each column:

  \begin{itemize}[$\bullet$]

  \item For each row below the main diagonal (starting at the bottom of the column):

    \begin{itemize}[$\bullet$]

    \item If the leading entry of this row is not zero (i.e. if its absolute value is within a given tolerance):

      \begin{itemize}[$\bullet$]

      \item Compute $c$ and $s$ using the entry in the current row and column and the entry immediately above it.

      \item Use $c$ and $s$ to construct the matrix $G$.

      \item Get a slice of $R$ of the current row and the row above it that includes the columns from the current column onward.
      Multiply it in place by $G$ to zero out the leading nonzero entry of the current row.

      \item Get a slice of $Q$ of the current row and the row above it and apply $G$ to it as well. (Strictly speaking, you do not need to operate over these entire rows, but the slicing needed to avoid the extra computation is a little more involved, so we will not include that here.)

      \end{itemize}

    \end{itemize}

  \end{itemize}

\item Return $Q\trp$ and $R$.

\end{itemize}
\end{comment}

\begin{problem}
\label{prob:Givens}
Write a function that computes the Givens triangularization of a matrix, using Algorithm \ref{Alg:givens}.
Assume that at the $ij^{th}$ stage of the algorithm, $a_{ij}$ will be nonzero.
\end{problem}

\begin{problem}
\label{prob:givens_hessenberg}
Modify your solution to Problem \ref{prob:Givens} to compute the Givens triangularization of an upper Hessenberg matrix, making the following changes:
\begin{enumerate}
\item Iterate through the first subdiagonal from left to right. (These are the only entries that need to be zeroed out.)
\item Line 10 of Algorithm \ref{Alg:givens} updates $Q$ with the current Givens rotation $G$. Decrease the number of entries of $Q$ that are modified in this line. Do this by replacing $Q[i-1:i+1, :]$ with $Q[i-1:i+1, :k_i]$ where $k_i$ is some appropriately chosen number (dependent on $i$) such that $Q[i-1:i+1, k_i:]=0$.
\end{enumerate}
Hint: Here is how to generate a random upper Hessenberg matrix on which to test your function.
The idea is to generate a random matrix and then zero out all entries below the first subdiagonal.
\begin{lstlisting}
import numpy as np
import scipy.linalg as la
from math import sqrt
A = np.random.rand(500, 500)

# We do not need to modify the first row of A
# la.triu(A[1:]) zeros out all entries below the diagonal of A[1:]
A[1:] = la.triu(A[1:])

# A is now upper Hessenberg
\end{lstlisting}

\begin{comment}
What is the computational order of  complexity for this problem?
Approximately for what $m$ is your implementation as fast as the general QR decomposition built in to \li{scipy.linalg} for computing the QR decomposition of an upper Hessenberg matrix?
\end{comment}
\end{problem}

\begin{comment}
\begin{problem}
\label{prob:givens_hessenberg_modified}
You may have noticed that matrix multiplication by $Q$ is generally a $\mathcal{O} \left( n^3 \right)$ algorithm, while the application of these individual Givens rotations is a $\mathcal{O} \left( n^2 \right)$ algorithm.
Write a modified version of your solution to Problem \ref{prob:givens_hessenberg} called \li{givens2_mod} which returns
an $(n-1) \times 2 \times 2$ array containing the computed values for $G$ at each step in the algorithm in the order in which they are applied to the upper Hessenberg array $H$.

Write two more functions \li{apply_Q} and \li{apply_QT} which, using the matrix of Givens rotations, perform left multiplication
by $Q$ and $Q^{-1}$, respectively, on some other input array $B$.
Left multiplication by $Q^{-1}$ can be done by applying each of the Givens rotations to $B$ the same way you did to $H$ to compute its QR factorization.
Left multiplication by $Q$ can be done by applying the transpose of the Givens rotations to their corresponding portions of $B$, but in the reverse order.
Notice that you will have to apply each Givens rotation across the full width of the rows it operates on since you do not know anything about the content of $B$.

For around what size of matrices is direct multiplication by $Q$ slower than this method of multiplying by $Q$?
For timing purposes, make a random upper Hessenberg matrix, compute its QR decomposition using the function you just wrote and your solution to Problem \ref{prob:givens_hessenberg}, then time how long it takes to left multiply a random square array by $Q$ using the function you just wrote and the \li{dot} method of NumPy arrays.

Note: the functions you just wrote can be used to perform right multiplication as well since $B Q = \left(Q\trp B\trp \right)\trp$ and $B Q\trp = \left( Q B\trp \right)\trp$.
\end{problem}
\end{comment}

\section*{Least Squares}

A linear system $A\x=\b$ is \emph{overdetermined} if it has no solutions.
In this situation, the \emph{least squares solution} is a vector $\widehat{\x}$ hat is ``closest'' to a solution.
By definition, $\widehat{\x}$ is the vector such that $A\widehat{\x}$ will equal the projection of $\b$ onto the range of $A$.
We can compute $\widehat{\x}$ by solving the \emph{Normal Equation} $A\trp A\widehat{\x} = A\trp \b$ (see [TODO: ref textbook] for a derivation of the Normal Equation).

\subsection*{Solving the normal equation}
If $A$ is full rank, we can use its QR decomposition to solve the normal equation.
In many applications, $A$ is usually full rank, including when least squares is used to fit curves to data.

Let $A=QR$ be the QR decomposition of $A$, so $R = \left(\begin{array}{c}R_0\\
0\\ \end{array} \right)$
where $R_0$ is $n \times n$, nonsingular, and upper triangular.
It can be shown that $\widehat{\x}$ is the least squares solution to $A\x=\b$ if and only if $R_0\widehat{\x} = (Q\trp \b)[:n].$
Here, $(Q\trp \b)[:n]$ refers to the first $n$ rows of $Q\trp \b$.
Since $R$ is upper triangular, we can solve this equation quickly with back substitution.

\begin{problem}
Write a function that accepts a matrix $A$ and a vector $b$ and returns the least squares solution to $Ax=b$.
Use the QR decomposition as outlined above.
Your function should use SciPy's functions for QR decomposition and for solving triangular systems, which are \li{la.qr()} and \li{la.solve_triangular()}, respectively.
\end{problem}

\subsection*{Using least squares to fit curves to data}
The least squares solution can be used to find the curve of a chosen type that best fits a set of points.

\subsubsection*{Example 1: Fitting a line}
For example, suppose we wish to fit a general line $y=mx+b$ to the data set $\{(x_k, y_k)\}_{k=1}^n$.
When we plug the constants $(x_k, y_k)$ into the equation $y=mx+b$, we get a system of linear equations in the unknowns $m$ and $b$.
This system corresponds to the matrix equation
\[
\begin{pmatrix}
x_1 & 1\\
x_2 & 1\\
x_3 & 1\\
\vdots & \vdots\\
x_n & 1
\end{pmatrix}
\begin{pmatrix}
m\\
b
\end{pmatrix}=
\begin{pmatrix}
y_1\\
y_2\\
y_3\\
\vdots\\
y_n
\end{pmatrix}.
\]
Because this system has two unknowns, it is guaranteed a solution if it has two or fewer equations.
In applications, there will usually be more than two data points, and these will probably not lie in a straight line, due to measurement error.
Then the system will be overdetermined.
The least squares solution to this equation will be a slope $\widehat{m}$ and $y$-intercept $\widehat{b}$ that produce a line $y = \widehat{m}x+\widehat{b}$ which best fits our data points.

%DO: spring constant as an example of this
% circle fit
% mention in this situation A will usually be full rank.
%todo: least squares and invertibiility.

Let us do an example with some actual data. Imagine we place different loads on a spring and measure the displacement, recording our results in the table below.
%TODO: get data points that are not so close to an actual line
\begin{table}
\begin{tabular}{c|c|c|c|c|c|c|c}
displacement (cm)& 1.04  &2.03  &2.95  &3.92  &5.06  &6.00  &7.07  \\ \hline
load (dyne) & 3.11&  6.01&  9.07&  11.99 &  15.02&  17.91&  21.12\\
\end{tabular}
\end{table}

Hooke's law from physics says that the displacement $x$ should be proportional to the load $F$, or $F = kx$ for some constant $k$.
The equation $F=kx$ describes a line with slope $k$ and $F$-intercept 0.
So the setup is similar to the setup for the general line we discussed above, except we already know $b=0$
When we plug our seven data points $(x,F)$ pairs into the equation $F=kx$, we get seven linear equations in $k$, corresponding to the matrix equation
\[
\begin{pmatrix}
1.04\\
2.03\\
2.95\\
3.92\\
5.06\\
6.00\\
7.07\\
\end{pmatrix}
\begin{pmatrix}k\end{pmatrix} =
\begin{pmatrix}
3.11 \\
6.01\\
9.07\\
11.99\\
15.02\\
17.91\\
21.12\\
\end{pmatrix}.
\]
We expect such a linear system to be overdetermined, and in fact it is: the equation is $1.04k = 3.11$ which implies $k=2.99$, but the second equation is $2.03k = 6.01$ which implies $k=2.96$.

We can't solve this system, but its least squares solution is a ``best'' choice for $k$.
We can find the least squares solution with the SciPy function \li{linalg.lstlsq()}.
This function returns a tuple of several values, the first of which is the least squares solution.
\begin{lstlisting}
>>> A = np.vstack([1.04,2.03,2.95,3.92,5.06,6.00,7.07])
>>> b = np.vstack([3.11,6.01,9.07,11.99,15.02,17.91,21.12])
>>> k = la.lstsq(A, b)[0]
>>> k
array([[ 2.99568294]])
\end{lstlisting}
Hence, to two decimal places, $k = 3.00$.
We plot the data against the best-fit line with the following code, whose output is in Figure \ref{fig:spring_fit}

\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> x0 = np.linspace(0,8,100)
>>> y0 = k[0]*x0
>>> plt.plot(A,b,'*',x0,y0)
>>> plt.show()
\end{lstlisting}

\begin{figure}
\includegraphics[width=\textwidth]{line_lstsq}
\caption{The graph of the spring data together with its linear fit.}
\label{fig:spring_fit}
\end{figure}

%TODO: find more interesting data and make a sample plot
\begin{problem}
Load the \li{linepts} array from the file \texttt{data.npz}. The following code stores this array as \li{linepts}.
\begin{lstlisting}
linepts = np.load('data.npz')['linepts']
\end{lstlisting}
The \li{linepts} array has two columns corresponding to the $x$ and $y$ coordinates of some data points.
\begin{enumerate}
\item Use least squares to fit the line $y=mx+b$ to the data.
\item Plot the data and your line on the same graph.
\end{enumerate}
\end{problem}

%
%\section*{General Line Fitting}
%
%Suppose that we wish to fit a general line, that is $y=m x+b$, to the data set
%$\{(x_k,y_k)\}^n_{k=1}$.  Assume that the line does not cross through the origin,
%as in the previous example.  Then we seek both a slope and a $y$-intercept.
%In this case, we set up the following linear system $A x = b$, or more precisely
%\[
%\begin{pmatrix}
%x_1 & 1\\
%x_2 & 1\\
%x_3 & 1\\
%\vdots & \vdots\\
%x_n & 1
%\end{pmatrix}
%\begin{pmatrix}
%m\\
%b
%\end{pmatrix}=
%\begin{pmatrix}
%y_1\\
%y_2\\
%y_3\\
%\vdots\\
%y_n
%\end{pmatrix}.
%\]
%Note that $A$ has rank $2$ as long as not all of the $x_k$ values are the same.
%Hence, the least squares solution
%is given by
%$$
%\widehat{x} = (A^HA)^{-1}A^Hb.
%$$
%In what sense does this solution give us the best fit line for the data? Recall that since $A$ is injective,
%the matrix $A(A^HA)^{-1}A^H$ is an orthogonal projector onto the range of $A$, which means that
%$A(A^HA)^{-1}A^Hb = A\widehat{x}$ is the closest vector (with respect to the 2-norm) to $b$ that lies in the
%range of $A$. That is, $\widehat{x}$ minimizes the error between $Ax$ and $b$, where the error is given
%by the distance between these vectors, $\|b-Ax\|_2$. Another way to say this is that $\widehat{x}$ gives the
%values $m$ and $b$ for which the sum of the squares of the distances from each data point $y_k$ to the value
%$y = mx_k + b$ is as small as possible.

\subsubsection*{Example 2: Fitting a circle}
Now suppose we wish to fit a general circle to a data set $\{(x_k, y_k)\}_{k=1}^n$. Recall that the equation of a circle with radius $r$ and center $(c_1,c_2)$ is
\begin{equation}
\label{circle}
(x-c_1)^2 + (y-c_2)^2 = r^2.
\end{equation}
What happens when we plug a data point into this equation? Suppose $(x_k, y_k)=(1,2)$.
\footnote{You don't have to plug in a point for this derivation, but it helps us remember which symbols are constants and which are variables.} Then
\begin{equation*}\label{equ:example}
5 = 2c_1+4c_2+(r^2-c_1^2-c_2^2).
\end{equation*}
To find $c_1$, $c_2$, and $r$ with least squares, we need \emph{linear} equations.
Then Equation \ref{equ:example} above is not linear because of the $r^2$, $c_1^2$, and $c_2^2$ terms.
We can do a trick to make this equation linear: create a new variable $c_3$ defined by $c_3 = r^2-c_1^2-c_2^2$.
Then Equation \ref{equ:example} becomes
\[
5=2c_1+4c_2+c_3,
\]
which \emph{is} linear in $c_1$, $c_2$, and $c_3$. Since $r^2 = c_3+c_1^2+c_2^2$, after solving for the new variable $c_3$ we can also find $r$.

For a general data point $(x_k, y_k)$, we get the linear equation
\[
2c_1x_k+2c_2y_k+c_3=x_k^2+y_k^2.
\]
Thus, we can find the best-fit circle from the least squares solution to the matrix equation

\begin{equation}\label{equ:circle_fit}
\begin{pmatrix}
2 x_1 & 2 y_1 & 1\\
2 x_2 & 2 y_2 & 1\\
\vdots & \vdots & \vdots \\
2 x_n & 2 y_n & 1
\end{pmatrix}
\begin{pmatrix}
c_1\\
c_2\\
c_3
\end{pmatrix}=
\begin{pmatrix}
x_1^2 + y_1^2\\
x_2^2 + y_2^2\\
\vdots\\
x_n^2 + y_n^2
\end{pmatrix}.
\end{equation}
If the least squares solution is $\widehat{c_1}, \widehat{c_2}$, $\widehat{c_3}$, then the best-fit circle is
\[
(x-\widehat{c_1})^2 + (y-\widehat{c_2})^2 = \widehat{c_3}+\widehat{c_1}^2+\widehat{c_2}^2.
\]

Let us use least squares to find the circle that best fits the following nine points:
%TODO: get data points that are not so close to an actual circle
\begin{table}
\begin{tabular}{c||c|c|c|c|c|c|c|c|c}
$x$& 134  &104 &34  &-36  &-66  &-36  &34 &104 &  134  \\ \hline
$y$& 76&  146&  176&  146 &  76&  5& -24 & 5 & 76\\
\end{tabular}
\end{table}

We enter them into Python as a $9\times 2$ array.
\begin{lstlisting}
>>> P = np.array([[134,76],[104,146],[34,176],[-36,146],
                  [-66,76],[-36,5],[34,-24],[104,5],[134,76]])
\end{lstlisting}

We compute $A$ and $b$ according to Equation \ref{equ:circle_fit}.
\begin{lstlisting}
>>> A = np.hstack((2*P, np.ones((9,1))))
>>> b = (P**2).sum(axis=1)
\end{lstlisting}

Then we use SciPy to find the least squares solution.
\begin{lstlisting}
>>> c1, c2, c3 = la.lstsq(A, b)[0]
\end{lstlisting}

We can solve for $r$ using the relation $r^2 = c_3+c_1^2+c_2^2$.
\begin{lstlisting}
>>> r = sqrt(c1**2 + c2**2 + c3)
\end{lstlisting}

A good way to plot a circle is to use polar coordinates.
Using the same variables as before, the equation for a general circle is $x=r\cos(\theta)+c_1$ and $y=r\sin(\theta)+c_2$.
With the following code we plot the data points and our best-fit circle using polar coordinates.
The resulting image is Figure \ref{fig:circle}.
\begin{lstlisting}
# In the polar equations for a circle, theta goes from 0 to 2*pi.
>>> theta = np.linspace(0,2*np.pi,200)
>>> plt.plot(r*np.cos(theta)+c1,r*np.sin(theta)+c2,'-',P[:,0],P[:,1],'*')
>>> plt.show()
\end{lstlisting}

\begin{figure}
\includegraphics[width=\textwidth]{circle.pdf}
\caption{The graph of the some data and its best-fit circle.}
\label{fig:circle}
\end{figure}

\begin{comment}
\begin{problem}
Write a function \li{fitCircle} that does the following.
Load the \texttt{circlepts} array from \texttt{data.npz}.
This consists of two columns corresponding to the $x$ and $y$ values of a given
data set.  Use least squares to find the center and radius of the circle that best
fits the data.  Then plot the data points and the circle on the same graph.
The function should return nothing.
\end{problem}
\end{comment}

%TODO: figure out how to plot this problem
\begin{problem}
\leavevmode
\begin{enumerate}
\item Load the \texttt{ellipsepts} array from \texttt{data.npz}. This array has two columns corresponding to the $x$ and $y$ coordinates of some data points.
\item Use least squares to fit an ellipse to the data.
The general equation for an ellipse is
\[
ax^2 + bx + cxy + dy + ey^2 = 1.
\]
You should get  $0.087$, $-0.141$,  $0.159$, $-0.316$, $0.366$ for $a, b, c, d,$ and $e$ respectively.
%\item Plot the data and your line on the same graph.
\end{enumerate}
\end{problem}

%TODO: keep this?
\begin{comment}
In these Least Squares problems, we have found best fit lines and ellipses relative to the 2-norm.
It is possible to generalize the idea of best fit curves relative to other norms.
See Figure \ref{Fig:ellipse} for an illustration of this.

\begin{figure}[h]
\label{ellipsefit}
\centering
\includegraphics[width=\textwidth]{ellipsefit.pdf}
\caption{Fitting an ellipse using different norms.}
\label{Fig:ellipse}
\end{figure}
\end{comment}

\begin{comment}
\section*{Loading Data from .npz Files}
For Least Squares problems as well as in many other contexts, loading data is often a necessary step before
proceeding with further analysis. Here we briefly review another data format in Python and the commands used
to load the data.

A \li{.npz} file is a compressed binary file that contains an archive of NumPy data structures.
A given file may therefore contain several arrays, each array associated with a unique string that identifies it.
When you load a \li{.npz} file in Python, a dictionary-like object is returned, and you can access the data by
providing the appropriate key. Note that when you load a \li{.npz} file, you must also be sure to close it when
you are finished. This is taken care of automatically if you use the \li{with ... as} keywords.

As an example, suppose that we have a file named \li{grades.npz} that contains several arrays, each giving the
homework scores of a particular student in a particular class. Assuming that one of the arrays is associated with
the key \li{'Abe'}, we can load this array in the following way:

\begin{lstlisting}
>>> with np.load('grades.npz') as grades:
>>>     abe_grades = grades['Abe']
>>> abe_grades
array([ 10.,  10.,  10.,  10.,  10.,  10.,  10.,  10.,  10.,  10.])
\end{lstlisting}

You will need to apply this technique in the next problem.

\end{comment}


\subsubsection{The Cholesky factorization}

Another important theorem is the existence of a specific factorization of positive definite matrices. It is called the Cholesky Factorization and factors the matrix into the product of an upper triangular matrix and its transpose.

\index{Cholesky factorization!positive definite}
\index{positive definite!Cholesky factorization}
\begin{theorem}{Cholesky factorization}{cholesky-factorization}
Let $A$ be a positive definite matrix. Then
there exists an upper triangular matrix $U$ whose main diagonal entries are positive, such that $A$ can be written

\begin{equation*}
A=
U^TU
\end{equation*}
This factorization is unique.
\end{theorem}

The process for finding such a matrix $U$ relies on simple row operations.

\begin{procedure}{Finding the Cholesky factorization}{finding-cholesky}
Let $A$ be a positive definite matrix. The matrix $U$ that creates the Cholesky Factorization can be found through two steps.
\begin{enumerate}
\item Using only type $3$ elementary row operations (multiples of rows added to other rows) put $A$ in upper triangular form. Call this matrix $\hat{U}$. Then $\hat{U}$ has positive entries on the main diagonal.
\item Divide each row of $\hat{U}$ by the square root of the diagonal entry in that row. The result is the matrix $U$.
\end{enumerate}
\end{procedure}

Of course you can always verify that your factorization is correct by multiplying $U$ and $U^T$ to ensure the result is the original matrix $A$.

Consider the following example.

\begin{example}{Cholesky factorization}{cholesky}
Show that
$A=\begin{mymatrix}{rrr}
9 & -6 & 3 \\ -6 & 5 & -3 \\ 3 & -3 & 6
\end{mymatrix}$
is positive definite, and find the Cholesky factorization of $A$.
\end{example}

\begin{solution}
First we show that $A$ is positive definite. By Theorem~\ref{thm:positive-matrix-determinant-Ak} it suffices to show that the determinant of each submatrix is positive.
\[ A_{1}=\begin{mymatrix}{c} 9 \end{mymatrix}
\mbox{ and }
A_{2}=\begin{mymatrix}{rr} 9 & -6 \\ -6 & 5 \end{mymatrix},\]
so $\det(A_{1})=9$ and $\det(A_{2})=9$.
Since $\det(A)=36$, it follows that $A$ is positive definite.

Now we use Procedure~\ref{proc:finding-cholesky} to find the Cholesky Factorization. Row reduce (using only type $3$ row operations) until an upper triangular matrix is obtained.
\[ \begin{mymatrix}{rrr}
9 & -6 & 3 \\ -6 & 5 & -3 \\ 3 & -3 & 6
\end{mymatrix}
\rightarrow
\begin{mymatrix}{rrr}
9 & -6 & 3 \\ 0 & 1 & -1 \\ 0 & -1 & 5
\end{mymatrix}
\rightarrow
\begin{mymatrix}{rrr}
9 & -6 & 3 \\ 0 & 1 & -1 \\ 0 & 0 & 4
\end{mymatrix}
\]

Now divide the entries in each row by the square root of the diagonal
entry in that row, to give

\[ U=\begin{mymatrix}{rrr}
3 & -2 & 1 \\ 0 & 1 & -1 \\ 0 & 0 & 2
\end{mymatrix}
\]

You can verify that $U^TU = A$.
\end{solution}

\begin{example}{Cholesky factorization}{cholesky}
Let $A$ be a positive definite matrix given by
\begin{equation*}
\begin{mymatrix}{ccc}
3 & 1 & 1 \\
1 & 4 & 2 \\
1 & 2 & 5
\end{mymatrix}
\end{equation*}
Determine its Cholesky factorization.
\end{example}

\begin{solution}
You can verify that $A$ is in fact positive definite.

To find the Cholesky factorization we first row reduce to an upper triangular matrix.
\[
\begin{mymatrix}{ccc}
3 & 1 & 1 \\
1 & 4 & 2 \\
1 & 2 & 5
\end{mymatrix}
\rightarrow
\begin{mymatrix}{ccc}
3 & 1 & 1 \\
0 & \vspace{0.05in}\frac{11}{3} & \vspace{0.05in}\frac{5}{3} \\
0 & \vspace{0.05in}\frac{5}{3}  & \vspace{0.05in}\frac{14}{5}
\end{mymatrix}
\rightarrow
\begin{mymatrix}{ccc}
3 & 1 & 1 \\
0 & \vspace{0.05in}\frac{11}{3} & \vspace{0.05in}\frac{5}{3} \\
0 & 0 & \vspace{0.05in}\frac{43}{11}
\end{mymatrix}
\]

Now divide the entries in each row by the square root of the diagonal entry in that row and simplify.
\[
U = \begin{mymatrix}{ccc}
\sqrt{3} & \vspace{0.05in}\frac{1}{3}\sqrt{3} & \vspace{0.05in}\frac{1}{3}\sqrt{3}  \\
0  & \vspace{0.05in}\frac{1}{3}\sqrt{3}\sqrt{11} &  \vspace{0.05in}\frac{5}{33}\sqrt{3}\sqrt{11} \\

0  & 0 & \vspace{0.05in}\frac{1}{11}\sqrt{11}\sqrt{43}
\end{mymatrix}
\]
\end{solution}


\subsection{The Absolute Value}

\begin{definition}{Absolute Value}{Absolute Value}
	\label{def:AbsoluteValue}
The \dfont{absolute value} of a number $x$ is written as $|x|$ and represents
the \ifont{distance} $x$ is from zero. Mathematically, we define it as follows:
$$|x|=\left\{\begin{array}{cl}
x, & \mbox{if $x\geq 0$,}\\
-x, & \mbox{if $x<0$.}\\
\end{array}\right.$$
\end{definition}

Thus, if $x$ is a negative real number, then $-x$ is a positive real number.
The absolute value does \ifont{not} just turn minuses into pluses.
That is, $|2x-1|\neq 2x+1$.
You should be familiar with the following properties.\\

\begin{formulabox}[Absolute Value Properties]
\begin{enumerate}
	\item $|x|\geq 0$.
	\item $|xy|=|x||y|$.
	\item $\displaystyle{\left|\frac{1}{x} \right|=\frac{1}{|x|} }$ \hspace{2mm} when $x\neq 0$.
	\item $|-x|=|x|$.
	\item $|x+y|\leq |x|+|y|$. This is called the \dfont{triangle inequality}.
	\item $\sqrt{x^2}=|x|$.
\end{enumerate}
\end{formulabox}

\begin{example}{$\sqrt{x^2}=|x|$}{sqrtabs}
Observe that $\sqrt{(-3)^2}$ gives an answer of $3$, not $-3$.
\end{example}

When solving inequalities with absolute values, the following are helpful. %(here $a>0$ is a positive number):
%\begin{itemize}
%	\item $|x|=a$ means $x=\pm a$.
%	\item $|x|\leq a$ means $x\geq -a$ \ifont{and} $x\leq a$  (that is, $-a\leq x\leq a$).
%	\item $|x|\geq a$ means $x\leq -a$ \ifont{or} $x\geq a$.
%\end{itemize}
%
%In the above, we do require $a>0$, otherwise we end up with false expressions.
%For example, $|x|=-1$ has no solutions because $|x|$ is always positive (or zero),
%so it will never equal $-1$. On the other hand, $|x|=1$ has the solutions $x=\pm 1$.

\medskip
{\bf Case 1: $a>0$.}\vspace{-0.1cm}
\begin{itemize}\setlength{\itemsep}{0 in}
\item $|x|=a$ has solutions $x=\pm a$.
\item $|x|\leq a$ means $x\geq -a$ {\bf and} $x\leq a$  (that is, $-a\leq x\leq a$).
\item $|x|<a$ means $x< -a$ {\bf and} $x< a$  (that is, $-a< x< a$).
\item $|x|\geq a$ means $x\leq -a$ {\bf or} $x\geq a$.
\item $|x|> a$ means $x< -a$ {\bf or} $x> a$.
\end{itemize}

{\bf Case 2: $a<0$.}\vspace{-0.1cm}
\begin{itemize}\setlength{\itemsep}{0 in}
\item $|x|=a$ has no solutions.
\item Both $|x|\leq a$ and $|x|<a$ have no solutions.
\item Both $|x|\geq a$ and $|x|>a$ have solution set $\{x|x\in\R\}$.
\end{itemize}

{\bf Case 3: $a=0$.}\vspace{-0.1cm}
\begin{itemize}\setlength{\itemsep}{0 in}
\item $|x|=0$ has solution $x=0$.
\item $|x|<0$ has no solutions.
\item $|x|\leq 0$ has solution $x=0$.
\item $|x|>0$ has solution set $\{x\in\R|x\neq 0\}$.
\item $|x|\geq 0$ has solution set $\{x|x\in\R\}$.
\end{itemize}


\section{Dirichlet eigenvalues and eigenfunctions on the annulus \label{sec:annul_dir_exact}}
In this section, we compute some of the Dirichlet eigenvalues
corresponding to a subset of the radially symmetric eigenfunctions
on the annulus.
In polar coordinates $(r,\theta)$, consider the annulus defined by
$R_{1}<r<R_{2}$.
Suppose that $\bu$ is of the form
\begin{equation}
\bu = \nabla^{\perp}\left( \alpha H_{0}(kr) + \beta J_{0}(kr) \right) \, ,
\label{eq:annulus_direig}
\end{equation}
and $p=0$.

Clearly, this pair satisfies the osciallatory Stokes equation
with parameter $k$, since $J_{0}(kr)$ and $H_{0}(kr)$
satisfy the Helmholtz equation on the annulus.

Let $\hat{r},\hat{\theta}$ denote the unit vectors in polar coordinates.
A simple calculation shows that
\begin{equation}
\begin{aligned}
u_{r} &= \bu \cdot \hat{r}  = 0 \, \\
u_{\theta} &= \bu \cdot \hat{\theta} = k(\alpha  H_{0}'(kr) + \beta J_{0}'(kr)) \, .
\end{aligned}
\end{equation}

This in particular implies that on $r=R_{1}$,
$u_{\theta}$ takes on the constant value,
$k(\alpha H_{0}'(kR_{1}) + \beta J_{0}'(kR_{1}))$.
Similarly, on $r=R_{2}$,
$u_{\theta}$ takes on the constant value,
$k(\alpha H_{0}'(kR_{2}) + \beta J_{0}'(kR_{2}))$.

Thus, if $k$ satisfies,
\begin{equation}
H_{0}'(kR_{1}) J_{0}'(kR_{2}) - H_{0}'(kR_{2})J_{0}'(kR_{1}) = 0 \, ,
\end{equation}
and for those values of $k$ if
$\alpha,\beta$ are non-zero solutions to system of equations
\begin{equation}
\begin{bmatrix}
H_{0}'(kR_1) & J_{0}'(kR_{1}) \\
H_{0}'(kR_{2}) & J_{0}'(kR_{2})
\end{bmatrix}
\begin{bmatrix}
\alpha \\
\beta
\end{bmatrix}
=
\begin{bmatrix}
0 \\
0
\end{bmatrix}
\, ,
\end{equation}
then $k$ is a Dirichlet eigenvalue and $\bu$
defined by~\cref{eq:annulus_direig}
is the corresponding eigenfunction.

\section{Neumann eigenvalues and eigenfunctions on the unit disk}
In this section, we derive an analytical expression which
can be used to compute some of the radially symmetric
Neumann eigenvalues on the unit disk for the Stokes operator.

Suppose that $\bu$ is of the form
\begin{equation}
\bu = \nabla^{\perp} \jkr \, ,
\end{equation}
and the pressure is given by $p=0$, as
$\bu$ satisfies $(\Delta + k^2)\bu = 0$.

Then, the surface traction $\bt$ on the disk of radius $r$
is given by
\begin{equation}
\bt =
\left( -\frac{k}{r^2}\jpkr  + \frac{k^2}{r} \jppkr\right)
\begin{bmatrix}
\sint \\
-\cost
\end{bmatrix} \, .
\end{equation}

Thus, $k$ which satisfies
\begin{equation}
  \label{eq:neu_roots}
-k \jpk + k^2 \jppk = 0 \, ,
\end{equation}
are Neumann eigenvalues on the unit disk.
The first $4$ roots of the \cref{eq:neu_roots} are
in \cref{tab:neu_roots}.
\begin{table}
  \centering
  \begin{tabular}{c}
    k  \\ \hline
    5.135622301840683 \\
    8.417244140399865 \\
    11.61984117214906 \\
    14.79595178235126
  \end{tabular}
  \caption{Roots of \cref{eq:neu_roots}.}
  \label{tab:neu_roots}
\end{table}


\lab{Algorithm}{Line Sweep}{Line Sweep}
\label{lab:Alg_Linesweep}

\objective{Learn about and implement a basic line sweep algorithm}

\section*{General Line Sweep Algorithms}

Line sweep algorithms are a significant group of algorithms with a variety of applications.
In the strictest sense, line sweep algorithms typically use a priority queue and binary tree to lower the temporal complexity associated with certain tasks.
Some notable examples include Fortune's Algorithm for calculating Voronoi diagrams and Andrew's Algorithm for computing the convex hull of a set of points.
In this lab, we will explore a basic line sweep algorithm that allows efficient computation of the Closest Pair problem.
Given a set of points, find the distance between the closest pair of points.

\section*{Na\"ive Implementation}

The obvious way of doing this is to simply compute the distance between each point and each of the other points.
Roughly speaking, this requires some constant multiple of $n^2$ computations where $n$ is the number of points.
This can be coded as follows:
\begin{lstlisting}
def reallybadmindist(X):
    r = ((X[0]-X[1])**2).sum()**.5
    for i in range(len(X)):
        for j in range(len(X)):
            if i != j:
                r = min(r,((X[i]-X[j])**2).sum()**.5)
    return r
\end{lstlisting}
Since distance is symmetrical, we can compare each point with the points that follow it in the list we are given (which will reduce
the total number of distances to be computed).
This is a slightly better implementation:
\begin{lstlisting}
def multidist(p0,p1):
    l=len(p0)
    return sum([(p0[i]-p1[i])**2 for i in range(l)])**.5
def badmindist(X):
    l=len(X)
    r=multidist(X[0],X[1])
    for i in xrange(l):
        for j in xrange(i+1,l):
            d=multidist(X[i],X[j])
            if d<r:
                r=d
    return r
\end{lstlisting}

Since, on average, we iterate through about half of the list for each point added, this algorithm will have complexity $O(n^2)$ (we are only scaling by a constant factor).
It is faster than the truly na\"ive method, but essentially just as inefficient.
For very small numbers of points, this algorithm works, but it quickly becomes inefficient as we increase the number of points processed.
What we would like to do is go from $O(n^2)$ to something better.
To do this, we have to think about the problem differently.
This can be done by using one of the key techniques of computational geometry-a line sweep algorithm.

\section*{A Simplified Line Sweep Algorithm}

The concept behind a line sweep algorithm is breaking a global problem into a sequence of smaller problems that can be solved locally.
We sweep the domain space with a line.
This line divides the domain space into an explored region and an unexplored region.
As the line sweeps, events occur, be it whether the line encounters a point in the data set
or some other type of event.
We compute the solution at the very beginning of the line sweep and as new events happen, we compute the change that these events have on our current solution.
We only need to consider the events in the explored region or on the sweep line as they are the only events that affect the current solution.
Sweep line algorithms typically exhibit a temporal complexity of $O(n \log n)$, which is far better than $O(n^2)$.

Line sweep algorithms are good for doing all sorts of proximity based computations.
In the example above, we took advantage of the fact that distance was symmetric ($d(a,b)=d(b,a)$).
In a line sweep version of this algorithm, we also take advantage of the fact that any points further than the current smallest minimum distance along a given axis do not need to be considered.
This is really just an application of the triangle inequality.
The secret to the line sweep solution for the closest pair problem is to actually have two sweep lines with a small distance between them.
As we process the points in our list, we can decrease the distance between the lines so that we only process points that could possibly give a smaller distance than the smallest distance we have already encountered.
By only processing the points between our two sweep lines, we avoid processing many points needlessly.

For a general line sweep algorithm, it is common to use a priority queue to order the items that need to be processed.
A good priority queue can be found in the library \li{Queue} that is included in Python.
In this case all we need to do is sort the points by $X$ value.
This means it will be easier to make a sorted copy of the array using Numpy's built in functions.
Something like \li{X=Y.take(Y[:,0].argsort(),axis=0)} or \li{Y[Y[:,0].argsort()]} will do this for you.

We track the points in the area between the two sweep lines with a list.
Each time we advance the sweep lines, we update this list so it always contains the points between the sweep lines (adding or removing points where necessary).
A point stays in the list if its distance from the leading sweep line is less than the current minimum distance.
Conceptually, this is how we place the second sweep line.

Assuming we have already ordered all our points by $x$ coordinate, we can perform the line sweep algorithm as follows:

\vspace{5mm}
\begin{compactenum}[1.]
\item
Process the first two points and add them to the active list.
Set current minimum distance to the distance between these two points.
\item
Get the next point in the priority queue.
\item
Update the active list so that it contains only the points between the two sweep lines.
\item
Compute the distance between the current point and all the points in the active list.
\item
Compare the smallest distance found with the current minimum distance.
Change the current minimum distance if needed.
\item
Repeat steps 2 to 5 for all remaining points in the priority queue
\end{compactenum}
\vspace{5mm}

When the algorithm ends, the current minimum distance will be the distance between the closest pair of points.

This algorithm can be illustrated as follows:
\begin{figure}[H]
\includegraphics[width = .8\textwidth]{simple0.pdf}
\caption{After processing the first two points, we process the third point and change the current minimum distance accordingly.}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .8\textwidth]{simple1.pdf}
\caption{This change in the minimum thus far is reflected in how we form the actives list for the next point we process.
We remove the points from the top of the queue that are too far away in the $x$ direction to have a distance less than the current minimum.
We then process anything that is left.}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .8\textwidth]{simple5.pdf}
\caption{A few points further in we actually hit the minimum distance.}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .8\textwidth]{simple7.pdf}
\caption{After iterating over all the points in the set, we have the final minimum distance desired.}
\end{figure}

\begin{problem}
Write a Python function that implements the above algorithm.
Have your code accept a metric function as a parameter.
This metric function should take a point and an array of points as arguments and return an array of distances from the point to each of the points in the array.
Test your function's speed.
How does it scale as you increase the number of points?
How does it scale as you increase the number of dimensions?
\end{problem}

\section*{A Line Sweep Algorithm}

You may have noticed that we are not exploiting all the symmetry of the problem in the previous algorithm.
It would likely speed things up if we could also slice away the points that have $y$ values that could not possibly yield the minimum distance.
The simplest way to do this in this case is to take advantage of some algebraic symmetry.
As we compute the distance between two points, we have to calculate the difference between each of the coordinates of the points in question.
If at any time the difference has an absolute value greater than the minimum distance we have encountered thus far, we know we don't need to finish processing the point.
This is, theoretically, like slicing along the other axes in order to further reduce the ``active'' list of points.
A similar result could be obtained by using a list, binary tree, or some other data structure to keep the list of active points sorted by $y$ value, but in this case the insertion and deletion operations involved are more costly.

This algorithm can be illustrated as follows:

Note: in this case we sweep along the $x$ axis from right to left.

\begin{figure}[H]
\includegraphics[width = .8\textwidth]{sweep0.pdf}
\caption{In processing the first point, we find find that we must reduce the radius.
Notice that we only need to consider the points that lie in the box formed by the red an blue lines.}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .8\textwidth]{sweep13.pdf}
\caption{This is where we actually hit the minimum distance.}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .8\textwidth]{sweep22.pdf}
\caption{After processing through all the points, we are guaranteed to have hit the minimum distance already, so our minimum distance thus far is our final return value.}
\end{figure}

Since this version of the algorithm depends so heavily on array lookups, it will be much faster if it is implemented in Cython.

\begin{problem}
Implement the algorithm above in Cython.
Time it against the naive implementation at the beginning of this lab and against the simplified version you coded above.
The new version should be faster, but speed depends heavily on how you have implemented it, so you may have some optimization left to do.
Time how long it takes your new function to process ten million points in two dimensions.
Using benchmarks from the first implementation we gave you, estimate how long (in years) it would take that function to process that many points.
How many times faster is the good implementation?
\end{problem}


\subsection{Multivariate Adaptive Regression Splines (MARS)}

A linear model looks like:

\(\hat y =c+\sum_i x_i\theta_i\)

MARS instead produces a linear model for subsets of X.

\(\hat y =c+\sum_i B_j(x_i,a_j)\theta_i\)

Where:

\begin{itemize}
\item \(B_j=max(0, x_i-a_j)\); or
\item \(B_j=-max(0, a_j-x_i)\)
\end{itemize}

This is trained using a forward pass and a backward pass.

\subsubsection{Forward pass}

\subsubsection{Backward pass}


\section{Types of Transformations}

% Similarity
\subsection{Similarity Transformations}
\label{sec:similaritytrans}

If $\mathbf{T}$ is a nonsingular matrix, then a similarity transformation is defined as:

\begin{equation}
\mathbf{A} = \mathbf{TBT^{-1}}
\end{equation}

And $\mathbf{A}$ and $\mathbf{B}$ are said to be \textit{similar}.

% Affine
\subsection{Affine Transformations}
\label{sec:affine}

Affine transformations are the combination of a linear map and a translation, which has the form $f(\mathbf{x}) = \mathbf{A}\mathbf{x} + \mathbf{b}$.

\begin{equation}
f: V \rightarrow W
\end{equation}

Where $V$ and $W$ are vector spaces. Affine transformations can be expresses as matrices by adding an entry with a constant to the vectors that describe a point in space. For example, for $\mathbf{x} \in \mathbb{R}^n$,  the affine transform $f(\mathbf{x}) = \mathbf{A}\mathbf{x} + \mathbf{b}$ with $A\in\mathbb{R}^{n,n}$ and $x,b \in \mathbb{R}^{n}$ can be expressed as the product of a rectangular matrix $\mathbf{M}$ and a vector $\mathbf{c}$ as:

\begin{equation}
\mathbf{A}\mathbf{x} + \mathbf{b} = \underbrace{\left[\begin{array}{c|c} \mathbf{A} & \mathbf{b} \end{array}\right]}_{\mathbf{M}} \underbrace{\left[\begin{array}{c} \mathbf{x} \\ 1\end{array} \right]}_{\mathbf{c}}
\end{equation}

Where $\mathbf{c}^T = \left[x_1,x_2,x_3,...,x_n,1\right]$ and $\mathbf{M} \in \mathbb{R}^{n,n+1}$.

\subsection{Unitary Transformations}
Unitary transformations are transformations that preserve the inner product, i.e. $\hat{U}x \cdot \hat{U}y = x \cdot y$. As linear transformations, they are represented by unitary matrices (cf. section \ref{sec:unitary}). Unitary transformations include translations, reflections and rotations.

\subsection{Multilinear Maps}

A multilinear map acts on several vectors in a way that is linear in each of its arguments. A $k$-linear map acts on $k$ vectors, where $k=2$ are bilinear maps and $k=1$ are linear maps.

\begin{equation}
f: V_1 \times V_2 \times ... \times V_n \rightarrow W
\end{equation}

Where $V_1, V_2, ... , V_n$ and $W$ are vector spaces. An example would be the addition or subtraction of two or more vectors.

\subsection{Multilinear Forms}
Multilinear forms are multilinear maps that have a scalar output. An example is the dot product between two vectors, or summing over the elements of one or more vectors.

\begin{equation}
f: V_1 \times V_2 \times ... \times V_n \rightarrow K
\end{equation}

Where $V_1, V_2, ... , V_n$ and $K$ is a scalar field.


\chapter{k-Means Clustering}
\label{ch:kmeans-clustering}

\newthought{Hierarchical clustering is not suitable for larger data sets} due to the prohibitive size of the distance matrix: with 30 thousand objects, the distance matrix already has almost one billion elements. An alternative approach that avoids using the distance matrix is k-means clustering.

K-means clustering randomly selects \textit{k} centers (with k specified in advance). Then it alternates between two steps. In one step, it assigns each point to its closest center, thus forming k clusters. In the other, it recomputes the centers of the clusters. Repeating these two steps typically converges quite fast; even for big data sets with millions of data points it usually takes just a couple of ten or hundred iterations.

Orange's Educational add-on provides a widget \widget{Interactive k-Means}, which illustrates the algorithm.

Use the \widget{Paint Data} widget to paint some data - maybe five groups of points. Feed it to Interactive k-means and set the number of centroids to 5. You may get something like this.

\marginnote{Try rerunning the clustering from new random positions and observe how the centers conquer the territory. Exciting, isn't it?}

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{interactive-kmeans.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure}

Keep pressing Recompute Centroids and Reassign Membership until the plot stops changing. With this simple, two-dimensional data it will take just a few iterations; with more points and features, it can take longer, but the principle is the same.

How do we set the initial number of clusters? That's simple: we choose the number that gives the optimal clustering.

% This should likely go out as this is explained in silhouette.
Well then, how do we define the optimal clustering? This one is a bit harder. We want small distances between points in the same cluster and large distances between points from different clusters. Pick one point, and let A be its average distance to the data points in the same cluster and let B represent the average distance to the points from the closest other cluster. (The closest cluster? Just compute B for all other clusters and take the lowest value.) The value (B - A) / max(A, B) is called silhouette; the higher the silhouette, the better the point fits into its cluster. The average silhouette across all points is the silhouette of the clustering. The higher the silhouette, the better the clustering.

Now that we can assess the quality of clustering, we can run k-means with different values of parameter k (number of clusters) and select k which gives the largest silhouette.

For this, we abandon our educational toy and connect Paint Data to the widget k-Means. We tell it to find the optimal number of clusters between 2 and 8, as scored by the Silhouette.

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{kmeans-silhouette.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure}

\newpage

Works like a charm.

Except that it often doesn't. First, the result of k-means clustering depends on the initial selection of centers. With unfortunate selection, it may get stuck in a local optimum. We solve this by re-running the clustering multiple times from random positions and using the best result. Second, the silhouette sometimes fails to correctly evaluate the clustering. Nobody's perfect.

Time to experiment. Connect the Scatter Plot to k-Means. Change the number of clusters. See if the clusters make sense. Could you paint the data where k-Means fails? Or where it works really well?

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{smiley-face.png}
    \caption{$\;$} % empty caption for proper pagesetting
\end{figure}


\subsection{Paraboloids}
\noindent
The paraboloid looks like a parabola that has been rotated and extruded about its axis of symmetry.
It is radially symmetric, and its level curves are circles.
Paraboloids have the form $z = ax^2 + by^2$ where $a,b \in \mathbb{R}$.

\begin{figure}[H]
	\centering
	\includegraphics[width = 0.3\textwidth]{./Images/differentialMultivariableCalculus/paraboloid.png}
	\caption{A paraboloid}
\end{figure}


\section{Exercises in Limit Laws of Statistics}\label{S:xsLimitLaws}% {S:xsCFs}% {S:xsExpectationsOfRVs} %S:xsMultivariateRVs

\begin{ExerciseList}

\Exercise
Suppose you plan to obtain a simple random sequence (SRS) --- also known as independent and identically distributed (IID) sequence --- of $n$ measurements from an instrument.
This instrument has been calibrated so that the distribution of measurements made with it have population variance of $1/4$.
Your boss wants you to make a point estimate of the unknown population mean from a SRS of sample size $n$.
He also insists that the tolerance for error has to be $1/10$ and the probability of meeting this tolerance should be just above $95\%$.
Use CLT to find how large should $n$ be to meet the specifications of your boss.
\Answer
~\\
%Using the CLT implied Equation (61) in the lecture notes and further noting from the Standard Normal Table that when $z_{\alpha/2}=2$ we get the desired $1-\alpha=0.9772>0.95$ we get
%\[
%n = \left( \sqrt{(V(X_1)} z_{\alpha/2}) / \epsilon \right)^2 = \left( (\sqrt{1/4}\times 2) / (1/10) \right)^2
%= \left( ((1/2)\times 2) / (1/10) \right)^2 = 10^2 = 100
%\]
We want $1-\alpha = 0.95$, and from the standard Normal Table we know that the corresponding $z_{\alpha/2}=1.96$.
Then we can get the right sample size $n$ from the CLT implied Equation (61) in the lecture notes, which is,
\[
n = \left( \sqrt{(V(X_1)} z_{\alpha/2}) / \epsilon \right)^2 \enspace ,
\]
as follows:
\begin{eqnarray*}
n &= \left( \sqrt{(V(X_1)} z_{\alpha/2}) / \epsilon \right)^2 = \left( (\sqrt{1/4}\times 1.96) / (1/10) \right)^2\\
&= \left( ((1/2)\times 1.96) / (1/10) \right)^2 = (0.98 \times 10)^2 = 9.8^2 = 96.04
\end{eqnarray*}
Finally, by rounding $96.04$ up to the next largest integer we need $n=97$ measurements to meet the specifications of your boss (at least up to the approximation provided by the CLT).

\Exercise
Suppose the collection of RVs $X_1,X_2, \ldots, X_n$ model the number of errors in $n$ computer programs named $1,2,\ldots,n$, respectively.  Suppose that the RV $X_i$ modeling the number of errors in the $i$-th program is the $\textrm{Poisson}(\lambda=5)$ for any $i=1,2,\ldots,n$.  Further suppose that they are independently distributed.  Succinctly, we suppose that
\[
X_1,X_2,\ldots,X_n \overset{\IID}{\sim} \poisson(\lambda=5) \ .
\]
Suppose we have $n=125$ programs and want to make a probability statement about $\overline{X}_{125}$ which is the average error per program out of these $125$ programs.  Since $E(X_i) = \lambda=5$ and $V(X_i)=\lambda=5$, we want to know how often our sample mean $\overline{X}_{125}$ differs from the expectation of $5$ errors per program.
Using the CLT find the $P(\overline{X}_{125} < 5.5)$.
\Answer
~\\
By CLT, $\frac{\sqrt{n}(\overline{X}_n - \E(X_1))}{\sqrt{\V(X_1)}} \rightsquigarrow Z \sim \mathrm{Normal}(0,1)$.
So we need to apply the ``standardization'' to both sides of the inequality that is defining the event of interest:
$$\{\overline{X}_n < 5.5\} \enspace ,$$
in order to find its probability $\P(\overline{X}_n < 5.5)$.
~\\
\begin{eqnarray}
\P(\overline{X}_n < 5.5)
&=& P \left( \frac{\sqrt{n}(\overline{X}_n - \E(X_1))}{\sqrt{\V(X_1)}} < \frac{\sqrt{n}(5.5-\E(X_1))}{\sqrt{\V(X_1)}} \right) \notag \\
&\approxeq& P \left( Z < \frac{\sqrt{n}(5.5-\lambda)}{\sqrt{\lambda}} \right) \qquad \text{{\scriptsize [
since we know/assume that $\E(X_1)=\V(X_1)=\lambda$]}} \notag \\
&=& P \left( Z < \frac{\sqrt{125}(5.5-5)}{\sqrt{5}} \right) \qquad \text{{\scriptsize [Since, $\lambda=5$ and $n=125$ in this Example]}} \notag \\
&=& \P(Z \leq 2.5) = \Phi(2.5) = 0.9938\ . \notag
\end{eqnarray}
(source: Wasserman, {\em All of Statistics}, Springer, p.~78, 2003)

\Exercise
What is the distribution of $\sum_{i=1}^n{X_i}/n$ as $n \to \infty$ when $X_i \overset{IID}{\sim} \uniform(-10,10)$?
\Answer
HINT: Use the LLN after finding the population mean of $X_i$.
\Exercise
What is the distribution of $\sum_{i=1}^n{X_i}/\sqrt{\V(X_i)n}$ as $n \to \infty$ when $X_i \overset{IID}{\sim} \uniform(-10,10)$?
\Answer
HINT: Use the CLT after finding the population mean and variance of $X_i$.
\end{ExerciseList}