\documentclass[11pt]{article}

\usepackage{parskip}
\usepackage{geometry}
\usepackage[utf8]{inputenc}
\usepackage{amsmath, amssymb}
\usepackage{subcaption}
\usepackage{hyperref}

\geometry{
  a4paper,
  bottom=2.5cm,
  right =2.5cm,
  left  =2.5cm,
  top   =2.5cm,
}

\begin{document}
\title{t-SNE Notes}
\author{Pavlin Poličar}
\date{}
\maketitle

\section{t-SNE}

t-SNE was presented in \cite{maaten2008visualizing} and aims to preserve local structure of high dimensional spaces $X$ with some low dimensional embedding $Y$. First for each point $i$, we find its nearest nearest neighbours and compute the probability of this point $p_j$ based on the PDF of a Gaussian centred on the point $i$:

\begin{equation}\label{eq:sne_pij}
p_{j \mid i} = \frac{\exp{\left (- || \mathbf{x}_i - \mathbf{x}_j ||^2 / 2\sigma_i^2 \right )}}{\sum_{k \neq i}\exp{\left (- || \mathbf{x}_i - \mathbf{x}_k ||^2 / 2\sigma_i^2 \right )}}
\end{equation}

where $\sigma_i$ is the bandwidth of the Gaussian density. These bandwidths are controlled by the ``perplexity'' parameter. Perplexity can be thought of as a continuous analogue to the number $k$-nearest neighbours:

\begin{equation}
Perp(P_i) = 2^{H(P_i)}
\end{equation}

where $H(P_i)$ is the Shannon entropy of the distribution $P_i$.

t-SNE actually doesn't use Equation~\ref{eq:sne_pij} directly, but symmetrizes this conditional probability, so the actual $p_{ij}$s used by t-SNE are

\begin{equation}
p_{ij} = \frac{p_{j\mid i} + p_{i \mid j}}{2}
\end{equation}

In their experiments, van der Maaten et al. found that this doesn't affect embedding quality and simplifies the gradient expression.

Similarly, we represent the embedding $Y$ as a probability distribution. In the original SNE paper \cite{hinton2003stochastic}, a Gaussian was used, however this often led to the crowding problem, where all the points were clumped into a single ball in a single point in space. t-SNE, as the name would suggest, uses a Student-t distribution, therefore the probability density of $Y$ is

\begin{equation}
q_{ij} = \frac{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{-1}}{\sum_{k \neq l}\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )^{-1}}
\end{equation}

We now have two probability distributions over the local point affinities. Now we'd like some way to match these two distributions, so the local structure of $X$ is reflected in $Y$. A natural way of doing this is to use Kullback-Leibler divergence (from here on referred to as the KL divergence), which is defined as

\begin{equation}
KL(P \mid \mid Q) = \sum_{ij} p_{ij} \log \frac{p_{ij}}{q_{ij}}
\end{equation}

Our goal is to minimize this error $C$, so we can take the derivative and obtain

\begin{equation}
\frac{\partial C}{\partial \mathbf{y}_i} = 4 \sum_{j \neq i} \left ( p_{ij} - q_{ij} \right ) \left ( \mathbf{y}_i - \mathbf{y}_j \right ) \left ( 1 + || \mathbf{y}_i - \mathbf{y}_j || ^2 \right )^{-1}
\end{equation}

This is t-SNE in essence. In practice various tricks are used to speed up convergence e.g. using a momentum term helps a lot. The embedding $Y$ is typically initialized using an isotropic Gaussian with small variance (e.g. 0.01). Often times, PCA is used for initialization. This can sometimes be problematic if the PCA embedding provides very scattered embeddings (sometimes most points are clumped to one side with very long stretched out tails). In these cases, using a random initialization produces better embeddings.

\section{Performance improvements}
It quickly became apparent that t-SNE, while nice, was infeasible to run for larger data sets, because of its quadratic time complexity $\mathcal{O}(n^2)$ (due to the normalization term in $q_{ij}$).

For convenience, we will write the gradient in a different form seen in many papers, that makes the attractive and repulsive forces clearer.

\begin{align}
\frac{\partial C}{\partial \mathbf{y}_i} &= 4 \sum_{j \neq i} \left ( p_{ij} - q_{ij} \right ) \left ( \mathbf{y}_i - \mathbf{y}_j \right ) \left ( 1 + || \mathbf{y}_i - \mathbf{y}_j || ^2 \right )^{-1}
\intertext{Notice that the right most term is just the unnormalized $q_{ij}$}
&= 4 \sum_{j \neq i} \left ( p_{ij} - q_{ij} \right ) \left ( \mathbf{y}_i - \mathbf{y}_j \right ) \left ( 1 + || \mathbf{y}_i - \mathbf{y}_j || ^2 \right )^{-1} \frac{Z}{Z}
\intertext{Where $Z$ is the normalization term of $Q$: $Z = \sum_{k \neq l}\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )^{-1}$}
&= 4 \sum_{j \neq i} \left ( p_{ij} - q_{ij} \right ) q_{ij} Z \left ( \mathbf{y}_i - \mathbf{y}_j \right ) \\
&= 4 \left (\sum_{j \neq i} p_{ij} q_{ij} Z \left ( \mathbf{y}_i - \mathbf{y}_j \right ) -\sum_{j \neq i} q_{ij}^2 Z \left ( \mathbf{y}_i - \mathbf{y}_j \right ) \right ) \\
\intertext{which can in turn be throught of as attractive and repulsive forces}
&= 4 \left ( F_{\text{attr}} + F_{\text{rep}} \right )
\end{align}

\subsection{Landmark points}
In fact, van der Maaten and Hinton provide a solution to this in their original paper: instead of visualizing all the points, embed only a sample of carefully chosen landmark points. The points must be carefully chosen because a random subset may not properly describe the manifold. First, we construct the k-neighbourhood graph on all the points. Next, they approximate the $P$ of the landmark points using random walks across the neighbourhood graph. Then, we proceed with t-SNE on the landmark points.

\subsection{Approximating P}

An observation made in \cite{van2014accelerating} was that since we use a Gaussian kernel for $P$, points further than 3 standard deviations from the mean have almost zero probabilities, and as such, do not affect the KL divergence term. Therefore, no harm would come if we simply ignored these terms. In practice, this means that we only compute the $p_{ij}$ terms for $\left \lfloor 3u \right \rfloor$ neighbours, where $u$ is the perplexity.

In \cite{van2014accelerating}, exact nearest neighbours are used. These can be efficiently computed in $\mathcal{O}(n \log n)$ using tree structures, thus reducing the complexity from $\mathcal{O}(n^2)$ needed for pairwise distances.

The preferred exact nearest neighbour method are vantage point trees (also referred to as VP trees). \cite{yianilos1993data} presented VP trees and compared their performance to another popular tree based nearest neighbour search method -- KD trees. VP trees were shown to require far fewer queries when dealing with high dimensions, as t-SNE often does.\cite{van2014accelerating} also provide a comparison with dual-trees, where VP trees, again, perform favourably.

More recently, it was shown in \cite{linderman2017efficient} that approximate nearest neighbours perform just as well. Approximate nearest neighbour algorithms are often orders of magnitude faster than exact nearest neighbour search, allowing us to scale this step to much larger data than before.

\subsection{Barnes-Hut}

Having drastically improved the complexity of $F_\text{attr}$, we are still left with quadratic $\mathcal{O}(n^2)$ complexity for $F_\text{rep}$, required by the normalization term $Z$.

\cite{van2014accelerating} notice that computing $F_\text{rep}$ can be posed as an N-body simulation problem. This problem has been addressed physics simulation community and can be efficiently solved in $\mathcal{O}(n \log n)$ time using Barnes-Hut trees. The main idea behind this approximation is that clusters of points far away for the current point $i$ will have similar contribution, therefore we can summarize entire regions of space (denoted cells in the following) by computing the center of mass of the region $\mathbf{y}_{\text{cell}}$, computing the interaction between $i$ and $\mathbf{y}_{\text{cell}}$ and adding this interaction up $N_{\text{cell}}$ times, where $N_{\text{cell}}$ is the number of points in the given region, given they are far enough from our query point $i$. The space is split into square regions and represented by a space splitting tree (a quad-tree in 2D and an oct-tree in 3D) which can be built in linear time.

The ``far enough'' is determined by a parameter $\theta$, which controls how accurate our estimations are. If the following relation holds, then the cell is summarized

\begin{equation}
\frac{r_\text{cell}}{|| \mathbf{y}_i - \mathbf{y}_{\text{cell}} || ^2} < \theta
\end{equation}

where $r_\text{cell}$ represents the length of the diagonal of the cell. Larger values of $\theta$ produce more accurate estimates. Setting $\theta$ to 0 computes all the pairwise interactions as the condition can never be met. Scikit-learn recommends values between 0.2 and 0.8, as anything above and below that quickly result in long computation time and large error,  respectively.

It is worth noting that this approach scales fairly well for 1, 2 and 3 dimensions, but further than that, the complexity becomes prohibitively expensive. This is not really an issue, since we humans can only perceive 3 dimensions, and most visualizations are 2D.

\subsection{FFT Accelerated Interpolation}

We can write an equivalent expression for the repulsive forces

\begin{align}
F_\text{rep} &= \sum_{j \neq i} q_{ij}^2 Z \left ( \mathbf{y}_i - \mathbf{y}_j \right )
\intertext{Plugging in the expressions for $q_{ij}$ and $Z$}
&= \sum_{j \neq i} \frac{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{-2}}{\sum_{k \neq l}\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )^{-2}} \frac{\left ( \mathbf{y}_i - \mathbf{y}_j \right )}{\sum_{k \neq l}\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )} \\
\intertext{Putting the top and bottom terms together}
&= \left ( \sum_{j \neq i} \frac{\mathbf{y}_i - \mathbf{y}_j}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{2}} \right )
\bigg/
\left( \sum_{k \neq l} \frac{1 + || \mathbf{y}_k - \mathbf{y}_l ||^2}{\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )^{2}} \right) \\
&= \left ( \sum_{j \neq i} \frac{\mathbf{y}_i - \mathbf{y}_j}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{2}} \right )
\bigg/
\left( \sum_{k \neq l} \frac{1}{\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )} \right)
\end{align}

We can also write an expression for each term of $\mathbf{y}_i$ individually:
\begin{equation}
F_{\text{rep}, i}(m) = \left ( \sum_{j \neq i} \frac{\mathbf{y}_i(m) - \mathbf{y}_j(m)}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{2}} \right )
\bigg/
\left( \sum_{k \neq l} \frac{1}{\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )} \right)
\end{equation}

where $\mathbf{y}_i(m)$ denotes the $m^{\text{th}}$ component of $\mathbf{y}$ i.e. $m \in \{1, 2\}$ in the 2D case.

\cite{linderman2017efficient} make the acute observation that the repulsive forces $F_\text{rep}$ can be written as $s + 2$ sums of the form

\begin{equation}
\phi(\mathbf{y}_i) = \sum_j K (\mathbf{y}_i, \mathbf{y}_j) q_{ij}
\end{equation}

where $K(y, z)$ is either the Cauchy kernel or the squared Cauchy kernel and $s$ is the dimensionality of $Y$

\begin{equation}
K_1(y, z) = \frac{1}{\left( 1 + || \mathbf{y} - \mathbf{z} ||^2 \right)}, \quad\text{or}\quad K_2(y, z) = \frac{1}{ \left( 1 + || \mathbf{y} - \mathbf{z} ||^2 \right) ^2}
\end{equation}

To make the sums concrete, consider the 2D case:

\begin{align}
\phi_{1, i} &= \sum_{j \neq i} \frac{1}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 \right)} \notag \\
\phi_{2, i} &= \sum_{j \neq i} \frac{\mathbf{y}_j(1)}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 \right)^2} \notag \\
\phi_{3, i} &= \sum_{j \neq i} \frac{\mathbf{y}_j(2)}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 \right)^2} \notag \\
\phi_{4, i} &= \sum_{j \neq i} \frac{1}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 \right)^2} \notag
\end{align}

the the repulsive forces can be expressed in terms of these 4 sums as follows:

\begin{align}
F_{\text{rep}, i}(1) &= \left ( \sum_{j \neq i} \frac{\mathbf{y}_i(1) - \mathbf{y}_j(1)}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{2}} \right )
\bigg/
\left( \sum_{k \neq l} \frac{1}{\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )} \right) \notag \\
&= (\phi_{2, i} - \mathbf{y}_{i}(1)\phi_{4, i}) / Z, \\
F_{\text{rep}, i}(2) &= \left ( \sum_{j \neq i} \frac{\mathbf{y}_i(2) - \mathbf{y}_j(2)}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{2}} \right )
\bigg/
\left( \sum_{k \neq l} \frac{1}{\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 \right )} \right) \notag \\
&= (\phi_{3, i} - \mathbf{y}_{i}(2)\phi_{4, i}) / Z,
\end{align}

where

\begin{align}
Z &= \sum_j \phi_{1, j}
\end{align}

The key idea in this approach is that since we have smooth kernels $K_1$ and $K_2$, we can approximate them using polynomial interpolation. Of course, the choice of interpolants is entirely up to us, but we we evaluate our kernel functions at these points and interpolate our true data using these. To make things computationally efficient, we can set the interpolants to be equispaced points on the space spanned by the data.

This is very convenient, because the kernels in question are all translation invariant and when we evaluate them at the interpolants, then the kernel matrix $K$ will be Toeplitz. This means that it is enough to evaluate the Kernel for the left-most point in space in 1D. In 2d, our $K$ is actually a 3D tensor, but is again, Toeplitz.

Linear algebra tells us we can embed any Toeplitz matrix into a circulant matrix. This is desirable, because now we can perform matrix-vector multiplication in the frequency domain in linear time, with the slowest part being the FFT and IFFT transforms in $\mathcal{O}(n \log n)$ time.

Finally, having evaluated the repulsive forces at the interpolants, we just need to interpolate the forces on our true data. This can be done in linear time $\mathcal{O}(n)$.

Doing this, we have successfully made the overall complexity independent of $N$, and have shifted the brunt of the work onto the number of chosen interpolation points, so the time complexity will rely heavily on that. In practice, we split the input space into equally sized intervals, and then have 3 interpolation points in each interval. While we could increase the number of interpolation points, it is preferable to increase the number of intervals (due to the Runge phenomenon in interpolation). Increasing the number of interpolation points also increases the accuracy of the approximation, but comes at a computation cost.

Like the Barnes-Hut variant, this method becomes very inefficient for higher dimensions, as the number of interpolation points needed scales exponentially with $d$. In practice, this isn't an issue because most often, we want to inspect 2D embeddings.

\section{Implementation details}

\subsection{Perplexity}

The following section explains how perplexity is formulated so the code can run efficiently. Perplexity is defined as

\begin{align}
\text{Perplexity}(P_i) &= 2^{H(P_i)}
\intertext{where $H$ is the Shannon entropy of a discrete distribution}
H(P_i) = -\sum_i p_{j \mid i} \log_2 (p_{j \mid i})
\intertext{In code, the following is more practical to avoid computing $2^{x}$ whereas perplexity stays fixed:}
\log(\text{Perplexity}(P_i)) &= -\sum_i p_{j \mid i} \log (p_{j \mid i})
\end{align}

Remember that $P_i$ is just a Gaussian distribution centered on point $i$, given by
\begin{align}
p_i(d_i) &= \frac{1}{\sqrt{2 \pi} \sigma} \exp \left ( -\frac{d_{ij}^2}{2 \sigma^2} \right )
\intertext{however, since we'll be performing row-normalization by hand, something proportional is sufficient}
&\sim \exp \left ( -\frac{d_{ij}^2}{2 \sigma^2} \right )
\end{align}

In most implementations this Gaussian is parameterized with $\beta = 1 / 2\sigma^2$ and therefore we compute $\exp \left ( -d_{ij}^2 \beta \right )$ in practice. In our case, we actually compute $ \frac{1}{\sigma} \exp \left ( -d_{ij}^2 \beta \right )$ because we allow a multiscale approach, which mixes several Gaussians together. We also reparameterize our distribution to use the more interpretable precision $\tau = 1 / \sigma^2$ instead of $\beta$. Therefore our probability density is given by

\begin{equation}
p_i(d_i) \sim \sqrt{\tau} \exp \left ( -\frac{d_{ij}^2 \tau}{2} \right )
\end{equation}

We now plug in our parametrization into the entropy and arrive at a convenient form which can be coded efficiently.

\begin{align}
H_i &= -\sum_j \frac{\sqrt{\tau} \exp \left ( -d_{ij}^2 \tau / 2 \right ) }{\sum_k \sqrt{\tau} \exp \left ( -d_{ik}^2 \tau / 2 \right )} \log \left ( \frac{\sqrt{\tau} \exp \left ( -d_{ij}^2 \tau / 2 \right ) }{\sum_k \sqrt{\tau} \exp \left ( -d_{ik}^2 \tau / 2 \right )} \right ) \\
\intertext{The first term is just $p_{j\mid i}$ and we can split up the log into two parts}
&= -\sum_j p_{j\mid i} \left [ \log \left ( \sqrt{\tau} \exp \left ( -d_{ij}^2 \tau / 2 \right ) \right ) - \log \left ( \sum_k \sqrt{\tau} \exp \left ( -d_{ik}^2 \tau / 2 \right ) \right ) \right ]
\intertext{Notice now that the first term in the square brackets almost has the form $\log (\exp (x))$. For clarity, we will also denote the normalization sum as $Z$.}
&= -\sum_j \left [ p_{j\mid i} \left ( \frac{1}{2} \log \tau - d_{ij}^2 \tau / 2 \right ) \right ] + \sum_j p_{j\mid i} \log Z \\
&= -\frac{1}{2} \log \tau \sum_j p_{j\mid i} + \frac{\tau}{2} \sum_j p_{j\mid i} d_{ij}^2 + \sum_j p_{j\mid i} \log Z
\intertext{We move the first term to the end to make the sign unmissable. Since $p_i$ is a proper probability distribution, its elements sum up to 1, leaving us with}
&= \frac{\tau}{2} \sum_j p_{j\mid i} d_{ij}^2 + \log Z-\frac{1}{2} \log \tau
\end{align}

This can be computed in two passes over the data. The first pass computes the unnormalized probabilities $\tilde{p}_{j\mid i}$ and accumulate the normalization constant $Z$. In the second pass, the first term can be computed.

In other implementation e.g. scikit-learn, the expression is computed without $\sqrt{\tau}$. It's easy to see that the result will be similar (and indeed, this is used in their code), but without the $-1/2 \log \tau$ term and parameterized with $\beta = \tau / 2$.

\subsection{Fast KL Divergence}

During computation of negative gradients, we do not know the value of the normalization term $Z$ during intermediate steps. Therefore, in order to compute the KL divergence of the embedding, we would need at least two passes over the data points, first to compute the unnormalized $q_{ij}$s, and secondly to normalize them and compute the KL divergence. By rewriting the KL divergence in terms of unnormalized $q_{ij}$s, we can compute the entire error with a single pass over the data points by accumulating the $\sum_{ij} p_{ij}$ and $\sum_{ij}q_{ij}$ in the first pass.

\begin{align}
KL(P \mid \mid Q) &= \sum_{ij} p_{ij} \log \frac{p_{ij}}{q_{ij}} \\
&= \sum_{ij} p_{ij} \log \left ( p_{ij} \frac{Z}{\hat{q}_{ij}} \right )
\intertext{where $\hat{q}_{ij}$ denotes the unnormalized values $q_{ij}$}
&= \sum_{ij} p_{ij} \log \frac{p_{ij}}{\hat{q}_{ij}} + \sum_{ij} p_{ij} \log Z
\end{align}

Therefore the first term requires a single pass over all $i, j$s and the second term can be computed in constant time if we accumulate the sums of $P$ and $Q$.

This is already included in most software packages e.g. scikit-learn.

\subsection{KL Divergence with exaggeration}

The implemented optimization methods don't have a notion of exaggeration, they simply take an affinity matrix $P$ containing the probabilities of points $j$ appearing close to $i$. Exaggeration is used to scale $P$ by some constant factor $\alpha$ (this means that entries in the affinity matrix $P$ are not proper probabilities) to help separate clusters in the beginning of the optimization. These methods also compute the KL divergence during optimization (for efficiency), and, as such, the error is incorrect because we don't account for the scaling $\alpha$.

This section derives a simple correction for the KL divergence error term so we can get the true error of the embedding even when $P$ is exaggerated.

\begin{align}
KL(P \mid \mid Q) &= \sum_{ij} p_{ij} \log \frac{p_{ij}}{q_{ij}}
\intertext{We need to introduce the scaling i.e. exaggeration factor $\alpha$ to every $p_{ij}$ term, so we multiply some terms by $1 = \alpha/\alpha$.}
&= \sum_{ij} \frac{\alpha}{\alpha}p_{ij} \log \frac{\alpha p_{ij}}{\alpha q_{ij}} \\
\intertext{Exaggeration means that the $p_{ij}$ terms get multiplied by $\alpha$, so we need to find an expression for the KL divergence that includes only $\alpha p_{ij}$ and $q_{ij}$ and some other factor that will correct for $\alpha$.}
&= \frac{1}{\alpha} \sum_{ij} \alpha p_{ij} \left ( \log \frac{\alpha p_{ij}}{q_{ij}} - \log \alpha \right ) \\
&= \frac{1}{\alpha} \left ( \sum_{ij} \alpha p_{ij} \log \frac{\alpha p_{ij}}{q_{ij}} \right ) - \frac{1}{\alpha} \left ( \sum_{ij} \alpha p_{ij} \log \alpha \right )
\intertext{We notice in the first term is exactly the KL divergence where $p_{ij}$s are scaled by $\alpha$. We also notice in the second term that $\sum_{ij} P_{ij} = 1$ and that $\alpha$ cancels out, leaving us with}
&= \frac{1}{\alpha} \left ( \sum_{ij} \alpha p_{ij} \log \frac{\alpha p_{ij}}{q_{ij}} \right ) - \log \alpha
\end{align}

The first term is computed by the gradient method (since it only knows about the scaled $P$), the second term can easily be computed post-optimization, allowing us to get the correct KL divergence.

\subsection{Variable Degrees of Freedom}

Kobak \textit{et al.}~\cite{kobak2019heavy} suggest that using variable degrees of freedom can be used to improve embeddings.

Standard t-SNE uses the t-distribution with a single degree of freedom. This is defined as

\begin{equation}
q_{ij} \propto \left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 / \alpha \right )^{-\alpha} = \frac{1}{\left( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 / \alpha \right)^\alpha }.
\end{equation}

In standard t-SNE $\alpha=1$ so this simplifies to the standard formulation
\begin{equation}
q_{ij} \propto \left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 \right )^{-1} = \frac{1}{1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 }
\end{equation}
where we have omitted the normalization constant.

The gradient of the t-SNE loss function then becomes
\begin{align}
\frac{\partial C}{\partial \mathbf{y}_i} &= 4 \sum_{j \neq i} \left ( p_{ij} - q_{ij} \right ) q_{ij}^{1/\alpha} \left ( \mathbf{y}_i - \mathbf{y}_j \right )
\end{align}
where $q_{ij}$ is, again, the unnormalized kernel between points $i$ and $j$.

Decomposing this into the attractive and repulsive forces gives us
\begin{align}
\mathbf{F}_{\text{attr}} &= 4 \sum_j p_{ij} q_{ij}^{1/\alpha} (\mathbf{y}_i - \mathbf{y}_j), \\
\mathbf{F}_{\text{rep}} &= - 4 \sum_j q_{ij}^{\frac{\alpha+1}{\alpha}} / Z (\mathbf{y}_i - \mathbf{y}_j).
\end{align}

See the original publication for more details.

\subsubsection{Implementation}

Adapting the implementation for computing the attractive forces and the Barnes-Hut repulsive forces is straightforward. Adapting the interpolation based computation of repulsive forces is a bit more involved.

The direct implementation of the approach described in the paper leads to a solution requiring two different kernels. We describe the 1D case, but the extension to the 2D case is straightforward.

\begin{align}
\mathbf{F}_\text{rep} &= \sum_{j \neq i} q_{ij}^{\frac{\alpha+1}{\alpha}} Z \left ( \mathbf{y}_i - \mathbf{y}_j \right ) \\
&=
\sum_{j \neq i} \left( \frac{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 / \alpha \right )^{-\alpha}}{\sum_{k \neq l} \left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 / \alpha \right )^{-\alpha}} \right) ^{\frac{\alpha+1}{\alpha}}
\frac{\mathbf{y}_i - \mathbf{y}_j}{\sum_{k \neq l}\left ( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 / \alpha \right )^{\alpha}} \\
&=
\sum_{j \neq i} \frac{\mathbf{y}_i - \mathbf{y}_j}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 / \alpha \right )^{\alpha \left( \frac{\alpha+1}{\alpha} \right)}}
\bigg/
\left( \sum_{k \neq l} \frac{\left( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 / \alpha \right)^\alpha}{\left( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 / \alpha \right)^{\alpha \left(\frac{\alpha+1}{\alpha} \right) }} \right) \\
&=
\sum_{j \neq i} \frac{\mathbf{y}_i - \mathbf{y}_j}{\left ( 1 + || \mathbf{y}_i - \mathbf{y}_j ||^2 / \alpha \right )^{\alpha + 1}}
\bigg/
\left( \sum_{k \neq l} \frac{1}{\left( 1 + || \mathbf{y}_k - \mathbf{y}_l ||^2 / \alpha \right)} \right)
\end{align}

Evaluating this sum using the interpolation scheme would require two separate kernels with three terms
\begin{align}
\phi_{1,j} &= \sum_{j \neq i} \frac{1}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 / \alpha \right)^{\alpha+1}}, \\
\phi_{2,j} &= \sum_{j \neq i} \frac{\mathbf{y}_j}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 / \alpha \right)^{\alpha+1}}, \\
\phi_{3,j} &= \sum_{j \neq i} \frac{1}{\left( 1 + || \mathbf{y}_j - \mathbf{y}_i ||^2 / \alpha \right)}.
\end{align}
Then, we can calculate the necessary quantities
\begin{align}
N_i &= \mathbf{y}_i \phi_{1,j} - \phi_{2,j} \\
Z &= \sum_j \phi_{3,j}
\end{align}
where $N_i$ is the unnormalized numerator of the repulsive forces.

%
\section{Transform}

\subsection{Direct optimization}

\subsection{General framework of cost functions}
\cite{bunte2012general}

\subsection{MDS interpolation}
MDS Interpolation~\cite{bae2010dimension}. A similar approach might be able to be applied to t-SNE. In essence, they run MDS on a sample of points. Then for each new point, we compute the k-nearest neighbours and optimize the stress function w.r.t. only those points. In their paper, they derive equations that can be used for efficient optimization via majorization.

\subsection{Kernel t-SNE}
\cite{gisbrecht2012out} claim to outperform direct mapping t-SNE using a direct kernel mapping. This paper is not very useful. The graph is misleading and the table at the end is informative, but run only on small datasets. Their subsequent paper is much better and throughout.

In \cite{gisbrecht2015parametric}, kernel t-SNE is described in more detail and parameters are chosen in a more principled manner.

Describes how to integrate class labels into embedding using Fischer information.

The issue of kernel t-SNE is that we have to compute the inverse of the interaction matrix K. We can use P as the interaction matrix, and P is sparse, but the inverse of that is very dense, and for any reasonably sized data set, this is unfeasable.

\bibliography{references}
\bibliographystyle{apalike}

\end{document}


\subsection{Own-price elasticity of demand}

We have our Marshallian demand function:

\(x_i=x_{di}(I, \mathbf p)\)

The derivative of this with respect to price is the additional amount consumed after prices increase.

\(\dfrac{\delta }{p_i}x_{di}(I, \mathbf p)\)

For the Cobb-Douglas utility function, this is:

\(\dfrac{\delta }{p_i}x_{di}(I, \mathbf p)\)

In addition to the derivative, we may be interested in the elasticity. That is, the proportional change in output after a change in price.

\(\xi_i =\dfrac{\dfrac{\Delta x_i}{x_i}}{\dfrac{\Delta p_i}{p_i}}\)

\(\xi_i =\dfrac{\Delta x_i}{\Delta p_i}\dfrac{p_i}{x_i}\)

For the point-price elasticity of demand we evaluate infintesimal movements.

\(\xi_i =\dfrac{\delta x_i}{\delta p_i}\dfrac{p_i}{x_i}\)

\subsection{Constant price elasticity of demand}

If the point-price elasticity of demand is constant we have:

\(\xi_i =\dfrac{\delta x_i}{\delta p_i}\dfrac{p_i}{x_i}=c\)

This means that small changes in the price at low level cause large changes in quantity.

\subsection{Arc-price elasticity of demand}

We may have price changes which are non-infintesimal.

\(E_d=\dfrac{\Delta Q/\bar Q}{\Delta P/\bar P}\)

Where \(\bar Q\) and \(\bar P\) are the mid-points between the start and end.


\documentclass[../main.tex]{subfiles}

\begin{document}
        \section{Practice Problems}
        \begin{example}{}{}
                Consider the curve described by \(\vec{r}=\left(\frac{t^3}{3}-2t\right)\i + \left(\frac{t^3}{3}+2t\right)\j + \sqrt{2}t^2\k\) for \(t\in\mathbb{R}\). Find the arc length from \(t=0\) to \(t=1\).
                \tcblower
                First, we find that
                \[
                \vec{r}'(t)=\left(t^2+2\right)\i + \left(t^2+2\right)\j +2\sqrt{2}t\j
                \]
                so
                \[
                |\vec{r}'(t)|^2=\left(t^2+2\right)^2 + \left(t^2+2\right)^2 +(2\sqrt{2}t)^2 = 2\left(t^2+2\right)^2
                \]
                which gives us \(|\vec{r}'(t)|=\sqrt{2}(t^2+2)\), and thus the arc length is given by
                \[
                \int_0^1\sqrt{2}(t^2+2)\,dt = \frac{7\sqrt{2}}{3}
                \]
        \end{example}
        \begin{example}{}{}
                For the above curve, find the curvature at \(t=0\).
                \tcblower
                We previously found
                \[
                \vec{r}'(t)=\left(t^2+2\right)\i + \left(t^2+2\right)\j +2\sqrt{2}t\j
                \]
                so we may calculate
                \[
                \vec{r}''(t)=2t\i + 2t\j + 2\sqrt{2}\k
                \]
                The curvature at \(t=0\) is then given by
                \[
                \kappa = \frac{|\vec{r}'(0)\times\vec{r}''(0)|}{|\vec{r}'(0)|^3}
                \]
                so we may begin by calculating
                \[
                \vec{r}'(0)\times\vec{r}''(0)=4\sqrt{2}\i+4\sqrt{2}\j
                \]
                which has magnitude \(|\vec{r}'(0)\times\vec{r}''(0)|=8\). Finally, we calculate \(|\vec{r}'(0)|=2\sqrt{2}\), so we obtain the final solution
                \[
                \kappa = \frac{8}{\sqrt{8}^3}=\frac{1}{\sqrt{8}}
                \]
        \end{example}
        \begin{example}{}{}
                Verify that \(\frac{d}{dt}\vec{u}'(t)\times\vec{u}''(t)=\vec{u}'(t)\times\vec{u}'''(t)\).
                \tcblower
                By the derivative rule for the cross product, we have
                \[
                \frac{d}{dt}\vec{u}'(t)\times\vec{u}''(t)=\vec{u}''(t)\times\vec{u}''(t) + \vec{u}'(t)\times\vec{u}'''(t)
                \]
                but for any vector we always have \(\vec{v}\times\vec{v}=0\), so this simplifies to
                \[
                \frac{d}{dt}\vec{u}'(t)\times\vec{u}''(t)=\vec{u}'(t)\times\vec{u}(t)'''
                \]
        \end{example}
        \begin{example}{}{}
                Expand and simplify \(\frac{d}{dt}\left((\vec{u}\times\vec{u}')\cdot(\vec{u}'\times\vec{u}'')\right)\).
                \tcblower
                \begin{align*}
                        \frac{d}{dt}\left((\vec{u}\times\vec{u}')\cdot(\vec{u}'\times\vec{u}'')\right)&=
                        (\vec{u}\times\vec{u}')'\cdot(\vec{u}'\times\vec{u}'')+(\vec{u}\times\vec{u}')\cdot(\vec{u}'\times\vec{u}'')'\\
                        &=(\vec{u}'\times\vec{u}'+\vec{u}\times\vec{u}'')\cdot(\vec{u}'\times\vec{u}'')\\
                        &\quad+(\vec{u}\times\vec{u}')\cdot(\vec{u}''\times\vec{u}''+\vec{u}'\times\vec{u}''')\\
                        &=(\vec{u}\times\vec{u}'')\cdot(\vec{u}'\times\vec{u}'')+(\vec{u}\times\vec{u}')\cdot(\vec{u}'\times\vec{u}''')
                \end{align*}
        \end{example}
        \begin{example}{}{}
                Consider a smooth vector valued function \(\vec{r}\) for which there exists a point \(P\) which lies on every normal plane to  \(\vec{r}\). Prove that the curve lies on a sphere.
                \tcblower
                First, note that we may take \(P=0\) without loss of generality, as any curve with such a point my be translated such that that \(P=0\) without changing the geometric properties of the curve.

                The normal plane to the curve at \(\vec{r}(t_0)\) must be perpendicular to \(\vec{r}'(t_0)\), so we have
                \[
                \vec{r}'(t)\cdot((x-x_0)\i+(y-y_0)\j+(z-z_0)\k)=0
                \]
                but since \(P=0\) must always satisfy the above, it must be that
                \[
                \vec{r}'(t)\cdot(-x_0\i+-y_0\j+-z_0\k)=\vec{r}'(t)\cdot(-\vec{r}(t))=0
                \]
                which we have previously shown requires that \(\vec{r}\) lies on the surface of a sphere.
        \end{example}
        \begin{example}{}{}
                Consider the parabola described by \(\vec{r}(t)=t\i+\frac{t^2}{2]\j}\), with unit tangent vector \(\vec{T}(t)\). Find a parameterization \(\vec{r}(u)\) such that \(|\frac{d\vec{T}}{du}|=1\).
                \tcblower
                First, we find that \(\vec{r}'(t)=\i+t\j\), so \(|\vec{r}'(t)|=\sqrt{1+t^2}\) and
                \[
                \vec{T}(t)=\frac{1}{\sqrt{1+t^2}}(\i+t\j)
                \]

                The solution may then be completed by noting that since we require that
                \[
                \left|\frac{d\vec{T}}{du}\right|=1
                \]
                it must be that
                \[
                \frac{du}{dt}=|\vec{T}'(t)|
                \]
                which may be integrated to find an expression for \(u\). This process is elided for brevity.
        \end{example}
\end{document}


\documentclass[../main.tex]{subfiles}
\begin{document}
\chapter{Bit Manipulation}
Many books on algorithmic problem solving seems forget about one topic--bit and bit manipulation. Bit is how data is represented and saved on the hardware. Thus knowing such concept and bit manipulation using Python sometimes can also help us device more efficient algorithms, either space or time complexity in the later Chapter.

For example, how to convert a char or integer to bit, how to get each bit, set each bit, and clear each bit. Also, some more advanced bit manipulation operations. After this, we will see some examples to show how to apply bit manipulation in real-life problems.
%%%%%%%%%%%%%%%%%%%%%%Bit operators%%%%%%%%%%%%
\section{Python Bitwise Operators}
\label{sec_basic_bit_operator}
Bitwise operators include <<, >>, \&, |, \~, \^. All of these operators operate on signed or unsigned numbers, but instead of treating that number as if it were a single value, they treat it as if it were a string of bits. Twos-complement binary is used for representing the singed number.

Now, we introduce the six bitwise operators.
\paragraph{x \texttt{<<} y} Returns $x$ with the bits shifted to the left by $y$ places (and new bits on the right-hand-side are zeros). This is the same as multiplying $x$ by $2^y$.

\paragraph{x \texttt{>>} y} Returns $x$ with the bits shifted to the right by $y$ places. This is the same as dividing $x$ by $2^y$, same result as the $//$ operator. This right shift is also called \textit{arithmetic right shift}, it fills in the new bits with the value of the sign bit.

\paragraph{x \texttt{\&} y} "Bitwise and". Each bit of the output is 1 if the corresponding bit of $x$ AND of $y$ is 1, otherwise it's 0. It has the following property:
\begin{lstlisting}[language=Python]
# keep 1 or 0 the same as original
1 & 1 = 1
0 & 1 = 0
# set to 0 with & 0
1 & 0 = 0
0 & 0 = 0
\end{lstlisting}

\paragraph{x \texttt{|} y} "Bitwise or".  Each bit of the output is 0 if the corresponding bit of $x$ AND of $y$ is 0, otherwise it's 1.
\begin{lstlisting}[language=Python]
# set to 1 with | 1
1 | 1 = 1
0 | 1 = 1

# keep 1 or 0 the same as original
1 | 0 = 1
0 | 0 = 0
\end{lstlisting}

\paragraph{$\thicksim x
$} Returns the complement of x - the number you get by switching each 1 for a 0 and each 0 for a 1. This is the same as $-x - 1$(really?).

\paragraph{x $\wedge$ y} "Bitwise exclusive or". Each bit of the output is the same as the corresponding bit in $x$ if that bit in $y$ is 0, and it's the complement of the bit in $x$ if that bit in $y$ is 1. It has the following basic properties:
\begin{lstlisting}[language=Python]
# toggle 1 or 0 with ^ 1
1 ^ 1 = 0
0 ^ 1 = 1

# keep 1 or 0 with ^ 0
1 ^ 0 = 1
0 ^ 0 = 0
\end{lstlisting}
Some examples shown:
\begin{lstlisting}
A = 5 = 0101, B = 3 = 0011
 A ^ B = 0101 ^ 0011 = 0110 = 6
 \end{lstlisting}
More advanced properties of XOR operator include:
 \begin{lstlisting}
a ^ b = c
c ^ b = a

n ^ n = 0
n ^ 0 = n
eg. a=00111011, b=10100000 , c= 10011011, c ^b= a
 \end{lstlisting}

\paragraph{Logical right shift} The logical right shift is different to the above right shift after shifting it puts a 0 in the most significant bit. It is indicated with a $>>>$ operator n Java. However, in Python, there is no such operator, but we can implement one easily using \textbf{bitstring} module padding with zeros using $>>=$ operator.
\begin{lstlisting}[language=Python]
>>> a = BitArray(int=-1000, length=32)
>>> a.int
-1000
>>> a >>= 3
>>> a.int
536870787
\end{lstlisting}

%%%%%%%%%%%%%%%%%%%%%%%Useful Python function%%%%%%%%%%%%%%%%%%%%
\section{Python Built-in Functions}
\label{sec_bitwise_built_in_function}
\paragraph{bin()} The bin() method takes a single parameter \textbf{num}- an integer and return its \textit{binary string}. If not an integer, it raises a TypeError exception.
\begin{lstlisting}[language=Python]
a = bin(88)
print(a)
# output
# 0b1011000
\end{lstlisting}
However, bin() doesn't return \textit{binary bits} that applies the two's complement rule. For example, for the negative value:
\begin{lstlisting}[language=Python]
a1 = bin(-88)
# output
# -0b1011000
\end{lstlisting}
\paragraph{int(x, base = 10)} The int() method takes either a string x  to return an integer with its corresponding base. The common base are: 2, 10, 16 (hex).
\begin{lstlisting}[language=Python]
b = int('01011000', 2)
c = int('88', 10)
print(b, c)
# output
# 88 88
\end{lstlisting}

\paragraph{chr()} The chr() method takes a single parameter of integer and return a character (a string) whose Unicode code point is the integer. If the integer i is outside the range, ValueError will be raised.
\begin{lstlisting}[language=Python]
d = chr(88)
print(d)
# output
# X
\end{lstlisting}
\paragraph{ord()} The ord() method takes a string representing one Unicode character and return an integer representing the Unicode code point of that character.
\begin{lstlisting}[language=Python]
e = ord('a')
print(e)
# output
#  97
\end{lstlisting}

%%%%%%%%%%%%%%%%Two's complement Binary%%%%%%%%%%%%%%%%%%%%%%
\section{Twos-complement Binary}
Given 8 bits, if it is unsigned, it can represent the values 0 to 255 (1111,1111). However, a two's complement 8-bit number can only represent positive integers from 0 to 127 (0111,1111) because the most significant bit is used as sign bit: '0' for positive, and '1' for negative.
\begin{equation}
  \sum_{i=0}^{N-1} 2^i =  2^{(N-1)}+2^{(N-2)}+...+2^2+2^1+2^0= 2^N-1
\end{equation}
The twos-complement binary is the same as the classical binary representation for positive integers and differs slightly for negative integers. Negative integers are represented by performing Two's complement operation on its absolute value: it would be $(2^N-n)$ for representing $-n$ with N-bits.
Here, we show Two's complement binary for eight-bit signed integers in Fig.~\ref{fig:twos_complement}.
\begin{figure}
    \centering
    \includegraphics[width=0.6\columnwidth]{fig/eight_bit_two_complement.png}
    \caption{Two's Complement Binary for Eight-bit Signed Integers.}
    \label{fig:twos_complement}
\end{figure}
\paragraph{Get Two's Complement Binary Representation}
In Python, to get the two's complement binary representation of a given integer, we do not really have a built-in function to do it directly for negative number. Therefore, if we want to know how the two's complement binary look like for negative integer we need to write code ourselves. The Python code is given as:
\begin{lstlisting}[language=Python]
bits = 8
ans = (1 << bits) -2
print(ans)
# output
# '0b11111110'
\end{lstlisting}
There is another method to compute: inverting the bits of n (this is called \textbf{One's Complement}) and adding 1. For instance, use 8 bits integer 5, we compute it as the follows:
\begin{align}
\label{five}
    5_{10} &= {0000, 0101}_2, \\
    {-5}_{10} &= {1111, 1010}_2 + 1_2, \\
    {-5}_{10} &= {1111, 1011}_2
\label{five_complement}
\end{align}
To flip a binary representation, we need expression x XOR '1111,1111', which is $2^N-1$. The Python Code is given:
\begin{lstlisting}[language=Python]
def twos_complement(val, bits):
    # first flip implemented with xor of val with all 1's
    flip_val = val ^ (1 << bits - 1)
    #flip_val = ~val we only give 3 bits
    return bin(flip_val + 1)
\end{lstlisting}

\paragraph{Get Two's Complement Binary Result}
In Python, if we do not want to see its binary representation but just the result of two's complement of a given positive or negative integer, we can use two operations $-x$ or $\thicksim +1$. For input 2, the output just be a negative integer -2 instead of its binary representation:
\begin{lstlisting}[language=Python]
def twos_complement_result(x):
    ans1 = -x
    ans2 = ~x + 1
    print(ans1, ans2)
    print(bin(ans1), bin(ans2))
    return ans1
# output
# -8 -8
# -0b1000 -0b1000
\end{lstlisting}
This is helpful if we just need two's complement result instead of getting the binary representation.

%%%%%%%%%%%%%%%%%%%%%%%Useful operation%%%%%%%%%%%%%%%%%%%%
\section{Useful Combined Bit Operations}
\label{sec_useful_bit_combination}

For operations that handle each bit, we first need a \textit{mask} that only set that bit to 1 and all the others to 0, this can be implemented with arithmetic left shift sign by shifting 1 with 0 to n-1 steps for n bits:
\begin{lstlisting}[language=Python]
mask = 1 << i
\end{lstlisting}
\paragraph{Get ith Bit} In order to do this, we use the property of AND operator either 0 or 1 and with 1, the output is the same as original, while if it is and with 0, they others are set with 0s.
\begin{lstlisting}[language=Python]
# for n bit, i in range [0,n-1]
def get_bit(x, i):
    mask = 1 << i
    if x & mask:
        return 1
    return 0
print(get_bit(5,1))
# output
# 0
\end{lstlisting}
Else, we can use left shift by i on x, and use AND with a single 1.
\begin{lstlisting}[language=Python]
def get_bit2(x, i):
    return x >> i & 1
print(get_bit2(5,1))
# output
# 0
\end{lstlisting}

\paragraph{Set ith Bit} We either need to set it to 1 or 0. To set this bit to 1, we need matching relation: $1->1, 0->1$. Therefore, we use operator |. To set it to 0: $1->0, 0->0$. Because 0 \& 0/1 = 0, 1\&0=1, 1\&1 = 1, so we need first set that bit to 0, and others to 1.
\begin{lstlisting}[language=Python]
# set it to 1
x = x | mask

# set it to 0
x = x & (~mask)
\end{lstlisting}

\paragraph{Toggle ith Bit} Toggling means to turn bit to 1 if it was 0 and to turn it to 0 if it was one. We will be using 'XOR' operator here due to its properties.
\begin{lstlisting}[language=Python]
x = x ^ mask
\end{lstlisting}

\paragraph{Clear Bits} In some cases, we need to clear a range of bits and set them to 0, our base mask need to put 1s at all those positions, Before we solve this problem, we need to know a property of binary subtraction. Check if you can find out the property in the examples below,
\begin{lstlisting}[numbers=none]
1000-0001 = 0111
0100-0001 = 0011
1100-0001 = 1011
\end{lstlisting}

The property is, the difference between a binary number n and 1 is all the bits on the right of the rightmost 1 are flipped including the rightmost 1. Using this amazing property, we can create our mask as:
\begin{lstlisting}[language=Python]
# base mask
i = 5
mask = 1 << i
mask = mask -1
print(bin(mask))
# output
# 0b11111
\end{lstlisting}
With this base mask, we can clear bits: (1) All bits from the most significant bit till i (leftmost till ith bit) by using the above mask. (2) All bits from the lest significant bit to the ith bit by using $\thicksim mask$ as mask. The Python code is as follows:
\begin{lstlisting}[language=Python]
# i i-1 i-2 ... 2 1 0, keep these positions
def clear_bits_left_right(val, i):
    print('val', bin(val))
    mask = (1 << i) -1
    print('mask', bin(mask))
    return bin(val & (mask))
\end{lstlisting}
\begin{lstlisting}[language=Python]
# i i-1 i-2 ... 2 1 0, erase these positions
def clear_bits_right_left(val, i):
    print('val', bin(val))
    mask = (1 << i) -1
    print('mask', bin(~mask))
    return bin(val & (~mask))
\end{lstlisting}
Run one example:
\begin{lstlisting}[numbers=none]
print(clear_bits_left_right(int('11111111',2), 5))
print(clear_bits_right_left(int('11111111',2), 5))
val 0b11111111
mask 0b11111
0b11111
val 0b11111111
mask -0b100000
0b11100000
\end{lstlisting}
\paragraph{Get the lowest set bit } Suppose we are given '0010,1100', we need to get the lowest set bit and return '0000,0100'. And for 1100, we get 0100.  If we try to do an AND between 5 and its two's complement as shown in Eq.~\ref{five} and \ref{five_complement}, we would see only the right most 1 bit is kept and all the others are cleared to 0. This can be done using expression $x \&(-x)$, $-x$ is the two's complement of $x$.
\begin{lstlisting}[language=Python]
def get_lowest_set_bit(val):
    return bin(val & (-val))
print(get_lowest_set_bit(5))
# output
# 0b1
\end{lstlisting}
Or, optionally we can use the property of subtracting by 1.
\begin{lstlisting}
x ^ (x & (x -1))
\end{lstlisting}
\paragraph{Clear the lowest set bit} In many situations we want to strip off the lowest set bit for example in Binary Indexed tree data structure, counting number of set bit in a number. We use the following operations:
\begin{lstlisting}[language=Python]
def strip_last_set_bit(val):
    print(bin(val))
    return bin(val & (val - 1))
print(strip_last_set_bit(5))
# output
# 0b101
# 0b100
\end{lstlisting}

% \paragraph{Update Bits:} $mask = ~(1<<i)$, use this to clear at first, $(num\&mask) | (value<<i)$

 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%      python implementation          %%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Applications}
\label{chapter_bit_section_bitwise}
% \paragraph{Code Implementation of Two's Complement} For One's complement we simply flip all bits. For Two's complement, we traverse all bits from Least significant bit (LSB) and we flip this bit: we flip all 1's until we find our first 0, and flip it to 1. And all the left digits just do the basic flip.
% \begin{lstlisting}[language=Python]
% def twos_complement(val, bits):
%     # first flip implemented with xor of val with all 1's
%     flip_val = val^(2**bits-1)
%     #flip_val = ~val we only give 3 bits
%     return bin(flip_val+1)
% print(twos_complement(5, 8))
% # 0b11111011
% \end{lstlisting}

% \begin{lstlisting}[language=Python]
% def twos_complement2(val, bits):
%     zeroFound = False
%     ans = 0
%     mask = 1
%     for i in range(bits):
%         b = (val & (mask))!=0 # get ith bit
%         b = not b # flipped
%         if not zeroFound:
%             if not b: # found zero, flip to one, else flip to zero: no operation needed
%                 print('found')
%                 ans = ans | (mask) # set ith bit
%                 zeroFound = True
%         else:
%             if b:
%                 ans = ans | (mask)
%         mask = mask << 1  # change mask to the next bit
%     return bin(ans)
% print(twos_complement2(5, 8))
% # output
% # 0b11111011
% \end{lstlisting}

\paragraph{Recording States} Some algorithms like Combination, Permutation,  Graph Traversal require us to record states of the input array. Instead of using an array of the same size, we can use a single integer, each bit's location indicates the state of one element with same index in the array. For example, we want to record the state of an array with length 8. We can do it like follows:
\begin{lstlisting}[language=Python]
used = 0
for i in range(8):
    if used &(1<<i): # check state at i
        continue
    used = used | (1<<i)  # set state at i used
    print(bin(used))
\end{lstlisting}
It has the following output
\begin{lstlisting}[numbers=none]
0b1
0b11
0b111
0b1111
0b11111
0b111111
0b1111111
0b11111111
\end{lstlisting}

\paragraph{XOR Single Number}
\begin{examples}[resume]
\item \textbf{136. Single Number(easy).}  Given a non-empty array of integers, every element appears twice except for one. Find that single one. \textit{Note: Your algorithm should have a linear runtime complexity. Could you implement it without using extra memory?}
\begin{lstlisting}[numbers=none]
Example 1:

Input: [2,2,1]
Output: 1

Example 2:

Input: [4,1,2,1,2]
Output: 4
\end{lstlisting}

\textbf{Solution: XOR.} This one is kinda straightforward. You’ll need to know the  properties of XOR as shown in Section~\ref{sec_basic_bit_operator}.
\begin{lstlisting}
n ^ n = 0
n ^ 0 = n
\end{lstlisting}
Therefore, we only need on variable to record the state which is initialize with 0: the first time to appear x = n, second time to appear x = 0. the last element x will be the single number. To set the statem we can use XOR.
\begin{lstlisting}[language = Python]
def singleNumber(self, nums):
    """
    :type nums: List[int]
    :rtype: int
    """
    v = 0
    for e in nums:
        v = v ^ e
    return v
\end{lstlisting}
\item \textbf{137. Single Number II} Given a non-empty array of integers, every element appears three times except for one, which appears exactly once. Find that single one. \textit{Note: Your algorithm should have a linear runtime complexity. Could you implement it without using extra memory?}
\begin{lstlisting}[language=Python]
Example 1:

Input: [2,2,3,2]
Output: 3

Example 2:

Input: [0,1,0,1,0,1,99]
Output: 99
\end{lstlisting}

\textbf{Solution: XOR and Two Variables.} In this problem, because all element but one appears three times. To record the states of three, we need at least two variables.  And we initialize it to a = 0, b = 0. For example, when 2 appears the first time, we set a = 2, b = 0; when it appears two times, a = 0, b = 2; when it appears three times, a = 0, b = 0. For number that appears one or two times will be saves either in a or in b. Same as the above example, we need to use XOR to change the state for each variable. We first do a = a XOR v, b = XOR v, we need to keep a unchanged and set b to zero. We can do this as a = a XOR v \& $\thicksim$  b; b = b XOR v \& $\thicksim$  a.
\begin{lstlisting}[language=Python]
def singleNumber(self, nums):
    """
    :type nums: List[int]
    :rtype: int
    """
    a = b = 0
    for num in nums:
        a = a ^ num & ~b
        b = b ^ num & ~a
    return a|b
\end{lstlisting}

\item \textbf{421. Maximum XOR of Two Numbers in an Array (medium).}  Given a non-empty array of numbers, $a_0, a_1, a_2,... , a_{n-1}$, where $0 \leq a_i < 2^{31}$. Find the maximum result of $a_i$ XOR $a_j$, where $0 \leq i, j < n$. Could you do this in $O(n)$ runtime?
\begin{lstlisting}[numbers=none]
Example:
Input: [3, 10, 5, 25, 2, 8]

Output: 28
Explanation: The maximum result is 5 \^ 25 = 28.
\end{lstlisting}
\textbf{Solution 1: Build the Max bit by bit.} First, let's convert these integers into binary representation by hand.
\begin{lstlisting}[numbers=none]
3   0000, 0011
10  0000, 1011
5   0000, 0101
25  0001, 1001
2   0000, 0010
8   0000, 1000
\end{lstlisting}
If we only look at the highest position i where there is one one and all others zero. Then we know the maximum XOR $m$ has 1 at that bit. Now, we look at two bits: i, i-1. The possible maximum XOR for this is append 0 or 1 at the end of $m$, we have possible max 11, because for XOR, if we do XOR of m with others,  $m XOR a = b$, if b exists in these possible two sets, then max is possible and it become  $m<<1+1$.  We can carry on this process,  the following process is showed as follows:
answer\^ 1 is the possible max,
\begin{lstlisting}
def findMaximumXOR(self, nums):
    """
    :type nums: List[int]
    :rtype: int
    """
    answer = 0
    for i in range(32)[::-1]:
        answer <<= 1 # multiple it by two
        prefixes = {num >> i for num in nums} # shift right for n, divide/2^i, get the first (32-i) bits
        answer += any((answer+1) ^ p in prefixes for p in prefixes)
    return answer
\end{lstlisting}

\textbf{Solution 2: Use Trie.}
\begin{lstlisting}[language=Python]
def findMaximumXOR(self, nums):
    def Trie():
        return collections.defaultdict(Trie)

    root = Trie()
    best = 0

    for num in nums:
        candidate = 0
        cur = this = root
        for i in range(32)[::-1]:
            curBit = num >> i & 1
            this = this[curBit]
            if curBit ^ 1 in cur:
                candidate += 1 << i
                cur = cur[curBit ^ 1]
            else:
                cur = cur[curBit]
        best = max(candidate, best)
    return best
\end{lstlisting}
\end{examples}
\paragraph{With Mask}
\begin{examples}[resume]
\item \textbf{190. Reverse Bits (Easy).}Reverse bits of a given 32 bits unsigned integer.
\begin{lstlisting}[numbers=none]
Example 1:

Input: 00000010100101000001111010011100
Output: 00111001011110000010100101000000
Explanation: The input binary string 00000010100101000001111010011100 represents the unsigned integer 43261596, so return 964176192 which its binary representation is 00111001011110000010100101000000.

Example 2:

Input: 11111111111111111111111111111101
Output: 10111111111111111111111111111111
Explanation: The input binary string 11111111111111111111111111111101 represents the unsigned integer 4294967293, so return 3221225471 which its binary representation is 10101111110010110010011101101001.
\end{lstlisting}

\textbf{Solution: Get Bit and Set bit with mask.} We first get bits from the most significant position to the least significant position. And get the bit at that position with mask, and set the bit in our 'ans' with a mask indicates the position of (31-i):
\begin{lstlisting}[language=Python]
# @param n, an integer
# @return an integer
def reverseBits(self, n):
    ans = 0
    for i in range(32)[::-1]: #from high to low
        mask = 1 << i
        set_mask = 1 << (31-i)
        if (mask & n) != 0: #get bit
            #set bit
            ans |= set_mask
    return ans
\end{lstlisting}

\item \textbf{201. Bitwise AND of Numbers Range (medium).}Given a range [m, n] where $0 \leq m \leq n \leq 2147483647$, return the bitwise AND of all numbers in this range, inclusive.
\begin{lstlisting}[numbers=none]
Example 1:

Input: [5,7]
Output: 4

Example 2:

Input: [0,1]
Output: 0
\end{lstlisting}

\textbf{Solution 1: O(n) do AND operation.} We start a 32 bit long 1s. The solution would receive LTE error.
\begin{lstlisting}[language=Python]
def rangeBitwiseAnd(self, m, n):
    """
    :type m: int
    :type n: int
    :rtype: int
    """
    ans = int('1'*32, 2)
    for c in range(m, n+1):
        ans &= c
    return ans
\end{lstlisting}

\textbf{Solution 2: Use mask, check bit by bit. } Think, if we AND all, the resulting integer would definitely smaller or equal to $m$. For example 1:
\begin{lstlisting}[numbers=none]
0101 5
0110 6
0111 7
\end{lstlisting}
We start from the least significant bit at 5, if it is 1, then we check the closest number to 5 that has 0 at the this bit. It would be 0110. If this number is in the range, then this bit is offset to 0. We then move on to check the second bit. To make this closest number: first we clear the least i+1 positions in m to get 0100 and then we add it with $1 << (i+1)$ as 0010 to get 0110.
\begin{lstlisting}[language=Python]
def rangeBitwiseAnd(self, m, n):
    ans = 0
    mask = 1
    for i in range(32): # [::-1]:
        bit = mask & m != 0
        if bit:
            # clear i+1, ..., 0
            mask_clear = (mask<<1)-1
            left = m & (~mask_clear)
            check_num = (mask << 1) + left
            if check_num < m or check_num > n:
                ans |= 1 << i
        mask = mask << 1
    return ans

\end{lstlisting}

\textbf{Solution 3: Use While Loop.} We can start do AND of n with (n-1). If the resulting integer is still larger than m, then we keep do such AND operation.
\begin{lstlisting}[language=Python]
def rangeBitwiseAnd(self, m, n):
    ans=n
    while ans>m:
        ans=ans&(ans-1)
    return ans
\end{lstlisting}
\end{examples}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%Exerciese
\section{Exercises}
\begin{enumerate}
\item Write a function to determine the number of bits required to convert integer A to integer B.
\begin{lstlisting}[language = Python]
def bitswaprequired(a, b):
  count = 0
  c = a^b
  while(c != 0):
    count += c & 1
    c = c >> 1
  return count
print(bitswaprequired(12, 7))
\end{lstlisting}

\item \textbf{389. Find the Difference (easy).} Given two strings $s$ and $t$ which consist of only lowercase letters. String $t$ is generated by random shuffling string s and then add one more letter at a random position. Find the letter that was added in $t$.
\begin{lstlisting}[numbers=none]
Example:
Input:
s = "abcd"
t = "abcde"

Output:
e
Explanation:
'e' is the letter that was added.
\end{lstlisting}
\textbf{Solution 1: Use Counter Difference.} This way we need $O(M+N)$ space to save the result of counter for each letter.
\begin{lstlisting}[language=Python]
def findTheDifference(self, s, t):
    s = collections.Counter(s)
    t = collections.Counter(t)
    diff = t - s
    return list(diff.keys())[0]
\end{lstlisting}
\textbf{ Solution 2: Single Number with XOR.}  Using bit manipulation and with $O(1)$ we can find it in $O(M+N)$ time, which is the best BCR:
\begin{lstlisting}[language=Python]
def findTheDifference(self, s, t):
    """
    :type s: str
    :type t: str
    :rtype: str
    """
    v = 0
    for c in s:
        v = v ^ ord(c)
    for c in t:
        v = v ^ ord(c)
    return chr(v)

\end{lstlisting}

\item \textbf{50. Pow(x, n) (medium).} for n, such as 10, we represent it as 1010, if we have a base and an result, we start from the least significant position, each time we move, the base because base*base, and if the value if 1, then we multiple the answer with the base.

% Now consider a range

% [m = 0bxyz0acd, n=0bxyz1rst]

% here xyzpacdrst all are digits in base 2.

% We can find two numbers that are special in the range [m, n]
% \begin{lstlisting}
% (1) m' = 0bxyz0111
% (2) n' = 0bxyz1000
% \end{lstlisting}

% The bitwise AND of all the numbers in range [m, n] is just the bitwise AND of the two special number
% \begin{lstlisting}
% rangeBitwiseAnd(m, n) = m' & n' = 0bxyz0000
% \end{lstlisting}

% This tells us, the bitwise and of the range is keeping the common bits of m and n from left to right until the first bit that they are different, padding zeros for the rest.
% \begin{lstlisting}[language = Python]
% def rangeBitwiseAnd(self, m, n):
%         """
%         :type m: int
%         :type n: int
%         :rtype: int
%         """
%         i = 0
%         while m != n:
%             m >>= 1
%             n >>= 1 #find the common bits, i counts how many zeros we need
%             i += 1
%         return n << i # common bits then we shift i left
% \end{lstlisting}
\end{enumerate}
\end{document}


\section{Coordinate Transformations}

Table \ref{tablecoord} contains coordinate transformations between Cartesian, cylindrical, and spherical coordinates and vector fields, \cite{ulaby1999fundamentals}. The routines in Table \ref{tablecoord} overload the built-in Matlab functions of the same names in order to use $\theta$, $\phi$ orderings and definitions consistent with most scattering textbooks (physics convention, rather than mathematics convention which is what Matlab uses).  The same routines transform between either coordinate points or vector fields. For example, transforming Cartesians points $(x,y,z)$ to spherical coordinates $(r,\theta,\phi)$ is done as
\begin{verbatim}
[r, th, phi] = cart2sph(x,y,z)
\end{verbatim}

Vector transformations always need the coordinates of the input vector field. For example, transforming a Cartesian vector field $(A_x,A_y,A_z)$ located at points $(x,y,z)$ to spherical vector components $(A_r,A_{\theta},A_{\phi})$ is done as
\begin{verbatim}
[Ar, Ath, Aphi] = cart2sph(x,y,z,Ax,Ay,Az)
\end{verbatim}

Input arrays can be any equal size and $\tan^{-1}$ is always computed with \texttt{atan2}. Unit vectors can be created by setting one of the input vector components equal to 1 and the others to 0.  For example, the Cartesian unit vectors at a point $(x,y,z)$ expressed in spherical coordinates are computed as
\begin{verbatim}
[x_r, x_th, x_phi] = cart2sph(x,y,z,1,0,0)
[y_r, y_th, y_phi] = cart2sph(x,y,z,0,1,0)
[z_r, z_th, z_phi] = cart2sph(x,y,z,0,0,1)
\end{verbatim}

\clearpage
\newpage

\bgroup
\def\arraystretch{1.25}
\begin{table}[h]
\caption{Coordinate Transforms}
\begin{center}
\begin{tabular}{|c| l | l |}
\hline
\multicolumn{1}{|c|}{Routine} & \multicolumn{1}{|c|}{Point Transforms} & \multicolumn{1}{|c|}{Vector Field Transforms} \\
\hline
\texttt{cart2cyl} &
\threearray{\rho}{\sqrt{x^2 + y^2}}{\phi}{\tan^{-1}(y/x)}{z}{z}  &
\threearray{A_{\rho}}{A_x \cos\phi + A_y\sin\phi}{A_{\phi}}{-A_x\sin\phi + A_y\cos\phi}{A_z}{A_z}  \\
\hline
\texttt{cyl2cart} &
\threearray{x}{\rho\cos\phi}{y}{\rho\sin\phi}{z}{z} &
\threearray{A_x}{A_{\rho}\cos\phi - A_{\phi}\sin\phi}{A_y}{A_{\rho}\sin\phi + A_{\phi}\cos\phi}{A_z}{A_z} \\
\hline
\texttt{cart2sph} &
\threearray{r}{\sqrt{x^2 + y^2 + z^2}}{\theta}{\tan^{-1}(\sqrt{x^2+y^2}/z)}{\phi}{\tan^{-1}(y/x)} &
\threearray{A_r}{A_x \st\cos\phi + A_y \sin\theta\sin\phi + A_z \cos\theta}{A_{\theta}}{A_x \ct\cos\phi + A_y \cos\theta\sin\phi - A_z \sin\theta}{A_{\phi}}{-A_x\sin\phi + A_y\cos\phi}\\
\hline
\texttt{sph2cart} &
\threearray{x}{r\st\cos\phi}{y}{r\st\sin\phi}{z}{r\ct} &
\threearray{A_x}{A_r \st\cos\phi + A_{\theta} \ct\cos\phi - A_{\phi} \sin\phi}{A_y}{A_r \st\sin\phi + A_{\theta} \cos\theta\sin\phi + A_{\phi} \cos\phi}{A_z}{A_r\ct - A_{\theta}\st} \\
\hline
\texttt{cyl2sph} &
\threearray{r}{\sqrt{\rho^2 + z^2}}{\theta}{\tan^{-1}(\rho/z)}{\phi}{\phi} &
\threearray{A_r}{A_{\rho} \st + A_z\ct}{A_{\theta}}{A_{\rho} \ct - A_z\st}{A_{\phi}}{A_{\phi}} \\
\hline
\texttt{sph2cyl} &
\threearray{\rho}{r\st}{\phi}{\phi}{z}{r\ct} &
\threearray{A_{\rho}}{A_r\st + A_{\theta}\ct}{A_{\phi}}{A_{\phi}}{A_z}{A_r\ct - A_{\theta}\st} \\
\hline
\end{tabular}
\end{center}
\label{tablecoord}
\end{table}
\egroup

\paragraph{Routine \texttt{cart2cyl} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/cart2cyl.m}
}
\clearpage
\newpage
\paragraph{Routine \texttt{cyl2cart} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/cyl2cart.m}
}
\paragraph{Routine \texttt{cart2sph} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/cart2sph.m}
}
\paragraph{Routine \texttt{sph2cart} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/sph2cart.m}
}
\paragraph{Routine \texttt{cyl2sph} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/cyl2sph.m}
}
\paragraph{Routine \texttt{sph2cyl} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/sph2cyl.m}
}
\paragraph{Helper Routine \texttt{checkargs} }
{\footnotesize
\VerbatimInput{\code/CoordinateTransforms/checkargs.m}
}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Alternating Series}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to alternating series and absolute or conditional convergence.
\end{abstract}
\maketitle

\begin{exercise}
For the infinite series
\[ \sum_{n=2}^\infty \frac{(-1)^n}{ \ln n}, \]
the function $1/\ln n$ \wordChoice{\choice[correct]{is}\choice{is not}} positive and \wordChoice{\choice[correct]{is}\choice{is not}}  decreasing when $n \geq 2$. Furthermore it \wordChoice{\choice[correct]{does}\choice{does not}} tend to zero as $n \rightarrow \infty$. The alternating series test \wordChoice{\choice[correct]{does}\choice{does not}} apply and \wordChoice{\choice[correct]{implies convergence}\choice{implies divergence}\choice{says nothing about the series}}.
\end{exercise}

\begin{exercise}
For the infinite series
\[ \sum_{n=2}^\infty \frac{(-1)^n}{ \ln n + (-1)^n \sqrt{\ln n}}, \]
the function $1/(\ln n+(-1)^n \sqrt{\ln n})$ \wordChoice{\choice[correct]{is}\choice{is not}} positive and \wordChoice{\choice{is}\choice[correct]{is not}} decreasing when $n \geq 2$. Furthermore it \wordChoice{\choice[correct]{does}\choice{does not}} tend to zero as $n \rightarrow \infty$. The alternating series test \wordChoice{\choice{does}\choice[correct]{does not}} apply and \wordChoice{\choice{implies convergence}\choice{implies divergence}\choice[correct]{says nothing about the series}}.
\end{exercise}

\begin{exercise}
For the infinite series
\[ \sum_{n=3}^\infty \frac{(-1)^n}{ \ln n - (\ln n)^{-1/2}}, \]
the function $1/(\ln n - (\ln n)^{-1/2})$ \wordChoice{\choice[correct]{is}\choice{is not}} positive and \wordChoice{\choice[correct]{is}\choice{is not}} decreasing when $n \geq 3$. Furthermore it \wordChoice{\choice[correct]{does}\choice{does not}} tend to zero as $n \rightarrow \infty$. The alternating series test \wordChoice{\choice[correct]{does}\choice{does not}} apply and \wordChoice{\choice[correct]{implies convergence}\choice{implies divergence}\choice{says nothing about the series}}.
\end{exercise}

\begin{exercise}
For the infinite series
\[ \sum_{n=2}^\infty \frac{(-1)^n (n-1)^2}{n^2}, \]
the function $1/(\ln n+(-1)^n \sqrt{\ln n})$ \wordChoice{\choice[correct]{is}\choice{is not}} positive and \wordChoice{\choice[correct]{is}\choice{is not}} decreasing when $n \geq 2$. Furthermore it \wordChoice{\choice{does}\choice[correct]{does not}} tend to zero as $n \rightarrow \infty$. The alternating series test \wordChoice{\choice{does}\choice[correct]{does not}} apply and \wordChoice{\choice{implies convergence}\choice{implies divergence}\choice[correct]{says nothing about the series}}.
\end{exercise}

\begin{exercise}
For the infinite series
\[ \sum_{n=1}^\infty \frac{\cos \ln n}{ e^n}, \]
the function $1/(e^n)$ \wordChoice{\choice[correct]{is}\choice{is not}} positive and \wordChoice{\choice[correct]{is}\choice{is not}} decreasing when $n \geq 1$. Furthermore it \wordChoice{\choice[correct]{does}\choice{does not}} tend to zero as $n \rightarrow \infty$. The alternating series test \wordChoice{\choice{does}\choice[correct]{does not}} apply and \wordChoice{\choice{implies convergence}\choice{implies divergence}\choice[correct]{says nothing about the series}}.
\begin{hint}
First determine whether the series is alternating.
\end{hint}
\end{exercise}

\begin{exercise}
Does the alternating series test apply to the series?
\[ \sum_{n=5}^\infty \frac{\cos n \pi}{ \sqrt{n}} \]
\begin{multipleChoice}
\choice[correct]{Yes}
\choice{No, it's not alternating}
\choice{No, the terms are not decreasing}
\choice{No, the terms do not go to zero}
\end{multipleChoice}
If yes, for what minimum value of $N$ can you be certain that
\[ \sum_{n=5}^{N-1} \frac{\cos n \pi}{\sqrt{n}} \]
differs from the sum of the series by at most $10^{-3}$? If no such $N$ exists, write N/A.
\[ N \geq \answer{10^{6}}. \]
\begin{hint}
If the alternating series test applies, we would need the magnitude (i.e., absolute value) of the first term \textit{not} included in the partial sum to be no greater than $10^{-3}$.
\end{hint}
\end{exercise}

\begin{exercise}
Find an interval of length $\frac{1}{4}$ which contains the sum of the infinite series
\[ \sum_{n=1}^\infty \frac{(-1)^{n-1}}{n^2} \]
\[ \text{Interval }  = \left[ \answer{\frac{3}{4}}, \answer{1} \right] \]
\begin{hint}
Partial sums of an alternating series also alternate above and below the sum of the series itself.
\end{hint}
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[seriesacd001]

For each series below, determine whether it converges absolutely (A), converges conditionally (C), or diverges (D). \offline{Show how you used convergence tests to arrive at your answer.} \[\text{I: } \sum_{n = 1}^\infty\frac{\cos n \pi}{\sqrt[3]{n+3}} \qquad \text{II: } \sum_{n = 1}^\infty\frac{(-1)^n}{n^2+3} \qquad \text{III: } \sum_{n = 2}^\infty\frac{\cos n \pi}{\ln (n^2 + 1)}\]
\begin{multiplechoice}
\choice{I: C, II: D, III: D}
\choice[correct]{I: C, II: A, III: C}
\choice{I: A, II: C, III: A}
\choice{I: D, II: C, III: D}
\choice{I: C, II: D, III: C}
\choice{I: C, II: A, III: A}
\end{multiplechoice}
\begin{feedback}
I: converges conditionally. The value of \(\cos n \pi\) alternates \(\pm 1\). The terms \((n+3)^{-1/3}\) decrease to zero, so the series converges by the alternating series test. The series is not absolutely convergent because the \(p\)-series with \(p = -1/3\) is divergent.

II: converges absolutely. The series converges absolutely by direct comparison to a \(p\)-series with \(p=2\).

III: converges conditionally. The series converges by the alternating series test because \(1/\ln (n^2+1)\) decreases to \(0\) as \(n \rightarrow \infty\) and \(\cos n \pi\) alternates in value between \(+1\) and \(-1\). However, \(1/\ln (n^2+1) \geq 1/n\) for all large \(n\), so by direct comparison to the harmonic series, the series is not absolutely convergent. Therefore the convergence is conditional.
\end{feedback}

\end{question}

\begin{question}%%%%%[seriesacd004]

For each series below, determine whether it converges absolutely (A), converges conditionally (C), or diverges (D). \offline{Show how you used convergence tests to arrive at your answer.} \[\text{I: } \sum_{n = 1}^\infty\frac{(-1)^n n^2}{2 n^2+1} \qquad \text{II: } \sum_{n = 1}^\infty\frac{1}{1 + n^3 e^{-n}} \qquad \text{III: } \sum_{n = 1}^\infty\frac{(-1)^n n + 2}{n^2}\]
\begin{multiplechoice}
\choice{I: D, II: D, III: D}
\choice{I: D, II: A, III: C}
\choice{I: C, II: C, III: A}
\choice{I: A, II: C, III: D}
\choice[correct]{I: D, II: D, III: C}
\choice{I: D, II: A, III: A}
\end{multiplechoice}
\begin{feedback}
I: diverges. The series diverges because \(n^2/(n^2+1) \rightarrow 1\), meaning that the terms do not go to zero. The \(n\)-th term divergence test implies divergence.

II: diverges. The series diverges because \(n / (n + n^3 e^{-n}) \rightarrow 1\) (because \(n^3 e^{-n} \rightarrow 0\)). By the limit comparison theorem, this means the series has the same behavior as a \(p\)-series with \(p=1\), which means it diverges.

III: converges conditionally. The series converges because it is the sum of two convergent series: one with terms \((-1)^n / n\) (which is a convergent series by the alternating series test because \(1/n\) decreases to zero) and a second with terms \(2/n^2\) (which is a convergent \(p\)-series). However, the series is not absolutely convergent, because \[ \left| \frac{(-1)^n n + 2}{n^2} \right| = \frac{n + (-1)^n2}{n^2}\] for \(n \geq 2\), which is a sum of a {\it divergent} \(p\)-series with \(p=1\) and an absolutely convergent alternating \(p\)-series with \(p=2\). Thus the series is conditionally convergent.
\end{feedback}

\end{question}

\begin{question}%%%%%[2019AltSerError]

Which of the following intervals contains the value of the infinite series
\[ \sum_{n=0}^\infty \frac{(-1)^n}{n+1}? \]
\begin{multiplechoice}
\choice{\(\displaystyle \left[ \frac{1}{4}, \frac{1}{3} \right]\)}
\choice{\(\displaystyle \left[ \frac{1}{3}, \frac{1}{2} \right]\)}
\choice{\(\displaystyle \left[ \frac{1}{2}, \frac{7}{12} \right]\)}
\choice[correct]{\(\displaystyle \left[ \frac{7}{12}, \frac{5}{6} \right]\)}
\choice{\(\displaystyle \left[ \frac{5}{6}, \frac{11}{12} \right]\)}
\choice{\(\displaystyle \left[ \frac{11}{12}, \frac{7}{6} \right]\)}
\end{multiplechoice}
\begin{feedback}
The function \(1/(n+1)\) is positive and decreases to zero, so by the Alternating Series Test, we know that partial sums alternate above and below the actual value of the sum.  In particular, if we call the value of the sum \(L\), then
\[
\begin{aligned}
1 & \geq  L \\
1 - \frac{1}{2} & \leq  L \\
1 - \frac{1}{2} + \frac{1}{3} & \geq L \\
1 - \frac{1}{2} + \frac{1}{3} - \frac{1}{4} & \leq L
\end{aligned}
\]
and so on. The last two inequalities together imply that \(L\) belongs to the interval \([\frac{7}{12},\frac{5}{6}]\).
\end{feedback}

\end{question}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2017C.12]

Determine whether the following series converge absolutely (A), converge conditionally (C), or diverge (D). \offline{For full credit be sure to explain your reasoning and specify which tests were used.}
\[ \sum_{n=2}^\infty \frac{(-1)^n 2^{2n}}{3^n} \ \ \ \ \sum_{n=2}^\infty \frac{(-1)^n}{\sqrt{n}} \]
\begin{multiplechoice}
\choice{both A}
\choice{one A, the other C}
\choice{one A, the other D}
\choice{both C}
\choice[correct]{one C, the other D}
\choice{both D}
\end{multiplechoice}

\end{question}

\end{document}


\paragraph{\underline{Discrete}} \hspace{0pt}

\vspace{10pt} \noindent \textbf{Bernoulli}
$ X \sim \distbernoulli(p) $, $0 < p < 1$

$p$: probability of success
\begin{gather*}
P(X=1)=p \\ %\mbox{ , }
P(X=0)=1-p
\end{gather*}

$ \mathbb{E}X = p$, $\var X = p(1-p) $

\vspace{10pt} \noindent \textbf{Binomial}
$X \sim \distbinomial(n,p)$, $n \in \mathbb{N}_+$, $0 < p < 1$

$n$: number of tests/trials ($\distbinomial(1,p) \equiv \distbernoulli(p)$)

$p$: probability of success
\begin{gather*}
P(x=k) = \binom{n}{k} p^k(1-p)^{n-k}
\end{gather*}

$ \mathbb{E}X = np$, $\var X = np(1-p) $

\vspace{10pt} \noindent \textbf{Poison}
$X \sim \distpoisson(\lambda)$, $\lambda > 0$

In some sense, it's a generalization of the Binomial distribution.
\begin{gather*}
P(X=k) = \frac{\lambda^k}{k!}e^{-\lambda} \mbox{ , } k = 0,1,2,\ldots
\end{gather*}

$ \mathbb{E}X = \lambda$, $\var X = \lambda $

\vspace{10pt} \noindent \textbf{Geometric}
$X \sim \distgeometric(p)$
\begin{gather*}
P(X=k) = (1-p)^{k-1}p
\end{gather*}

if $k=1,2,\ldots$ then $\mathbb{E}X = \frac{1}{p}$, $\var X = \frac{1-p}{p^2}$

if $k=0,1,2,\ldots$ then $\mathbb{E}X = \frac{1-p}{p}$, $\var X = \frac{1-p}{p^2}$

\vspace{10pt} \noindent \textbf{Hypergeometric}
$X \sim \disthypergeometric(N,M,n)$
\begin{gather*}
P(X=k) \frac{ \binom{M}{k} \binom{N-M}{n-k} }{ \binom{N}{n} }
\end{gather*}

$ \max\{ 0, M-N+n \} \leq  k \leq \min\{ n, M \} $

\vfill
\columnbreak

\paragraph{\underline{Continuous}} \hspace{0pt}

\vspace{10pt} \noindent \textbf{Uniform}
$X \sim \distuniform[a,b]$, $a,b \in R$, $a < b$

$a$: lower limit

$b$: upper limit
\begin{gather*}
f(x) = \begin{cases}
\frac{1}{b-a} \mbox{ if } x \in [a,b] \\
0 \mbox{ otherwise}
\end{cases}
= \frac{1}{b-a} I_{[a,b]}(x)
\end{gather*}

$ \mathbb{E}X = \frac{a+b}{2}$, $\var X = \frac{(b-a)^2}{12} $

\vspace{10pt} \noindent \textbf{Normal}
$X \sim \distnormal(\mu, \sigma)$

$\mu$: mean

$\sigma$: standard deviation
\begin{gather*}
f(x) = \frac{1}{\sqrt{2\pi} \sigma} \exp{ \left\{ - \frac{(X-\mu)^2}{2 \sigma^2} \right\} }
\end{gather*}

$ \mathbb{E}X = \mu$, $\var X = \sigma^2$

\vspace{10pt} \noindent \textbf{Exponential}
$X \sim \distexponential(\lambda)$, $\lambda > 0$

decreasing \eqref{eq:expdecreasing} and increasing \eqref{eq:expincreasing}:
\begin{gather*}
f(x) = \lambda e^{-\lambda x}
\label{eq:expdecreasing} \tag{a}
\end{gather*}

\vspace{-30pt} \begin{gather*}
f(x) = 1 - e^{-\lambda x}
\label{eq:expincreasing} \tag{b}
\end{gather*}

\vspace{10pt} \noindent \textbf{Gamma}
$X \sim \distgamma(\alpha, \beta)$, $\alpha, \beta > 0$

\nodata

\vspace{10pt} \noindent \textbf{Chi-square}
$X \sim \distchisquare_n$

$n$: number of degrees of freedom

\vspace{10pt} \noindent \textbf{Student's t-distribution}
$X \sim \diststudentt^{[n]}$

$n$: number of degrees of freedom

Also known as simply t-distribution.

\vspace{10pt} \noindent \textbf{F-distribution}
$X \sim \distf^{[n]}$

$n$: number of degrees of freedom

Also known as Fisher-Snedecor distribution.


\section{The \accumulate algorithm}
\Label{sec:accumulate}

The \accumulate algorithm in the \cxx Standard Library \cite[\S 29.8.2]{cxx-17-draft} computes
the sum of an given initial value and the elements in a range.
%
Our version of the original signature
reads:

\begin{lstlisting}[style=acsl-block]

  value_type
  accumulate(const value_type* a, size_type n, value_type init);
\end{lstlisting}

The result of \accumulate shall equal the value
$\displaystyle{ \mathtt{init} + \sum_{i = 0}^{\mathtt{n}-1} \mathtt{a}[i] }$.
This implies that \accumulate will return \inl{init} for an empty range.

%\clearpage

\subsection{The logic function \Accumulate}

As in the case of \specref{counti}  we specify \accumulate by first defining the
\emph{logic function} \logicref{Accumulate} that formally defines
the summation of elements in an array.

\input{Listings/Accumulate.acsl.tex}

With this definition the following equation holds for $n \geq 0$
\begin{align}
\Label{eq:accumulate}
    \mathtt{Accumulate}(\mathtt{a}, \mathtt{n}, \mathtt{init})
    &= \mathtt{init} + \sum_{i = 0}^{\mathtt{n-1}} \mathtt{a}[i]
\end{align}

The predicate \logicref{AccumulateBounds} that we will subsequently use
in order to compactly express requirements that exclude numeric
overflows while accumulating value.
This predicate states that  for $0 \leq i < n$ the \emph{partial sums}
%
\begin{gather}
\Label{eq:accumulate1}
\mathtt{init} + \sum_{k = 0}^{\mathtt{i}} \mathtt{a}[k]
\end{gather}
%
do not overflow.
If one of them did, one couldn't guarantee that the result of \isoc implementation
of \accumulate equals the mathematical description of \Accumulate.

%\clearpage

\subsection{\AccumulateDefault ---a variant of \Accumulate}

The following listing shows another version of \logicref{Accumulate},
called \logicref{AccumulateDefault}.

\input{Listings/AccumulateDefault.acsl.tex}

The function \AccumulateDefault uses~\inl{a[0]} as default value of \inl{init}.
Thus, for \AccumulateDefault we have

\begin{align}
\Label{eq:accumulate-default}
    \mathtt{AccumulateDefault}(\mathtt{a}, \mathtt{n})
    &= \sum_{i = 0}^{\mathtt{n-1}} \mathtt{a}[i]
\end{align}
We will use this version for the specification of the algorithm \specref{partialsum}.

This listing also includes additional properties of observable
\AccumulateDefault behavior, here given as a lemmas.
It also contains the predicate \logicref{AccumulateDefaultBounds}
with corresponding numeric limits for the predicate~\AccumulateDefault.

%\clearpage

\subsection{Formal specification of \accumulate}

Using the logic function \Accumulate and the predicate \AccumulateBounds,
the specification of \accumulate is then as simple
as shown in the following listing.

\input{Listings/accumulate.h.tex}

\clearpage

\subsection{Implementation of \accumulate}

The following listing shows an implementation of the
\accumulate function with corresponding loop annotations.

\input{Listings/accumulate.c.tex}

Note that loop invariant \inl{partial} claims that in the $i$-th iteration step \inl{result}
equals the accumulated value of Equation~\eqref{eq:accumulate1}.
This depends on the property \inl{bounds} of \specref{accumulate} which expresses that
there is no numeric overflow when updating the variable \inl{init}.

\clearpage


\chapter{Regression}
Almost all of them have closed-form solutions.
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%% Least Squares Regression %%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Least Squares Regression}
%%%%%%%% Notation
\subsection{Notations}
\begin{itemize}
    \item $X$: $N \times d$ matrix of data.
    \item $x^{(i)}$: $i$-th sample, $d$ dimensional feature vector, suppose $x_0 = 1$.
    \item $y$: $N$ dimensional output.
    \item  $w$: $d \times 1$ parameter.
\end{itemize}
%%%%%%%% Model
\subsection{Model}
\[ f(X;w) = Xw \]
\begin{remark}
    If $X$ is invertible, then we are done.
    \[ w = X^{-1}y \]
    Unfortunately $X$ is usually not invertible, it is usually not even a square matrix. So we need optimizaiton-based approach to solve this.
\end{remark}
%%%%%%%% Loss Function
\subsection{Loss Function}\label{section:LeastSquareLossFunc}
\[ J(w)  = \frac{1}{2}\| Xw - y \|_2^2 = \frac{1}{2}\sum_{i=1}^N(w^Tx^{(i)} - y^{(i)})^2 \]
Our goal is to minimize the loss function
\[ \min_w \quad J(w) \]
%%%%%%%% Gradient Descent Update Rule
\subsection{Descent Method}
$J(w)$ is convex, and can be optimized by gradient descent.
\[ \nabla J(w) = X^T(Xw-y) \]
\[ w_{i+1} = w_i - \alpha \nabla J(w) \]
where $\alpha$ is the step size.
%%%%%%%% Closed Form Solution
\subsection{Closed Form Solution}
$J(w)$ is convex, and we can calculate the closed-form solution.

Let $\nabla J(w) = 0$.
\[ X^TXy = X^Ty \]
\[ w = (X^TX)^{-1}X^Ty \]
\begin{remark}
    If $X^TX$ is not invertible, we can use its pseudo-inverse, which is defined as
\end{remark}
\begin{definition}[Pseudo-Inverse]
    Let $A = U\Sigma V^T$ be the SVD of matrix $A \in \mathbb{R}^{m\times n}$, with $\mathbf{Rank}(A) = r$. The pseudo-inverse of $A$ is defined as
    \[ A^{\dagger} = V\Sigma^{-1}U^T \in \mathbb{R}^{n\times m} \]
\end{definition}
\begin{remark}
    Pseudo-Inverse does not necessarily give a good solution for lack of numerical stability.
\end{remark}
%%%%%%%%
\subsection{Comparision}
\paragraph{Gradient Descent} is easy to implement and converges relatively fast.
\paragraph{Closed-form solution} is straightforward but may be numerically unstable.
%%%%%%%% Geometric Interpretation
\subsection{Geometric Interpretation}
\[ \hat{y} = Xw = X(X^TX)^{-1}X^Ty \]
\begin{align*}
    \hat{y} - y &= X(X^TX)^{-1}X^Ty - y \\
    &= (X(X^TX)^{-1}X^T - I)y
\end{align*}
Multiplying $X^T$ on both sides yields
\begin{align*}
    X^T(\hat{y}-y) &= X^T(X(X^TX)^{-1}X^T - I)y \\
    &= (X^TX(X^TX)^{-1}X^T - X^T)y \\
    &= 0
\end{align*}
The error $\hat{y} - y$ is perpendicular to the space spanned by samples $x^{(i)}$.
%%%%%%%% Probability Interpretation
\subsection{Probability Interpretation}\label{section:LeastSquareProbability}
We make the following assumptions
\begin{itemize}
    \item $y$ is linear with $x$ plus some random error $\varepsilon^{(i)}$.
    \[ y^{(i)} = w^Tx^{(i)} + \varepsilon \]
    \item $\varepsilon^{(i)}$ has a Gaussian distribution.
    \[ \varepsilon^{(i)} \sim \mathcal{N}(0,1) \]
\end{itemize}
Substitute $\varepsilon$ with $w^Tx-y$, and the probaility density function is
\[ p(y^{(i)}|x^{(i)};w) = \frac{1}{\sqrt{2\pi}}\exp\left(-\frac{(w^Tx^{(i)}-y^{(i)})^2}{2}\right) \]
We maximize the (log) likelihood
\[ L(w) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\|w^Tx^{(i)}-y^{(i)}\|^2}{2}\right) \]
\[ l(w) = \log L(w) = constant - \sum_{i=1}^N \frac{1}{2}\|w^Tx^{(i)}-y^{(i)}\|^2\]
Maximizing the log likelihood is equivalent to minimizing the last term. So the problem becomes
\[ \min_{w} \quad \frac{1}{2}\sum_{i=1}^N\|w^Tx^{(i)} - y^{(i)}\|^2 \]
This is exactly the loss function in section \ref{section:LeastSquareLossFunc}.
%%%%%%%% Regularization
\subsection{Regularization}
\[ \min_w \| Xw - y \|^2 + \epsilon\|w\|^2 \]
where $\epsilon$ is a small positive value, and the closed form solution can be derived from
\[ (X^TX + \epsilon I)w = X^Ty \]
Adding $\epsilon$ sometimes makes the inverse of matrix more stable.
\begin{remark}
    Adds a contraint that the norm of $w$ cannot be too large. But regularization still does not guarantee a good solution when the matrix is not invertible.

    Regularization is also a general way of preventing overfitting.
\end{remark}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%% Ridge Regression %%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Ridge Regression}
%%%%%%%% Model
\subsection{Model}
\begin{align*}
    \min_w \quad &\| Xw - y \|_2^2 \\
    \text{s.t.} \quad &\|w\|_2^2 \le t
\end{align*}
It can be rewritten in the form of Penalized Residual Sum of Squares, PRSS
\[ \textrm{PRSS}(w) = \|Xw-y\|_2^2 + \lambda\|w\|_2^2 \]

\subsection{Solving Ridge Regression}
Ridge regression is also a convex optimization problem. It can be solved by using numerical methods or deriving closed form solutions.
\[ \nabla\textrm{PRSS}(w) = -2X^T(y-Xw) + 2\lambda w \]
So
\[ w = (X^TX + \lambda I)^{-1}X^Ty \]
\begin{remark}
    Even if $X^TX$ is singular, by choosing a proper $\lambda$ we can make $(X^TX + \lambda I)$ nonsingular.
\end{remark}
%%%%%%%% Regularization Revisited
\subsection{Regularization Revisited}
$\lambda$ is a regularization hyperparameter.
\begin{itemize}
    \item When $\lambda = 0$, the penalty term has no effect and Ridge regression reduces to ordinary least squares.
    \item When $\lambda \to \infty$, $w\to 0 $.
    \item A proper $\lambda$ can be chosen via cross-validation.
\end{itemize}
%%%%%%%% Probability Interpretation
\subsection{Probability Interpretation}
In addtion to the assumptions made in Section \ref{section:LeastSquareProbability}, if we further assume
\[ w \sim \mathcal{N}(0,\sqrt{\lambda}) \]
Then by Bayes's Rule
\[ p(w|y) = \frac{p(y|w)\cdot p(w)}{p(y)} \]
By Maximum-A-Posteriori estimation
\[ \hat{w} = \arg \max_w p(w|y) = \arg \max_w \log p(y|w)\cdot p(w) \]
This yields the loss function of Ridge Regression.
%%%%%%%% Geometric Interpretation
\subsection{Geometric Interpretation}
The feasible region of Ridge Regression is a $l_2$-norm ball (circle in 2D and sphere in 3D). We are looking for the smallest value of objective function in the norm ball.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%% LASSO Regression %%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{LASSO}
\emph{Least Absolute Shrinkage and Selection Operator}.
%%%%%%%% Model
\subsection{Model}
\begin{align*}
    \min_w \quad &\| Xw - y \|_2^2 \\
    \text{s.t.} \quad &\|w\|_1 \le t
\end{align*}
The optimization problem can be reformulated as
\[ \min_w \quad \| Xw - y \|_2^2 + \lambda \|w\|_1 \]
%%%%%%%% Solving LASSO
\subsection{Solving LASSO}
LASSO can be solved, but solving it is nontrivial. There are a number of algorithms to solve LASSO.
\paragraph{Quasi-Gradient Method.}
Let
\[
\nabla |w|_1 =
\begin{cases}
1 \quad &w > 0\\
-1 \quad &w < 0\\
0 \quad &w = 0
\end{cases}
\]
\paragraph{A Simple Method.}
Define error $r$ by
$$ r = y - Xw = y - \sum_{i=1}^D x_iw_i $$
\begin{enumerate}
    \item Initialize $r = y$; $w_1 = w_2 = \dots = w_d = 0$.
    \item Find an $x_j$ that correlates with $r$.
    \item Update $w_j = w_j +\delta_j$, where $\delta_j = \varepsilon\cdot\mathrm{sgn}\langle r, X_j \rangle$.
    \item Update $r = r-\delta\cdot x_j$.
    \item Repeat step 2-4 until convergence
\end{enumerate}
\paragraph{Soft Threshold Method.}
Use Taylor expansion.
%%%%%%%% Probabilistic Interpretation
\subsection{Probabilistic Interpretation}
In addition to the assumptions made in \ref{section:LeastSquareProbability}, we further assume that $w$ has a Gamma distribution.

The minimizing $w$ is the MAP estimator.
%%%%%%%% Geometric Interpretation
\subsection{Geometric Interpretation}
The feasible region of LASSO is a $l_1$-norm ball (diamond in 2D). We look for the smallest value in the diamond-shaped region.


\section{Space Curves}\label{sec:SpaceCurves}

We have already seen that a convenient way to describe a line in three
dimensions is to provide a vector that ``points to'' every point on
the line as a parameter $t$ varies, like
$$\langle 1,2,3\rangle+t\langle 1,-2,2\rangle
=\langle 1+t,2-2t,3+2t\rangle.$$
Except that this gives a particularly simple geometric object, there
is nothing special about the individual functions of $t$ that make up
the coordinates of this vector---any vector with a parameter, like
$\langle f(t),g(t),h(t)\rangle$, will describe some curve in three
dimensions as $t$ varies through all possible values.

\begin{example}{Decribing Curves}{helixes}
Describe the curves
$\langle \cos t,\sin t,0\rangle$,
$\langle \cos t,\sin t,t\rangle$, and
$\langle \cos t,\sin t,2t\rangle$.
\end{example}
\begin{solution}
As $t$ varies, the first two coordinates in all three functions
trace out the points on the unit circle, starting
with $(1,0)$ when $t=0$ and proceeding counter-clockwise around the
circle as $t$ increases. In the first case, the $z$ coordinate is
always 0, so this describes precisely the unit circle in the $x$-$y$
plane. In the second case, the $x$ and $y$ coordinates still describe
a circle, but now
the $z$ coordinate varies, so that the
height of the curve matches the value of $t$. When $t=\pi$, for
example, the resulting vector is $\langle -1,0,\pi\rangle$. A bit of
thought should convince you that the result is a helix. In the third
vector, the $z$ coordinate varies twice as fast as the parameter $t$,
so we get a stretched out helix. Both are shown in
Figure~\ref{fig:helixes}. On the left is the first helix, shown for
$t$ between 0 and $4\pi$; on the right is the second helix, shown for
$t$ between 0 and $2\pi$. Both start and end at the same point, but
the first helix takes two full ``turns'' to get there, because its $z$
coordinate grows more slowly.
\end{solution}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize5cm\epsfbox{images/helix1.eps}}} at 0 0
\put {\hbox{\epsfxsize5cm\epsfbox{images/helix2.eps}}} at 2 0
\endpicture}}
\caption{Two helixes. \label{fig:helixes}}
\end{figure}

A vector expression of the form $\langle f(t),g(t),h(t)\rangle$ is called
a \dfont{vector function}; it is a function from
the real numbers $\R$ to the set of all three-dimensional vectors.
We can alternately think of it as three separate functions,
$x=f(t)$, $y=g(t)$, and $z=h(t)$, that describe points in space. In
this case we usually refer to the set of equations as
\dfont{parametric equations} for the curve, just
as for a line. While the parameter $t$ in a vector function might
represent any one of a number of physical quantities, or be simply a
``pure number'', it is often convenient and useful to think of $t$ as
representing time. The vector function then tells you where in space
a particular object is at any time.

Vector functions can be difficult to understand, that is, difficult to
picture. When available, computer software can be very helpful. When
working by hand, one useful approach is to consider the
``projections'' of the curve onto the three standard coordinate
planes. We have already done this in part: in
Example~\ref{exa:helixes} we noted that all three curves project to
a circle in the $x$-$y$ plane, since $\langle \cos t,\sin t\rangle$ is
a two dimensional vector function for the unit circle.

\begin{example}{Projections onto a Plane}{projectionsontoplaneexample}
Graph the projections of $\langle \cos t,\sin
t,2t\rangle$ onto the $x$-$z$ plane and the $y$-$z$ plane.
\end{example}
\begin{solution}
The two dimensional vector function for the
projection onto the $x$-$z$ plane is $\langle \cos t, 2t\rangle$, or in
parametric form, $x=\cos t$, $z=2t$. By eliminating $t$ we get the
equation $x=\cos(z/2)$, the familiar curve shown on the left in
Figure~\ref{fig:helix projections}. For the projection onto the $y$-$z$ plane, we start
with the vector function $\langle \sin t, 2t\rangle$, which is the
same as $y=\sin t$, $z=2t$. Eliminating $t$ gives $y=\sin(z/2)$, as
shown on the right in Figure~\ref{fig:helix projections}.
\end{solution}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <15truemm,4truemm>
\setplotarea x from -1.25 to 1.25, y from 0 to 14
\axis left shiftedto x=0 ticks length <2pt> withvalues {$2\pi$} {$4\pi$} /
  at 6.28 12.57 / /
\axis bottom ticks length <2pt> numbered from -1 to 1 by 1 /
\arrow <4pt> [0.35, 1] from 0 14 to 0 14.05
\arrow <4pt> [0.35, 1] from 1.25 0 to 1.3 0
\put {$x$} [tl] <3pt,0pt> at 1.3 0
\put {$z$} [br] <0pt,3pt> at 0 14.05
\plot
1.000 0.000 0.995 0.209 0.978 0.419 0.951 0.628 0.914 0.838 0.866
1.047 0.809 1.257 0.743 1.466 0.669 1.676 0.588 1.885 0.500 2.094
0.407 2.304 0.309 2.513 0.208 2.723 0.105 2.932 0.000 3.142 -0.105
3.351 -0.208 3.560 -0.309 3.770 -0.407 3.979 -0.500 4.189 -0.588 4.398
-0.669 4.608 -0.743 4.817 -0.809 5.027 -0.866 5.236 -0.914 5.445
-0.951 5.655 -0.978 5.864 -0.995 6.074 -1.000 6.283 -0.995 6.493
-0.978 6.702 -0.951 6.912 -0.914 7.121 -0.866 7.330 -0.809 7.540
-0.743 7.749 -0.669 7.959 -0.588 8.168 -0.500 8.378 -0.407 8.587
-0.309 8.796 -0.208 9.006 -0.105 9.215 0.000 9.425 0.105 9.634 0.208
9.844 0.309 10.053 0.407 10.263 0.500 10.472 0.588 10.681 0.669 10.891
0.743 11.100 0.809 11.310 0.866 11.519 0.914 11.729 0.951 11.938 0.978
12.147 0.995 12.357 1.000 12.566 /
\setcoordinatesystem units <15truemm,4truemm> point at -4 0
\setplotarea x from -1.25 to 1.25, y from 0 to 14
\axis left shiftedto x=0 ticks length <2pt>  withvalues {$2\pi$} {$4\pi$} /
  at 6.28 12.57 / /
\axis bottom ticks length <2pt> numbered from -1 to 1 by 1 /
\arrow <4pt> [0.35, 1] from 0 14 to 0 14.05
\arrow <4pt> [0.35, 1] from 1.25 0 to 1.3 0
\put {$y$} [tl] <3pt,0pt> at 1.3 0
\put {$z$} [br] <0pt,3pt> at 0 14.05
\plot 0.000 0.000 0.105 0.209 0.208 0.419 0.309 0.628 0.407 0.838 0.500
 1.047 0.588 1.257 0.669 1.466 0.743 1.676 0.809 1.885 0.866 2.094
 0.914 2.304 0.951 2.513 0.978 2.723 0.995 2.932 1.000 3.142 0.995
 3.351 0.978 3.560 0.951 3.770 0.914 3.979 0.866 4.189 0.809 4.398
 0.743 4.608 0.669 4.817 0.588 5.027 0.500 5.236 0.407 5.445 0.309
 5.655 0.208 5.864 0.105 6.074 0.000 6.283 -0.105 6.493 -0.208 6.702
 -0.309 6.912 -0.407 7.121 -0.500 7.330 -0.588 7.540 -0.669 7.749
 -0.743 7.959 -0.809 8.168 -0.866 8.378 -0.914 8.587 -0.951 8.796
 -0.978 9.006 -0.995 9.215 -1.000 9.425 -0.995 9.634 -0.978 9.844
 -0.951 10.053 -0.914 10.263 -0.866 10.472 -0.809 10.681 -0.743 10.891
 -0.669 11.100 -0.588 11.310 -0.500 11.519 -0.407 11.729 -0.309 11.938
 -0.208 12.147 -0.105 12.357 0.000 12.566 /
\endpicture}}
\caption{The projections of $\langle \cos t,\sin
t,2t\rangle$ onto the $x$-$z$ and $y$-$z$ planes. \label{fig:helix projections}}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:SpaceCurves}}

\begin{enumialphparenastyle}

\begin{ex}
Investigate the curve ${\bf r}=\langle \sin t,\cos t,\cos
8t\rangle$.
\end{ex}

\begin{ex}
Investigate the curve
${\bf r}=\langle t\cos t,t\sin t,t\rangle$.
\end{ex}

\begin{ex}
Investigate the curve
${\bf r}=\langle t,t^2,\cos t\rangle$.
\end{ex}

\begin{ex}
Investigate the curve
${\bf r}=\langle \cos(20t)\sqrt{1-t^2},\sin(20t)\sqrt{1-t^2},t\rangle$
\end{ex}

\begin{ex}
Find a vector function for the curve of intersection of
$x^2+y^2=9$ and $y+z=2$.
\begin{sol}
	$\langle 3\cos t, 3\sin t, 2-3\sin t\rangle$
\end{sol}
\end{ex}

\begin{ex}
A bug is crawling outward along the spoke of a wheel that lies along
a radius of the wheel. The bug is crawling at 1 unit per second and
the wheel is rotating at 1 radian per second. Suppose the wheel lies
in the $y$-$z$ plane with center at the origin, and at time $t=0$ the
spoke lies along the positive $y$ axis and the bug is at the origin.
Find a vector function ${\bf r}(t)$
for the position of the bug at time $t$.
\begin{sol}
	$\langle 0,t\cos t,t\sin t\rangle$
\end{sol}
\end{ex}

\begin{ex}
What is the difference between the parametric curves
$f(t)=\langle t, t, t^2 \rangle$, $g(t)=\langle t^2, t^2, t^4
\rangle$, and $h(t)=\langle \sin(t), \sin(t), \sin^2(t) \rangle$as $t$
runs over all real numbers?
\end{ex}

\begin{ex}
Plot each of the curves below in 2 dimensions, projected
onto each of the three standard planes (the $x$-$y$, $x$-$z$, and
$y$-$z$ planes).
\begin{enumerate}
	\item	$f(t)=\langle t, t^3, t^2 \rangle$, $t$ ranges over all real numbers
	\item	$f(t)=\langle t^2, t-1, t^2+5 \rangle$  for $0\leq t \leq 3$
\end{enumerate}
\end{ex}

\begin{ex}
Given points $A=(a_1, a_2, a_3)$ and $B=(b_1, b_2, b_3)$, give
parametric equations for the line {\em segment} connecting $A$ and
$B$. Be sure to give appropriate $t$ values.
\end{ex}

\begin{ex}
With a parametric plot and a set of $t$ values, we can associate
a `direction'.  For example, the curve $\langle \cos t, \sin t
\rangle$ is the unit circle traced counterclockwise.  How can we amend
a set of given parametric equations and $t$ values to get the same
curve, only traced backwards?
\end{ex}

\end{enumialphparenastyle}


\subsection{Basic Topology}
\begin{defn}
A topological space is a set $X$ together with a collection $\tau$
of subsets called \emph{open sets }such that\end{defn}
\begin{itemize}
\item $\emptyset$ and $X$ are open
\item Every union of open sets is open
\item Every finite intersection of open sets is open
\end{itemize}
$\tau$ is called a \emph{topology} on $X$. Closed sets in $X$ are
complements of open sets.
\begin{defn}
Let $X$ and $Y$ be topologial spaces and $f:X\to Y$ a function.
Then $f$ is continuous if $f^{-1}(U)=\{x\in X\mid f(x)\in U\}$ is
open in $X$ if $U$ is open in $Y$.
\end{defn}

\begin{defn}
If $X$ is a topological space and $A\subseteq X$, then the \emph{subspace
topology} on $A$ are the sets $U\cap A$ where $U$ open in $X$.
\end{defn}

\begin{defn}
If $X,Y$ are topological spaces, then $X\times Y=\{(x,y)\mid x\in X,y\in Y\}$
has a \emph{product toplogy}, where open sets in $X\times Y$ are
unions of products $U\times V$ where $U$ open in $X$, $V$ open
in $Y$.
\end{defn}

\begin{defn}
A \emph{homeomorphism} between topological spaces $X$ and $Y$ is
a continuos bijection $f:X\to Y$ whose inverse is also continuous.

If such an $f$ exists we say that $X$ and $Y$ are \emph{homeomorphic:
$X\approx Y$.}
\end{defn}

\begin{defn}
A topological space $X$ is \emph{connected} if it is \emph{not} the
disjoint union of two open subsets, i.e.

\begin{align*}
X & =A\sqcup B
\end{align*}

with $A,B$ open implies that either $A=\emptyset$ or $B=\emptyset$.

Equivalently, a space $X$ is connected if the only subsets of $X$
that are both open and closed are $X$ and $\emptyset$.\end{defn}
\begin{thm}
If $f:X\to Y$ is continuous and $X$ is connected then $f(X)$ is
connected.
\end{thm}

\begin{thm}
If $\{Y_{i}\}_{i}$ are connected sets in a topological space $X$,
and $Y_{i}\cap Y_{j}\neq\emptyset$ for all $i,j$ then $\bigcup_{i}Y_{i}$
is connected.

Note: Any homeomorphism $f:X\to Y$ takes cut points of $X$ to cut
points of $Y$.
\end{thm}

\begin{defn}
A path in $X$ is a continuous map $\alpha:I\to X$. We say that $\alpha$
joins $\alpha(0)$ to $\alpha(1)$.
\end{defn}

\begin{defn}
A topological space $X$ is called \emph{Hausdorff} if given any two
ppoints $x,y\in X$ with $x\neq y$, then there exists disjoint open
sets $U$ and $V$ with $x\in U$, $y\in V$.
\end{defn}

\begin{defn}
An $n$-dimensional manifold is a Hausdorff topological space which
is locally homeomorphic to $\mathbb{R}^{n}$.
\end{defn}

\begin{defn}
An \emph{open cover} of a topological space $X$ is a colelction of
open sets whose union is $X$. A \emph{subcover} is a subset of this
collection which still covers $X$.

A topological space $X$ is called \emph{compact }if every open cover
has a finite subcover. $A\subset X$ is compact if it is compact in
the subspace topology.\end{defn}
\begin{thm}
~\end{thm}
\begin{itemize}
\item If $f:X\to Y$ is continuous and $X$ is compact, then $f(X)$ is
compact.
\item If $X$ compact, $A\subset X$ is closed, then $A$ is compact.\end{itemize}
\begin{thm}
(Heine-Borel) A subspace of $\mathbb{R}^{n}$ is compact iff it is
closed and bounded.
\end{thm}

\begin{thm}
Products of compact spaces are compact.
\end{thm}

Note: Compactness is a topological invariant.

\begin{thm}
If $X$ is a Hausdorff topological space, then every compact subset
$A$ is closed.
\end{thm}

\begin{thm}
If $f:X\to Y$ continuous, $X$ compact, $Y$ Hausdorff, then\end{thm}
\begin{itemize}
\item $f$ is a closed map.
\item If $f$ is a bijection, then $f$ is a homeomorphism.
\end{itemize}