\section{Differential Equations}
Differential equations are equations, or sometimes systems of equations where we are given how a function relates to its derivatives and would like to find satisfying functions or families of functions.
Differential equations land themselves well to modeling real-world phenomena and are still an area of active mathematical study.
We'll see some very basic differential equations and how what we've learned about integrals can be used to solve them.

\subsection{Separable Differential Equations}
If we can break up a first-order (just involving a first derivative) ordinary differential equation into the following form,
\begin{equation*}
	\dd{y}{x} = f(x)g(y)
\end{equation*}
then we say the differential equation is separable and may be able to be solved using a technique called separation of variables.
The steps to solve a separable differential equation are:
\begin{enumerate}
	\item Write the equation in differential form (i.e using $\dd{y}{x}$)
	\item Separate the variables ($\frac{\d{y}}{g(y)}=f(x)\d{x}$)
	\item Integrate both sides
	\item Solve for $y$ in terms of $x$, if possible
	\item Find the general solution
	\item Find a particular solution, given any initial conditions
\end{enumerate}

\begin{example}
	Solve the following differential equation.
	\begin{equation*}
		\dd{y}{x} = (xy)^2, y(1)=1.
	\end{equation*}
\end{example}
\begin{answer}
	Separating and integrating,
	\begin{align*}
		\dd{y}{x} &= (xy)^2 = x^2y^2 \\
		\frac{\d{y}}{y^2} &= x^2\d{x} \\
		\frac{1}{y} &= \frac{x^3}{3} + C. \\
	\end{align*}

	Solving for $C$,
	\begin{align*}
		\frac{-1}{1} &= \frac{1^3}{3} + C \\
		C &= \frac{-4}{3}.
	\end{align*}

	Solving for $y$,
	\begin{align*}
		\frac{-1}{y} &= \frac{x^3}{3} - \frac{4}{3} \\
		y &= \frac{-3}{x^3 - 4}.
	\end{align*}
\end{answer}

\subsubsection{Exponential Growth \& Decay}
We can use differential equations to model growth and decay.

\begin{example}
	Imagine we have some money in a bank account earning interest.
	The more money in the bank, the more the more the account will receive in interest.
	So, the rate of growth of the account value is proportional to the current account value.
	In the language of differential equations,
	\begin{equation*}
		\dd{y}{t} = ky.
	\end{equation*}
	where $k$ is some constant of proportionality.
\end{example}
\begin{answer}
	This equation is separable, so we'll try to solve it using separation of variables.
	\begin{align*}
		\frac{\d{y}}{t} &= k\d{x} \\
		\ln{\abs{y}} &= kt + C \\
		\abs{y} &= Ce^{kt} \\
		y &= Ce^{kt} \\
		y(0) &= Ce^{k\cdot 0} = C \\
		y &= y_0e^{kt}.
	\end{align*}
\end{answer}

This is exactly the equation for continually compounding interest: $y_0$ is the principle and $k$ is an interest rate.

Note that $k$ could theoretically be negative, meaning the amount would decrease proportionally to the remaining amount.
\begin{example}
	The half-life of Pu-239 is 24360 years.
	Suppose that 10g of Pu-239 were released in a nuclear accident, how long would it take to decay to 1g?
\end{example}
\begin{answer}
	Our "principle" is 10g.
	We know that half-life follows the exponential decay differential equation, so it's modeled by $y = 10e^{kt}$.
	We also know that $y(24360)=5$, which should give us enough information to solve for $k$.
	\begin{align*}
		5 &= 10e^{k\cdot 24360} \\
		\frac{1}{2} &= e^{k\cdot 24360} \text{ (see the ``half" in half-life?)} \\
		\ln{\frac{1}{2}} &= 24360k \\
		-\ln{2} &= 24360k \\
		k &= \frac{-\ln{2}}{24360}.
	\end{align*}

	We now can plug $k$ back into our equation to get the full model.
	\begin{align*}
		y &= 10e^{\frac{-\ln{2}}{24360}t} \\
		&= 10\left(e^{\ln{2}}\right)^{\frac{-t}{24360}} \\
		&= 10\cdot2^{-\frac{-t}{24360}}.
	\end{align*}

	We can now plug in 1 for $y$ and solve for $t$
	\begin{align*}
		1 &= 10\cdot2^{-\frac{t}{24360}} \\
		\frac{1}{10} &= 2^{-\frac{t}{24360}} \\
		\log_{2}{\frac{1}{10}} &= -\frac{t}{24360} \\
		\log_{2}{10} &= \frac{t}{24360} \\
		t &= 24360\log_{2}{10} \approx 80922\text{yr}.
	\end{align*}

	If you're familiar with half-life equations, this is exactly $t=t_{1/2}\log_{2}{\frac{N_0}{N_f}}$.
\end{answer}

\subsubsection{Logistic Growth \& Decay}
Although our differential equations assuming that growth is proportional to amount work well for things like bank accounts, bacteria or radioactive particles that can grow and decay without limits, that model is a little too simplistic to model populations that are limited by resources.

\begin{example}
	Imagine that we have a population of animals in a forest.
	If the forest has lots of resources to support to animal population, then they can grow basically like normal.
	However, as the population grows, resources become more scarce, so population growth would slow down, or some of the population would starve.
	We can model this with the following differential equation.
	\begin{equation*}
		\dd{P}{t} = kP(M-P)
	\end{equation*}
	where $k$ is some constant of proportionality and $M$ is some maximum population before growth starts to decline.
\end{example}
\begin{answer}
	This differential equation is also separable, so let's try to solve it.
	\begin{align*}
		\dd{P}{t} &= kP(M-P) \\
		\frac{\d{P}}{P(M-P)} &= k\d{t} \\
		\d{P}\left(\frac{1/M}{P}+\frac{1/M}{M-P}\right) &= k\d{t} \text{ (using partial fractions)} \\
		\frac{1}{M}\left(\ln{\abs{P}} - \ln{\abs{M-P}}\right) &= kt + C \\
		\ln{\abs{\frac{P}{M-P}}} &= Mkt + C \\
		\frac{P}{M-P} &= Ce^{Mkt} \text{ (b/c $P \geq 0$)} \\
		P &= CMe^{Mkt} - CPe^{Mkt} \\
		P\left(1+Ce^{Mkt}\right) &= CMe^{Mkt} \\
		P &= \frac{CMe^{Mkt}}{1+Ce^{Mkt}} \\
		&= \frac{M}{1+Ce^{-Mkt}}.
	\end{align*}
\end{answer}

\begin{figure}[H]
	\label{logistic}
	\centering
	\includegraphics[width=0.5\textwidth]{./applications_integrals/logistic_growth.png}
	\caption{\hyperref{}{}{}{Logistic Growth}}
\end{figure}

Looking at a graph of this function, we can see that it starts growing like an exponential curve but begins to flatten, obtaining a maximum value of $M$, which is called the carrying capacity.
The population is growing the fastest when $P=M/2$, which you can verify by finding the global maxima of $\dd{P}{t}$ using a first derivative test.

\subsection{Slope Fields \& Euler's Method}
Unfortunately, not all differential equations are as easy to solve as separable differential equations.
In fact, some are impossible to get nice, closed-form solutions.
\subsubsection{Slope Fields}
We may still be able to visualize what the graph of a solution might look like by drawing lines that have the same slope as a solution.
Any solution that passes through the points where we draw the sloped lines must be tangent to these lines, meaning a solution will follow the ``flow" of these lines.

\begin{figure}[H]
	\label{slope_field}
	\centering
	\includegraphics[width=0.75\textwidth]{./applications_integrals/slope_field.png}
	\caption{\hyperref{}{}{}{Slope field of $\dd{y}{x}=-x/y$ with a possible solution}}
\end{figure}

\subsubsection{Euler's Method}
If your differential equation also has an initial condition, you can start at the initial point and follow the slope field to find an approximate solution.
This is what Euler's Method tries to accomplish.
It can approximate the value of a solution at some $x$ value by starting at some point, usually given by the initial condition, and iteratively taking small steps of size $\Delta x$ in the direction determined by the slope field.
\begin{align*}
	x_{n+1} &= x_n + \Delta x \\
	y_{n+1} &= y_n + \Delta x\dd{y}{x}_{(x_n,y_n)}.
\end{align*}

The smaller the steps, the more accurate the approximation.
\begin{figure}[H]
	\label{eulers_method}
	\centering
	\includegraphics[width=0.55\textwidth]{./applications_integrals/Eulers-Approximation.png}
	\includegraphics[width=0.35\textwidth]{./applications_integrals/Eulers-Approximation2.png}
	\caption{\hyperref{https://calcworkshop.com/first-order-differential-equations/eulers-method-table/}{}{}{Calc Workshop - Euler's Method};\hspace{5pt}\hyperref{https://en.wikipedia.org/wiki/Euler\_method}{}{}{Wikipedia - Euler Method}}
\end{figure}

\begin{example}
	Given that $\dd{y}{x}=3-x$ and $y(4)=2$, approximate the value of $y(5)$ using Euler's method with increments of $\Delta x = 0.25$.
\end{example}
\begin{answer}
	\begin{table}[H]
		\begin{center}
			\begin{tabular}{|c|c|c|c|c|}
				\hline
				$(x,y)$ & $\dd{y}{x}$ & $\Delta x$ & $\Delta y = \Delta x\dd{y}{x}$ & $(x+\Delta x, y+\Delta y)$ \\
				\hline
				$(4,2)$ & $-1$ & $0.25$ & $-0.25$ & $(4.25,1.75)$ \\
				\hline
				$(4.25,1.75)$ & $-1.25$ & $0.25$ & $-0.3125$ & $(4.5,1.4375)$ \\
				\hline
				$(4.5,1.4375)$ & $-1.5$ & $0.25$ & $-0.375$ & $(4.75,1.0625)$ \\
				\hline
				$(4.75,1.0625)$ & $-1.75$ & $0.25$ & $-0.4375$ & $(5,0.625)$ \\
				\hline
			\end{tabular}
		\end{center}
	\end{table}

	So, Euler's Method yields an approximate\footnote{The actual value of the solution is $(5,0.5)$, so not too far off.} value of $(5,0.625)$.
\end{answer}


\section{Support vector machines}

Key idea:
\begin{itemize}
  \item find the widest separating strip between the points
belonging to the two classes.
  \item Find a vector $w$ that is perpendicular to the median line to the
strip.
  \item For any unknown $u$, classify as positive example if
    $f(u) = w\cdot u + b \geq 0$
\end{itemize}

\subsection{Derivation of result}

Assume the following constraints:

\begin{math}
  c_i = \begin{cases}
    f(x_{+}) = w \cdot x_{+} + b \geq 1 & for \medspace x_{+} \\
    f(x_{-}) = w \cdot x_{-} + b \leq -1 & for \medspace x_{-}
  \end{cases}
\end{math}

The constraint can be generalized introducing:

\begin{math}
  y_i = \begin{cases}
    1 & x_{+} \\
    -1 & x_{-}
  \end{cases}
\end{math}

So to obtain:
\begin{math}
  c_i: y_i(w \cdot x_i + b) - 1 \geq 0
\end{math}

Also, assume that the constraint is satisfied with equality for
observations at the edge of the strip.

\paragraph{Finding the width of the strip} For any two points at
the edge of the strip, one negative and one positive observation,
it holds that:

\begin{equation}
  \begin{aligned}
    width &= (x_{+} - x_{-}) \cdot \frac{w}{\|w\|} \\
          &= (1-b) + (1+b) \\
          &= \frac{2}{\|w\|}
  \end{aligned}
\end{equation}

\paragraph{Objective} Maximize the width of the strip,
reformulated as:
\begin{equation}
  objective = \frac{1}{2}\|w\|^2
\end{equation}

\subsection{Constrained optimization}

The problem can be formulated as:

\begin{equation}
  L(w) = \frac{1}{2}\|w\|^2 -
  \sum_{i}{\alpha_{i}\left[ y_i(w \cdot x_i + b) - 1 \right]}
\end{equation}

\begin{equation}
  \begin{cases}
    \frac{\varDelta L}{w}: w - \sum{\alpha_{i}y_ix_i} = 0  \\
    \frac{\varDelta L}{b}: \sum_{i}{\alpha_{i}y_i } = 0
  \end{cases}
\end{equation}

Plugging the first f.o.c. into the Lagrangean, we obtain:
\begin{equation}
  L = \sum_i{a_i} - \frac{1}{2}
    \sum_{i}\sum_{j}{a_ia_jy_iy_j (x_i \cdot x_j)}
\end{equation}

Then check the sign of:

\begin{equation}
  f(u) = \left(\sum{a_i y_i (x_i u)}\right) + b
\end{equation}

Thus:
\begin{itemize}
  \item $w$ is a linear sum of the samples.
  \item Learning ($w$) depends only on the the dot product
    of observation pairs
  \item Classification depends only on the the dot product
    of observations with the unkown
  \item Exclusive reliance on dot products enables approach
    to problems in which samples cannot be separated by a
    straight line
\end{itemize}

When the classes are not linearly separable, handle
non-linearities by applying a transformation $\Phi$ - the
\textbf{kernel} to the dot products of the samples,
so to solve the problem in a higher dimensional space.

Common kernels are:
\begin{itemize}
  \item Polynomial: $(u \cdot v + 1)^n$
  \item Radial basis: $e^{-\frac{u-v}{\sigma}}$
\end{itemize}


\section{Composite Numerical Integration}
To calculate an aribitrary integral $\int_a^bf(x)\D x$, choose an even integer $n$, subdivide the interval $[a,b]$ into $n$ subinterval, and apply Simpson's rule on each consecutive pair of subintervals. With $h=(b-a)/h$ and $x_j=a+jh$, for $j=0,1,\ldots,n$, we have
\begin{align*}
    \int_a^bf(x)\D x &= \sum_{j=1}^{n/2}\int_{x_{2j-2}}^{x_{2j}}f(x)\D x \\
    &= \sum_{j=1}^{n/2}\left\{\frac{h}{3}\left[f(x_{2j-2})+4f(x_{2j-1})+f(x_{2j})\right]-\frac{h^5}{90}f^{(4)}(\xi_j)\right\} \\
    &= \frac{h}{3}\left[f(x_0)+2\sum_{j=1}^{n/2-1}f(x_{2j})+4\sum_{j=1}^{n/2}f(x_{2j-1})+f(x_n)\right] - \frac{h^5}{90}\sum_{j=1}^{n/2}f^{(4)}(\xi_j) \\
    &= \frac{h}{3}\left[\cdots\cdots\right]-\frac{h^5}{90}\left(\frac{n}{2}\right)f^{(4)}(\mu) \\
    &= \frac{h}{3}\left[\cdots\cdots\right]-\frac{(b-a)}{180}h^4f^{(4)}(\mu) \\
\end{align*}

\begin{theo}[Composite Simpson's Rule]
Let $f\in C^4[a,b]$, $n$ be even, $h=\frac{b-a}{n}$, and $x_j=a+jh$ for $j=0,1,\ldots,n$.
There exists a $\mu\in(a,b)$ s.t. with $n$ subintervals
\begin{align*}
\int_a^bf(x)\D x =& \frac{h}{3}\left[f(a)+2\sum_{j=1}^{n/2-1}f(x_{2j})+4\sum_{j=1}^{n/2}f(x_{2j-1})+f(b)\right] \\
&- \frac{b-a}{180}h^4f^{(4)}(\mu).
\end{align*}
\end{theo}

\begin{theo}[Composite Trapezoidal Rule]
Let $f\in C^2[a,b]$, $n$ be even, $h=\frac{b-a}{n}$, and $x_j=a+jh$ for $j=0,1,\ldots,n$.
There exists a $\mu\in(a,b)$ s.t. with $n$ subintervals
\[
\int_a^bf(x)\D x=\frac{h}{2}\left[f(a)+2\sum_{j=1}^{n-1}f(x_j)+f(b)\right] - \frac{b-a}{12}h^2f''(\mu).
\]
\end{theo}

\begin{theo}[Composite Midpoint Rule]
Let $f\in C^2[a,b]$, $n$ be even, $h=\frac{b-a}{n+2}$, and $x_j=a+(j+1)h$ for $j=-1,0,\ldots,n+1$.
There exists a $\mu\in(a,b)$ s.t. with $n+2$ subintervals
\[
\int_a^bf(x)\D x=2h\sum_{j=0}^{n/2}f(x_{2j}) + \frac{b-a}{6}h^2f''(\mu).
\]
\end{theo}

\subsection{Round-off Error Stability}
\begin{align*}
    e(h) &= \abs{\frac{h}{3}\left[e_0+2\sum_{j=1}^{n/2-1}e_{2j}+4\sum_{j=1}^{n/2}e_{2j-1}+e_n\right]} \\
    &\leq \frac{h}{3}\left[\abs{e_0}+2\sum_{j=1}^{n/2-1}\abs{e_{2j}}+4\sum_{j=1}^{n/2}\abs{e_{2j-1}}+\abs{e_n}\right] \\
    &\leq \frac{h}{3}\left[\varepsilon+2(\frac{n}{2}-1)\varepsilon+4(\frac{n}{2})\varepsilon+\varepsilon\right]=\frac{h}{3}3h\varepsilon = nh\varepsilon \\
    &= (b-a)\varepsilon.
\end{align*}
If the round-off errors are uniformly bounded by $\varepsilon$.


\subsection{Gaussian Process Regression}
\label{supp: GPR}

A \textit{Gaussian process} is a collection of random variables, any finite number of which have a joint Gaussian distribution.
We define mean function $\mu(\mathbf{x})$  and covariance function $k(\mathbf{x}, \mathbf{x}^\prime)$ of a real process $f(\mathbf{x})$ as
\begin{align}
    \mu(\mathbf{x}) &= \mathbb{E}[f(\mathbf{x})]\\
    k(\mathbf{x}, \mathbf{x}^\prime) &= \mathbb{E}[(f(\mathbf{x}) - \mu(\mathbf{x}))(f(\mathbf{x}^\prime) - \mu(\mathbf{x}^\prime))].
\end{align}

A Gaussian process is specified by its mean function and convariance function as $f(\mathbf{x}) \sim \mathcal{G} \mathcal{P}\left(\mu(\mathbf{x}), k\left(\mathbf{x}, \mathbf{x}^{\prime}\right)\right)$.
We consider the case where the observations are noisy, i.e. $\{(\mathbf{x}_i, y_i)| i = 1, \dots, n\}$, where $y_i = f(\mathbf{x}_i) + \epsilon$ with $\epsilon \sim \mathcal{N}(0, \alpha^2)$.
The Gaussian noise is independent identically distributed, and the prior on the noisy observations is then $\operatorname{cov}\left(y_{p}, y_{q}\right)=k\left(\mathbf{x}_{p}, \mathbf{x}_{q}\right)+\alpha^{2} \delta_{p q}$,
where $\delta_{pq}$ is a Kronecker delta which is one if $p = q$ and zero otherwise.
It is equivalent to a diagonal matrix $\alpha^2 I$ on the kernel matrix evaluated on the training points.

For $n_\ast$ test points $X_\ast$, we assume the prior over the functions values as a random Gaussian vector $\mathbf{f}_\ast \sim \mathcal{N}(\mathbf{0}, K(X_\ast, X_\ast))$.
Then the joint distribution of the observed target values and the function values at the test points under the prior as
\begin{align}
    \left[\begin{array}{l}\mathbf{y} \\ \mathbf{f}_{*}\end{array}\right] \sim \mathcal{N}\left(\mathbf{0},\left[\begin{array}{cc}K(X, X)+\alpha^{2} I & K\left(X, X_{*}\right) \\ K\left(X_{*}, X\right) & K\left(X_{*}, X_{*}\right)\end{array}\right]\right)
\end{align}
where $K(X, X_\ast)$ denotes the $n \times n_\ast$ covariance/Kernel matrix evaluated at all pairs of training and testing points, similarly for other kernel matrices.
Then the posterior of the test points (i.e. predictive distributions) is given by the conditional distribution $\mathbf{f}_\ast | X, \mathbf{y}, X_\ast \sim \mathcal{N}(\bar{\mathbf{f}}_\ast, cov(\mathbf{f}_\ast))$, where
\begin{align}
   \overline{\mathbf{f}}_{*} & \triangleq \mathbb{E}\left[\mathbf{f}_{*} \mid X, \mathbf{y}, X_{*}\right]=K\left(X_{*}, X\right)\left[K(X, X)+\alpha^{2} I\right]^{-1} \mathbf{y} \\
   \label{Eq: predicted variance}
   \operatorname{cov}\left(\mathbf{f}_{*}\right) &=K\left(X_{*}, X_{*}\right)-K\left(X_{*}, X\right)\left[K(X, X)+\alpha^{2} I\right]^{-1} K\left(X, X_{*}\right)
\end{align}
For noisy test targets $\mathbf{y}_\ast$, we can compute the predictive distribution by adding $\alpha^2 I$ to the variance term $cov(\mathbf{f}_\ast)$ in Eq. (\ref{Eq: predicted variance}).
% We now introduce the marginal likelihood (or evidence) $p(\mathbf{y}|X)$, is which the integral of the likelihood times the prior
% \begin{align}
%     p(\mathbf{y} \mid X)=\int p(\mathbf{y} \mid \mathbf{f}, X) p(\mathbf{f} \mid X) d \mathbf{f}
% \end{align}

\subsection{Choices of Kernels}

The choice of covariance function is critical for the performance of Gaussian process regression, we show a number of different string kernels tested in this study below:

\begin{itemize}
    \item \textit{Spectrum Kernel.}
    \begin{align}
        k_\ell^{\text{Spec}}(X, X^\prime) =\left\langle\phi_{\ell}^{\mathrm{Spec}}(\mathbf{x}), \phi_{\ell}^{\mathrm{Spec}}\left(\mathbf{x}^{\prime}\right)\right\rangle = \phi_{\ell}^{\mathrm{Spec}}(\mathbf{x})^T \phi_{\ell}^{\mathrm{Spec}}\left(\mathbf{x}^{\prime}\right).
    \end{align}
     where $\mathbf{x}, \mathbf{x}^\prime$ are two RBS sequences in $\mathcal{D}$ over an alphabet $\Sigma$. We denote the number of letters in the alphabet as $|\Sigma|$.
    $\phi_{\ell}^{\mathrm{spec}}(\mathbf{x})$ maps the sequence $X$ into a $|\Sigma|^\ell$ dimensional feature space, where each dimension is the count of the number of one of the $|\Sigma|^\ell$ possible strings $s$ of length $\ell$.
    Let $X, X^\prime$ be two metrics which include $n$ sequences, and $\Phi_d^{Spec}(X) \in \mathbb{R}^{n \times |\Sigma|^{\ell}}$, then the spectrum kernel over metrics is
    \begin{align}
         K_\ell^{\text{Spec}}(X, X^\prime) = \Phi_{\ell}^{\mathrm{Spec}}(X) \Phi_{\ell}^{\mathrm{Spec}}\left(X^{\prime}\right)^T.
    \end{align}

    \item \textit{Sum of Spectrum Kernel,} considers weighted sum over different parts of the string.

    \item \textit{Mixed Spectrum Kernel,} considers weighted sum over different substring length, with $\beta_d = \frac{2(\ell - d + 1)}{\ell(\ell+1)}$,
        \begin{align}
            k_\ell^{MixedSpec}(\mathbf{x}, \mathbf{x}^\prime)
            = \sum_{d=1}^{\ell} \beta_d k_d^{Spec}(\mathbf{x}, \mathbf{x}^\prime)
        \end{align}
    \item \textit{Weighted Degree Kernel,} considers positional information. WD kernel counts the match of kmers at corresponding positions in two sequences.
    For sequences with fixed length $L$ and weighted degree kernel considers substrings starting at each position $l = 1, ..., L$, with $\beta_d = \frac{2(\ell - d + 1)}{\ell(\ell+1)}$, \\
    \begin{align}
        k_\ell^{WD}(\mathbf{x}, \mathbf{x}^\prime)
        &= \sum_{d=1}^{\ell} \beta_d \sum_{l=1}^{L-d+1} \gamma_l k_d^{Spec}(\mathbf{x}_{[l:l+d]}, \mathbf{x}_{[l:l+d]}^\prime)\\
        &= \sum_{d=1}^{\ell} \beta_d \sum_{l=1}^{L-d+1} \gamma_l \phi_d^{Spec}(\mathbf{x}_{[l:l+d]})^T \phi_d^{Spec}(\mathbf{x}_{[l:l+d]}^\prime)\\
        &= \sum_{d=1}^{\ell} \beta_d \sum_{l=1}^{L-d+1} \gamma_l \mathbb{I}(\mathbf{x}_{[l:l+d]} = \mathbf{x}_{[l:l+d]}^\prime),
    \end{align}
    where $\mathbb{I}(\text{true}) = 1$ and 0 otherwise.

    \item \textit{Weighted Degree Kernel With Shift.}
    \begin{align}
        k_\ell^{WDS}(\mathbf{x}, \mathbf{x}^\prime)
        &= \sum_{d=1}^{\ell} \beta_d \sum_{l=1}^{L-d+1} \gamma_l \sum_{s = 0, s + l \leq L}^{S(l)} \delta_s
        \left(k_d^{Spec}(\mathbf{x}_{[l+s:l+s+d]}, \mathbf{x}_{[l:l+d]}^\prime) + (k_d^{Spec}(\mathbf{x}_{[l:l+d]}, \mathbf{x}_{[l+s:l+s+d]}^\prime)\right)\\
        &= \sum_{d=1}^{\ell} \beta_d \sum_{l=1}^{L-d+1} \gamma_l \sum_{s = 0, s + l \leq L}^{S(l)} \delta_s
        \left(\mathbb{I}(\mathbf{x}_{[l+s:l+s+d]} = \mathbf{x}_{[l:l+d]}^\prime) + (\mathbb{I}(\mathbf{x}_{[l:l+d]}= \mathbf{x}_{[l+s:l+s+d]}^\prime)\right),
    \end{align}
    where $\beta_d = \frac{2(\ell - d + 1)}{\ell(\ell+1)}, \delta_s = \frac{1}{2(s+1)}$, $\gamma_l$ is a weighting over the position in the
    sequence, where we choose to use a uniform weighting over the sequences, i.e. $\gamma_l = 1/L$. $S(l)$ determines the shift
    range at position $l$.
\end{itemize}

\textbf{From kernel to distance}:
$$d(\mathbf{x}, \mathbf{x}^\prime) = \sqrt{k(\mathbf{x}, \mathbf{x}) + k(\mathbf{x}^\prime, \mathbf{x}^\prime) - 2 k(\mathbf{x}, \mathbf{x}^\prime)} $$

\subsubsection{Normalisation of Kernel}
\label{supp: Normalisation of Kernel}

As part of data pre-processing,
the range of all features should be normalised so that each feature contributes approximately proportionately to the predictive model.
The kernel matrix is represented by the inner product of the underlying feature vectors, it needs to be normalised before being used in the downstream regression models.
Up-scaling (down-scaling) features can be understood as down-scaling (up-scaling) regularizers such that they penalise the features less (more).

Here we consider two approaches for kernel normalisation: centering and unit norm.
We will show how to convert the normalisation in terms of feature vectors to normalisation in terms of kernel matrices.
As defined before, consider $\mathbf{x}, \mathbf{x}^\prime$ are two RBS sequences in $\mathcal{D}$ over an alphabet $\Sigma$.
We denote $\phi(\mathbf{x}_i)$ as a column feature vector of sequence $\mathbf{x}_i$,
where a feature function $\phi: \mathbf{x} \rightarrow \mathbb{R}^d$. Assume there is total of $n$ sequences in the data $X$ ($n'$ sequences in the data $X'$).
We illustrate centering and unit norm normalisation below.

\begin{itemize}
    \item Centering.
    Defining the mean vector as $\bar{\Phi}(X) = \frac{1}{n} \sum_{s = 1}^n \phi(\mathbf{x}_s) \in \mathbb{R}^d$, the centered feature vector $\phi^C(\mathbf{x}_i) \in \mathbb{R}^d$ of $\mathbf{x}_i$ is
    \begin{align}
        \phi^{C}(\mathbf{x}_i) = \phi(\mathbf{x}_i) - \bar{\Phi}(X) = \phi(\mathbf{x}_i) - \frac{1}{n'} \sum_{s = 1}^{n'} \phi(\mathbf{x}_s).
    \end{align}
    The corresponding centering kernel value between $\mathbf{x}_i$ and $\mathbf{x}_j$ is then
    \begin{align}
        k^C(\mathbf{x}_i, \mathbf{x}_j) &= <\phi^C(\mathbf{x}_i), \phi^C(\mathbf{x}_j)>\\
        &= \left( \phi(\mathbf{x}_i) - \frac{1}{n} \sum_{s = 1}^n \phi(\mathbf{x}_s)\right)^T \left( \phi(\mathbf{x}_j) - \frac{1}{n'} \sum_{s' = 1}^{n'} \phi(\mathbf{x}_{s'})\right)\\
        &= \phi(\mathbf{x}_i)^T \phi(\mathbf{x}_j) - \left( \frac{1}{n} \sum_{s = 1}^n \phi(\mathbf{x}_s)\right)^T \phi(\mathbf{x}_j) - \phi(\mathbf{x}_i)^T \left(\frac{1}{n} \sum_{s' = 1}^{n'} \phi(\mathbf{x}_{s'})\right) + \left( \frac{1}{n} \sum_{s = 1}^n \phi(\mathbf{x}_s)\right)^T \left(\frac{1}{n'} \sum_{s' = 1}^{n'} \phi(\mathbf{x}_{s'})\right)\\
        &= k(\mathbf{x}_i, \mathbf{x}_j) - \frac{1}{n} \sum_{s=1}^n k(\mathbf{x}_s, \mathbf{x}_j) - \frac{1}{n'} \sum_{s'=1}^{n'} k(\mathbf{x}_i, \mathbf{x}_{s'}) + \frac{1}{n^2} \sum_{s = 1}^n \sum_{s'=1}^{n'} k(\mathbf{x}_s, \mathbf{x}_{s'})
    \end{align}

    \item Unit Norm. Define the ($l_2$) norm of a feature vector as $||\phi(\mathbf{x})|| = \sqrt{\sum_{m = 1}^d \phi_d(\mathbf{x})^2} = \sqrt{k(\mathbf{x}, \mathbf{x})} \in \mathbb{R}^+$, then the unit norm feature vector $\phi^{UN}(\mathbf{x}_i) \in \mathbb{R}^d$ of $\mathbf{x}_i$ is
    \begin{align}
        \phi^{UN}(\mathbf{x}_i) = \frac{\phi(\mathbf{x}_i)}{||\phi(\mathbf{x}_i)||}.
    \end{align}
    The corresponding unit norm kernel value between $\mathbf{x}_i$ and $\mathbf{x}_j$ is then
    \begin{align}
        k^{UN}(\mathbf{x}_i, \mathbf{x}_j) &= <\frac{\phi(\mathbf{x}_i)}{||\phi(\mathbf{x}_i)||}, \frac{\phi(\mathbf{x}_j)}{||\phi(\mathbf{x}_j)||}>\\
        &= \frac{\phi(\mathbf{x}_i)^T \phi(\mathbf{x}_j)}{||\phi(\mathbf{x}_i)|| \times ||\phi(\mathbf{x}_j)||}\\
        &= \frac{k(\mathbf{x}_i, \mathbf{x}_j)}{\sqrt{k(\mathbf{x}_i, \mathbf{x}_i)  k(\mathbf{x}_j, \mathbf{x}_j)}}
    \end{align}

     \item Unit Variance.
    After the centering and unit norm normalisation, the kernel matrix is unit variance as well.
    In the following, we show transformations of the unit variance (with centering) normalisation.
    Define the variance vector ${Var}(\Phi(X)) = \frac{1}{n} \sum_{s=1}^n ||\phi(\mathbf{x}_s) - \bar{\Phi}(X)||^2 = \frac{1}{n} \sum_{s=1}^n ||\phi(\mathbf{x}_s) - \sum_{s'=1}^n \left(\phi(\mathbf{x}_s')\right)||^2 = \frac{1}{n} \sum_{s=1}^n  k^C(\mathbf{x}_s, \mathbf{x}_s)  \in \mathbb{R}$, the unit variance feature vector $\phi^{UV}(\mathbf{x}_i) \in \mathbb{R}^d$ of $\mathbf{x}_i$ is
    \begin{align}
        \phi^{UV}(\mathbf{x}_i) = \frac{\phi(\mathbf{x}_i)}{\sqrt{Var(\Phi(X))}}.
    \end{align}
    The corresponding kernel representation is
    \begin{align}
        k^{UV}(\mathbf{x}_i, \mathbf{x}_j) &= <\frac{\phi(\mathbf{x}_i)}{\sqrt{Var(\Phi(X))}}, \frac{\phi(\mathbf{x}_j)}{\sqrt{Var(\Phi(X'))}}>\\
        &= \frac{\phi(\mathbf{x}_i)^T \mathbf{x}_j}{\sqrt{Var(\Phi(X)) Var(\Phi(\mathbf{X'}))}}\\
        &= \frac{k(\mathbf{x}_i, \mathbf{x}_j)}{\sqrt{ \frac{1}{n} \sum_{s=1}^n  k^C(\mathbf{x}_s, \mathbf{x}_s)  \frac{1}{n} \sum_{s'=1}^{n'}  k^C(\mathbf{x}_{s'}, \mathbf{x}_{s'})}}
    \end{align}
    After centering and unit norm, $ \frac{1}{n} \sum_{s=1}^n  k^C(\mathbf{x}_s, \mathbf{x}_s) = k(\mathbf{x}_i, \mathbf{x}_i)$, which implies that after centering and unit norm, the kernel matrix is already unit variance normalised.
\end{itemize}
For the Gaussian Process regression, we make of use of two kernel matrices: the kernel function between the training data itself, i.e. $K(X_{train}, X_{train})$; and
the kernel function taking the training data and testing data as inputs, i.e. $K(X_{test}, X_{train})$.
%It is straightforward to normalise a square kernel which same input, i.e. $n = n'$ and $k(\mathbf{x}_i, \mathbf{x}_i),  k(\mathbf{x}_j, \mathbf{x}_j)$ taken from the diagonal of the matrix.
%The second one (between train and test) is a little bit tricky. The different is not only that $n \neq n'$.
We will state two ways of normalisation those two kind of matrices:
\begin{itemize}
    \item Normalise training and testing data separately.
    This approach is preferred for most of the machine learning algorithms since it follows the rule that we have no information about testing data while training.
    Then for centering, one should subtract the mean vector over the training data for both kinds of matrices.
    For unit norm normalisation, when one calculates $K^{UN}(X_{test}, X_{train})$, the two terms inside of square root: $k(\mathbf{x}_i, \mathbf{x}_i)$ is taken from $K(X_{test}, X_{test})[i,i]$, and $k(\mathbf{x}_j, \mathbf{x}_j)$ is taken from $K(X_{train}, X_{train})[j,j]$.

    \item Normalise training and testing data together, i.e. normalise $K(X_{train+test}, X_{train+test})$, then extra the parts we need from the normalised matrix.
    This approach is suitable in a case where one already knows the whole of testing features.
    For centering, one should subtract the mean vector over the whole matrix $\Phi(X_{train+test})$.
    The unit norm normalisation is the same as in the previous case.
\end{itemize}

For our experiment, we fix the design space before training, i.e. the testing features are already known before testing.
So we choose to normalise the kernel matrix over the training and testing data together,
by first applying centering and then unit norm normalisation.

\subsection{Batch Recommendation}

For recommending RBS sequences to label, we consider the Upper Confidence Bound (UCB) algorithm,
%which is based on the \textit{optimism in the face of uncertainty},
selecting RBS sequences with the maximum upper confidence bound at round $t$, i.e.
\begin{align}
    \operatorname{argmax}_{\mathbf{x}_i \in \mathcal{D}} \left( \mu_{t-1}(\mathbf{x}_i) + \beta_t \sigma_{t-1}(\mathbf{x}_i)\right),
    \tag{\ref{Eq: GPUCB}}
\end{align}
where $\beta_t$ is a hyperparmeter balancing the exploitation and exploration,
$\mu_t(\mathbf{x}_i), \sigma_t(\mathbf{x}_i)$ are the predicted mean and standard deviation at round $t$ for the sequence $\mathbf{x}_i$.

Since labelling sequences is time-consuming, it is unrealistic to recommend sequence sequentially (i.e. one-by-one) and waiting for the label after each prediction.
Therefore we consider recommending sequences in batch and using Gaussian Process Batch Upper Confidence Bound (GP-BUCB) algorithm  \cite{desautels2014parallelizing}.
With batches of size $B$, the feedback mapping $fb[t] = \lfloor(t-1) / B\rfloor B$, i.e.
\begin{align}
    \mathrm{fb}[t]=\left\{\begin{array}{cl}
    0 & : t \in\{1, \ldots, B\} \\
    B & : t \in\{B+1, \ldots, 2 B\} \\
    2 B & : t \in\{2 B+1, \ldots, 3 B\} \\
    & \vdots
    \end{array}\right.
\end{align}

A key property of Gaussian Process regression is that the predictive variance in Eq. (\ref{Eq: predicted variance}) only depends on observed points (i.e. features), but not on the labels of these observed points.
So one can compute the posterior variance without actually observing the labels.
The GP-BUCB policy is to select sequences that
\begin{align}
    \operatorname{argmax}_{\mathbf{x}_i \in \mathcal{D}} \left( \mu_{fb[t]}(\mathbf{x}_i) + \beta_t \sigma_{t-1}(\mathbf{x}_i)\right).
\end{align}
And only update $y_{t^{\prime}}=f\left(\boldsymbol{x}_{t^{\prime}}\right)+\varepsilon_{t^{\prime}} \text { for } t^{\prime} \in\{\mathrm{fb}[t]+1, \ldots, \mathrm{fb}[t+1]\}$ at the end of each batch ($\mathrm{fb}[t]<\mathrm{fb}[t+1]$).
This is equivalent to sequential GP-UCB with \textit{hallucinated observations} $\boldsymbol{y}_{\mathrm{fb}[t]+1: t-1}=\left[\mu_{\mathrm{fb}[t]}\left(\boldsymbol{x}_{\mathrm{fb}[t]+1}\right), \ldots, \mu_{\mathrm{fb}[t]}\left(\boldsymbol{x}_{t-1}\right)\right]$, while the posterior variance decreases.

\subsection{Machine Learning Design Pipeline}
\label{sec: ML design pipeline}
In this section, we report in details about our machine learning design pipeline.
The raw TIR is calculated as a derivative of GFP fluoresence divided by OD600 of culture over 4h counting from the start of log phase of growth.

\mengyan{explain somewhere why we have a different pipeline for round 0.}
\mengyan{add some analysis on literature data?
to verify 1) our library is "better" 2) it's hard to do ML on available library. some analysis in my mind (for literature data): number of data points, number of replicates (I remembered there only one replicates, if not, then std), label histogram, coverage of RBS.
And we need to address the label in literature data and ours is "different", in the way that our pipeline is different for bandit-0 design}

\textbf{Bandit-0: } the design of round-0 is based on the literature data \cite{jervis2018machine}.
We first normalise the raw TIR to values between 0 and 1.
We applied the Gaussian Process Regression with noise parameter $\alpha = 1e-10$.
We chose to use one of the basic string kernels, the \textit{spectrum kernel} \cite{leslie2001spectrum} to process the core 6bp and dot product kernel \cite{Rasmussen2004} (with one-hot embedding) to process the 7bp flanking sequences both upstream and downstream of the core sequence.
The design size is 60 with UCB parameter $\beta = 1$.\\
\textbf{Bandit-1 to Bandit-3: }
the t + 1 round design is based on the $t^th$ ($t \geq 1$) round result, where each sequence has 6 replicates with TIR labels.
We pre-processed the data by taking a logarithm transformation and standardisation of the raw TIR label for each replicates respectively.
After normalisation, each replicate has zero mean and unit variance.
For prediction, we use Gaussian process regression, with training on all normalised replicates and predicting on the design space (6-base core part design) except known sequences.
We assume the observation are noisy, where the noise is under centered normal distribution with standard deviation $\alpha$.
We model the covariance matrix using the weighted degree kernel with shift.
We normalise the kernel with centering and unit norm in terms of the whole kernel constructed by both first round result and design space.
The hyperparameter for kernel, including maximum substring length  $l$, maximum shift length $s$, and the noise standard deviation $\alpha$ of Gaussian process model are choose based on 10-repeat 5-fold cross validation.
We choose $l=6, s= 1, \alpha = 2$ for the second round design.
For recommendation, we use batch upper confidence bound introduced by GP-BUCB algorithm \cite{desautels2014parallelizing}.
The upper confidence bound is constructed by predicted mean plus 2 predicted standard deviation.
We recommend 90 sequences from the design space.

\subsection{Statistics of the core vs non-core regions of the RBS}

In our study, we have tested a set of sequnces designed to confirm the notion that changing bases within the RBS core (6 bases) is statistically more influencing the TIR than the changes made outside the core.
This hypothesis has been build based on reported biases towards certain bases present in the core of the RBS but absent outside of it.
For example, according to \cite{SHULTZABERGER2001} there is a strong bias towards A and G bases in the core region of the RBS.
Similarly outside of the 6 bases of the core in the wider 20 bp context of the RBS there is no significant bias towards any particular base which suggest that these bases do not contribute to the overal TIR of a given RBS.
This effect is shown in (Figure \ref{fig:core_vs_noncore}), which shows results for our set X of sequences.
The value of Welch's t-test between the mean TIR in core and non-core groups is -4.8780 with p-value $<$ 0.0001 and 34 degrees of Freedom.

\begin{figure}[!ht]
    \centering
    \includegraphics[scale=0.4]{plots/Supplementary/core_vs_noncore.pdf}
    \caption{\textbf{Comparison of base change impact on TIR in core versus non-core region.} The core region is highligthed in light green and the lines are rolling averages for each base. The top dotted line shows the TIR for the benchmark sequence, where dots represent a change at a given position to a given base, which is colour coded.}
    \label{fig:core_vs_noncore}
\end{figure}

\subsection{Other Evaluation}

\textbf{How much more efficient we are by using GP-UCB?}

random = {set of TIR that are in UNI and PPM}
bandit = {set of TIR that are in bandit 0,1,2,3}

The ratio of the max random to the max bandit: 0.67/1.34 = 0.5
The ratio of the 0.9 quantile random to the 0.9 quantile bandit: 0.49/0.82 = 0.598

we can also calculate an average regret of TIRs which are higher than SD? The regret for one sequences is calculated based on the difference between the benchmark (1) and TIR.

reciprocal would make a better story. I.e. bandit is 2x better than random.

\textbf{zoom-in tSNE?}

1) supplementary website.
2) eLife is now offering submissions of interactive papers with Jupyter notebooks. More of a trivia than useful information in our case.
https://elifesciences.org/labs/ad58f08d/introducing-elife-s-first-computationally-reproducible-article

\subsection{Mixture of Absolute Gaussian}

The raw TIR scores in each round are positive and  have two modes.
We assume they are mixture of two absolute Gaussian distributions.
If we can fit the mixture of absolute Gaussian distributions, we can answer the following questions:
\begin{itemize}
    \item is the left peak decreasing in height with each round?
	\item is the left peak going left?
	\item Is the right peak increasing in height with each round?
	\item is the right peak going right?
\end{itemize}

The density of absolute Gaussian (also called \textit{folded normal distribution}) is
\begin{align}
    f_{Y}\left(x ; \mu, \sigma^{2}\right)=\frac{1}{\sqrt{2 \pi \sigma^{2}}} e^{-\frac{(z-\mu)^{2}}{2 \sigma^{2}}}+\frac{1}{\sqrt{2 \pi \sigma^{2}}} e^{-\frac{(x+\mu)^{2}}{2 \sigma^{2}}}
\end{align}
for $x \geq 0$ and $0$ everywhere else.
The mean and variance of the absolute Gaussian distribution is
\begin{align}
    \mu_{Y} &=\sigma \sqrt{\frac{2}{\pi}} \exp \left(\frac{-\mu^{2}}{2 \sigma^{2}}\right)+\mu \operatorname{erf}\left(\frac{\mu}{\sqrt{2 \sigma^{2}}}\right);\\
    \sigma_{Y}^{2} &=\mu^{2}+\sigma^{2}-\mu_{Y}^{2},
\end{align}
where $\operatorname{erf}$ is the error function
$\operatorname{erf} z=\frac{2}{\sqrt{\pi}} \int_{0}^{z} e^{-t^{2}} d t$.

We consider the model of the mixture of two absolute Gaussian distributions as

\begin{figure}[!ht]
    \centering
    \includegraphics[scale = 0.25]{plots/Supplementary/clustering.png}
    \caption{TSNE of RBS sequences in design space with clustering. The distance is calculated based on the weighted degree kernel on RBS sequences. Colours indicate different groups, shapes indicate different clusters.}
\end{figure}


\subsection{Leverage}

\subsubsection{Introduction}

Leverage measures how much the predicted value of \(y_i\), \(\hat y_i\), changes as \(y_i\) changes.

We have:

\(\mathbf y = \mathbf X \theta +\mathbf u\)

\(\hat \theta =X(X^TX)^{-1}X^Ty\)

\(\hat \theta =P_Xy\)

The leverage score is defined as:

\(h_i=P_{ii}\)


\chapter{Tensor Product Spaces}
So far we have considered the tensor product space $V^* \otimes V^*$, constructed from a vector space $V$ over a field $\mathbb{F}$.
Elements of this tensor product space map from $V \times V$ to $\mathbb{F}$, and has $16$ basis vectors $\vec{e}^\mu \otimes \vec{e}^\nu$, meaning any given tensor can be written as $T_{\mu\nu} \vec{e}^\mu \otimes \vec{e}^\nu$.
We can (almost trivially) generalize this specific type of tensor product space to arbitrary combinations of $V$ and $V^*$.
This works exactly as you think it does; if we want to map from $V \times V \times V$ to $\mathbb{F}$, we construct a tensor in the following way:
\[ T_{\lambda\mu\nu} \in V^* \otimes V^*\otimes V^* : V \times V \times V \to \mathbb{F}. \]
We can do the exact same thing if we want to map 4-tuples of vectors in $V$ to the real numbers; such a tensor would be an element of the tensor product space $V^* \otimes V^* \otimes V^* \otimes V^*$, and would have dimension $n^4$, where $n = \dim(V)$. This is a prime example of why Einstein summation is so useful; once we start talking about tensor product spaces with multiple constituent parts, we start needing to sum over lots of different indices; the tensor $T_{\lambda\mu\nu}$ would more properly be written as
\[ \sum_{i=0}^{\lambda}\sum_{j=0}^{\mu}\sum_{k=0}^{\nu} T_{\lambda\mu\nu} \vec{e}^\lambda \otimes \vec{e}^\mu \otimes \vec{e}^\nu, \]
which graphically represents the exact same information as the condensed form as long as we've agreed upon what basis to use beforehand.

\subsection{Types of Simple Tensors}
So far, we've dealt with tensors which are elements of the $n$\textsuperscript{th} tensor product of the dual space; they map $n$-tuples of \emph{vectors} to a scalar field.
However, there is nothing stopping us from considering tensors which map $n$-tuples of \emph{covectors} to a scalar field.
Such tensors would be elements of the $n$\textsuperscript{th} tensor product of the vector space.
\begin{align*}
    T_{\mu\nu} &\in V^* \otimes V^* : V \times V \to \mathbb{F} \tag{vectors $\to$ scalar field} \\
    T^{\mu\nu} &\in V \otimes V : V^* \times V^* \to \mathbb{F} \tag{covectors $\to$ scalar field}
\end{align*}
Notice that tensors which map tuples of covectors to a scalar field are identified with superscripts, not subscripts.
This is because the basis for such a tensor is the tensor product of basis vectors of $V$, not $V^*$, and so to sum over them, we must identify the components by upper indicies, not lower, as shown below:
\[ T^{\mu\nu} \equiv T^{\mu\nu} \vec{e}_\mu \otimes \vec{e}_\nu. \]
Let's compare how these tensors act on their arguments. First, the ``regular'' tensors we've dealt with up until this point, which map from vectors to the scalar field. Let $T_{\mu\nu}$ be such a tensor, build from $V$ and $\mathbb{F}$, and let it act on the two vectors $A^\alpha \vec{e}_\alpha,\,B^\beta \vec{e}_\beta \in V$.
\begin{align*}
    T_{\mu\nu} \vec{e}^\mu \otimes \vec{e}^\nu \left( A^\alpha \vec{e}_\alpha, B^\beta \vec{e}_\beta \right) &= T_{\mu\nu} \langle \vec{e}^\mu, A^\alpha \vec{e}_\alpha \rangle \langle \vec{e}^\nu, B^\beta \vec{e}_\beta \rangle \\
    &= T_{\mu\nu} \left( A^\alpha \delta^\mu_\alpha \right) \left( B^\beta \delta^\nu_\beta \right) \\
    &= T_{\mu\nu}A^\mu B^\nu.
\end{align*}
Next, consider the tensor $T^{\mu\nu}$, built from the same $V$ and $\mathbb{F}$ as before. Let $A_\alpha \vec{e}^\alpha,\,B_\beta \vec{e}^\beta \in V^*$ be the two arguments for this tensor.
\begin{align*}
    T^{\mu\nu} \vec{e}_\mu \otimes \vec{e}_\nu \left(A_\alpha \vec{e}^\alpha,B_\beta \vec{e}^\beta\right) &= T^{\mu\nu} \langle A_\alpha \vec{e}^\alpha, \vec{e}_\mu \rangle \langle B_\beta \vec{e}^\beta, \vec{e}_\nu \rangle \\
    &= T^{\mu\nu} \left(A_\alpha \delta^\alpha_\mu\right) \left(B_\beta \delta^\beta_\nu\right) \\
    &= T^{\mu\nu}A_\mu B_\nu.
\end{align*}
The two tensors act in almost identical ways, which makes sense as $V$ are $V^*$ are so tightly coupled.
Notice also that, if we start with a tensor whose components have two lower indices ($T_{\mu\nu}$), we end up summing over the product of it and coefficients with \emph{matching upper} indices ($A^\mu B^\nu$); the same goes for tensors with upper indices; they get coefficients with \emph{matching lower} indices.

\subsection{Mixed Tensors}
At this point, we've considered tensors which map from the Cartesian product of vectors spaces to a scalar field, and we've considered tensors which map from the Cartesian product of dual spaces to a scalar field; we will now combine them to get a \emph{mixed tensor}, which maps $n$-tuples of vectors and covectors to a field.

Let's start with our trusty vector space $V$ and its dual space $V^*$, both defined over a scalar field $\mathbb{F}$, and construct the following two tensors:
\begin{align}
    \tensor{T}{_{\mu}^\nu} \vec{e}^\mu \otimes \vec{e}_\nu \in V^* \otimes V : V \times V^* \to \mathbb{F} \\
    \tensor{T}{^{\mu}_{\nu}} \vec{e}_\mu \otimes \vec{e}^\nu \in V \otimes V^* : V^* \times V \to \mathbb{F}\label{eqn:correct-notation}
\end{align}
The first maps a vector and a covector ordered pair to $\mathbb{F}$; the second maps a covector and a vector ordered pair to $\mathbb{F}$.
Although they look very similar, they are \emph{entirely different tensors}.
The order of the arguments and the order of their definition matters.
It's also important to note that the scalar components of the tensor convey this information; the order of the indices indicates the order of the tensor products, so
\begin{align*}
    \tensor{T}{_{\mu}^\nu} \implies \text{a basis of }\vec{e}^\mu \otimes \vec{e}_\nu, \\
    \tensor{T}{^{\mu}_\nu} \implies \text{a basis of }\vec{e}_\mu \otimes \vec{e}^\nu.
\end{align*}
Let's take a closer look at how a tensor similar to \eqref{eqn:correct-notation} would operate.
It's first argument must be a covector (remember, linear map) of the form $A_\alpha e^\alpha$, while its second argument must be a vector, like $B^\beta e_\beta$.
\begin{align*}
    \tensor{T}{^{\mu}_{\nu}} \vec{e}_\mu \otimes \vec{e}^\nu \qty(A_\alpha \vec{e}^\alpha, B^\beta \vec{e}_\beta) &= \tensor{T}{^{\mu}_{\nu}} \langle A_\alpha \vec{e}^\alpha, \vec{e}_\mu \rangle \langle \vec{e}^\nu, B^\beta \vec{e}_\beta \rangle \\
    &= \tensor{T}{^{\mu}_{\nu}} \qty(A_\alpha \delta^\alpha_\mu) \qty(B^\beta \delta_\beta^\nu) \\
    &= \tensor{T}{^{\mu}_{\nu}} A_\mu B^\nu
\end{align*}
Notice how the order of the remaining coefficients matches that of the tensor's indices: first, $\mu$ and then $\nu$.


\section{Clustering}

\subsection{Clustering}
\subsection{A clustering Objective}
$G_{j}= \{ i| c_{i}=j \}$ where $G_{j}$ is set of all indices i for which $c_{i}=j$\\
Group representatives: We want each rep to be close to the vectors, i.e $||x_{i}-z_{c_{i}}||$ where $x_{i}$ is in group $j=c_{i}$ so $z_{c_{i}}$ is the representative vector associated with data vector $x_{i}$ \\
$J^{clust}=(||x_{1}-z_{c_{1}}||^2 + ...+||x_{N}-z_{c_{N}}||^2)/N$\\
Partitioning vectors with representatives fixed: $||x_{i}-z_{c_{i}}||=\min\limits_{j=1,....,k} ||x_{i}-z_{j}||$\\
$J^{clust}=\Big(\min\limits_{j=1,....,k} ||x_{1}-z_{j}||^{2}+....+\min\limits_{j=1,....,k} ||x_{N}-z_{j}||^{2}\Big)/N$\\
Optimizing group representatives with assignment fixed: $J^{clust}=J_{1}+...+J_{k}$ where $J_{j}=\big(1/N\big)\sum\limits_{i\in G_{j}}||x_{i}-z_{j}||^{2}$\\
To minimize mean square distance $z_{j}=\big(1/|G_{j}|\big))\sum\limits_{i\in G_{j}}x_{i}$ where $|G_{j}|$ is standard notation for number of elements in set $G_{j}$
\subsection{The k-means algorithm}
\subsection{Examples}
\subsection{Applications}


\subsection{\(L_p\) norms (\(p\)-norms)}
\subsubsection{\(L^P\) norm}

This generalises the Euclidian norm.

\(||x||_p=(\sum_{i=1}^{n}|x|^p_i)^{1/p}\)

This can defined for different values of \(p\). Note that the absolute value of each element in the vector is used.

Note also that:

\(||x||_2\)

Is the Euclidian norm.

\subsubsection{Taxicab norm}

This is the \(L^1\) norm. That is:

\(||x||_1=\sum_{i=1}^{n}|x|_i\)

\subsubsection{Angles}

\subsubsection{Cauchy-Schwarz}


\subsection*{Ex.29.4-1}

We have the following linear program
$$
\begin{aligned}
 & \max & 18x_1 && + && 12.5x_2 &
\\
 & \ \text{s.t.} & x_1 && + && x_2 & \leq 20
\\
 & & x_1 && && & \leq 12
\\
 & & && && x_2 & \leq 16
\\
 & & && && x_1,x_2 & \geq 0
\end{aligned}
$$
To form the dual, we need to
\\
- change the maximization to a minimization,
\\
- switch the coefficients of the objective function and the coefficients on the right-hand sides of the inequalities, and
\\
- replace every $\leq$ with $\geq$.

The original linear program (in standard form) can be rewritten in a compact form as
$$
\begin{aligned}
 & \max & c^Tx &
\\
 & \ \text{s.t.} & Ax & \leq b
\\
 & & x & \geq 0,
\end{aligned}
$$
where $c=\begin{pmatrix}18 \\12.5 \end{pmatrix}$, $A=\begin{pmatrix}1 & 1 & 0 \\1 & 0 & 1 \end{pmatrix}$, $b=\begin{pmatrix}20 \\12 \\16 \end{pmatrix}$, and $x=\begin{pmatrix}x_1 \\x_2 \end{pmatrix}$.

These vectors and matrix are exactly what we need for the dual:
$$
\begin{aligned}
 & \min & 20y_1 && + && 12y_2 && + && 16y_3 &
\\
 & \ \text{s.t.} & y_1 && + && y_2 && && & \geq 18
\\
 & & y_1 && && && + && y_3 & \geq 12.5
\\
 & & && && && && y_1,y_2,y_3 & \geq 0
\end{aligned}
$$


\subsection{Kernels and images}

The kernel of a linear operator is the set of vectors such that:

\(Mv=0\)

The kernel is also called the nullspace.

This can be shown as \(\ker (M)\)

The image of a linear operator is the set of vectors \(w\) such that:

\(Mv=w\).

This can be shown as \(\Im (M)\)

We also know that:

\(span (M)=\ker (M)+\Im (M)\)


\section*{Ex.2.4}
\subsection*{What is the expected number of elements in $L[h(x)]$ if $x \in S$}

The expected number of elements is
\begin{align*}
E\left[ \abs*{L[(h(x))]} \right]
       &= \sum_{y\in S} P(h(y) = h(x)) \\
       &= 1 + \sum_{y\in S\backslash\{x\}} P(h(y) = h(x)) \\
       &\leq 1 + \frac{n-1}{m} < 2
\end{align*}
The first equality is the same as in the notes. For the next equality we used the assumption that $x\in S$, which will give a collision with probability 1. Hence that element is extracted. The last equality comes from the fact that $h$ is universal hence each collision probability is less than $1/m$ and there are $n-1$ elements in the sum.

\emph{What is the bound if $h$ is 2 universal?}
\\
The calculations would be the same, the only thing that will change is where the inequality sign is. For 2 universal the probability is $P(h(x)=h(y))\leq \frac{2}{m}$. There are $n-1$ of those probabilities in the sum, hence the result would be
$$
E\left[ \abs*{L[(h(x))]} \right]  \leq 1 + \frac{2(n-1)}{m}
$$


\subsection{Tensor contraction}

We have a vector \(v\in V\) and \(w\in V^*\).

\(\mathbf v=\sum_i v^i \mathbf e_i\)

\(\mathbf w=\sum_i w_i \mathbf f^i\)

\(\mathbf w\mathbf v=[\sum_i v^i \mathbf e_i][\sum_i w_i \mathbf f^i]\)

\(\mathbf w\mathbf v=\sum_i \sum_j [v^i \mathbf e_i][w_j \mathbf f^j]\)

\(\mathbf w\mathbf v=\sum_i \sum_j v^i w_j \mathbf e_i\mathbf f^j\)

We use the dual basis so:

\(\mathbf w\mathbf v=\sum_i \sum_j v^i w_j \mathbf e_i\mathbf e^j\)

\(\mathbf w\mathbf v=\sum_i \sum_j v^i w_j \delta_i^j\)

We can see that this value is unchanged when there is a change in basis.

What if these were both from \(V\)?

\(\mathbf v=\sum_i v^i \mathbf e_i\)

\(\mathbf w=\sum_i w^i \mathbf e_i\)

\(\mathbf w\mathbf v=[\sum_i v^i \mathbf e_i][\sum_i w^i \mathbf e_i]\)

\(\mathbf w\mathbf v=\sum_i \sum_j v^i w^j \mathbf e_i\mathbf e_i\)

This term is dependent on the basis, and so we do not contract.

So if we have \(v_iw^i\), we can contract, because the result (calculated from the components) does not depend on the basis.

But if we have \(v_iw_i\), the result (calcualted from the components) will change depending on the choice of basis.

We define a new object

\(c=\sum_i w^iv_i\)

This new term, c, does not depend on \(i\), and so we have contracted the index.


\subsection{Random variables}

\subsubsection{Defining variables}

We have a sample space, \(\Omega \). A random variable \(X\) is a mapping from the sample space to the real numbers:

\(X: \Omega \rightarrow \mathbb{R}\)

We can then define the set of elements in \(\Omega \). As an example, take a coin toss and a die roll. The sample space is:

\(\{H1,H2,H3,H4,H5,H6,T1,T2,T3,T4,T5,T6\}\)

A random variable could give us just the die value, such that:

\(X(H1)=X(T1)=1\)

We can define this more precisely using set-builder notation, by saying the following is defined for all	 \(c\in \mathbb{R}\):

\(\{\omega |X(\omega )\le c\}\)

That is, for any number random variable map \(X\), there is a corresponding subset of \(\Omega \) containing the \(\omega \)s in \(\Omega \) which map to less than \(c\).

\subsubsection{Multiple variables}

Multiple variables can be defined on the sample space. If we rolled a die we could define variables for

\begin{itemize}
\item Whether it was odd/even
\item Number on the die
\item Whether it was less than 3
\end{itemize}

With more die we could add even more variables

\subsubsection{Derivative variables}

If we define a variable \(X\), we can also define another variable \(Y=X^2\).


\section{Implementation}
\label{sec:implementation}

\subsection{Dataset Generation}
We use an artificially generated dataset with $P$ examples and $N$ features with normally distributed values.
The labels are not chosen randomly, but determined using a teacher weights vector $\bm{\mathsf{w}}^*$ as $S^\mu = sign(\mathsf{w}^{*} \cdotp \xi^\mu)$.
The obtained dataset is by construction linearly separable for $\bm{\mathsf{w}} = \bm{\mathsf{w}}^*$.

We chose $\bm{\mathsf{w}}^*$ such that its norm is constant, for example $||\bm{\mathsf{w}}^*||^2 = N$.
Since we initialize $\bm{\mathsf{w}} = \overrightarrow{0}$, we have that the distance between the student and the teacher perceptron is constant:
\begin{gather*}
    \bm{\mathsf{w}} = \overrightarrow{0} \in \mathbb{R}^N \\
    distance(\bm{\mathsf{w}}, \bm{\mathsf{w}}^*) = || \bm{\mathsf{w}} - \bm{\mathsf{w}}^* || = || \bm{\mathsf{w}}^*|| = \sqrt{N}
\end{gather*}
For simplicity, we chose $\bm{\mathsf{w}^*} = \overrightarrow{1} \in \mathbb{R}^N$.

\subsection{Perceptron Training}
To implement the MinOver algorithm we started from the code written for the Rosenblatt perceptron in the last assignment.
As for the Rosenblatt perceptron, we initialize the weights vector of the student perceptron with zeros.
Then, we run the training procedure for number of epochs (where $epochs = n_{max} * P$).
At each epoch, we compute the stability of each example in the dataset (for performance reasons and to avoid the division by the norm of $\bm{\mathsf{w}}$, we parallelize the computation as matrix operations instead of iterating over the elements in the dataset) and update the weights using the example with the minimum stability.
We terminate the training if the last update of the weights (normalized by the current norm) is smaller than a certain threshold (we chose $min_{update} = 0.001$), since the update is in practice never equal to zero for numerical problems.

The algorithm differs from the Rosenblatt's one since it performs an update at each step, whether the perceptron is already giving the correct classification or not (it could not have reached the optimum stability).

\subsection{Experiments}
For a fixed value of $P$ and $N$, $n_D$ independent datasets are generated.
A new perceptron is trained on each dataset for at most $epochs = n_{max} * P$ epochs.
For each dataset, we compute the generalization error $\epsilon_g(t_{max})$ as:
\begin{equation}
    \epsilon_g(t_{max}) = \frac{1}{\pi} \arccos \bigg(\frac{\bm{\mathsf{w}}(t_{max})\;\cdotp \bm{\mathsf{w}}^*}{|| \bm{\mathsf{w}}(t_{max}) || \cdot || \bm{\mathsf{w}}^* ||} \bigg)
    \label{eq:generalization_error}
\end{equation}
$\epsilon_g(t_{max})$ measures the distance of the student from the teacher perceptron and gives an indication on the accuracy for the classification of previously unseen data.

Multiple experiments are run for different values of $P$ and $N$ in order to compute $\epsilon_g(t_{max})$ as a function of $\alpha = P / N$.
In other words, we study the accuracy of the perceptron in classifying previously unseen inputs as a function of the rate between the number of examples and the dimension of the input.

\subsection{Noisy dataset}
In real classification problems the data contains some noise.
It is therefore interesting to investigate the performances of the MinOver and Rosenblatt training algorithms on noisy data.
The implementation consists in a small change of the function that generates the dataset (\cref{code:dataset_noise}).
We generate $P$ random values and then define a noise vector which contains a label $-1$ if the the randomly generated number is smaller than $\lambda$ and $+1$ otherwise.
The real labels are then multiplied element-wise with the noise vector.


\subsection{Zero to the zero power}
The following example draws a graph of the function $f(x)=|x^x|$.
The graph shows why the convention $0^0=1$ makes sense.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
f(x) = abs(x^x)
xrange = (-2,2)
yrange = (-2,2)
draw(f,x)
\end{Verbatim}

\begin{center}
\includegraphics[scale=0.2]{zerozero.png}
\end{center}

We can see how $0^0=1$ results in a continuous line through $x=0$.
Now let us see how $x^x$ behaves in the complex plane.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
f(t) = (real(t^t),imag(t^t))
xrange = (-2,2)
yrange = (-2,2)
trange = (-4,2)
draw(f,t)
\end{Verbatim}

\begin{center}
\includegraphics[scale=0.2]{zerozero2.png}
\end{center}


%
% Chapter 2.1
%

\section*{2.1 Derivative and Rates of Change}

\subsection*{Tangents}

The \textbf{tangent line} to the curve \(y=f(x)\) at the point \(P(a, f(a))\) is the line through \(P\) with slope
$$m=\lim_{x \to a}\frac{f(x)-f(a)}{x-a} \Leftrightarrow m=\lim_{h \to 0}\frac{f(a+h)-f(a)}{h}$$
as long as the limit exists.

\subsection*{Velocities}

$$\text{average velocity}=\frac{\text{displacement}}{\text{time}}=\frac{f(a+h)-f(a)}{h}$$\\
Instantaneous velocity at time \(t = a\) is defined as
$$v(a) = \lim_{h \to 0}\frac{f(a+h)-f(a)}{h}$$

\subsection*{Derivatives}

The derivative of a function \(f\) at a number \(a\), denoted by \(f'(a)\), is
$$f'(a)=\lim_{h \to 0}\frac{f(a+h)-f(a)}{h}$$
if the limit exists.

\subsection*{Rates of Change}

The average rate of change of \(y\) with respect to \(x\) over the interval \([x_1, x_2]\) is the difference quotient
$$\frac{\Delta x}{\Delta y}=\frac{f(x_2)-f(x_1)}{x_2-x_1}$$
\\
The instantaneous rate of change of \(y\) with respect to \(x\) is
$$\lim_{\Delta x \to 0}\frac{\Delta y}{\Delta x}=\lim_{x_2 \to x_1}\frac{f(x_2)-f(x_1)}{x_2-x_1}$$
The derivative \(f'(a)\) is the instantaneous rate of change of \(y=f(x)\) with respect to \(x\) when \(x=a\).


\section*{Ex.3.5}
\subsection*{Estimate the size of the symmetric difference}
The symmetric difference is $(B\backslash C)\cup (C \backslash B)$.

We have that
$$
(B\backslash C)\cup (C \backslash B) = (B \cup C)\backslash (C \cap B)
$$
Hence
\begin{align*}
S_{h,t}((B\backslash C)\cup (C \backslash B))
  &= S_{h,t}((B \cup C)\backslash (C \cap B))\\
  &= ((S_{h,t}(B) \cup S_{h,t}(C))\backslash (S_{h,t}(C) \cap S_{h,t}(B)))
\end{align*}
Hence the estimate would be
$$
[\left|S_{h,t}(B) \cup S_{h,t}(C)\right| - \left|S_{h,t}(C) \cap S_{h,t}(B)\right|]\frac{m}{t}
$$


\subsubsection{Flux}
\noindent
We can rewrite the surface integral because $\hat{n} = \frac{\vec{r_u} \times \vec{r_v}}{\norm{\vec{r_u} \times \vec{r_v}}}$ and $\mathrm{d}s = \norm{\vec{r_u} \times \vec{r_v}}$.

[INSERT IMAGE]

\noindent
So,
\begin{equation*}
	\iint\limits_{S}{\vec{F} \cdot \mathrm{d}s} = \iint\limits_{D}{\left(\vec{F}\circ\vec{r}\right) \cdot \left(\vec{r_u} \times \vec{r_v}\right)\mathrm{d}u\mathrm{d}v}
\end{equation*}
where $D$ is the domain of $S$ in uv-space (a uv-plane), and $\vec{r}(u,v)$ parameterizes the surface $S$. This quantity is called the ``directed surface area integral'' or ``flux'' through $S$.\\

\noindent
Flux tells us how much a vector field penetrates a surface. If the field is parallel to the surface (perpendicular to the normal vector) then the flux is 0. As the field and normal vector to the surface become more aligned, the flux increases. \\

\noindent
Flux has many practical applications. One of the fundamental equations governing electricity and magnetism talks about electric flux: the amount of an electric field that goes through a surface. We will investigate this equation later.\\

\noindent
For example, let an electric field be $\vec{E} = \langle x, y, 0 \rangle \text{ N}/ \text{C}$. Compute the electric flux, $\Phi_{E}$, through the portion of the paraboloid $z=25-x^2-y^2$ above the xy-plane. Assume a unit distance of 1 meter.
\indent
We will take advantage of a paraboloid's radial symmetry and use cylindrical coordinates.
\begin{equation*}
	\vec{r}(u,v) = \langle u\cos{v}, u\sin{v}, 25 - u^2 \rangle \text{ where } 0 \leq u \leq 5 \text{ and } 0 \leq v \leq 2\pi\end{equation*}
\begin{equation*}
	\vec{r_u} = \langle \cos{v}, \sin{v}, -2u \rangle \text{ and } \vec{r_v}=\langle -u\sin{v}, u\cos{v}, 0 \rangle
\end{equation*}
\begin{equation*}
	\vec{r_u} \times \vec{r_v} = \langle 2u^2\cos{v}, 2u^2\sin{v}, 0 \rangle
\end{equation*}
\begin{equation*}
	\vec{E}\circ\vec{r} = \langle u\cos{v}, u\sin{v}, 0 \rangle
\end{equation*}
\begin{equation*}
	\left(\vec{E}\circ\vec{r}\right) \cdot (\vec{r_u}) \times \vec{r_v}) = 2u^3
\end{equation*}
\begin{equation*}
	\Phi_{E} = \int_{0}^{2\pi}{\int_{0}^{5}{2u^3\mathrm{d}u}\mathrm{d}v} = 2\pi\left(\frac{u^4}{2}\right)\rvert_{0}^{5} = 625\text{ Nm}^2/\text{C}
\end{equation*}


\section{Derivatives of Inverse Trig Functions}
Inverse trig functions are defined so that composing them with their corresponding trig function gives you the identity.
For example $\arcsin{(\sin{x})} = x$.
The inverse of a function $f$ can be obtained by reflecting the graph of $f$ across the line $y=x$, effectively swapping the $x$ and $y$ coordinates of each point along the graph.
For functions that repeat $y$ values (i.e are not injective), we have to limit the domain of the inverse functions so they don't repeat $x$ values.
Since the trig functions don't have any cusps or corners, neither will the inverse trig functions.
The slope $f^{-1}$ will the the reciprocal of the slope of $f$, since the change in $x$ in $f$ becomes the change in $y$ of $f^{-1}$ and the change in $y$ of $f$ becomes the change in $x$ of $f^{-1}$.
\begin{equation*}
	\dd{f^{-1}}{x}\biggr\rvert_{f(a)} = \frac{1}{\dd{f}{a}\bigr\rvert_{a}}
\end{equation*}

This leads us to a theorem that will help us derive the derivatives of the inverse trig functions.
\begin{theorem}
	If $f$ is differentiable at every point along an interval $I$ and its derivative is never 0 along $I$, then $f^{-1}$ exists and is differentiable on every point in $f(I)$.
\end{theorem}

\subsection{$\arcsin$, $\arctan$, and $\arcsec$}
Let's apply this theorem and our differentiation rules to find the derivative of $y = \arcsin{x}$.
From $-\frac{\pi}{2} < x < \frac{\pi}{2}$, the derivative of $f(x)=\sin{x}$, $f^\prime(x)=\cos{x}$ is never 0.
So, we know by the previous theorem that $f^{-1}(x)=\arcsin{x}$ exists and is differentiable along every point in $\sin(-\frac{\pi}{2} < x < \frac{\pi}{2}) = -1 < x < 1$.
So, we can rearrange the equation and implicitly differentiate to find the derivative of $\arcsin{x}$ in this interval.
\begin{align*}
	y &= \arcsin{x} \\
	\sin{y} &= x \\
	\cos{(y)}y^\prime &= 1 \\
	y^\prime &= \frac{1}{\cos{y}} \\
	&= \frac{1}{\sqrt{1-\sin^2{y}}} \\
	&= \frac{1}{\sqrt{1-x^2}}, -1 < x < 1.
\end{align*}

We can use the same method to find the derivative of $\arctan{x}$, which is differentiable for all real numbers.
\begin{align*}
	y &= \arctan{x} \\
	\tan{y} &= x \\
	\sec^2{(y)}y^\prime &= 1 \\
	y^\prime &= \frac{1}{\sec^2{y}} \\
	&= \frac{1}{1+\tan^2{y}} \\
	&= \frac{1}{1+x^2}.
\end{align*}

Although we generally use the same method for $\arcsec$, which is differentiable for $\abs{x}>1$, we have to be careful with out trig identities.
\begin{align*}
	y &= \arcsec{x} \\
	\sec{y} &= x \\
	\sec{(y)}\tan{(y)}y^\prime &= 1 \\
	y^\prime &= \frac{1}{\sec{(y)}\tan{(y)}} \\
	&= \frac{1}{x\tan{(y)}} \\
	&= \frac{1}{\pm x\sqrt{\sec^2{y}-1}} \\
	&= \frac{1}{\pm x\sqrt{x^2-1}} \\
	&= \begin{cases}
		\frac{1}{x\sqrt{1-x^2}} & x > 1 \\
		\frac{-1}{x\sqrt{1-x^2}} & x < -1
	\end{cases} \\
	&= \frac{1}{\abs{x}\sqrt{1-x^2}}, \abs{x} > 1.
\end{align*}

\subsection{$\arccos$, $\arccot$, and $\arccsc$}
Now that we've derived derivatives of $\arcsin$, $\arctan$, and $\arcsec$, finding the derivatives of the arc-co-functions is much easier.
Since trig functions and their co-functions are $\pi/2$ radians different on the $x$-axis, arc-trig functions will be $\pi/2$ radians different on the y-axis.
\begin{equation*}
	\text{arccxx{($x$)}} = \frac{\pi}{2} - \text{arcxxx}{(x)}.
\end{equation*}
Using our constant and sum and difference derivative rules, we can derive that the derivatives of the arc-co-functions are simply the opposite of the derivatives of their corresponding arc-functions.
\begin{table}[H]
	\begin{center}
		\begin{tabular}{ l l l }
			$\begin{aligned}\dd{}{x}\arccos{x}=\frac{-1}{\sqrt{1-x^2}},-1<x<1\end{aligned}$ & $\begin{aligned}\dd{}{x}\arccot{x}=\frac{-1}{1+x^2}\end{aligned}$ & $\begin{aligned}\dd{}{x}\arccsc{x}=\frac{-1}{\abs{x}\sqrt{1-x^2}},\abs{x}>1\end{aligned}.$ \\
		\end{tabular}
	\end{center}
\end{table}


\part{Antiderivatives}

\chapter{The Indefinite Integral}

\section{The Indefinite Integral}

How do we undo a derivative? If we were given the derivative of a function $ f'(x) $, how could we find the original function $ f(x) $? The answer is called the antiderivative of $ f(x) $, which we will denote by the associated capital letter $ F(x) $. \\

Another way to think about this question is: "What function do I have to take the derivative of in order to get the answer?" The antiderivative of $ f(x) = 2x $ is $ x^2 $. \\

But what about $ F(x) = x^2 + 1 $? This works too! In fact, since when we take the derivative of a constant, we get zero. We could have chosen any constant. As a result, we report our antiderivative in its most general form $ x^2 + C $. The constant $ C $ is an important part of the antiderivative. \\

Notationally, we denote the operation of take the antiderivative (known as integration) as: \\

$$
    \int 2x \ dx = x^2 + C
$$

This is also called the indefinite integral of the function $ f(x) $, or sometimes just the integral of $ f(x) $, where $ f(x) $ is called the integrand. \\

That was a pretty simple example, so how do we find antiderivatives of more complicated expressions? In much the same way as we did with derivatives, we can generate a set of rules for finding antiderivatives, derived simply by thinking of our familiar derivative rules in reverse. \\

\section{Basic Antiderivative Rules}

The power rule for derivatives multiplies by the power and then subtracts one from the power. Reversing these operations means that we add one to the power and divide by the new power. \\

\subsection{Reverse Power Rule}

\begin{theorem}[Reverse Power Rule]
    \begin{align}
        \int x^n \ dx = {x^{n+1} \over n+1} + C,\ where\ n \ne -1
    \end{align}
\end{theorem}

\begin{exercise}\nonumber
    Find the anitiderivatives. \\

    (a)
    \begin{align}
         & \int x^6 \ dx \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int \sqrt[4]{t} \ dt \\
        \\
        \\
        \\
    \end{align}

    (c)
    \begin{align}
         & \int {1 \over x^{5/3}} \ dx \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\subsection{Antiderivative of Zero}

\begin{theorem}[Antiderivative of Zero]
    \begin{align}
        \int 0 \ dx = C
    \end{align}
\end{theorem}

\subsection{Antiderivative of a Constant}

\begin{theorem}[Antiderivative of a Constant]
    \begin{align}
        \int k \ dx = kx + C,\ where\ k\ is\ any\ constant
    \end{align}
\end{theorem}

\begin{exercise}\nonumber
    Find the anitiderivatives.

    \begin{align}
         & \int \pi \ dx \\
        \\
    \end{align}
\end{exercise}

\subsection{Multiplicative Constants}

\begin{theorem}[Multiplicative Constantst]
    \begin{align}
        \int kf(x) \ dx = k \int f(x) \ dx
    \end{align}
\end{theorem}

\begin{exercise}\nonumber
    Find the anitiderivatives. \\

    (a)
    \begin{align}
         & \int 4x^7 \ dx \\
        \\
        \\
        \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int {\pi \over \sqrt{t}} \ dt \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\subsection{Sum / Difference}

\begin{theorem}[Sum / Difference]
    \begin{align}
        \int (f(x) \pm g(x)) \ dx = \int f(x) \ dx \pm \int g(x) \ dx
    \end{align}
\end{theorem}

\begin{exercise}\nonumber
    Find the anitiderivatives. \\

    (a)
    \begin{align}
         & \int (3x^2 + 5) \ dx \\
        \\
        \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int \left({1 \over x^3} - {2 \over x^2}\right) \ dx \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\subsection{Trigonometric Functions}

\begin{theorem}[Trigonometric Functions]
    \begin{align}
        \int sin(x) \ dx       & = -cos(x) + C \\
        \int cos(x) \ dx       & = sin(x) + C  \\
        \int csc^2(x) \ dx     & = -cot(x) + C \\
        \int sec^2(x) \ dx     & = tan(x) + C  \\
        \int sec(x)tan(x) \ dx & = sec(x) + C  \\
        \int csc(x)cot(x) \ dx & = -csc(x) + C
    \end{align}
\end{theorem}

\subsection{Exponential / Logarithmic}

\begin{theorem}[Exponential / Logarithmic]
    \begin{align}
        \int a^xln(a) \ dx         & = a^x + C      \\
        \int e^xln(e) \ dx         & = e^x + C      \\
        \int {1 \over xln(a)} \ dx & = log_a|x| + C \\
        \int {1 \over x} \ dx      & = ln|x| + C
    \end{align}
\end{theorem}

\begin{exercise}\nonumber
    Find the anitiderivatives. \\

    (a)
    \begin{align}
         & \int 4^xln(4) + 5e^x - {6 \over x} \ dx \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int 3^z \ dz \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

Sometimes we need to manipulate the integral a little bit before we can apply the rules. \\

\begin{exercise}\nonumber
    Find the anitiderivatives. \\

    (a)
    \begin{align}
         & \int {2s^3 - 5s^4 \over 3s^2} \ ds \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int \left({1 \over x} + {1 \over x^2}\right)\left(3 + 2x^2\right) \ dx \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\chapter{Chain Rule in Reverse}

\section{Chain Rule in Reverse}

The derivative of $ f(u(x)) $ is $ f'(u(x))u'(x) $, so \\

\begin{theorem}[Chain Rule in Reverse]
    \begin{align}
        \int f'(u(x))u'(x) \ dx = f(u(x)) + C
    \end{align}
\end{theorem}

Notice that in the integration, the $ u'(x) $ piece disappears, being absorbed back into $ f(x) $. The steps for finding the antiderivative of composition functions are as follows: \\

\begin{enumerate}
    \item
          Identify the core layer $ u(x) $. \\

    \item
          Identify the derivative of the core layer $ u'(x) $. \\

    \item
          Identify the outer layer $ f' $, and integrate $ f' $ leaving $ u(x) $ inside. \\
\end{enumerate}

\begin{exercise}\nonumber
    Find the anitiderivatives. \\

    (a)
    \begin{align}
         & \int (6x^2 + 1)sin(2x^3 + x) \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int sec^2(4t) \ dt \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (c)
    \begin{align}
         & \int 4x^3(3x^4 - 1)^{14} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (d)
    \begin{align}
         & \int {e^{1 \over x} \over 4x^2} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\begin{exercise}\nonumber
    Integrate in one step. \\

    \begin{align}
         & \int e^{-2t} + sin(3t) + cos\left({1 \over 4}t\right) \ dt \\
        \\
        \\
    \end{align}
\end{exercise}

\chapter{The Method of Substitution}

\section{The Method of Substitution}

The idea behind the method of substitution is to change a difficult integral in terms of one variable into an easier integral in terms of some other variable using a substitution. \\

\begin{theorem}[The Method of Substitution]
    \begin{align}
        \int f'(u(x)) \ {du \over dx} = \int f'(u) \ du
    \end{align}
\end{theorem}

\begin{exercise}\nonumber
    The method of substitution.

    \begin{align}
        \int (6x + 4)(3x^2 + 4x)^5 \ dx
    \end{align}

    \begin{enumerate}
        \item
              Identify the core layer $ u(x) = 3x^2 +4x $. \\

        \item
              Find the derivative of the core $ {du \over dx} = 6x + 4 $. \\

        \item
              Transform from an integral in $ x $ to an integral in the new variable $ u $ using the change of variable theorem. \\
              \begin{align}
                   & \int (6x + 4)(3x^2 + 4x)^5 \ dx \\
                   & = \int {du \over dx}u^5 \ dx    \\
                   & = \int u^5 \ du                 \\
                   & = {u^6 \over 6} + C
              \end{align}

        \item
              Convert back to the original variable by substituting $ u(x) $ back in.
              \begin{align}
                   & {u^6 \over 6} + C             \\
                   & = {(3x^2 + 4x)^6 \over 6} + C
              \end{align}
    \end{enumerate}
\end{exercise}

\begin{exercise}\nonumber
    Calculate using the method of substitution. \\

    (a)
    \begin{align}
         & \int sin(x)e^{5cos(x)} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int {x \over (5x + 7)^3} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (c)
    \begin{align}
         & \int xsec(3x^2)tan(3x^2) \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (d)
    \begin{align}
         & \int {-{1 \over t^2} + 1 \over \sqrt{{1 \over t} + t}} \ dt \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (e)
    \begin{align}
         & \int (1 + 900x)^{1/15000} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (f)
    \begin{align}
         & \int sin(\theta)(cos^3\theta - cos^5\theta) \ d\theta \\
        \\
        \\
        \\
    \end{align}

    (g)
    \begin{align}
         & \int {7x \over 4x^2 + 9} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (h)
    \begin{align}
         & \int (2x + 5) \cdot \sqrt[3]{3x + 1} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\chapter{Definite Integrals}

\section{Riemann Sum}

Suppose we wanted to find the area underneath the graph of a straight line that lies above the x-axis, between $ x = a $ and $  x = b $. Since we have formulas for finding the area of basic shapes, we can easily figure this out. \\

\begin{figure}[H]
    \centering
    \begin{tikzpicture}
        \fill[yellow](1,0) -- (1,2) -- (3.5,4.5) to[out=45,in=35] (4,4.8) -- (4,0) -- cycle;
        \draw[thick,-latex] (-2,0) -- (6,0);
        \draw[thick,-latex] (0,-1) -- (0,6);
        \draw[very thick,black] (-0.8,0.2) -- (3.5,4.5) to[out=45,in=135] node[pos=0.5,above,font=\large]{$ f(x) $} (5,4.5);
        \draw[very thick,dashed,gray]
        (1,0) node[below,black] {$a$} -- (1,2)
        (4,0) node[below,black] {$b$} -- (4,4.8);
    \end{tikzpicture}
\end{figure}

But what about finding the area underneath the graph of a general curve $ y = f(x) $ and above the x-axis between $  x = a $ and $ x = b $? \\

Bernhard Riemann's idea was to carve up the desired area into rectangle and user their area to estimate the true area. He called this the Riemann Sum and it goes as follows: \\

\begin{enumerate}
    \item
          Create a partition $ P $, dividing up the interval $ [a, b] $ into n subintervals $ I_1, I_2, \dots, I_n $. \\

    \item
          Choose an x-value (called it $ x_k $) in each subinterval $ I_k $. \\

    \item
          For each $ x_k $ we choose, draw a rectangle with height $ f(x_k) $ and a width spanning $ I_k $ ($ \Delta x_k $). \\
\end{enumerate}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=1.2]
        \def\a{1.7}
        \def\b{5.7}
        \def\c{3.7}
        \def\L{0.5} % width of interval

        \pgfmathsetmacro{\Va}{2*sin(\a r+1)+4} \pgfmathresult
        \pgfmathsetmacro{\Vb}{2*sin(\b r+1)+4} \pgfmathresult
        \pgfmathsetmacro{\Vc}{2*sin(\c r+1)+4} \pgfmathresult

        \draw[->,very thick] (-0.5,0) -- (7,0) coordinate (x axis) node[below] {$x$};
        \draw[->,very thick] (0,-0.5) -- (0,7) coordinate (y axis) node[left] {$y$};
        \foreach \f in {1.7,2.2,...,6.2} {\pgfmathparse{2*sin(\f r+1)+4} \pgfmathresult
                \draw[fill=blue!20] (\f-\L/2,\pgfmathresult |- x axis) -- (\f-\L/2,\pgfmathresult) -- (\f+\L/2,\pgfmathresult) -- (\f+\L/2,\pgfmathresult |- x axis) -- cycle;}
        \node at (\a-\L/2,-5pt) {\footnotesize{$a$}};
        \node at (\b+\L/2+\L,-5pt) {\footnotesize{$b$}};
        \draw[blue,very thick,smooth,samples=100,domain=1.45:6.2] plot(\x,{2*sin(\x r+1)+4});
    \end{tikzpicture}
\end{figure}

The area of rectangle $ k $ is:
$$
    f(x_k) \cdot \Delta x_k
$$

The total area of all the rectangles is:
$$
    f(x_1) \cdot \Delta x_1 + f(x_2) \cdot \Delta x_2 + \dots + f(x_n) \cdot \Delta x_n = \sum_{k=1}^{n} f(x_k) \cdot \Delta x_k
$$

\begin{exercise}\nonumber
    Use the Riemann Sum to estimate the area below $ y = sin({1 \over 2}x) $ and above the x-axis, between $ x = 0 $ and $ x = 2\pi $. Use a partition of 4 subintervals. \\

    \begin{figure}[H]
        \centering
        \begin{tikzpicture}[
                declare function={
                        f(\x)=sin(deg(0.5*\x));
                    },
                xscale=0.5
            ]
            \begin{axis}[
                    axis lines = middle,
                    xtick ={1,4},
                    ytick ={0},
                    xticklabels = {$a$,$b$},
                    ymin = -0.2,
                    ymax = 2,
                    xmin = -0.2,
                    xmax = 7,
                    x=2cm,y=2cm,
                    axis line style = thick,
                ]

                \addplot [
                    domain=0:6.3,
                    samples=300,
                    line width=1pt,
                    fill=red, draw=none,
                    fill opacity=0.2
                ] {f(x)} \closedcycle;

                \addplot [
                    domain=0:6.3,
                    samples=300,
                    line width = 1pt, red
                ] {f(x)};

                \addplot [
                    ycomb, thick, red,
                    no markers,
                    samples at={1,3,5}
                ] {f(x)};
            \end{axis}
            \draw[dashed,red] (2.4, 1.3) -- (0.3, 1.3);
            \draw[dashed,red] (6.1, 2.4) -- (2.4, 2.4) -- (2.4, 0);
            \draw[dashed,red] (10.2, 1.55) -- (6.4, 1.55);
            \draw[dashed,red] (12.9, 0.4) -- (10.4, 0.4);
        \end{tikzpicture}
    \end{figure}

    \begin{align}
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

So, what happens as make the rectangles skinnier? \\

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=2+sin(deg(\x-2))+sin(deg(3*\x))/2+sin(deg(5*\x))/8 + sin(deg(7*\x))/28;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=1:4,
                samples=300,
                line width=1pt,
                fill=red, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1,2,...,4}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=2+sin(deg(\x-2))+sin(deg(3*\x))/2+sin(deg(5*\x))/8 + sin(deg(7*\x))/28;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=1:4,
                samples=300,
                line width=1pt,
                fill=red, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1,1.5,...,4}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=2+sin(deg(\x-2))+sin(deg(3*\x))/2+sin(deg(5*\x))/8 + sin(deg(7*\x))/28;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=1:4,
                samples=300,
                line width=1pt,
                fill=red, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1,1.1,...,4}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

As the width of the rectangles decreases, the accuracy of the estimate increases. So, what happens if we let the width of all the rectangles in the partition approach 0? \\

Notationally:

$$
    \lim\limits_{\substack{n \rightarrow \infty \\ \left\|p\right\| \rightarrow 0}} \sum_{k=1}^{n} f(x_k) \cdot \Delta x_k
$$

We should get the exact area, that is, our estimate is no longer just an estimate. \\

Notationally, instead of limit, we write it as: \\

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_{x = a}^{b} f(x) \ dx
    \end{align}
\end{theorem}

\section{Definite Integral}

For all of the following, suppose that $ k $, $ a $, $ b $, and $ c $ are constants with $ a < b < c $, and that $ f $ and $ g $ are integrable functions on the domain of integration. \\

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_a^b k \ dx = (b - a)k
    \end{align}
\end{theorem}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=2;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=1:4,
                samples=300,
                line width=1pt,
                fill=red, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1,4}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_a^a f(x) \ dx = 0
    \end{align}
\end{theorem}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=0.5 * sin(deg(\x)) + 2;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1},
                ytick ={0},
                xticklabels = {$a$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_b^a f(x) \ dx = -\int_a^b f(x) \ dx
    \end{align}
\end{theorem}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=0.5 * sin(deg(\x)) + 2;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=1:4,
                samples=300,
                line width=1pt,
                fill=red, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1,4}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_a^c f(x) \ dx = \int_a^b f(x) \ dx + \int_b^c f(x) \ dx
    \end{align}
\end{theorem}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=0.5 * sin(deg(\x)) + 2;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,2,4},
                ytick ={0},
                xticklabels = {$a$,$b$,$c$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=1:2,
                samples=300,
                line width=1pt,
                fill=red, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=2:4,
                samples=300,
                line width=1pt,
                fill=blue, draw=none,
                fill opacity=0.2
            ] {f(x)} \closedcycle;

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                ycomb, thick, red,
                no markers,
                samples at={1,2,4}
            ] {f(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_a^b kf(x) \ dx = k \int_a^b f(x) \ dx
    \end{align}
\end{theorem}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=0.5 * sin(deg(\x)) + 2;
                    g(\x)=0.5 * sin(deg(\x)) + 1;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {g(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{theorem}[Definite Integral]
    \begin{align}
        \int_a^b (f(x) \pm g(x))  \ dx = \int_a^b f(x) \ dx \pm \int_a^b g(x) \ dx
    \end{align}
\end{theorem}

\begin{figure}[H]
    \centering
    \begin{tikzpicture}[scale=0.6,
            declare function={
                    f(\x)=0.5 * sin(deg(\x)) + 1;
                    g(\x)=2;
                    h(\x)=0.5 * sin(deg(\x)) + 3;
                }
        ]
        \begin{axis}[
                axis lines = middle,
                xtick ={1,4},
                ytick ={0},
                xticklabels = {$a$,$b$},
                ymin = -0.2,
                ymax = 3.7,
                xmin = -0.2,
                xmax = 5.2,
                x=2cm,y=2cm,
                axis line style = thick,
            ]

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {f(x)};

            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {g(x)};
            \addplot [
                domain=0:5,
                samples=300,
                line width = 1pt, red
            ] {h(x)};
        \end{axis}
    \end{tikzpicture}
\end{figure}

\begin{exercise}\nonumber
    Evaluate. \\

    (a)
    \begin{align}
         & \int_1^2 {1 \over t} \ dt \\
        \\
        \\
        \\
    \end{align}

    (b)
    \begin{align}
         & \int_{-2}^1 x^3 \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\                   \\
    \end{align}

    (c)
    \begin{align}
         & \int_1^5 s(s^2 + 1) \ ds \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (d)
    \begin{align}
        \int_{\sqrt{ln(2)}}^{\sqrt{ln(4)}} xe^{x^2} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (e)
    \begin{align}
         & \int_x^{x^2} sin(t) \ dt \\
        \\
        \\
        \\
        \\
        \\
    \end{align}

    (f)
    \begin{align}
        \int_5^{13} (x + 1)\sqrt{2x - 1} \ dx \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\chapter{Area Under a Curve}

\section{Area Under a Curve}

\begin{exercise}\nonumber
    Find the area below the curve $ f(x) = sin(x) $ and above the x-axis between $ x = {\pi \over 2} $ and $ x = \pi $. \\

    \begin{figure}[H]
        \centering
        \begin{tikzpicture}[scale=0.8,yscale=0.2]
            \draw[->] (-1,0) -- (8,0) node[right] {$ x $};
            \draw[->] (0,-10) -- (0,30) node[above] {$ y $};
            \draw[very thick,color=red,domain=-1:7] plot (\x,{-2 * (\x - 3.14) ^ 2 + 20});
            \draw[-,very thick,red] (3.14, 0) -- (3.14,20);
            \node at (1.5,-2) [] {$ \pi \over 4 $};
            \node at (3.14,-2) [] {$ \pi \over 2 $};
            \node at (4.8,-2) [] {$ 3\pi \over 4 $};
            \node at (6,-2) [] {$ \pi $};
        \end{tikzpicture}
    \end{figure}

    \begin{align}
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

What if we have a more interesting situation where many curves are involved? For instance, how do we find the area between two curves $ f $ and $ g $? \\

\begin{figure}[H]
    \centering
    \begin{tikzpicture}
        \begin{axis}[
                xlabel={$x$},
                ylabel={$y$},
                xtick={-3,0,1},
                ytick={2},
                samples=100,
                domain=-4:1,
                xmin=-4,xmax=2,
                ymin=-1,ymax=3,
            ]
            \addplot[name path=func, red, very thick, mark=none, ] {sqrt(-x+1)};
            \addplot[name path=line, red, very thick] {2};
            \addplot fill between[
                    of = func and line,
                    soft clip={domain=-3:0},
                    every even segment/.style  = {gray,opacity=.4}
                ];
            \draw[thick,dashed,brown] (axis cs:-3,0) -- (axis cs:-3,2);
        \end{axis}
    \end{tikzpicture}
\end{figure}

$$
    \text{Area under Upper} - \text{Area under Lower} = \text{Area expected}
$$

\begin{exercise}\nonumber
    Calculate the area bounded by $ y = 2x + 1 $ and $ y = x^2 - 2x - 3 $. \\

    \begin{figure}[H]
        \centering
        \begin{tikzpicture}[scale=0.8,yscale=0.4]
            \draw[->] (-2,0) -- (7,0) node[right] {$ x $};
            \draw[->] (0,-5) -- (0,15) node[above] {$ y $};
            \draw[very thick,color=red,domain=-2:6] plot (\x,{2 * (\x) + 1});
            \draw[very thick,color=red,domain=-2:5] plot (\x,{(\x) ^ 2 - 2 * \x - 3});
        \end{tikzpicture}
    \end{figure}

    \begin{align}
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\begin{exercise}\nonumber
    Calculate the area bounded by $ y = -x $, $ y = \sqrt{x} $, and $ y = -x^2 + 2 $ between $ x = 0 $ and $ x = 2 $. \\

    \begin{figure}[H]
        \centering
        \begin{tikzpicture}[scale=0.8]
            \draw[->] (-2,0) -- (5,0) node[right] {$ x $};
            \draw[->] (0,-4) -- (0,4) node[above] {$ y $};
            \draw[very thick,color=red,domain=-2:4] plot (\x,{-(\x)});
            \draw[very thick,color=red,domain=0:5,samples=300] plot (\x,{\x ^ (1/2)});
            \draw[very thick,color=red,domain=-2:2.3] plot (\x,{-(\x) ^ 2 + 2});
            \draw[dashed, very thick,color=blue] (1, 1) -- (1, -1);
            \node at (0.6, 0.2) [blue] {$ A_1 $};
            \node at (1.3, -0.5) [blue] {$ A_2 $};
        \end{tikzpicture}
    \end{figure}

    \begin{align}
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
    \end{align}
\end{exercise}

\begin{exercise}\nonumber
    Find the area bounded by $ y^2 = x + 4 $ and $ y = {1 \over 2}x + {1 \over 2} $. \\

    \begin{figure}[H]
        \centering
        \begin{tikzpicture}[scale=0.8]
            \draw[->] (-5,0) -- (6,0) node[right] {$ x $};
            \draw[->] (0,-3) -- (0,3) node[above] {$ y $};
            \draw[very thick,color=red,domain=-4:6] plot (\x,{0.5 * \x + 0.5});
            \draw[very thick,color=red,domain=-4:6,samples=300] plot (\x,{(\x + 4) ^ (1/2)});
            \draw[very thick,color=red,domain=-4:6,samples=300] plot (\x,{-(\x + 4) ^ (1/2)});
            \draw[dashed,blue] (-3,1) -- (0,1) -- (0,0.5) -- (-3,0.5) -- (-3,1);
        \end{tikzpicture}
    \end{figure}

    \begin{align}
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
        \\
                                         \\
                                         \\
                                         \\
                                         \\
    \end{align}
\end{exercise}


\subsection{First-order conditions}

We have a utility function:

\(U=f(\mathbf x)\)

And the constraint:

\(\sum_i (x_i-c_i)p_i\le 0\)

This gives us the constrained optimisation problem:

\(L=f(\mathbf x)-\lambda \sum_i (x_i-c_i)p_i\)

The first-order conditions are:

\(L_{x_i}=\dfrac{\delta }{\delta x_i}f(\mathbf x)-\lambda p_i=0\)

Or:

\(\lambda = \dfrac{MU(x_i)}{p_i}\)

This means for any pair we have:

\(\dfrac{MU(x_i)}{p_i}=\dfrac{MU(x_j)}{p_j}\)

For the Cobb-Douglas utility function the first-order conditions are:

\(\dfrac{\delta }{\delta x_1}f(\mathbf x) = \dfrac{1}{x_1}\alpha_1\prod_i x_i^{\alpha_i}\)

\(L_{x_i}=\dfrac{\delta }{\delta x_i}f(\mathbf x)-\lambda p_i=0\)


\subsection{Power Rule}
\begin{lemma}
	Let $f(x) = x^n$ where $n$ is a positive integer. Then
	\begin{equation}
		f^\prime(x) = nx^{n-1}.
	\end{equation}
\end{lemma}
\begin{proof}
	Applying the limit definition of the derivative,
	\begin{equation*}
		f^\prime(x) = \lim_{h \to 0}{\frac{(x+h)^n - x^n}{h}}.
	\end{equation*}
	Applying the binomial theorem (think Pascal's Triangle),
	\begin{align*}
		f^\prime(x) &= \lim_{h \to 0}{\frac{x^n + nhx^{n-1} + \ldots + {n \choose k}h^{k}x^{n-k} + \ldots + h^n - x^n}{h}} \\
		&= \lim_{h \to 0}{nx^{n-1} + \ldots + {n \choose k}h^{k-1}x^{n-k} + \ldots + h^{n-1}} \\
		&= nx^{n-1}.
	\end{align*}
\end{proof}

We'll revisit the power rule after we discover some more detailed rules to show that $n$ can be any real number, not just a positive integer.


\subsection{Group order}

For finite groups, each element \(e\) has:

\(e^n=I\)

For some \(n\in \mathbb{N}\)

Where \(I\) is the identity element.

The order of the group is the smallest value of \(n\) such that that holds for all elements.

For example in the multiplicative group \(G=\{-1,1\}\) the order is \(2\).

Or:

\(|G|=2\)

Additionally

\(|-1|=2\)

\(|1|=1\)


\subsection{Cosets and normal subgroups}

A coset is defined between a group and a subgroup of the group.

For a group \(G\), and its subgroup \(H\):

\begin{itemize}
\item The left coset is \(\{gH\}\)
\item The right coset is \(\{Hg\}\)
\end{itemize}

For \(\forall g\in G\).

For abellian groups, the left and right cosets are the same.

The left and right cosets can also be the same, even if the group \(G\) is not abelian.

\subsubsection{Normal subgroups}

If the left and right cosets are the same then \(H\) is a normal subgroup.

\subsubsection{Cosets divide a group.}

Consider two left cosets, \(aH\) and \(bH\), with a common element.

This means that \(ah_i=bh_j\).

We can use this to get:

\(a=bh_jh_i^{-1}\)

\(b=ah_ih_j^{-1}\)

We know that:

\(ah\in aH\)

\(bh\in bH\)

So:

\(bh_jh_i^{-1}h\in aH\)

\(ah_ih_j^{-1}h\in bH\)

And so:

\(bH\subset aH\)

\(aH\subset bH\)

Therefore:

\(aH=bH\)

\subsubsection{Example 1}

Consider the group \(\{-1,1\},\times \)

For the subgroup \(\{1\},\times \), the left coset is \(\{gH\}=\{1,-1\}\).

The right coset is the same.

\subsubsection{Example 2}

Consider the group of integers and addition: \((Z,+)\)

For subgroup \((mZ,+)\), the left and right cosets are the same because the group is abelian.

The coset of the subgroup is the subgroup multiplied by each element in \(G\).

This is \(mZ\), \(mZ+1\), \(mZ+2\) and so on.

Once we reach \(mZ+m\) this has looped, and is already a coset, so we only need the sets upto \(mZ+m-1\).