% !TEX root = Main.tex
\section{Non-Negative Matrix Factorization}
\subsection*{pLSA}
\textbullet co-occurence matrix $X=x_{ij}$ \#occurences of word $w_j$ in doc $d_i$.
\textbullet $p(w|d) = \sum_z p(w,z|d) = \sum_z p(w|d,z)p(z|d) = \sum_z p(w|z)p(z|d)$.\\
\textbullet log-likelihood: $\sum_{i,j} x_{i,j}\log p(w_j|d_i)$ \\
E-Step (optimal q):\\
$q_{zij} = p(z|w_j,d_i)=\frac{p(w_j|z)p(z|d_i)}{\sum_{k=1}^K p(w_j|k)p(k|d_i)}$ \\
M-Steps:\\
$p(z|d_i) = \frac{\sum_j x_{ij}q_{zij}}{\sum_j x_{ij}}, p(w_j|z) = \frac{\sum_i x_{ij}q_{zij}}{\sum_{i,l}x_{il}q_{zil}}$\\

\subsection*{NMF Algorithm for quadratic cost function}
$\mathbf{X} \in \mathbb{Z}^{N \times M}_{\geq 0}$, NMF: $\mathbf{X} \approx \mathbf{U^\top V}, x_{ij}$

$\min_{\mathbf{U}, \mathbf{V}} J(\mathbf{U}, \mathbf{V}) = \frac{1}{2} \|\mathbf{X} - \mathbf{U}^\top\mathbf{V}\|_F^2 = \\
\frac{1}{2}\sum(x_{ij}-u_i^\top v_j)^2$\\
s.t. $\forall i,j,z:u_{zi},v_{zj} \geq 0 $

1. init: $\mathbf{U}, \mathbf{V} = rand()$ 2. repeat for $\mathit{maxIters}$:\\
3. upd. $(\mathbf{VV}^\top)\mathbf{U} = \mathbf{VX}^\top$, proj. $u_{zi} = \max \{ 0, u_{zi} \}$\\
4. upd. $(\mathbf{UU}^\top)\mathbf{V} = \mathbf{UX}$, proj. $v_{zj} = \max \{ 0, v_{zj} \}$ \\
\textbullet vector form: \\
$(\sum_i u_iu_i^\top)v_j=\sum_i x_{ij}u_i$, $(\sum_j v_jv_j^\top)u_i=\sum_j x_{ij}v_j$ \\
note for derivation: If want matrix form, use trace def. If want vector only form, use sum representation of objective.


\subsubsection{Implicit Mass Transfer}

On its inner boundary, the Lumped Parameter model uses the fixed concentration
Dirichlet boundary condition directly in its solution such that,
\begin{align}
C_{k,in}(t_n) &= C(z, t_n)|_{z=r_j}.
\end{align}

The resulting mass transfer into the external component $k$ containing the Lumped
Parameter model is calculated by taking the integral of that concentration
profile over the volume,

\begin{align}
m_{jk}(t_n) &=\int C(z,t_n)dV_k - \int C(z, t_{n-1})dV_k.
\end{align}

In the similar case of the One Dimensional Permeable Porous Medium Model,
the Dirichlet boundary condition at the boundary is also used directly in
the solution as $C_0$ such that,

\begin{align}
  C_{k,0}(t_n) &= C(z, t_n)|_{z=r_j}.
\end{align}

The mass transfer on the inner boundary is again calculated by taking an
integral of that profile over the volume,

\begin{align}
m_{jk}(t_n) &=\int C(z,t_n)dV_k - \int C(z, t_{n-1})dV_k.
\end{align}


\chapter{Linear Regression}
\label{ch:linear-regression}

For a start,\marginnote{In the \widget{Paint Data} widget, remove the C2 label from the list. If you have accidentally left it while painting, don't despair. The class variable will appear in the \widget{Select Columns} widget, but you can "remove" it by dragging it into the Available Variables list.} let us construct a very simple data set. It will contain just one continuous input feature (let's call it x) and a continuous class (let's call it y). We will use Paint Data, and then reassign one of the features to be a class using \widget{Select Columns} and moving the feature y from "Features" to "Target Variable". It is always good to check the results, so we are including \widget{Data Table} and \widget{Scatter Plot} in the workflow at this stage. We will be modest this time and only paint 10 points and will use Put instead of the Brush tool.

\begin{marginfigure}
    \centering
    \includegraphics[width=\linewidth]{lin-reg-workflow1.png}
    \caption{$\;$}
\end{marginfigure}

We would like to build a model that predicts the value of target variable y from the feature x. Say that we would like our model to be linear, to mathematically express it as $h(x)=\theta0+\theta1x$. Oh, this is the equation of a line. So we would like to draw a line through our data points. The $\theta$0 is then an intercept and $\theta$1 is a slope. But there are many different lines we could draw. Which one is the best one? Which one is the one that fits our data the most? Are they the same?

\begin{figure*}[h]
    \centering
    \newcommand{\paint}{\includegraphics[scale=0.45]{paint-data.png}}
    \newcommand{\selcol}{\includegraphics[scale=0.45]{select-columns.png}}
    \infinitewidthbox{
    \stackinset{r}{-0.35\linewidth}{t}{+0.1\linewidth}{\selcol}{\paint}\hspace{6cm}
    }
\end{figure*}

The question above requires us to define what a good fit is. Say, this could be the error the fitted model (the line) makes when it predicts the value of y for a given data point (value of x). The prediction is h(x), so the error is $h(x) - y$. We should treat the negative and positive errors equally, plus, let us agree, we would prefer punishing larger errors more severely than smaller ones. Therefore, we should square the errors for each data \marginnote{Do not worry about the strange name of the \widget{Polynomial Regression}, we will get there in a moment.} point and sum them up. We got our objective function! Turns out that there is only one line that minimizes this function. The procedure that finds it is called linear regression. For cases where we have only one input feature, Orange has a special widget in the Educational add-on called \widget{Polynomial Regression}.

\begin{marginfigure}
    \centering
    \includegraphics[width=\linewidth]{lin-reg-workflow2.png}
    \caption{$\;$}
\end{marginfigure}

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{pol-regression-first.png}
    \caption{$\;$}
\end{figure}

Looks ok. Except that these data points do not appear exactly on the line. We could say that the linear model is perhaps too simple for our data set. Here is a trick: besides the column x, the widget Polynomial Regression can add columns x2, x3… xn to our data set. The number n is a degree of polynomial expansion the widget performs. Try setting this number to higher values, say to 2, and then 3, and then, say, to 8. With the degree of 3, we are then fitting the data to a linear function $h(x) = \theta0 + \theta1x + \theta1x2 + \theta1x3$.

\newpage

The trick we have just performed (adding higher order features to the data table and then performing linear regression) is called polynomial regression. Hence the name of the widget. We get something reasonable with polynomials of degree 2 or 3, but then the results get really wild. With higher degree polynomials, we totally overfit our data.

\begin{figure*}[h!]
    \centering
    \newcommand{\second}{\includegraphics[scale=0.35]{pol-regression-second.png}}
    \newcommand{\eigth}{\includegraphics[scale=0.35]{pol-regression-eigth.png}}
    \infinitewidthbox{
    \stackinset{r}{-0.35\linewidth}{t}{+0.2\linewidth}{\eigth}{\second}\hspace{6cm}
    }
\end{figure*}

\marginnote{It is quite surprising to see that linear regression model can result in fitting non-linear (univariate) functions. That is, the functions with curves, such as those on the figures. How is this possible? Notice though that the model is actually a hyperplane (a flat surface) in the space of many features (columns) that are the powers of x. So for the degree 2, $h(x)=\theta0+\theta1x+\theta1x2$ is a (flat) hyperplane. The visualization gets curvy only once we plot $h(x)$ as a function of x.}

Overfitting is related to the complexity of the model. In polynomial regression, the models are defined through parameters $\theta$. The more parameters, the more complex the model. Obviously, the simplest model has just one parameter (an intercept), ordinary linear regression has two (an intercept and a slope), and polynomial regression models have as many parameters as is the degree of the polynomial. It is easier to overfit with a more complex model, as this can adjust to the data better. But is the overfitted model really discovering the true data patterns? Which of the two models depicted in the figures above would you trust more?


% !TeX root = ./main.tex
% chktex-file 46
% !TeX spellcheck = en-GB
% !TeX encoding = utf8

\subsection{Graphs and graph convolutions}

Graphs are sets of nodes connected by edges, as shown in Fig.~\ref{fig:graph_example}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.4\columnwidth]{img/graph_example.pdf}
	\caption{Example graph where nodes are individuals and edges are their contacts at time step $t$.}%
	\label{fig:graph_example}
\end{figure}

We model partially infected populations as a graph, where each individual (interchangeably called agent) is a node. Edges of this graph model contacts between two agents. The dynamics of infections throughout the population is described by graph convolutions. The definition of a graph convolution~\cite{Kipf2017SemiSupervisedCW} for here is

\begin{equation}
	\label{eq:graph_convolution}
	h_{v_i}^{(l+1)} = \sum_{j\in A(i)} h_{v_j}^{(l)}
\end{equation}

with $h_{v_i}^{(l)}$ denoting the feature vector of node $i$ of iteration $(l)$ and $A(i)$ as all neighbours of node $i$ as described by the adjacency matrix $A$. This formulation is equivalent to the matrix formulation $h^{(l+1)} = A h^{(l)}$ with $A$ as adjacency matrix as shown in~\ref{sec:consistency}. We consider here adjacency matrices without diagonal elements.

Each agent $i$ is modeled by $D$ features, $h_{v_i} \in \mathbb{R}^D$. Therefore, the feature matrix, $h^{(l)}$, consists of all agents' features at time $(l)$ and is thereby of dimension $N\times D$ where there are $N$ agents in the population and each agent is described by $D$ features. A three dimensional feature space is used in this work, $D=3$, modeling three possible health states. The unit vectors of this space are interpreted as following:
\begin{itemize}
	\item $\vec{e}_0$: susceptible state
	\item $\vec{e}_1$: infected state
	\item $\vec{e}_2$: recovered state
\end{itemize}
A uniform distribution over these possible states expresses complete uncertainty of the health state of an agent.

\subsection{SIR Model}
Our basic stochastic SIR model relies on the assumptions that every person in an environment can be modeled as a point value on a grid, which has a location (i.e.\ GPS coordinates) and an infection state. These states can be either \textit{susceptible} (S), \textit{infected} (I), \textit{recovered} (R) or in advanced models also \textit{under quarantine} (Q) or \textit{dead} (D). All individuals, here called agents, have a probability (here called diffusion rate $d$) to make a step on the grid per time step on a predefined grid. The movement is a random walk pattern. In the case that some agents meet at the same location, disease spreading can occur. An infected agent spreads the disease with the probability $\beta$ to all the agents in its close vicinity (same location on the grid). Furthermore, recovery is covered by taking a recovery rate into account, i.e.\ a probability $\gamma$ to recover from the disease per time step. If an infected agent recovers from the disease, the state of the agent changes from \textit{infected} to \textit{recovered}, which is definite (no double infections). The process ends, when no infected agents are left. For more background the reader is referred to~\cite{weiss2013sir} and~\cite{epstein2009modelling}.


\subsection{Hilbert spaces}\label{subsec:hilbert_spaces}

\begin{definition}\label{def:hilbert_space}
  A \term{Hilbert space} is an \hyperref[def:inner_product_space]{inner product space} which is also a \hyperref[def:complete_metric_space]{complete metric space} with the metric induced by the inner product.
\end{definition}

\begin{definition}\label{def:orthonormal_system}
  A set of vectors \( A \) in a Hilbert space \( X \) is called an \term{orthonormal system} if \( A \)
  \begin{equation*}
    \inprod x y \coloneqq \begin{cases}
      1, & x = y,    \\
      0, & x \neq y.
    \end{cases}
  \end{equation*}

  It is a special case of an \hyperref[def:orthogonality]{orthogonal system}. We are usually interested in \term{orthogonal bases}.
\end{definition}


\documentclass[en,12pt]{elegantpaper}

\begin{document}
    \section*{6}
    \noindent  For Poisson distribution,
    \[
        f(x=k|\lambda)=\frac{\lambda^k}{k!}e^{-\lambda}=1/k!\exp\left(k\log\lambda-\lambda\right),
    \]
    \[
        f(X=K|\lambda)=1\Big/\prod k_i!\exp\left(\log\lambda\sum_{i=1}^n(k_i)-n\lambda\right).
    \]
    So, \(T(X)=\sum_{i=1}^{n} X_{i}\) is sufficient and complete for \(\lambda\). Let $\log\lambda=\theta$,
    \[
        \mathbb{E}(T)=\left(ne^\theta\right)'=n\lambda.
    \]
    Hence, UMVU for $\lambda$ is $T/n$. Let $h(t)$ be UMVU for $\lambda^r$, then
    \[
        \mathbb{E}(h(T))=\sum_{i=0}^\infty\frac{h(i)n^i}{i!}\lambda^i=e^{n\lambda}\lambda^r=\sum_{i=0}^\infty\frac{(n\lambda)^i}{i!}\lambda^r.
    \]
    Then, compare the coefficients for both sides, when $t<r$,
    \[
        h(t)=0.
    \]
    But when $t\geqslant r$,
    \[
        h(t)=\frac{t!}{n^r(t-r)!}.
    \]
\end{document}


\documentclass[12pt, letterpaper]{article}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{anysize}

\marginsize{.5in}{.5in}{.5in}{.5in}

% I found that pdflatex, my favorite weapon-of-choice, was goofing my
% margins.  This will fix it:
% Force pdflatex to use correct paper size.
\special{papersize=8.5in,11in}
\setlength{\pdfpageheight}{\paperheight}
\setlength{\pdfpagewidth}{\paperwidth}

\begin{document}

\noindent
{\LARGE \textbf{The Crank-Nicholson Scheme}}

Earlier versions of our heat equation code used an explicit scheme, i.e. one
where  $U(t+\Delta t, x)$ is only a function of $U(t, x\pm\Delta x)$.  This
gave us a somewhat reliable scheme but easily became unstable if our selection
of $\Delta t$ and $\Delta x$ was poor.  \emph{Implicit schemes} are ones where
$U(t+\Delta t, x)$ is a function of $U(t, x\pm\Delta x)$ \emph{and} a function
of $U(t+\Delta t, x\pm\Delta x)$, i.e. we need to know information about the
future state to get to the future state.  This sounds self-contradictory, but
it is both possible and yields a scheme that is unconditionally stable.
Let's start with our heat equation:

\begin{equation}
  \label{heat}
  \frac{\partial U}{\partial t} = c^{2} \nabla U
\end{equation}

\noindent
Again, note that in my examples, the diffusion coefficient is defined as
$c^{2}$, in other places it is simply $c$.

Previously, we discretized this equation by using the forward difference
approximation for the time derivative and a central difference approximation
for the spatial second derivative. This gave us one term that was a function
of $t+\Delta t$; it was trivial to solve for this term.  However, something
interesting happens when we use central difference approximations to yield
the time and space derivative not at time $t$ but at time
$t+\frac{\Delta t}{2}$:

\begin{equation}
  \label{disct}
  \frac{\partial U(x,t+\frac{\Delta t}{2})}{\partial t} =
  \frac{U(x, t+\Delta t) - U(x, t)}{\Delta t} + \mathcal{O}(\Delta t^{2})
\end{equation}

\noindent
For the space discretization at $t+\frac{\Delta t}{2}$, we'll average the
central difference approximation for $U(x,t+\Delta t)$ and
$U(x,t)$:

\begin{align}
  \label{discx}
  \frac{\partial^{2} U(x,t+\frac{\Delta t}{2})}{\partial x^{2}} =
  \frac{1}{2\Delta x}\\
  \nonumber & \left(U(x-\Delta x,t+\Delta t) -
  2U(x,t+\Delta t) + U(x+\Delta x,t+\Delta t) + \right.\\
  \nonumber & \left. U(x-\Delta x,t) -
  2U(x,t) + U(x+\Delta x,t)\right) \\
  \nonumber & + \mathcal{O}(\Delta x^{2})
\end{align}

\noindent
Substituting Equation \ref{disct} and \ref{discx} into Equation \ref{heat}
yields

\begin{equation}
  \label{heat_disc}
  \frac{U_{i,j+1}-U_{i,j}}{k} = c^{2}\frac{U_{i-1,j+1}-2U_{i,j+1}+U_{i+1,j+1}+
    U_{i-1,j}-2U_{i,j}+U_{i+1,j}}{2h^{2}}
\end{equation}

\noindent
...where the following definitions were made:

\begin{equation}
  \label{subs}
  \begin{array}{c}
  k=\Delta t \\
  h=\Delta x \\
  U_{i+n, j+m} = U(x+n\Delta x, t+m\Delta t)
  \end{array}
\end{equation}

We have a mix of forward terms ($j+1$) and current terms ($j$).  Let's first
make the following definition:

\begin{equation}
  \label{r}
  r \equiv \frac{c^{2}k}{h^{2}}
\end{equation}

\noindent
...and now place all of the forward terms on the left and the current terms
on the right:

\begin{equation}
  \label{cn}
  -rU_{i-1,j+1} + (2+2r)U_{i,j+1} - rU_{i+1,j+1} =
  rU_{i-1,j} + (2-2r)U_{i,j} +rU{i+1,j}
\end{equation}

\noindent
This is the \textbf{Crank-Nicholson} method for solving the diffusion equation.
It is implicit and unconditionally stable.

In the domain $x=[x_{0}:x_{0}+n\Delta x]$ (or, rather more simply, $i=[1:n]$),
when we have enforced boundary conditions at $i=1$ and $i=n$, \emph{we have
$n$ equations and $n$ unknowns}.  Any $j+1$ term that is not on the boundary
is an unknown.  Casting this into matrix notation makes the situation far
more clear:

\begin{equation}
  \label{matrix}
  \begin{array}{c}
  \begin{bmatrix}
    2+2r & -r   & & & & &\\
    -r   & 2+2r & -r & & &\mathbf{0} &\\
         &   -r & 2+2r & -r & & &\\
         & & & \ddots & & &\\
     & \mathbf{0}& & & -r   & 2+2r & -r \\
     & & & & &   -r & 2+2r
  \end{bmatrix}
  \begin{bmatrix}
    U_{1,j+1} \\ U_{2,j+1}\\U_{3,j+1}\\ \vdots \\U_{n-1,j+1} \\ U_{n,j+1}
  \end{bmatrix}
  =\\
  \\
  \begin{bmatrix}
    2-2r & r   & & & & &\\
    r   & 2-2r & r & & &\mathbf{0} &\\
    &   r & 2-2r & r & & &\\
    & & & \ddots & & &\\
    & \mathbf{0}& & & r   & 2-2r & r \\
    & & & & &   r & 2-2r
  \end{bmatrix}
  \begin{bmatrix}
    U_{1,j} \\ U_{2,j}\\U_{3,j}\\ \vdots \\U_{n-1,j} \\ U_{n,j}
  \end{bmatrix}
  \end{array}
\end{equation}

The $\mathbf{A}\overline{U_{j+1}} = \mathbf{B}\overline{U_{j}}$ form of Equation
\ref{matrix} makes the final task clear.  Matrices $\mathbf{A}$ and $\mathbf{B}$
must be constructed appropriately.  Then, the only unknown at any given
iteration is vector $\overline{U_{j+1}}$.  We simply solve for it:
$\overline{U_{j+1}} =\mathbf{A}^{-1}\mathbf{B}\overline{U_{j}}$, where
$\mathbf{A}^{-1}$ is the inverse of $\mathbf{A}$.
  Note that the matrices are sparse and \textbf{tri-diagonal}.  They only have
non-zero elements along $i=j$, $i-1=j$, and $i+1=j$ diagonals.

%\section{Implementation in FORTRAN}
%%%%%%%%%%%%%%
\end{document}


\chapter{Project Description}

\section{Basics}

%Can put  this into the introduction
We use the following notation:
\begin{itemize}
\item Matrices are written in uppercase bold e.g. $\mathbf{X}$.
\item Vectors are written in lowercase bold e..g $\mathbf{x}$.
\item scalars are written in lowercase or uppercase. Lowercase indicates that it is a counting variable and uppercase that it is one of the limits in an finite set.
\item For all functions , e.g. $f(x)$, where $x = \mathbf{X}$ , we apply the function element wise.
\item Dot product is indicated by simple concatenation of two matrices/vectors e.g. $\mathbf{X} \mathbf{y}$.
\item Element wise multiplication is indicated as $\mathbf{X} \otimes \mathbf{Z}$.
\end{itemize}

Neural networks are a statistical model for regression or classification. Even though naturally they do not produce any classification output, we can model the regressed output to be used as a classifier.

Overall we have two different modes: Training and evaluation. In the training phase we use the back propagation algorithm to train the network and adjust it so that it produces output, which is close to our target output.

Furthermore a neural network contains $L$ hidden layers, which are called hidden since their output is not directly observable.

The weights are in our simple case randomly initialized and then updated using the back propagation rule.

A basic neural network has therefore the following parameters:
\begin{itemize}
\item The weights from an neuron $k$ in layer $a$ to a neuron $j$ in layer $b$
\item The input to the network, usually a vector $\mathbf{x}$ with $k$ values
\item The target which should be estimated, usually a vector $\mathbf{t}$ with $q$ values.
\item The learning rate $\eta$, which is the indicator how fast the network learns.
\end{itemize}

Moreover we can add some advanced techniques as the momentum to speed up the training the momentum

\subsection{Gradient Descent}
Gradient descent is a first-order optimization algorithm. To find a local minimum of a function using gradient descent, it takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point. If instead one takes steps proportional to the positive of the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent \cite{Kiwiel2001,Qian1999}.

Gradient descent is based on the observation that if the multivariable function $F(\mathbf{x})$ is defined and differentiable in a neighborhood of a point $\mathbf{a}$, then $F(\mathbf{x})$ decreases fastest if one goes from $\mathbf{a}$ in the direction of the negative gradient of F at $\mathbf{a}, -\nabla F(\mathbf{a})$ \cite{Yuan1999}. It follows that, if
\begin{equation}
\mathbf{b} = \mathbf{a}-\gamma\nabla F(\mathbf{a})
\end{equation}
for $\gamma$ small enough, then $F(\mathbf{a})\geq F(\mathbf{b})$. With this observation in mind, one starts with a guess $\mathbf{x}_0$ for a local minimum of F, and considers the sequence $\mathbf{x}_0, \mathbf{x}_1, \mathbf{x}_2, \dots$ such that
\begin{equation}
\mathbf{x}_{n+1}=\mathbf{x}_n-\gamma_n \nabla F(\mathbf{x}_n),\ n \ge 0.
\end{equation}

Finally we have \cite{Cauchy1847}:
\begin{equation}
F(\mathbf{x}_0)\ge F(\mathbf{x}_1)\ge F(\mathbf{x}_2)\ge \cdots,
\end{equation}

\subsection{Minibatch Stochastic Gradient Descent}

Both statistical estimation and machine learning consider the problem of minimizing an objective function that has the form of a sum:
\begin{equation}
Q(w) = \sum_{i=1}^n Q_i(w)
\end{equation}

where the parameter $w^*$ which minimizes Q(w) is to be estimated. Each summand function $Q_i$ is typically associated with the $i$-th observation in the data set (used for training).

When used to minimize the above function, a standard (or "batch") gradient descent method would perform the following iterations :
\begin{equation}
w := w - \eta \nabla Q(w) = w - \alpha \sum_{i=1}^n \nabla Q_i(w)
\end{equation}
where $\eta$ is the learning rate.

But since in many cases, the update of one gradient would take to estimate the gradient over the whole dataset, the computation would take too much time. Often times we only need a rough estimate of the gradient, not a precise one.

What stochastic gradient descent does is to remove the sum and only uses one sample to estimate the gradient.
\begin{equation}
w := w - \eta \nabla Q(w) = w - \alpha \nabla Q(w)
\end{equation}
In practice this turns out to be less effective than batch gradient descent, since we only get a very rough estimate of the gradient.
Therefore the minibatch gradient descent technique is used. The difference here is that while batch gradient descent uses $n$ samples and stochastic gradient descent uses only $1$, minibatch gradient uses $b$ training samples, where $b << n$. Typical sizes for $b$ are in the range if $b \in {2,\ldots,100}$.

Therefore in minibatch stochastic Gradient descent we randomly subset the training set and only take some samples out of it and estimate the gradient out of the seen sample set.

The important thing to do is to randomize the training set, since otherwise only a small portion of the training set will be seen and leads to wrong gradients.

\subsection{The Forward Backward procedure in detail}

To train the network we use the standard back-propagation algorithm, which essentially does a forward and a backward pass of the network in a row.
Firstly we initialize the weights in a matrix.
\begin{align}
\mathbf{W}_{j \times k } = \begin{bmatrix}
w_{11} & w_{12} & \ldots & w_{1k}\\
w_{21} & \ddots & & \vdots\\
\vdots & & \ddots  & \vdots\\
w_{j1} & \ldots & \ldots  & w_{jk}\\
\end{bmatrix},
\mathbf{x}_{1 \times k} = \begin{bmatrix}
a_{1} \\
\vdots\\
\vdots \\
a_{k}
\end{bmatrix}
\end{align}

\paragraph{Forward propagation} is one of the two passes the network needs to do ( hence it's name ).
In forward propagation the network calculates the predicted output of the network.

The output of the hidden layer $j$ of the network can be calculated as the weighted sum of the inputs
\begin{align}
\mathbf{nnet}_j = \sum_{k=1}^{n}w_{kj}x_k = \mathbf{W} \mathbf{x}
\end{align}

Even though this method should already work, we add a bias to every node to increase the learning speed and void that the network stops learning.

\begin{align}
\mathbf{nnet}_j = \sum_{k=1}^{n}w_{kj}x_k + b_k = \mathbf{W} \mathbf{x} + \mathbf{b}
\end{align}r

Later we see that back-propagation needs some variables, which can be already precomputed during the forward step. These two variables are the output $\mathbf{o}_j$ of layer $j$ and the derivative of the output w.r.t to the weighted sum $\mathbf{nnet}_j$. We represent the derivatives in a matrix $\mathbf{D}$ and the output of the current layer
\begin{align}
\mathbf{D}_j = \frac{\mathbf{o}_j}{\mathbf{nnet}_j}
\end{align}
In case of sigmoid activation function, we obtain:
\begin{align}
\mathbf{D}_j = \varphi \left( \mathbf{nnet}_j \right) \left( 1 - \varphi \left(\mathbf{nnet}_j \right) \right)
\end{align}
For the outputs, we simply feed forward the network and store the output $\mathbf{o}_j$ in out buffered lists.
\begin{align}
\mathbf{o}_j = \varphi \left( \mathbf{Wx+b} \right)
\end{align}

As soon as the feed forward is done, we produced an output of the network, which we can denote as $\mathbf{o}_L$. Now we need to decide how close our output is to the targeted output $\mathbf{t}$. For this we use a cost function in usual cases it is sufficient to use $MSE$ as such, which is defined as:
\begin{equation}
\label{eq:mse}
\mathbf{MSE} = \frac{1}{2} \left( \mathbf{o}_L - \mathbf{t} \right) ^2
\end{equation}

Here (\ref{eq:mse}) we need to make sure that the last output layer has the same dimensions as the target output.

\paragraph{Backpropagation} starts when all layers are trained. The idea behind back propagation is that the hidden layers do not produce any output. So we cannot modify their weights directly, since we don’t know how large the error was. With back propagation we calculate the error at the output layer $L$ and then propagate this error to the hidden layers back. Therefore we update all $L-1$ weight matrices and biases.

At first we calculate the differences between the target output and the estimated output.

\begin{align}
\boldsymbol{err} = \mathbf{o}_L - y\\
\boldsymbol{\delta}_L &= \mathbf{err} \otimes \mathbf{D}_L\\
\nabla \mathbf{W}_L &= \mathbf{o}_L \boldsymbol{\delta}_L^T
\end{align}

We store both, the deltas and the nablas to later update the weights ( with the $\nabla$s) and the biases (using the $\mathbf{\delta}$).
From here on we then calculate the other layers backwards, we do:
\begin{align}
\boldsymbol{\delta}_i &= \mathbf{D}_i \otimes \left( \mathbf{W}_{i+1}^T \boldsymbol{\delta}_{i+1} \right)\\
\nabla \mathbf{W}_i &= \mathbf{o}_i \boldsymbol{\delta}_i^T
\end{align}
Whereas we again store the deltas and the nablas to later update the weights.

To update the weights, we use the usual gradient descent update rule:
\begin{align}
\mathbf{W}^{*} = \mathbf{W} - \alpha \nabla \mathbf{W}_i^T\\
\end{align}

To update the biases, we use essentially the same update rule, but only consider the given $\boldsymbol{\delta}_{i}$s.
\begin{equation}
\mathbf{b}^{*} = \mathbf{b} - \alpha \boldsymbol{\delta}_i^T\\
\end{equation}

Finally, if we would like to improve the converging speed we can apply momentum. Momentum adds extra "velocity" towards the gradient curve, by using the last estimated value of the gradient $\mathbf{W}^{*}_{i-1}$ ($i$ denotes the current iteration) and applying on that the momentum ($ \alpha$) as:

\begin{equation}
\mathbf{W}_{i+1} = \mathbf{W}_i - \eta \nabla \mathbf{W}_i + \alpha \mathbf{W}_i
\end{equation}

The momentum is initialized with 0 and takes effect after one iteration of gradient descent.

\section{Implementation details}

OpenCL offers multiple implementation types. It is natively written in C, but has also a C++ wrapper onboard. We used in our project the C++ wrapper since it is more naturally to use that in a C++ project.
First one needs to include the necessary header into any class.

\begin{lstlisting}[caption=OpenCL C++ header]
#include <CL/cl.hpp>
\end{lstlisting}

\paragraph{The interface to OpenCL} was written in C++11 and makes heavily use of the current variadic args feature. For this small scale task, the interface is definitely too complex, but it can be used for any other OpenCL task.

\label{lst:mult}
\begin{lstlisting}[caption=Example Kernel function]
__kernel void mult(const int wSrc, __global const float* A,__global const float* B,__global float* output)
{
   const int idx = get_global_id(0);
   const int idy = get_global_id(1);

   output[idy*wSrc+idx] = A[idy*wSrc+idx] * B[idy*wSrc+idx];
}
\end{lstlisting}
OpenCL uses externally defined functions ( coined as kernels ) to compile the code and run it, during the runtime. Therefore one needs to write a kernel for its needs. An example kernel can be seen at \ref{lst:mult}. In our case we defined various kernels, for dot products and other matrix operations.

Since OpenCL is a multi device and platform GPU/CPU interface, one needs to first figure out which kind of platform (e.g. NVIDIA,AMD,Intel) the current machine is using and then decide which accelerator will be used \ref{lst:plat}.

\label{lst:plat}
\begin{lstlisting}[caption=Get Platforms and devices]
void exampleplatform(const char * programpath){
std::vector<cl::Platform> all_platforms;
cl::Platform::get(&all_platforms);
//Assume having only one platform
cl::Platform defaultplatform = all_platforms[0];
std::vector<cl::Device> all_gpu_devices;
//CL_DEVICE_TYPE_GPU can also be CL_DEVICE_TYPE_CPU for CPU
defaultplatform.getDevices(CL_DEVICE_TYPE_GPU, &all_gpu_devices);
//Assume having only one device / take the first
cl::Device device = all_gpu_devices.front();
//Init the current context with the device
cl::Context(device);
//We wrote a helper function here to get the string data out of the kernel file
const char* content = util::file_contents(programpath);
}
\end{lstlisting}

Moreover since OpenCL is compiled during runtime, one needs to extract the code from the kernel file (e.g. kernel.cl) and give this content to the OpenCL.

As soon as the context is initialized we can run our kernels on the device. The problem here is that the kernel is defined with it's own parameters and types, but we dont know these during the compile time.
To have a universal interface we used the variadicargs feature to allow the programmer a very straight forward way to use any kernel function.
\label{lst:kernel}
\begin{lstlisting}[caption=Kernel usage]
void runKernel(const char *kernelname){
cl::Program::Sources sources;
//contents is the already read out content of the kernels file (e.g. kernels.cl)
sources.push_back(std::make_pair(contents,strlen(contents)+1));
//Init the program with the context and the source
cl::Program program(context,sources);
//Build the program on the device
program.build({device});
cl::CommandQueue queue(context,device);
//The operator allows us to set args to the kernel
cl::Kernel kernel_operator(program,kernelname);
//kernel_operator allows us to send arguments to the kernel by calling
//kernel_operator.setArgs(ARGNUMBER,ARGUMENT);

//Init space on the device using cl::Buffers
//Send Arguments to the device ....
// quene.enqueueWriteBuffer() .....
//Wait until the arguments did arrive on the device
queue.finish();

//Execute!
cl::Event event;
queue.enqueueNDRangeKernel(kernel_operator,cl::NullRange,cl::NDRange(10),cl::NullRange,NULL,&event);
//Wait until the execution has finished
event.wait();
// Read out the results by calling
// quene.enqueueReadBuffer()
}
\end{lstlisting}

The usual Kernel execution can be seen in \ref{lst:kernel}. Even though it is recommended to use OpenCL in this fashion, we cannot cope with different arguments for the kernel. Therefore we would need to init a device and context every time ( or at least as a singleton ) and then rewrite the argument passing for every parameter independently. This would take a lot of time, if the GPU is used extensively.

Our solution to that problem is as follows:

\begin{lstlisting}[caption=Add arguments to kernel dynamic way]
class OpenCL{

//Constructor ..... etc
//	Hook for the iteration
template<std::size_t P=0,typename... Tp>
	typename std::enable_if<P == sizeof...(Tp), void>::type addkernelargs(std::tuple<Tp ...>&& t,cl::Kernel &k,cl::CommandQueue &,std::vector<cl::Buffer> &outputbuffers) const{
	// Do nothing
	}

//  Start of the iteration
template<std::size_t P = 0, typename... Tp>
	typename std::enable_if< P < sizeof...(Tp), void>::type addkernelargs(std::tuple<Tp...> && t,cl::Kernel &kernel,cl::CommandQueue &,std::vector<cl::Buffer> &outputbuffers) const{
		 // Type
        typedef typename std::tuple_element<P, std::tuple<Tp...>>::type type;

        // Add the value of the current item from std::get<P> to the args in kernel
        // This function decides which type the kernel arg is
        addkernelarg(P, std::get<P>(t), kernel,queue,outputbuffers);

        // Recurse to get the remaining args
        addkernelargs<P + 1, Tp...>(std::forward<std::tuple<Tp...>>(t), kernel,queue,outputbuffers);
	}

//	Adding Std::vector as type to the kernel args list
	template<typename T>
	void addkernelarg(std::size_t i, std::vector<T> const & arg, cl::Kernel & kernel,cl::CommandQueue &) const{
		cl::Buffer buffer(this->context,CL_MEM_READ_WRITE,arg.size()*sizeof(T));
		queue.enqueueWriteBuffer(buffer,CL_FALSE,0,sizeof(T)*arg.size(),&(arg[0]));
		kernel.setArg(i,buffer);

	}

//	Adding any array into the kernel args
	template<typename T,std::size_t N>
	void addkernelarg(std::size_t i, T const (& arg)[N], cl::Kernel & kernel,cl::CommandQueue &) const{
		cl::Buffer buffer(this->context,CL_MEM_READ_WRITE,N*sizeof(T));
		queue.enqueueWriteBuffer(buffer,CL_FALSE,0,sizeof(T)*N,&arg);
		kernel.setArg(i,buffer);
	}

	// Adding any constant to the kernel
	template<typename T>
	void addkernelarg(std::size_t i, T const & arg, cl::Kernel & kernel,cl::CommandQueue &) const{
		cl::Buffer buffer(this->context,CL_MEM_READ_WRITE,arg.size()*sizeof(T));
		queue.enqueueWriteBuffer(buffer,CL_FALSE,0,sizeof(T)*arg.size(),&(arg[0]));
		kernel.setArg(i,buffer);
	}

}
\end{lstlisting}

This recipe can be used to do the same actions for the reading buffers out after the transfer has finished.
Therefore we can create a highly dynamic wrapper class for OpenCL.

\subsection{Using the Interface}


\section{Mathematical Preliminaries}
\label{sec:prelim}
In this paper, vector- and tensor-valued quantities
are denoted by bold letters (e.g. $\bh$ and $\mathbf{T}$).
Subscript indices of non-bold characters (e.g. $h_j$ or $T_{ij\ell}$)
are used to denote the entries within a vector or tensor.
We use the standard Einstein summation convention; i.e.,
there is an implied sum taken over the repeated indices of
any term (e.g. the symbol $a_{j} b_{j}$ is used to represent the sum
$\sum_{j} a_{j} b_{j}$).
If $\xx = (x_1,x_2)^\intercal$, then $\xx^\bot = (-x_2,x_1)^\intercal$.
Similarly, $\nabla^\bot = (-\partial_{x_2},\partial_{x_1})^\intercal$.
Upper-case script characters (e.g. $\mathcal{K}$) are reserved for
operators on Banach spaces, with $\mathcal{I}$ denoting the
identity. Given a set $X$, we denote the closure of $X$
by $\overline{X}$.

In the rest of the paper,
suppose that $\Omega$ is a multiply connected region defined by
the intersection of a simply connected domain $\Omega_{0}$
and the exteriors of a finite collection of bounded
simply connected domains $\{ \Omega_{i} \}_{i=1}^{m}$.
Let $\Gamma_{i}$, denote the boundary of $\Omega_{i}$, and
let $\Gamma = \cup_{i=0}^{m} \Gamma_{i}$ denote the boundary of $\Omega$.
Let $\bnu(\bx)$ denote the outward normal at $\bx \in \Gamma$.

\section{Stokes eigenvalue problem}
The Stokes Dirichlet eigenvalue problem is to find values $k^2$ such that
\begin{equation}
\begin{aligned}
  -\Delta \uu + \nabla p &= k^2 \uu \quad \textrm{in} \quad
  \Omega \label{eq:ostokes} \; , \\
  \nabla \cdot \uu &= 0 \; , \\
  \bu &=0 \quad \textrm{on} \quad \Gamma \, ,
\end{aligned}
\end{equation}
has a non-trivial solution $(\bu,p)$.
Since the Stokes eigenfunction $\bu$ is divergence free,
and satisfies
$\int_{\Gamma_{i}} \bu \cdot \bnu \, ds = 0$
for all $i=0,1,2\ldots m$, there exists a single valued
stream function $\psi$ associated with $\bu$, i.e.,
$\bu = \nabla^{\perp} \psi$ for all $\bx \in \Omega$.

The Stokes eigenvalue problem can be reformulated as a buckling
eigenvalue problem with gradient boundary conditions for
the stream function $\psi$.
Plugging in $\bu = \nabla^{\perp} \psi$ and applying $\nabla^{\perp} \cdot$
to~\cref{eq:ostokes}, we observe that $\psi$ satisfies the following
differential equation.

\begin{equation}
\begin{aligned}
\Delta (\Delta + k^2) \psi &= 0 \, \quad \textrm{in} \quad \Omega \, , \\
\nabla \psi &=0 \, ,\quad \textrm{in} \quad \Gamma \, .
\end{aligned}
\end{equation}

\section{Connection to buckling eigenvalues}
It is clear that if $w$ is a buckling eigenfunction with
eigenvalue $k^2$, then $\bu = \nabla^{\perp} w$ is a Stokes eigenfunction
with the same eigenvalue.
Thus, for a given region $\Omega$,
the buckling eigenvalues are a subset of the Stokes eigenvalue.

However, the converse is not necessarily true.
The reason is the following.
As before, suppose that $\bu$ is a Stokes eigenfunction
with eigenvalue $k^2$.
Note that a single valued stream function $\psi$ associated with
$\bu$ always exists.
Moreover, on each component $\Gamma_{i}$,
$\psi$ satisfies $\partial_{\tau} \psi = 0$, which implies that
$\psi$ is a constant on $\Gamma_{i}$, for each $i=0,1,2\ldots m$.
If the domain is simply connected, then since $\psi$ is only
well-defined upto a constant, we can adjust to constant term
to ensure that $\psi$ also satisfies $\psi = 0$ on $\Gamma_{0}$.
However, on multiply connected domains, this is not always
the case since $\psi$ might be different constants on different
boundary components.
In the following lemma, we identify necessary and sufficient
conditions on the stream function $\psi$ to guarantee that
$\psi$ (upto a constant) is also a buckling eigenfunction.

\begin{lem}
\label{lem:mainlem}
Suppose that $k^2$ is a Stokes eigenvalue for the region $\Omega$ and
let $\bu$ be the corresponding eigenfunction.
Then, the stream function associated with the velocity $\bu$ defined by
$\psi = \left(\nabla^{\perp} \right)^{-1} \bu$ exists and is single valued.
Furthermore, $k^2$ is also a buckling eigenvalue if and only if
there exists a constant $c$ such that
\begin{equation}
\frac{1}{\Gamma_{j}}\int_{\Gamma_{j}} \psi\, ds = c \quad \forall j=0,1,\ldots m \, .
\end{equation}
\end{lem}

\begin{proof}
Since $\bu$ is a Stokes eigenfunction, it satisfies
\begin{equation}
\int_{\Gamma_{j}} \bu \cdot \bn ds = 0 \quad j=1,2\ldots n \, .
\end{equation}
Thus, there exists a single valued stream function associated with the
velocity field $\bu$ such that
\begin{equation}
\nabla^{\perp} w = \bu \, .
\end{equation}
Furthermore, since $\bu$ satisfies the oscillatory Stokes equations,
$w$ satisfies
\begin{align}
\Delta (\Delta + k^{2} ) w &= 0 \, , \quad \bx \in \Omega \\
\nabla^{\perp} w = \bu &=0 \, ,\quad \bx \in \Gamma \, ,
\end{align}
In particular $\psi= c_{j}$ for $\bx \in \Gamma_{j}$ for
some constant $c_{j}$.
Thus $k$ is also a buckling eigenvalue if and only if
\begin{equation}
c_{j} = \frac{1}{|\Gamma_{j}|}\int_{\Gamma_{j}} w ds =  c \, .
\end{equation}
The buckling eigenfunction is then given by $\psi -c$.
\end{proof}

\section{Computing the stream function}
Given $k^2$ to be a Stokes eigenvalue, in this section, we describe
an integral formulation for computing the eigenfunction $\bu$
and the corresponding stream function $\psi$.

Following the approach in~\cite{askhameig2019}, we
first reformulate the oscillatory Stokes equation
as an integral equation on the boundary using the combined
field representation.

Let $\GG(\bx,\by)$ denote the Stokeslet for the oscillatory
Stokes equation and $T_{ij\ell}(\bx,\by)$ denote the corresponding
Stresslet.
These are given by the formulae
\begin{align}
  \GG &= - \II \Delta \Gbh + \nabla \otimes \nabla \Gbh
\label{eq:ostokeslet} \\
  T_{ij\ell} &= - \partial_{x_j} \Glap \delta_{i\ell}
  + \partial_{x_\ell} \left ( -\Delta \Gbh \delta_{ij} +
  \partial_{x_i} \left(\partial_{x_j} \Gbh \right) \right)
  \nonumber \\
  & \qquad+ \partial_{x_i} \left ( -\Delta \Gbh \delta_{\ell j} +
  \partial_{x_\ell} \left(\partial_{x_j} \Gbh \right) \right)
  \; . \label{eq:ostress}
\end{align}
$\Gbh$ is the Green's function for the oscillatory
biharmonic equation given by
\begin{equation}
  \Gbh(\xx,\yy) = \frac{1}{k^2}
  \left (\frac{1}{2\pi} \log |\xx-\yy| +
  \frac{i}{4} H_0^{(1)}(k|\xx-\yy|) \right ) \, ,
  \label{eq:Gbh}
\end{equation}
where $k$ is the Helmholtz parameter in the oscillatory biharmonic equation,
and $H_{0}^{1}(r)$ is the Hankel function of the first kind of order zero,
and $\Glap(\xx,\yy)$ is the Laplace Green's function given by
\begin{equation}
  \Glap(\xx,\yy) = \frac{1}{2\pi} \log |\xx-\yy| \; . \nonumber
\end{equation}

Given a density $\bmu$, the oscillatory Stokes single and double layer
potentials denoted by $\bS[\bmu](\bx)$ and $\bD[\bmu](\bx)$ respectively,
are given by
\begin{align} \label{eq:singlelayer}
  \bS [\bmu] (\xx) &= \int_\Gamma \GG (\xx,\yy) \bmu(\yy)
  \, dS(\yy) \; , \\
  \bD [\bmu] (\xx) &= \int_\Gamma \left ( \TT_{\cdot,\cdot,\ell}(\xx,\yy)
  \nu_\ell(\yy)\right )^\intercal \bmu(\yy) \, dS(\yy) \; .
\end{align}
When solving the velocity boundary value problem for the oscillatory
Stokes equation on multiply connected domains, the velocity
is represented as a bomcinbed field layer potential, i.e.
setting $\bu = (i\eta \bS_{k} + \bD_{k})[\bmu]$, where
$\eta>0$ is a constant and $\bmu$ now is an unknown density.
By construction, this velocity field satisfies the oscillatory
Stokes equation.
On imposing the velocity boundary condition, and using the jump
conditions for the layer potentials (see~\cref{askhameig2019},
for example)
we obtain the following integral equation for the unknown density
$\bmu$,
\begin{equation}
\label{eq:inteq0}
(\cI - 2\cD_{k} -2i \eta \cS_{k})\bmu = 0 \, \quad \textrm{on} \quad \Gamma \,,
\end{equation}
where $\cS_{k}$ and $\cD_{k}$ are the restrictions of the layer potentials
$\bS_{k}$ and $\bD_{k}$ on the boundary $\Gamma$ respectively, and are given
by
\begin{align}
  \cS [\bmu] (\xx) &= \int_\Gamma \GG (\xx,\yy) \bmu(\yy)
  \, dS(\yy) \\
  \cD [\bmu] (\xx) &= \pv \int_\Gamma \left ( \TT_{\cdot,\cdot,\ell}(\bx,\by)
  \nu_\ell(\yy)
  \right )^\intercal \bmu(\yy) \, dS(\yy) \; .
\end{align}
Here \pv indicates that the integral is to be
evaluated in the principal value sense.
The integral equation~\cref{eq:inteq0} is known to be rank-deficient
for all values of $k$ due to the divergence-free condition
on $\bu$, and a standard approach is to use the following integral
equation instead

\begin{equation}
\label{eq:inteq}
(\cI - 2\cD_{k} -2i \eta \cS_{k}-2\cW)\bmu = 0 \, \quad \textrm{on} \quad \Gamma \,,
\end{equation}
where $\cW$ is the operator given by
\begin{equation}
\cW[\bmu](\bx) = \frac{\bnu(x)}{|\Gamma|}\int_{\Gamma} \bmu(\by) \cdot \bnu(\by) \, ds \, .
\end{equation}

In~\cite{askhameig2019}, the authors show that the operator
$\cI -2\cD_{k} -2i \cS_{k} -2\cW$ is not invertible if and only
if $k^2$ is a Stokes eigenvalue.
Furthermore if $\bmu$ is a non-zero null-vector of $\cI - 2\cD_{k} -2i \cS_{k}
-2 \cW$, then
$\bu = (i \eta \bS_{k} + \bD_{k})[\bmu]$ is the Stokes eigenfunction.

In order to verify the conditions in~\cref{lem:mainlem}, we
need to be able to compute the stream function associated with
the velocity $\bmu$ defined above.
If $\bmu = \mu_{\tau} \btau + \mu_{\nu} \bnu$, where $\btau=\bnu^{\perp}$
is the positively oriented tangential vector then the
layer potentials $\bS_{k}[\bmu]$ and $\bD_{k}[\bmu]$ can be rewritten as
\begin{align}
\bS_{k}[\bmu] &=
\nabla^{\perp}
\int_{\Gamma} (\partial_{\tau} \Gbh(\xx,\yy) \mu_{\tau}(\yy) -
\partial_{\nu} \Gbh(\xx,\yy) \mu_{\nu}(\yy)) \, ds \\
\bD_{k}[\bmu] &=
-\nabla
\int_{\Gamma} \Glap(\xx,\yy) \mu_{\nu}(\yy) \, ds
+ \nabla^{\perp}
\int_{\Gamma} \big( 2 \partial_{\nu\tau} \Gbh(\xx,\yy) \mu_{\nu}(\yy)
+ \\
& \hspace*{35ex}(\partial_{\tau\tau} -\partial_{\nu\nu})\Gbh(\xx,\yy) \mu_{\tau}(\yy)
\big) \, ds \, .
\end{align}

From the expressions above, it is clear that the stream function
$\psi_{S}[\bmu]$ associated with the single layer potential is given
by
\begin{equation}
\psi_{S}[\bmu] =
\int_{\Gamma} (\partial_{\tau} \Gbh(\xx,\yy) \mu_{\tau}(\yy) -
\partial_{\nu} \Gbh(\xx,\yy) \mu_{\nu}(\yy)) \, ds \, .
\end{equation}

The situation with the double layer potential is a little trickier.
We write the double layer potential as $\bD_{k}[\bmu] = \nabla \phi_{D}[\bmu]
+ \nabla^{\perp} \xi_{D}[\bmu]$,
where
\begin{align}
\label{eq:phiddef}
\phi_{D}[\bmu] &=
-\int_{\Gamma} \Glap(\xx,\yy) \mu_{\nu}(\yy) \, ds  \, , \\
\xi_{D}[\bmu] &=
\int_{\Gamma} \big( 2 \partial_{\nu\tau} \Gbh(\xx,\yy) \mu_{\nu}(\yy)
+ (\partial_{\tau\tau} -\partial_{\nu\nu})\Gbh(\xx,\yy) \mu_{\tau}(\yy) ) \,
ds \, .
\end{align}

If we can find a function $\tilde{\psi}_{D}[\bmu]$ such
that
\begin{equation}
\label{eq:perptogradperp}
\nabla^{\perp} \tilde{\psi}_{D}[\bmu] = \nabla \phi_{D} [\bmu] \, ,
\end{equation}
then the stream function $\psi_{D}[\bmu]$
associated with the double layer potential would
be given by $\psi_{D}[\bmu] = \tilde{\psi}_{D}[\bmu] + \xi_{D}[\bmu]$.
However, on multiply connected domains, it is not necessarily true,
that given a function $\phi_{D}[\bmu]$ of the form~\cref{eq:phiddef}
there exists a single valued function $\tilde{\psi}_{D}[\bmu]$
which satisfies the condition~\cref{eq:perptogradperp} for
every $\bmu$.
Turns out, in the case where $\bmu$ satisfies the conditions
\begin{equation}
\label{eq:inteqconst}
\int_{\Gamma_{i}} \mu_{\nu} \, ds = 0 \, ,\quad i=0,1,2\ldots m \, ,
\end{equation}
then there exists a single valued function $\tilde{\psi}_{D}[\bmu]$
which satisfies~\cref{eq:perptogradperp} (see~\cite{rachh2015integral},
for example).
In the following lemma, we show that every null-vector of
$\cI - 2\cD_{k} -2i \eta \cS_{k} - 2\cW$ satisfies the above integral
constraints.

\begin{lem}
Suppose $\bmu$ is a null-vector of
$\cI - 2\cD_{k} - 2i \eta \cS_{k} -2\cW$, then
$\bmu$ satisfies~\cref{eq:inteqconst}.
\end{lem}
\begin{proof}
Insert proof here
\end{proof}


\subsection{Preordered sets}\label{subsec:preordered_sets}

\begin{definition}\label{def:preordered_set}\mcite[def. 2.14]{OpenLogicFull}
  A \term{preordered set} is a set \( \mscrP \) endowed with a \hyperref[def:binary_relation/reflexive]{reflexive} and \hyperref[def:binary_relation/transitive]{transitive} \hyperref[def:binary_relation]{binary relation} \( \leq \). The relation itself is called a \term{preorder}.

  It is conventional to use the same symbol \( \leq \) as for \hyperref[def:partially_ordered_set]{partial orders}, however the lack of \hyperref[def:binary_relation/antisymmetric]{antisymmetry} may be confusing --- see \fullref{ex:preorder_nonuniqueness}.

  We define \( \geq \) as the \hyperref[def:binary_relation/converse]{inverse relation} of \( \leq \).

  Preordered sets have the following metamathematical properties:
  \begin{thmenum}
    \thmitem{def:preordered_set/theory} Consider a \hyperref[def:first_order_language]{first-order language} \( \mscrL \) with two \hyperref[rem:first_order_formula_conventions/infix]{infix} binary predicate symbols --- \( \leq \) and \( \geq \).

    The theory of preordered sets is a \hyperref[def:first_order_theory]{first-order theory} in \( \mscrL \) consisting of the axioms \eqref{eq:def:binary_relation/reflexive} and \eqref{eq:def:binary_relation/transitive} for \( \leq \) and the compatibility axiom
    \begin{equation}\label{eq:def:preordered_set/theory}
      (\xi \leq \eta) \leftrightarrow (\eta \geq \xi).
    \end{equation}

    \thmitem{def:preordered_set/homomorphism} A \hyperref[def:first_order_homomorphism]{homomorphism} from \( (\mscrP, \leq_\mscrP) \) to \( (\mscrQ, \leq_\mscrQ) \) is, explicitly, a function \( f: \mscrP \to \mscrQ \) such that
    \begin{equation}\label{eq:def:preordered_set/homomorphism}
      x \leq_\mscrP y \T{implies} f(x) \leq_\mscrQ f(y).
    \end{equation}

    These are precisely the \hyperref[eq:def:partially_ordered_set/homomorphism/nonstrict]{nonstrict monotone maps}.

    \thmitem{def:preordered_set/submodel} Since the theory contains only positive formulas over a language with no functional symbols, any subset \( A \) of the domain of a preordered set \( \mscrP \) becomes a preordered set with the induced preorder \( \leq_A \) defined as the \hyperref[def:binary_relation/restriction]{restriction} of \( \leq_\mscrP \) to only elements of \( A \).

    \thmitem{def:preordered_set/trivial} The \hyperref[thm:substructures_form_complete_lattice/bottom]{trivial preordered set} is the empty set (see \fullref{rem:empty_models} regarding allowing empty sets as first-order structures).

    \thmitem{def:preordered_set/category}  We denote the \hyperref[def:category_of_small_first_order_models]{category of \( \mscrU \)-small models} for the theory of preordered sets by \( \ucat{PreOrd} \).

    This category is equivalent to that of \( \mscrU \)-small thin categories --- see \fullref{thm:order_category_isomorphism/preordered}.

    \thmitem{def:preordered_set/duality} We define the \term{dual preordered set} of \( (\mscrP, \leq) \) as \( (\mscrP, \geq) \).

    The \term{principle of duality} states that the formula \( \varphi \) is derivable in the \hyperref[def:preordered_set/theory]{theory of preordered sets} if and only if the dual formula \( \varphi^{-1} \), in which we swap all instances of \( \leq \) and \( \geq \), is also derivable. Observe that \( \varphi \) is satisfied in a preordered set if and only if \( \varphi^{-1} \) is satisfied in the dual preordered set.

    There is a actually a very simple proof. If \( \varphi \) is derivable in the theory, it is satisfied by every preordered set. Let \( \mscrP \) be a preordered set. Then \( \varphi \) is satisfied in both \( (\mscrP, \leq) \) and its dual \( (\mscrP, \geq) \). But if \( \varphi \) is valid for the dual preordered set \( (\mscrP, \geq) \), its dual formula \( \varphi^{-1} \) is valid for the the double dual preordered set, which is \( (\mscrP, \leq) \). Since \( (\mscrP, \leq) \) was chosen arbitrarily, the dual formula \( \varphi^{-1} \) is valid for every preordered set and, so it belongs to the theory of preordered sets.

    The dual of the dual formula of \( \varphi \) is obviously \( \varphi \). The actual replacement can be formalized by performing the \hyperref[def:first_order_substitution/term_in_formula]{simultaneous substitution}
    \begin{equation*}
      \begin{aligned}
        \varphi^{-1} \coloneqq \varphi[
          &\xi_1 \leq \eta_1 \mapsto \xi_1 \geq \eta_1, &&\xi_1 \geq \eta_1 \mapsto \xi_1 \leq \eta_1, \\
          &\vdots                                       &&\vdots \\
          &\xi_n \leq \eta_n \mapsto \xi_n \geq \eta_n, &&\xi_n \geq \eta_n \mapsto \xi_n \leq \eta_n]
      \end{aligned}
    \end{equation*}
    for all pairs \( (\xi_k, \eta_k) \) of free variables in \( \varphi \).

    Another form of this duality is formalized in \fullref{thm:order_category_isomorphism}.
  \end{thmenum}
\end{definition}

\begin{definition}\label{def:directed_set}\mcite[8]{Engelking1989}
  A \hyperref[def:preordered_set]{preordered set} \( \mscrP \) is called a \term{directed set} if every finite subset of \( \mscrP \) has an \hyperref[def:partially_ordered_set_extremal_points/upper_and_lower_bounds]{upper bound}, i.e. for all \( x, y \in \mscrP \) there must exist \( z \in \mscrP \) such that \( x \leq z \) and \( y \leq z \). We do not care how many upper bounds exist and how they are related, we simply need one upper bound to exist for every pair of elements of \( \mscrP \).

  There is no established name for the relation itself.

  Directed sets are used to define nets in topological spaces, see \fullref{def:topological_net}.
\end{definition}

\begin{definition}\label{def:cofinal_set}
  A subset \( A \) of a preordered set \( (\mscrP, \leq) \) is called \term{cofinal} if for every \( x \in \mscrP \) there exists some \( y \in A \) such that \( x \leq y \).
\end{definition}

\begin{example}\label{ex:def:cofinal_set}
  We list several examples of \hyperref[def:cofinal_set]{cofinal} and non-cofinal sets.

  \begin{itemize}
    \item In a finite set like \( \set{ 0, 1, 2 } \), the set \( \set{ 2 } \) containing the maximum is cofinal. This is generalized by \fullref{thm:partially_ordered_cofinal_equivalences}.

    \item Consider the set \( \BbbZ \) of integers. Clearly the set \( 2\BbbZ \) of even integers is cofinal. This is generalized by \fullref{thm:totally_ordered_cofinal_equivalences}.

    \item Cofinal sets are important in topology because it is used to define \hyperref[def:net_convergence]{convergence of nets}.

    \item \hyperref[def:regular_cardinal]{Regular cardinals} are equal to their own \hyperref[def:cofinality]{cofinality}.
  \end{itemize}
\end{example}


\documentclass{article}

\usepackage{style/preamble}
\usepackage{style/mytikz}
\usepackage{parskip}

\begin{document}
  \title{Problem Set 1 - The Riemann Sphere}
  \date{}
  \maketitle

    \begin{mdframed}
      Corrected open mapping theorem statement:
      \begin{theorem}[Open mapping theorem]
        \label{th:OpenMappingTheorem}
        If $f:U \rightarrow \bbc$ is a \emph{non-constant} complex differentiable then for any open set $V \subseteq U$, the set $f(V)$ is an open subset of $\bbc$.
      \end{theorem}
    \end{mdframed}

    \section*{Holomorphic functions on $\bbp^1$}
    A complex differentiable function $f: X \rightarrow \bbc$ is called a \emph{holomorphic} function on $X$.

    The Riemann sphere $\bbp^1$ is the set $\bbc \cup \set{\infty}$.
    We write $\bbp^1$ as the union of two sets
    \begin{align*}
      U_1 = \bbc && U_2 = \bbc \setminus \set{0} \cup \set{\infty}
    \end{align*}
    The set $U_1$ is the standard complex plane, but the set $U_2$ is not.
    We can turn $U_2$ into the complex plane by using the following function
    \begin{align*}
      \varphi_2 : U_2 &\longrightarrow \bbc \\
      z &\mapsto z^{-1} \mbox{ if } z \neq \infty \\
      \infty &\mapsto 0
    \end{align*}
    Thus we can think of $\bbp^1$ as two copies of the complex plane ($U_1$ and $\varphi_2(U_2)$) glued together.

    A function $f: \bbp^1 \rightarrow \bbc$ is defined as a pair of functions
    \begin{align*}
      f_1 : U_1 \rightarrow \bbc && f_2 : U_2 \rightarrow \bbc
    \end{align*}
    such that $f_1$ and $f_2$ agree on $U_1 \cap U_2$.

    We can only make sense of the complex differentiable functions when both the source and target are open subsets of $\bbc$. For this reason, we define holomorphic functions on $\bbp^1$ as follows.

    A function $f: \bbp^1 \rightarrow \bbc$ is holomorphic if
    \begin{enumerate}
      \item $f|_{U_1}$ is holomorphic,
      \item $f \circ \varphi_2^{-1}|_{\varphi(U_2)}$ is holomorphic.
    \end{enumerate}
    \begin{equation*}
      \begin{tikzcd}
        U_1 \ar[r,"f|_{U_1}"] &  \bbc\\
        U_2 \ar[d, "\varphi_2"] \ar[r, "f"] & \bbc \\
        \bbc \ar[u, "\varphi_2^{-1}", bend left] \ar[ur, "f \circ \varphi_2^{-1}", swap, dashed]
      \end{tikzcd}
    \end{equation*}

    \begin{qbox}
      Check that defining a function $f: \bbp^1 \rightarrow \bbc$ is equivalent to defining a pair of functions
      \begin{align*}
        f_1: U_1 &\longrightarrow \bbc \\
        f_2 \circ \varphi_2^{-1}: \varphi(U_2) &\longrightarrow \bbc \\
      \end{align*}
      such that
      \begin{align*}
        f_1(z) = f_2 \circ \varphi_2^{-1}(z^{-1}) \mbox{ whenever } z \neq 0.
      \end{align*}
      Note that the source of both $f_1$ and $f_2 \circ \varphi_2^{-1}$ is $\bbc$.
    \end{qbox}

    It is kinda hard to come up with examples because of the following theorem.

    \begin{theorem}
      The only holomorphic functions on $\bbp^1$ are the constant functions.
    \end{theorem}
    The proof is in the following exercises.

    A subset $V$ of a topological space $X$ is compact if it has the following property.
    \begin{quote}
      Every infinite sequence has a convergent subsequence i.e. for every infinite sequence of points $a_1, a_2, \dots $ in $V$ there exists a subsequence $a_{n_1}, a_{n_2}, a_{n_3}, \dots$ which converges to a point in $V$.
    \end{quote}
    % For subsets of $\bbr^n$ or $\bbc^n$ this is equivalent to saying that $X$ is closed and bounded.

    \begin{qbox}
      Prove that compact subsets $V$ of a (nice) topological space $X$ are closed i.e. if a sequence of points $a_1, a_2, \dots $ in $V$ converges to point $a \in X$ then $a$ is in $V$.
    \end{qbox}

    \begin{qbox}
      Prove that every compact subset $V$ of $\bbc$ is bounded i.e. there exists a real number $M$ such that $z < M$ for all $z \in V$.
    \end{qbox}

    \begin{qbox}
      \label{q:imageOfCompact}
      Use the fact that every infinite sequence has a convergent subsequence to argue that for any continuous function $g: X \rightarrow Y $ the image of a compact set is compact.
    \end{qbox}

    Assume the following fact:
    \begin{quote}
      The Riemann sphere is a compact topological space.
    \end{quote}
    One way to see this is that the Riemann sphere is topologically isomorphic to the sphere $S^2$ in $\bbr^3$ (hence the name) which is a closed and bounded subset of $\bbr^3$. It is not hard to show that such subsets are compact.

    \begin{qbox}
      Argue that the image of any continuous function $f: \bbp^1 \rightarrow \bbc $ is bounded.
    \end{qbox}

    \begin{qbox}
      Using Liouville's theorem, argue that if $f$ is a holomorphic function on $\bbp^1$ then $f|_{U_1}$ is a constant function.
    \end{qbox}
    \begin{qbox}
      Using continuity, argue that if $f$ is a holomorphic function on $\bbp^1$ then $f$ is a constant function.
    \end{qbox}

    \section*{Meromorphic functions on $\bbc$}
    A complex differentiable function $f: X \rightarrow \bbp^1$ is called a \emph{meromorphic} function on $X$.

    We can only make sense of the complex differentiable functions when both the source and target are open subsets of $\bbc$. For this reason, we define meromorphic functions on $X$ as follows.

    A function $f: X \rightarrow \bbp^1$ is meromorphic if
    \begin{enumerate}
      \item $f$ is holomorphic when restricted to $f^{-1}(U_1)$,
      \item $\varphi_2 \circ f$ is holomorphic when restricted to ${f^{-1}(U_2)}$.
    \end{enumerate}
    \begin{equation*}
      \begin{tikzcd}
        f^{-1}(U_1) \ar[rr,"f"] & &  U_1 = \bbc\\
        f^{-1}(U_2) \ar[rr, "f"] \ar[rrrr, "\varphi_2 \circ f", swap, bend right] & & U_2 \ar[rr, "\varphi_2"] & & \bbc
      \end{tikzcd}
    \end{equation*}

    It gets tedious to keep track of all the inverses and the sources and targets.
    We use the following shorthand notation to simplify the clutter.
    Let $\varphi_1: U_1 \rightarrow \bbc$ be the identity function, $\varphi_1(z) = z$.
    Then a function $f: X \rightarrow \bbp^1$ is meromorphic if the two functions
    \begin{enumerate}
      \item $\varphi_1 \circ f$,
      \item $\varphi_2 \circ f$,
    \end{enumerate}
    are holomorphic wherever they make sense.
    \begin{equation*}
      \begin{tikzcd}
        f^{-1}(U_1) \ar[rrrr, "\varphi_1 \circ f", swap, bend right] \ar[rr,"f|_{f^{-1}(U_1)}"] &  &  U_1 \ar[rr, "\varphi_1"] & & \bbc\\\\
        f^{-1}(U_2) \ar[rr, "f|_{f^{-1}(U_2)}"] \ar[rrrr, "\varphi_2 \circ f", swap, bend right] &  & U_2 \ar[rr, "\varphi_2"] & & \bbc
      \end{tikzcd}
    \end{equation*}

    \begin{qbox}
      Which of the following functions are meromorphic functions on $\bbc$?
      \begin{enumerate}
        \item $f(z) = z$
        \item \begin{align*}
            f(z) = \begin{cases}
                    z^{-1} & \mbox{ if } z \neq 0\\
                    \infty & \mbox{ if } z = 0
                  \end{cases}
          \end{align*}
        \item \begin{align*}
            f(z) = \begin{cases}
                    e^{1/z} & \mbox{ if } z \neq 0\\
                    \infty & \mbox{ if } z = 0
                  \end{cases}
          \end{align*}
      \end{enumerate}
    \end{qbox}

    \section*{Meromorphic functions on $\bbp^1$}
    \begin{qbox}
      Show that a function $f: \bbp^1 \rightarrow \bbp^1$ is meromorphic if the four functions
      \begin{enumerate}
        \item $\varphi_1 \circ f \circ \varphi_1^{-1}$,
        \item $\varphi_1 \circ f \circ \varphi_2^{-1}$,
        \item $\varphi_2 \circ f \circ \varphi_1^{-1}$,
        \item $\varphi_2 \circ f \circ \varphi_2^{-1}$,
      \end{enumerate}
      are holomorphic wherever they make sense.
    \end{qbox}

    \begin{qbox}
      Let $p(z)$ and $q(z)$ be polynomials with no common roots. Assume that $q(z)$ is not the 0 polynomial.

      Show that the following function is a meromorphic function on $\bbp^1$.
      \begin{align*}
        f(z) = \begin{cases}
          \dfrac{p(z)}{q(z)} & \mbox{ if } z \neq \infty, q(z) \neq 0, \\
          \infty & \mbox{ if } z \neq \infty, q(z) = 0, \\
          \lim \limits_{z \rightarrow \infty} \dfrac{p(z)}{q(z)} & \mbox{ if } z = \infty.
      \end{cases}
      \end{align*}
      Such a function is called a \emph{rational function}. It is common to simply write $f(z) = \dfrac{p(z)}{q(z)}$.
    \end{qbox}

    Turns out these are all the meromorphic functions on $\bbp^1$.
    \begin{theorem}
      Every meromorphic function on $\bbp^1$ is a rational function.
    \end{theorem}
    The following exercises provide the proof of this theorem.

    Let $f: \bbp^1 \rightarrow \bbp^1$ be a meromorphic function.
    Let $\calz = f^{-1}(0) \cap \bbc$ and $\calp = f^{-1}(\infty) \cap \bbc$. $\calz$ is called the set of zeroes and $\calp$ is called the set of poles.
    \begin{qbox}
      Using the isolated zeroes property of complex differentiable functions argue that both the sets $\calz$ and $\calp$ are isolated i.e. for every point $x \in \calz$ there exists a neighborhood  $U$ of $x$ such that $U \cap \calz = \set{x}$. Similarly, for $\calp$.
    \end{qbox}
    \begin{qbox}
      Using the fact that every infinite sequence in a compact set has a convergent subsequence, and that $\bbp^1$ is compact, argue that both $\calz$ and $\calp$ are finite sets.
    \end{qbox}
    Let $\calz = \set{z_1, \dots, z_m}$ and $\calp = \set{p_1, \dots, p_n}$.
    Assume the following fact for now. We'll prove it in class tomorrow.
    \begin{quote}
      The function
      \begin{align*}
        g(z) = f(z) \cdot \dfrac{(z-p_1)^{k_1} \dots (z-p_n)^{k_n}}{(z-z_1)^{\ell_1} \dots (z-z_m)^{\ell_m}}
      \end{align*}
      is meromorphic and has no zeroes or poles, for some positive integers $k_1, \dots, k_n$ and $\ell_1, \dots, \ell_m$.
    \end{quote}
    \begin{qbox}
      Check that the open mapping theorem \ref{th:OpenMappingTheorem} extends verbatim to meromorphic functions on $\bbp^1$.
    \end{qbox}
    \begin{qbox}
      Using the open mapping theorem and Q.\ref{q:imageOfCompact} argue that $g$ is either a constant function or the image of $g$ is all of $\bbp^1$.\hint{You will need to use the fact that the only open and closed subsets of $\bbp^1$ are the empty set and $\bbp^1$ itself.}
    \end{qbox}
    Because the only zero or pole of $g$ is at $\infty$ (which can be one or the other) the image of $g$ cannot be all of $\bbp^1$. Hence, $g$ is a constant function i.e. $g(z) = c$ for some $c \in \bbc$. Hence,
    \begin{align*}
      &&
      f(z) \cdot \dfrac{(z-p_1)^{k_1} \dots (z-p_n)^{k_n}}{(z-z_1)^{\ell_1} \dots (z-z_m)^{\ell_m}}
      = c \\
      \implies
      &&
      f(z) = c\dfrac{(z-z_1)^{\ell_1} \dots (z-z_m)^{\ell_m}}{(z-p_1)^{k_1} \dots (z-p_n)^{k_n}}.
    \end{align*}
\end{document}


\input{PreambleCommon}
\input{../WeekTitles}

\begin{document}
\setfont
\pagestyle{fancy}
\renewcommand{\Week}{5 }
\renewcommand{\WeekTitle}{\WeekTitleFive }

\fancyhead[LE,RO]{Week \Week}  % default, usually only for first page
\fancyfoot{}
\sectionbox{Week \#\Week: \WeekTitle}

\vspace{5mm}
\goals
\begin{itemize}
\item Recognize the family of functions that can be solved with the technique of
integration by substitution.
\item Solve integration problems using the technique of substitution.
\item Recognize the family of functions that can be solved with the technique of
integration by parts.
\item Solve integration problems using the technique of integration by parts.
\end{itemize}
\vspace{5mm}

\topic{Integration Method - Guess and Check}

We now return to the challenge of finding a {\em formula} for an
anti-derivative function.  We saw simple cases last week, and now we
will extend our methods to handle more complex integrals.

\section*{Anti-differentiation by Inspection:\\ The Guess-and-Check Method}

\vsc

Often, even if we do not see an anti-derivative immediately, we can
make an educated guess and eventually arrive at the correct answer.

\vsc

\newpage
\problem Based on your knowledge of derivatives, what should the
  anti-derivative of $\cos(3x)$, $\ds \int \cos(3x) ~dx$, look like?

\vfill

\vfill

\newpage

\problem Find $\ds \int e^{3x-2} ~dx$.

\vfill

\newpage

Both of our previous examples had {\em linear} `inside'
  functions.  Here is an integral with a {\em quadratic} `inside' function:
$$\ds{\int x e^{-x^2} ~dx}$$
\problem Evaluate the integral.  \vfill \vfill

Why was it important that there be a factor $x$ in front
  of $e^{-x^2}$ in this integral?

\vfill

\newpage
\topic{Integration Method - Substitution}
\section*{Integration by Substitution}

We can formalize the guess-and-check method by defining an {\em
  intermediate variable} the represents the ``inside'' function.

\problem Show that $\ds \int x^3 \sqrt{x^4 + 5} ~dx = {1 \over 6} (x^4
  + 5)^{3/2} + C$.

\vfill
\vfill

\newpage
$$\ds \int x^3 \sqrt{x^4 + 5} ~dx = {1 \over 6} (x^4
  + 5)^{3/2} + C$$

\problem Relate this result to the {\bf chain rule}.

\vfill

\vsc

\newpage

\problem Now use the {\bf method of substitution} to evaluate $\ds
  \int x^3 \sqrt{x^4 + 5}~dx$

\vfill
\vfill
\vfill
\vfill

\newpage

\topic{Substitution Integrals - Example 1}

{\bf Steps in the Method Of Substitution}

\begin{enumerate}[1.]
\item Select a simple function $w(x)$ that appears in the integral.
  \begin{itemize}
  \item Typically, you will also see $w'$ as a {\bf factor} or {\bf
      multiplier} in the integrand as well.
  \end{itemize}
\item Find $\ds \frac{dw}{dx}$ by differentiating.  Re-write it in the
  form $\ldots dw = ~dx$
  \item Rewrite the integral using only $w$ and $dw$ (no $x$ nor
    $dx$).
  \begin{itemize}
  \item If you can now evaluate the integral, the substitution
was effective.
\item If you cannot remove all the $x$'s, or the integral became
  harder instead of easier, then either try a different substitution,
  or a different integration method.
  \end{itemize}
\end{enumerate}

\newpage

\problem Find $\ds{ \int \tan(x) ~dx}$.

\vsc

\vfill
\vfill
\vfill
\vfill

\newpage

Though it is not required unless specifically requested, it can be reassuring to check the answer.

\problem Verify that the anti-derivative you found is correct.

\vfill
\newpage

\topic{Substitution Integrals - Example 2}
\problem Find $\ds{\int x^3 e^{x^4 - 3} dx}$.

\newpage
\topic{Substitution Integrals - Example 3}

\problem For the integral, $$ \int{\frac{e^x - e^{-x}}{(e^x + e^{-x})^2}}
  dx \ \ $$ both $w = e^{x} - e^{-x}$ and $w = e^{x} + e^{-x}$ are
  seemingly reasonable substitutions.

\Question{Which substitution will change
  the integral into the simpler form?}

  \begin{enumerate}[1.]
  \item  $ w = e^x - e^{-x}$
  \item  $ w = e^x + e^{-x}$
  \end{enumerate}

\newpage

\problem Compare both substitutions in practice.
$$ \int{\frac{e^x - e^{-x}}{(e^x + e^{-x})^2}}~dx$$
\begin{center}
\begin{tabular}{l|r}
   with $ w = e^x - e^{-x}$ ~~~~~ &~~~~~
   with $ w = e^x + e^{-x}$  \\
~& \\
~& \\
~& \\
~& \\
~& \\
~& \\
~& \\
~& \\
\end{tabular}
\end{center}

\newpage
\topic{Substitution Integrals - Example 4}
\problem Find $\ds \int \frac{\sin(x)}{1 + \cos^2(x)} dx$.

\newpage

\topic{Substitutions and Definite Integrals}

\section*{Using the Method of Substitution for Definite Integrals}

If we are asked to evaluate a {\bf definite} integral such as
$$
\int_0^{\pi/2} \frac{\cos x}{1+\sin x} dx \ \ ,
$$

where a substitution will ease the integration, we have two methods
for handling the limits of integration ($x=0$ and $x=\pi/2$).
\begin{enumerate}[a)]
\item When we make our substitution, convert both the {\em variables}
  $x$ and the {\em limits} (in $x$) to the new variable; or
\item do the integration while keeping the limits explicitly in terms
  of $x$, writing the final integral back in terms of the original $x$
  variable as well, and {\em then} evaluating.
\end{enumerate}

\newpage

\problem Use method (a) (converting both the integral and the limits
to the new variable) to evaluate the integral
$$\int_0^{\pi/2} \frac{\cos x}{1+\sin x} dx$$

\vfill

\newpage
\problem Use method (b) (converting back to $x$'s to evaluate at the
end points) to evaluate
$$ \int_9^{64} \frac{\sqrt{1 + \sqrt{x}}}{\sqrt{x}} dx \ \ .
$$

\newpage

\topic{Integration Method - By Parts}

\section*{Integration by Parts}

So far in studying integrals we have used
\begin{itemize}
\item direct anti-differentiation, for relatively simple functions,
  and
\item integration by substitution, for some more complex integrals.
\end{itemize}

However, there are many integrals that can't be evaluated with these
techniques.

\problem Try to find $\ds \int x e^{4x} ~dx$.

\vfill

\newpage

This particular integral can be evaluated with a different integration
technique, {\bf integration by parts.}  This rule is related to the
{\bf product rule} for derivatives.

\problem Expand
\begin{align*}
\frac{d}{dx} \left(u v\right) =
\end{align*}

Integrate both sides with respect to $x$ and simplify.

\vfill

Express $\ds \int u \frac{dv}{dx} ~dx$ relative to the other terms.

\vfill

\newpage
\begin{boxnote}
  {\bf Integration by Parts} \\
  For short, we can remember this formula as
\begin{align*}
\ds \int udv = uv - \int vdu
\end{align*}
\end{boxnote}

\vspace{3mm} Integration by parts:
\begin{itemize}
\item Choose a part of the integral to be $u$, and the remaining part
  to be $dv$.
\item {\bf Differentiate} $u$ to get $du$.
\item {\bf Integrate} $dv$ to get $v$.
\item Replace $\ds \int u ~dv$ with $\ds uv - \int v du$.
\item Hope/check that the new integral is easier to evaluate.
\end{itemize}

\newpage

\problem Use integration by parts to evaluate $\ds \int x e^{4x} ~dx$.

\vfill
\vfill
\vfill

\newpage
\problem Verify that your anti-derivative is correct.

\vfill

\newpage
\topic{Integration By Parts - Examples}
\subsection*{Integration By Parts - Examples}

\begin{boxnote}
{\bf Guidelines for selecting $u$ and $dv$}

\begin{itemize}
\item Ensure you can actually integrate the $dv$ part by itself, then
\item Try to select $u$ and $dv$ so that either
\begin{itemize}
\item $u'$ is simpler than $u$ or
\item $\int dv$ is simpler than $dv$
\end{itemize}
\end{itemize}
\end{boxnote}

\newpage
\problem Find $\ds \int x  \cos x ~ dx$.

\vfill

\newpage
\problem Now evaluate the slightly more challenging integral $$\ds \int x^2\cos x ~ dx$$

\vfill

\newpage

\hfill $\ds \int x^2\cos x ~ dx$

\newpage

\topic{Integration By Parts - Definite Integrals}
\subsection*{Integration By Parts - Definite Integrals}
When using integration by parts to evaluate {\em definite} integrals,
you need to apply the limits of integration to the {\bf entire}
anti-derivative that you find.

\problem Evaluate $\ds \int_{0}^{\pi} x  \sin 4x ~ dx$

\newpage

Don't forget that $dv$ does not require any other factors besides
$dx$.  That can help when there is only a single factor in the
integrand.

\problem Find the area under the graph of $\ln x$ between $x=1$ and
$x=2$.

\newpage

General integration advice:
\begin{itemize}
\item Look for a substitution in your integral first - they are the simplest method to use,
and usually the most obvious.
\item Only try integration by parts if substitution fails.
\item With all methods, you may need to {\bf experiment} with your
  choice of $u, dv$, or your substitution.
\end{itemize}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS622: Theory of Formal Languages
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 4}

Construct deterministic finite automata that accept the following languages over the alphabet $A = \{a,b,c\}$:
\begin{enumerate}[label=(\alph*)]
	\item The set of all words that begin with $ab$ and end with $ba$.
	\item The set $\{bab\}$.
	\item The set $A^* - \{bab\}$.
	\item The set of all words $x \in A^*$ that contain at least three $a$s.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]

	\item
	The \textit{dfa} that recognizes the language of the set of words that begin with $ab$ and end with $ba$ represented by $abA^*ba \cup \{aba\}$ is shown in Figure \ref{fig:DR5}.

	\begin{figure}[H]\centering
		\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,semithick]
			\tikzstyle{final}=[circle,thick,draw=black,fill=gray!40,text=black]
			\node[state] (2) {$q_2$};
			\node[state,initial] (0) [above left of =2] {$q_0$};
			\node[state] (1) [below left of=2] {$q_1$};
			\node[state] (3) [right of=2] {$q_3$};
			\node[state] (4) [above right of=3] {$q_4$};
			\node[state, final] (5) [below right of=3] {$q_5$};
			\path
				(0) edge [bend left] node {a} (2)
					edge [bend right] node {b,c} (1)
				(1) edge [loop left] node {a,b,c} (1)
				(2) edge [bend left]  node {a,c} (1)
					edge [bend left]  node {b} (3)
				(3) edge [loop left] node {b} (3)
					edge [bend right] node {c} (4)
					edge [bend left]  node {a} (5)
				(4) edge [loop right] node {a,c} (4)
					edge [bend right]  node {b} (3)
				(5) edge [bend right]  node {a,c} (4)
					edge [bend left]  node {b} (3);
		\end{tikzpicture}
		\caption{Graph of a \textit{dfa} accepting the set of words that begin with $ab$ and end with $ba$}
		\label{fig:DR5}
	\end{figure}

	\item
	The \textit{dfa} that recognizes the language of the set $\{bab\}$ is shown in Figure \ref{fig:DR6}.

	\begin{figure}[H]\centering
		\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,semithick]
			\tikzstyle{final}=[circle,thick,draw=black,fill=gray!40,text=black]
			\node[state,initial] (0) {$q_0$};
			\node[state] (1) [above right of=0] {$q_1$};
			\node[state] (2) [below right of=1] {$q_2$};
			\node[state] (3) [above right of=2] {$q_3$};
			\node[state, final] (4) [below right of=3] {$q_4$};
			\path
				(0) edge [bend left] node {b} (1)
					edge [bend right] node {a,c} (2)
				(1) edge [bend left=15] node {a} (3)
					edge [bend right] node {b,c} (2)
				(2) edge [loop below] node {a,b,c} (2)
				(3) edge [bend left] node {b} (4)
					edge [bend left] node {a,c} (2)
				(4) edge [bend left] node {a,b,c} (2);
		\end{tikzpicture}
		\caption{Graph of a \textit{dfa} accepting the set $\{bab\}$}
		\label{fig:DR6}
	\end{figure}

	\item
	the \textit{dfa} that recognizes the language of the set $A^* - \{bab\}$ is shown in Figure \ref{fig:DR7}.

	\begin{figure}[H]\centering
		\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,semithick]
			\tikzstyle{final}=[circle,thick,draw=black,fill=gray!40,text=black]
			\node[state,initial,final] (0) {$q_0$};
			\node[state,final] (1) [above right of=0] {$q_1$};
			\node[state,final] (2) [below right of=1] {$q_2$};
			\node[state,final] (3) [above right of=2] {$q_3$};
			\node[state] (4) [below right of=3] {$q_4$};
			\path
				(0) edge [bend left] node {b} (1)
					edge [bend right] node {a,c} (2)
				(1) edge [bend left] node {a} (3)
					edge [bend right] node {b,c} (2)
				(2) edge [loop below] node {a,b,c} (2)
				(3) edge [bend left] node {b} (4)
					edge [bend left] node {a,c} (2)
				(4) edge [bend left] node {a,b,c} (2);
		\end{tikzpicture}
		\caption{Graph of a \textit{dfa} accepting the set $A^* -\{bab\}$}
		\label{fig:DR7}
	\end{figure}

	\item
	The set of all words that contain at least three $a$s can be described as set of all words of the form $A^*aA^*aA^*aA^*$.
	\textit{dfa} that recognizes such language would have a final state $q_3$ to be reached from $q_0$ by three symbols $a$.
	Proposed \textit{dfa} is shown in Figure \ref{fig:DR8} where state $q_i$ is reached by the words with at least $i$ symbol $a$.

	\begin{figure}[H]\centering
		\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,semithick]
			\tikzstyle{final}=[circle,thick,draw=black,fill=gray!40,text=black]
			\node[state,initial] (0) {$q_0$};
			\node[state] (1) [right of=0] {$q_1$};
			\node[state] (2) [right of=1] {$q_2$};
			\node[state,final] (3) [right of=2] {$q_3$};
			\path
				(0) edge [loop below] node {b,c} (0)
					edge [bend left] node {a} (1)
				(1) edge [loop below] node {b,c} (1)
					edge [bend left] node {a} (2)
				(2) edge [loop below] node {b,c} (2)
					edge [bend left] node {a} (3)
				(3) edge [loop below] node {a,b,c} (3);
		\end{tikzpicture}
		\caption{Graph of a \textit{dfa} accepting the set $A^*aA^*aA^*aA^*$}
		\label{fig:DR8}
	\end{figure}

\end{enumerate}


\subsection{Taylor Polynomials}\label{sec:Taylor}
We can go beyond first order derivatives to create polynomials approximating a function as closely as we wish, these are called $Taylor$ $Polynomials$.

While our linear approximation $L(x)=f'(a)(x-a)+f(a)$ at a point $a$ was a polynomial of degree 1 such that both $L(a)=f(a)$ and $L'(a)=f'(a)$, we can now form a polynomial
\[ T_n(x)=a_0+a_1(x-a)+a_2(x-a)^2+a_3(x-a)^3+\dots +a_n(x-a)^n \]
which has the same first $n$ derivatives at $x=a$ as the function $f$.

By successively computing the derivatives of $T_n$, we obtain:
\[ \begin{array}{l}
a_0 = f(a)=\frac{f(a)}{0!}\\
a_1 = \frac{f'(a)}{1!}\\
a_2 = \frac{f''(a)}{2!}\\
\cdots \\
a_k = \frac{f^{(k)}(a)}{k!}\\
\cdots a_n =\frac{f^{(n)}(a)}{n!}
\end{array} \]
where $f^{(k)}(x)$ is the $k^{th}$ derivative of $f(x)$, and
$n!=n(n-1)(n-2)\ldots (2)(1) $, referred to as \ifont{factorial} notation.

Here is an example.

\begin{example}{Approximate e using Taylor Polynomials}{approximate e using taylor polynomials}
Approximate $e^x$ using Taylor polynomials at $a=0$, and use this to approximate $e$.
\end{example}

\begin{solution}
In this case we use the function $f(x)=e^x$ at $a=0$, and therefore
\[ T_n(x)=a_0+a_1x+a_xx^2+a_3x^3+\ldots +a_nx^n \]

Since all derivatives $f^{(k)}(x)=e^x$, we get:
\[ \begin{array}{l}
a_0=f(0)=1 \\
a_1=\frac{f'(0)}{1!}=1 \\
a_2=\frac{f''(0)}{2!}=\frac{1}{2!} \\
a_3=\frac{f'''(0)}{3!}=\frac{1}{3!} \\
\cdots \\
a_k=\frac{f^{(k)}(0)}{k!}=\frac{1}{k!} \\
\cdots \\
a_n=\frac{f^{(n)}(0)}{n!}=\frac{1}{n!}
\end{array} \]
Thus
\[ \begin{array}{l}
T_1(x)=1+x=L(x) \\
T_2(x)=1+x+\frac{x^2}{2!} \\
T_3(x)=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}
\end{array} \]
and in general
\[ T_n(x)=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\cdots +\frac{x^n}{n!}. \]

Finally we can approximate $e=f(1)$ by simply calculating $T_n(1)$. A few values are:
\[ \begin{array}{l}
T_1(1)=1+1=2 \\
T_2(1)=1+1+\frac{1^2}{2!}=2.5 \\
T_4(1)=1+1+\frac{1^2}{2!}+\frac{1^3}{3!}=2.\overline{6} \\
T_8(1)=2.71825396825 \\
T_{20}(1)=2.71828182845
\end{array} \]

We can continue this way for larger values of $n$, but $T_{20}(1)$ is already a pretty good approximation of $e$, and we took only 20 terms!
\end{solution}
%
%\subsubsection{Taylor's Theorem}\label{subsubsec:TaylorsTheoremsubsubsec}
%We have now seen how a polynomial is used to approximate a function $f$ near $a$. But how close is our approximation to the actual function? That is to say, can we measure the error in our approximation?
%
%\begin{theorem}{Taylor's Theorem}{TaylorTheorem}
%Suppose $f$ is defined and has $n+1$ continuous derivatives on an open interval $I$ containing $a$. Then for each $x$ in the interval,
%\[\ds f(x)=\left[\sum_{k=0}^n\frac{f^{(k)}(a)}{k!}(x-a)^k\right]+R_{n+1}(x)\]
%where the error term is
%\[R_{n+1}(x)=\frac{f^{(n+1)}(z)}{(n+1)!}(x-a)^{n+1}\]
%for some $z$ between $a$ and $x$.
%\end{theorem}
%
%The form for the error $R_{n+1}(x)$ is called the \dfont{Lagrange Formula} for the remainder. Notice that this error term looks quite similar to how we would expect the next term in the Taylor Polynomial of $f(x)$ to look. The main difference is that $f^{(n+1)}$ is evaluated at a point $z$ and not necessarily at the center $a$, and we do not know what $z$ might be.
%
%Notice as well that when $n=0$, Taylor's Theorem is precisely the Mean Value Theorem, which is mainly how we would prove Taylor's Theorem (but will not prove here). The following corollary is useful when applying the theorem.
%
%\begin{corollary}{Corollary to Taylor's Theorem}{TaylorTheoremCorollary}
%Let $\mu_{n+1}$ be the maximum of $|f^{(n+1)}(x)|$ on some interval containing the center $a$. Then for any $x$ in this interval, the error of the Taylor Polynomial is bounded as follows.
%\[\ds |f(x)-T_n(x)|\leq \frac{\mu_{n+1}}{(n+1)!}|x-a|^{n+1}.\]
%\end{corollary}
%
%Observe that if $f^{(n+1)}(x)=0$, then the Taylor Polynomial of degree $n$ is an exact approximation of the function $f$. Why? If the $(n+1)$\textsuperscript{th} derivative of a function is 0, then $f$ is simply a polynomial of degree at most $n$.
%
%Let's look at an example:
%
%\begin{example}{Approximate Square Root}{ApproxSquareRootTaylorTheorem}
%Approximate $\sqrt{11}$ to accuracy of at least 0.01.
%\end{example}
%\begin{solution}
%The 2nd degree Taylor Polynomial of $f(x)=\sqrt{x}$ centered at $x=9$ is given by:
%\[\ds T_2(x)=3+\frac{1}{6}(x-9)-\frac{1}{216}(x-9)^2.\]
%For our purposes a 2nd degree polynomial should be sufficient to produce our desired level of accuracy, which we will see shortly. If we require greater accuracy, we can simply increase the degree of the Taylor Polynomial. At $x=11$ this polynomial approximation gives $T_2(10)=3.3\overline{148}$.
%
%Now we must determine the accuracy of our approximation. We first find that the third derivative of $\sqrt{x}$ if $f'''(x)=\frac{3}{8}x^{-5/2}$, and that this function's maximum in the interval $[9,11]$ occurs at $x=9$ (since it is a positive decreasing function). Thus, the maximum value is $\mu_3=f'''(9)=\frac{3}{8}\cdot\frac{1}{243}=\frac{3}{1944}=\frac{1}{648}$. Applying the corollary we get
%\begin{align*}
%\left|\sqrt{11}-3.3\overline{148}\right|&\leq\frac{1}{648}\frac{1}{3!}(11-9)^3	\\
%&\leq\frac{1}{486}\approx 0.0020576
%\end{align*}
%Therefore, the approximation $3.3\overline{148}$ is guaranteed to be accurate to at least $\frac{1}{486}$, which is less than 0.01.
%\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:Taylor}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
Find the 5\textsuperscript{th} degree Taylor polynomial for $f(x)=\sin x$ around $a=0$.
\begin{enumerate}
	\item	Use this Taylor polynomial to approximate $\sin (0.1)$.
	\item	Use a calculator to find $\sin (0.1)$. How does this compare to our approximation in part (a)?
\end{enumerate}
\begin{sol}
$T_5(x)=x-\frac{x^3}{3!}+\frac{x^5}{5!}$
\begin{enumerate}
	\item	$\sin (0.1)\approx T_5(0.1)\approx 0.10016675$
	\item	$\sin (0.1)=0.0998334\ldots$ using a calculator. Our approximation is accurate to $0.10016675-0.0998334\ldots =0.000\bar{3}$.
\end{enumerate}
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Suppose that $f^{\prime\prime}$ exists and is continuous on $[1,2]$.
Suppose also that $\left\vert f^{\prime\prime}(x)\right\vert \leq \frac{1}{4}$
for all $x$ in $(1,2)$. Prove that if we use the linearization $y=L(x)$
of $y=f(x)$ at $x=1$ as an approximation of $y=f(x)$ near $x=1$,
then our estimated value of $f(1.2)$ is
guaranteed to have an accuracy of at least 0.01, i.e., our estimate will lie
within 0.01 units of the true value.
\end{ex}

%%%%%%%%%%
\begin{ex}
Find the 3\textsuperscript{rd} degree Taylor polynomial for $f(x)=\frac{1}{1-x}-1$ around $a=0$. Explain why this approximation would not be useful for calculating $f(5)$.
\begin{sol}
	$T_3(x)=x+x^2+x^3$. The point $x=5$ is not close to $x=0$, and $f$ is not continuous at $x=1$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Consider $f(x)=\ln x$ around $a=1$.
\begin{enumerate}
	\item	Find a general formula for $f^{(n)}(x)$ for $n\geq 1$.
	\item	Find a general formula for the Taylor Polynomial, $T_n(x)$.
\end{enumerate}
\begin{sol}
\begin{enumerate}
	\item	$f^{(n)}(x)=\frac{(-1)^{(n-1)}(n-1)!}{x^n}$
	\item	$T_n(x)=\ln (1)+\displaystyle\sum_{i=1}^{n} \frac{\big(\frac{(-1)^{(i-1)}(i-1)!}{1^n}\big)}{i!}(x-1)^i=\displaystyle\sum_{i=1}^{n} \bigg(\frac{(-1)^{(i-1)}(i-1)!}{i!}\bigg)(x-1)^i$ since $\ln (1)=0$ and $1^n=1$.
\end{enumerate}
\end{sol}
\end{ex}

%% % % % % % % % % %
%% Exercises for Taylor's Theorem
%% % % % % % % % % %
%\begin{ex}
%Approximate $\ln(1.3)$ to accuracy of at least 0.0001.
%\end{ex}
%
%\begin{ex}
%Determine a general inequality describing the error of Taylor Polynomial approximations of $f(x)=\sin(x)$ and $g(x)=\cos(x)$. \ifont{(Hint: Build the inequality in Corollary~\ref{cor:TaylorTheoremCorollary})}
%\begin{sol}
%	\begin{align*}
%	|\sin(x)-T_n(x)|&\leq\frac{1}{(n+1)!}x^{n+1}	\\
%	|\cos(x)-T_n(x)|&\leq\frac{1}{(n+1)!}x^{n+1}
%	\end{align*}
%\end{sol}
%\end{ex}
%
%% % % % % % % % % %

\end{enumialphparenastyle}


\chapter{Trignometric Functions}

As mentioned earlier, in a right triangle where one angle is $\theta$,
the sine of $\theta$ is the length of the side opposite $\theta$
divided by the length of the hypotenuse.

The sine function is defined for any real number. We treat that real number
$\theta$ as an angle, we draw a ray from the origin out to the unit
circle.  The $y$ value of that point is the sine. So, for example,
the $\sin(\frac{4\pi}{3})$ is $-\sqrt{3}/2$

\begin{tikzpicture}[declare function={angle=240;},bullet/.style={inner
    sep=1pt,fill,draw,circle,solid}, scale=3]
    % Axis
    \draw[thick,-stealth,black] (-1.2,0)--(1.2,0) node[right] {$x$}; % x axis
    \draw[thick,-stealth,black] (0,-1.2)--(0,1.2) node[left] {$y$}; % y axis
    % Rest
    \draw (0,0) circle (1);
    \draw[thick] (0,0) -- (angle:1.0) node [midway, right] {1};
    \draw[sdkblue] (-0.1, 0.32) node[above] {$\theta = \frac{4\pi}{3}\text{ radians} = 240^\circ$};
    \draw[-stealth,sdkblue] (0.3,0) arc (0:angle:0.3);
    \draw[dashed, black] (-0.7, -0.866) -- (0.05, -0.866) node[right] {$\sin(\theta) = -\sqrt{3}/2$}; % horizontal
    \filldraw[black] (angle:1.0) circle(1pt);
\end{tikzpicture}

(Note that in this section, we will be using radians instead of
degrees unless otherwise noted. While degrees is more familiar to most
people, engineers and mathematicians nearly always use radians when
solving problems. Your calculator should have a radians mode and a
degrees mode. You want to be in radians mode.)

Similarly, we define cosine using the unit circle: to find the cosine
of $\theta$, we draw a ray from the origin at the angle $\theta$. The
$x$ component of the point where the ray intersects the unit circle is
the cosine of $\theta$.

\begin{tikzpicture}[declare function={angle=240;},bullet/.style={inner
    sep=1pt,fill,draw,circle,solid}, scale=3]
    % Axis
    \draw[thick,-stealth,black] (-1.2,0)--(1.2,0) node[right] {$x$}; % x axis
    \draw[thick,-stealth,black] (0,-1.2)--(0,1.2) node[left] {$y$}; % y axis
    % Rest
    \draw (0,0) circle (1);
    \draw[thick] (0,0) -- (angle:1.0) node [midway, right] {1};
    \draw[sdkblue] (0.1, 0.32) node[above] {$\theta = \frac{4\pi}{3}\text{ radians} = 240^\circ$};
    \draw[-stealth,sdkblue] (0.3,0) arc (0:angle:0.3);
    \draw[dashed, black]  (-0.5, -0.95) -- (-0.5, 0.05) node[left, above] {$\cos(\theta) = -0.5$}; % horizontal
    \filldraw[black] (angle:1.0) circle(1pt);
\end{tikzpicture}

From this description, it is easy to see why $\sin(\theta)^2 +
\cos(\theta)^2 = 1$.  They are the legs of a right triangle with a
hypotenuse of length 1.

It should also be easy to see why $\sin(\theta) = \sin(\theta +
2\pi)$: Each time you go around the circle, you come back to where
you started.

Can you see why $\cos(\theta) = \sin(\theta + \pi/2)$? Turn the picture sideways.

\section{Graphs of sine and cosine}

Here is a graph of $y = \sin(x)$:

\begin{tikzpicture}[
tl/.style = {% tick labels
    fill=white, inner sep=1pt, font=\scriptsize,
            },                        ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.5] (-6.6,-1.2) grid (6.6,1.2);

% y tick label
\foreach \y in {-1, -1/2, 1/2, 1}{\node[tl,left=1mm] at (0,\y) {$\y$};}
% x tick label
\foreach \x [count=\xx from -4] in
       {-2\pi,
        -\frac{3\pi}{2},
        -\pi,
        -\frac{\pi}{2},
        { },
         \frac{\pi}{2},
         \pi,
         \frac{3\pi}{2},
         2\pi
        }{\node[tl,below=1mm] at (3*0.5235*\xx,0) {$\x$};}
% axes
    \draw[->,thick] (-6.5,0) -- (6.5,0) node[right] {$x$};
    \draw[->,thick] (0,-1.25) -- (0, 1.25) node[above] {$y$};
% curve
\draw[<->,thick,draw=black,
      domain=-6.5:6.5,samples=300,variable=\x]
      plot (\x,{sin(deg{\x})});
\end{tikzpicture}

It looks like waves, right? It goes forever to the left and
right. Remembering that $\cos(\theta) = \sin(\theta + \pi/2)$, we can
guess what the graph of $y = \cos(x)$ looks like:

\begin{tikzpicture}[
tl/.style = {% tick labels
    fill=white, inner sep=1pt, font=\scriptsize,
            },                        ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.5] (-6.6,-1.2) grid (6.6,1.2);

% y tick label
\foreach \y in {-1, -1/2, 1/2, 1}{\node[tl,left=1mm] at (0,\y) {$\y$};}
% x tick label
\foreach \x [count=\xx from -4] in
       {-2\pi,
        -\frac{3\pi}{2},
        -\pi,
        -\frac{\pi}{2},
        { },
         \frac{\pi}{2},
         \pi,
         \frac{3\pi}{2},
         2\pi
        }{\node[tl,below=1mm] at (3*0.5235*\xx,0) {$\x$};}
% axes
    \draw[->,thick] (-6.5,0) -- (6.5,0) node[right] {$x$};
    \draw[->,thick] (0,-1.25) -- (0, 1.25) node[above] {$y$};
% curve
\draw[<->,thick,draw=black,
      domain=-6.5:6.5,samples=300,variable=\x] plot (\x,{cos(deg{\x})});
\end{tikzpicture}

\section{Plot cosine in Python}

Create a file called \filename{cos.py}:

\begin{Verbatim}
import numpy as np
import matplotlib.pyplot as plt

until = 8.0

# Make a plot of cosine
thetas = np.linspace(0, until, 32)
cosines = []
for theta in thetas:
    cosines.append(np.cos(theta))

# Plot the data
fig, ax = plt.subplots()
ax.plot(thetas, cosines, 'r.', label="Cosine")
ax.set_title("Cosine")
plt.show()
\end{Verbatim}

This will plot 32 points on the cosine wave between 0 and 8.  When you
run it, you should see something like this:

\includegraphics[width=0.8\textwidth]{cospy.png}

\section{Derivatives of sine and cos}

Here is a wonderful property of sine and cosine functions: At any point $\theta$, the slope of the sine graph at $\theta$ equals $cos(\theta)$.

For example, we know that $\sin(4\pi/3) = -(1/2)\sqrt{3}$ and
$\cos(4\pi/3) = -1/2$. If we drew a line tangent to the sine curve at
this point, it would have a slope of -1/2:

\begin{tikzpicture}[
tl/.style = {% tick labels
    fill=white, inner sep=1pt, font=\scriptsize,
            },                        ]
% grid
\draw[sdkblue, very thin, xstep=0.5235, ystep=0.5] (-1.25,-1.7) grid (6.6,1.2);

% y tick label
\foreach \y in {-3/2, -1, -1/2, 1/2, 1}{\node[tl,left=1mm] at (0,\y) {$\y$};}
% x tick label
\foreach \x [count=\xx from -1] in
       {-\frac{\pi}{2},
        { },
         \frac{\pi}{2},
         \pi,
         \frac{3\pi}{2},
         2\pi
        }{\node[tl,below=1mm] at (3*0.5235*\xx,0) {$\x$};}
% axes
    \draw[->,thick] (-1.25,0) -- (6.5,0) node[right] {$x$};
    \draw[->,thick] (0,-1.5) -- (0, 1.25) node[above] {$y$};
% curve
\draw[<->,thick,draw=black,
      domain=-1.75:6.5,samples=300,variable=\x]
      plot (\x,{sin(deg{\x})});
\filldraw[black] (4.188790204786391,-0.866025403784439) circle(2pt);
\draw[->, thick, draw=red] (4.188790204786391,-0.866025403784439) -- (5.188790204786391,-1.366025403784439) node [right] {slope = -1/2} ;
\end{tikzpicture}

We say ``The derivative of the sine function is the cosine function.''

Can you guess the derivative of the cosine function? For any $\theta$, the slope of the graph of the $\cos(\theta)$ is $-\sin(\theta)$.

\section{A weight on a spring}

Let's say you fill a rollerskate with heavy rocks and attach it to the
wall with a stiff spring.  If you push the skate toward the wall a
release it, it will roll back and forth. Engineers would say ``The skate will oscillate.''

Intuitively, you can probably guess:
\begin{itemize}
\item If the spring is stronger, the skate will oscillate more times per minute.
\item If the rocks are lighter, the skate will oscillate more times per minute.
\end{itemize}

The force that the spring exerts on the skate is proportional to how
far its length is from its relaxed length.. When you buy a spring, the
manufacturer advertises its ``spring rate'', which is in pounds per
inch or newtons per meter.  If a spring has a rate of 5 newtons per
meter, that means that if stretch or compress it 10 cm, it will push
back with a force of 0.5 newtons. If you stretch or compress it 20 cm,
it will push back with a force of 1 newton.

Let's write a simulation of the skate-on-a-spring.  Duplicate \filename{cos.py}, and name the new copy \filename{spring.py}.  Add code to implement the simulation:

\begin{Verbatim}[commandchars=\\\{\}]
import numpy as np
import matplotlib.pyplot as plt

until = 8.0

\textbf{# Constants}
\textbf{mass = 100 # kg}
\textbf{spring_constant = -1 # newtons per meter displacement}
\textbf{time_step = 0.01 # s}

\textbf{# Initial state}
\textbf{displacement = 1.0 # height above equilibrium in meters}
\textbf{velocity = 0.0}
\textbf{time = 0.0 # seconds}

\textbf{# Lists to gather data}
\textbf{displacements = []}
\textbf{times = []}

\textbf{# Run it for a little while}
\textbf{while time <= until:}
\textbf{    # Record data}
\textbf{    displacements.append(displacement)}
\textbf{    times.append(time)}

\textbf{    # Calculate the next state}
\textbf{    time += time_step}
\textbf{    displacement += time_step * velocity}
\textbf{    force = spring_constant * displacement }
\textbf{    acceleration = force / mass}
\textbf{    velocity += acceleration}

# Make a plot of cosine
thetas = np.linspace(0, until, 32)
cosines = []
for theta in thetas:
    cosines.append(np.cos(theta))

# Plot the data
fig, ax = plt.subplots()
\textbf{ax.plot(times, displacements, 'b', label="Displacement")}
ax.plot(thetas, cosines, 'r.', label="Cosine")

\textbf{ax.set_title("Weight on Spring vs. Cosine")}
\textbf{ax.set_xlabel("Time (s)")}
\textbf{ax.set_ylabel("Displacement (m)")}
\textbf{ax.legend()}
plt.show()
\end{Verbatim}
When you run it, you should get a plot of your spring and the cosine graph on the same plot.

\includegraphics[width=0.8\textwidth]{springpy.png}

The position of the skate is following a cosine curve. Why?

Because for a sine or cosine waves happen whenever the acceleration of
an object is proportional to -1 times its displacement. Or in symbols:

$$a \propto - p$$

where $a$ is acceleration and $p$ is the displacement from equilibrum.

Remember that if you take the derivative of the displacement, you get
the velocity.  And if you take the derivative of that, you get
acceleration.  So, the weight on the spring must follow a function $f$ such that

$$f(t) \propto - f''(t)$$

Remember that the derivative of the $\sin(\theta)$ is $\cos(\theta)$.

And the derivative of the $\cos(\theta)$ is $- \sin(\theta)$

Thus these sorts of waves have an almost-magical power: their
acceleration is proportional to -1 times their displacement.

Thus sine waves of various magnitudes and frequencies are ubiquitous
in nature and technology.

\section{Integral of sine and cosine}

If we take the area between the graph and the $x$ axis of the cosine
function (and if the function is below the $x$ axis, it counts as
negative area), from 0 to $4\pi/3$, we find that it is equal to
$-(1/2)\sqrt{3}$

\begin{tikzpicture}[
tl/.style = {% tick labels
    fill=white, inner sep=1pt, font=\scriptsize,
            },                        ]

% y tick label
\foreach \y in {-1, -1/2, 1/2, 1}{\node[tl,left=1mm] at (0,\y) {$\y$};}
% x tick label
\foreach \x [count=\xx from -1] in
       {-\frac{\pi}{2},
        { },
         \frac{\pi}{2},
         \pi,
         \frac{3\pi}{2},
         2\pi
        }{\node[tl,below=1mm] at (3*0.5235*\xx,0) {$\x$};}
       % axes
       \draw[->,thick] (-1.25,0) -- (6.5,0) node[right] {$x$};
       \draw[->,thick] (0,-1.25) -- (0, 1.25) node[above] {$y$};
       % curve
       \draw[<->,thick,draw=black, domain=-1.75:6.5,samples=300,variable=\x] plot (\x,{cos(deg{\x})});
       \fill[sdkblue, domain=0:1.57,samples=100, variable=\b]
       (0, 1)
       -- plot (\b,{cos(deg(\b))})
       -- (0, 0)
       -- cycle;
       \fill[red, domain=1.57:4.188790204786391,samples=100, variable=\b]
       (1.57, 0)
       -- plot (\b,{cos(deg(\b))})
       -- (4.188790204786391, 0)
       -- cycle;
       \draw[thick, draw=black] (4.188790204786391, 1) -- (4.188790204786391,-1) node [right]{area=$-(1/2)\sqrt{3}$};
\end{tikzpicture}

We say ``The integral of the cosine function is the sine function.''


\subsection{Power series}

of the form:

\(\sum_{n=0}a_n(x-c)^n\)

\subsubsection{Smoothness of power series}

Power series are all smooth. That is, they are infinitely differentiable.