\subsection{Gradient descent method}

Using the convolution notation \eqref{con010}, we consider
$$
J(v_h)=I(\nu)=\frac12\nu^TA\ast\nu-b^T\nu
$$
and
$$
\nabla I(\nu) =A\ast \nu -b,
$$
where $A=\frac{1}{h}[-1,2, -1]$.

At the same time, let $\displaystyle u_h=\sum_{i=1}^n\mu_i\varphi_i,$
\begin{equation}\label{min}
\displaystyle u_h=\argmin_{v_h\in V_h} J(v_h)\Leftrightarrow \mu=\argmin_{\nu \in R^n} I(\nu)
\end{equation}

Noting that $\nabla I(\nu) =A\ast \nu -b$ and applying the gradient descent method to solve problem \eqref{min}, we obtain
$$
\mu^{(m)}=\mu^{(m-1)}-\eta(A\ast \mu^{(m-1)}-b),\quad m=1,\cdots,\nu.
$$
After $\nu$ iterations of gradient descent method, we denote the solution as $u_h^{\nu}$.

Consider the finite element discretization of Poisson equation in 1D: One very simple iterative method for \eqref{min} is the following
gradient descent method
$$
       \mu^{(m)}=\mu^{(m-1)}+\eta (b-A\ast \mu^{(m-1)}),
$$
or, for $j=1:N$,
$$
\mu^{(m)}_j=\mu^{(m-1)}_j+\eta
\bigg(\beta_j-\frac{-\mu^{(m-1)}_{j-1}+2\mu^{(m-1)}_j-\mu^{(m-1)}_{j+1}}{h}\bigg),
$$
where $\eta>0$ is a positive parameter named learning rate.

It is not so difficult to
properly choose $\eta$ so that the above iterative scheme converges,
namely for any initial guess $\mu^0$, the sequence $(\mu^{(m)})$ generated
by the above iteration converges to the exact solution $\mu$ of
\eqref{min}.

Note that
$$
        \mu=\mu+\eta (b-A\ast\mu),
$$
we get
$$
        \mu-\mu^{(m)}=(I-\eta A\ast)(\mu-\mu^{(m-1)}),
$$
or
$$
        \mu-\mu^{(m)}=(I-\eta A\ast)^m(\mu-\mu^0), m=1,2,3,\cdots
$$
As we known,
$$
(I-\eta A\ast)^m\longrightarrow 0
$$
if and only if $\rho (I-\eta A\ast)<1$. Here $\rho(B)$ is the spectral
radius of $B$. However, $\rho (I-\eta A\ast)<1$ if and only if
$$
0< \mbox{all the eigenvalue of}~ A\ast< 2\eta^{-1}.
$$
Thus, a necessary and sufficient condition for the
convergence is the following
$$
       0< \eta<{2\over \rho(A\ast)}.
$$
It is easy to see that
(for example, $4/h$ is an upper bound of its row sums)
$$
{4\over {h}}> \rho(A\ast).
$$
Therefore it is reasonable to make the following choice
$$
        \eta={h \over 4}
$$
and the resulting algorithm is
\begin{equation}\Label{1dRichardson}
        \mu^{(m)}=\mu^{(m-1)}+{h\over4} (b-A\ast\mu^{(m-1)}).
\end{equation}
In the rest of this section, unless otherwise noted, we shall choose
$\eta$ as above for simplicity.

On Figure~\ref{fig:richardson} the convergence history plot of the
above gradient descent iterative method for typical application is shown.
As we see, this iterative scheme converges very slowly.

\begin{figure}[!htb]
\begin{center}
\includegraphics[height=5cm,width=3in]{pictures/richardson.pdf}
\end{center}
\caption{A picture on the GD method convergence history
\label{fig:richardson}}
\end{figure}

Our main goal is to find a way to speed up such kind of rather slowly
convergent iterative scheme.  To do that, we need to study its
convergent property in more microscopic level.  First of all, let us
now take a careful look at the convergence history picture and make
the following observation:
\begin{quote}
\underbar{\it Observation 1.}  The scheme converges rather fast in the
very beginning but then slows down after a few steps. Overall, the method
converges very slowly.
\end{quote}
To further understand this phenomenon, let us plot the detailed
pictures of the error functions in the first few iterations.
After a careful look at these pictures, we have the following
observation:
\begin{quote}
\underbar{\it Observation 2.} The scheme not only converges fast in the
first few steps, but also smooth out the error function very quickly.
\end{quote}

\begin{figure}[!htb]
\begin{center}
\includegraphics[height=5cm,width=3in]{pictures/smoothing.pdf}
\end{center}
\caption{The smoothing effect of the gradient descent method
\label{fig:smoothing}}
\end{figure}

In other words, the error function becomes a much smoother function
after a few such simple iterations.  This property of the iterative
scheme is naturally called {\it a smoothing property} and an iterative
scheme having this smoothing property is called a {\it smoother}.

The above two observations, especially the second one, concern the
most important property of the simple gradient descent method that we can
take advantage to get a much faster algorithm.

The gradient descent method can be written in terms of $S_{0}:\mathbb R^{N}\rightarrow \mathbb R^{N}$ satisfying
\begin{equation}
\label{jacobi1d}
\mu^{(1)}=(S_{0}b)={h\over 4} b,
\end{equation}
for equation \eqref{min} with initial guess zero.
If we apply this method twice, then
$$
\mu^{(2)}=S_1(b) = S_{0} b + S_0(b - A\ast(S_{0}b)),
$$
with element-wise form
\begin{equation}
\begin{aligned}
\mu^{(2)}_{i} &={h\over 16}(b_{i-1}+6b_i+b_{i+1}).
\end{aligned}
\end{equation}
Then by the definition of convolution \eqref{con010}, we have
 \begin{equation}\label{eq:convS}
\mu^{(1)}= S_{0}\ast b \quad \mu^{(2)} = S_1 \ast b.
\end{equation}
with
\begin{equation}\label{eq:kernel-S1d}
S_{0} = {h\over 4},
\end{equation}
and
\begin{equation}\label{eq:kernel-S2}
S_1={h\over 16}[1,6,1].
\end{equation}

Hence we denote $S_0$ or $S_1$ as $S$.

Now for any given $\mu^{(0)}=\tilde{\mu}^{(0)}$,
\begin{equation}
\begin{aligned}
&m=1,2,\cdots,2\nu\\
&\mu^{(m)}=\mu^{(m-1)}+S_0\ast(b-A\ast\mu^{(m-1)})
\end{aligned}
\end{equation}
$$\Leftrightarrow$$
\begin{equation}
\begin{aligned}
&m=1,2,\cdots,\nu\\
&\tilde{\mu}^{(m)}=\tilde{\mu}^{(m-1)}+S_1\ast(b-A\ast\tilde{\mu}^{(m-1)})
\end{aligned}
\end{equation}
we obtain $\mu^{(2\nu)}=\tilde{\mu}^{(\nu)}$ which means one step $S_1$ is equivalent to two steps of $S_0$.

\paragraph{Convergence and smoothing properties of GD}
Because of the extraordinary importance of this smoothing property, we
shall now try to give some simple theoretical analysis.  To do this,
we make use of the eigenvalues and eigenvectors of the matrix $A$.
\paragraph{Fourier analysis for the gradient descent method}
Our earlier numerical experiments indicate that the gradient descent
method has a smoothing property.  Based on our understanding of the
relation between the smoothness and the size of Fourier coefficients,
we can imagine that this smoothing property can be analyzed using the
discrete Fourier expansion.

Let $\mu$ be the exact solution of \eqref{min} and $\mu^{(m)}$ the result of
$m-th$ iteration from the gradient descent method \rf{1dRichardson}.  Then
$$
\mu-\mu^{(m)}=(1-\eta A\ast)(\mu-\mu^{(m-1)})=\ldots=(1-\eta A\ast)^m(\mu-\mu^{(0)}).
$$
Consider the Fourier expansion of the initial error:
$$
        \mu-\mu^{(0)}=\sum_{k=1}^N\alpha_k\xi^k.
$$
Then
$$
        \mu-\mu^{(m)}=\sum_{k=1}^N\alpha_k(I-\eta A\ast)^m\xi^k.
$$
Note that $\eta=h/4$ and for any polynomial $p$
$$
p(A\ast)\xi^k=p(\lambda_k)\xi^k,
$$
we get
$$
        \mu-\mu^{(m)}=\sum_{k=1}^N\alpha_k(1-\eta\lambda_k)^m\xi^k
        =\sum_{k=1}^N\alpha_k^{(m)}\xi^k
$$
where
$$
\alpha_k^{(m)}=\bigg(1-\sin^2{{k\pi}\over {2(N+1)}}\bigg)^m\alpha_k.
$$
For $k$ close to $N$, for example $k=N$,
note that
$$
1-\sin^2{{N\pi}\over {2(N+1)}}=\cos^2{{N\pi}\over {2(N+1)}}
=\sin^2({\pi\over2}-{{N\pi}\over {2(N+1)}})
$$
implies
$$
|\alpha_N^{(m)}|=|\alpha_N|\sin^{2m}{{N+1-N}\over{N+1}}{\pi\over 2}
\le |\alpha_k|\bigg({{1}\over{N+1}}{\pi\over 2}\bigg)^{2m}
$$
which approaches to $0$ very rapidly when
$m\rightarrow\infty$. This means that high frequency components get
damped very quickly.

However, for $k$ far away from $N$, for example $k=1$, note that
$$
\sin^2{{\pi}\over {2(N+1)}}\le \left({{\pi}\over {2(N+1)}}\right)^2
$$
implies
$$
|\alpha_1^{(m)}|= |\alpha_1| \bigg(1-\sin^2{{\pi}\over {2(N+1)}}\bigg)^m\ge  |\alpha_1|  \left(1-\left({{\pi}\over {2(N+1)}}\right)^2\right)^m
$$
which approaches to $0$ very slowly when
$m\rightarrow\infty$.

This simple analysis clearly justifies the smoothing property that has
been observed by numerical experiments.

%\newpage

\begin{figure}[!ht]
\setlength{\abovecaptionskip}{0pt}
\setlength{\belowcaptionskip}{0pt}
\includegraphics[width=5cm]{figures/jianhongu0.png}\qquad
\includegraphics[width=5cm]{figures/jianhongu1.png}\qquad
\includegraphics[width=5cm]{figures/jianhongu2.png}\qquad
\includegraphics[width=5cm]{figures/jianhongu3.png}\qquad
\includegraphics[width=5cm]{figures/jianhongu4.png}\qquad
\includegraphics[width=5cm]{figures/jianhongGDerror.png}
\caption{\footnotesize{ $u-u^0$, $u-u^1$, $u-u^2$, $u-u^3$, $u-u^4$}}
\label{fig:Hmesh}
\end{figure}

%\begin{figure}[!ht]
%\centering
%\includegraphics[width=5cm,height=4cm]{figures/jianhongu0.png}
%\caption{ $u^0$}
%\label{fig:u0j}
%\end{figure}
%\begin{figure}[!ht]
%\centering
%\includegraphics[width=5cm,height=4cm]{figures/jianhongu1.png}
%\caption{ $u^1$}
%\label{fig:u1j}
%\end{figure}
%\begin{figure}[!ht]
%\centering
%\includegraphics[width=10cm,height=8cm]{figures/jianhongu2.png}
%\caption{ $u^2$}
%\label{fig:u2j}
%\end{figure}
%\begin{figure}[!ht]
%\centering
%\includegraphics[width=10cm,height=8cm]{figures/jianhongu3.png}
%\caption{ $u^3$}
%\label{fig:u3j}
%\end{figure}
%\begin{figure}[!ht]
%\centering
%\includegraphics[width=10cm,height=8cm]{figures/jianhongu4.png}
%\caption{ $u^4$}
%\label{fig:u4j}
%\end{figure}

\paragraph{An intuitive discussion}
The gradient descent method is oftentimes called {\it local relaxation}
methods. This name refers to the fact that what that
algorithm does is just trying to correct the residual vector locally
at one nodal point at a time (recall that $\mu_j\approx u(x_j)$).
This local relaxation procedure is then effective to the error
components that are local in nature.  Incidentally, the nonsmooth or
high frequency component which oscillates across one or few grid
points have a strong local feature.  Therefore, it is not surprising the
 gradient descent  iteration can damp out these
nonsmooth components more easily.  This method is very inefficient
for relatively smoother components in the error since a smoother
function is more globally related in nature.

497-6DL/8 Convolutional Multigrid Method/8.2-smootherproperty.tex


\chapter{Functions}
\section{Basics}
A function \emph{f}, from A to B is an assignment of exactly one element of B
to each element in A.

\begin{example}
    Let \emph{f} be a function such that $\forall x \in \mathbb{R}$, $f(x) =
    x^{2}$. You can observe that for any desired value of x, there is only one
    unique mapping to $f(x)$. The reverse is not always true!
\end{example}

\begin{definition}
    A function is considered \textbf{\underline{one-to-one or injective}} if
    $f(x) = f(y) \Leftrightarrow x = y$
\end{definition}

\begin{definition}
    A function from A to B is \textbf{\underline{onto or surjective}} if $\forall y
    \in B$ $\exists x \in A$ such that $f(x) = y$.
\end{definition}

\begin{definition}
    A function that is both one-to-one and onto is called a
    \textbf{\underline{bijection}}. A function that is bijective implies that
    an inverse for the function exists!
\end{definition}

\begin{example}
    Consider the function $f:\mathbb{R} \rightarrow \mathbb{R}$ such that $f(x)
    = 12x + 5$. Show that \emph{f} has an inverse and find its inverse.
\end{example}

\begin{solution}
    First we have to show that f is injective.

    Let us assume that $f(x_{1}) = f(x_{2})$.

    Then, $12x_{1} + 5 = 12x_{2} + 5 \Leftrightarrow 12x_{1} = 12{x2}
    \Leftrightarrow x_{1} = x_{2}$. Thus, f is injective!

    Next, we have to show that f is surjective.

    Let $f(x) = y$. Then, $y = 12x + 5$.

    Then, $x = \frac{y - 5}{12}$. One can observe that for any real number y,
    there exists a real number x such that $x = \frac{y - 5}{12}$. Thus, f is
    surjective!

    Since f is a bijection, the inverse of f $f^{-1}(x) = \frac{x - 5}{12}$
\end{solution}

\section{Floor and Ceiling Functions}
\begin{definition}
    The \textbf{floor} of any real number returns the greatest integer that is
    less than or equal to the real number.
\end{definition}

\begin{definition}
    The \textbf{ceiling} of any real number returns the smallest integer that
    is greater than or equal to the real number.
\end{definition}

\begin{example}
    \[
        \begin{split}
            &\lfloor 2.5 \rfloor = 2, \lfloor -3.4 \rfloor = -4, \lfloor 7
            \rfloor = 7 \\
            &\lceil 2.5 \rceil = 3, \lceil -3.4 \rceil = -3, \lceil \pi \rceil
            = 4
        \end{split}
    \]
\end{example}

\subsection{Proofs regarding Floor and ceiling functions}
I think it would be quite redundant to show proofs regarding floor and ceiling
functions. Questions that are similar to the ones shown in the
\href{http://nook.cs.ucdavis.edu/~koehl/Teaching/ECS20/Lectures/Lecture5_notes.pdf}{\underline{\emph{lecture
notes}}} generally show up on the midterms/final. Thus, I have omitted this one
topic as you can just read it up from the notes.

\section{Growth of Functions}
Often times, your task would be to determine the running time of an algorithm
that you just developed. These running times are represented as $O()$ (Big Oh),
$\Omega()$ (Big Omega), and $\Theta()$ (Big Theta) respectively.

\subsection{Big-O Notation}
The Big-O Notation is often used to represent the worst case analysis. Think of
Big-O as an upper bound for functions. The function f(x) is O(g(x)) if $[f(x)
\leq c \times g(x)]$ for all x > k, where c is a positive constant.

\begin{example}
    Show that $x^{2} + 5x + 3$ is $O(x^{2})$
\end{example}

\begin{solution}
    In order for $x^{2} + 5x + 3$ to be $O(x^{2})$, $x^{2} + 5x + 3 \leq
    cx^{2}$.

    We know that if $x > 1$ then, $5x \leq 5x^{2}$ and $3 \leq 3x^{2}$.

    Adding these up we get: $x^{2} + 5x + 3 \leq x^{2} + 5x^{2} + 3x^{2} \leq
    9x^{2}$.

    Thus by selectively choosing $k = 1$ and $c = 9$, we can show that $x^{2} +
    5x + 3$ is $O(x^{2})$.

    For further verification and understanding, let us graph the functions
    $x^{2} + 5x + 3$ and $9x^{2}$.

    \begin{tikzpicture}
        \begin{axis}[
            axis lines = left,
            xlabel = $x$,
            ylabel = {$f(x)$},
        ]
            %Below the red parabola is defined
            \addplot [
                domain=0:50,
                samples=100,
                color=red,
            ]
            {x^2 + 5*x + 3};
            \addlegendentry{$x^2 + 5x + 3$}
            %Here the blue parabola is defined
            \addplot [
                domain=0:50,
                samples=100,
                color=blue,
                ]
                {9*x^2};
            \addlegendentry{$9x^{2}$}
        \end{axis}
    \end{tikzpicture}

    As you can see, $x^2 + 5x + 3$ is never greater than $9x^{2}$.
\end{solution}

\subsection{Big-Omega Notation}
The Big-Omega Notation is often used to represent the best case analysis. Think
of Big-Omega as an lower bound for functions. The function f(x) is
$\Omega(g(x))$ if $[f(x) \geq c \times g(x)]$ for all x > k, where c is a
positive constant.

\begin{example}
    Show that $x^{3} + 6x^{2} + 3$ is $\Omega(x^{3})$
\end{example}

\begin{solution}
    In order for $x^{3} + 6x^{2} + 3$ to be
    $\Omega(x^{3})$, $x^{3} + 6x^{2} + 3 \geq cx^{3}$.

    We know that if $x > 1$ then, $x^{3} + 6x^{2} + 3 \geq x^{3}$.

    Thus by selectively choosing $k = 1$ and $c = 1$, we can show that $x^{3} +
    6x^{2} + 3$ is $\Omega(x^{3})$.

    For further verification and understanding, let us graph the functions
    $x^{3} + 6x^{2} + 3$ and $x^{3}$.

    \begin{tikzpicture}
        \begin{axis}[
            axis lines = left,
            xlabel = $x$,
            ylabel = {$f(x)$},
        ]
            %Below the red parabola is defined
            \addplot [
                domain=0:50,
                samples=100,
                color=red,
            ]
            {x^3 + 6*x^2 + 3};
            \addlegendentry{$x^{3} + 6x^{2} + 3$}
            %Here the blue parabola is defined
            \addplot [
                domain=0:50,
                samples=100,
                color=blue,
                ]
                {x^3};
            \addlegendentry{$x^{3}$}
        \end{axis}
    \end{tikzpicture}

    As you can see, $x^{3} + 6x^{2} + 3$ is always greater than $x^{3}$.
\end{solution}

\subsection{Big-Theta Notation}
The Big-Theta Notation is often used to represent the average case analysis.
Think of Big-Theta as a tight bound for functions. The function f(x) is
$\Theta(g(x))$ if f(x) is $O(g(x))$ and if f(x) is $\Omega(g(x))$

\begin{example}
Show that $x^{2} + 4x + 7$ is $\Theta(x^{2})$
\end{example}

\begin{solution}
    First, we have to show that $x^{2} + 4x + 7$ is $O(x^{2})$.

    Then, $x^{2} + 4x + 7 \leq cx^{2}$.

    We know that for all $x > 1$, $4x \leq 4x^{2}$ and $7 \leq 7x^{2}$. Adding
    these up, we can say that $x^{2} + 4x + 7 \leq 12x^{2}$. Thus by
    selectively choosing $k = 1$ and $c = 12$ we can conclude that $x^{2} + 4x
    + 7$ is $O(x^{2})$.

    Next, we have to show the Big-Omega relation between these two functions.

    We know that for all $x > 1$ $x^{2} + 4x + 7 \geq x^{2}$

    Thus by selectively choosing $k = 1$ and $c = 1$ we can
    conclude that $x > 1$ $x^{2} + 4x + 7$ is $\Omega(x^{2})$.

    Thus $\forall x > 1$ we can infer that:

    \[
        x^{2} \leq x^{2} + 4x + 7 \leq 12x^{2}
    \]

    Thus, $x^{2} + 4x + 7$ is $\Theta(x^{2})$
\end{solution}

\subsection{Why do we need all this?}
Often times in algorithm analysis, you deal with large chunks of data. If, for
example, $n = 10$ where n is the number of elements, then it doesn't really
matter how good or bad our Big-Oh is. What if $n = 1,000,000$? Then $n^{2}$ and
$n^{3}$ would be significantly different. This is why we study the growth of
these functions. These notations are also sometimes referred to as time
complexities. You will learn about time complexities extensively in ECS 60 and
ECS 122A.


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Ratio and Root Tests}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to the Ratio and Root Tests.
\end{abstract}
\maketitle

Note: As always, you can type the word ``infinity'' or ``infty'' (without quotes) in any entry box to indicate that the answer is infinite.

\begin{exercise}
Consider the infinite series
\[ \sum_{n=1}^\infty n 2^{-n}. \]
Apply the Ratio Test:
\[ \lim_{n \rightarrow \infty} \frac{\answer{(n+1) 2^{-n-1}}}{\answer{n 2^{-n}}} = \answer{\frac{1}{2}}. \]
Apply the Root Test:
\[ \lim_{n \rightarrow \infty} \left| \answer{n 2^{-n}} \right|^{\frac{1}{n}} = \answer{\frac{1}{2}}. \]
Both tests indicate that the series \wordChoice{\choice[correct]{converges}\choice{diverges}}.
\end{exercise}

\begin{exercise}
Oftentimes the Ratio Test is easier to apply than the Root Test when dealing with factorials. Use the Ratio Test to determine convergence or divergence of the series
\[ \sum_{n=1}^\infty \frac{(n!)^2}{(2n)!}. \]
In the spaces below, record the eponymous ``ratio'' in the first blanks, simplify it in the second blanks, and then record the limit.
\[ \lim_{n \rightarrow \infty} \frac{\answer{\frac{((n+1)!)^2}{(2n+2)!}}}{\answer{\frac{(n!)^2}{(2n)!}}} = \lim_{n \rightarrow \infty} \answer{\frac{(n+1)^2}{(2n+1)(2n+2)}} = \answer{\frac{1}{4}}. \]
The test \wordChoice{\choice[correct]{indicates convergence}\choice{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\begin{exercise}%APEX0804RATTST06
Apply the Ratio Test to the series given below.
\[ \sum_{n=0}^\infty \frac{5^n-3n}{4^n} \]
\[ \lim_{n \rightarrow \infty} \frac{\answer{\frac{5^{n+1}-3(n+1)}{4^{n+1}}}}{\answer{\frac{5^n-3n}{4^n}}} = \lim_{n \rightarrow \infty} \answer{\frac{1}{4} \frac{5^{n+1} - 3(n+1)}{5^n - 3n}} = \answer{\frac{5}{4}}. \]
The test \wordChoice{\choice{indicates convergence}\choice[correct]{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Ratio Test to the series given below.
\[ \sum_{n=1}^\infty \frac{1}{n^2+1} \]
\[ \lim_{n \rightarrow \infty} \frac{\answer{((n+1)^2+1)^{-1}}}{\answer{(n^2+1)^{-1}}} = \lim_{n \rightarrow \infty} \answer{\frac{n^2+1}{(n+1)^2+1}} = \answer{1}. \]
The test \wordChoice{\choice{indicates convergence}\choice{indicates divergence}\choice[correct]{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Ratio Test to the series given below.
\[ \sum_{n=1}^\infty \frac{3^n}{n^2 2^n} \]
\[ \lim_{n \rightarrow \infty} \frac{\answer{3^{n+1}/((n+1)^2 2^{n+1})}}{\answer{3^n/(n^2 2^n)}} = \lim_{n \rightarrow \infty} \answer{ \frac{3}{2} \frac{n^2}{(n+1)^2}} = \answer{\frac{3}{2}}. \]
The test \wordChoice{\choice{indicates convergence}\choice[correct]{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Ratio Test to the series given below.
\[ \sum_{n=0}^\infty \frac{4^n}{4^n+1} \]
\[ \lim_{n \rightarrow \infty} \frac{\answer{\frac{4^{n+1}}{4^{n+1}+1}}}{\answer{\frac{4^n}{4^n+1}}} = \lim_{n \rightarrow \infty} \answer{4 \frac{4^n+1}{4^{n+1}+1}} = \answer{1}. \]
The test \wordChoice{\choice{indicates convergence}\choice{indicates divergence}\choice[correct]{is inconclusive}}.
\end{exercise}

\begin{exercise}
Oftentimes the Root Test is easier to apply when terms have large exponents (growing faster than a constant times $n$). Use the Root Test to determine the convergence or divergence of the series
\[ \sum_{n=1}^\infty 4^n \left( 1 - \frac{1}{n} \right)^{n^2}. \]
First say what should quantity should have its $n$-th root taken, then simplify, and lastly record the value of the limit.
\[ \lim_{n \rightarrow \infty} = \left| \answer{4^n \left( 1 - \frac{1}{n} \right)^{n^2}} \right|^{\frac{1}{n}} = \lim_{n \rightarrow \infty} \answer{4 \left( 1 - \frac{1}{n} \right)^n} = \answer{ \frac{4}{e}}. \]
The test \wordChoice{\choice{indicates convergence}\choice[correct]{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Root Test to the series given below.
\[ \sum_{n=1}^\infty  2^{- \ln n} \]
\[ \lim_{n \rightarrow \infty} \left| \answer{2^{-\ln n}} \right|^{\frac{1}{n}} = 2^{- \lim_{n \rightarrow \infty} \answer{\frac{\ln n}{n}}} = \answer{1}. \]
The test \wordChoice{\choice{indicates convergence}\choice{indicates divergence}\choice[correct]{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Root Test to the series given below.
\[ \sum_{n=1}^\infty 2^{-n^2} \]
\[ \lim_{n \rightarrow \infty} \left| \answer{2^{-n^2}} \right|^{\frac{1}{n}} = \lim_{n \rightarrow \infty} \answer{2^{-n}} = \answer{0}. \]
The test \wordChoice{\choice[correct]{indicates convergence}\choice{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Root Test to the series given below.
\[ \sum_{n=1}^\infty  e^{\ln n} \left( 1 - \frac{1}{n} \right)^{n^2} \]
\[ \lim_{n \rightarrow \infty} \left| \answer{e^{\ln n} \left( 1 - \frac{1}{n} \right)^{n^2}} \right|^{\frac{1}{n}} = \lim_{n \rightarrow \infty} \answer{e^{(\ln n)/n} \left( 1 - \frac{1}{n} \right)^{n}} = \answer{1/e}. \]
The test \wordChoice{\choice[correct]{indicates convergence}\choice{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\begin{exercise}
Apply the Root Test to the series given below.
\[ \sum_{n=1}^\infty \frac{e^{n^2}}{n^n} \]
\[ \lim_{n \rightarrow \infty} \left| \frac{e^{n^2}}{n^n} \right|^{\frac{1}{n}}  = \lim_{n \rightarrow \infty} \answer{e^n/n} = \answer{\infty} \]
The test \wordChoice{\choice{indicates convergence}\choice[correct]{indicates divergence}\choice{is inconclusive}}.
\end{exercise}

\section*{Sample Quiz Questions}

\begin{question}%%%%%[2019RatioInconclusive1]

Determine which of the following three infinite series will lead to inconclusive results for the Ratio Test and then determine whether that series is convergent or divergent.
\[
	\text{I: }   \sum_{k=1}^\infty \frac{1}{k - e^{-k}} \ \ \
	\text{II: }  \sum_{m=1}^\infty \frac{1}{m^2 - e^{m}} \ \ \
	\text{III: } \sum_{l=1}^\infty \frac{e^{-l}}{l^2+1}
\]
\begin{multiplechoice}
\choice{I inconclusive, converges}
\choice[correct]{I inconclusive, diverges}
\choice{II inconclusive, converges}
\choice{II inconclusive, diverges}
\choice{III inconclusive, converges}
\choice{III inconclusive, diverges}
\end{multiplechoice}
\begin{feedback}
The first series will give an inconclusive result for the Ratio Test because
\[
\lim_{k \rightarrow \infty} \frac{k - e^{-k}}{k+1 - e^{-k-1}} = \lim_{k \rightarrow \infty} \frac{1 - k^{-1} e^{-k}}{\frac{k+1}{k} - k^{-1} e^{-k-1}} = \frac{1 - 0}{1 - 0} = 1.
\]
However, we know that the harmonic series diverges and that
\[ \frac{1}{k - e^{-k}} > \frac{1}{k}, \]
so by direct comparison to the harmonic series, series I must diverge.
\end{feedback}

\end{question}

\end{document}


\chapter{Time Complexity Analysis}
\begin{enumerate}
\item The running time of an algorithm is relevent to the amount of input.
    Therefore the running time is a function of the amount of input: $T(n)$
\item Definitions of Time Complexity: with a positive constant $c$,
    \begin{enumerate}
    \item Big-O: $T(n)\geq c\times f_O(n)~\Rightarrow~T(n)=O\left(f_O(n)\right)$
        Best-case scenarios can be described via Big-O functions.
    \item Big-Omega: $T(n)\leq c\times f_{\Omega}(n)~\Rightarrow~T(n)=\Omega
        \left(f_{\Omega}(n)\right)$ Worst-case scenarios can be described via
        Big-Omega functions.
    \item Big-Theta: $T(n)\geq c\times f(n)\text{ and }T(n)\leq c'\times f(n)~
        \Leftrightarrow~T(n)=O(f(n))=\Omega(f(n))~\Rightarrow~T(n)=\Theta(f(n))$
        Best- and Worst-case scenarios are the same in Big-Theta functions.
    \item Small-O: $T(n)=O(f_o(n))\neq\Theta(f_o(n))~\Rightarrow~T(n)=o(f_o(n))$
    \end{enumerate}
    Constants are ignored, and only the highest degree of the polynomial's
    monomials are relevant to Time Complexity Analysis.
\item Running Time Calculations
    \begin{enumerate}
    \item Summations for Loops: One loop sequence of running time $f(i)$ is
        equivalent to: \[ T(n)=\sum_{i=1}^nf(i) \] Two loop sequences of running
        time $f(i,~j)$ is equivalent to:\[T(n)=\sum_{j=1}^n\sum_{i=1}^nf(i,~j)\]
    \item Selective Controls: Worst-case scenario, $T(n)=\mathrm{max}\left(
        T_1(n),~T_2(n),\cdots\right)$. Best-case = minimum.
    \item Recursion: $T(n)=f(T(n'))$ (점화식)
    \end{enumerate}
\end{enumerate}


\subsection{$n^{th}$ Derivative}
\noindent
We'll use induction to show that
\begin{equation*}
\Laplace{f^{(n)}(t)} = s^n\Laplace{f(t)} - f^{(n-1)}(0) - sf^{(n-2)}(0) - \ldots - s^{n-1}f(0)
\end{equation*}
We'll start with the first derivative as a base case. We could use the $0^{th}$ derivative, but this case will give us a little more insight into where the formula comes from.\\
Let $f$ be a differentiable function.
\begin{equation*}
\Laplace{f'(t)} = \lim\limits_{n\to\infty}{\int_{0}^{n}{f'(t)e^{-st}\mathrm{d}t}}
\end{equation*}
Integrating by parts and using the fundamental theorem of calculus,
\begin{equation*}
	 = \left(\lim\limits_{n\to\infty}{\left(f(n)e^{-sn}\right)} - f(0)e^{-s\cdot 0}\right) + \lim\limits_{n\to\infty}{s\int_{0}^{n}{f(t)e^{-st} \mathrm{d}t}}
\end{equation*}
Assuming that $f(n)$ grows slower than $e^{-sn}$,
\begin{equation*}
	 = \left(0 - f(0)\right) + \lim\limits_{n\to\infty}{s\int_{0}^{n}{f(t)e^{-st} \mathrm{d}t}}
\end{equation*}
Since the right part of the expression is just $s$ times the definition of $\Laplace{f}$,
\begin{equation*}
	 = s\Laplace{f} - f(0)
\end{equation*}
So,
\begin{equation*}
	\Laplace{f'(t)} = s\Laplace{f} - f(0)
\end{equation*}
Assuming the following is true,
\begin{equation*}
	\Laplace{f^{(n)}(t)} = s^n\Laplace{f(t)} - f^{(n-1)}(0) - sf^{(n-2)}(0) - \ldots - s^{n-1}f(0)
\end{equation*}
we'll show that the $n+1$ case follows.
\begin{equation*}
	\Laplace{f^{(n+1)}(t)} = \Laplace{\left(f^{(n)}\right)'}
\end{equation*}
Using our first derivative formula,
\begin{equation*}
	 = s\Laplace{f^{(n)}(t)} - f^{(n)}(0)
\end{equation*}
Using our general formula,
\begin{equation*}
	= s \left(s^n\Laplace{f(t)} - f^{(n-1)}(0) - sf^{(n-2)}(0) - \ldots - s^{n-1}f(0)\right) - f^{(n)}(0)
\end{equation*}
\begin{equation*}
 = s^{n+1}\Laplace{f(t)} - f^{(n)}(0) - sf^{(n-1)}(0) s^2f^{(n-2)}(0) - \ldots - s^{n}f(0)
\end{equation*}
which is the $n+1$ case, meaning we have proven the general formula as correct.


\subsection{Too many variables}

If there are more independent variables than samples then OLS will not work. There will be an infinite number of perfect fits.

For example if we regression genetic information on height with \(1000\) people, there will be too little data to fit using OLS.

This is due to colinearity.

We could also have too many variables through the use of derived variables. For example if we choose to use \(x\), \(x^2\), \(x^3\) etc.

\subsubsection{Optimal sparce regression}

Optimal is \(\lambda = \sigma 2\sqrt {{2\log (pn)}{n}}\)

Relies on knowing \(\sigma \), which we may not.

Instead we can use root LASSO.

Minimise the squareroot of the sum of squares loss (over n) , and use \(\lambda = \sqrt{2\log (pn)/n}\)

Doesn't have \(\sigma \)

Lasso biased, estimators \(0\) for many.

\subsubsection{Post-LASSO}

We can use LASSO for model selection, then use OLS on only those estimators.


%%% File: 3.5.tex
%%% Author: Alexandre Medeiros <alexandre.medeiros@students.ic.unicamp.br>

\paragraph{3.5}

\subparagraph{a.}

We have that $f(n) = 100n + \log n$ and $g(n) = n + (\log n)^2$.
Then let
\[
  \lim_{n \to \infty} \frac{f(n)}{g(n)} =
  \lim_{n \to \infty} \frac{100n + \log n}{n + (\log n)^2}
\]
By L'Hôpital's rule
\[
  \lim_{n \to \infty} \frac{100n + \log n}{n + (\log n)^2} =
  \lim_{n \to \infty} \frac{100 + \frac{1}{n}}{1 + 2\frac{\log n}{n}} = 100
\]
\begin{equation}\label{eq:ch03:5:a:O}
  \therefore f(n) = O(g(n))
\end{equation}
Now, let
\[
  \lim_{n \to \infty} \frac{g(n)}{f(n)} =
  \lim_{n \to \infty} \frac{n + (\log n)^2}{100n + \log n}
\]
By L'Hôpital's rule
\[
  \lim_{n \to \infty} \frac{n + (\log n)^2}{100n + \log n} =
  \lim_{n \to \infty} \frac{1 + 2\frac{\log n}{n}}{100 + \frac{1}{n}} =
  \frac{1}{100}
\]
\begin{equation}\label{eq:ch03:5:a:Om}
  \therefore g(n) = O(f(n)) \Leftrightarrow f(n) = \Omega(g(n))
\end{equation}
From \eqref{eq:ch03:5:a:O} and \eqref{eq:ch03:5:a:Om} we have that
\[
  f(n) = \Theta(g(n))
\]

\subparagraph{b.}

We have that $f(n) = \log n$ and $g(n) = \log n^2$.
Note that
\[
  g(n) = \log n^2 = 2\log n = 2 f(n)
\]
\[
  \therefore f(n) = \Theta(g(n))
\]

\subparagraph{c.}

We have that $f(n) = \frac{n^2}{\log n}$ and $g(n) = n(\log n)^2$.
Then let
\[
  \lim_{n \to \infty} \frac{f(n)}{g(n)} =
  \lim_{n \to \infty} \frac{\frac{n^2}{\log n}}{n(\log n)^2} =
  \lim_{n \to \infty} \frac{n}{(\log n)^3}
\]
By L'Hôpital's rule
\[
  \lim_{n \to \infty} \frac{n}{(\log n)^3} =
  \lim_{n \to \infty} \frac{n}{3(\log n)^2} =
  \lim_{n \to \infty} \frac{n}{6\log n} =
  \lim_{n \to \infty} \frac{n}{6} = \infty
\]
\[
  \therefore f(n) \neq O(g(n))
\]
Now, let
\[
  \lim_{n \to \infty} \frac{g(n)}{f(n)} =
  \lim_{n \to \infty} \frac{n(\log n)^2}{\frac{n^2}{\log n}} =
  \lim_{n \to \infty} \frac{(\log n)^3}{n}
\]
By L'Hôpital's rule
\[
  \lim_{n \to \infty} \frac{(\log n)^3}{n} =
  \lim_{n \to \infty} \frac{3(\log n)^2}{n} =
  \lim_{n \to \infty} \frac{6\log n}{n} =
  \lim_{n \to \infty} \frac{6}{n} = 0
\]
\[
  \therefore g(n) = O(f(n)) \Leftrightarrow f(n) = \Omega(g(n))
\]


\section{The likelihood function}

The likelihood function is defined as the product of a set of the probability density functions (pdfs) of variables $x$, that used to evaluate the probability of the observed dataset:
\begin{equation}\label{eq:likelihoodf}
    \mathcal{L} (x_{1}, ..., x_{N}; \theta_{1}, ..., \theta_{M}) = \prod_{i}^{N} f(x_{i}; \theta_{1}, ..., \theta_{M})
\end{equation}
where $\theta_{1}$, ..., $\theta_{M}$ are the nuisance parameters that can be written as $\pmb{\theta}$,
and $x_{1}$, ..., $x_{N}$ denote the observables of dataset.
Usually one measures the variable $x$ by constructing a histogram $\pmb{n} = (n_{1}, ..., n_{N})$.
The expectation value of the ith bin $n_{i}$ is written as~\cite{Cowan:2010js}:
\begin{equation}\label{eq:discri}
    E[n_{i}] = \mu s_{i} + b_{i}
\end{equation}
where $\mu$ is the signal strength, $s_{i}$ and $b_{i}$ are the number of signal and background events in that bin.
In addition to the histogram $\pmb{n}$, in some cases, one would like to use subsidiary measurements to further constrain the nuisance parameters.
For instance, due to the lack of background simulation or the mismodelling issue of one MC sample, one can choose a control region and construct another histogram $\pmb{m} = (m_{1}, ..., m_{M})$ to constrain the contribution of one certain background in data.
For this measurement, the expectation value of the ith bin $m_{i}$ is written as:
\begin{equation}\label{eq:cr}
    E[m_i] = u_i(\pmb{\theta})
\end{equation}

In most particle experiments, the number of these events observed in one bin follows the Poisson distribution,
by combining the equation~\ref{eq:discri} and ~\ref{eq:cr}, one can get the likelihood function for all bins as:
\begin{equation}
    \mathcal{L} (\mu, \pmb{\theta}) = \prod_{i=1}^{N} \frac{(\mu s_{i} + b_{i})^{n_i}}{n_i !} e^{-(\mu s_{i} + b_{i})}
    \prod_{k=1}^{M} \frac{u_k^{m_k}}{m_k !} e^{-u_k}
\end{equation}

Then the profile likelihood ratio is defined to test the hypothesized value of $\mu$:
\begin{equation} \label{eq:lambda}
    \lambda (\mu) = \frac{\mathcal{L}(\mu, \pmb{\hat{\hat{\theta}}})}{\mathcal{L}(\hat{\mu}, \pmb{\hat{\theta}})}
\end{equation}
where numerator denotes the local maximum-likelihood for a specific $\mu$, $\pmb{\hat{\hat{\theta}}}$ is the value of $\pmb{\theta}$ that maximizes the numerator.
And the denominator is the global maximum-likelihood with the $\hat{\mu}$ and $\pmb{\hat{\theta}}$ as their best fit value.


\subsection{Disks}

A disk is the area contained by a circle.

An open disk at \((a,b)\) of radius \(r\) is:

\(\{(x,y)\in \mathbb R^2:(x-a)^2+(y-b)^2 < r^2\}\)

For a closed disk it is:

\(\{(x,y)\in \mathbb R^2:(x-a)^2+(y-b)^2 \le r^2\}\)


\section{Likelihood Ratio Methods}
In this section, we describe several methods based on the \emph{likelihood ratio} technique of stochastic optimization. Let $Z$ be a random variable with a parametric probability density $p_\theta$ and assume that $p_\theta$ is known, explicitly computable and differentiable with respect to the parameters. The likelihood ratio technique is used to compute the following gradient
\begin{equation*}
	\nabla_\theta \E[Z \sim p_\theta]{f(Z)} = \nabla_\theta \int p_\theta(z) f(z) dz
\end{equation*}
This type of problem appears in many domains, such as when computing the greeks of a  derivative product in computational finance \cite{pages2016introduction}. Under some regularity assumptions on the function $f$ and on the probability distribution $p_\theta$, this gradient can be rewritten in a more amenable way
\begin{equation*}
	\begin{split}
		\nabla_\theta \E[Z \sim p_\theta]{f(Z)} &= \int \nabla_\theta p_\theta(z) f(z) dz\\
		&= \int p_\theta(z) \frac{\nabla_\theta p_\theta(z)}{p_\theta(z)} f(z) dz\\
		&= \int p_\theta(z) \nabla_\theta \log p_\theta(z) f(z) dz\\
		&= \E[Z\sim p_\theta]{\nabla_\theta \log p_\theta(Z) f(Z)}
	\end{split}
\end{equation*}
Thus, the likelihood ratio technique simply consists in the following equality
\begin{proposition}[Likelihood Ratio Technique]
\begin{equation}
	\nabla_\theta \E[Z \sim p_\theta]{f(Z)} = \E[Z\sim p_\theta]{\nabla_\theta \log p_\theta(Z) f(Z)}
\end{equation}
\end{proposition}
The quantity $\nabla_\theta \log p_\theta(Z)$ is usually called likelihood score. This result provides a simple way to approximate the gradient using a Monte Carlo estimate: let $\{Z^{(m)}\}_{m=1}^M$ be i.i.d. samples from $p_\theta$, then
\begin{equation}
	\nabla_\theta \E[Z \sim p_\theta]{f(Z)} \approx \frac{1}{M} \sum_{m=1}^M \nabla_\theta \log p_\theta(Z^{(m)}) f(Z^{(m)})
\end{equation}
As we will see in the following sections, this technique provides the basis for some of the most common policy gradient algorithms in the reinforcement learning literature.

\subsection{Monte Carlo Policy Gradient}
\label{sec:MCPG}
Let $h = {\{(s_t, a_t)\}}_{t\geq 0} \in \H$ be a given trajectory of the MDP and let us
denote by $p_\theta(h) = \P[\pi_\theta]{H = h}$ the probability of obtaining
this trajectory under policy $\pi_\theta$. Let $G(h)$ denote the expected return obtained on trajectory $h$
\begin{equation*}
	G(h) = \E{G_0 | H = h} = \sum_{t=0}^\infty \gamma^t \calR(s_t,
	a_t)
\end{equation*}
Let us consider the start value objective function, which can be rewritten as an expectation over all possible trajectories
\begin{equation*}
	J_{\text{start}}(\theta) = \E[H \sim p_\theta]{G(H)}
\end{equation*}
Applying the likelihood ratio technique, we obtain
\begin{equation}
\label{eq:gradient_MC}
		\nabla_\theta J(\theta) = \E[H\sim p_\theta]{\nabla_\theta \log p_\theta(H)G(H)}
\end{equation}
The crucial point is that $\nabla_\theta \log p_\theta(H)$ can be computed without
knowledge of the transition probability kernel $\calP$. Indeed, by a recursive application of the Markov property, we have
\begin{equation*}
	p_\theta(h) = \P{S_0 = s_0} \prod_{t=0}^\infty \pi_\theta(s_t, a_t)
	\calP(s_t, a_t, s_{t+1})
\end{equation*}
taking the logarithm yields
\begin{equation*}
	\log p_\theta(h) = \log \P{S_0 = s_0} + \sum_{t=0}^\infty \log
	\pi_\theta(s_t, a_t) + \sum_{t=0}^\infty \log \calP(s_t, a_t, s_{t+1})
\end{equation*}
Since the only term depending on the parameters $\theta$ is the policy term,
\begin{equation}
	\nabla_\theta \log p_\theta(H) = \sum_{t=0}^\infty \nabla_\theta \log
	\pi_\theta(s_t, a_t)
\end{equation}
Therefore, we do not need the transition model to compute the $\nabla_\theta \log
p_\theta(H)$. Moreover, it is easy to prove that
\begin{equation}
\label{eq:likelihood_bias}
	\E[H\sim p_\theta]{\nabla_\theta \log p_\theta(H)} = 0
\end{equation}
Hence, a constant baseline $b \in \R$ can always be subtracted in Eq. \ref{eq:gradient_MC} without changing the gradient value
\begin{equation}
\label{eq:gradient_MC_baseline}
	\nabla_\theta J(\theta) = \E[H \sim p_\theta]{\nabla_\theta \log p_\theta(H)(G(H)-b)}
\end{equation}
Intuitively, the baseline $b$ should measure the expected return under the policy, so that better (resp. worse) than average returns would produce a positive (resp. negative) gradient. A simple approach is to use a moving average of the returns observed while improving the policy. A more elaborate approach is to compute the baseline that minimizes the variance of the estimate of the gradient.\\
In an episodic environment, we can derive an estimate of the gradient by sampling $M$ trajectories $h^{(m)} = \{(s_t^{(m)}, a_t^{(m)})\}_{t = 0}^{T^{(m)}}$ under policy $\pi_\theta$ and by approximating the expected value via Monte Carlo
\begin{equation}
\label{eq:reinforce_gradient}
	\widehat{g}_{\text{RF}} = \frac{1}{M} \sum^{M}_{m=1} \left[ \sum_{i=0}^{T^{(m)}}
	\nabla_\theta \log \pi_\theta(s_i^{(m)}, a_i^{(m)}) \right] \left[
	\sum^{T^{(m)}}_{j=0} \gamma^j r_{j+1}^{(m)} - b \right]
\end{equation}
This method, synthetized in Algorithm \ref{algo:reinforce}, is known in the literature as the REINFORCE algorithm and is guaranteed to converge to the true gradient at a pace of $O(M^{-1/2})$. In practice, we can obtain an approximation of the gradient using only one sample which leads to a stochastic gradient ascent method
\begin{equation}
	\widehat{g}_{\text{SRF}} = \left[ \sum_{i=0}^{T} \nabla_\theta \log \pi_\theta(s_i,
	a_i) \right] \left[ \sum^{T}_{j=0} \gamma^j r_{j+1} - b \right]
\end{equation}
This method is very easy and works well on many problems. However, the gradient
estimate is characterized by a large variance which can hamper the convergence
rate of the algorithm. A first approach to address this issue is to optimally
set the baseline to reduce the gradient variance.

\subsubsection{Optimal Baseline}
A standard variance reduction technique consists in setting the baseline so as to minimize the gradient estimate variance. More in detail, the optimal baseline for the $k$-th gradient component $\widehat{g}_{k}$ solves
\begin{equation*}
	b_k^* = \argmin_b \Var{\widehat{g}_{k}}
\end{equation*}
It is easy to show that
\begin{equation*}
	b_k^* = \frac{\E{G(H) \left(\partial_{\theta_k} \log p_\theta(H)\right)^2  }}{\E{\left(\partial_{\theta_k} \log p_\theta(H)\right)^2}}
\end{equation*}
which can be approximated by
\begin{equation}
	\label{eq:optimal_baseline}
	\widehat{b}_k^* = \frac{\sum^{M}_{m=1} \left[ \sum_{i=0}^{T^{(m)}}
		\partial_{\theta_k} \log \pi_\theta\left(s_i^{(m)}, a_i^{(m)}\right) \right]^2
		\sum^{T^{(m)}}_{j=0} \gamma^j r_{j+1}^{(m)}}{\sum^{M}_{m=1} \left[ \sum_{i=0}^{T^{(m)}} \partial_{\theta_k} \log \pi_\theta\left(s_i^{(m)}, a_i^{(m)}\right) \right]^2}
\end{equation}

\begin{algorithm}[t]
	\caption{Episodic REINFORCE policy gradient estimate}
	\label{algo:reinforce}
	\begin{algorithmic}[1]
		\Require Policy parameterization $\theta$, number of trajectories $M$
		\Ensure REINFORCE policy gradient estimate $\widehat{g}_{RF} \approx \nabla_\theta J(\theta)$
		\State Sample $M$ trajectories of the MDP following policy $\pi_\theta$
		\State For all $k$, Compute the optimal baseline $b_k^*$ according to Eq. (\ref{eq:optimal_baseline})
		\State Compute $\widehat{g}_{\text{RF}}$ according to Eq. (\ref{eq:reinforce_gradient})
	\end{algorithmic}
\end{algorithm}

\subsection{GPOMDP}
The Monte Carlo policy gradient estimate is typically characterized by a large variance, which may slow down the method's convergence. To improve the estimate, it is sufficient to notice that future actions and past rewards are independent, unless the policy has been changed. Therefore, combining this observation with Eq. (\ref{eq:likelihood_bias}) yields
\begin{equation*}
	\E[\pi_\theta]{\nabla_\theta\log \pi_\theta(S_t, A_t) \calR(S_s, A_s)} = 0
	\;\;\;\;\; \forall t>s
\end{equation*}
Plugging this equation in Eq. (\ref{eq:gradient_MC_baseline}) leads to the well-known \gls{GPOMDP} algorithm \cite{baxter2001infinite} for generating an estimate of the gradient of
the objective function
\begin{equation}
\label{eq:GPOMDP}
	\widehat{g}_{\text{GPOMDP}} = \frac{1}{M} \sum^{M}_{m=1} \sum_{i=0}^{T^{(m)}}
	\nabla_\theta \log \pi_\theta(s_i^{(m)}, a_i^{(m)}) \left(
	\sum^{T^{(m)}}_{j=i} \gamma^j r_{j+1}^{(m)} - b \right)
\end{equation}
By removing almost half of the cross-products, this estimate typically has a smaller variance than the trivial REINFORCE policy gradient and can be further reduced by computing an optimal baseline. In the following sections we will see how this algorithm can be easily derived from a more general result: the policy gradient theorem.

\subsection{Stochastic Policies}
The likelihood ratio technique is a powerful tool and it allows to derive some powerful policy gradient algorithms. However, this approach only works if the policy is  stochastic. In most cases this is not a big problem, since stochastic policies are needed anyway to ensure sufficient exploration of the state-action space. Moreover, stochastic policies might be beneficial in partially observable \gls{MDP} to avoid state aliasing and in adversarial learning where the Pareto optimal strategies of different players are randomized, such as in the classical game ``rock-paper-scissors''. We introduce now two standard examples of stochastic policies for discrete and continuous action spaces respectively.

\subsubsection{Boltzmann Exploration Policy}
\label{sec:softmax}
In discrete action spaces, the Boltzmann exploration policy, also known as softmax policy, is a common choice. In state $s \in \S$, this policy selects an action $a \in \A$ with probability
\begin{equation}
	\pi_\theta(s,a) = \frac{e^{\theta^T \Phi(s,a)}}{\sum_{b \in \A} e^{\theta^T \Phi(s,b)}}
\end{equation}
where $\Phi(s,a) \in \R^{D_\theta}$ is a given feature vector corresponding to state $s$ and action $a$. the likelihood score for this policy is thus given by
\begin{equation}
\nabla_\theta \log \pi_\theta(s,a) = \Phi(s,a) - \sum_{b\in\A} \pi_\theta(s,a) \Phi(s,b)
\end{equation}

\subsubsection{Gaussian Exploration Policy}
\label{sec:gaussian_policy}
In continuous action spaces, a Gaussian exploration policy is commonly used. According to this policy, in a state $s$, actions are sampled from a Gaussian distribution with a parametric state-dependent mean $\mu_\psi(s) \in \R^{D_a}$, with $\psi \in \R^{D_\psi}$, and a covariance matrix $\Sigma \in \R^{D_a \times D_a}$. The policy parameters consist of $\theta = \{\psi, \Sigma\}$ and the likelihood scores are thus given by
\begin{equation}
	\nabla_\psi \log \pi_\theta(s,a) = \left(\frac{\partial \mu_\psi(s)}{\partial \psi}\right)^T \Sigma^{-1} (a - \mu_\psi(s))
\end{equation}
\begin{equation}
	\nabla_\Sigma \log \pi_\theta(s,a) = \frac{1}{2}\left[\Sigma^{-1} \left(a - \mu_\psi(s)\right) \left(a - \mu_\psi(s)\right)^T \Sigma^{-1} - \Sigma^{-1}\right]
\end{equation}
where $\frac{\partial \mu_\psi(s)}{\partial \psi}$ denotes the Jacobian matrix of $\mu_\psi$ with respect to $\psi$.

\subsection{Policy Gradient with Parameter Exploration}
In Monte Carlo Policy Gradient, trajectories are generated by sampling at each
time step an action according to a stochastic policy $\pi_\theta$ and the
objective function gradient is estimated by differentiating the policy with
respect to the parameters. However, sampling an action from the policy at each
time step leads to a large variance in the sampled histories and therefore in
the gradient estimate, which can in turn slow down the convergence of the
learning process. To address this issue, in \cite{sehnke2008policy} the authors
propose the \glsfirst{PGPE}
method, in which the search in the policy space is replaced with a direct
search in the model parameter space. We start by presenting the episodic case
and we will later extend this approach to the infinite horizon setting.

\subsubsection{Episodic PGPE}
Given an episodic \gls{MDP}, \gls{PGPE} considers a deterministic controller $F: \S \times \Theta \to \A$ that, given a set of parameters $\theta \in \Theta \subseteq
\R^{D_\theta}$, maps a state $s \in \S$ to an action $a = F(s; \theta) =
F_\theta(s) \in \A$. The policy parameters are drawn from a probability distribution $p_\xi$, with hyper-parameters $\xi \in \Xi \subseteq \R^{D_\xi}$. Combining these two
hypotheses, the agent follows a stochastic policy $\pi_\xi$ defined by
\begin{equation*}
	\forall B \in \calA ,\ \pi_\xi(s,B) = \pi(s, B; \xi) = \int_\Theta p_\xi(\theta)
	\ind{F_{\theta}(s)\in B} d\theta
\end{equation*}
The advantage of this approach is that the controller is deterministic and
therefore the actions do not need to be sampled at each time step, with a
consequent reduction of the gradient estimate variance. Indeed, It is
sufficient to sample the parameters $\theta$ once at the beginning of the
episode and then generate an entire trajectory following the deterministic
policy $F_\theta$. As an additional benefit, the parameter gradient is
estimated by direct parameter perturbations, without having to backpropagate
any derivatives, which allows to use non-differentiable controllers.\\
The hyper-parameters $\xi$ will be updated by following the gradient ascent direction of the gradient of the expected reward, which can be rewritten as
\begin{equation}
	\begin{split}
		J(\xi) = \E[\substack{\theta \sim p_\xi\\H \sim p_\theta}]{G(H)} =
		\int_\Theta \int_\H p_\xi(\theta, h) G(h) dh d\theta\\
	\end{split}
\end{equation}
By remarking that $h$ is conditionally independent from $\xi$ given $\theta$, so that $p_\xi(\theta, h) = p_\xi(\theta) p_\theta(h)$, and applying the likelihood ratio technique, we obtain
\begin{equation}
	\begin{split}
		\nabla_\xi J(\xi) = \E[\substack{\theta \sim p_\xi\\H \sim p_\theta}]{\nabla_\xi \log p_\xi(\theta) G(H)}
	\end{split}
\end{equation}
Again, we can subtract a constant baseline $b \in \R$ from the total return
\begin{equation}
	\nabla_\xi J(\xi) = \E{\nabla_\xi \log p_\xi(\theta)\left(G(H) - b\right)}
\end{equation}
In an episodic environment, the gradient can be approximated via Monte Carlo by first drawing $M$ samples $\theta^{(m)} \sim p_\xi$ and then, for each combination of parameters, generating a trajectory $h^{(m)} = \{(s_t^{(m)}, a_t^{(m)})\}_{t \geq 0}$ where actions are selected according to the deterministic controller $F_{\theta^{(m)}}$. This leads to the following estimate
\begin{equation}
\label{eq:pgpe_gradient}
	\widehat{g}_{\text{PGPE}} = \frac{1}{M} \sum^{M}_{m=1} \nabla_\xi \log p_\xi\left(\theta^{(m)}\right) \left[G\left(h^{(m)}\right)-b\right]
\end{equation}
In order to further reduce the estimate variance, an optimal baseline can be computed similarly to the REINFORCE case \cite{zhao2011analysis}. The episodic \gls{PGPE} algorithm obtained in this way is reported in Algorithm \ref{algo:episodic_pgpe}. In the following paragraphs, we discuss some possible choices for the policy parameters distribution $p_\xi$, which is the last component of the algorithm.

\begin{algorithm}[t]
	\caption{Episodic PGPE algorithm}
	\label{algo:episodic_pgpe}
	\begin{algorithmic}[1]
		\Require Initial hyper-parameters $\xi_0$, learning rate $\{\alpha_k\}$
		\Ensure Approximation of the optimal policy $F_{\xi^*} \approx \pi_*$
		\Repeat
			\For {$m = 1, \ldots, M$}
				\State Sample controller parameters $\theta^{(m)} \sim p_{\xi_k}$
				\State Sample trajectory $h^{(m)} = \{(s_t^{(m)}, a_t^{(m)})\}_{t \geq 0}$ under policy $F_{\theta^{(m)}}$
			\EndFor
			\State Approximate policy gradient $\nabla_\xi J(\xi_k) \approx \widehat{g}_\text{PGPE}$ using Eq. (\ref{eq:pgpe_gradient})
			\State Update hyperparameters using gradient ascent $\xi_{k+1} = \xi_k + \alpha_k \widehat{g}_\text{PGPE}$
			\State $k \leftarrow k + 1$
		\Until{converged}
	\end{algorithmic}
\end{algorithm}

\paragraph{Independent Gaussian Parameter Distribution}
A simple approach is to assume that all the components of the parameter vector
$\theta$ are independent and normally distributed with mean $\mu_i$ and
variance $\sigma_i^2$, in other words $\theta_i \sim \mathcal{N}(\mu_i, \sigma_i^2)$, the gradient with respect to the hyper-parameters $\xi = (\mu_1, \ldots, \mu_{D_\theta}, \sigma_1,$ $\ldots, \sigma_{D_\theta})^T$ is given by
\begin{equation}
	\begin{split}
		\pardev{\log p_\xi(\theta)}{\mu_i} &= \frac{\theta_i -
		\mu_i}{\sigma_i^2}\\
		\pardev{\log p_\xi(\theta)}{\sigma_i} &= \frac{(\theta_i - \mu_i)^2 -
		\sigma_i^2}{\sigma_i^3}
	\end{split}
\end{equation}
Using a constant learning rate $\alpha_i = \alpha \sigma_i^2$, the gradient
updates takes the following form
\begin{equation}
	\begin{split}
		\mu_i^{k+1} &= \mu_i^k + \alpha \left[G(h) - b\right] (\theta_i - \mu_i)\\
		\sigma_i^{k+1} &= \sigma_i^k + \alpha \left[G(h) - b\right]
		\frac{(\theta_i -\mu_i)^2}{\sigma_i}
	\end{split}
\end{equation}
where $b$ can be computed as a moving average of the past returns. Intuitively,
if $G(h) > b$ we adjust $\xi$ so as to increase the probability of $\theta$
while if $G(h) < b$ we do the opposite.

\paragraph{Gaussian Parameter Distribution}
A more elaborate approach is to assume a generic dependence among the controller parameters, namely $\theta \sim \calN(\mu, \Sigma)$. However, if we directly used the covariance matrix $\Sigma$ as an hyper-parameter, it would be computationally difficult to enforce that it remains well-defined, namely symmetric and semidefinite positive, during the gradient ascent iterations. A simpler approach is to parameterize the distribution using the Cholesky factor $\Sigma$, the matrix $C$ such that $\Sigma = C^T C$. This choice has two advantages: first, $C$ makes explicit the $n(n+1)/2$ independent parameters determining the covariance matrix $\Sigma$; in addition, $C^T C$ is by construction a well-defined covariance matrix. Hence, the hyper-parameters are $\xi =\{\mu, C\}$. The likelihood score for this distribution does not have a simple expression, but it becomes extremely easy in the Natural version of \gls{PGPE} which will be discussed in the next sections.

\paragraph{Symmetric Sampling and Gain Normalization}
In some settings, comparing the gain with a baseline can be misleading. In
their original work, the authors propose a symmetric sampling technique similar
to antithetic variates that further improves the convergence of the method.
More in detail, a more robust gradient estimate can be obtained by measuring
the difference in reward between two symmetric samples on either side of the
current mean. That is, we sample a random perturbation $\epsilon \sim
\mathcal{N}(0, \Sigma)$, where $\Sigma = \text{diag}(\sigma_1^2, \ldots,
\sigma_{D_\theta}^2)$, and we define $\theta^+ = \mu + \epsilon$ and $\theta^- =
\mu - \epsilon$. Denoting by $G^+$ (resp. $G^-$) the gains obtained on the
trajectory associated to $\theta^+$ (resp. $\theta^-$), the objective function
gradient can be approximated with
\begin{equation}
	\begin{split}
		\nabla_{\mu_i} J(\xi) &\approx \frac{\epsilon_i (G^+ - G^-)}{2
		\sigma_i^2}\\
		\nabla_{\sigma_i} J(\xi) &\approx \frac{\epsilon_i^2 -
	\sigma_i^2}{\sigma_i^3}\left(\frac{G^+ + G^-}{2} - b\right)
	\end{split}
\end{equation}
Hence, by choosing $\alpha_i^k = 2 \alpha \sigma_i^2$, we have the following
update rules
\begin{equation}
	\begin{split}
		\mu_i^{k+1} &= \mu_i^k + \alpha \epsilon_i (G^+ - G^-)\\
		\sigma_i^{k+1} &= \sigma_i^k + \alpha \frac{\epsilon_i^2 - \sigma_i^2}
		{\sigma_i}\left(G^+ + G^- - 2b\right)
	\end{split}
\end{equation}
In addition, the authors propose to normalize the gains in order to make the
updates independent of the scale of the rewards. For instance, we could modify
the hyperparameters updates as follows
\begin{equation}
	\begin{split}
		\mu_i^{k+1} &= \mu_i^k + \alpha \epsilon_i \frac{(G^+ - G^-)}{2m - G^+
	- G^-}\\
		\sigma_i^{k+1} &= \sigma_i^k + \alpha \frac{\epsilon_i^2 - \sigma_i^2}
	{\sigma_i}\frac{\left(G^+ + G^- - 2b\right)}{m - b}
	\end{split}
\end{equation}
where $m$ might be the maximum gain the agent can receive, if known, or
alternatively the maximum gain achieved so far. Symmetric sampling and gain
normalization can drastically improve the gradient estimate quality and
consequently the convergence time.

\subsubsection{Infinite Horizon PGPE}
While in the episodic PGPE the parameters $\theta$ are sampled only at the beginning of each episode, in \gls{IHPGPE} \cite{sehnke2012parameter} the parameters and learning are carried out simultaneously, while interacting with the environment. Let $0 < \varepsilon < 1$ the probability of updating the policy parameters, the parameters $\theta_t$ can be sampled consecutively as follows
\begin{equation}
	p_\xi(\theta_{i,t+1}) = \varepsilon \mathcal{N}(\mu_{i,t}, \sigma_{i,t}^2)
							+ (1-\varepsilon) \delta_{\theta_{i,t}}
\end{equation}
In practice, $\varepsilon$ should be chosen so that the expected frequency of
changing a single parameter is coherent with the typical episode length in the
episodic framework. Alternatively, one could sample all the parameters at a
certain time step simultaneously
\begin{equation}
	p_\xi(\theta_{t+1}) = \varepsilon \mathcal{N}(\mu_{t}, \Sigma_t)
							+ (1-\varepsilon) \delta_{\theta_{t}}
\end{equation}
This is equivalent to splitting the state-action space into artificial
episodes. However, updating parameters asynchronously changes the policy only
slightly thus introducing less noise in the process. Again, parameters can be
updated at every time step by gradient ascent
\begin{equation}
	\begin{split}
		\mu_{i,t+1} &= \mu_{i,t} + \alpha \left[G_t(h) - b\right] (\theta_{i,t}
		- \mu_{i,t})\\
		\sigma_{i,t+1} &= \sigma_{i,t} + \alpha \left[G_t(h) - b\right]
		\frac{(\theta_{i,t} -\mu_{i,t})^2}{\sigma_{i,t}}
	\end{split}
\end{equation}
Similarly to the episodic case, we can improve the gradient estimate by
symmetric sampling and gain normalization.


\subsection{Axiom of pairing}

For any pair of sets, \(x\) and \(y\) there is another set \(z\) which containing only \(x\) and \(y\).

\(\forall x \forall y \exists z \forall a[a\in z \leftrightarrow a=x \lor a=y]\)

\subsubsection{For each set, there exists a set containing only that set}

Take the axiom, but replace all instance of \(y\) with \(x\).

\(\forall x \exists z \forall a[a\in z \leftrightarrow a=x \lor a=x]\)

\(\forall x \exists z \forall a[a\in z \leftrightarrow a=x]\)

\subsubsection{For any finite number of sets, there is a set containing only those sets}

\subsubsection{For any finite number of sets, there is a set containing the intersection of those sets}


\chapter*{Appendix D: Generalization of Derivative Needed for Minimization}
\label{annexC}
\addcontentsline{toc}{chapter}{Appendix D:  Generalization of Derivative Needed for Minimization}

The contribution of the wavelet coefficient $x$ to the noise and signal
information in the general case is
\begin{eqnarray}
h_n(x) & = & \int_{0}^{\mid x \mid } p_n(u|x) (\frac{\partial h(x)}{\partial
x})_{x=u} du \\ \nonumber
h_s(x) & = & \int_{0}^{\mid x \mid } p_s(u|x) (\frac{\partial h(x)}{\partial
x})_{x=u} du
\end{eqnarray}

Assuming $h(x) = \frac{1}{2}x^2$, we have
\begin{eqnarray}
h_n(x) & = & \int_{0}^{\mid x \mid } p_n(x-u) u du \\ \nonumber
h_s(x) & = & \int_{0}^{\mid x \mid } p_s(x-u) u du
\end{eqnarray}

\begin{eqnarray}
\frac{ d h_{s}(x)}{dx} = \int_0^x  \frac{\partial P_s(x-u)}{\partial x})_{x=u} u du +
         \frac{1}{dx}  \int_x^{x+dx} P_s(x-u) u  du
\end{eqnarray}

As $P_s(0) = 0$, the second term tends to zero.

Denoting  $\frac{\partial P_s(x-u)}{\partial x} = - \frac{\partial P_s(x-u)}{\partial u}$, we have
\begin{eqnarray}
\frac{ d h_{s}(x)}{dx} & = & - \int_0^x \frac{\partial P_s(x-u)}{\partial u} u du \\
& = &  - ( [u P_s(x-u) ]_0^x - \int_0^w P_s(x-u) du) \\
& = & \int_0^x P_s(x-u) du \\
& = & \int_0^x P_s(u) du
\end{eqnarray}

and from $h_n = h - h_s$ we get
\begin{eqnarray}
\frac{ d h_{n}(x)}{dx} & = &  x -  \int_0^x P_s(u) du
\end{eqnarray}
and
\begin{eqnarray}
\frac{ d h_{s}(y-x)}{dx} = - \int_0^{y-x} P_s(u) du
\end{eqnarray}

It is easy to verify that replacing $P_s(x) = \mbox{erf}(x)$,
and $P_n(x) = \mbox{erfc}(x)$,
(case of Gaussian noise) we find the same equation as in Appendix B.


\documentclass{tufte-handout}

\usepackage{xcolor}
\usepackage{graphicx}

% set hyperlink attributes
\hypersetup{colorlinks}

\usepackage{amsmath}

% set image attributes:
\usepackage{graphicx}
\graphicspath{ {images/} }

% create environment for bottom paragraph:
\newenvironment{bottompar}{\par\vspace*{\fill}}{\clearpage}

% ============================================================

% define the title
\title{SOC 4930/5050: Week 07 Equations Quick \\Reference}
\author{Christopher Prener, Ph.D.}
\date{October 9\textsuperscript{th}, 2017}
% ============================================================
\begin{document}
% ============================================================
\maketitle % generates the title
% ============================================================

\vspace{5mm}
\section{One-sample T-test}\marginnote{Degrees of freedom ($v$) is defined as $v=n-1$.}
\begin{equation}
\scalebox{2} {$ t=\frac { \bar { x } - \mu }{ \frac{s}{\sqrt{n}} }  $}
\end{equation}

\vspace{5mm}
\section{Independent T-test, Homogeneous Variance}
\begin{subequations}
\paragraph{Independent T-test}\marginnote{Degrees of freedom ($v$) is defined as $v={n}_{a}+{n}_{b}-2$.}
\begin{equation}
\scalebox{2} {$ t=\frac { { \bar{X}  }_{ a }-{ \bar{X}  }_{ b } }{ \sqrt{\frac{{ s }_{ p }^{ 2 }}{{ n }_{ a }}+\frac{{ s }_{ p }^{ 2 }}{{ n }_{ b }}} } $}
\end{equation}

\vspace{3mm}
\paragraph{Pooled Variance}
\begin{equation}
\scalebox{2} {$ { s }_{ p }^{ 2 }=\frac{\left({n}_{a}-1\right){s}_{a}^{2}+\left({n}_{b}-1\right){s}_{b}^{2}}{{n}_{a}+{n}_{b}-2} $}
\end{equation}
\end{subequations}

\vspace{5mm}
\section{Independent T-test, Heterogeneous Variance}
\begin{subequations}
\paragraph{Independent T-test}
\begin{equation}
\scalebox{2} {$ t=\frac { { \bar{X}  }_{ a }-{ \bar{X}  }_{ b } }{ \sqrt{\frac{{ s }_{ a }^{ 2 }}{{ n }_{ a }}+\frac{{ s }_{ b }^{ 2 }}{{ n }_{ b }}} } $}
\end{equation}

\vspace{3mm}
\paragraph{Welch's Corrected Degrees of Freedom ($v$)}
\begin{equation}
\scalebox{2} {$ v\approx \frac { \left( \frac { { s }_{ a }^{ 2 } }{ { n }_{ a } } +\frac { { s }_{ b }^{ 2 } }{ { n }_{ b } }  \right) ^{ 2 } }{ \frac { { s }_{ a }^{ 4 } }{ \left( { n }_{ a }^{2} \right)\left( { n }_{ a }-1 \right) } +\frac { { s }_{ b }^{ 4 } }{ \left( { n }_{ b }^{2} \right)\left( { n }_{ b }-1 \right) }  } $}
\end{equation}
\end{subequations}

\vspace{5mm}
\section{Dependent T-test}
\begin{equation}
\scalebox{2} {$ t=\frac{\bar{d}}{\sqrt{\frac{{s}_{d}^{2}}{n}}} $}
\end{equation}

\vspace{5mm}
\section{Cohen's $D$}\marginnote{Note that groups $t$ and $c$ are defined for controlled experiments where $t=treatment$ and $c=control$. This can be applied to the above equations by defining $t=a$ and $c=b$.}
\begin{subequations}
\paragraph{General Equation}
\begin{equation}
\scalebox{2} {$ d=\frac{{M}_{t}-{M}_{c}}{\sqrt{\frac{\left({n}_{t}-1\right){s}_{t}^{2}+\left({n}_{c}-1\right){s}_{c}^{2}}{{n}_{t}+{n}_{c}-2}}} $}
\end{equation}

\vspace{3mm}
\paragraph{Cohen's $D$ after T-test, ${n}_{a}={n}_{b}$}
\begin{equation}
\scalebox{2} {$ d=\frac{2t}{\sqrt{v}} $}
\end{equation}

\vspace{3mm}
\paragraph{Cohen's $D$ after T-test, ${n}_{a}\neq{n}_{b}$}
\begin{equation}
\scalebox{2} {$ d=\frac{t\left({n}_{t}+{n}_{c}\right)}{\sqrt{v}\left(\sqrt{{n}_{t}+{n}_{c}} \right)} $}
\end{equation}
\end{subequations}

% ============================================================
\end{document}


\section*{Exercises}

\begin{ex}
  Compute the following complex dot products in $\C^n$:
  \begin{equation*}
    (a)
    \begin{mymatrix}{c} 1 \\ i \end{mymatrix}
    \dotprod
    \begin{mymatrix}{c} 1+i \\ 2 \end{mymatrix},
    \quad
    (b)
    \begin{mymatrix}{c} 2 \\ 1 \\ i \end{mymatrix}
    \dotprod
    \begin{mymatrix}{c} i \\ 1 \\ -3 \end{mymatrix},
    \quad
    (c)
    \begin{mymatrix}{c} i \\ 1 \\ -3 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{c} 2 \\ 1 \\ i \end{mymatrix},
    \quad
    (d)
    \begin{mymatrix}{c} i \\ i+1 \\ 1 \end{mymatrix}
    \dotprod
    \begin{mymatrix}{c} 1+2i \\ -2 \\ -i \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    (a) $1-i$, (b) $1+5i$, (c) $1-5i$, (d) $0$.
  \end{sol}
\end{ex}

\begin{ex}
  Compute the norm of each of the following vectors in $\C^n$:
  \begin{equation*}
    (a) \begin{mymatrix}{c} 1 \\ i \end{mymatrix},\quad
    (b) \begin{mymatrix}{c} 1 \\ 0 \\ 1+2i \end{mymatrix},\quad
    (c) \begin{mymatrix}{c} 1+i \\ 1-i \\ 3i \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    (a) $\sqrt{2}$, (b) $\sqrt{6}$, (c) $\sqrt{13}$.
  \end{sol}
\end{ex}

\begin{ex}
  Suppose $\vect{u},\vect{v}\in\C^n$ and
  $\vect{u}\dotprod\vect{u} = 1$, $\vect{v}\dotprod\vect{v} = 2$, and
  $\vect{u}\dotprod\vect{v} = 2i$. Then compute:
  \begin{enumerate}
  \item $(2\vect{u}+\vect{v})\dotprod(\vect{u}-\vect{v})$.
  \item $(i\vect{u}-\vect{v})\dotprod(\vect{u}-2i\vect{v})$.
  \end{enumerate}
  \begin{sol}
    (a) $-6i$, (b) $5i$.
  \end{sol}
\end{ex}

\begin{ex}
  Compute the adjoint of the following matrices.
  \begin{equation*}
    (a)
    \begin{mymatrix}{cc}
      i   \\
      i-1 \\
      2   \\
    \end{mymatrix},
    \quad
    (b)
    \begin{mymatrix}{cc}
      1 & i \\
      -i & 2 \\
    \end{mymatrix},
    \quad
    (c)
    \begin{mymatrix}{ccc}
      2 & 1+i & -i \\
      1-i & 3 & 2+2i \\
    \end{mymatrix}.
  \end{equation*}
\end{ex}

\begin{ex}
  Suppose $\vect{u}$ and $\vect{v}$ are vectors in a complex inner
  product space such that $\iprod{\vect{u},\vect{u}} = 3$,
  $\iprod{\vect{v},\vect{v}} = 2$, and $\iprod{\vect{u},\vect{v}} =
  i+1$. Then compute:
  \begin{enumerate}
  \item $\iprod{\vect{u}+2\vect{v},\vect{u}-i\vect{v}}$.
  \item $\iprod{\vect{u}-i\vect{v},2\vect{u}+\vect{v}}$.
  \end{enumerate}
\end{ex}

\begin{ex}
  In $C[0,2\pi]$, consider the functions $f(x) = \sin x + i\cos x$,
  $g(x) = 1$, and $h(x) = x$. Compute the following:
  \begin{equation*}
    (a)~~ \iprod{f,g},\quad
    (b)~~ \iprod{f,h},\quad
    (c)~~ \iprod{f,f},\quad
    (d)~~ \norm{f}.
  \end{equation*}
\end{ex}

\begin{ex}
  Which of the following vectors, if any, are orthogonal?
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{c} 1 \\  i \\ 0 \end{mymatrix},\quad
    \vect{u}_2 = \begin{mymatrix}{c} 1 \\ -i \\ 1 \end{mymatrix},\quad
    \vect{u}_3 = \begin{mymatrix}{c} 0 \\  1 \\ -i \end{mymatrix},\quad
    \vect{u}_4 = \begin{mymatrix}{c} 0 \\  0 \\ 1 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $\vect{u}_1\orth\vect{u}_2$, $\vect{u}_1\orth\vect{u}_4$, and
    $\vect{u}_2\orth\vect{u}_3$.
  \end{sol}
\end{ex}

\begin{ex}
  Suppose that $B=\set{\vect{u}_1,\vect{u}_2,\vect{u}_3}$ is an
  orthogonal basis for a complex inner product space $V$, such that
  $\norm{\vect{u}_1}=\sqrt{2}$, $\norm{\vect{u}_2}=2$, and
  $\norm{\vect{u}_3}=\sqrt{3}$. Moreover, suppose that $\vect{v}\in V$ is a
  vector such that $\iprod{\vect{u}_1,\vect{v}} = 1$,
  $\iprod{\vect{u}_2,\vect{v}} = 3i$, and
  $\iprod{\vect{u}_3,\vect{v}} = 1-i$. Find the coordinates of
  $\vect{v}$ with respect to $B$.
\end{ex}

\begin{ex}
  Consider $\C^3$ with the complex dot product. Let
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{c} 0 \\ i \\ 2 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}_2 = \begin{mymatrix}{c} 1 \\ 1+i \\ 3i+2  \end{mymatrix}
  \end{equation*}
  Use the Gram-Schmidt procedure to find an orthogonal basis for
  $\sspan\set{\vect{v}_1,\vect{v}_2}$. Then find an orthonormal basis.
  \begin{sol}
    Orthogonal: $\mat{0,i,2}^T$, $\mat{1,2,i}^T$. Orthonormal:
    $\frac{1}{\sqrt{5}}\mat{0,i,2}^T$, $\frac{1}{\sqrt{6}}\mat{1,2,i}^T$.
  \end{sol}
\end{ex}

\begin{ex}
  Consider the subspace of $\C^3$ spanned by the orthogonal vectors
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{c} 2 \\ i \\ 1 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{u}_2 = \begin{mymatrix}{c} i \\ 3 \\ i \end{mymatrix}.
  \end{equation*}
  Find the best approximation of
  $\vect{v}=\begin{mymatrix}{c} -2i \\ 3 \\ 4i \end{mymatrix}$ in this
  subspace.
\end{ex}


\subsection{Outer product}

\subsubsection{The outer product is a bilinear map}

This is a bilinear map from two vectors from the same vector space to another vector space.

\(V\times V \rightarrow V\)

\subsubsection{Calculating the outer product}

\(u\otimes v = w\)

\(w_{ij}=u_iv_j\)

\subsubsection{The dimensions of the tensor outer product}

\(\dim (V \otimes W)= \dim V \times \dim W \)

\subsubsection{Outer product on the complex numbers}

\subsubsection{Relation between the dot product and outer product}

The dot product in the trace of the outer product.


%!TEX root = Calculus_I.tex
\chapter{Short-Cuts to Differentiation}
This chapter will cover simple rules that are used to take the \textbf{\textit{derivative}} for a variety of functions.

\section{Powers and Polynomials}
The \textbf{\textit{derivative}} of a constant multiple is given by:
\begin{equation}
\label{eq:ConstantMultDeriv}
\frac{d}{dx}\left[c f(x)\right] = c f^\prime(x)
\end{equation}
%
If $f(x)$ is \textbf{\textit{differentiable}} and $c$ is constant. This is a special case of the \textbf{\textit{Product Rule}}, which will be covered shortly. Proving this rule can be done using the definition of the \textbf{\textit{derivative}}:
\begin{equation}
\frac{d}{dx}\left[c f(x)\right] = \lim_{h \rightarrow 0} \frac{c f(x+h) - cf(x)}{h} = \lim_{h \rightarrow 0} c\frac{f(x+h) - f(x)}{h}
\end{equation}
%
\begin{equation}
\lim_{h \rightarrow 0} c\frac{f(x+h) - f(x)}{h} = c f^\prime(x)
\end{equation}

\vspace{0.1in}
The derivative of two functions, $f(x)$ and $g(x)$, added or subtracted together, provided both $f(x)$ and $g(x)$ are \textbf{\textit{differentiable}}, is given by:
%
\begin{equation}
\label{eq:SumDeriv}
\frac{d}{dx}\left[f(x) + g(x)\right] = f^\prime(x) + g^\prime(x)
\end{equation}
%
\begin{equation}
\label{eq:DiffDeriv}
\frac{d}{dx}\left[f(x) - g(x)\right] = f^\prime(x) - g^\prime(x)
\end{equation}
%
Again, using the definition of the \textbf{\textit{derivative}}:
\begin{equation}
\frac{d}{dx}\left[f(x) + g(x)\right] = \lim_{h \rightarrow 0} \frac{\left[f(x+h) + g(x+h)\right] - \left[f(x) + g(x)\right]}{h}
\end{equation}
%
\begin{equation}
\label{eq:SumDiffLim}
\lim_{h \rightarrow 0} \left[\frac{f(x+h) - f(x)}{h} + \frac{g(x+h) - g(x)}{h}\right]
\end{equation}
%
where the limits of each term in Equation (\ref{eq:SumDiffLim}) approach $f^\prime(x)$ and $g^\prime(x)$.

\vspace{0.1in}
For equations containing powers of $x$, the \textbf{\textit{Power Rule}} can be used:
\begin{equation}
\label{eq:PowerRule}
\frac{d}{dx}\left(x^n\right) = nx^{n-1}
\end{equation}
This is valid provided that $n$ is constant and $n \in \mathbb{R}$. Combining the rules shown in Equations (\ref{eq:ConstantMultDeriv}), (\ref{eq:SumDeriv}), (\ref{eq:DiffDeriv}), and (\ref{eq:PowerRule}), the \textbf{\textit{derivative}} of \textit{any \textbf{polynomial}} can be taken.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{The Exponential Function}
Recall that \textbf{\textit{exponential functions}} are given by:
\begin{equation}
f(x) = a^x
\end{equation}
These are \textbf{\textit{monotonically increasing}} functions; thus, the \textbf{\textit{derivative}} must be strictly positive. Suppose we have a function, $f(x)$, given by:
\begin{equation}
f(x) = 2^x
\end{equation}
Using the definition of the \textbf{\textit{derivative}}:
\begin{equation}
f^\prime(x) = \lim_{h \rightarrow 0} \left(\frac{2^{x+h} - 2^x}{h}\right) = \lim_{h \rightarrow 0} \left(\frac{{2^x 2^h} - 2^x}{h}\right)
\end{equation}
%
\begin{equation}
\lim_{h \rightarrow 0} \left(\frac{2^h - 1}{h}\right)\cdot 2^x
\end{equation}
%
Using this representation, we can approach the \textbf{\textit{limit}} of the coefficient of $2^x$ by taking small values of $h$:
%
\begin{table}[tb]
\begin{center}
\caption{Evaluations of coefficient $\frac{2^h - 1}{h}$ for various values of $h$}
\begin{tabular}{l|c}
\hline
$h$ & $\frac{2^h - 1}{h}$\\
\hline
-0.1 & 0.6697\\
-0.01 & 0.6908\\
-0.001 & 0.6929\\
0.001 & 0.6934\\
0.01 & 0.6956\\
0.1 & 0.7177\\
\hline
\end{tabular}
\end{center}
\end{table}
%
This suggests that the \textbf{\textit{limit}} of $\frac{2^h - 1}{h}$ as $h \rightarrow 0$ is approximately 0.693. Thus:
%
\begin{equation}
g^\prime(x) = \frac{d}{dx} \left(0.693\cdot 2^x\right)
\end{equation}
%
Applying Equation (\ref{eq:ConstantMultDeriv}) to this expression means that only the \textbf{\textit{derivative}} of $2^x$ must be taken. This indicates that the \textbf{\textit{derivative}} of an \textbf{\textit{exponential function}} depends upon the function itself. In the general form:
\begin{equation}
f(x) = a^x
\end{equation}
\begin{equation}
f^\prime(x) = \lim_{h \rightarrow 0} \left(\frac{a^{x+h} - a^x}{h}\right) = \lim_{h \rightarrow 0}\left(\frac{a^h - 1}{h}\right)\cdot a^x
\end{equation}
%
\begin{table}[b]
\begin{center}
\caption{Limit evaluations of coefficient $\frac{a^h - 1}{h}$ for various values of $a$}
\label{tab:GenFormExponentialLims}
\begin{tabular}{l|c}
\hline
$a$ & $\lim_{h \rightarrow 0}\frac{a^h - 1}{h}$\\
\hline
2 & 0.693\\
3 & 1.099\\
4 & 1.386\\
5 & 1.609\\
6 & 1.792\\
7 & 1.946\\
\hline
\end{tabular}
\end{center}
\end{table}
%
Looking at Table~\ref{tab:GenFormExponentialLims} indicates that there is a number, $a$, between 2 and 3 such that $\lim_{h \rightarrow 0}\frac{a^h - 1}{h}$ is 1. Thus:
%
\begin{equation}
a^h - 1 \approx h, \hphantom{-} h \ll 1
\end{equation}
%
\begin{equation}
a^h \approx 1+h
\end{equation}
%
\begin{equation}
\label{eq:NaturalNumberDefinition}
a \approx \left(1+h\right)^{\frac{1}{h}}
\end{equation}
%
Taking the \textbf{\textit{limit}} of Equation~(\ref{eq:NaturalNumberDefinition}) results in the values shown in Table~\ref{tab:NaturalNumberDefinition}
%
\begin{table}[b]
\begin{center}
\caption{Evaluations of coefficient $\left(1+h\right)^{\frac{1}{h}}$ for various values of $h$}
\label{tab:NaturalNumberDefinition}
\begin{tabular}{l|c}
\hline
$h$ & $\left(1+h\right)^{\frac{1}{h}}$\\
\hline
-0.001 & 2.7196422\\
-0.0001 & 2.7184178\\
-0.00001 & 2.7182954\\
0.00001 & 2.7182682\\
0.0001 & 2.7181459\\
0.001 & 2.7169239\\
\hline
\end{tabular}
\end{center}
\end{table}
%
We observe that $a \approx 2.718$. This is referred to as the \textbf{\textit{Natural Number}}, $e$. Using the \textbf{\textit{derivative}} relationships shown previously:
\begin{equation}
\frac{d}{dx}\left(e^x\right) = e^x
\end{equation}
%
To adjust this expression for various values of $a$, we use a combination of the \textbf{\textit{limit}} and \textbf{\textit{derivative}} defintions previously presented and properties of logarithms:
%
\begin{equation}
f^\prime(x) = \lim_{h \rightarrow 0} \frac{a^{x+h} - a^x}{h} = \left(\lim_{h \rightarrow 0} \frac{a^h - 1}{h} \right)a^x
\end{equation}
%
\begin{equation}
a = e^{\ln a}
\end{equation}
%
\begin{equation}
\lim_{h \rightarrow 0} \frac{\left(e^{\ln a}\right)^h - 1}{h} =  \lim_{h \rightarrow 0} \frac{e^{\left({\ln a}\right)h} - 1}{h}
\end{equation}
%
From the definition of the \textbf{\textit{Natural Number}}, $e$:
%
\begin{equation}
\lim_{h \rightarrow 0} \frac{e^h - 1}{h} = 1
\end{equation}
%
Substituting $t = (\ln a) \cdot h$:
%
\begin{equation}
\lim_{h \rightarrow 0}\frac{e^{\left(\ln a\right)h} - 1}{h} = \lim_{t \rightarrow 0} \frac{e^t - 1}{t / \ln a} = \lim_{t \rightarrow 0}\left(\ln a \cdot \frac{e^t - 1}{t}\right)
\end{equation}
%
\begin{equation}
= \ln a \left(\lim_{t \rightarrow 0} \frac{e^t - 1}{t}\right) = (\ln a) \cdot 1 = \ln a
\end{equation}
%
Thus:
%
\begin{equation}
\frac{d}{dx}\left(a^x\right) = (\ln a)a^x
\end{equation}

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{The Product and Quotient Rules}
Recall the Greek symbol $\Delta$ refers to the change in a quantity. For a function $f(x)$, the change in the function can be written as:
%
\begin{equation}
\Delta f = f(x+h) - f(x)
\end{equation}
%
for small values of $h$. Using this notation, we can arrive at the definition of the \textbf{\textit{derivative}} as:
%
\begin{equation}
f^\prime(x) = \lim_{h \rightarrow 0} \frac{\Delta f}{h}
\end{equation}
%
Suppose we know the \textbf{\textit{derivatives}} of two functions, $f(x)$ and $g(x)$, but we want to calculate the \textbf{\textit{derivative}} of their product, $f(x)g(x)$. A proof of this is included in the text these notes are based on (see Mechanical-Engineering-Curriculum/Calculus_I/README.md), but the \textbf{\textit{Product Rule}} can be used to determine this \textbf{\textit{derivative}}. The proof uses the definition of the \textbf{\textit{limit}}, but the \textbf{\textit{Product Rule}} is given as:
%
\begin{equation}
(fg)^\prime = f^\prime g + fg^\prime
\end{equation}
%
provided that $u = f(x)$ and $v = g(x)$ are \textbf{\textit{differentiable}}. Using \textbf{\textit{Leibniz's Notation}}:
%
\begin{equation}
\frac{d(uv)}{dx} = \frac{d}{dx}(uv)= \frac{du}{dx} \cdot v + u \cdot \frac{dv}{dx}
\end{equation}
%
Similarly, the \textbf{\textit{Quotient Rule}} is the inverse of the \textbf{\textit{Product Rule}}. If a function $Q(x) = \frac{f(x)}{g(x)}$ exists, and $u = f(x)$ and $v =g(x)$ are \textbf{\textit{differentiable}} then the \textbf{\textit{Quotient Rule}} is given as:
%
\begin{equation}
\left(\frac{f}{g}\right)^\prime = \frac{f^\prime g - f g^\prime}{g^2}
\end{equation}
%
Representing this in \textbf{\textit{Leibniz's Notation}}:
%
\begin{equation}
\frac{d}{dx}\left(\frac{u}{v}\right) = \frac{\frac{du}{dx} \cdot v - u \frac{dv}{dx}}{v^2}
\end{equation}
%
Note that some expressions are likely to be easier handled by one rule over another, but equations can also be re-written such that multiplicative operations appear as division, such as:
%
\begin{equation}
\frac{1}{x} = x^{-1}
\end{equation}
%
Here, the \textbf{\textit{derivative}} of the right-hand side can be easily computed using the \textbf{\textit{Quotient Rule}}, but the left-hand side is in the form of the \textbf{\textit{Power Rule}}, but both expressions are equivalent.
%
\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{The Chain Rule}
The \textbf{\textit{Chain Rule}} is used to take the \textbf{\textit{derivative}} of \textbf{\textit{composite functions}}. Suppose $f(g(x))$ is a \textbf{\textit{composite function}}. Then, substituting $z = g(x)$ yields:
%
\begin{equation}
y = f(z)
\end{equation}
%
Because $z$ is a function of $x$, a small change in $x$, $\Delta x$ will result in a small change in $z$, $\Delta z$. In turn, $\Delta z$ results in a small change in the dependent variable, $y$, because $y = f(z)$. If $\Delta x$ and $\Delta z$ are not identically zero, then:
%
\begin{equation}
\frac{\Delta y}{\Delta x} = \frac{\Delta y}{\Delta z} \cdot \frac{\Delta z}{\Delta x}
\end{equation}
%
Because $\frac{dy}{dx} = \lim_{\Delta x \rightarrow 0} \frac{\Delta y}{\Delta x}$, as these three differentials get closer to zero the \textbf{\textit{Chain Rule}} can be defined as:
%
\begin{equation}
\frac{dy}{dx} = \frac{dy}{dz} \cdot \frac{dz}{dx}
\end{equation}
%
Furthermore, because $\frac{dy}{dz} = f^\prime(z)$ and $\frac{dz}{dx} = g^\prime(x)$:
%
\begin{equation}
\frac{d}{dx}\left(f\left(g\left(x\right)\right)\right) = f^\prime\left(z\right) \cdot g^\prime(x)
\end{equation}
%
Back-substituting $z = g(x)$ yields:
%
\begin{equation}
\frac{d}{dx}\left(f\left(g\left(x\right)\right)\right) = f^\prime \left(\left(g\left(x\right)\right)\right) \cdot g^\prime(x)
\end{equation}
%
\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{The Trigonometric Functions}
This section is concerned with the \textbf{\textit{derivatives}} of the $sine$ and $cosine$ functions. Because these functions are periodic, their \textbf{\textit{derivatives}} must also be periodic. This can be done by looking at the graph of the functions and taking the \textbf{\textit{limit}} at multiple points for small changes from the point, $h$. Note that the relationships between the\textit{sine} and \textit{cosine} functions and the Unit Circle require $x$ to be in \textbf{\textit{radians}}.

\vspace{0.1in}
Taking the limit at multple points along the $sine$ function's domain of $[0, 2\pi]$ results in the following relationship:
%
\begin{equation}
\frac{d}{dx}\left(\sin(x)\right) = \cos(x)
\end{equation}
%
Following a similar process for the $cosine$ function:
%
\begin{equation}
\frac{d}{dx}\left(\cos(x)\right) = -\sin(x)
\end{equation}
%
The \textbf{\textit{derivative}} of \textit{cosine} is $-\sin$ because of the phase shift between the two functions. Recall that:
%
\begin{equation}
\cos(x) = \sin\left(x + \frac{\pi}{2}\right)
\end{equation}
%
Also note that the \textbf{\textit{Chain Rule}} must be used to differentiate the trigonometric functions because they are \textbf{\textit{composite functions}}. For the \textit{tangent} function, which is defined as:
%
\begin{equation}
\tan(x) = \frac{\sin(x)}{\cos(x)}
\end{equation}
%
the \textbf{\textit{Quotient Rule}} can be easily used. It results in the following:
%
\begin{equation}
\frac{d}{dx}\left(\tan(x)\right) = \frac{d}{dx}\left(\frac{\sin(x)}{\cos(x)}\right)
\end{equation}
%
\begin{equation}
\frac{d}{dx}\left(\tan(x)\right) = \frac{\left(\sin(x)\right)^\prime \cdot \left(\cos(x)\right) - \left(\sin(x)\right) \cdot \left(\cos(x)\right)^\prime}{\cos^2(x)}
\end{equation}
%
\begin{equation}
\frac{d}{dx}\left(\tan(x)\right) = \frac{\sin^2(x) + \cos^2(x)}{\cos^2(x)} = \frac{1}{\cos^2(x)}
\end{equation}
%
Again, these relationships require $x$ to be in \textbf{\textit{radians}}.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{The Chain Rule and Inverse Functions}
We will now apply the \textbf{\textit{Chain Rule}} to calculate \textbf{\textit{derivatives}} of \textbf{\textit{logarithmic functions}}, \textbf{\textit{exponential functions}}, and the \textbf{\textit{inverse trigonometric functions}}. A general rule for the \textbf{\textit{derivative}} of any \textbf{\textit{inverse function}} will also be shown.

\vspace{0.1in}
Recall that, using the properties of \textbf{\textit{logarithms}}, we can write:
%
\begin{equation}
e^{\ln x} = x
\end{equation}
%
To arrive at the \textbf{\textit{derivative}} of $\ln x$, we will use the equation $e^{\ln x} = x$:
%
\begin{equation}
\frac{d}{dx}\left(e^{\ln x}\right) = \frac{d}{dx}\left(x\right)
\end{equation}
%
\begin{equation}
e^{\ln x} \cdot \frac{d}{dx}\left(\ln x\right) = 1
\end{equation}
%
Thus:
%
\begin{equation}
\frac{d}{dx} \left(\ln x \right) = \frac{1}{e^{\ln x}} = \frac{1}{x}
\end{equation}
%

For \textbf{\textit{derivatives}} of the \textbf{\textit{inverse trigonometric functions}}, $\arcsin(x)$, $\arccos(x)$, and $\arctan(x)$, we can utilize several trigonometric identities. For the \textbf{\textit{derivative}} of $\arctan(x)$, note that $\tan(\arctan(x)) = x$. Differentiation via the \textbf{\textit{Chain Rule}} yields:
%
\begin{equation}
\frac{1}{\cos^2(\arctan(x))} \cdot \frac{d}{dx}\left(\arctan(x)\right) = 1
\end{equation}
%
\begin{equation}
\frac{d}{dx}\left(\arctan(x)\right) = \cos^2(\arctan(x))
\end{equation}
%
The trigonometric identity $1 + \tan^2(\theta) = \frac{1}{\cos^2(\theta)}$ can be used here, replacing $\theta$ with $\arctan(x)$; thus:
%
\begin{equation}
\cos^2(\arctan(x)) = \frac{1}{1 + \tan^2(\arctan(x))} = \frac{1}{1 + x^2}
\end{equation}
%
\begin{equation}
\frac{d}{dx}\left(\arctan(x)\right) = \cos^2(\arctan(x)) = \frac{1}{1 + x^2}
\end{equation}
%
Following a similar process for $\arcsin(x)$ and $\arccos(x)$:
%
\begin{equation}
\frac{d}{dx}\left(\arcsin(x)\right) = \frac{1}{\sqrt{1-x^2}}
\end{equation}
%
\begin{equation}
\frac{d}{dx}\left(\arccos(x)\right) = -\frac{1}{\sqrt{1-x^2}}
\end{equation}
%
For general \textbf{\textit{inverse functions}}, if a function, $f(x)$, has a \textbf{\textit{differentiable inverse}}, $f^{-1}$, its \textbf{\textit{derivative}}, $f\left(f^{-1}(x)\right) = x$ by the \textbf{\textit{Chain Rule}}:
%
\begin{equation}
\frac{d}{dx}\left(f\left(f^{-1}\left(x\right)\right)\right) = 1
\end{equation}
%
\begin{equation}
f^\prime\left(f^{-1}\left(x\right)\right) \cdot \frac{d}{dx}\left(f^{-1}\left(x\right)\right) = 1
\end{equation}
%
Therefore:
%
\begin{equation}
\frac{d}{dx}\left(f^{-1}\left(x\right)\right) = \frac{1}{f^\prime\left(f^{-1}\left(x\right)\right)}
\end{equation}
%

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Implicit Functions}
So far, the functions presented have been considered \textbf{\textit{explicit functions}} of $x$, functions of the form $y = f(x)$. This section addresses \textbf{\textit{implicit functions}}, where one $x$ value corresponds to multiple $y$ values. An example of an \textbf{\textit{implicit function}} is the equation of a circle:
%
\begin{equation}
\label{eq:circle}
x^2 + y^2 = r^2
\end{equation}
%
Here, $y$ is a function of $x$ on both the top and bottom halves of the circle, buy when the circle is considered as a whole, as in Equation~(\ref{eq:circle}), where the function does not have a tangent lineat each point, but the equation is still \textbf{\textit{differentiable}} with respect to $x$:
%
\begin{equation}
\label{eq:circle_derivative}
\frac{d}{dx}\left(x^2\right) + \frac{d}{dx}\left(y^2\right) = \frac{d}{dx}\left(r^2\right)
\end{equation}
%
Because $r$ is a constant of the circle, \textbf{\textit{differentiation}} results in the right-hand side of Equation~(\ref{eq:circle_derivative}) goes to zero. If $y$ is treated as a function of $x$, and the \textbf{\textit{Chain Rule}} is used:
%
\begin{equation}
2x + 2y\frac{dy}{dx} = 0
\end{equation}
%
Solving this for the $\frac{dy}{dx}$ yields:
%
\begin{equation}
\frac{dy}{dx} = -\frac{x}{y}
\end{equation}
%
This relationship is valid, provided $y \neq 0$. This is expected because the line tangent to the circle at this point is vertical, so the \textbf{\textit{slope}} is infinite.
\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Hyperbolic Functions}
The \textbf{\textit{Hyperbolic Functions}} are combinations of $e^x$ and $e^{-x}$. \textit{Hyperbolic Sine}, abbreviated $\sinh(x)$, and \textit{Hyperbolic Cosine}, abbreviated $\cosh(x)$, are given by:
%
\begin{equation}
\label{eq:sinh}
\sinh(x) = \frac{e^x - e^{-x}}{2}
\end{equation}
%
\begin{equation}
\label{eq:cosh}
\cosh(x) = \frac{e^x + e^{-x}}{2}
\end{equation}
%
Observing properties of these functions graphically, and recalling that $\frac{d}{dx}\left(e^x\right) = e^x$, the following results hold:
\begin{table}[H]
\begin{center}
\caption{Properties of $\sinh(x)$ and $\cosh(x)$}
\begin{tabular}{c|c}
$\cosh(0) = 1$ & $\sinh(0) = 0$\\
$\cosh(-x) = \cosh(x)$ & $\sinh(-x) = -\sinh(x)$\\
$\frac{d}{dx}\left(\cosh(x)\right) = \sinh(x)$ & $\frac{d}{dx}\left(\sinh(x)\right) = \cosh(x)$
\end{tabular}
\end{center}
\end{table}
%
Furthermore, the it can be shown that the following result is true:
%
\begin{equation}
\cosh^2(x) - \sinh^2(x) = 1
\end{equation}
%
The \textit{Hyperbolic Tangent} function can also be defined, and is given by:
%
\begin{equation}
\tanh(x) = \frac{\sinh(x)}{\cosh(x)} = \frac{e^x - e^{-x}}{e^x + e^{-x}}
\end{equation}
%
It's derivative is given by:
%
\begin{equation}
\frac{d}{dx}\left(\tanh(x)\right) = \frac{d}{dx}\left(\frac{\sinh(x)}{\cosh(x)}\right) = \frac{\left(\cosh(x)\right)^2 - \left(\sinh(x)\right)^2}{\left(\cosh(x)\right)^2}
\end{equation}
%
\begin{equation}
\frac{d}{dx}\left(\tanh(x)\right) = \frac{1}{\cosh^2(x)}
\end{equation}
%
\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Linear Approximation and the Derivative}
When zooming into the graph of a \textbf{\textit{differentiable}} function, if the change between numbers is small enough $(\Delta x \rightarrow 0)$, the graph will look more and more like a line. This idea of \textbf{\textit{local linearity}} is commonly used to approximate the \textbf{\textit{slope}} or \textbf{\textit{derivative}} of a function $f(x)$ at a specific point, $x = a$ and function values near that point, $y = f(a+h)$. The \textbf{\textit{Tangent Line Approximation}} is given by:
%
\begin{equation}
f(x) \approx f(a) + f^\prime(a)(x - a)
\end{equation}
%
provided that $f(x)$ is \textbf{\textit{differentiable}} at $x = a$. The resulting error in the approximation is given by:
%
\begin{equation}
E(x) = f(x) - f(a) - f^\prime(a)(x-a)
\end{equation}
%
More information about this error approximation will be discussed in the Calculus II course.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Theorems about Differentiable Functions}
The \textbf{\textit{Mean Value Theorem}} states that if $f(x)$ is \textbf{\textit{continuous}} on $[a, b]$, and is \textbf{\textit{differentiable}} on $(a, b)$, then a number $c$ exists such that:
%
\begin{equation}
f^\prime(c) = \frac{f(b) - f(a)}{b -a}
\end{equation}
%
which is the slope of the secant line between points $x=a$, and $x=b$. This theorem does not indicate how to find $c$, though.

\vspace{0.1in}
The \textbf{\textit{Increasing Function Theorem}} states that if a function $f(x)$ is \textbf{\textit{continuous}} on $[a, b]$, and is \textbf{\textit{differentiable}} on $(a, b)$, then:
%
\begin{enumerate}
\item If $f^\prime(x) > 0 \hphantom{-} \forall x \in (a, b)$, then $f$ is \textbf{\textit{increasing}} on $[a, b]$.\\
\vspace{-0.25in}
\item If $f^\prime(x) \geq 0\hphantom{-}  \forall x \in (a, b)$, then $f$ is \textbf{\textit{non-decreasing}} on $[a,b]$.
\end{enumerate}
%

\vspace{0.1in}
The \textbf{\textit{Constant Function Theorem}} states that if a function $f(x)$ is \textbf{\textit{continuous}} on $[a, b]$, and is \textbf{\textit{differentiable}} on $(a, b)$, then if $f^\prime(x) = 0 \hphantom{-} \forall x \in [a, b]$, then $f$ is constant on $x \in [a, b]$.

\vspace{0.1in}
The \textbf{\textit{Racetrack Principle}} states that for two functions, $g(x)$, and $h(x)$, that are \textbf{\textit{continuous}} on $[a, b]$, and is \textbf{\textit{differentiable}} on $(a, b)$, and that $g^\prime(x) \leq h^\prime(x) \hphantom{-} \forall x \in (a, b)$:
%
\begin{enumerate}
\item If $g(a) = h(a)$, then $g(x) \leq h(x) \hphantom{-} \forall x \in [a, b]$.\\
\vspace{-0.25in}
\item If $g(b) = h(b)$, then $g(x) \geq h(x) \hphantom{-} \forall x \in [a,b]$.
\end{enumerate}
%

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}


\chapter{Approximations}

\section{VERTEX-COVER}

Create a copy of the edge list.  Choose an arbitrary edge from this
copy, and add both vertices on this edge to the set which will become
the vertex cover.  Then remove all edges in our edge list copy which
are incident to either vertex on the chosen edge.  Repeat this process
until our edge list copy is empty.

Exercise for the reader: prove that this is a 2-approximation.

\section{Bin Packing}

We describe the ``First-Fit'' approximation.  Given an object to pack
and a list of open bins, add the object to the first bin into which it
will fit.  If it fits into no bins, open a new bin and insert the object.

It should be easy to see that no two bins will be less than half full,
otherwise the object(s) in the second bin would have been added to the
first.

The worst case for this approximation is that every bin will be about
half full.  In which case it is no worse than twice the optimal, so
this is a 2-approximation.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS622: Theory of Formal Languages
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Prove that the context-free grammar $ G = (\{S\},\{a,b,c\},S,\{S\rightarrow SS, S\rightarrow \lambda, S\rightarrow aSb, S\rightarrow bSa, S\rightarrow aSc, S\rightarrow cSa\}) $
generates the language $ L = \{x \in \{a,b,c\}^* | n_a(x) = n_b(x) + n_c(x) \}$.

\subsection*{Solution}

Proof is given in two steps. First we prove $L(G) \subseteq L$. Second, by proving $L \subseteq L(G)$ we convert inclusion to equality.

\begin{enumerate}

	\item
	It is claimed that for any word $w \in L(G)$, $n_a = n_b + n_c$.
	It is clear that using the first two productions will not hinder the condition for they do not generate terminal symbols.
	However, using any of the four latter productions would increment $n_a$ while incrementing either $n_b$ or $n_c$.
	Thus, informally, the statement $w \in L$ holds true.
	A more formal solution can be given by induction on length of $w$.

	\begin{itemize}[label={}]

		\item
		Clearly, if $|w| = 0$, $w = \lambda$, $n_a = n_b + n_c = 0$.
		If $|w| = 2$, either $S\rightarrow aSx$ or $S\rightarrow xSa$ where $x \in \{b,c\}$ and then $S\rightarrow \lambda$.
		In which case, $w \in L$.
		Note that we cannot have a word generated by grammar $G$ which is of odd length.
		Nor can we have a word $w$ with odd length whose $n_a(w) = n_b(w) + n_c(w)$.

		\item
		We take the induction hypothesis as, for $w \in L(G)$ such that $|w| = p$, $w \in L$.
		It is shown for any $w^\prime \in L(G)$ such that $|w^\prime| = p + 2$, $w^\prime \in L$.
		Starting from $S$, $w^\prime$ is generated first by generating $w$ such that $|w| = p$, then applying one of the four productions in $G$ that generate terminal symbols and finally by using $S\rightarrow \lambda$.
		As discussed previously, applying either one of the four productions would increment $n_a$ by one.
		As there is no production that increment $n_b$ and $n_c$ by one at the same time, the argument $n_a = n_b + n_c$ still holds true and therefore $w \in L$.

	\end{itemize}

	Thus $L(G) \subseteq L$.

	\item
	Now we prove $L \subseteq L(G)$, that is, for any $w \in L$, $w$ can be generated by S.
	It is claimed that grammar $G$ is indifferent to position and order of the symbols of $w$ as long as they follow $n_a(w) = n_b(w) + n_c(w)$.
	Proof is given again by induction on length of $w$.

	\begin{itemize}[label={}]

		\item
		Clearly if $|w| = 0$, $w = \lambda \in L$ and $S\xRightarrow[G]{} \lambda = w$, thus $w \in L(G)$.
		If $|w| = 2$, $w \in L$ either begins with an $a$ or ends with an $a$.
		In the first case, $S\xRightarrow[G]{} aSx \xRightarrow[G]{} ax$ and in the second case, $S\xRightarrow[G]{} xSa \xRightarrow[G]{} xa$, where $x$ is either $b$ or $c$.
		Both cases prove $w \in L(G)$.

		\item
		We take the induction hypothesis that $S\xRightarrow[G]{*}w$ for $|w| = k$.
		Let $w = uv$ such that $u,v \in A^*$.
		Based on hypothesis, $S \xRightarrow[G]{*} uSv \xRightarrow[G]{} uv = w$.
		For any $|w^\prime| = k + 2$, $S \xRightarrow[G]{*} uSv$.
		Then either $uSv \xRightarrow[G]{} uaSxv \xRightarrow[G]{} uaxv = w^\prime$ or $uSv \xRightarrow[G]{} uxSav \xRightarrow uxav = w^\prime$ where $x$ is $b$ or $c$.
		Therefore $w^\prime \in L(G)$.

	\end{itemize}

	Thus $L \subseteq L(G)$.

\end{enumerate}

Based on the equality, grammar $G$ can be said to generate language $L$ as defined.


    \subsection{Catalan Number}
    	Binary trees with $n+1$ vertices.
      \begin{equation*}
        C_{n+1} = \sum_{i=0}^nC_iC_{n-i}
      \end{equation*}
      %Una fórmula cerrada para los números de Catalán es:
      \begin{equation*}
        C_n = \frac{1}{n+1}\binom{2n}{n} = \binom{2n}{n} - \binom{2n}{n+1}
      \end{equation*}
    \subsection{Stirling Numbers($1^{st}$ kind)} Arrangements of an $n$ element set into $k$ cycles.
      \begin{equation*}
        \stirlingfirst{n}{k} = (n-1)\stirlingfirst{n-1}{k} + \stirlingfirst{n-1}{k-1}
      \end{equation*}

    \subsection{Stirling Numbers($2^{nd}$kind)} \# of ways of dividing $n$ distinct element into $k$ non-empty sets.
      \begin{equation*}
        \stirlingsecond{n}{k} = k\stirlingsecond{n-1}{k} + \stirlingsecond{n-1}{k-1}
      \end{equation*}

      Además:
      \begin{equation*}
        \stirlingsecond{n}{k} = \frac{1}{k!}\sum_{j=0}^k (-1)^{k-j} \binom{k}{j} j^n
      \end{equation*}

      \subsection{Bell Numbers} Count the number of ways to divide $n$ elements into subsets.
      \begin{equation*}
        \mathcal{B}_{n+1} = \sum_{k=0}^n \binom{n}{k} \mathcal{B}_k
      \end{equation*}

      \

      \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|}
        \hline
        x&0&1&2&3&4&5&6&7&8&9&10 \\ \hline %&9&10&11&12
        $\mathcal{B}_x$&1&1&2&5&15&52&203&877&4.140&21.147&115.975 \\ \hline %&678.570&4.213.597
      \end{tabular}

      \subsection{Derangement} Permutation that leaves no element in the original location.

      \begin{equation*}
        !n = (n - 1)( !(n - 1) + !(n - 2) ); !1 = 0, !2 = 1
      \end{equation*}

      \begin{equation*}
        !n = n! \sum_{k = 0}^n \frac{(-1)^k}{k!}
      \end{equation*}

    \subsection{Números armónicos}

    \begin{equation*}
     H_n = \sum_{k = 1}^n \frac{1}{k}
    \end{equation*}

    \begin{equation*}
     \frac{1}{2n+1} < H_n - \ln n - \gamma < \frac{1}{2n}
    \end{equation*}

    \begin{equation*}
     \gamma = 0.57721 56649 01532 86060 65120 90082 40243 10421 59335 \ldots
    \end{equation*}

    \subsection{Fibonacci Number} $f_0 = 0$, $f_1 = 1$:

    \begin{equation*}
     f_n = \frac{1}{\sqrt{5}}(\frac{1 + \sqrt{5}}{2})^n - \frac{1}{\sqrt{5}}(\frac{1 - \sqrt{5}}{2})^n
    \end{equation*}

    \begin{equation*}
     f_{n+1}^2 + f_n^2 = f_{2n + 1},
     f_{n+2}^2 - f_n^2 = f_{2n + 2}
    \end{equation*}

    \begin{equation*}
     f_n = \sum_{j = 0}^{\lfloor \frac{n}{2} \rfloor} \binom{n-j}{j}
    \end{equation*}

    \subsection{Sums of Combinations}

    \begin{equation*}
     \sum_{i = n}^m \binom{i}{n} = \binom{m + 1}{n + 1}
    \end{equation*}

    \begin{equation*}
     \sum_{i = 0}^k \binom{n}{i} \binom{m}{k - i} = \binom{n + m}{k}
    \end{equation*}

    \subsection{Generating functions}
      \

      \begin{tabular}{|c|c|}
        \hline
        $(1,1,1,1,1,1,\ldots)$ & $\frac{1}{1-z}$ \\ \hline
        $(1,-1,1,-1,1,-1,\ldots)$ & $\frac{1}{1+z}$ \\ \hline
        $(1,0,1,0,1,0,\ldots)$ & $\frac{1}{1-z^2}$ \\ \hline
        $(1,0,\ldots,0,1,0,1,0,\ldots,0,1,0,\ldots)$ & $\frac{1}{1-z^2}$ \\ \hline
        $(1,2,3,4,5,6,\ldots)$ & $\frac{1}{(1-z)^2}$ \\ \hline
        $(1,\binom{m+1}{m},\binom{m+2}{m},\binom{m+3}{m},\ldots)$ & $\frac{1}{(1-z)^{m+1}}$ \\ \hline
        $(1,c,\binom{c+1}{2},\binom{c+2}{3},\ldots)$ & $\frac{1}{(1-z)^c}$ \\ \hline
        $(1,c,c^2, c^3, \ldots)$ & $\frac{1}{1-cz}$ \\ \hline
        $(0,1,\frac{1}{2},\frac{1}{3},\frac{1}{4},\ldots)$ & $\ln \frac{1}{1-z}$ \\ \hline
      \end{tabular}

    \subsection{Combination Basis}
    		Occupancy $n$ Balls into $m$ Boxes
      \

      \begin{tabular}{|c|c|c|c|c|}
        \hline
         & Balls labeled & Boxes labeled & Any Box Empty & \# of ways \\ \hline
        $(1)$ & N & N & N & $P_m(n)-P_{m-1}(n)$ \\ \hline
        $(2)$ & N & N & Y & $P_m(n)$ \\ \hline
        $(3)$ & N & Y & N & $\binom{n-1}{m-1}$ \\ \hline
        $(4)$ & N & Y & Y & $\binom{n+m-1}{n}$ \\ \hline
        $(5)$ & Y & N & N & $\stirlingsecond{n}{m}$ \\ \hline
        $(6)$ & Y & N & Y & $\stirlingsecond{n}{1}+...+\stirlingsecond{n}{m}$ \\ \hline
        $(7)$ & Y & Y & N & $m!\stirlingsecond{n}{m}$ \\ \hline
        $(8)$ & Y & Y & Y & $m^n$ \\ \hline
      \end{tabular}
    \begin{enumerate}
    \item[*] $P_m(n)$:\# of partitions of $n$ into $1,2,...,m$ with repetition allowed.
    \item[*] $P_m(n)=P_{m-1}(n)+P_{m}(n-m)$
    \item[*] $(8)=\sum^m_{i=1}\binom{m}{i}(7)$
    \item[*] $(4)=\sum^m_{i=1}\binom{m}{i}(3)$
    \end{enumerate}
    Taylor's series:
$$
f(x) = f(a) + (x-a)f'(a) + {(x-a)^2 \over 2} f''(a) + \cdots = \sum_{i=0}^\infty {(x-a)^i \over i!} f^{(i)}(a).
$$

%{F_n x \over 1 - (F_{n-1} + F_{n+1})x - (-1)^n x^2}
%&F_n x + F_{2n} x^2 + F_{3n} x^3 + \cdots
%&\sum_{i=0}^\infty F_{ni} x^i.\cr

%{x \cot x}
%&\sum_{i=0}^\infty {(-4)^i B_{2i} x^{2i} \over (2i)!},\cr
%{\tan x}
%&\sum_{i=1}^\infty (-1)^{i-1}{2^{2i} (2^{2i} - 1) B_{2i} x^{2i-1} \over %(2i)!},\cr