\section{Limits}
Limits are the foundation of calculus.
A lot of other things commonly used in calculus are made with the help of limits.
For example the derivative of \(x^2\) is \(2x\).
This can be proven by using limits.
A limit of a function is the value of the function when it approaches a value.

\subsection{Limit law}

Given two limits \(\lim_{x \to a} f(x)\) and \(\lim_{x \to a} g(x)\) we can say that:

\begin{table}[h]
	\centering
	\begin{tabular} {c}
    \(\lim_{x \to a} [f(x) + g(x)] = \lim_{x \to a} f(x) + \lim_{x \to a} g(x)\) \\\hline
    \(\lim_{x \to a} [f(x) - g(x)] = \lim_{x \to a} f(x) - \lim_{x \to a} g(x)\) \\\hline
    \(\lim_{x \to a} [c f(x)] = c \lim_{x \to a} f(x)\) \\\hline
    \(\lim_{x \to a} [f(x)g(x)] = \lim_{x \to a} f(x) * \lim_{x \to a} g(x)\) \\\hline
    \(\lim_{x \to a} \frac{f(x)}{g(x)} = \frac{\lim_{x \to a} f(x)}{\lim_{x \to a} g(x)} \text{if} \lim_{x \to a} g(x) \neq 0\) \\
	\end{tabular}
	\caption{Limit laws}
\end{table}

\subsection{Finding a limit}
\begin{figure}[H]
  \begin{center}
    \begin{tikzpicture}
      \begin{axis}[
          axis x line=center,
          axis y line=center,
          xtick={-5,-3,...,5},
          ytick={0,0.5,...,2},
          xlabel=$x$,
          ylabel=$y$,
          xmin=-5,
          xmax=5,
          ymin=0,
          ymax=2,
          samples=5
        ]
        \addplot [color=blue] {(x-1)/(x-1)};
        \legend{\((x-1)/(x-1)\)}
      \end{axis}
    \end{tikzpicture}
  \end{center}
\end{figure}

The simplest way of approximating the limit,
is to evaluate the function with an \(x\) close to the limit you want.
Take for example the function graphed above.
\(f(1)\) is obviously not defined, you cannot solve it that way.
But you can approximate it by filling \(1.00000001\) into the function.
This gives \(1\), which is the limit.

\subsection{Doing it properly}
This does not always work, so it's better to rewrite the function.
For example:

\[
  f(x) =
  \frac{x-1}{x-1} =
  \frac{1}{1} =
  1
\]
Obviously this is a simple function, other functions will not be this simple.
For example, solving the following function will look like this:

\begin{dmath}
  \lim_{x \to 0} \frac{\sqrt{x^2 + 9} - 3}{x^2} =
  \lim_{x \to 0} \frac{\sqrt{x^2 + 9} - 3}{x^2} * \frac{\sqrt{x^2 + 9} + 3}{\sqrt{x^2 + 9} + 3} =
  \lim_{x \to 0} \frac{(x^2 + 9) - 9}{x^2(\sqrt{x^2 + 9} + 3)} =
  \lim_{x \to 0} \frac{1}{\sqrt{x^2 + 9} + 3} =
  \frac{1}{\sqrt{0^2 + 9} + 3} =
  \frac{1}{6}
\end{dmath}

\subsection{Continuity}
A function is {\bf continuous} iff

\begin{itemize}
  \item \(f(a)\) is defined
  \item \(\lim_{x \to a} f(x)\) exits
  \item \(\lim_{x \to a} f(x) = f(a)\)
\end{itemize}

So, \(f\) is continuous at \(a\) if \(f(x)\) approaches \(f(a)\).
If this is not the case we can say that te function is {\bf discontinuous at a}.

\subsection{Limits at infinity}
If you take an infinity positive or negative number as the limit you can get the horizontal asymptotes of a function.
Take for example the following function:

\begin{figure}[H]
  \begin{center}
    \begin{tikzpicture}
      \begin{axis}[
          axis x line=center,
          axis y line=center,
          xtick={-20,-10,...,20},
          ytick={-2,-1,...,2},
          xlabel=$x$,
          ylabel=$y$,
          xmin=-20,
          xmax=20,
          ymin=-2,
          ymax=2,
          samples=75
        ]
        \addplot [color=blue,domain=-20:20] {(x^2-1)/(x^2+1)};
        \legend{\((x^2-1)/(x^2+1)\)}
      \end{axis}
    \end{tikzpicture}
  \end{center}
\end{figure}

This function probably has a horizontal limit at \(y=1\).
But you don't know for sure.
You can use the limit of infinity to prove them.
As constants don't really matter when approaching infinity you can remove them.
So you'll get the following:

\[
  \lim_{x \to \infty} \frac{x^2-1}{x^2+1} =
  \frac{\infty^2-1}{\infty^2+1} =
  \frac{\infty^2}{\infty^2} =
  \frac{1}{1} =
  1
\]
This gets the horizontal asymptote on the right.
If you want to get the limit on the left you'll have to use negative infinity.

\[
  \lim_{x \to -\infty} \frac{x^2-1}{x^2+1} =
  \frac{-\infty^2-1}{-\infty^2+1} =
  \frac{-\infty^2}{-\infty^2} =
  \frac{1}{1} =
  1
\]
In this example it's also \(y=1\), as both asymptotes are the same.
One important thing to keep in mind when solving infinite limits is that \(\frac{1}{\infty} = 0\).


\subsection{Belief update}
\label{sec:belief}

Having estimated $P_{i,t}$ in the first phase, the second phase is rather simple. Ignoring normalizing constants, \Cref{eq:new-obj} tells us that the pdf of the skill posterior can be obtained as the pointwise product of the pdfs of the skill prior and the performance model. When both factors are differentiable and log-concave, then so is their product. Its maximum is the new rating $\mu_{i,t}$; let's see how to compute it for the same two specializations of our model.

\paragraph{Gaussian performance model}
When the skill prior and performance model are Gaussian with known means and variances, multiplying their pdfs yields another known Gaussian. Hence, the posterior is compactly represented by its mean $\mu_{i,t}$, which coincides with the MAP and rating; and its variance $\sigma_{i,t}^2$, which is our \textbf{uncertainty} regarding the player's skill.

\paragraph{Logistic performance model}
When the performance model is non-Gaussian, the multiplication does not simplify so easily. By \Cref{eq:new-obj}, each round contributes an additional factor to the belief distribution. In general, we allow it to consist of a collection of simple log-concave factors, one for each round in which player $i$ has participated. Denote the participation history by
\[\cH_{i,t} := \{k\in\{1,\ldots,t\}:i\in\mathcal P_k\}.\]

Since each player can be considered in isolation, we'll omit the subscript $i$. Specializing to the logistic setting, each $k\in\cH_t$ contributes a logistic factor to the posterior, with mean $p_k$ and variance $\beta_k^2$. We still use a Gaussian initial prior, with mean and variance denoted by $p_0$ and $\beta_0^2$, respectively. Postponing the discussion of skill evolution to \Cref{sec:skill-drift}, for the moment we assume that $S_k=S_0$ for all $k$. The posterior pdf, up to normalization, is then
\begin{align}
&\pi_0(s) \prod_{k\in\cH_t} \Pr(P_k=p_k \mid S_k=s) \nonumber
\\&\propto \exp\left( -\frac{(s-p_0)^2}{2\beta_0^2} \right) \label{eq:posterior}
\prod_{k\in\cH_t} \sech^{2}\left( \frac\pi{\sqrt{12}} \frac{s-p_k} {\beta_k} \right).
\end{align}

Maximizing the posterior density amounts to minimizing its negative logarithm. Up to a constant offset, this is given by
\begin{align*}
L(s) &:= L_2\left(\frac{s-p_0}{\beta_0}\right)
+ \sum_{k\in\cH_t} L_R\left(\frac{s-p_k}{\beta_k}\right),
\\\text{where }L_2(x) &:= \frac 12 x^2\text{ and }
L_R(x) := 2\ln\left(\cosh \frac{\pi x}{\sqrt{12}}\right).
\end{align*}
\begin{equation}
\label{eq:loss}
\text{Thus, }L'(s) = \frac{s-p_0}{\beta_0^2} + \sum_{k\in\cH_t} \frac{\pi}{\beta_k\sqrt{3}} \tanh \frac{(s-p_k)\pi}{\beta_k\sqrt{12}}.
\end{equation}

$L'$ is continuous and strictly increasing in $s$, so its zero is unique: it is the MAP $\mu_t$. Similar to what we did in the first phase, we can solve for $\mu_t$ with binary search or other root-solving methods.

We pause to make an important observation. From \Cref{eq:loss}, the rating carries a rather intuitive interpretation: Gaussian factors in $L$ become $L_2$ penalty terms, whereas logistic factors take on a more interesting form as $L_R$ terms. From \Cref{fig:l2-lr-plot}, we see that the $L_R$ term behaves quadratically near the origin, but linearly at the extremities, effectively interpolating between $L_2$ and $L_1$ over a scale of magnitude $\beta_k$
%\aram{cite literature to justify this claim, and the next one? It would take more space to derive it ourselves}.

It is well-known that minimizing a sum of $L_2$ terms pushes the argument towards a weighted mean, while minimizing a sum of $L_1$ terms pushes the argument towards a weighted median. With $L_R$ terms, the net effect is that $\mu_t$ acts like a robust average of the historical performances $p_k$. Specifically, one can check that
\[\mu_t = \frac{\sum_k w_k p_k}{\sum_k w_k}, \text{ where } w_0 := \frac{1}{\beta_0^2} \text{ and }\]
\begin{equation}
\label{eq:average}
w_k := \frac{\pi}{(\mu_t-p_k)\beta_k\sqrt{3}}\tanh\frac{(\mu_t-p_k)\pi}{\beta_k\sqrt{12}} \text{ for }k\in\cH_t.
\end{equation}

$w_k$ is close to $1/\beta_k^2$ for typical performances, but can be up to $\pi^2/6$ times more as $|\mu_t-p_k| \rightarrow 0$, or vanish as $|\mu_t-p_k| \rightarrow\infty$. This feature is due to the thicker tails of the logistic distribution, as compared to the Gaussian, resulting in an algorithm that resists drastic rating changes in the presence of a few unusually good or bad performances. We'll formally state this \emph{robustness} property in \Cref{thm:robust}.

%Empirically, contest performances have indeed been seen to have thick tails, more like the logistic than the Gaussian (TODO citation).

\paragraph{Estimating skill uncertainty} While there is no easy way to compute the variance of a posterior in the form of \Cref{eq:posterior}, it will be useful to have some estimate $\sigma_t^2$ of uncertainty. There is a simple formula in the case where all factors are Gaussian. Since moment-matched logistic and normal distributions are relatively close (c.f. \Cref{fig:l2-lr-plot}), we apply the same formula:
\begin{equation}
\label{eq:variance}
\frac{1}{\sigma_t^2} := \sum_{k\in\{0\}\cup\cH_t}\frac{1}{\beta_k^2}.
\end{equation}


\subsection{Free-Marginal Multirater Kappa}
\label{chp:fundamentals:sec:inter_rater_agreement:subsec:free_marginal_multirater_kappa}

Popular kappa statistics, such as Fleiss' kappa \parencite{Fleiss:1971}, are influenced by \textit{bias} and \textit{prevalence} which can cause low kappa values despite high agreement \parencite{Randolph:2005, Sim:2005}.
Therefore their interpretation is not straightforward.
Further, Fleiss' kappa assumes that the raters know beforehand how to distribute their votes over the possible categories.
This limitation is referred to as \textit{fixed marginals} \parencite{Brennan:1981}.
According to \textcite{Sim:2005}, prevalence influences the kappa coefficient if the proportion of agreements on one attribute differs strongly to the proportion of another attribute.
Bias is defined as the degree to which raters disagree on an attribute \parencite{Sim:2005}.

To counteract those drawbacks, \textcite{Randolph:2005} introduces a \textit{Free-Marginal Multirater Kappa} $\kappa_{free}$ which does not suffer the drawbacks of prevalence and bias.
Further, his approach allows free marginals, meaning that raters are not restricted on how often they assign a subject to a specific class \parencite{Brennan:1981}.
\textcite{Randolph:2005} suggests to use his kappa metric when some marginals are not fixed.
He points out that the number of categories must be chosen carefully, since each category which is theoretically not necessary, falsely inflates the kappa's value.

\Citeauthor{Randolph:2005}'s kappa \parencite{Randolph:2005} follows the same formula as the previous kappa coefficients.
From the overall observed agreement $P_o$ the agreement merely expected by chance is subtracted, the result is then divided by the maximally adjusted chance agreement $1-P_e$.
This yields the same formula as for the previous kappa \cref{eq:Cohens_kappa,eq:fleiss_kappa,eq:Scotts_pi}:

\begin{equation}\label{eq:Randolphs_kappa}
    \kappa_{free} = \frac{P_o - P_e}{1 - P_e}
\end{equation}

It also uses the observed agreement similar to \citeauthor{Fleiss:1971}  $P_o=\bar{P}$ (\cref{eq:fleiss_P_bar}).
However, its $P_e$ is defined as

\begin{equation}\label{eq:Randolphs_Pe}
P_e = \frac{1}{C}
\end{equation}

with $C$ being the number of different classes.
When plugging the test data from \cref{tab:cohens_kappa_sample_data} into \cref{eq:Randolphs_kappa}, \citeauthor{Fleiss:1971}' observed agreement $\bar{P} = 0.7$ can be reused.
With \citeauthor{Randolph:2005}'s $P_e = 0.5$ the overall kappa yields again $\kappa_{free} = \frac{0.7 - 0.5}{1 - 0.5} = 0.4$.


% Expectations
\section{Expectations}

\subsection{Law of the Unconscious Statistician}
Given a random variable $X$ with distribution $f(x)$ and some function $r(x)$, the expected value $\mathbb{E}(r)$ is given by:

\begin{equation}
\mathbb{E}(r) = \int r(x) \mathrm{d}F_X(x)
\end{equation}

\subsection{Linearity of Expeced Value}
\begin{equation}
\mathbb{E}\left(\sum_i a_i X_i \right) = \sum_i a_i \mathbb{E}(X_i)
\end{equation}

% Moments
\section{Moments, Central Moments}

The $k$th moment is $\mathbb{E}X^k$. The $k$th central moment is $\mathbb{E}(X-\mu)^k$.

% Variance
\section{Variance}

In one dimension:
\begin{equation}
\sigma^2 =\mathbb{V}(X) = \mathbb{E}(X-\mu)^2 = \int (x-\mu)^2 \mathrm{d}F(x)
\end{equation}

The standard deviation is $\mathrm{sd}(X) = \sqrt{\mathbb{V}(X)}$.

\begin{itemize}
\item $\mathbb{V}(X) = \mathbb{E}(X^2) - \mu^2$.
\item If $a$ and $b$ are constants, $\mathbb{V}(aX+b) = a^2 \mathbb{V}(X)$.
\item If $X_1,...,X_n$ are independent and $a_1,...,a_n$ are constants, then $\mathbb{V}\left(\sum^n_{i=1}a_i X_i\right) = \sum_{i=1}^n a_i^2 \mathbb{V}(X_i)$
\end{itemize}

% Higher Dimensions
\section{Mean and Variance of Vector Valued Random Variables}
The mean is simply $\mathbb{E}(\mathbf{X}) = (...,\mathbb{E}(X_i),...,) $. In higher dimensions, the variance is a matrix with entries $\Sigma_{i,j}= \mathrm{Cov}(X_i,X_j)$.

\begin{itemize}
\item $\mathbb{E}(\mathbf{a}^T \mathbf{X}) = \mathbf{a}^T\mathbf{\mu}$
\item $\mathbb{E}(\mathbf{A}\mathbf{X}) = \mathbf{A\mu}$
\item $\mathbb{V}(\mathbf{a}^T \mathbf{X}) = \mathbf{a}^T\mathbf{\Sigma}\mathbf{a}$
\item $\mathbb{V}(\mathbf{AX}) = \mathbf{A\Sigma A}^T$
\end{itemize}

% Covariance
\section{Covariance, Correlation}
\begin{equation}
\mathrm{Cov}(X,Y) = \mathbb{E}(XY) - \mathbb{E}X\mathbb{E}Y
\end{equation}

% Sample Mean, Sample Variance
\section{Sample Mean and Sample Variance}
If $X_1,...,X_n$ are random variables, the \textit{sample mean} is:

\begin{equation}
\overline{X}_n = \frac{1}{n}\sigma_i X_i
\end{equation}

The \textit{sample variance} is:

\begin{equation}
S^2_n = \frac{1}{n-1} \sum^n_{i=1} (X_i - \overline{X}_n)^2
\end{equation}

The sample mean and sample variance are random variables. The mean and variance are fixed properties of the underlying distribution.

\begin{equation}
\begin{array}{l}
\mathbb{E}(\overline{X}_n) = \mu\\
\mathbb{V}(\overline{X}_n) = \frac{\sigma^2}{n}\\
\mathbb{E}(S_n^2) = \sigma^2\\
\end{array}
\end{equation}

\section{Law of Iterated Expectation}

Given two random variables $X$ and $Y$, the law of iterated expectation states:

\begin{equation}
\mathbb{E}(X) = \mathbb{E}\mathbb{E}(X|Y)
\end{equation}

\subsection{Applied to Variance}
\begin{equation}
\mathbb{V}(Y) = \mathbb{EV}(Y|X) + \mathbb{VE}(Y|X)
\end{equation}


\documentclass[10pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}

\newtheorem{definition}{Definition}
\newtheorem{conjecture}{Conjecture}
\newtheorem{theorem}{Theorem}

\newtheorem{algorithm}{Algorithm}

\newcommand{\htn}{\hat{\theta}_n}

\begin{document}

\section{Types of convergence}

\begin{definition}
Convergence in distribution is defined via
\begin{equation}
F_{X_n} \rightarrow F_X \quad \text{as } n \rightarrow \infty
\end{equation}
\end{definition}

\begin{definition}
	Convergence in probability is defined via
	\begin{equation}
	P(|X_n - X| > \epsilon) \quad \text{as } n \rightarrow \infty \  \forall \epsilon
	\end{equation}
\end{definition}

\begin{definition}
	Convergence almost surely is defined via
	\begin{equation}
	P(X_n \rightarrow X,  n \rightarrow \infty) = 1
	\end{equation}
\end{definition}

\textbf{Almost surely implies in probability implies in distribution!}

\begin{theorem}[Slutsky]

	Suppose $X_n \xrightarrow{d} X$ and $Y_n \xrightarrow{p} a$ where $a$ is a constant. Then
	\begin{enumerate}
		\item $X_n Y_n \xrightarrow{d} aX$
		\item $X_n + Y_n \xrightarrow{d} X+ a$
	\end{enumerate}
\end{theorem}

\section{Fundamental laws}
\begin{theorem}[Weak Law of Large Numbers]
	Let $X_1, X_2, \dots , X_n$ be i.i.d distributed random variables with a finite mean $\mu_X$.  Let $S_n = \sum_1^n X_i$.
	Then
	\begin{equation}
	\lim_{n \to \infty} P\left( \lvert \frac{S_n}{n} - \mu_X \rvert > \epsilon) \right)  = 0 \quad \forall \epsilon
	\end{equation}

\end{theorem}

\begin{theorem}[Central Limit Theorem]
	Let $X_1, X_2, \dots , X_n$ be i.i.d distributed random variables with a finite mean $\mu_X$ and finite variance $\sigma_X^2$.
	Then the sample mean $\bar X_n = \sum_1^n X_i /n$ will follow an approximately normal distribution with mean $\mu_X$, and variance $\sigma^2_X/n$. Alternatively,
	\begin{equation}
	\bar X_n \xrightarrow{d} N(\mu_X, \frac{\sigma^2_X}{n})
	\end{equation}
\end{theorem}

\section{Basics}

\begin{definition}
A \textbf{statistical model} is a pair $(E, P_\theta) \quad \theta\in \Theta$ where $E$ is the sample sapce, $P$ is a family of probability distributions and $\theta$ is the parameter set describing the distributions.
\end{definition}
\begin{itemize}
	\item When $\Theta \in R^d$ we call the model parameteric.
	\item When $\Theta$ is infinite dimensional the model is non-parametric
	\item When $\Theta = \Theta_1 \times \Theta_2$ where $\Theta_1$ is infinite dimesnional and $\Theta_2$ is finite dimensional the model is called semi-parametric.
\end{itemize}

\begin{definition}
A \textbf{statistic} is any measurabable function of the sample.  An \textbf{estimator} of $\theta$ is a statistic $\hat \theta_n =  \hat \theta_n(X_1, X_2, \dots , X_n)$ which does not depend on $\theta$.
\end{definition}

\begin{definition}
An estimator is \textbf{weekly consistent} if $\lim_{n \to \infty} \hat \theta_n = \theta$.
\end{definition}

\begin{definition}
The \textbf{bias} of an estimator is given by
\begin{equation}
\text{bias} = E[\hat \theta_n] - \theta
\end{equation}
\end{definition}

\begin{definition}
The \textbf{variance} of an estimator is given by\begin{equation}
\text{variance} = E \left[ (\hat \theta_n - E[\hat \theta_n])^2 \right]
\end{equation}
\end{definition}

\begin{definition}
The \textbf{quadratic risk} of an estimator is given by
\begin{align}
R &= E \left[ (\htn - \theta)^2 \right] = E \left[ (\hat \theta_n - E[\htn] + E[\htn] - \theta )^2  \right] \\
&=  \text{variance} + \underbrace{ E \left[ \hat \theta_n -
 E[\htn]  \right]}_{=0} (E[\htn] - \theta) + \text{bias}^2 \\
& = \text{variance} + \text{bias}^2
\end{align}
\end{definition}

\end{document}


\subsection{The Second Derivative Test}\label{sec:SecondDer}
The basis of the first derivative test is that if the derivative
changes from positive to negative at a point at which the derivative
is zero then there is a local maximum at the point, and similarly for
a local minimum. If $f'$ changes from positive to negative it is
decreasing; this means that the derivative of $f'$, $f''$, might be negative,
and if in fact $f''$ is negative then $f'$ is definitely
decreasing. From this we determine that there is a local maximum at the point in question. Note
that $f'$ might change from positive to negative while $f''$ is
zero, in which case $f''$ gives us no information about the critical
value. Similarly, if $f'$ changes from negative to positive there is a
local minimum at the point, and $f'$ is increasing. If $f''>0$ at the
point, this tells us that $f'$ is increasing, and so there is a local
minimum.

\begin{example}{Second Derivative}{secondderivative}
Consider again $f(x)=\sin x + \cos x$,  with $f'(x)=\cos x-\sin x$ and
$ f''(x)=-\sin x -\cos x$. Use the second derivative test to determine which critical points are local maximum or minima.
\end{example}
\begin{solution}
Since $\ds f''(\pi/4)=-\sqrt{2}/2-\sqrt2/2=-\sqrt2<0$,
we know there is a local maximum at $\pi/4$. Since
$\ds f''(5\pi/4)=-(-\sqrt{2}/2)-(-\sqrt2/2)=\sqrt2>0$, there is a local
minimum at $5\pi/4$.
\end{solution}

When it works, the second derivative test is often the easiest way to
identify local maximum and minimum points. Sometimes the test fails,
and sometimes the second derivative is quite difficult to evaluate; in
such cases we must fall back on one of the previous tests.

\begin{example}{Second Derivative}{secondderivativetwo}
Let $\ds f(x)=x^4$ and $\ds g(x)=-x^4$. Classify the critical points of $f(x)$ and $g(x)$ as either maximum or minimum.
\end{example}
\begin{solution}
The derivatives for $f(x)$ are $f'(x)=4x^3$ and $\ds f''(x)=12x^2$.
Zero is the only critical value, but $f''(0)=0$, so
the second derivative test tells us nothing. However, $f(x)$ is
positive everywhere except at zero, so clearly $f(x)$ has a local
minimum at zero.

On the other hand, for $g(x)=-x^4$, $g'(x)=-4x^3$ and $g''(x)=-12x^2$. So $g(x)$ also has zero
as its only only critical value, and the second derivative is again zero, but $-x^4$ has a local maximum at zero.
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:SecondDer}}

\begin{enumialphparenastyle}

Find all local maximum and minimum points by the second derivative test.

%%%%%%%%%%
\begin{ex}
 $\ds y=x^2-x$
\begin{sol}
 min at $x=1/2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=2+3x-x^3$
\begin{sol}
 min at $x=-1$, max at $x=1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^3-9x^2+24x$
\begin{sol}
 max at $x=2$, min at $x=4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^4-2x^2+3$
\begin{sol}
 min at $x=\pm 1$, max at $x=0$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^4-4x^3$
\begin{sol}
 min at $x=1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=(x^2-1)/x$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^2-(1/x^2)$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $y=\cos(2x)-x$
\begin{sol}
 min at $x=7\pi/12+n\pi$, max at $x=-\pi/12+n\pi$, for integer $n$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = 4x+\sqrt{1-x}$
\begin{sol}
 max at $x=63/64$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = (x+1)/\sqrt{5x^2 + 35}$
\begin{sol}
 max at $x=7$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y= x^5 - x$
\begin{sol}
 max at $\ds -5^{-1/4}$, min at $\ds 5^{-1/4}$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = 6x +\sin 3x$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = x+ 1/x$
\begin{sol}
 max at $-1$, min at $1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = x^2+ 1/x$
\begin{sol}
 min at $\ds 2^{-1/3}$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = (x+5)^{1/4}$
\begin{sol}
 none
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = \tan^2 x$
\begin{sol}
 min at $n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y =\cos^2 x - \sin^2 x$
\begin{sol}
 max at $n\pi$, min at $\pi/2+n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = \sin^3 x$
\begin{sol}
 max at $\pi/2+2n\pi$, min at $3\pi/2+2n\pi$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\newpage
\section[Differentiation]{\hyperlink{toc}{Differentiation}}

\subsection{Derivatives}
\begin{definition}{Derivatives}{5.1}
    Let $f: [a, b] \mapsto \RR$, and $x \in [a, b]$. We then define the \textbf{derivative} of $f$ at $x$ as:
    \begin{align*}
        f'(x) = \lim_{t\rightarrow x} \frac{f(t) - f(x)}{t - x}
    \end{align*}
    If the limit exists. Alternative notations for the derivative are given by:
    \begin{align*}
        \dpd{f}{x}(x) \text{ or } \dod{}{x}f(x) \text { or } \left.\dod{}{y}f(x)\right|_{y = x}
    \end{align*}
\end{definition}
As an interpretation of the derivative, take $[a, b]$ to be a metric space, with $x$ a limit point of $[a, b] \setminus \set{x}$. Then, $g(t) = \frac{f(t) - f(x)}{t - x}$ is a function from $[a, b] \setminus \set{x} \mapsto \RR$. If $x \in (a, b)$, then the above definition of the derivative agrees with the definition of $f'(x)$ from first year calculus. If $x = a$ or $x = b$, then the above definition agrees with the definition of the one-sided derivative from first year calculus. Note that we will not discuss in this class cases where the domain gets more complicated (i.e. not just closed intervals of $\RR$).

\begin{theorem}{}{5.2}
    Let $f:[a,b] \mapsto \RR$, let $x \in [a, b]$, and suppose $f'(x)$ exists. Then, $f$ is continuous at $x$.
\end{theorem}
\begin{nproof}
    For $t \neq x$, we can write:
    \begin{align*}
        f(t) = f(x) + (f(t) - f(x)) = f(x) + \frac{f(t) - f(x)}{t - x}(t - x)
    \end{align*}
    Taking the limit of $t \rightarrow x$, we then have that:
    \begin{align*}
        \lim_{t \rightarrow x} f(t) = \lim_{t \rightarrow x} \left(f(x) + \frac{f(t) - f(x)}{t - x}(t - x) \right) = \lim_{t \rightarrow x} f(x) + \lim_{t \rightarrow x} \frac{f(t) - f(x)}{t - x} \lim_{t \rightarrow x} (t - x)
    \end{align*}
    Where in the last line we invoke Theorem \ref{thm:4.4}. Evaluating the limits on the RHS by using the existence of the derivative of $f$ at $x$, we have
    \begin{align*}
        \lim_{t \rightarrow x} f(t) = f(x) + f'(x)\cdot (0) = f(x)
    \end{align*}
    So we conclude that $f$ is continuous at $x$ by Theorem \ref{thm:4.6}. \qed
\end{nproof}
The interpretation is that differentiability at $x \in (a, b)$ implies continuity of $f$ at $x$, and the left/right differentiability of $f$ at $a/b$ implies the left/right continuity of $f$ at $a/b$. We have wrapped the proof of all these cases into one!

Note that the converse of the above theorem is not true. As a simple example, take $f(x) = \abs{x}$ on $[-1, 1]$, which is continuous at $x = 0$ (it can be verified that $\lim_{x \rightarrow 0}f(x) = f(0) = 0$) but is not differentiable there (the left/right handed limits of the difference quotient do not agree and hence the derivative does not exist). In Chapter 7, we will construct a function that is continuous everywhere and differentiable nowhere!

NWe will now proceed to prove a series of theorems that have been seen in first year, but using our new/rigorous definitions.

\begin{theorem}{Sum, Product, and Quotient Rules}{5.3}
    Let $f, g: [a, b] \mapsto \RR$. Let $x \in [a, b]$ and suppose $f$ and $g$ are differentiable at $x$. Then, $f + g$, $f - g$, $f\cdot g$ are differentiable at $x$, and so is $\frac{f}{g}$ provided $g(x) \neq 0$. Furthermore:
    \begin{enumerate}
        \item $(f+g)'(x) = f'(x) + g'(x)$
        \item $(fg)'(x) = f'(x)g(x) + f(x)g'(x)$
        \item $\left(\frac{f}{g}\right)'(x) = \frac{f'(x)g(x) - f(x)g'(x)}{(g(x))^2}$
    \end{enumerate}
\end{theorem}
\begin{nproof}
    \begin{enumerate}
        \item Follows immediately from the additive property of limits (Theorem \ref{thm:4.4}).
        \item Let $h = fg$. We then have that:
        \begin{align*}
            h(t) - h(x) = f(t)\left[g(t) - g(x)\right] + g(x)\left[f(t) - f(x)\right]
        \end{align*}
        For $t \neq x$, we can divide both sides by $t-x$ to obtain:
        \begin{align*}
            \frac{h(t) - h(x)}{t - x} = f(t)\frac{g(t) - g(x)}{t-x} + g(x)\frac{f(t) - f(x)}{t - x}
        \end{align*}
        Taking the limit of $t \rightarrow x$ on both sides, we obtain:
        \begin{align*}
            h'(x) = f(x)g'(x) + f'(x)g(x)
        \end{align*}
        as desired.
        \item Let $h(t) = \frac{f(t)}{g(t)}$. Then:
        \begin{align*}
            h(t) - h(x) &= \frac{f(t)}{g(t)} - \frac{f(x)}{g(x)}
            \\ &= \frac{1}{g(t)g(x)}\left(f(t)g(x) - g(t)f(x)\right)
            \\ &= \frac{1}{g(t)g(x)}\left[g(x)\left(f(t) - f(x)\right) - f(x)\left(g(t) - g(x)\right)\right]
        \end{align*}
        For $t \neq x$, we can divide both sides by $t - x$ to get:
        \begin{align*}
            \frac{h(t) - h(x)}{t - x} = \frac{1}{g(t)g(x)}\left[g(t)\frac{f(t) - f(x)}{t - x} - f(x)\frac{g(t) - g(x)}{t - x}\right]
        \end{align*}
        Taking the limit as $t \rightarrow x$ on both sides, we obtain the desired expression. \qed
    \end{enumerate}
\end{nproof}

\newpage
\noindent As an exercise, one can prove by induction (applying 5.3(b)) that $(f_1f_2f_3\ldots f_n)'(x)$ (where $f_i: [a, b] \mapsto \RR$ and each $f_i'(x)$ exists) is given by:
\begin{align*}
    f_1'(x)f_2(x)\ldots f_n(x) + \cdots + f_1(x)f_2(x)\ldots f'_n(x).
\end{align*}
Note that as a corollary of this, we get that if $f(x) = x^n$, then $f'(x) = nx^{n-1}$ and we hence recover the familiar power rule from first year calculus!

\setcounter{rudin}{4}
\begin{theorem}{Chain Rule}{5.5}
    Let $f: [a, b] \mapsto \RR$, $x \in [a, b]$, and suppose $f$ is differentiable at $x$. Suppose furthermore that $f([a, b])$ is contained in some interval $I$. Let $g: I \mapsto \RR$ and suppose $g$ is differentiable at $f(x)$. Then, $g \circ f: [a, b] \mapsto \RR$ is differentiable at $x$, and furthermore:
    \begin{align*}
        (g \circ f)'(x) = g'(f(x))f'(x)
    \end{align*}
\end{theorem}
\begin{nproof}
    Define $h(t) = g \circ f(t)$ for $a \leq t \leq b$, $t \neq x$. We cna then write:
    \begin{align*}
        f(t) - f(x) = (t-x)\left[f'(x) + u(t)\right]
    \end{align*}
    For a function $u(t)$ with $\lim_{t \rightarrow x} u(t) = 0$. Now defining $y = f(x)$, we write:
    \begin{align*}
        g(s) - g(y) = (s-y)\left[g'(y) + r(s)\right]
    \end{align*}
    For a function $r(s)$ with $\lim_{s \rightarrow y}r(s) = 0$. Hence, we have that:
    \begin{align*}
        h(t) - h(x) &= g(f(t)) - g(f(x))
        \\ &= \left(f(t) - f(x)\right)\left(g'(y) + r(s)\right)
        \\ &= (t- x)\left[f'(x) + u(t)\right]\left(g'(y) + r(s)\right)
    \end{align*}
    Dividing both sides by $t - x$, we obtain:
    \begin{align*}
        \frac{h(t) - h(x)}{t - x} = \left[f'(x) + u(t)\right]\left(g'(y) + r(s)\right)
    \end{align*}
    We now take the limit of $t \rightarrow x$ on both sides. $\lim_{t \rightarrow x} u(t) = 0$, and $f$ is differentiable and hence continuous at $x$, so $s = f(t) \rightarrow y$ as $t \rightarrow x$. Thus, $r(s) \rightarrow 0$ as $t \rightarrow x$, and in conclusion:
    \begin{align*}
        h'(x) = (g \circ f)'(x) =  f'(x)g'(y) = g'(f(x))f'(x)
    \end{align*}
    as desired. \qed
\end{nproof}

\subsection{MVT}

\setcounter{rudin}{6}
\begin{definition}{Local Maxima/Minima}{5.7}
    Let $X$ be a metric space. Let $f: X \mapsto \RR$, and let $x \in X$. We say that $x$ is a \textbf{local maximum} of $f$ if there exists $\delta > 0$ such that $f(y) \leq f(x)$ for all $y \in N_{\delta}(x)$. A \textbf{local minimum} is defined similarly, with $f(y) \geq f(x)$ instead.
\end{definition}
\noindent For a metric space $X$ equipped with the discrete metric, all points $x \in X$ are simultaneously local maxima and minima. To see this, take any $0 < \delta \leq 1$.

\begin{theorem}{}{5.8}
    Let $f: [a, b] \mapsto \RR$. Let $x \in [a, b]$ and suppose that $f'(x)$ exists, and $f$ is either a local maximum or local minimum of $f$. Then, $f'(x) = 0$.
\end{theorem}
\begin{nproof}
    Suppose $x$ is a local minimum. Then, there exists $\delta > 0$ such that $N_{\delta}(x) \subset [a, b]$, and $f(y) \geq f(x)$ for all $y \in N_{\delta}(x)$. Thus, if $x < y < x + \delta$, then:
    \begin{align*}
        \frac{f(y) - f(x)}{y - x} \geq 0 \implies f'(x) \geq 0
    \end{align*}
    Conversely, if $x - \delta < y < x$, then:
    \begin{align*}
        \frac{f(y) - f(x)}{y - x} \leq 0 \implies f'(x) \leq 0
    \end{align*}
    So taken together we obtain that $f'(x) = 0$. An identical argument is used for the case of a local maximum. \qed
\end{nproof}

\begin{ntheorem}{: Rolle's Theorem}{}
    Let $f: [a, b] \mapsto \RR$ be continuous, and suppose $f$ is differentiable on $(a, b)$. If $f(a) = f(b)$, then there exists $x \in (a, b)$ such that $f'(x) = 0$.
\end{ntheorem}
\begin{nproof}
    Since $[a, b]$ is compact and $f$ is continuous, by the EVT (Theorem \ref{thm:4.16}) $f$ attains its maximum on $[a, b]$, that is, there exists $c \in [a, b]$ such that $f(y) \leq f(x)$ for all $y \in [a, b]$. If $c \in (a, b)$, then by Theorem \ref{thm:5.8}, $f'(c) = 0$ and we are done. Next, suppose $c = a$ or $c = b$. Again by the EVT, $f$ attains its minumum on $[a, b]$, that is, there exists $d \in [a, b]$ such that $f(y) \geq f(d)$ for all $y \in [a, b]$. If $d \in (a, b)$, then by Theorem \ref{thm:5.8}, $f'(d) = 0$ and we are done. Suppose then that $d = a$ or $d = b$. Since $f(a) = f(b)$, we therefore obtain that $f(a) = f(b) = f(c) = f(d)$ and the maximum/minimum values agree. Hence, $f(y) = f(a)$ for all $y \in [a, b]$, so $f'(y) = 0$ for all $y \in [a, b]$. So, the desired $x$ may be any point in $[a, b]$. \qed
\end{nproof}

\begin{figure}[htbp]
    \centering
    \begin{tikzpicture}[scale = 2]
        \draw[-latex, very thick] (0, 0) -- (0, 2);
        \draw[-latex, very thick] (0, 0) -- (2, 0);
        \draw[] (1, 1.5) parabola (0.5, 0.2);
        \draw[] (1, 1.5) parabola (1.5, 0.2);
        \filldraw[] (0.5, 0.2) circle (1pt);
        \filldraw[] (1.5, 0.2) circle (1pt);
        \draw[dashed, red] (0, 1.51) -- (1.85, 1.51);
        \draw[] (0.5, 0) -- (0.5, -0.15);
        \node[below] at (0.5, -0.15) {$a$};
        \draw[] (1.5, 0) -- (1.5, -0.15);
        \node[below] at (1.5, -0.12) {$b$};
        \draw[] (1, 0) -- (1, -0.15);
        \node[below] at (1, -0.16) {$x$};
    \end{tikzpicture}

    \caption{A simple parabolic function that demonstrates Rolle's Theorem.}
    \label{fig21}
\end{figure}

\newpage
\setcounter{rudin}{9}
\begin{theorem}{Mean Value Theorem}{5.10}
    Let $f: [a, b] \mapsto \RR$ be continuosu and differentiable on $(a, b)$. Then, there exists $x \in (a, b)$ such that $f(b) - f(a) = f'(x)(b - a)$.
\end{theorem}
\noindent The visual interpretation of this theorem is that there exists $x \in (a, b)$ such that the slope of the tangent line to $f$ at $x$ is equal to the secant line slope between $(a, f(a))$ and $(b, f(b))$. The idea of the proof is to rotate one's head such that the sectant line is horizontal; one is then able to apply Rolle's Theorem!

\begin{nproof}
    Define $h(y) = f(y) = \frac{f(b) - f(a)}{b - a}(y - a)$. $h$ is continuous on $[a, b]$ and differentiable on $(a, b)$ (being a sum of continuous/differentiable functions). We have that $h(a) = f(a) - 0 = f(a)$, and $h(b) = f(b) - \frac{f(b) - f(a)}{b - a}(b - a) = f(a)$. Applying Rolle's Theorem to $h$, there exists $x \in (a, b)$ such that $h'(x) = 0$. Therefore, $h'(x) = 0 = f'(x) - \frac{f(a) - f(b)}{b - a} = 0$, and we conclude that $f(b) - f(a) = f'(x)(b - a)$ for some $x \in (a, b)$. \qed
\end{nproof}

\begin{figure}[htbp]
    \centering
    \begin{tikzpicture}[scale=2]
    \draw[-latex, very thick] (0, 0) -- (0, 2);
    \draw[-latex, very thick] (0, 0) -- (2, 0);
    \draw [] (0.5 , 0.25) to [ curve through ={(1, 1.25)}] (1.5,1.5);
    \filldraw[] (0.5, 0.25) circle (1pt);
    \filldraw[] (1.5, 1.5) circle (1pt);
    \draw[dashed] (0.5, 0.25) -- (1.5, 1.5);
    \draw[dashed, red] (0, 0.075) -- (1, 1.325) -- (1.42,1.85);
    \draw[] (0.5, 0) -- (0.5, -0.15);
    \node[below] at (0.5, -0.15) {$a$};
    \draw[] (1.5, 0) -- (1.5, -0.15);
    \node[below] at (1.5, -0.12) {$b$};
    \draw[] (0.775, 0) -- (0.775, -0.15);
    \node[below] at (0.775, -0.16) {$x$};
    \end{tikzpicture}
    \caption{A simple continuous function that demonstrates the MVT.}
    \label{fig22}
\end{figure}

\begin{theorem}{}{5.11}
    Let $f: [a, b] \mapsto \RR$ be differentiable on $(a, b)$. Then:
    \begin{enumerate}
        \item If $f'(x) \geq 0$ for all $x \in (a, b)$, then $f$ is monotonically increasing.
        \item If $f'(x) = 0$ for all $x \in (a, b)$, then $f$ is constant.
        \item If $f'(x) \leq 0$ for all $x \in (a, b)$, then $f$ is monotonically decreasing.
    \end{enumerate}
\end{theorem}
\begin{nproof}
    If $a < x < y < b$, by the mean value theorem, there exists $z \in (x, y)$ such that:
    \begin{align*}
        f(y) - f(x) = f'(z)(y - x)
    \end{align*}
    Note that $y - x > 0$ by construction.
    \begin{enumerate}
        \item If $f'(x) \geq 0$ for all $x \in (a, b)$, then $f'(z) \geq 0$, showing that $f(y) - f(x) \geq 0$ and hence that $f$ is monotonically increasing.
        \item If $f'(x) = 0$ for all $x \in (a, b)$, then $f'(z) = 0$, showing that $f(y) - f(x) = 0$ and hence that $f$ is constant on $(a, b)$.
        \item If $f'(x) \leq 0$ for all $x \in (a, b)$, then $f'(z) \leq 0$, showing that $f(y) - f(x) \leq 0$ and hence that $f$ is monotonically decreasing. \qed
    \end{enumerate}
\end{nproof}

\subsection{Taylor's Theorem}
\setcounter{rudin}{13}
\begin{definition}{Higher Order Derivatives}{5.14}
If $f$ is differentiable in a neighbourhood of $x$, then we may compute a second order derivative:
\begin{align*}
    \lim_{t \rightarrow x} \frac{f'(t) - f'(x)}{t - x} = (f')'(x) = f''(x)
\end{align*}
We can then continue this process to obtain $f^{(3)}(x), f^{(4)}(x), \ldots, f^{(n)}(x)$.
\end{definition}
\begin{ndef}{: \texorpdfstring{$C^n(I, \RR)$}{Cn(I, R)}}{}
    If $f$ is continuous in $I$, we can write $f \in C^{0}(I, \RR)$. If $f$ is differentiable in a neighbourhood $I$ and the derivative $f'$ is continuous in $I$, then we write $f \in C^{1}(I, \RR)$. In general, $f \in C^{n}(I, \RR)$ denotes the $n$th derivative of $f$ is continuous in $I$. Note that where it is clear from context, we may drop the $\RR$ and just write $C^n(I)$.
\end{ndef}
\noindent Recall that for a function $f$ continuous and differentiable on $(x_0, x)$, the Mean Value Theorem proved the existence of some $\tilde{x}$ such that:
\begin{align*}
    f(x) = f(x_0) + f'(\tilde{x})(x - x_0)
\end{align*}
This gives us a natural method to build up approximations for functions; we can start with a constant approximation $f(x_0)$, then add a linear term $f'(\tilde{x})(x - x_0)$, then add on a quadratic term $(x - x_0)^2$ and so on. The following theorem gives us a way to construct these approximations and bound their error.

\begin{theorem}{Taylor's Theorem}{5.15}
    Let $I$ be a neighbourhood of $x_0$, and $f \in C^p(I)$. Then, for any $n < p$, we have that:
    \begin{align*}
        f(x) = \sum_{j=0}^n \frac{f^{(j)}(x_0)}{j!}(x- x_0)^j + \frac{f^{(n+1)}(\tilde{x})}{(n+1)!}(x - x_0)^{n+1}
    \end{align*}
    Where $\tilde{x} = x_0 + \lambda(x - x_0)$ for some $\lambda \in (0, 1)$ (i.e. $\tilde{x} \in (x_0, x)$). Note that $\tilde{x}$ depends on $x, x_0$, and $n$.
\end{theorem}
\begin{nproof}
    For $n = 0$, the claim reduces to the Mean Value Theorem. Then, let $n \geq 1$. Let $A$ be a constant that depends on $x, x_0$, and $n$, and let $P_n(x) = \sum_{j=0}^n \frac{f^{(j)}(x_0)}{j!}(x-x_0)^j$. Then, we can write:
    \begin{align*}
        f(x) = P_n(x) + A(x-x_0)^{n+1}
    \end{align*}
    We need to show that we can express $A$ as relating to the derivative, namely, that there exists $\tilde{x}$ such that $A = \frac{f^{(n+1)}(\tilde{x})}{(n+1)!}$. Let $g(t) = f(t) - P_n(t) - A(t - x_0)^{n+1}$ with $t \in I$. Then, $g \in C^{p}(I)$. For $n < p$, we then have that:
    \begin{align*}
        g^{(n+1)}(t) = f^{(n+1)}(t) - 0 - A(n+1)!(t - x_0)
    \end{align*}
    We claim that there exists $\tilde{x} \in (x_0, x)$ such that $g^{(n+1)}(\tilde{x}) = 0$. To see this, consider that $P^{(j)}(x_0) = f^{(j)}(x_0)$ for $j = 0, 1, \ldots, n$, so $g(x_0) = 0$, and furthermore:
    \begin{align*}
        g'(x_0) = g''(x_0) = g^{(3)}(x_0) = \ldots = g^{(n)}(x_0) = 0
    \end{align*}
    Moreover y the choice of $n$, we have that $g(x) = 0$. Hence, by Rolle's Theorem, there exists a point $x_1$ between $x_0$ and $x$ such that $g'(x_1) = 0$. Similarly, repeating the argument above, there exists an $x_2$ between $x_0$ and $x_1$ such that $g''(x_2) = 0$. Repeating this process up to $g^{(n)}$, we have that $g^{(n)}(x_n) = 0$, for some $x_0 < x_{n} < x_{n-1} < \cdots < x$ and in turn, there exists $x_{n+1} \in (x_0, x_n)$ such that $g^{(n+1)}(x_{n+1}) = 0$. Setting $\tilde{x} = x_{n+1}$, the claim is shown. \qed
\end{nproof}

\noindent As an example, we consider the Taylor series of the function $f(x) = \cos(x)$ (We will formally define this function later on, but for now, let us assume its familar properties and derivatives). We then have that:
\begin{align*}
    f^{(j)}(0) = \begin{cases}
        (-1)^m & \text{if $j = 2m$}
        \\ 0 & \text{if $j = 2m + 1$}
    \end{cases}
\end{align*}
If we have the sum run from $j = 0$ to some $j = n$, let us then try to estimate the rest. Let $\tilde{x} \in (0, x)$. Then, letting the error term be represented by $\e$, we have that:
\begin{align*}
    \e = \abs{\frac{f^{(n+1)}(\tilde{x})}{(n+1)!}(x-x_0)^{n+1}} \leq \frac{\abs{x}^{n+1}}{(n+1)!}
\end{align*}
And we observe that $\linf \frac{\abs{x}^{n+1}}{(n+1)!} = 0$ and hence the error $\e$ goes to zero in the $n \rightarrow \infty$ limit. Therefore, the difference between $\cos(x)$ and its Taylor polynomial vanishes quickly for any $x$, and the Taylor series converges for all $x$. Taking the limit of the sum, we have that:
\begin{align*}
    f(x) = \cos(x) = \linf \left(P_{n}(x) + \frac{f^{(n+1)}(x_0)}{(n+1)!}(x - x_0)^{n+1}\right) = \sum_{j=0}^\infty \frac{(-1)^{2m}}{m!}x^{2m} = 1 - \frac{x^2}{2} + \frac{x^4}{4} - \frac{x^6}{6} + \ldots
\end{align*}
A question of interest might be how many terms do we need in the Polynomial such that our error is less than $10^{-6}$, say, for estimating the value of $\cos(\frac{\pi}{12})$. In other words, we want to find the $m$ such that:
\begin{align*}
    \frac{1}{2m!}\left(\frac{\pi}{12}\right)^{2m} \leq 10^{-6}
\end{align*}
Rearranging, we require:
\begin{align*}
    (2m)!\left(\frac{12}{\pi}\right)^{2m} > 10^6
\end{align*}
Making a table of the value of the LHS as a function of $m$, we have:
\begin{table}[htbp]
    \centering\begin{tabular}{c|c}
    $m$ & $(2m)!\left(\frac{12}{\pi}\right)^{2m}$
    \\ \hline
    1 & $\approx 29$
    \\ 2 & $\approx 5110$
    \\ 3 & $\approx 2.23 \times 10^{6}$
    \end{tabular}
\end{table}
So we see that three terms are sufficient for a good approximation in this case (and as stated before, the series converges very quickly)!

A natural question of interest is the convergence of the sum in the $N \rightarrow \infty$ limit, that is, the convergence of the power series $\sum_{n = 0}^\infty \frac{f^{(n)}(x_0)}{n!}(x - x_0)^n$. We are also interested are interested for when $f(x) = \sum_{n = 0}^\infty \frac{f^{(n)}(x_0)}{n!}(x - x_0)^n$ holds (that is, when is a function equal to its Taylor series)? These turn out to be distinct questions; in particular, there are functions whose power series converge for all $x$ but are equal to their power series nowhere (except at $x_0$ where equality must hold). This motivates the following definition:
\begin{ndef}{: Analyticity}{}
    A function $f$ is \textbf{analytic} if $f(x) = \sum_{n=0}^\infty \frac{f^{(n)}(x_0)}{n!}(x - x_0)^n$ in a neighbourhood of $x_0$.
\end{ndef}
\noindent To motivate this definition, it will help to study a function which is \textit{not} analytic. Consider the function:
\begin{align*}
    f(x) = \begin{cases}
        \exp(-\frac{1}{x}) & x > 0
        \\ 0 & x \leq 0
    \end{cases}
\end{align*}
\begin{figure}[htbp]
    \centering
    \begin{tikzpicture}
        \draw[latex-latex, very thick] (-2, 0) -- (2, 0);
        \draw[-latex, very thick] (0, 0) -- (0, 2);
        \draw[<-, blue, thick] (-1.5, 0) -- (0.02, 0);
        \draw[->, blue, thick, smooth, samples = 100, domain=0.01:1.5, variable = \x] plot(\x, {exp(-1/\x)});
    \end{tikzpicture}
    \caption{Plot of $f$.}
    \label{fig23}
\end{figure}

\noindent $f$ is continuous everywhere by construction. It is infinitely differentiable at $x = 0$, but it is only equal to its Taylor series around $x_0 =0$ for $x \leq 0$. To see this, we observe that:

\begin{align*}
    f'(x) = \frac{1}{x^2}\exp(-\frac{1}{x})
\end{align*}
\begin{align*}
    f^{(n)}(x) = \frac{P_n(x)}{x^{2n}}\exp(-\frac{1}{x})
\end{align*}
We have that $f^{(n)}(x) \rightarrow 0$ as $x \rightarrow 0$ for all $n$ as the exponential dominates the polynomial singularity. We hence have that $f \in C^\infty$. The taylor polynomial at $x_0 = 0$ however, as we have that $f(0) = 0$ and $f^{(n)}(0) = 0$, leading to:
\begin{align*}
    \sum_{n=0}^N \frac{f^{(n)}(0)}{n!}x^n = 0
\end{align*}
for all $N$. We therefore have that the series converges for all $x$, but is only equal to $f$ for $x \leq 0$; hence it is not analytic (as there exists no neighbourhood around $x_0= 0$ for which $f$ is equal to its Taylor series).

We now consider a function $\chi(x)$ defined as $\chi(x) = \frac{f(x)}{f(x) + f(1 - x)}$. This function is alsoz continuous, its denominator is never zero, and $\chi \in C^\infty$. We observe that $\chi(x) = 0$ for $x \leq 0$ and $\chi(x) = 1$ for $x \geq 1$, and overall the function looks much like a step function:

\begin{figure}[htbp]
    \centering
    \begin{tikzpicture}
        \draw[latex-latex, very thick] (-2, 0) -- (2, 0);
        \draw[-latex, very thick] (0, 0) -- (0, 2);
        \draw[<-, blue, thick] (-1.5, 0) -- (0.02, 0);
        \draw[blue, smooth, samples = 100, domain=0.01:0.99, variable = \x, thick] plot(\x, {exp(-1/\x)/(exp(-1/\x) + exp(-1/(1-\x)))});
        \draw[blue, ->, thick] (0.98, 1) -- (1.5, 1);
    \end{tikzpicture}
    \caption{Plot of $\chi$.}
    \label{fig24}
\end{figure}
Indeed, this function can be used as a ``cutoff''/``switch'' function that behaves much like a step function (except it is infinitely differentiable).

A question becomes whether such a function could be analytic. The answer turns out to be no, and the proof we leave as an exercise. As a sketch, consider an analytic function $g$ such that $g(x) = 0$ for $x \leq x_0$ and $g(x) \neq 0$ for $x > x_0$. One can derive a contradiction by considering the Taylor series expansion about $x_0$ and then using the assumed analyticity of $g$.

Note that in a sense, Taylor's Theorem is the culmination of a sequence of theorems we have proven in the course. Roughly, the sequence was as follows:
\begin{enumerate}[1)]
    \item $f: X \mapsto Y$ and $K \subset X$, then $f(K)$ compact (Theorem \ref{thm:4.14})
    \item Extreme Value Theorem: If $f: X \mapsto \RR$ with $f$ continu$(X, \tau)$ and $(Y, \rho)$ ous and $K$ compact, then $f$ realizes its supremum and infimum on $K$. (Theorem \ref{thm:4.16})

    \item Rolle's Theorem
    \item Mean Value Theorem (Theorem \ref{thm:5.10})
    \item Taylor's Theorem (Theorem \ref{thm:5.15})
\end{enumerate}
A question that arises is could we have gone through this sequence of proofs with just the rational numbers ($\QQ$)? The intuitive answer is no, but it may be interesting to see where along this chain the logic breaks down.

The first step that looks immediately questionable is step 2; the supremum/infimum is not well-defined for all subsets of $\QQ$, so we might be able to find a breakdown there. To this end, we consider the set $A = \set{q \in \QQ: q > 0, q^2 < 2}$ that arises in Example \ref{exam:1.1a} and try to find a function $f: K \mapsto \QQ$ with $K \subset \QQ$ compact such that $f(K) = A$. An idea would be to try $f(x) = \sqrt{x}$, with $K = \set{q \in \QQ: \exists r: r^2 = q} \cap [0, 2]$ but this doesn't work as $K$ is not compact. Trying another attempt, $K = [1, 2] \cap \QQ$ with $f(x) = \sin(x)$ does not work either as $\sin(x)$ is not necessarily rational, and moreover, $[1, 3] \cap \QQ$ is not a compact set as not all Cauchy sequences in $S$ converge! Another attempt would be $K = [1, 2] \cap \QQ = X$ with $f(q) = \abs{q^2 - 2}$ where it would seem as though $f(K) = \set{r \in \QQ: 0 < r \leq 2}$ provides a good counterexample, but this fails for the same reason as $K$ is not compact. Finding a valid counterexample for the EVT is therefore difficult.

An easier break along the chain to find is with Rolle's Theorem. One can consider the function $f(x) = x^2 - 2$ which can break the Intermediate value theorem as $0$ is not contained in the image if the domain is $\QQ$. We can dress this up to construct a counterexample for Rolle's Theorem.

\subsection{Local Behavior of Functions}
\begin{ntheorem}{: Second Derivative Test}{}
    Suppose $f \in C^{3}(I)$ where $I$ is a neighbourhood of $x_0$. Furthermore, suppose that $f'(x_0) = 0$. If $f''(x_0) > 0$, then $x_0$ is a local minimum. Conversely, if $f''(x_0) < 0$, then $x_0$ is a local maximum.
\end{ntheorem}
\begin{nproof}
    By Taylor's Theorem (Theorem \ref{thm:5.15}), if we let $x = x_0 + h$ for $h > 0$, there exists some $\tilde{x} = x_0 + \lambda h$ with $\lambda \in (0, 1)$ such that:
    \begin{align*}
        f(x) = f(x_0) + f'(x_0)(x - x_0) + \frac{1}{2}f''(x_0)(x - x_0)^2 + \frac{1}{6}f^{(3)}(\tilde{x})(x - x_0)^3
    \end{align*}
    Using that $f'(x_0) = 0$, we have:
    \begin{align*}
        f(x) - f(x_0) = h^2\left(\frac{1}{2}f''(x_0) + \frac{1}{6}f^{(3)}(x_0 + \lambda h) h\right)
    \end{align*}
    Let $0 < \e < \abs{\frac{1}{2}f''(x_0)}$. Then, by the assumed continuity of $f^(3)$, we have that there exists $\delta > 0$ such that $\abs{h} < \delta$ implies $\abs{\frac{1}{6}f^{(3)}(x_0 + \lambda h)h} < \e$. Hence, for sufficiently small $h$, we have that:
    \begin{align*}
        \sgn\left(f(x) - f(x_0)\right) = \sgn(f''(x_0))
    \end{align*}
    So we conclude that if $f'(x_0) = 0$ and $f''(x_0) > 0$ then $x_0$ is a local minimum, and if $f''(x_0) < 0$, then $x_0$ is a local maximum. \qed
\end{nproof}

\begin{ndef}{: Convex Functions}{}
    Let $f: (a, b) \mapsto \RR$ is \textbf{convex} if for all $x, y \in (a, b)$ with $a < x < y < b$ and for all $\lambda \in (0, 1)$,
    \begin{align*}
        f(\lambda x + (1 - \lambda)y) \leq \lambda f(x) + (1 - \lambda)f(y)
    \end{align*}
\end{ndef}
\noindent Note that an alternative definition of convexity is that for any $x, y$ with $x < y$, the function evaluated at some point $z \in (x, y)$ will always be below the average of the function at $x, y$. That is:
\begin{align*}
    f(z) \leq \frac{f(x) + f(y)}{2}
\end{align*}
\begin{figure}[htbp]
    \centering
    \begin{tikzpicture}[scale=2]
        \draw[-latex, very thick] (-2.5, 0) -- (-2.5, 2);
        \draw[-latex, very thick] (-2.5, 0) -- (-0.5, 0);
        \draw[] (-1.5, 0.5) parabola (-2, 1.5);
        \draw[] (-1.5, 0.5) parabola (-1, 1.5);
        \filldraw[] (-2, 1.5) circle (1pt);
        \filldraw[] (-1, 1.5) circle (1pt);
        \draw[] (-2, 0) -- (-2, -0.15);
        \node[below] at (-2, -0.15) {$a$};
        \draw[] (-1, 0) -- (-1, -0.15);
        \node[below] at (-1, -0.13) {$b$};
        \draw[] (-1.7, 0) -- (-1.7, -0.15);
        \node[below] at (-1.7, -0.16) {$x$};
        \draw[] (-1.15, 0) -- (-1.15, -0.15);
        \node[below] at (-1.15, -0.15) {$y$};
        \filldraw[] (-1.7, 0.66) circle (0.5pt);
        %\node[left] at (-2.5, 0.66) {$f(x)$};
        \filldraw[] (-1.15, 1) circle (0.5pt);
        %\node[left] at (-2.5, 1) {$f(y)$};
        \draw[dashed] (-1.7, 0.66) -- (-1.15, 1);
        \draw[-latex, very thick] (0.5, 0) -- (0.5, 2);
        \draw[-latex, very thick] (0.5, 0) -- (2.5, 0);
        \draw[] (1.5, 1.5) parabola (1, 0.5);
        \draw[] (1.5, 1.5) parabola (2, 0.5);
        \filldraw[] (1, 0.5) circle (1pt);
        \filldraw[] (2, 0.5) circle (1pt);
        \draw[] (1, 0) -- (1, -0.15);
        \node[below] at (1, -0.16) {$a$};
        \draw[] (2, 0) -- (2, -0.15);
        \node[below] at (2, -0.13) {$b$};
        \draw[] (1.3, 0) -- (1.3, -0.15);
        \node[below] at (1.3, -0.16) {$x$};
        \draw[] (1.85, 0) -- (1.85, -0.15);
        \node[below] at (1.85, -0.15) {$y$};
        \filldraw[] (1.3, 1.34) circle (0.5pt);
        \filldraw[] (1.85, 1) circle (0.5pt);
        \draw[dashed] (1.3, 1.34) -- (1.85, 1);
    \end{tikzpicture}
    \caption{Visualization of a convex and non-convex function. For the upwards facing parabola, the function (and hence all points $(\lambda x + (1- \lambda) y, f(\lambda x + (1- \lambda) y))$ for $\lambda \in (0, 1)$) always lies below the line connecting $(x, f(x))$ and $(y, f(y))$ for $a < x < y < b$ and hence it is convex. For the downwards facing parabola, this is no longer true and the function is not convex (it is instead\textit{concave}).}
    \label{fig25}
\end{figure}

\noindent Another equivalent way of defining convexity is to say that $f$ is convex if and only if for all $a < s < t < y < b$:
\begin{align*}
    \frac{f(t) - f(s)}{t - s} \leq \frac{f(u) - f(t)}{u - t}
\end{align*}

\begin{figure}[htbp]
    \centering
    \begin{tikzpicture}[scale=2]
        \draw[-latex, very thick] (0, 0) -- (0, 2);
        \draw[-latex, very thick] (0, 0) -- (2, 0);
        \draw[] (1, 0.5) parabola (0.5, 1.5);
        \draw[] (1, 0.5) parabola (1.5, 1.5);
        \filldraw[] (0.5, 1.5) circle (1pt);
        \filldraw[] (1.5, 1.5) circle (1pt);
        \draw[] (0.5, 0) -- (0.5, -0.15);
        \node[below] at (0.5, -0.16) {$a$};
        \draw[] (1.5, 0) -- (1.5, -0.15);
        \node[below] at (1.5, -0.13) {$b$};
        \draw[] (0.75, 0) -- (0.75, -0.15);
        \node[below] at (0.75, -0.16) {$s$};
        \draw[] (1.25, 0) -- (1.25, -0.15);
        \node[below] at (1.25, -0.17) {$u$};
        \draw[] (1, 0) -- (1, -0.15);
        \node[below] at (1, -0.14) {$t$};
        \filldraw[] (1, 0.5) circle (0.5pt);
        \filldraw[] (0.75, 0.755) circle (0.5pt);
        \filldraw[] (1.25, 0.755) circle (0.5pt);
        \draw[dashed] (0.75, 0.755) -- (1, 0.5);
        \draw[dashed] (1.25, 0.755) -- (1, 0.5);
    \end{tikzpicture}

    \caption{Visualization of the alternative definition of convexity. For any $a < s < t < u < v$, the slope of the line segment joining $s$ and $t$ is less than the slope of the line segment joining $t$ and $u$.}
    \label{fig26}
\end{figure}

\begin{ntheorem}{}{}
    \begin{enumerate}[(a)]
        \item Assume that $f: (a, b) \mapsto \RR$ is convex. Then, $f$ is continuous.
        \item Assume that $f \in C^1(a, b)$. Then, if $f$ is convex, $f'$ is increasing.
        \item Assume that $f \in C^2(a, b)$. Then $f$ convex implies $f'' \geq 0$.
    \end{enumerate}
\end{ntheorem}
\begin{nproof}
    \begin{enumerate}
        \item Let $[c, d] \subset (a, b)$ and $a < c_1 < c < x < y < d < d_1 < b$. By convexity, we have that:
        \begin{align*}
            \frac{f(y) - f(x)}{y - x} \leq \frac{f(d) - f(y)}{d - y} \leq \frac{f(d_1) - f(d)}{d_1 - d}
        \end{align*}
        and also that:
        \begin{align*}
            \frac{f(y) - f(x)}{y - x} \geq \frac{f(x) - f(c)}{x - c} \geq \frac{f(c) - f(c_1)}{c - c_1}
        \end{align*}
        We therefore have that:
        \begin{align*}
            \set{\abs{\frac{f(y) - f(x)}{y - x}}: c < x < y < b} < M
        \end{align*}
        for some $M \in \RR$. Therefore, $\abs{f(y) - f(x)} < M\abs{y - x}$ for all $x, y \in (c, d)$. This holds for all $[c, d] \subset (a, b)$, showing the continuity of $f$.
        \item Let $f$ be convex, and let $a < c < x < y < d < b$. Then, by convexity we have that:
        \begin{align*}
            \frac{f(x) - f(c)}{x - c} \leq \frac{f(y) - f(x)}{y - x} \leq \frac{f(d) - f(y)}{d - y}
        \end{align*}
        Ignoring the central term in the inequality, and taking the limit as $x \rightarrow c$ and $ y \rightarrow b$, we have that:
        \begin{align*}
            f'(c) \leq f'(b)
        \end{align*}
        So we conclude that $f'$ is increasing on $(a, b)$.
        \item If $f$ is convex, $f'$ is increasing by (b). Then, we have that for any $a < x < y < b$:
        \begin{align*}
            \frac{f(y) - f(x)}{y - x} \geq 0
        \end{align*}
        And hence taking $y \rightarrow x$ we have that $f'(x) \geq 0$. \qed
    \end{enumerate}
\end{nproof}

\begin{ncorollary}{}{}
    If $f \in C^3(I)$ and $f'(x_0) = 0$ and $f''(x_0) \neq 0$, $x_0$ is a local minimum if $f$ is convex in a neighbourhood of $x_0$.
\end{ncorollary}


\subsection{Uniform cost search}

Modify BFS to prioritise cost not depth. expand node with lowest path cost. could be "deep".

This is the same as Dijkstra’s algorithm.

Can do this in algo by using heaps

Informed: No

Time: \(O(b^{?})\)

Space: \(O(b^{?})\)

Complete: Yes

Optimal: Yes


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\newcommand\set[1]{\left\{#1\right\}}

\begin{document}
\section{Exercise 7}

Given 5 women and 9 men.
Let $\mathbb P(F)$ denote the probability that a female member is chosen.
Let $\mathbb P(F\,|\,F)$ denote the probability that a female member is chosen, given that one woman was already chosen before.

We assume independent events with uniform distribution when choosing members.

\subsection{$\mathbb P(F)$}
%
In 5 of 14 cases, the chosen member will be female.
\[ \mathbb P(F) = \frac{5}{14} \]

\begin{verbatim}
def prob(s):
  women, men = 5, 9
  p = '1.0'
  for x in s:
    if x.upper() == 'F':
      p += ' * ({} / {})'.format(women, men + women)
      women -= 1
    elif x.upper() == 'M':
      p += ' * ({} / {})'.format(men, men + women)
      men -= 1
  return p

# prob('FFMM')
\end{verbatim}

\subsection{$\mathbb P(F|F)$}
%
As one woman is chosen, we are left with $4$ out of $13$ cases.
\[ \mathbb P(F|F) = \frac{4}{13} \]

\subsection{Probability that two women are chosen}
%
\begin{align*}
  &\mathbb P(\text{FFMM}) + \mathbb P(\text{FMFM}) + \mathbb P(\text{FMMF}) + \mathbb P(\text{MFFM}) + \mathbb P(\text{MFMF}) + \mathbb P(\text{MMFF}) \\
  &= \frac{5}{14} \cdot \frac{4}{13} \cdot \frac{9}{12} \cdot \frac{8}{11}
  + \frac{5}{14} \cdot \frac{9}{13} \cdot \frac{4}{12} \cdot \frac{8}{11}
  + \frac{5}{14} \cdot \frac{9}{13} \cdot \frac{8}{12} \cdot \frac{4}{11} \\
  &+ \frac{9}{14} \cdot \frac{5}{13} \cdot \frac{4}{12} \cdot \frac{8}{11}
  + \frac{9}{14} \cdot \frac{5}{13} \cdot \frac{8}{12} \cdot \frac{4}{11}
  + \frac{9}{14} \cdot \frac{8}{13} \cdot \frac{5}{12} \cdot \frac{4}{11} \\
  &= \frac{1440}{24024} + \frac{1440}{24024} + \frac{1440}{24024} + \frac{1440}{24024} + \frac{1440}{24024} + \frac{1440}{24024} \\
  &= 6 \cdot \frac{1440}{24024} = \frac{8640}{24024} = \frac{360}{1001}
\end{align*}

\subsection{Probability that three women are chosen}
%
\begin{align*}
  &\mathbb P(\text{FFFM}) + \mathbb P(\text{FFMF}) + \mathbb P(\text{FMFF}) + \mathbb P(\text{MFFF}) \\
  &= \frac{5}{14} \cdot \frac{4}{13} \cdot \frac{3}{12} \cdot \frac{9}{11}
  + \frac{5}{14} \cdot \frac{4}{13} \cdot \frac{9}{12} \cdot \frac{3}{11}
  + \frac{5}{14} \cdot \frac{9}{13} \cdot \frac{4}{12} \cdot \frac{3}{11}
  + \frac{9}{14} \cdot \frac{5}{13} \cdot \frac{4}{12} \cdot \frac{3}{11} \\
  &= \frac{540}{24024} + \frac{540}{24024} + \frac{540}{24024} + \frac{540}{24024} \\
  &= 4 \cdot \frac{540}{24024} = \frac{2160}{24024} = \frac{90}{1001}
\end{align*}

\subsection{Probability that two or three women are chosen}
%
\[ \frac{360}{1001} + \frac{90}{1001} = \frac{450}{1001} \]

This corresponds to,
\[ \frac{{5 \choose 2} {8 \choose 2}}{{14 \choose 4}} + \frac{{5 \choose 3} {9 \choose 1}}{{14 \choose 4}} \]

\end{document}


%\newpage
In this chapter, we first introduce some widely used convolutional operations,  some examples of convolution filters and their performance, and then some popular convolutional neural network (CNN) models.
\section{Convolutional operations}

\subsection{Images as matrix}\label{sec:functions}
An image can be viewed as a piecewise constant function on a grid.  Images with different
resolutions can then be viewed as functions on grids of different
sizes.  The use of such multiple-grids is a main technique used in the
standard multigrid method for solving discretized partial differential
equations, and it can also be interpreted as a main ingredient used in
convolutional neural networks (CNN) for image calssification.

An image can be viewed as a function on a grid \cite{krizhevsky2012imagenet} on
a rectangular  domain $\Omega\in \mathcal R^2$.  Without loss of generality,
 we assume that the grid, $\mathcal T$, is of size
$$
m=2^{s}, \quad n=2^{t}
$$
for some integers $s, t\ge 1$.
Starting from $\mathcal T_1=\mathcal T$,  we consider a sequence of
coarse grids with $J=\min (s,t)$ (as depicted in Fig.~\ref{mugrid-bi} with $J=4$):
\begin{equation}
\label{grids}
\mathcal T_1, \mathcal T_2, \ldots, \mathcal T_J
\end{equation}
such that ${\cal T}_\ell$ consist of $m_\ell\times n_\ell$ grid
points, with
\begin{equation}
\label{mn-ell}
 m_\ell=2^{s-\ell+1},~~ n_\ell=2^{t-\ell+1}.
\end{equation}

\begin{figure} \label{mugrid-bi}
\begin{center}
\setlength{\unitlength}{0.445mm}
\begin{picture}(45,45)(50,0)
\linethickness{0.1mm}
\multiput(-20,0)(2.5,0){17}{\line(0,1){40}}
\multiput(-20,0)(0,2.5){17}{\line(1,0){40}}
\multiput(28,0)(5,0){9}{\line(0,1){40}}
\multiput(28,0)(0,5){9}{\line(1,0){40}}
\multiput(77,0)(10,0){5}{\line(0,1){40}}
\multiput(77,0)(0,10){5}{\line(1,0){40}}
\multiput(126,0)(20,0){3}{\line(0,1){40}}
\multiput(126,0)(0,20){3}{\line(1,0){40}}
\end{picture}
\setlength{\unitlength}{0.5mm}
\end{center}
$$
\hskip0.05 in \mathcal T_1\hskip 0.7in \mathcal T_2\hskip 0.7in  \mathcal T_3\hskip 0.7in \mathcal T_4
$$
\caption{multilevel grids for piecewise constant functions (images)}
\end{figure}
Here, please note that each element in this grid can be viewed as
a pixel or an image or an element in a matrix.

\subsection{Convolution operation with one channel}
For simplicity of exposition, we denote
\begin{equation}\label{eq:size_f}
m = m_1 = 2^{s},  \quad n = n_1 = 2^t.
\end{equation}
Recall the definition of convolution in Definition \ref{def:convolution}.  In image processing, a kernel, convolution matrix, or mask is a small matrix, denoted by $K$ below. It is used for blurring, sharpening, embossing, edge detection, and more. This is accomplished by doing a convolution between a kernel and an image, denoted by $g$ below. Convolution is the process of adding each element of the image to its local neighbors, weighted by the kernel.
\begin{definition}
A convolution defined on $\mathbb{R}^{m\times n}$ is a linear mapping
$K\ast: \mathbb{R}^{m\times n}\mapsto \mathbb{R}^{m\times n}$ defined with padding,
for any $g \in \mathbb{R}^{m\times n}$ by:
%We first consider $\theta$ a convolution operator (with stride $1$)
%and padding:
\begin{equation}\label{con01}
[K \ast g]_{i,j} = \sum_{p,q=-k}^k K_{p, q} g_{i + p, j + q}, \quad i=1:m, j = 1:n.
\end{equation}
\end{definition}
The convolution maps the original  image $g$ to a modified one $K\ast g$ with the same size, and each element $[K \ast g]_{i,j}$ in the resulting image $K\ast g$ is a weighted average of elements in the original image $g$ with weights $K_{p,q}$.
The weights in \eqref{con01} constitute  a kernel matrix
\begin{equation}
K \in \mathbb{R}^{(2k+1) \times (2k+1)},
\end{equation}
where $k$ is often taken as a small integer.
Here we note that the indices for the entries in $K$ are given in a special way.
For example, if $k=1, K\in \mathbb R^{3\times 3}$, and
$$
K=\begin{pmatrix}
	K_{-1,-1} &K_{-1,0} &K_{-1,1} \\
	K_{0,-1} &K_{0,0} &K_{0,1} \\
	K_{1,-1} &K_{1,0} &K_{1,1} \\
	\end{pmatrix},
$$
for we may have the following 2D Laplacian kernel
\begin{equation}\label{key}
K=\begin{pmatrix}
0 &-1 &0\\
-1 &4&-1 \\
0 &-1 &0 \\
\end{pmatrix}.
\end{equation}
Here padding means how $ g_{i+ p, j + q}$ is defined
when $(i+ p, j + q)$ is out of $1:m$ or $1:n$.
The following three choices are often used
\begin{equation}\label{eq:padding}
g_{i + p, j + q} = \begin{cases}
0,  \quad &\text{zero padding}, \\
f_{(i + p)\pmod{m}, (s + q)\pmod{n}},  \quad &\text{periodic padding}, \\
f_{|i-1 +p|, |j -1  +q|},  \quad &\text{reflected padding}, \\
\end{cases}
\end{equation}
if
\begin{equation}
i + p \notin \{1, 2, \dots, m\} ~\text{or} ~  j+ q \notin \{1, 2, \dots, n\}.
\end{equation}
Here $ d \pmod{m} \in \{1, \cdots, m\} $  means the remainder when $d$ is divided by $m$.

Here is a diagram for convolution with one channel (and also stride one).
\begin{figure}[H]
	\begin{center}
		\includegraphics[width=0.5\textwidth]{figures/Conv0padding}
	\end{center}
\end{figure}
In \cite{bramble1977higher}, the authors show that by ``averaging" the values of
the finite element solution $u_h$ of elliptic problem using convolution
 in the neighborhood of a point $x$ they may construct an approximation to the true solution $u(x)$
 which is often a better approximation than $u_h(x)$ itself. The ``averaging" operator showed in \cite{bramble1977higher}
 is just the convolution operator defined here and does not depend on the specific elliptic operator involved.

\subsection{Convolution with stride (one channel)}
Recall the definition  of convolution with stride $2$ in Definition \ref{def:convolution2}, namely,
for $g \in \mathbb{R}^{m\times n}$, convolution with stride $2$ is defined as
\begin{equation}\label{stride_2}
[K \ast_2 g]_{i,j} = \sum_{p,q=-k}^k K_{p,q} g_{2i + p-1, 2j + q-1},
\quad i = 1: \lfloor \frac{m+1}{2}\rfloor , j = 1: \lfloor \frac{n+1}{2} \rfloor.
\end{equation}
Note that the convolution with stride $2$ maps the original image with size $2m\times 2n$ to a new one with smaller size $m\times n$, namely lower resolution.
\begin{lemma}
The convolution with stride $2$ can be written as:
\begin{equation}\label{eq:convstride_2_1}
K \ast_2 g = \mathcal S( K\ast g),
\end{equation}
where $\mathcal S$ is a stride operator defined by:
\begin{equation}\label{eq:strideopdim}
\mathcal S: \mathbb{R}^{m \times n} \mapsto \mathbb{R}^{\frac{m+1}{2} \times \frac{n+1}{2}},
\end{equation}
with
\begin{equation}\label{eq:strideop}
[\mathcal S(g)]_{i,j} = g_{2i-1, 2j-1}, \quad i = 1: \lfloor \frac{m+1}{2}\rfloor , j = 1: \lfloor \frac{n+1}{2} \rfloor.
\end{equation}
\end{lemma}

\begin{example}
The so-called average pooling with kernel size $3 \times 3$ and stride 2 means
\begin{equation}
K \ast_2\quad \mbox{with}\quad
K = \frac{1}{9}
\begin{pmatrix}
1 & 1 & 1 \\
1 & 1 & 1 \\
1 & 1 & 1
\end{pmatrix}.
\end{equation}
\end{example}

We note that, in general, for any given integer $s\ge1$, a convolution with stride
$s$ for $g \in \mathbb{R}^{m\times n}$  can be  defined as:
	\begin{equation}\label{stride}
	[K \ast_s g]_{i,j} = \sum_{p,q=-k}^k K_{p,q} g_{s(i-1) + p+1, s(j-1) + q+1},
	\quad i = 1: \lfloor  \frac{m+1}{s}\rfloor, j = 1: \lfloor  \frac{n+1}{s}\rfloor.
	\end{equation}
	Here $ \lfloor  \frac{m}{s}\rfloor$ denotes the biggest integer that less than $\frac{m}{s}$. Similarly, the convolution with stride $s$ maps the original image with size $sm\times sn$ to a new one with smaller size $m\times n$.
	The following is a diagram for stride $2$.
\begin{figure}[H]
\begin{center}
	\includegraphics[width=0.35\textwidth]{figures/PoolingLayer1}
\end{center}
\end{figure}

\subsection{Convolutional operations with multi-channel}
In many applications, we need to deal with images with multiple channels. A typical example is the RGB image, where each RGB channel emphasizes different aspects of the original images. Then the inputs become a three-dimensional tensor with size $3\times m\times n$. We refer to this axis, with a size of 3, as the channel dimension.

When the input data contain multiple channels, we need to construct a convolution kernel with the same number of input channels as the input data, so that it can perform cross-correlation with the input data.
One important class of linear mapping is the so-called convolution:
$$
\theta: \mathbb{R}^{c\times m\times n} \mapsto \mathbb{R}^{h\times m\times n},
$$
where $m\times n$ is called the spatial dimension or resolution, $c$ and $h$ are corresponding
to input and output channels. Let $f\in \mathbb{R}^{c\times m\times n}$ and the $t$-th channel $[f]_t \in \mathbb{R}^{m\times n}$.
The operation is defined by
\begin{equation}\label{conv-1}
[\theta(f)]_{s} = \sum_{t=1}^{c}\mathbf{K}_{s,t} \ast [f]_t + b_s
\bm{1}  \in \mathbb{R}^{m\times n}, \quad s = 1:h,
\end{equation}
where $\bm{1}  \in \mathbb{R}^{m\times n} $ is a
$m\times n$ matrix with all elements being $1$,
and $\mathbf{K}_{s,t}$ is a convolution operator which maps $ [f]_t\in \mathbb{R}^{m\times n}$ to $[\theta(f)]_{s} \in \mathbb{R}^{m\times n}$.
\begin{equation}\label{con1}
[\mathbf{K}_{s,t} \ast [f]_t]_{i,j} = \sum_{p,q=-k}^k K_{s,t; p,q} f_{t; i + p, j + q}, \quad i=1:m, j = 1:n.
\end{equation}
The coefficients kernel $\mathbf{K}_{s,t}$ in \eqref{con1} constitute  a kernel matrix
\begin{equation}
\mathbf{K}_{s,t} \in \mathbb{R}^{(2k+1) \times (2k+1)},
\end{equation}
where $k$ is often taken as small integers.

Here a more compact notation for multi-channel convolution can be written as
\begin{equation}
\theta(f) = \mathbf{K} \ast f + \mathbf b
\end{equation}
where
\begin{equation}\label{key}
f = \begin{pmatrix}
[f]_1 \\
[f]_2 \\
\vdots \\
[f]_c
\end{pmatrix},
\quad
\mathbf{K} = \begin{pmatrix}
K_{1,1} & K_{1,2} & \cdots & K_{1,c} \\
K_{2,1} & K_{2,2} & \cdots & K_{2,c} \\
\vdots & \vdots & \ddots& \vdots \\
K_{h,1} & K_{h,2} & \cdots & K_{h,c}
\end{pmatrix},
\quad
\mathbf b = \begin{pmatrix}
b_1 \bm 1\\
b_2 \bm 1\\
\vdots \\
b_h \bm 1\\
\end{pmatrix}=b\otimes \bm 1.
\end{equation}

Furthermore, we have the following natural extension of convolution with
stride for multi-channel by
\begin{equation}\label{conv-1}
[\theta(f)]_{s} = \sum_{t=1}^{c}\mathbf{K}_{s,t} \ast_2 [f]_t + b_s
\bm{1}  \in \mathbb{R}^{\tilde m \times \tilde n}, \quad s = 1:h,
\end{equation}
where
\begin{equation}\label{key}
\tilde m = \lfloor \frac{m+1}{2}\rfloor , \quad \tilde n = \lfloor \frac{n+1}{2} \rfloor,
\end{equation}

\subsection{Pooling operation in CNNs}
When processing images, we want to  gradually reduce the spatial resolution of our hidden representations, aggregating information so that the higher up we go in the network, the larger the receptive field (in the input) to which each hidden node is sensitive.

Finally, we introduce another type of important operation in CNNs -- pooling.
The key purpose for pooling operator is to reduce the spatial resolution of images (features) in a typical CNN models.
Basically, pooling is an operator
\begin{equation}\label{key}
T: \mathbb{R}^{c_1\times m_1\times n_1} \mapsto \mathbb{R}^{c_2 \times m_2\times n_2}.
\end{equation}
where
\begin{equation}\label{key}
m_2 = \lfloor \frac{m+1}{s}\rfloor , \quad n_2 = \lfloor \frac{n+1}{s} \rfloor,
\end{equation}
for any choice of $c_2 \ge 1$. Here $s$ is also called the stride in pooling operations. There are generally two types of pooling:

	\paragraph{Convolution with stride $s$ as pooling} In this case, it often happens that
	\begin{equation}\label{key}
	T = R \ast_s, \quad  (s = 2 \text{ for the main case }).
	\end{equation}
	Here $R$ can be learned or fixed such as average pooling as we discussed before.

	\paragraph{Nonlinear pooling} The most commonly used nonlinear pooling is called max-pooling,
	a max pooling with kernel size $(2k+1)\times (2k+1)$ and stride $s$ is is defined as
	\begin{equation}\label{key}
	[R_{\rm max}(f)]_{t; i,j} = \max \{ f_{t;s(i-1) + p+1, s(j-1) + q+1} ~|~ -k \le p,q \le k\},
	\end{equation}
	here $t$ means channel and $c_2 = c_1$ in this case.

Here is an example for max-pooling with kernel size $3\times3$ and stride $3$.
\begin{figure}[H]
\begin{center}
	\includegraphics[width=0.5\textwidth]{6DL/figures/3by3MaxPooling}
\end{center}
\end{figure}

\endinput

\subsection{Deconvolution with one channel}
For any linear mapping $\mathcal C: \mathbb{R}^{m\times n} \mapsto \mathbb{R}^{m'\times n'}$,
its transpose is the unique linear mapping $\mathcal C^\top: \mathbb{R}^{m'\times n'} \mapsto \mathbb{R}^{m\times n}$
satisfying
$$
(\mathcal C^\top u, v)_{l^2}=(u, \mathcal C v)_{l^2}~~~\forall~u\in \mathbb{R}^{m'\times n'} , v\in \mathbb{R}^{m\times n}
$$
Associated with any kernel $K$, a deconvolution is defined as the transpose of convolution
with stride $2$ with respect to the $l^2$-inner product as:
\begin{equation}\label{eq:def_deconv}
(u, K \ast_2^\top v)_{l^2}=(K \ast_2 u, v)_{l^2},
\end{equation}
with
\begin{equation}
u \in \mathbb{R}^{m \times n} \quad \text{and} \quad v \in \mathbb{R}^{\frac{m+1}{2} \times \frac{m+1}{2}}.
\end{equation}

\begin{lemma}\label{lemm:tilde-K}
	For any $K \in \mathbb{R}^{(2k+1) \times (2k+1)}$,
	\begin{equation}\label{eq:}
	K\ast_2^\top = {\tilde K}\ast \mathcal S^\top,
	\end{equation}
	where $\tilde K$ is defined as
	\begin{equation}\label{eq:def_tildeK}
	\tilde K_{p,q} = K_{-p, -q}, \quad p,q = -k:k.
	\end{equation}
	Intuitively, if we take $K_{0,0}$ as the center for the convolutional kernel $K$,
	then $\tilde K$ is the central symmetry of $K$.
	In 2D case, it can also be understood as the rotation of $\pi$ with respect to
	the center $K_{0,0}$.
\end{lemma}

Recalling the definition of deconvolution in \eqref{eq:def_deconv}, we have
\begin{equation}\label{eq:op_deconv}
\begin{aligned}
(u,  K \ast_2^\top v)_{l^2} &= (K \ast_2 u, v)_{l^2} = (\mathcal S \mathcal C_K u, v)_{l^2} \\
&= (u,  \mathcal C^\top_K \mathcal S^\top v)_{l^2},
\end{aligned}
\end{equation}
with definition
\begin{equation}\label{eq:de_stride_dim}
\mathcal S^\top:   \mathbb{R}^{\frac{m+1}{2} \times\frac{n+1}{2}} \mapsto \mathbb{R}^{m\times n},
\end{equation}
and
\begin{equation}\label{eq:de_stride}
[\mathcal S^\top (f)]_{i,j} =
\begin{cases}
0 \quad &\text{if i or j is even}, \\
f_{i/2, j/2}, \quad &\text{else}.
\end{cases}
\end{equation}

Thus to say, we have the simple version of the deconvolution for $K \ast $ as
\begin{equation}\label{eq:simple_deconv}
K \ast_2^\top v = \mathcal C_K^\top \circ \mathcal S^\top (v) = \mathcal C_{\tilde K} \circ \mathcal S^\top (v) = \tilde K \ast \mathcal S^\top (v),
\end{equation}
thus to say
\begin{equation}\label{eq:final}
K \ast_2^\top  = \tilde K \ast \mathcal S^\top.
\end{equation}

In short, we have the next decomposition
\begin{itemize}
	\item convolution with stride = stride $ \circ$ convolution,
	\item deconvolution with stride  = transposed convolution $\circ$ transposed stride = convolution with the central symmetry of original kernel $\circ$ transposed stride.
\end{itemize}

\begin{theorem}\label{thm:deconv_op}

	Let us consider
	\begin{equation}
	K=(K_{p,q}),~~p,q = -1, 0, 1.
	\end{equation}
	Then we have
	$$
	K \ast_2^\top v = \tilde K \ast \mathcal S^\top (v).
	$$
	As in \eqref{eq:de_stride} and the Lemma \ref{lemm:tilde-K}, we have the
	final version is
	\begin{equation}
	\label{eq:7}
	[K \ast_2^\top v ]_{2i,2j}=  K_{0,0}v_{i,j},
	\end{equation}
	with
	\begin{equation}
	\label{eq:9}
	[K \ast_2^\top v ]_{2i-1, 2j} = K_{0,1}v_{i-1,j} + K_{0,-1}v_{i,j}, \quad
	[K \ast_2^\top v ]_{2i, 2j-1} = K_{1,0}v_{i,j} + K_{-1,0}v_{i,j-1},
	\end{equation}
	and
	\begin{equation}
	%\begin{tiny}
	%{\scriptsize
	[K \ast_2^\top v ]_{2i-1, 2j-1}  =
	K_{1,1}v_{i,j} + K_{-1,1}v_{i-1,j} + K_{1,-1}v_{i,j-1} + K_{-1,-1}v_{i-1,j-1}.
	%\end{tiny}
	%}
	\end{equation}
\end{theorem}
\begin{remark}
	Deconvolution can obviously be also defined for general stride $s$, but we believe it is sufficient to use $s=2$
	in most applications.
\end{remark}


\section{Triple Integrals}\label{sec:TripleIntegrals}

It will come as no surprise that we can also do triple
integrals---integrals over a three-dimensional region. The simplest
application allows us to compute volumes in an alternate way.\index{triple integral}

To approximate a volume in three dimensions, we can divide the
three-dimensional region into small rectangular boxes, each
$\Delta x\times\Delta y\times\Delta z$ with volume
$\Delta x\Delta y\Delta z$. Then we add them all up and take the
limit, to get an integral:
\[\int_{x_0}^{x_1}\int_{y_0}^{y_1}\int_{z_0}^{z_1} dz\,dy\,dx.\]
Of course, if the limits are constant, we are simply computing the
volume of a rectangular box.

\begin{example}{Volume of a Box}{VolumeBox}
Compute the volume of the box with opposite corners at $(0,0,0)$ and $(1,2,3)$.
\end{example}
\begin{solution}
We use an integral to compute the volume of the box:
\[\int_0^1\int_0^2\int_0^3
dz\,dy\,dx=\int_0^1\int_0^2\left.z\right|_0^3 \,dy\,dx
=\int_0^1\int_0^2 3\,dy\,dx
=\int_0^1 \left.3y\right|_0^2 \,dx
=\int_0^1 6\,dx = 6.
\]
\end{solution}

Of course, this is more interesting and useful when the limits are not
constant.

\begin{example}{Volume of a Tetrahedron}{VolumeTetrahedron}
Find the volume of the tetrahedron with corners at $(0,0,0)$,
$(0,3,0)$, $(2,3,0)$, and $(2,3,5)$.
\end{example}
\begin{solution}
The whole problem comes down to correctly describing the region by
inequalities:
$0\le x\le 2$, $3x/2\le y\le 3$, $0\le z\le 5x/2$.
The lower $y$ limit comes from the equation of the line
$y=3x/2$ that forms one edge of the tetrahedron in the $x$-$y$ plane;
the upper $z$ limit comes from the equation of the plane $z=5x/2$ that
forms the ``upper'' side of the tetrahedron; see Figure~\ref{fig:tetrahedron}.
Now the volume is
\begin{align*}
\int_0^2\int_{3x/2}^3\int_0^{5x/2}dz\,dy\,dx
&=\int_0^2\int_{3x/2}^3\left.z\right|_0^{5x/2} \,dy\,dx	\\
&=\int_0^2\int_{3x/2}^3 {5x\over2}\,dy\,dx	\\
&=\int_0^2 \left.{5x\over2}y\right|_{3x/2}^3 \,dx	\\
&=\int_0^2 {15x\over2}-{15x^2\over4}\,dx	\\
&=\left. {15x^2\over4}-{15x^3\over12}\right|_0^2	\\
&=15-10=5.
\end{align*}
\end{solution}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <1.5truecm,1.5truecm>
\setplotarea x from 0 to 2.1, y from -1.1 to 1.1
\put {\hbox{\epsfxsize6cm\epsfbox{images/tetrahedron.eps}}} at 0 0
\endpicture}}
\caption{A tetrahedron.}
\label{fig:tetrahedron}
\end{figure}

Pretty much just the way we did for two dimensions we can use triple
integration to compute mass, center of mass, and various average quantities.

\begin{example}{Average Temperature in a Cube}{AvgTempCube}
Suppose the temperature at a point is given by
$T=xyz$. Find the average temperature in the cube with opposite
corners at $(0,0,0)$ and $(2,2,2)$.
\end{example}
\begin{solution}
In two dimensions we add up the temperature at ``each'' point and
divide by the area; here we add up the temperatures and divide by the
volume, $8$:
\begin{align*}
{1\over8}\int_{0}^2\int_{0}^2\int_{0}^2 xyz\,dz\,dy\,dx
&={1\over8}\int_{0}^2\int_{0}^2\left.{xyz^2\over2}\right|_0^2\,dy\,dx
={1\over16}\int_{0}^2\int_{0}^2 xy\,dy\,dx	\\
&={1\over4}\int_{0}^2\left.{xy^2\over2}\right|_0^2\,dx
={1\over8}\int_{0}^2 4x\,dx
={1\over2}\left.{x^2\over2}\right|_0^2
=1.
\end{align*}
\end{solution}

\begin{example}{Mass \& Center of Mass of Tetrahedron}{3dcenterofmass}
Suppose the density of an object is given by $xz$, and the
object occupies the tetrahedron with corners
$(0,0,0)$, $(0,1,0)$, $(1,1,0)$, and $(0,1,1)$. Find the mass and
center of mass of the object.
\end{example}
\begin{solution}
As usual, the mass is the integral of density over the region:
\begin{align*}
M&=\int_{0}^1\int_{x}^1\int_{0}^{y-x} xz\,dz\,dy\,dx
=\int_{0}^1\int_{x}^1 {x(y-x)^2\over2}\,dy\,dx
={1\over2}\int_{0}^1 {x(1-x)^3\over3}\,dx	\\
&={1\over6}\int_{0}^1 x-3x^2+3x^3-x^4\,dx
={1\over120}.
\end{align*}
We compute moments as before, except now there is a third moment:
\begin{align*}
M_{xy} &= \int_{0}^1\int_{x}^1\int_{0}^{y-x} xz^2\,dz\,dy\,dx
={1\over360},	\\
M_{xz} &= \int_{0}^1\int_{x}^1\int_{0}^{y-x} xyz\,dz\,dy\,dx
={1\over144},	\\
M_{yz} &= \int_{0}^1\int_{x}^1\int_{0}^{y-x} x^2z\,dz\,dy\,dx
={1\over360}.
\end{align*}
Finally, the coordinates of the center of mass are
$\bar x=M_{yz}/M=1/3$, $\bar y=M_{xz}/M=5/6$, and  $\bar
z=M_{xy}/M=1/3$.
\end{solution}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:TripleIntegrals}}

\begin{enumialphparenastyle}

\begin{ex}
Evaluate $\ds\int_{0}^{1}\int_{0}^{x}\int_{0}^{x+y}
2x+y-1 \,dz\,dy\,dx$.
\begin{sol}
$11/24$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\int_{0}^{2}\int_{-1}^{x^2}\int_{1}^{y}
xyz \,dz\,dy\,dx$.
\begin{sol}
$623/60$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\int_{0}^{1}\int_{0}^{x}\int_{0}^{\ln y}
e^{x+y+z}\,dz\,dy\,dx$.
\begin{sol}
$-3e^2/4+2e-3/4$
\end{sol}
\end{ex}

\begin{ex}
Evaluate
$\ds\int_{0}^{\pi/2}\int_{0}^{\sin\theta}\int_{0}^{r\cos\theta}
r^2\,dz\,dr\,d\theta$.
\begin{sol}
$1/20$
\end{sol}
\end{ex}

\begin{ex}
Evaluate
$\ds\int_{0}^{\pi}\int_{0}^{\sin\theta}\int_{0}^{r\sin\theta}
r\cos^2\theta\,dz\,dr\,d\theta$.
\begin{sol}
$\pi/48$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\int_{0}^{1}\int_{0}^{y^2}\int_{0}^{x+y}
x\,dz\,dx\,dy$.
\begin{sol}
$11/84$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\int_{1}^{2}\int_{y}^{y^2}\int_{0}^{\ln(y+z)}
e^x\,dx\,dz\,dy$.
\begin{sol}
$151/60$
\end{sol}
\end{ex}

\begin{ex}
Compute $\ds
\int_0^\pi\int_0^{\pi/2}\int_0^1 z\sin x+z\cos y\,dz\,dy\,dx$.
\begin{sol}
$\pi$
\end{sol}
\end{ex}

\begin{ex}
For each of the integrals in the previous exercises, give a
description of the volume (both algebraic and geometric) that is the
domain of integration.
\end{ex}

\begin{ex}
Compute $\ds\int\int\int
x+y+z\,dV$ over the region inside
$x^2+y^2+z^2\le 1$ in the first octant.
\begin{sol}
$\ds {3\pi\over16}$
\end{sol}
\end{ex}

\begin{ex}
Find the mass of a cube with edge length 2 and density equal
to the square of the distance from one corner.
\begin{sol}
$32$
\end{sol}
\end{ex}

\begin{ex}
Find the mass of a cube with edge length 2 and density equal
to the square of the distance from one edge.
\begin{sol}
$64/3$
\end{sol}
\end{ex}

\begin{ex}
An object occupies the volume of the upper hemisphere of
$x^2+y^2+z^2=4$ and has density $z$ at $(x,y,z)$. Find the center of mass.
\begin{sol}
$\bar x=\bar y=0$, $\bar z=16/15$
\end{sol}
\end{ex}

\begin{ex}
An object occupies the volume of the pyramid with corners at
$(1,1,0)$, $(1,-1,0)$, $(-1,-1,0)$, $(-1,1,0)$, and $(0,0,2)$ and has
density $x^2+y^2$ at $(x,y,z)$. Find the center of mass.
\begin{sol}
$\bar x=\bar y=0$, $\bar z=1/3$
\end{sol}
\end{ex}

\begin{ex}
Verify the moments $M_{xy}$, $M_{xz}$, and $M_{yz}$
of Example~\ref{exa:3dcenterofmass} by evaluating the
integrals.
\end{ex}

\begin{ex}
Find the region $E$ for which $\ds\iiint_E
  (1-x^2-y^2-z^2) \; dV$ is a maximum.
\end{ex}

\end{enumialphparenastyle}


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amsthm,amssymb,amsfonts}
\usepackage{enumitem}
\usepackage{tabu}
\usepackage{xcolor}
\usepackage{mathtools}
\usepackage{tcolorbox}
\usepackage{changepage}
\usepackage{kpfonts}
\usepackage{picture}
\usepackage{venndiagram}

\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\R}{\mathbb{R}}

\newcommand{\field}{\mathcal{F}}
\newcommand{\prob}[1]{\mathbb{P}(#1)}
\newcommand{\probtext}[1]{\mathbb{P}(\text{#1})}

\renewcommand\labelitemi{---}

\newenvironment{nscenter}
 {\parskip=0pt\par\nopagebreak\centering}
 {\par\noindent\ignorespacesafterend}

\def\SPSB#1#2{\rlap{\textsuperscript{\textcolor{black}{#1}}}\SB{#2}}

\begin{document}
\title{%
  MATH 487: Probability Theory Exam II\\
  \large University of Nebraska-Lincoln, Fall 2017}
\author{Jacob Shiohira}
\maketitle

\newpage

% ================================================================================================================================
% ================================================================================================================================
\begin{center}
\section*{Random Variables}
\end{center}

\noindent
In general, a random variable is a symbolic representation (variable) of the outcome of experiment that depends on chance.

\vspace*{.5cm}
\subsection*{Discrete Random Variables}
\noindent
A \textbf{discrete random variable} has a countable number of outcomes. The distribution of probabilities of a discrete random variable are measured by a \textit{probability mass function}, $p_X(x)$, such that for any $x \in \text{Range}(X)$,

\begin{equation*}
\prob{X = x} = p_X(x)
\end{equation*}

\noindent
where $p_X(x) > \geq 0$ and $\sum_{x} p_X(x) = 1$.

\subsection*{Continuous Random Variables}
\noindent
A \textbf{continuous random variable} has an infinitely uncountable number of outcomes. In other words, it can take on all values in a given interval. The distribution of probabilities of a continuous random variable are measured by a \textit{probability distribution function}, $f_X(x)$ such that for any $a,b \in \text{Range}(X)$,

\begin{equation*}
\prob{a \leq X \leq b} = \int_{a}^{b} f_X(x) dx
\end{equation*}

\noindent
where $f_X(x) > \geq 0$ and $\int_{- \infty}^{\infty} f_X(x) dx = 1$. Unlike a \textbf{probability mass function}, the probability of $f_X(x)$ such that $X = x$ is not defined in a \textbf{probability distribution function}.

\subsubsection*{PMF versus PDF}
\noindent
A \textbf{probability mass function} measures the total mass with the dimension of probability. A \textbf{probability distribution function} is essentially the measurement of probability on some continuum where $\rho (\frac{\text{probability}}{unit})$ is a non-constant density function $f_X(x)$.

\subsection*{Cumulative Distribution Functions}
A \textbf{cumulative distribution function}, or CDF, of a real-valued random variable $X$, or just distribution function of $X$, evaluated at $x$, is the probability that $X$ will take a value less than or equal to $x$. In other words, $\text{CDF}(x) = F_X(x)$ is equivalent to

\begin{align*}
\prob{X \leq x} = \sum_{x_i \leq x} p_X(x) dx && \text{in the Discrete Case} \\
\prob{X \leq x} = \int_{- \infty}^{x} f_X(x) dx && \text{in the Continuous Case}
\end{align*}

\noindent
General properties of the CDF are as follows,

\begin{enumerate}
\item $F(x) = \prob{X \leq x}$,
\item $0 \leq F(x) \leq 1$,
\item $F(x)$ is a non-decreasing function. If $a < b$, then $F(a) < F(b)$,
\item $\lim_{x \rightarrow \infty} F(x) = 1$ and $\lim_{x \rightarrow - \infty} F(x) = 0$,
\item $\prob{a \leq X \leq b} = F(b) - F(a)$,
\item $F'(x) = f_X(x)$.
\end{enumerate}

\begin{center}
\section*{Distributions}
\end{center}

\subsection*{Normal, Gaussian}
\noindent
Generally, the \textbf{normal} distribution is used $XXXX$. The probability distribution function is

\begin{align*}
N(\mu, \sigma^2) = \frac{1}{\sqrt{2 \pi} \sigma} e^{- \frac{(x - \mu)^2}{2 \sigma^2}} \quad \text{ for } \sigma > 0.
\end{align*}

\noindent
while the cumulative distribution function is

\begin{equation*}
\iota(x) = \int_{- \infty}^{x} \frac{1}{\sqrt{2 \pi}} e^{- \frac{\mu^2}{2}} du.
\end{equation*}

\noindent
Thus, $\text{E}(X) = \mu$ and $\text{Var}(X) = \sigma^2$.

\subsection*{Cauchy}
\noindent
Generally, the \textbf{Cauchy} distribution is used for resonance behavior and used to measure outliers in data sets. The probability distribution function is

\begin{equation*}
C(\theta, \sigma) = \frac{1}{\pi \sigma} \Bigg [ \frac{1}{1 + \big ( \frac{x- \mu}{\sigma}\big )^2} \Bigg ].
\end{equation*}

\noindent
while the cumulative distribution function is

\begin{equation*}
F(X) = \frac{1}{\pi} \text{arctan}\big ( \frac{x - x_0}{\gamma}\big ) + \frac{1}{2}
\end{equation*}

\noindent
The $\text{E}(X)$ and the $\text{Var}(X)$ are not defined for the Cauchy distribution.

\subsection*{Exponential}
\noindent
Generally, the \textbf{Exponential} distribution is used to describe timing of events, such as the times between customer arrivals, times to equipment failure, etc. The probability distribution function is

\begin{equation*}
E(\lambda) = \lambda e^{- \lambda x}
\end{equation*}

\noindent
while the cumulative distribution function is

\[ F(x) =  \begin{cases}
      1-e^{- \lambda x} & x \geq 0, \\
      0 & x < 0.
      \end{cases} \]

\noindent
Thus, $\text{E}(X) = \frac{1}{\lambda}$ and $\text{Var}(X) = \frac{1}{\lambda^2}$.

\subsection*{Gamma}
\noindent
Generally, the \textbf{Gamma} distribution is used to measure non-negative random variables such as rain fall amount, plant yields, time between earthquakes, etc. The probability distribution function is

\begin{equation*}
\Gamma ( \alpha, \beta) = \frac{\lambda^{\alpha}}{\Gamma(\alpha)} x^{\alpha - 1} e^{- \lambda x}
\end{equation*}

\noindent
while the cumulative distribution function is

\begin{equation*}
F(x) = \frac{\Gamma_x(\gamma)} {\Gamma(\gamma)} \qquad x \geq 0; \gamma > 0
\end{equation*}

\noindent
Thus, $\text{E}(X) = \frac{\alpha}{\lambda}$ and $\text{Var}(X) = \frac{\alpha}{\lambda^2}$. Note that $\Gamma(n) = (n-1)!$.

\subsection*{Bernoulli}
\noindent
Generally, the \textbf{Bernoulli} distribution is used $XXXX$. The probability distribution function is

\[ f_X(x) =  \begin{cases}
      \alpha & x = 1, \\
      1 - \alpha & x = 0.
      \end{cases} \]

\noindent
while the cumulative distribution function is

\[ F(x) =  \begin{cases}
      0 & x \leq 0, \\
      1 - \alpha & 0 < x < 1, \\
      1 & x \geq 1.
      \end{cases} \]

\noindent
Thus, $\text{E}(X) = p$ and $\text{Var}(X) = p(1-p)$.

\subsection*{Binomial}
\noindent
Generally, the \textbf{Binomial} distribution is used $XXXXX$. The probability distribution function is

\begin{equation*}
b(n,p,m) = {n \choose m} p^n (1-p)^{n-m}
\end{equation*}

\noindent
while the cumulative distribution function is

\begin{equation*}
F(x;p,n) = \sum_{i = 0}^{x} {n \choose i} p^i (1- p)^{n - i}
\end{equation*}

\noindent
Thus, $\text{E}(X) = np$ and $\text{Var}(X) = npq$.

\subsection*{Geometric}
\noindent
Generally, the \textbf{Geometric} distribution is used $XXXXX$. The probability distribution function is

\begin{equation*}
g(p, k) = q^{k-1} p
\end{equation*}

\noindent
while the cumulative distribution function is

\begin{equation*}
F(x) = 1- (1 - p)^{k}
\end{equation*}

\noindent
Thus, $\text{E}(X) = \frac{1}{p}$ and $\text{Var}(X) = \frac{1-p}{p^2}$.

\subsection*{Poisson}
\noindent
Generally, the \textbf{Poisson} distribution is used $XXX$. The probability distribution function is

\begin{equation*}
P(k, \lambda) = e^{- \lambda} \frac{\lambda^k}{k!}.
\end{equation*}

\noindent
The $\text{E}(X) = \lambda$ and $\text{Var}(X) = \lambda$.

\begin{center}
\section*{Joint, Marginal, Conditional Functions}
\end{center}

\subsection*{Discrete Random Variables}
\noindent
Consider the two probability mass functions, $f_X(x) = \prob{X = x}$ and $f_Y(y) = \prob{Y = y}$. The \textbf{joint probability mass function} for $X,Y$ is

\begin{equation*}
\mathbb{P}_{XY}(x,y) = \prob{X = x \cap Y = y}.
\end{equation*}

\noindent
Note that

\begin{equation*}
\sum_{x} \sum_{y} \mathbb{P}_{XY}(x,y) = \sum_{(x_i, y_j) \in R_{XY}} \mathbb{P}_{XY}(x_i,y_j) =  1,
\end{equation*}

\noindent
where $R_{XY} = \{ (x_i, y_j) \lvert x_i \in R_X, y_j \in R_Y \} = R_{X} \times R_{Y}$ where $R_X = {x_1, x_2, \ldots }$ and $R_Y = {y_1, y_2, \ldots }$.

\vspace*{.5cm}
\noindent
The \textbf{marginal probability mass function} for any $x \in R_X$ is

\begin{equation*}
\mathbb{P}_X(x) = \prob{X = x} = \sum_{y_j \in R_Y} \prob{X = x, Y = y_j} = \sum_{y_j \in R_Y} \mathbb{P}_{XY}(x,y_j).
\end{equation*}

\noindent
Likewise, for any $y \in R_Y$,

\begin{equation*}
\mathbb{P}_Y(y) = \prob{Y = y} = \sum_{x_i \in R_X} \prob{X = x_i, Y = y} = \sum_{x_i \in R_X} \mathbb{P}_{XY}(x_i,y).
\end{equation*}

\vspace*{.5cm}
\noindent
The \textbf{conditional probability mass function} of $Y$ given $X = x$ is

\begin{align*}
f_{Y \lvert x} \frac{f_{XY}(x,y)}{f_X(x)} \qquad \text{for } f_X(x) > 0.
\end{align*}

\noindent
In other words, $f_{Y \lvert x}$ is equivalent to the joint probability of $X$ and $Y$ divided by the marginal probability for $X$. A few common properties of the \textbf{conditional probability mass function} are as follows,

\begin{enumerate}
\item $f_{Y \lvert x}(y) \geq 0$,
\item $\sum_{y} f_{Y \lvert x} = 1$,
\item $f_{Y \lvert x}(y) = \prob{Y = y \lvert X = x}$.
\end{enumerate}

\subsection*{Continuous Random Variables}
% Basically the same... copy from discrete random variables.

\subsection*{Law of Total Probability}
\begin{equation*}
\prob{x} = f_X(x) = \int_{- \infty}^{\infty} f_{X \lvert Y} (x, y) f_Y(y) dy.
\end{equation*}

\subsection*{Joint Cumulative Distribution Function}
\noindent
For two Random Variables $X,Y$, the joint Cumulative Distribution Function is

\begin{equation*}
F_{XY}(x,y) = \prob{X \leq x, Y \leq y} = \prob{ (X \leq x) \cap (Y \leq y)}.
\end{equation*}

\subsection*{Marginal Cumulative Distribution Function}
\noindent
For any $x \in X$,

\begin{equation*}
F_X(x) = F_{XY}(x, \infty) = \lim_{y \rightarrow \infty} F_{XY}(x, y).
\end{equation*}

\noindent
For any $y \in Y$,

\begin{equation*}
F_Y(y) = F_{XY}(\infty, y) = \lim_{x \rightarrow \infty} F_{XY}(x, y).
\end{equation*}

\begin{center}
\item	\section*{Independent Random Variables}
\end{center}
Random variables $X,Y$ are independent \textit{if and only if}

\begin{equation*}
\prob{X \in A, Y \in B} = \prob{X \in A} \prob{Y \in B}
\end{equation*}

\noindent
for any subsets $A,B \in \R^2$.
\subsection*{Expected Value of Random Variables}
\noindent
The expected value of a random variable is also its mean. So, $\text{E}(X) = \mu$. Common properties of the expected value of random variables are as follows,

\begin{enumerate}
\item $\text{E}(X + Y) = \text{E}(X) + \text{E}(Y)$,
\item $\text{E}(cX) = c\text{E}(X)$,
\item If $X,Y$ are independent, $\text{E}(XY) = \text{E}(X) \text{E}(Y)$.
\end{enumerate}

\subsubsection*{Discrete Random Variables}
\noindent
In the case of a discrete random variable,

\begin{equation*}
\text{E}(X) = \sum_{k \in \Omega} k m(x)
\end{equation*}

\noindent
for distribution function $m(x)$.

\vspace*{.5cm}
\noindent
\textbf{Martingales} If $X$ is a random variable, $\Omega$ is a sample space, and $F_1, F_2, \ldots, F_r$ are events such that $F_i \cap F_j = \emptyset$ for $i \neq j$ and $\Omega = \bigcup_j F_j$, then

\begin{equation*}
\text{E}(X) = \sum_j \text{E}(X \lvert F_j) \prob{F_j}.
\end{equation*}

\subsubsection*{Continuous Random Variables}
% Basically the same... copy from discrete random variables.

\subsection*{Variance of Random Variables}
\noindent
Variance is how much the outcome of an experiment varies from the expected value, or mean, $\text{E}(X)$. Common properties of the variance of random variables are as follows,

\begin{enumerate}
\item $\text{Var}(X + b) = \text{Var}(X)$,
\item $\text{Var}(cX) = c^2 \text{E}(X)$,
\item If $X,Y$ are independent, $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$.
\end{enumerate}

\subsubsection*{Discrete Random Variables}
\noindent
Generally, for a distribution function $m(x)$,

\begin{equation*}
\text{Var}(X) = \text{E}((X-\mu)^2) = \sum_{x} (x - \mu)^2 m(x) = \text{E}(X^2) - \big [ \text{E}(X)^2 \big ].
\end{equation*}

\subsubsection*{Continuous Random Variables}
% Basically the same... copy from discrete random variables.

\begin{center}
\item \section*{Convolutions}
\end{center}
\end{document}