\subsection{Numerical Solutions for Burgers' Equation without Viscosity}

	To finish the section, we will study the problem (\ref{IVP_Burgers}) without viscosity, that is, when $\alpha = 0$. So the problem is the following
	\begin{align}
	\label{IVP_Inviscid}
		\left \lbrace \begin{array}{ll}
			\frac{\partial u}{\partial t} + \frac{1}{2} (u^2)_x = 0, \hspace{2mm} 0 < t \leq T, \hspace{2mm} x \in \mathcal{D} \\
			\\
			u(x, 0) = u_0(x), \hspace{2mm} x \in \mathcal{D}
		\end{array}  \right .
	\end{align}

	This problem, which seems much simpler, turns out to be very interesting, because it presents relevant characteristics regarding the physical interpretation of the problem in general, that is, the case with non-zero viscosity. \\

	The previous equation interprets the conservation of energy and is considered as a non-linear conservation law. We can understand this if we consider the function $ u $ as the speed of a fluid that conserves its energy with a flow density given by $f(u) = \frac{u^2} {2}$. \\

	The above can be shown considering that $u \in H^1_p [\mathcal{D}]$, and multiplying by $u$ and integrating both sides of the equation on the domain $\mathcal{D}$ to prove that $\| u \|$ does not change over time, that is,
	\begin{align*}
		\frac{1}{2} \frac{d}{dt} \displaystyle \int_{0}^{2 \pi} u^2(x, t) dx = - \int_{0}^{2 \pi} u^2(x, t) \frac{\partial u(x, t)}{\partial x} dx = - \frac{1}{3} u^3(x, t) \Big|^{2 \pi}_{0}.
	\end{align*}
	and because $u$ is periodic, we have to
	\begin{align*}
		\frac{1}{2} \frac{d}{dt} \| u(x, t) \|^2 = 0.
	\end{align*}
	This means that the energy is conserved with the same initial energy since we must bear in mind that this represents the kinetic energy of a fluid that travels with a velocity given by $u_0$. \\

	In any problem, these characteristics are of utmost importance, this is because we must consider the behavior of the solutions when they must be approached with some numerical method. In the literature, when a solution remains bounded in time, the problem is said to be well defined. In the analysis context, this assures us the existence of a continuously differentiable solution $u$, and that besides being unique, it is possible to approximate it. \\

	However, approximating the solutions to this problem using spectral methods can be more complicated if its characteristics are not well understood. Next, we will see that under a certain condition it is possible to find a single analytical solution and that otherwise, it may lose its uniqueness when developing discontinuities. \\

	First, let's define the curves $x = x(t)$ that start at a point $x_0 \in \mathbb{R}$, and satisfy the following problem
	\begin{align*}
		\left \lbrace \begin{array}{ll}
			x' (t) = u(x(t), t), \hspace{2mm} t > 0, \\
			\\
			x(0) = x_0.
		\end{array} \right .
	\end{align*}

	The solutions for each $x_0$ are known as characteristic curves, which pass through the solution $u = u (x (t), t)$. It is well known in the theory of differential equations that when $u(x, t)$ is locally Lipschitz in the variable $x$, and continues in the variable $t$, the above equation admits a single solution $x(t)$ for each $x_0 \in \mathbb{R}$. So, assuming the above we have that for a solution $ x (t) $ corresponding to a fixed $x_0 \in \mathbb{R}$, it satisfies the following
	\begin{align*}
		\frac{d}{dt} [u(x(t), t)] &= x' (t) u_x (x(t), t) + u_t (x(t), t) \\
		&= u(x(t), t) u_x (x(t), t) - u(x(t), t) u_x (x(t), t) = 0
	\end{align*}
	which tells us that the function $u(x (t), t)$ is independent of the variable $t$, remaining constant along the characteristic curve. Therefore, we have to
	\begin{align*}
		u(x(t), t) = u (x(0), 0) = u_0 (x_0)
	\end{align*}

	Furthermore, we have that the solution $ x (t) $ will be given by the following curve
	\begin{align*}
		x(t) = x_0 + u_0 (x_0) t, \hspace{2mm} t > 0.
	\end{align*}
	which allows us to write the solution $ u(x, t)$ as follows
	\begin{align}
	\label{Exact_Inviscid}
		u(x, t) = u_0(x_0), \hspace{2mm} x_0 = x - u_0(x_0) t
	\end{align}

	Note that the Lipschitz condition is necessary for the uniqueness of the above solution, since, conversely, if we set two different starting points $x_0, x_1$ such that $x_0 <x_1$, then its curves characteristics intersect for some $t$, that is,
	\begin{align*}
		x_0 + u_0(x_0) t = x_1 + u_0 (x_1) t,
	\end{align*}
	and using the mean value theorem we have that for some $c \in (x_0, x_1)$ the time $t$ is given by
	\begin{align*}
		t = \frac{x_1 - x_0}{u_0 (x_0) - u_0 (x_1)} = - \frac{1}{u'_0(c)},
	\end{align*}

	The above tells us that when these two curves intersect, the solution $u (x, t)$ cannot be continuous in the time $ t $ given by the previous equation. This is because $u_0 (x_0) \neq u_0 (x_1)$, and since the solution is constant over time we would have that $u (x, t) = u_0 (x_0) = u_0 (x_1)$, which is impossible. \\

	Therefore, the time $t$ for which two curves intersect represents a discontinuity, and that can be calculated by
	\begin{align}
	\label{shock_time}
			Tc = \min_{x \in \mathbb{R}} \left[  \frac{-1}{u'_0 (x)} \right]
	\end{align}
	and we can observe that the continuity of the solution $u$ is assured if $T_c < 0$, which depends only on the initial condition $u_0$. \\

	This type of information allows us to know the criteria that must be considered when implementing numerical methods, but also the physical interpretation of the problem can be useful. For example, the solution to this problem can be considered to simulate the evolution of the profile of a sea wave that deforms as it approaches the coast until it breaks. But when this occurs, the deformation of the wave stops precisely at the time $T_c$. \\

	We must consider that the problem without viscosity supposes that the fluid behaves in an ideal or perfect way, traveling as if they were separate sheets without rubbing. But in real life, fluids always have a certain degree of viscosity, that is, the fluid sheets can rub and cause energy dissipation, and for these cases, we could consider the problem with sufficiently small values of $\alpha$. So a question that naturally arises is what the behavior of the solutions looks like when $\alpha$ approaches zero. \\

	In chapter \ref{Introduction} we obtained the solution of the problem (\ref{IVP_Burgers}), which is given by (\ref{Exact_Solution}), and from this equation we can see that the solutions are infinitely differentiable for any value of $\alpha> 0$. Instinctively, we can notice that the solution approaches the solution of the equation without viscosity when $\alpha$ approaches zero. In fact, the solution obtained as a limit when $\alpha$ approaches zero is known as the entropy solution, which has been studied in \cite{Tadmor1989}, \cite{Maday1989} and in \cite{Kruzkov1970} has been proved that this solution is unique. \\

	In order to illustrate what we have previously discussed, we will consider the problem (\ref{IVP_Inviscid}) with the following initial condition function
	\begin{align}
	\label{IC_Inviscid}
		u_0 (x) = e^{-0.005x^2}, \hspace{3mm} x \in \mathcal{D}
	\end{align}
	where $\mathcal{D} = [x_L, x_R]$, and considering the interval $I = [0, T_c]$ for a value of $T_c$ given by (\ref{shock_time}). \\

	In the following simulations, we will use the Fourier-Galerkin method given in (\ref{Galerkin_Euler}) to obtain approximations with different values of $\alpha$, and we will see how they approximate the exact solution of the problem (\ref{IC_Inviscid} ) which was given in (\ref {Exact_Inviscid}).

	\newpage
	\begin{figure}[H]
		\centering
		\includegraphics[width=12cm]{burgers_equation/deterministic/numerical_experiments/inviscid/figures/small_alpha.png}
		\caption{Numerical solution for (\ref{IVP_Burgers}) using (\ref{Galerkin_Euler}) with $\alpha = 1.0 \times 10^{-5}$, $N=256$, and $\Delta t = 1.0 \times 10^{-3}$.}
		\vspace{2mm}
		\includegraphics[width=12.5cm]{burgers_equation/deterministic/numerical_experiments/inviscid/figures/small_alpha_T.png}
		\caption{Numerical solution for (\ref{IVP_Burgers}) using (\ref{Galerkin_Euler}) at the time $T_c$ with $\alpha = 1.0 \times 10^{-5}$, and $\Delta t = 1.0 \times 10^{-3}$. (b) Point-wise error of approximation.}
	\end{figure}
	\newpage
	\begin{figure}[H]
		\centering
		\includegraphics[width=13cm]{burgers_equation/deterministic/numerical_experiments/inviscid/figures/Numerical_Solution_Inviscid_T.png}
		\caption{(a) Exact solution for (\ref{IVP_Burgers}), and its approximations using (\ref{Galerkin_Euler}) at the time $Tc$ with initial condition $u_0(x) = e^{-0.005x^2}$, $x \in [-60, 60]$. (b) Pointwise error of approximation.}
		\label{convection_aprox_T}
		\includegraphics[width=11cm]{burgers_equation/deterministic/numerical_experiments/inviscid/figures/Numerical_Solution_Inviscid.png}
		\caption{Numerical approximation for  \ref{IVP_Burgers} using (\ref{Galerkin_Euler}) with $N=512$, $u_0(x) = e^{-0.005x^2}$, $x \in [-60, 60]$, and $t \in [0, Tc]$.}
	\end{figure}
	\newpage
	\begin{table}[H]
		\centering
		\begin{tabular}{lccc}
			\toprule
			\multicolumn{1}{c}{\hspace{6mm}\textbf{Expansion}} & \multicolumn{3}{c}{\textbf{Distance}} \\
			\hspace{12mm} $N$ & $\Delta t=1\times 10^{-2}$ & $\Delta t=1\times 10^{-3}$ & $\Delta t=1\times 10^{-4}$ \\
			\midrule
			\hspace{12mm} 16 & 0.285531 & 0.285732 & 0.285752 \\
			\midrule
			\hspace{12mm} 32 & 0.222737 & 0.223260 & 0.223312 \\
			\midrule
			\hspace{12mm} 64 & 0.160385 & 0.162782 & 0.163025 \\
			\midrule
			\hspace{12mm} 128 & 0.129297 & 0.133322 & 0.133733 \\
			\midrule
			\hspace{12mm} 256 & 0.083291 & 0.091449 & 0.092320 \\
			\bottomrule
		\end{tabular}
		\caption{Distance between exact solution for (\ref{IVP_Inviscid}) and the approximation for (\ref{IVP_Burgers}) with $\alpha = 1.0 \times 10^{-5}$.}
	\end{table}
	\begin{figure}[H]
		\centering
		\includegraphics[width=12cm]{burgers_equation/deterministic/numerical_experiments/inviscid/figures/varios_alphas.png}
		\caption{Exact solution for (\ref{IVP_Inviscid}) and different approximations using (\ref{Galerkin_Euler}) with $N=256$, and $\Delta t = 1.0 \times 10^{-3}$.}
	\end{figure}


%!TEX root =  ../main.tex
\subsection{Properties of Lines}

\objective{Classify linear and linear-like functions, and explain relationships of slopes.}

All non-vertical lines have slope, which is defined as rise over run, $\frac{\Delta x}{\Delta y}$
or $m$.  The initial value
of a linear equation might not be zero, but an ordered pair of the form $(0,b)$.  The variable
$b$ is called the $y-$intercept.  All together, this is the famous $y=mx+b$ form of lines.

\marginfig[-0.1in]{\chapdir/pics/ceiling_function}{The ceiling function.}
However, that formula only makes sense when the origin is within view and we might become
concerned with the output-value when the input is 0. In calculus and other areas of mathematics,
we are more
often simply interested in the slope and some arbitrary point, $(x_1,y_1)$.  Hence we find a majority
of time from here on out, the point-slope form of \index{linear!point-slope form}
linear equation is most helpful, $y-y_1=m(x-x_1)$,
a form which allows us to see the slope, and a (random?) point on the line very easily.
This form is nearly as easy to put in your TI-8* as your old friend slope-intercept: simply add $y_1$
to the other side, and you have a $y=$ form ready to be entered in your grapher.

\subsection{Parallel vs. Perpendicular}
If a line has slope expressed via the fraction in lowest terms $\frac{a}{b}$,
then any other fraction that reduces to the same ratio will produce a line parallel to the first.
Consider the sketch of how to make a line rotated $90^\circ$ clockwise or counterclockwise
to the first.  They will have slopes of $\frac{-b}{a}$ or $\frac{b}{-a}$, which are the same
things.  In short, perpendicular lines have opposite reciprocal slope.\index{linear!parallel}
\index{linear!perpendicular}

\subsection{Linear-like}
Several functions are linear in pieces, and are used in computer programming and other
systems of functions

\subsubsection{Ceiling, Floor, and Round}
Various forms of rounding are present in computer systems and your TI-8*.  Rounding down
in all cases, rounding up in call cases, and the familiar rounding to the closest.

\marginfig[-1.0in]{\chapdir/pics/floor_function}{The floor function}
\subsubsection{Modulus}
The most used function in this respect, and the foundation of an entire species of mathematics
(called Modular Arithmetic) is the modulus function.  It can be thought of as taking two arguments,
one is what to divide by, and the other is what to divide.  The function returns the remainder.  For
example, 10 mod 3 is 1, and 49 mod 7 is 0.  Consider the graph of y=x mod 5.

\subsubsection{Rates of Change}
A constant function is one that never changes (by definition).  \index{constant!derivative}
Algebraically, that means it is
of the form $y=c$, where $c$ is some numbers.  The slope is always zero and the graph
is always a horizontal line.  A vertical line is not a function and its slope is undefined.  Every
other line has a constant rate of change, it's slope.  In calculus terminology,
it's derivative is a constant.


\documentclass{article}
\usepackage[utf8]{inputenc}
\title{Hidden Golden Ratio Integral (@solvingtogether)}
\author{Shreenabh Agrawal}
\date{\today}
\usepackage{amsmath}
\usepackage{geometry}
\geometry{a4paper, portrait, margin=1in}
\usepackage{natbib}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{relsize}
\usepackage[makeroom]{cancel}
\begin{document}
\maketitle

\section{Question}

If $I$ = $$\mathlarger{\LARGE{{\int\limits_{0}^{1}}\frac{1-x^{2}}{x^{2}+\left(x^{2}+1\right)^{2}}\: dx}} $$
Then find $tan(I) + sec(I)$

\section{Solution}

Let us take substitution $x = tan \theta$ so that $dx\ =\ \sec^{2}\theta\ d\theta$, thus our integral becomes,
$$\mathlarger{\LARGE{\int\limits_{0}^{\pi/4}}\frac{1-\tan^{2}\theta}{\tan^{2}\theta\ +\ \sec^{4}\theta}\sec^{2}\theta\: d\theta\ } $$
Now converting $tan \theta$ and $sec \theta$ to respective $sin$ and $cos$ functions, the integral becomes,
$$\mathlarger{\LARGE{\int\limits_{0}^{\pi/4}\frac{\cos^{2}\theta-\sin^{2}\theta}{\sin^{2}\theta\ \cos^{2}\theta\ +\ 1}\ d\theta}} $$
Using double angle properties, the integral can be rewritten as:
$$\mathlarger{\LARGE{\int\limits_{0}^{\pi/4}\frac{\cos2\theta}{\frac{\sin^{2}2\theta}{4}+1} d\theta}} $$
Taking a final substitution $\sin2\theta=u$ so that $2\cos2\theta\ d\theta=du$,
$$\mathlarger{\LARGE{\int\limits_{0}^{1}\frac{2}{4+u^{2}}\ du}} $$
Plugging in the limits, we get,
$${I =\tan^{-1}\left(\frac{1}{2}\right)}$$
For the final answer,
$$\tan I+\sec I=\frac{1}{2}+\frac{\sqrt{5}}{2}$$
$$\boxed{\tan I+\sec I= \phi}$$
Where $\phi$ is the Golden Ratio.

\end{document}

Calculus/solvingtogether's questions/Hidden Golden Ratio Integral.tex


%!TEX root=ClassNotes.tex

\section{Differentiation}
\begin{remark}
	We'll now exclusively work with {\it continuous} functions. As such, we can compute the various limits in this section without requiring any $\epsilon, \delta$ arguments. We'll get back to the $\epsilon, \delta$'s much later. For now, we'll repeated use the following identities,
	\begin{align*}
		\lim \limits_{x \rightarrow a} f(x) + g(x)     & = \lim \limits_{x \rightarrow a} f(x) + \lim \limits_{x \rightarrow a} g(x)                                                                   \\
		\lim \limits_{x \rightarrow a} f(x) \cdot g(x) & = \lim \limits_{x \rightarrow a} f(x) \cdot \lim \limits_{x \rightarrow a} g(x)                                                               \\
		\lim \limits_{x \rightarrow a} f(x) / g(x)     & = \lim \limits_{x \rightarrow a} f(x) / \lim \limits_{x \rightarrow a} g(x)     & \mbox{ assuming }\lim \limits_{x \rightarrow a} g(x) \neq 0
	\end{align*}
	wherever the appropriate limits exist.
\end{remark}

\subsection{Derivative}

The derivative of a function measures the {\it instantaneous rate of change} of the function. Several quantities in physics can be naturally expressed as derivatives, which was the original motivation for defining them.
\begin{center}
	\begin{tabular}{l|l}
		Function  & Derivative   \\ \hline
		Position  & Velocity     \\
		Velocity  & Acceleration \\
		Potential & Force
	\end{tabular}
\end{center}

\begin{definition}
	The {\bf derivative} of $f(x)$ with respect to $x$, at $x=a$, is defined to be the limit
	\begin{align}
		\label{eq:def_derivative}
		f'(a)
		 & :=
		\lim \limits_{h \rightarrow 0}
		\dfrac{f(a+h) - f(a)}{h}
	\end{align}
	If $f'(a)$ exists then we say that $f(x)$ is {\bf differentiable} at $a$.
\end{definition}
One can rewrite Equation \eqref{eq:def_derivative} as
\begin{align}
	\label{eq:def_derivative2}
	f'(a)
	 & =
	\lim \limits_{x \rightarrow a}
	\dfrac{f(x) - f(a)}{x - a}
\end{align}
by making the substitution $h = x - a$.

In this form, it is more explicit that derivative is computing the {\it instantaneous rate of change}, as the numerator $f(x) - f(a)$ is the change in $f$ and the denominator $x - a$ is the change in $x$, so their ratio is the rate of change of $f(x)$ with respect to $x$. Taking the limit $x \rightarrow a$ makes this rate of change {\it instantaneous}. When we want to make the ``free variable'' more explicit, we write the derivative as
\begin{align*}
	\left.\dfrac{df}{dx}\right|_{x = a}
\end{align*}
This notation becomes more relevant in multivariable calculus when there are multiple variables with respect to which one can differentiate the function.

\begin{exercise}
	\label{q:derivatives_1}
	Using the definition, compute the derivatives of the following functions.
	\begin{enumerate}
		\item $f(x) = c$, where $c$ is a real number.
		\item $f(x) = x$.
		\item $f(x) = x^2$.
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Prove that if $f(x)$ is differentiable at $a$ then $f(x)$ is continuous at $x = a$.
\end{exercise}

\begin{theorem}
	\label{thm:derivative_rules}
	If $f$ and $g$ are differentiable at $a$ then\\
	\begin{align*}
		(f+g)'(a)                     & = f'(a) + g'(a)                                                                     \\\\
		(f\cdot g)'(a)                & = f'(a) \cdot g(a) + f(a) \cdot g'(a)                 &  & \mbox{\bf Product Rule}  \\\\
		\left(\dfrac{1}{g}\right)'(a) & = -\dfrac{g'(a)}{g(a)^2}                                                            \\\\
		\left(\dfrac{f}{g}\right)'(a) & = \dfrac{g(a) \cdot f'(a) - g'(a) \cdot f(a)}{g(a)^2} &  & \mbox{\bf Quotient Rule} \\\\
		(f\circ g)'(a)
		                              & = f'(g(a)) \cdot g'(a)                                &  & \mbox{\bf Chain Rule}    \\
	\end{align*}
	where we assume that $g(a) \neq 0$ wherever it is in the denominator.\\
\end{theorem}

\begin{exercise}
	Prove Theorem \ref{thm:derivative_rules}.

	The proof of the Product Rule requires a trick: add and subtract $f(a+h) g(a)$ to the numerator.

	For proving the Quotient Rule notice that $f/g = f \cdot (1/g)$.

	Feel free to read the proof of the Chain Rule from the book.
\end{exercise}

Because of these rules, once we know the derivatives of standard functions like trig functions, exponential functions etc., derivatives are fairly easy to compute. However, computing these standard derivatives is not easy using simply the definition (give it a try!).

Polynomials are the only functions whose derivatives we can compute right now. We'll get back to computing more complicated derivatives once we learn integration and the Fundamental Theorem of Calculus.

\begin{exercise} In this exercise we'll (almost) prove that
	\begin{align}
		\label{eq:polynomial_derivative}
		(x^n)' = n x^{n -1}
	\end{align}
	where $n$ is any real number.

	Note that Exercise \ref{q:derivatives_1} proves this for the cases $n = 0, 1, 2$.
	\begin{enumerate}
		\item Use the identity $(x^{n}) \cdot x = x^{n + 1}$ to prove \eqref{eq:polynomial_derivative} for all positive integers.\footnote{{\bf Optional:} Read about {\bf Mathematical Induction} and use it to write a rigorous proof of this statement.}
		\item Use the identity $\frac{1}{x^m} = x^{-m}$ to prove \eqref{eq:polynomial_derivative} for all negative integers.
		\item Use the $\left( x^{\frac{p}{q}}\right)^q = x^p$ to prove \eqref{eq:polynomial_derivative} for all rational numbers.
	\end{enumerate}
	This proves \eqref{eq:polynomial_derivative} for all rational numbers. We do not yet know enough to extend this proof to irrationals.
	\begin{enumerate}[resume]
		\item Can you think of a way to prove \eqref{eq:polynomial_derivative} for all real numbers? What fact {\it should} be true for your proof to work?
	\end{enumerate}
\end{exercise}

\subsection{Inverse Functions}
We say that functions $f$ and $g$ are {\bf inverses} of each other if the following identities hold:
\begin{align*}
	f (g(x)) & = x \\
	g(f(x))  & = x
\end{align*}

\begin{exercise}
	Verify that if $f$ and $g$ are inverses of each other and $f(x) = y$ then $x = g(y)$. (Hence the name inverse.)
\end{exercise}
This exercise is saying that to find the inverse of $f(x)$ we simply solve $f(x) = y$ for $x$ in terms of $y$, which then gives us $g$.

\begin{exercise}$ $
	\begin{enumerate}
		\item Verify that $f(x) = 2x + 1$ and $g(x) = (x-1)/2$ are inverses of each other. Draw their graphs.
		\item Verify that $f(x) = x^2$ and $g(x) = \sqrt{x}$ are inverses of each other. Draw their graphs.
		\item Verify that $f(x) = e^x$ and $g(x) = \ln x$ are inverses of each other.
		      Draw their graphs.
		\item What relation do you see between the graphs of inverses?
		\item What are the domains of these functions i.e. what is the set of real numbers where the these functions are defined? Why do you think some of the inverses are not defined everywhere?
	\end{enumerate}
\end{exercise}

The inverses of $\sin x$ and $ \cos x$ are denoted $\sin^{-1} (x)$ and $\cos^{-1} (x)$, respectively. More generally, the inverse of a function $f(x)$ is denoted by $f^{-1}(x)$ (which is different from $(f(x))^{-1}$). Note that if $f = g^{-1}(x)$ then $g = f^{-1}(x)$.

\begin{exercise}
	Using the Chain Rule show that
	\begin{align*}
		\left(f^{-1}(a)\right)'= \dfrac{1}{f'(f^{-1}(a))}
	\end{align*}
	Verify this explicitly for $f(x) = 2x+1$ and for $f(x) = x^2$.
\end{exercise}
We'll interpret this geometrically in the next section. We'll later on see that the derivatives of $\sin^{-1}(x)$ and $\ln x$ can expressed in terms of polynomials and radicals. This fact along with the above formula will then allow us to compute the derivatives of $\sin x$ and $e^x$.

\subsection{Mean Value Theorems}
Derivatives have several physical and geometric interpretations which make then widely applicable in all branches of science. We'll see a few of these interpretations in the next few sections.

In this section, we'll prove the analogues of Intermediate Value Theorem for differentiable functions.

\begin{definition}
	A number $c$ is an {\bf absolute maximum} of a function $f$ in the interval $[a,b]$ if $f(c) \ge f(x)$ for all $x \in [a,b]$.
\end{definition}
To prove the Mean Value Theorems we'll need the following {\bf Extreme Value Theorem}. We'll assume it without proof. As for the IVT, the proof of the Extreme Value Theorem fundamentally uses the Completeness of the Real Numbers.

\begin{theorem}[Extreme Value Theorem]
	\label{thm:extreme_value_theorem}
	For every continuous function $f$ and every closed interval $[a,b]$, there exists a real number $c \in [a,b]$ such that $c$ is the absolute maximum of $f$ on $[a,b]$.
\end{theorem}

\begin{remark}
	This theorem is false if we use open intervals. For example, $f(x) = 1/x$ has no absolute maximum on the interval $(0,1)$ even though it is continuous on it.
\end{remark}

\begin{exercise}
	Define an absolute minimum and state the corresponding Extreme Value Theorem.
\end{exercise}

% \begin{exercise}
% 	Let $f$ be a differentiable (and hence also continuous) function. On an interval $[a,b]$, by the Extremal Value Theorem, $f$ has an absolute maxima. Let $c$ be the absolute maximum. Assume that $a < c <b$.
% 	\begin{enumerate}
% 		\item Argue that the function $g(x) = \dfrac{f(x) - f(c)}{x - c}$ is $\ge 0$ for $ a < x < c$ and is $\le 0$ for $c < x < b$.
% 		\item Argue that $\lim \limits_{x \rightarrow c^-}g(x) \ge 0$ and $\lim \limits_{x \rightarrow c^+}g(x) \le 0$.
% 		\item Conclude that $f'(c) = 0$.
% 	\end{enumerate}
% \end{exercise}
%
% Thus we've proven the following theorem
% \begin{theorem}
% 	\label{thm:max_derivative}
% 	Let $f$ be a differentiable function. If $c$ is an absolute maximum for $f$ on the interval $[a,b]$ with $a < c < b$ then $f'(c) = 0$. Similar statement is true for an absolute minimum.
% \end{theorem}

Note that by our definitions an {\it absolute} max/min $c$ of $f$ over $[a,b]$ is also a {\it local} max/min if $a < c < b$. We've already shown that the derivative vanishes at a local max/min, so we have
\begin{theorem}
	\label{thm:max_derivative}
	Let $f$ be a differentiable function. If $c$ is an absolute max/min for $f$ on the interval $[a,b]$ with $a < c < b$ then $f'(c) = 0$.
\end{theorem}

This is all we'll be needing to prove the various Mean Value Theorems.

\begin{theorem}[Rolle's Theorem]
	Let $f$ be a differentiable function. For an interval $[a,b]$, if
	\begin{align*}
		f(a) = f(b)
	\end{align*}
	then there exists a real number $c \in (a,b)$ such that
	\begin{align*}
		f'(c) = 0
	\end{align*}
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.6\textwidth]{RollesTheorem.png}
	\end{figure}
\end{theorem}
\begin{proof}[Proof of Rolle's Theorem]
	This theorem almost immediately follows from Theorem \ref{thm:extreme_value_theorem} and Theorem \ref{thm:max_derivative}. The only issue is that for Theorem \ref{thm:max_derivative} we need $a < c < b$ but the Extreme Value Theorem only guarantees $a \le c \le b$. So we need a small argument to fill this gap.

	By the Extremal Value Theorem, $f$ has an absolute maximum $c_{\max}$ and an absolute minimum $c_{\min}$ on the interval $[a,b]$.

	\begin{exercise}$ $
		\begin{enumerate}
			\item Argue that if $f(c_{\max}) = f(c_{\min})$ then $f$ is a constant function on $[a,b]$.
		\end{enumerate}
		We've already shown that for a constant function, $f'(x)$ equals 0, hence for every $c \in (a,b)$, $f'(c) = 0$ and we're done.

		So now we'll assume that $f(c_{\max}) \neq f(c_{\min})$.

		\begin{enumerate}[resume]
			\item 		Argue that in this case either $a < c_{\max} < b$ or $a < c_{\min}< b$ (possibly both).
			\item 		Finish the proof of Rolle's Theorem.
		\end{enumerate}
	\end{exercise}
\end{proof}

\begin{theorem}[Mean Value Theorem]
	Let $f$ be a differentiable function. For any interval $[a,b]$ there exists a real number $c \in (a,b)$ such that
	\begin{align*}
		f'(c) = \dfrac{f(b) - f(a)}{b - a}
	\end{align*}
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.6\textwidth]{MeanValueTheorem.jpg}
	\end{figure}
\end{theorem}
\begin{proof}[Proof of Mean Value Theorem]
	Mean Value Theorem follows from the Rolle's theorem by an algebraic trick.
	We'll construct a new function $g(x)$ that satisfies the hypotheses of Rolle's theorem.
	\begin{exercise}$ $
		\begin{enumerate}
			\item Find a real number $r$ such that $f(a) - r a = f(b) - r b$.
		\end{enumerate}
		For this value of $r$ define
		\begin{align*}
			g(x) = f(x) - r x
		\end{align*}
		By construction, $g$ is differentiable and satisfies $g(a) = g(b)$ and hence satisfies the hypotheses for Rolle's theorem. So, there exists some real number $ a < c < b$ such that
		\begin{align*}
			g'(c) = 0
		\end{align*}
		\begin{enumerate}[resume]
			\item Show that for this $c$, $f'(c) = \dfrac{f(b) - f(a)}{b - a}$, thereby completing the proof of Mean Value Theorem.
		\end{enumerate}
	\end{exercise}
\end{proof}

We'll see later on that the Mean Value Theorem is a key ingredient in the proof of Taylor series approximation. In fact, Taylor series approximations can be thought of as a generalization of MVT for higher order derivatives.

\subsection{L'Hopital's Rule}
We'll now (almost) provide a proof of L'Hopital's Rule using a stronger version of the Mean Value Theorem.
\begin{theorem}[Easy L'Hopital's Rule]
	Let $f$, $g$ be differentiable functions. Assume that
	\begin{align*}
		f(a) = 0 = g(a) \mbox{ \quad and \quad } g'(a) \neq 0,
	\end{align*}
	where $a$ is a real number. Then
	\begin{align*}
		\lim \limits_{x \rightarrow a} \dfrac{f(x)}{g(x)} = \dfrac{f'(a)}{g'(a)}
	\end{align*}
\end{theorem}
\begin{exercise}
	Notice that $\dfrac{f(x)}{g(x)} = \dfrac{f(x) - f(a)}{g(x) - g(a)}$. Use this to prove the easy version of L'Hopital's Rule. \hint{Divide the numerator and denominator by $x - a$ and take $\lim \limits_{x \rightarrow a}$.}
\end{exercise}

\begin{theorem}[Strong Mean Value Theorem]
	Let $f$, $g$ be differentiable functions. Assume that $g(a) \neq g(b)$ for some real numbers $a$ and $b$. Then there exists a real number $c \in (a,b)$ such that
	\begin{align*}
		\dfrac{f'(c)}{g'(c)} = \dfrac{f(b) - f(a)}{g(b) - g(a)}
	\end{align*}
\end{theorem}
\begin{proof}[Proof of Strong Mean Value Theorem]
	The Strong Mean Value Theorem follows from the Rolle's theorem by the same algebraic trick. We'll construct a new function $h(x)$ that satisfies the hypotheses of Rolle's theorem.
	\begin{exercise}$ $
		\begin{enumerate}
			\item Find a real number $r$ such that $f(a) - r g(a) = f(b) - r g(b)$.
		\end{enumerate}
		For this value of $r$ define
		\begin{align*}
			h(x) = f(x) - r g(x)
		\end{align*}
		By construction, $h$ is differentiable and satisfies $h(a) = h(b)$ and hence satisfies the hypotheses for Rolle's theorem. So, there exists some real number $ a < c < b$ such that
		\begin{align*}
			h'(c) = 0
		\end{align*}
		\begin{enumerate}[resume]
			\item Show that for this $c$, $\dfrac{f'(c)}{g'(c)} = \dfrac{f(b) - f(a)}{g(b) - g(a)}$, thereby completing the proof of the Strong Mean Value Theorem.
		\end{enumerate}
	\end{exercise}
\end{proof}

\begin{exercise}
	Explain how the Mean Value Theorem is a special case of the Strong Mean Value Theorem and how the Rolle's theorem is a special case of the Mean Value Theorem.
\end{exercise}
\begin{theorem}[L'Hopital's Rule]
	Let $f$, $g$ be differentiable functions. Assume that
	\begin{align*}
		f(a) = 0 = g(a)
	\end{align*}
	where $a$ is a real number. Then
	\begin{align*}
		\lim \limits_{x \rightarrow a} \dfrac{f(x)}{g(x)} = \lim \limits_{x \rightarrow a} \dfrac{f'(x)}{g'(x)}
	\end{align*}
\end{theorem}

\begin{proof}[Idea of the proof]
	For any $b$, by the Strong Mean Value Theorem, there exists a $c \in (a,b)$ such that
	\begin{align*}
		\dfrac{f'(c)}{g'(c)}          & = \dfrac{f(b) - f(a)}{g(b) - g(a)}                                \\
		\implies \dfrac{f'(c)}{g'(c)} & = \dfrac{f(b)}{g(b)}               & (\mbox{as } f(a) = 0 = g(a))
	\end{align*}
	As $c$ has to lie between $a$ and $b$, when we take the limit $b \rightarrow a$ we must also have $c \rightarrow a$. Taking limits of both sides we get the required result.
	\begin{align*}
		\lim \limits_{c \rightarrow a} \dfrac{f'(c)}{g'(c)} = \lim \limits_{b \rightarrow a}  \dfrac{f(b)}{g(b)}
	\end{align*}
\end{proof}

\begin{remark}
	The last argument in the above proof is not rigorous. While the idea is correct, to make it rigorous we need generalized notions of limits called {\bf limit superior} ($\lim \sup$) and {\bf limit inferior} ($\lim \inf$). This is unfortunately beyond the scope of this class.
\end{remark}

There are many more variants of L'Hopital's Rule: for one-sided limits, for $a = \infty$, for $ \lim \limits_{x \rightarrow a} f(a)= \infty =\lim \limits_{x \rightarrow a} g(a) $ etc. They can all be proven using similar techniques.

\subsection{First Derivative Test}
In this section, we'll prove the First Derivative Test for differentiable functions.

\begin{definition}
	We say that a function $f$ is {\bf increasing} on an interval $[a,b]$, if for all $x$, $y$ in $[a,b]$ if $x < y$ then $f(x) \le f(y)$.
		We say that $f$ is {\bf strictly increasing} on $[a,b]$, if for all $x$, $y$ in $[a,b]$ if $x < y$ then $f(x) < f(y)$.
\end{definition}
\begin{exercise}
	Come up with a definition for when a function is {\bf decreasing}, {\bf strictly decreasing} on an interval.
\end{exercise}
\begin{exercise}
	Find the interval(s) on which the function $x^2$ is increasing/decreasing? What about the function $x^3$?
\end{exercise}

For differentiable functions, we can determine if a function is increasing or decreasing by calculating it's first derivative.
\begin{exercise}$ $
	\label{q:increasing_decreasing_derivative}
	\begin{enumerate}
		\item {\bf Optional:} Let $g(x)$ be a function such that $\lim \limits_{x\rightarrow c} g(x)$ exists for some real number $c$ in the interval $(a,b)$. Prove that if $g(x) \ge 0$ for all $x$ in $[a,b]$, then $\lim \limits_{x\rightarrow c} g(x)$ is also $\ge 0$.\hint{Proof by Contradiction.}\\ {\it (You can assume this part to be true if you choose not to prove it.)}
		\item Let $f(x)$ be an increasing function on $[a,b]$ and let $c \in (a,b)$. Prove that $g(x) \ge 0$ on the interval $[a,b]$ where $g(x)$ is the function
		      \begin{align*}
			      g(x) = \dfrac{f(x) - f(c)}{x - c}
		      \end{align*}
		\item Let $f(x)$ be a differentiable function that is increasing on the interval $[a,b]$. Prove that all $c \in (a,b)$, $f'(c) \ge 0$. Similarly, for decreasing functions.
	\end{enumerate}
\end{exercise}
Note that this agrees with the interpretation of a derivative as the {\it instantaneous} rate of change of a function: an increasing function should have a positive {instantaneous} rate of change and a decreasing function should have a negative {instantaneous} rate of change. The converse of the above statement is almost true, but for this we need an extra assumption on $f$ and the proof is a bit more intricate.

\begin{prop}
	\label{thm:increasing_positive_derivative}
	If $f'(x) > 0$ for all $c \in (a,b)$ then $f$ is strictly increasing on $(a,b)$. Similarly, for $f'(c) < 0$.
\end{prop}

\begin{exercise}
	Let $f'(x) > 0$ for all $c \in (a,b)$. Prove by Contradiction, that for all $x$, $y \in (a,b)$, if $x < y$ then $f(x) < f(y)$.\hint{Use the Mean Value Theorem.}
\end{exercise}

\begin{exercise}
	Use the above Proposition to find the range in which the function $x^n$ is increasing/decreasing, where $n$ is a positive integer.
\end{exercise}

\begin{definition}A real number $c$ is said to be a {\bf local maximum} of a function $f$ if there is some interval $(a,b)$ containing $c$ such that if $a < x < b$ then $f(c) \ge f(x)$.
\end{definition}
\begin{exercise}
	Come up with a definition for {\bf local minimum}.
\end{exercise}
\begin{exercise} Assume that the function $f(x)$ is differentiable and it's derivative $f'(x)$ is continuous. Using Proposition \ref{thm:increasing_positive_derivative} prove that if $c$ is a local maximum of $f(x)$ then $f'(c) = 0$.\hint{Proof by Contradiction.} Similarly for a local minimum.
\end{exercise}
\begin{remark}
	We need $f'(x)$ to be continuous to ensure that if $f'(c) > 0$ then $f'(x) > 0$ for every $ x \in (a,b)$, where $(a,b)$ is some interval containing $c$.
\end{remark}

Combining everything in this section we have proven the following Theorem, also called the {\bf First Derivative Test}.
\begin{theorem}[First Derivative Test]
	Let $f(x)$ be a differentiable function whose derivative $f'(x)$ is a continuous function.

	For a real number $c$,
	\begin{align*}
		f	\mbox{ has a local max/min at } c & \implies f'(c) = 0   \\
		f \mbox{ is increasing near } c    & \implies f'(c) \ge 0 \\
		f \mbox{ is decreasing near } c    & \implies f'(c) \le 0
	\end{align*}
	Conversely,
	\begin{align*}
		f'(c) > 0 & \implies f \mbox{ is increasing near } c \\
		f'(c) < 0 & \implies f \mbox{ is decreasing near } c
	\end{align*}
	where by ``near $c$'' means for some interval $(a,b)$ containing $c$.
\end{theorem}

A real number $c$ satisfying $f'(c) = 0$ is called a {\bf critical point} of $f$. The First Derivative Test says that a local max/min has to be a critical point, but it does not say that a critical point has to be a local max/min. A standard example of this failure is $c=0$ for the function $f(x) = x^3$.
\begin{align*}
	\mbox{Local min/max}  & \Rightarrow \mbox{Critical point}    \\
	\mbox{Critical point} & \not\Rightarrow \mbox{Local min/max}
\end{align*}
There are ways to fix this using higher derivatives. We'll do this later using Taylor Series approximations.

\subsubsection*{Optional Problems}
There are multiple ways of proving Proposition \ref{thm:increasing_positive_derivative}. However, all the methods require some additional techniques which we have not yet developed.

\begin{exercise}
	Try to write down a rigorous proof of Proposition \ref{thm:increasing_positive_derivative}. What statement {\it should} be true for your proof to work?
\end{exercise}

\subsection{Convexity and Concavity}
The second derivative of a function $f(x)$ is defined as
\begin{align*}
	f''(x) = \left(f'(x)\right)'
\end{align*}
If the second derivative exists then we say that a function is {\bf twice differentiable}.
More generally, the $n^{th}$ derivative $f^{(n)}(x)$ is defined as
\begin{align*}
	f^{(n)}(x) = \left(f^{(n-1)}(x)\right)'
\end{align*} If all derivatives exist then we say that the function is {\bf smooth} or {\bf infinitely differentiable}. The standard functions like polynomials, trig functions, exponentials, and logarithms are all smooth wherever they're defined.\\

If a function $f$ is twice differentiable, then the second derivative of $f$ measures it's convexity/concavity.
\begin{definition}
	A function $f$ is said to be {\bf convex} (or {\bf concave upwards}) on an interval, if for every real numbers $a$, $b$ in the interval, the graph of $f(x)$ lies below the line joining $(a,f(a))$ to $(b,f(b))$.
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.6\textwidth]{ConvexFunction.png}
	\end{figure}
\end{definition}
Convex functions play an important role in many areas of mathematics, especially in the study of optimization problems.\\

The following theorem establishes the connection between convexity and the second derivative.
\begin{theorem}
	\label{thm:convex_functions}
	Let $f$ be a twice differential function.

	If $f$ is convex on an interval then $f''(x) \ge 0$ on that interval. Conversely, if for every real number $x$ in some interval, $f''(x) > 0$, then $f$ is convex on that interval.
\end{theorem}

\begin{remark}
	To avoid clutter, we'll often drop the terms {\it on an interval, in some interval} etc. in the following proof. Just keep in mind that there is an ambient interval and all our variables belong to that interval.
\end{remark}

\noindent \begin{tabular}{|p{\textwidth}|}
  \hline \\{\it Once you have solved all the exercises below, submit your final solutions as one single logically coherent proof. Also include all the text that is in between the exercises, so that you yourself see how all the pieces fit together. As before, this is to practice writing complex proofs.}\\\\
  \hline
\end{tabular}

\begin{proof}
	Let $f$ be a twice differential function.
	\begin{exercise}
		For real numbers $a$, $b$ denote by $m_{a,b}$ the slope of the line joining the points $(a,f(a))$ and $(b,f(b))$.
		Find a formula for $m_{a,b}$.
	\end{exercise}

	\noindent We'll first prove the forward direction.\\
	\noindent {\bf Claim:} {\it 	If $f$ is convex on an interval then $f''(x) \ge 0$ on that interval.}

	We'll prove that $f'(x)$ is an increasing function which will then imply that $f''(x) \ge 0$ (as the derivative of an increasing function is $\ge 0$).\\

	Let $f$ be a convex function. Let $a < b$ be two real numbers.
	\begin{exercise}
		Let $a < x < y$ be real numbers.
		\begin{enumerate}
			\item {\it Geometrically} argue that $m_{a,x} < m_{a,y}$.
			\begin{figure}[H]
				\centering
				\includegraphics[width=0.6\textwidth]{ConvexFunctionProof1.png}
			\end{figure}
			\item Using the formula for $m_{a,x}$, $m_{a,y}$ and the above inequality argue that $$f'(a) \le m_{a,y}$$ for real numbers $a < y$.
		\end{enumerate}
	\end{exercise}

	\begin{exercise}
		Let $x < y < b$ be real numbers.
		\begin{enumerate}
			\item {\it Geometrically} argue that $m_{x,b} < m_{y,b}$.
			\item Using the formula for $m_{y,b}$, $m_{x,b}$ and the above inequality argue that $$m_{x,b} \le f'(b) $$ for real numbers $x < b$.
		\end{enumerate}
	\end{exercise}

	\begin{exercise}
		Combine the results from the previous two exercises to conclude that $f'(a) \le f'(b)$.
	\end{exercise}
	Hence $f'(x)$ is an increasing function and hence $f''(x) \ge 0$.\\

	\noindent We'll next prove the other direction. \\
	\noindent {\bf Claim:} {\it If for every real number $x$ in some interval, $f''(x) > 0$, then $f$ is convex on that interval.}

	We'll prove this by {\it contradiction}. Let $f$ be a function such that $f''(x) > 0$ on some interval. This implies that $f'(x)$ is an increasing function on that interval. This is the statement that we'll contradict.\\

	Assume on the contrary that $f$ is not convex. This means that for some real numbers $a < b$, there exists a number $c$ with $a < c < b$ such that the point $(c,f(c))$ lies above the line joining $(a,f(a))$ and $(b,f(b))$.

	\begin{exercise}
		Draw a picture.
	\end{exercise}
	Applying the Mean Value Theorem to the intervals $[a,c]$ and $[c,b]$ gives us real numbers $d_1 \in (a,c)$ and $d_2 \in (c,b)$ such that
	\begin{align*}
		f'(d_1) = m_{a,c} \mbox{ and } f'(d_2) = m_{c,b}
	\end{align*}

	\begin{exercise} $ $
		\begin{enumerate}
			\item Draw a picture.
			\item {\it Geometrically} argue that $f'(d_1) > f'(d_2)$.
			\item Why is this a contradiction?
		\end{enumerate}
	This completes a proof of the other direction, and hence of the main theorem.
	\end{exercise}
\end{proof}

\subsubsection{Second Derivative Test}
Let $f(x)$ be a twice differentiable function. If a real number $c$ is a local minimum of $f$ then $f$ is convex on some interval containing $c$. A similar statement is true when $c$ is a local maximum. Combining this observation with the previous Theorem and the First Derivative Test, we get the {\bf Second Derivative Test}.

\begin{theorem}[Second Derivative Test]
	Let $f(x)$ be a twice differentiable function whose second derivative $f''(x)$ is a continuous function.
	\begin{align*}
		f	\mbox{ has a local min at } c &\implies f'(c) = 0 \mbox{ and } f''(c) \ge 0 \\
			f	\mbox{ has a local max at } c &\implies f'(c) = 0 \mbox{ and } f''(c) \le 0
	\end{align*}
	Conversely,
	\begin{align*}
		f'(c) = 0 \mbox{ and } f''(c) > 0 & \implies f	\mbox{ has a local min at } c  \\
		f'(c) = 0 \mbox{ and } f''(c) < 0 & \implies f	\mbox{ has a local max at } c
	\end{align*}
\end{theorem}
We need continuity of $f''(x)$ so that $f''(c) > 0$ implies $f''(x)>0$ on some interval containing $c$.

\subsubsection*{Optional Problems}
\begin{exercise}$ $
	\begin{enumerate}
		\item Find an example of a function which is convex but $f''(x)$ is not always $> 0$.
		\item Why does the proof of Theorem \ref{thm:convex_functions} only proves  that {\it if $f$ is convex then $f''(x) \ge 0$} (instead of $f''(x) > 0$)?
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Several times in this section we used {\it geometric} arguments. This is less preferable than algebraic arguments as geometric arguments are not rigorous and hence susceptible to being incorrect.
	\begin{enumerate}
		\item Convert the definition of a convex function into an algebraic equation involving inequalities.
		\item Use this equation to rigorously prove that, say, $m_{a,x} < m_{a,y}$ if $a < x < y$.
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Let $f$ be a convex function on an interval containing the real numbers $a$, $b$. Show that for all $ t \in [0,1]$,
	\begin{align*}
		f(t a + (1-t) b) \le t f(a) + (1- t) f(b)
	\end{align*}
	This inequality is called {\bf Jensen's inequality} for convex functions.
\end{exercise}


\chapter{Optimization methods}
\label{chap:Optimization-methods}

\section{Convexity}
\label{sec:Convexity}

\begin{definition}
(\textbf{Convex set}) We say aset $\mathcal{S}$ is convex if for any $\vec{x}_1, \vec{x}_2 \in \mathcal{S}$, we have
\begin{equation}
\lambda\vec{x}_1+(1-\lambda)\vec{x}_2 \in \mathcal{S}, \forall \lambda \in [0,1]
\end{equation}
\end{definition}

\begin{definition}
(\textbf{Convex function}) A function $f(\vec{x})$ is called convex if its \textbf{epigraph}(the set of points above the function) defines a convex set. Equivalently, a function $f(\vec{x})$ is called convex if it is defined on a convex set and if, for any $\vec{x}_1, \vec{x}_2 \in \mathcal{S}$, and any $\lambda \in [0,1]$, we have
\begin{equation}
f(\lambda\vec{x}_1+(1-\lambda)\vec{x}_2) \leq \lambda f(\vec{x}_1)+(1-\lambda)f(\vec{x}_2)
\end{equation}
\end{definition}

\begin{definition}
A function $f(\vec{x})$ is said to be \textbf{strictly convex} if the inequality is strict
\begin{equation}
f(\lambda \vec{x}_1 + (1 - \lambda)\vec{x}_2) < \lambda f(\vec{x}_1) + (1 - \lambda)f(\vec{x}_2)
\end{equation}
\end{definition}

\begin{definition}
A function $f(\vec{x})$ is said to be (strictly) \textbf{concave} if $-f(\vec{x})$ is (strictly) convex.
\end{definition}

\begin{theorem}
If $f(x)$ is twice differentiable on $[a, b]$ and $f''(x) \geq 0$ on $[a, b]$ then $f(x)$ is convex on $[a, b]$.
\end{theorem}

\begin{proposition}
$\log(x)$ is strictly convex on $(0, \infty)$.
\end{proposition}

Intuitively, a (strictly) convex function has a “bowl shape”, and hence has a unique global minimum $x^*$ corresponding to the bottom of the bowl. Hence its second derivative must be positive everywhere, $\frac{\mathrm{d}^2}{\mathrm{d}x^2}f(x)>0$. A twice-continuously differentiable, multivariate function $f$ is convex iff its Hessian is positive definite for all $\vec{x}$. In the machine learning context, the function $f$ often corresponds to the NLL.

Models where the NLL is convex are desirable, since this means we can always find the globally optimal MLE. We will see many examples of this later in the book. However, many models of interest will not have concave likelihoods. In such cases, we will discuss ways to derive locally optimal parameter estimates.

\section{Gradient descent}
\label{sec:Gradient-descent}

\subsection{Stochastic gradient descent}

\begin{algorithm}[htbp]
	\SetKwInOut{Input}{input}\SetKwInOut{Output}{output}

    \Input{Training data $\mathcal{D}=\left\{(\vec{x}_i,y_i) | i=1:N\right\}$}
	\Output{A linear model: $y_i=\vec{\theta}^T\vec{x}$}
    $\vec{w} \leftarrow 0;\; b \leftarrow 0;\; k \leftarrow 0$\;
    \While{no mistakes made within the for loop}{
        \For{$i\leftarrow 1$ \KwTo $N$}{
			\If{$y_i(\vec{w} \cdot \vec{x}_i+b) \leq 0$}{
				$\vec{w} \leftarrow \vec{w}+\eta y_i \vec{x}_i$\;
				$b \leftarrow b+\eta y_i$\;
				$k \leftarrow k+1$\;
			}
		}
    }
\caption{Stochastic gradient descent}
\end{algorithm}

\subsection{Batch gradient descent}

\subsection{Line search}
The \textbf{line search}\footnote{\url{http://en.wikipedia.org/wiki/Line_search}} approach first finds a descent direction along which the objective function $f$ will be reduced and then computes a step size that determines how far $\vec{x}$ should move along that direction. The descent direction can be computed by various methods, such as gradient descent(Section \ref{sec:Gradient-descent}), Newton's method(Section \ref{sec:Newtons-method}) and Quasi-Newton method(Section \ref{sec:Quasi-Newton-method}). The step size can be determined either exactly or inexactly.

\subsection{Momentum term}

\section{Lagrange duality}

\subsection{Primal form}
Consider the following, which we'll call the \textbf{primal} optimization problem:
\begin{eqnarray}
xyz
\end{eqnarray}

\subsection{Dual form}

\section{Newton's method}
\label{sec:Newtons-method}
\begin{align}
f(\vec{x})& \approx f(\vec{x}_k)+\vec{g}_k^T(\vec{x}-\vec{x}_k)+\dfrac{1}{2}(\vec{x}-\vec{x}_k)^T\vec{H}_k(\vec{x}-\vec{x}_k) \nonumber \\
\text{where } & \vec{g}_k \triangleq \vec{g}(\vec{x}_k)=f'(\vec{x}_k), \vec{H}_k \triangleq \vec{H}(\vec{x}_k), \nonumber \\
              & \vec{H}(\vec{x}) \triangleq \left[\dfrac{\partial^2 f}{\partial x_i \partial x_j}\right]_{D \times D} \quad \text{(Hessian matrix)} \nonumber \\
f'(\vec{x})& = \vec{g}_k+\vec{H}_k(\vec{x}-\vec{x}_k)=0 \Rightarrow  \label{eqn:newton-stationary-point} \\
\vec{x}_{k+1}& = \vec{x}_k-\vec{H}_k^{-1}\vec{g}_k
\end{align}

\begin{algorithm}[htbp]
    Initialize $\vec{x}_0$ \\

	\While{(!convergency)} {
        Evaluate $\vec{g}_k=\nabla f(\vec{x}_k)$ \\
        Evaluate $\vec{H}_k=\nabla^2 f(\vec{x}_k)$ \\
        $\vec{d}_k=-\vec{H}_k^{-1}\vec{g}_k$ \\
        Use line search to find step size $\eta_k$ along $\vec{d}_k$ \\
        $\vec{x}_{k+1}=\vec{x}_k+\eta_k\vec{d}_k$
	}

\caption{Newton’s method for minimizing a strictly convex function}
\end{algorithm}

\section{Quasi-Newton method}
\label{sec:Quasi-Newton-method}
From Equation \ref{eqn:newton-stationary-point} we can infer out the \textbf{quasi-Newton condition} as follows:
\begin{align}
f'(\vec{x})-\vec{g}_k & = \vec{H}_k(\vec{x}-\vec{x}_k) \nonumber \\
\vec{g}_{k-1}-\vec{g}_k & = \vec{H}_k(\vec{x}_{k-1}-\vec{x}_k) \Rightarrow \nonumber \\
\vec{g}_k- \vec{g}_{k-1} & = \vec{H}_k(\vec{x}_k-\vec{x}_{k-1}) \nonumber \\
\vec{g}_{k+1}- \vec{g}_k & = \vec{H}_{k+1}(\vec{x}_{k+1}-\vec{x}_k) \quad \text{(quasi-Newton condition)} \label{eqn:quasi-Newton-condition}
\end{align}

The idea is to replace $\vec{H}_k^{-1}$ with a approximation $\vec{B}_k$, which satisfies the following properties:
\begin{enumerate}
\item{$\vec{B}_k$ must be symmetric}
\item{$\vec{B}_k$ must satisfies the quasi-Newton condition, i.e., $\vec{g}_{k+1} - \vec{g}_k= \vec{B}_{k+1}(\vec{x}_{k+1}-\vec{x}_k)$. \\
Let $\vec{y}_k=\vec{g}_{k+1}- \vec{g}_k$, $\vec{\delta}_k=\vec{x}_{k+1}-\vec{x}_k$, then
\begin{equation}
\vec{B}_{k+1}\vec{y}_k=\vec{\delta}_k \label{eqn:secant-equation}
\end{equation}
}
\item{Subject to the above, $\vec{B}_k$ should be as close as possible to $\vec{B_{k-1}}$.}
\end{enumerate}

Note that we did not require that $\vec{B}_k$ be positive definite. That is because we can show that it must be
positive definite if $\vec{B_{k-1}}$ is. Therefore, as long as the initial Hessian approximation $\vec{B}_0$ is positive definite,
all $\vec{B}_k$ are, by induction.

\subsection{DFP}
Updating rule:
\begin{equation}
\vec{B}_{k+1}=\vec{B}_k+\vec{P}_k+\vec{Q}_k
\end{equation}

From Equation \ref{eqn:secant-equation} we can get
\begin{equation*}
\vec{B}_{k+1}\vec{y}_k=\vec{B}_k\vec{y}_k+\vec{P}_k\vec{y}_k+\vec{Q}_k\vec{y}_k=\vec{\delta}_k
\end{equation*}

To make the equation above establish, just let
\begin{align*}
\vec{P}_k\vec{y}_k & = \vec{\delta}_k \\
\vec{Q}_k\vec{y}_k & = -\vec{B}_k\vec{y}_k
\end{align*}

In DFP algorithm, $\vec{P}_k$ and $\vec{Q}_k$ are
\begin{align}
\vec{P}_k &= \dfrac{\vec{\delta}_k\vec{\delta}_k^T}{\vec{\delta}_k^T\vec{y}_k} \\
\vec{Q}_k &= -\dfrac{\vec{B}_k\vec{y}_k\vec{y}_k^T\vec{B}_k}{\vec{y}_k^T\vec{B}_k\vec{y}_k}
\end{align}

\subsection{BFGS}
Use $\vec{B}_k$ as a approximation to $\vec{H}_k$, then the quasi-Newton condition becomes
\begin{equation}
\vec{B}_{k+1}\vec{\delta}_k=\vec{y}_k
\end{equation}

The updating rule is similar to DFP, but $\vec{P}_k$ and $\vec{Q}_k$ are different. Let
\begin{align*}
\vec{P}_k\vec{\delta}_k & = \vec{y}_k \\
\vec{Q}_k\vec{\delta}_k & = -\vec{B}_k\vec{\delta}_k
\end{align*}

Then
\begin{align}
\vec{P}_k &= \dfrac{\vec{y}_k\vec{y}_k^T}{\vec{y}_k^T\vec{\delta}_k} \\
\vec{Q}_k &= -\dfrac{\vec{B}_k\vec{\delta}_k\vec{\delta}_k^T\vec{B}_k}{\vec{\delta}_k^T\vec{B}_k\vec{\delta}_k}
\end{align}

\subsection{Broyden}
Broyden's algorithm is a linear combination of DFP and BFGS.


\subsection{Taylor Series}
\noindent
A Taylor series as a way of approximating a function about a point $x=a$ using polynomials. The first approximation just keeps the same value at $x=a$, the second approximation keep the same value and first derivative, etc.
\begin{definition}
	\begin{equation*}
		f(x) = f(a)+f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \ldots +  \frac{f^{(n)}(a)}{n!}(x-a)^n + \ldots
	\end{equation*}
\end{definition}
\noindent
If we approximate a function about $x=0$, we call this a Maclaurin series. Below are some common Maclaurin series, and their radii of convergence if applicable.
\begin{enumerate}[label=]
	\item \begin{equation*}
		e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \ldots
	\end{equation*}
	\item \begin{equation*}
		\sin{x} = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \ldots
	\end{equation*}
	\item \begin{equation*}
		\cos{x} = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \ldots
	\end{equation*}
	\item \begin{equation*}
		\frac{1}{1+x} = 1 - x + x^2 - \ldots \text{, where } \abs{x} < 1
	\end{equation*}
	\item \begin{equation*}
		\ln{(1+x)} = x - \frac{x^2}{2} + \frac{x^3}{3} - \ldots \text{, where } \abs{x} < 1
	\end{equation*}
\end{enumerate}

\input{./backgroundReview/singleVariableCalc/eulersIdentity.tex}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: ODE Applications}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to the application of ODEs to solve problems.
\end{abstract}
\maketitle

\begin{exercise}
Find the orthogonal trajectories of the given family (represented by the purple curves in the plot below):
\[ y = C e^{-x}. \]
Write your answer as $f(y) = g(x) + C$, where both $f(y)$ and $g(x)$ vanish at the origin. Moreover, you should multiply by constants if necessary so that $f(1) = 1/2$ and $g(1) = 1$.
\[ \answer{\frac{1}{2} y^2} = \answer{x} + C. \]
\begin{center}
\begin{image}
\includegraphics[width=4in]{images/ortho01.png}
\end{image}
\end{center}

\end{exercise}

\begin{exercise}
Find the orthogonal trajectories of the given family (represented by the purple curves in the plot below):
\[ y = x - \frac{x^3}{3} + C. \]
Write your answer as $y = f(x) + C$, where $f(0) = 0$
\[ y = \answer{\frac{1}{2} \ln \left| \frac{1-x}{1+x} \right|} + C. \]
\begin{center}
\begin{image}
\includegraphics[width=4in]{images/ortho02.png}
\end{image}
\end{center}

\end{exercise}

\begin{exercise}
Find the orthogonal trajectories of the given family:
\[ y = \tan \left( C + x + \frac{x^3}{3} \right) \]
Write your answer as $f(y) = g(x) + C$, where both $f(y)$ and $g(x)$ vanish at the origin. Moreover, you should multiply by constants if necessary so that $f(1) = 4/3$ and $g(1) = -\pi/4$.
\[ \answer{y + \frac{y^3}{3}} = \answer{- \arctan x} + C \]
\end{exercise}

\begin{exercise}
It takes about three days to defrost a 10 pound imitation frozen turkey in a home refrigerator. Suppose that the initial temperature is $-16^\circ$C and the ambient temperature is $2^\circ$C inside the refrigerator. If after exactly two days, the temperature of the frozen turkey is $0^\circ$C, find a formula for the temperature $y(t)$ (in degrees Celsius) as a function of time for all $t > 0$, where $t$ is measured in days
\[ y = \answer{ -18 e^{-t \ln 3} + 2}. \]
\end{exercise}

\begin{exercise}
At time $t = 0$, a cup of tea has temperature $46^\circ$C. Ten minutes later, its temperature is $34^\circ$C. Ten minutes after that, the temperature is then $28^\circ$C. Assuming the temperature obeys Newton's Law of cooling, what is the temperature (in degrees Celsius) as a function of time $y(t)$ for all $t > 0$, where $t$ is measured in minutes?
\[ y(t) = \answer{24 e^{-t \ln(2)/10} + 22}  \]
\begin{hint}
Suppose $A$ is the ambient temperature. We know that
\[ \frac{48-A}{34 - A} = \frac{34 - A}{28 - A} \]
because every ten minutes, the \textit{difference} between tea temperature and room temperature decreases by the same factor.
\end{hint}
\end{exercise}

\begin{exercise}
A 1000 liter tank is filled with a sugar solution: 200 kilograms of sugar dissolved in 1000 liters of pure water. The solution in the tank is kept thoroughly mixed at all times. At time $t = 0$, the attendants begin adding 1 liter per minute of dissolved sugar at a concentration of 0.1 kilograms per liter. At the same time, One liter per minute is drained from the tank to keep the overall volume of solution in the tank at 1000 liters. Find a function $S(t)$ for the total amount of sugar in the tank (in kilograms) a time $t > 0$, where $t$ is measured in minutes.
\[ S(t) = \answer{ 100 + 100 e^{-t/1000}}. \]
\end{exercise}

\begin{exercise}
A small flask contains 10mL of solvent in which 1 gram of substance X is initially dissolved. The solution is slowly drained at a rate of 3mL per hour while 4mL per hour of pure solvent is added (with the whole solution being kept thoroughly mixed). Find a formula $X(t)$ for the amount of substance $X$ (in grams) sill in the beaker at time $t$, where $t$ is measured in hours.
\[ X(t) = \answer{ \frac{1000}{(t+10)^3}}. \]
\end{exercise}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2016C.15]

A tank contains 100 gallons of water in which 300 pounds of salt are dissolved. At some initial time, workers begin pumping in fresh water, i.e., containing no salt, at a rate of 10 gallons per minute. During the process, the tank is kept well-mixed and 20 gallons per minute of the resulting saltwater are pumped out of the tank (in particular, note that the tank will be empty after 10 minutes). Find the total amount of salt in the tank (measured in pounds) which remains 9 minutes after the process starts.
\begin{multiplechoice}
\choice{\(1\)}
\choice{\(2\)}
\choice[correct]{\(3\)}
\choice{\(4\)}
\choice{\(5\)}
\choice{\(6\)}
\end{multiplechoice}

\end{question}

\end{document}


\subsection{Dentable sets}\label{subsec:dentable_sets}

\begin{definition}\label{def:dentability}\mcite[def. 5.1]{Phelps1993}
  A subset \( A \) of a Banach space \( X \) is called \term{dentable} if it admits slices of arbitrarily small diameter, i.e. for every \( \varepsilon > 0 \) there exist a functional \( x^* \in X^* \) and a diameter \( \alpha > 0 \), such that \( \diam S(x^*, A, \alpha) < \varepsilon \).

  Weak* dentability is defined in an obvious way.
\end{definition}

\begin{definition}\label{def:radon-nikodym-property}\mcite[def. 5.2]{Phelps1993}
  The space \( X \) is said to have the \term{Radon-Nikodym property (RNP)} if every nonempty bounded set \( A \) of \( X \) is dentable.
\end{definition}

\begin{proposition}\label{thm:weak_dentable_sets_are_dentable}
  Let \( X \) be a Banach space and \( A^* \subseteq X^* \) be a weak*-dentable set. Then \( A^* \) is dentable in \( X^* \).
\end{proposition}
\begin{proof}
  Let \( \varepsilon > 0 \) and let \( x \in X \) and \( \alpha > 0 \) be such that \( \diam S^*(x, A^*, \alpha) < \varepsilon \).
  We denote by \( J(x) \) the embedding of \( x \in X \) into the double-dual \( X^{**} \) and by \( T(J(x), A^*, \alpha) \) the slice of \( A^* \) in \( X^* \). We have that
  \begin{balign*}
    S^*(x, A^*, \alpha)
     & =
    \{ x^* \in A^* \colon \inprod {x^*} x > \sigma_{A^*}(x) - \alpha \}
    =    \\ &=
    \{ x^* \in A^* \colon \inprod {x^*} x > \sup \{ \inprod {y^*} x \colon y^* \in A^* \} - \alpha \}
    =    \\ &=
    \{ x^* \in A^* \colon \inprod {J(x)} {x^*} > \sup \{ \inprod {J(x)} {y^*} \colon y^* \in A^* \} - \alpha \}
    =
    T(J(x), A^*, \alpha),
  \end{balign*}

  Since \( J \) is an isometry, this equality implies that
  \begin{equation*}
    \diam T(J(x), A^*, \alpha) = \diam S(x, A^*, \alpha) < \varepsilon.
  \end{equation*}

  Hence, \( A^* \) admits arbitrarily small slices in \( X^* \), i.e. it is dentable in \( X^* \).
\end{proof}


\section{Camera Model}
\subsection{Camera Matrix}
\subsubsection{Pinhole camera model}
\includegraphics[width=\columnwidth]{pictures/pinholecamera}
Projection on image plane
$$\begin{pmatrix}
fX\\
fY\\
Z\\
\end{pmatrix} =
\underbrace{\begin{bmatrix}
	f&0&0&0\\
	0&f&0&0\\
	0&0&1&0\\
	\end{bmatrix} }_{= diag(f,f,1)[I|0]} \begin{pmatrix}
X\\
Y\\
Z\\
1\\
\end{pmatrix} $$
By dividing $(fX,fY,Z)^\top$ by $Z$ we get $(x,y,1)^\top$

\subsubsection{Principal Point offset}
Moving the point into the middle of the picture

$$\begin{pmatrix}
fX+Zp_x\\
fY+Zp_y\\
Z\\
\end{pmatrix} = \begin{bmatrix}
f&0&p_x&0\\
0&f&p_y&0\\
0&0&1&0\\
\end{bmatrix} \begin{pmatrix}
X\\
Y\\
Z\\
1\\
\end{pmatrix} $$
By dividing $(fX+Zp_x,fY+Zp_y,Z)^\top$ by $Z$ we get $(x+p_x,y+p_y,1)^\top$

\subsubsection{Kalibration matrix K}
Intrinsic camera parameters

$$ \underbrace{K = \begin{bmatrix}
f&0&p_x\\
0&f&p_y\\
0&0&1\\
\end{bmatrix}}_{simple: a=1 \text{ and } s=0}  \qquad \underbrace{K = \begin{bmatrix}
f&s&p_x\\
0&af&p_y\\
0&0&1\\
\end{bmatrix}}_{general}$$

$f$: focal length\\
$s$: skew between the sensor axes due to the sensor not being mounted perpendicular to the optical axis\\
$a$: aspect ratio between $f_x$ \& $f_y$

\subsubsection{Camera Rotation + Shift}
Extrinsic camera parameters
$$ [R|t] $$
$R$: 3x3 Rotation matrix\\
$t$: 3x1 Translation vector

\subsubsection{Camera Matrix / Projective Camera}
$$ P = K[R|t] $$

$$ \lambda \begin{bmatrix}
u\\
v\\
1\\
\end{bmatrix} = P \begin{bmatrix}
X\\
Y\\
Z\\
W\\
\end{bmatrix} = K[R|t] \begin{bmatrix}
X\\
Y\\
Z\\
W\\
\end{bmatrix} $$

Full matrix has 11 DOF (5 intrinsic +3 rot +3 trans)

3x4 matrix

$$ \tilde{P} = \begin{bmatrix}
K&0\\
0&1\\
\end{bmatrix} \begin{bmatrix}
R&t\\
0^T&1\\
\end{bmatrix} $$
4x4 matrix

$$P = K[R|-RC]$$

\subsection{Camera Calibration with Direct Linear Transform (DLT)}
The DLT method uses a set  of  control  points  whose  object  space/plane  coordinates  are  already  known.  The  control  points  are  normally  fixed  to  a  rigid  frame,  known  as  the  calibration  frame.    The  problem  is  essentially  to  calculate  the  mapping  between  the  2D  image  space  coordinates  (x,y)  and  the  3D  object  space  coordinates  (X,Y,Z).  For this 3D $\leftrightarrow$ 2D correspondence the mapping should take the form of a 3x4 projection matrix (P) such that $x = PX$.

$$ x = PX \rightarrow [x]_xPX = 0 $$

Denoting the rows of $P$ with $P^1, P^2, P^3$ we can rewrite above equation to

\includegraphics[width=0.7\columnwidth]{pictures/dlt}

Since the third equation is dependant on the first two, it can be discounted. So for every point X we get two equations.

The bracket is denoted with $A$ (2nx12) and the camera matrix with P (3x4)

$$ A P = 0 $$

P has 11 DOF $\rightarrow$ 5.5 points are needed for the minimal solution.
For $n \geq 6$ it is over-determined and is solved with SVD.

\subsubsection{Gold Standard algorithm}
Objective:\\
Given $n \geq 6$ 2D to 3D point correspondences, determine the Maximum Likelihood Estimation of the camera projection matrix P.\\

Algorithm:
\begin{enumerate}
	\item Linear Solution: Compute an initial estimate of P using a linear method.
		\begin{enumerate}
			\item Normalization $\tilde{X} = UX$, $\tilde{x} = Tx$
			\item DLT
		\end{enumerate}
	\item Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: $\min_P \sum_{i}d(\tilde{x}_i,\tilde{P}\tilde{X}_i)^2$
	\item Denormalization $P = T^{-1}\tilde{P}U$
\end{enumerate}


\subsection{Tangent Planes}
\noindent
Although the tangent lines at a point on a surface can all be different depending on from which direction one approaches a point, all of these tangent lines lie in the same plane, defining the tangent plane.
This means that the tangent plane to $z = f(x, y)$ at $(x_0,y_0)$ has the following properties:
\begin{itemize}
	\item The z-value of the tangent plane at $(x_0, y_0)$ is the same as $f(x_0, y_0)$.
	\item The value of the first-order partial derivatives of the tangent plane at $(x_0, y_0)$ should match those of $f(x_0, y_0)$.
\end{itemize}

\noindent
The general form of a plane at $(x_0, y_0, z_0)$ is
\begin{equation*}
	P(x,y) = A(x-x_0) + B(y-y_0) + z_0.
\end{equation*}
We want $P_x = f_x$ and $P_y = f_y$.
This means that $P_x = f_x = A$ and $P_y = f_y = B$.
Rewriting,
\begin{equation*}
	P(x,y) = f_x(x-x0) + f_y(y-y_0) + z_0.
\end{equation*}
The normal vector is $\langle \pm f_x,\pm f_y, \mp 1\rangle$.
So, the point normal form of the plane is
\begin{equation*}
	\langle -f_x, -f_y, 1\rangle \cdot \langle x-x_0, y-y_0, z-f(x_0,y_0) \rangle = 0.
\end{equation*}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\textwidth]{./Images/differentialMultivariableCalculus/tangent_plane.png}
	\caption{Tangent plane}
\end{figure}


%================================
\section{Cover and Basis}
%================================

%--------------------------------
\begin{definition}
	[cover]
	\label{def: cover}
	Let $(X, \mathcal T)$ be a topological space, and let $U \subseteq X$, then a family $\mathcal C \subseteq \mathcal P(X)$ is called a \textit{cover} of $U$ iff the union of all sets in $\mathcal C$ is a superset of $U$. That is,
	$$
	U \subseteq \bigcup \mathcal C.
	$$

	If $\mathcal C \subseteq \mathcal T$, then we call $\mathcal C$ an \textit{open cover} of $U$.

	Let $\mathcal S \subseteq \mathcal C$, iff the union of $\mathcal S$ is still a superset of $U$, then we call $\mathcal S$ a \textit{subcover} of $\mathcal C$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[basis]
	\label{def: basis}
	\label{theorem: alt def of basis}
	Let $(X, \mathcal T)$ be a topological space, and let $\mathcal B\subseteq \mathcal T$. $\mathcal B$ is a \textit{base} for $\mathcal T$ iff it satisfies the following properties.
	\begin{enumerate}[(i)]
		\item For each $x \in X$, there exists $B \in \mathcal B$ such that $x \in B$, i.e., $\bigcup \mathcal B = X$.
		\item For each $x \in B_1 \cap B_2$ where $B_1, B_2 \in \mathcal B$, there exists $B_3 \ni x$ such that $B_3 \subseteq B_1 \cap B_2$.
	\end{enumerate}
\end{definition}
%--------------------------------

%--------------------------------
\begin{note}
	A base for a topology is not necessarily closed under finite intersections. For example, let
	$$
	\mathcal I = \left\{ (a,b) : 0 < b - a < 1 \right\} \in \mathcal P(\mathbb R),
	$$
	and let
	$$
	\mathcal J = \left\{ (-\infty, 2), (-2, \infty) \right\} \in \mathcal P(\mathbb R).
	$$
	$\mathcal I \cup \mathcal J$ is a basis for the Euclidean topology on $\mathbb R$, but $\mathcal I \cup \mathcal J$ is not closed under finite intersection, for $\bigcap \mathcal J = (-2, 2)$ is not a member of $\mathcal I \cup \mathcal J$.
\end{note}
%--------------------------------

%--------------------------------
\begin{corollary}
	\label{coro: alt def of basis}
	Let $(X, \mathcal T)$ be a topological space and let $\mathcal B$ be a basis for $\mathcal T$. For all $U \in \mathcal T$, there exists $\mathcal I, \mathcal J \subseteq \mathcal B$ ($\mathcal J$ finite) such that
	$$
	U = \bigcup \mathcal I = \bigcap \mathcal J.
	$$
	% todo: POOF
\end{corollary}
%--------------------------------

%--------------------------------
\begin{example}
	Let $(\mathbb R, \mathcal T)$ be a topological space where $\mathcal T$ is induce from the standard Euclidean metric, and let $\mathcal B$ be the family of all open proper intervals in $\mathbb R$. $\mathcal B$ is a base for $\mathcal T$.

	However, if $\mathcal T$ is a discrete topology on $\mathbb R$, $\mathcal B$ is not a base for $\mathcal T$, for some open subsets of $\mathbb R$, such as $\{0\}$, is neither any arbitrary union nor any finite intersection of $\mathcal B$-sets.
\end{example}
%--------------------------------

%--------------------------------
\begin{theorem}
	Let $(X, \mathcal T)$ and $(X, \mathcal T')$ be topological spaces, and let $\mathcal B$ and $\mathcal B'$ be the basis for $\mathcal T$ and $\mathcal T'$ respectively. $\mathcal T'$ is finer than $\mathcal T$ iff for any $x \in X$ and for any $B \in \mathcal B'$ with $B \ni x$, there exists $B' \in \mathcal B'$ such that $x \in B' \subseteq B$.

	\begin{proof}
		For $\implies$. Let $B \in \mathcal B$. Clearly, $B \in \mathcal T$. Also, $B \in \mathcal T'$ for $\mathcal T \subseteq \mathcal T'$. By Corollary \ref{coro: alt def of basis}, there exists $\mathcal I' \subseteq \mathcal B'$ such that $B = \bigcup \mathcal I'$. Clearly, all $\mathcal I'$-sets are subsets of $B$.

		For $\impliedby$. For all $B \in \mathcal B$ and for all $x \in B$, there exists $B' \in \mathcal B'$ with $x \in B' \subseteq B$. Let $\mathcal J'$ denote all such $B'$, then we have $B = \bigcup \mathcal J'$. By Definition \ref{def: basis}, we have $B \in \mathcal B \subseteq \mathcal T'$. Above all, $\mathcal T \subseteq \mathcal T$.
	\end{proof}
\end{theorem}
%--------------------------------

%--------------------------------
\begin{note}
	In this theorem, it is not necessary that $\mathcal B \subseteq \mathcal B'$. For example, let
	$$
	\begin{aligned}
		& \mathcal B = \left\{ (a, b) \subseteq \mathbb R : b - a = 1 \right\}, \\
		& \mathcal B' = \left\{ (a, b) \subseteq \mathbb R : b - a = 2 \right\}.
	\end{aligned}
	$$
	Obviously, $\mathcal B \not \subseteq \mathcal B'$ and $\mathcal B' \not \subseteq \mathcal B$, but they generate exactly the same topologies on $\mathbb R$.

	For another example, let
	$$
	\begin{aligned}
		& \mathcal B = \left\{ \text{all open intervals in $\mathbb R$}\right\}, \text{ and} \\
		& \mathcal B' = \left\{ \text{all singletons in $\mathbb R$} \right\}.
	\end{aligned}
	$$
	In this case, also, $\mathcal B \not \subseteq \mathcal B'$ and $\mathcal B' \not \subseteq \mathcal B$, but $\mathcal B'$ generates the discrete topology which is the finest topology on $\mathbb R$.
\end{note}
%--------------------------------

%%--------------------------------
%\begin{theorem}
%	Let $(X, \mathcal T)$ be a topological space be genrated by a base $\mathcal B$. For all $U \in \mathcal T$, there is a $B \in \mathcal B$ such that $U \subseteq \mathcal B$.
%
%	\begin{proof}
%		By Definition \ref{def: generated by basis}, if $\mathcal T$ is generated by $\mathcal B$, then for all $U \in \mathcal T$, there is an finite set $I$, such that
%		$$
%		U = \bigcap_{i \in I} B_i, \quad B_i \in \mathcal B.
%		$$
%		Thus, for at least one $k \in I$, $U \subseteq B_k$.
%	\end{proof}
%\end{theorem}
%%--------------------------------
%
%
%%--------------------------------
%\begin{proposition}
%	Let $X$ be any set, and let $\mathcal T$ and $\mathcal T'$ be its topologies generated by basis $\mathcal B$ and $\mathcal B'$ respectively. Then $\mathcal T'$ is finer than $\mathcal T$ iff for any $B \in \mathcal B$, there is a $B' \in \mathcal B'$ such that $B' \subseteq B$.
%
%	\begin{proof}
%		If $\mathcal T$ is generated by $\mathcal B$, then for all $U' \in \mathcal T'$,
%		$$
%		U' = \bigcup_{j \in J} B_j',
%		$$
%		where $B_j \in \mathcal B$.
%
%		As $\mathcal T$ is generated by $\mathcal B$, then, certainly, $\mathcal B \subseteq \mathcal T$.
%
%		By the conditions we have, $\mathcal T \subseteq \mathcal T'$ iff for all $B \in \mathcal B$, there is $W' \in \mathcal T$ such that
%		$$
%		B = W' = \bigcup_{i \in I} B_i',
%		$$
%		where $B_i' \in \mathcal B'$. Certainly, all such $B_i'$ are contained in $B$.
%	\end{proof}
%\end{proposition}
%%--------------------------------
%
%
%%--------------------------------
%\begin{proposition}
%	Let $X$ be any set, and let $\mathcal T \subseteq \mathcal P(X)$. $\mathcal T$ is a topology on $X$ iff it generates itself.
%
%	\begin{proof}
%		If $\mathcal T$ is a topology on $X$, then, by Definition \ref{def: generated by basis}, any open set generated by $\mathcal T$ is still a member of $\mathcal T$. On the other hand, if $\mathcal T$ generates itself, then, $\emptyset$ and $X$ must be members of $\mathcal T$, and, by Definition \ref{def: generated by basis}, $\mathcal T$ is a topology on $X$.
%	\end{proof}
%\end{proposition}
%%--------------------------------

% todo: check this link to fill the propositions.
%
% https://en.wikipedia.org/wiki/Base_(topology)#Theorems
%
% check the "local base" theorem in this link.

%


\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{datetime}
\usepackage{lipsum}
\graphicspath{{images/}}

\setlength\parindent{0pt}	% no indent for entire document
\setcounter{MaxMatrixCols}{20}

\title{Convolution and Deconvolution}

\begin{document}

\begin{titlepage}
\author{Hao Huang}
\date{\today}
\maketitle
\end{titlepage}

\tableofcontents
\newpage

\section{2D Convolution}
\begin{center}
\includegraphics[scale=0.7]{conv}
\end{center}
The blue grid is input image ($4\times 4$), and the green grid is output image ($2\times 2$). The kernel size is $3\times 3$. Stride is 1 and padding is 0. The weight of the kernel at the localization $(i, j)$ is donated as $w_{i, j}$. The value of the image at the location $(i, j)$ is donated as $x_{i,j}$.
The kernel is:
\[
w=
\begin{bmatrix}
w_{0,0} & w_{0,1} & w_{0,0} \\
w_{1,0} & w_{1,1} & w_{1,2} \\
w_{2,0} & w_{2,1} & w_{2,2}
\end{bmatrix}
\]
Rewrite image $X$ as a vector:
$
\begin{bmatrix}
x_{0,0} & x_{0,1} & \dots & x_{3,2} & x_{3,3}
\end{bmatrix}^T
$.Therefore, convolution can be written as matrix multiplication:
\[
Y=w*X=WX=
\]
\[
\setlength{\arraycolsep}{2pt}
\renewcommand{\arraystretch}{1}
\begin{bmatrix}
w_{0,0} & w_{0,1} & w_{0,2} & 0 & w_{1,0} & w_{1,1} & w_{1,2} & 0 & w_{2,0} & w_{2,1} & w_{2,2} & 0 & 0 & 0 & 0 & 0\\
0 & w_{0,0} & w_{0,1} & w_{0,2} & 0 & w_{1,0} & w_{1,1} & w_{1,2} & 0 & w_{2,0} & w_{2,1} & w_{2,2} & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & w_{0,0} & w_{0,1} & w_{0,2} & 0 & w_{1,0} & w_{1,1} & w_{1,2} & 0  & w_{2,0} & w_{2,1} & w_{2,2} & 0\\
0 & 0 & 0 & 0 & 0 & w_{0,0} & w_{0,1} & w_{0,2} & 0 & w_{1,0} & w_{1,1} & w_{1,2} & 0 & w_{2,0} & w_{2,1} & w_{2,2}\\
\end{bmatrix}
\begin{bmatrix}
x_{0,0}\\ x_{0,1}\\ x_{0,2} \\ x_{0,3} \\ x_{1,0} \\x_{1,1} \\ x_{1,2} \\x_{1,3} \\ x_{2,0}\\ x_{2,1}\\ x_{2,2} \\ x_{2,3} \\ x_{3,0} \\x_{3,1} \\ x_{3,2} \\x_{3,3}
\end{bmatrix}
\]
$W$ is a doubly block circulant matrix (a special case of Toeplitz matrix).
Here we define:
\[
W_0=
\begin{bmatrix}
w_{0,0} & w_{0,1} & w_{0,2} & 0 \\ 0 & w_{0,0} & w_{0,1} & w_{0,2}
\end{bmatrix} \;
W_1=
\begin{bmatrix}
w_{1,0} & w_{1,1} & w_{1,2} & 0 \\ 0 & w_{1,0} & w_{1,1} & w_{1,2}
\end{bmatrix}
\]
\[
W_3=
\begin{bmatrix}
w_{2,0} & w_{2,1} & w_{2,2} & 0 \\ 0 & w_{2,0} & w_{2,1} & w_{2,2}
\end{bmatrix} \;
\bf{0}=
\begin{bmatrix}
0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0
\end{bmatrix}
\]
$W_0$,$W_1$,$W_2$, and $\bf{0}$ are all Toeplitz matrix. $W$ can be written as:
\[
W=
\begin{bmatrix}
W_0 & W_1 & W_2 & \bf{0} \\ \bf{0} & W_0 & W_1 & W_2
\end{bmatrix}
\]
Therefore, $W$ is a doubly block circulant matrix.

\section{2D Deconvolution}
In the above example, the size of $X$ is $16\times 1$ and the size of $Y$ is $4\times 1$. The dimension of $W$ is $4\times 16$. In order to recover the size of $Y$ to the size of $X$, we just need $W^TY$ (We cannot recover the value of $X$, but just size):
\[
Y^\prime = W^TY = W^T(WX)
\]
If we donate $Y$ as:$
\begin{bmatrix}
y_{0,0} & y_{0,1} & y_{1,0} & y_{1,1}
\end{bmatrix}^T
$, the deconvolution can be written as:
\[
Y^\prime = W^TY =
\begin{bmatrix}
w_{0,0} & 0 & 0 & 0 \\ w_{0,1} & w_{0,0} & 0 & 0 \\
w_{0,2} & w_{0,1} & 0 & 0 \\ 0 & w_{0,2} & 0 & 0 \\
w_{1,0} & 0 & w_{0,0} & 0 \\ w_{1,1} & w_{1,0} & w_{0,1} & w_{0,0}\\
w_{1,2} & w_{1,1} & w_{0,2} & w_{0,1} \\ 0 & w_{1,2} & 0 & w_{0,2}\\
w_{2,0} & 0 & w_{1,0} & 0 \\ w_{2,1} & w_{2,0} & w_{1,1} & w_{1,0}\\
w_{2,2} & w_{2,1} & w_{1,2} & w_{1,1} \\ 0 & w_{2,2} & 0 & w_{1,2}\\
0 & 0 & w_{2,0} & 0 \\ 0 & 0 & w_{2,1} & w_{2,0} \\
0 & 0 & w_{2,2} & w_{2,1} \\ 0 & 0 & 0 & w_{2,2}
\end{bmatrix}
\begin{bmatrix}
y_{0,0} \\ y_{0,1} \\ y_{1,0} \\ y_{1,1}
\end{bmatrix}
\]
If we rewrite it in convolution format $Y^\prime=w^\prime*Y$, where $w^\prime$ is the top-bottom AND left-right flip of the original filter $w$.
\begin{center}
\includegraphics[scale=0.5]{deconv}
\end{center}
In the above image, the blue grid is $Y$, and the green grid is $Y^\prime$. Stride is 1 and padding is 2.

We can write this process as:
\[
X -> Y = conv(X) -> deconv(Y) -> Y^\prime
\]
To keep $Y^\prime$ has the same size as $X$, we need to add paddings to $conv(X)$. Usually we add 0 paddings to $conv(X)$. There are two common padding strategries:
\begin{enumerate}
\item{Add paddings around an image}
\begin{center}
\includegraphics[scale=0.5]{spad}
\end{center}
\item{Add paddings inside an image}
\begin{center}
\includegraphics[scale=0.5]{ipad}
\end{center}
\end{enumerate}

These two strategies yield different results. How to choose this two strategies depends on which one can keep the image size.

\end{document}


\section{Integration along Lines, Surfaces, and Volumes}
\subsection{Line Integrals}
\begin{definition}
    For a vector field $\underline{F}(\underline{x})$ and a curve $C$ where $\underline{x}(t)$ travels through for $t\in [a,b]$, we define the line integral
    $$\int_C\underline{F}\cdot\mathrm d\underline{x}=\int_a^b\underline{F}(\underline{x}(t))\cdot\frac{\mathrm d\underline{x}}{\mathrm dt}\,\mathrm dt$$
\end{definition}
\begin{example}
    Let
    $$\underline{F}=\begin{pmatrix}
        x^2y\\
        yz\\
        2zx
    \end{pmatrix},C_1:[0,1]\ni t\mapsto\begin{pmatrix}
        t\\
        t\\
        t
    \end{pmatrix},C_2:[0,1]\ni t\mapsto\begin{pmatrix}
        t\\
        t\\
        t^2
    \end{pmatrix}$$
    ...
    We have
    $$\int_{C_1}\underline{F}\cdot\mathrm d\underline{x}=\frac{5}{4},\int_{C_2}\underline{F}\cdot\mathrm d\underline{x}=\frac{13}{10}$$
    Hence in general line integrals between two points depends on path.
\end{example}
\begin{example}
    In cylindral polars $(\rho,\phi,z)$, we consider $\underline{F}=\rho z\underline{e_{\phi}}$.
    Consider the line $C:[0,2\pi]\ni t\mapsto (a\cos t,a\sin t,t)^\top$.
    So $\rho=a,\phi=z=t$
    We have $\underline{F}\cdot\mathrm d\underline{x}=\rho^2 z\mathrm d\phi$, therefore
    $$\int_C\underline{F}\cdot\mathrm d\underline{x}=\int_0^{2\pi}a^2t\,\mathrm dt=2a^2\pi^2$$
\end{example}
In those cases where $C$ is closed, we write
$$\oint_C\underline{F}\cdot\mathrm d\underline{x}=\int_C\underline{F}\cdot\mathrm d\underline{x}$$
This is sometimes called the circulation of $\underline{F}$ over $C$.
\subsection{Conservative Forces and Exact Differentials}
$\underline{F}\cdot\mathrm d\underline{x}$ is an example of a differential form.
\begin{definition}
    We say $\underline{F}\cdot\mathrm d\underline{x}$ is exact if $\underline{F}\cdot\mathrm d\underline{x}=\mathrm df$ for some scalar function $f$.
    Equivalently, the differential form is exact iff $\underline{F}=\nabla f$ for a scalar function $f$.
    In this case, we say $\underline{F}$ is conservative.
\end{definition}
\begin{proposition}
    If $\underline{F}\cdot\mathrm d\underline{x}$ is exact, then
    $$\oint_C\underline{F}\cdot\mathrm d\underline{x}=0$$
    for any closed $C$.
\end{proposition}
\begin{proof}
    By exactness, $\underline{F}=\nabla f$ for a scalar function $f$.
    Suppose $C: [a,b]\ni t\mapsto\underline{x}(t)$.
    $$\oint_C\underline{F}\cdot\mathrm d\underline{x}=\int_a^b\nabla f(\underline{x}(t))\cdot\underline{x}^\prime(t)\,\mathrm dt=\int_a^b\frac{\mathrm d}{\mathrm dt}f(\underline{x}(t))\,\mathrm dt=f(\underline{x}(b))-f(\underline{x}(a))=0$$
    since $\underline{x}(b)=\underline{x}(a)$.
\end{proof}
\begin{remark}
    Consider the cylindral coordinate $(\rho,\phi,z)$, suppose $\underline{F}\cdot\mathrm d\underline{x}=\mathrm d\phi$ and $C:[0,2\pi]\ni t\mapsto (\cos t,\sin t,0)^\top$, then by calculation we have
    $$\oint_C \underline{F}\cdot\mathrm d\underline{x}=2\pi\neq 0$$
    It does not work!
    The reason for this is that $\phi\in\mathbb R/2\pi\mathbb Z$.
    So we are doing here is taking $\phi$ as a multivalued function instead of an actual function.
\end{remark}
Suppose we have a set of curvilinear coordinate $(u,v,w)=(u_1,u_2,u_3)$, and we have $\underline{F}\cdot\mathrm d\underline{x}=\theta_i\,\mathrm du_i$ where $\theta_i=\underline{F}\cdot\mathrm d\underline{x}/\mathrm du_i$.
If $\underline{F}\cdot\mathrm d\underline{x}$ is exact and is the differential $\mathrm df$, then $\theta_i=\partial f/\partial u_i$, and
$$\frac{\partial \theta_i}{\partial u_j}=\frac{\partial^2 f}{\partial u_i\partial u_j}=\frac{\partial^2 f}{\partial u_j\partial u_i}=\frac{\partial \theta_j}{\partial u_i}$$
\begin{definition}
    If the above condition is met, we say the differential form $\underline{F}\cdot\mathrm d\underline{x}$ is closed.
\end{definition}
So exact differentials are all closed.
\begin{theorem}
    If the domain of the vector field is simply connected, then any closed differential is exact.
\end{theorem}
\begin{proof}
    Not gonna do it.
\end{proof}
\begin{example}
    1. The differential $y\,\mathrm dx-x\,\mathrm dy$ is not closed hence not exact.\\
    2. Consider $C:[t_1,t_2]\ni t\mapsto (f,g,h)$ where $f,g,h$ are completely unintelligible functions such that $(f,g,h)(t_1)=(f,g,h)(t_2)$, then whatever they are, we always have
    $$\oint_C 3x^2y\,\mathrm dx+x^3\,\mathrm dy=0$$
    As the integrand is exact.
\end{example}
\begin{theorem}
    Suppose $\underline{F}\cdot\mathrm\underline{x}=\mathrm df$, then consider $C$ from $\underline{a}$ from $\underline{b}$, then we have
    $$\int_C\underline{F}\cdot\mathrm\underline{x}=f(\underline{b})-f(\underline{a})$$
\end{theorem}
\begin{proof}
    Obvious.
\end{proof}
\begin{example}
    Suppose $\underline{F}=m\underline{\ddot{x}}$ and $C:[a,b]\ni t\mapsto\underline{x}(t)$, we have
    $$\int_C\underline{F}\cdot\mathrm d\underline{x}=m\int_a^b\underline{\ddot{x}}\cdot\underline{\dot{x}}\,\mathrm dt=\left.\frac{1}{2}m|\underline{\dot{x}}|^2\right|_a^b$$
    If $\underline{F}=-\nabla V$, then we have the conservation of energy:
    $$\left.\frac{1}{2}m|\underline{\dot{x}}|^2\right|_a^b=\int_C\underline{F}\cdot\mathrm d\underline{x}=-\left.V(\underline{x}(t))\right|^b_a$$
    So $V(\underline{x})+m|\underline{\dot{x}}|^2/2$ conserves.
\end{example}
\subsection{Integration over Areas}
We want to extend our definition of Riemannian integrals to $\mathbb R^2$.
To do it, we partition our region $D$ into small cells $A_{ij}$ with area $\delta A_{ij}$ diameter at most $\epsilon$ and pick points $(x_i,y_j)\in A_{ij}$.
\begin{definition}
    We thus define the area integral by
    $$\int_Df\,\mathrm dA=\lim_{\epsilon\to 0}\sum_{i,j}f(x_i,y_j)\delta A_{ij}$$
    We say this integral exists if this limit is independent of the choice of the partition $A_{ij}$.
\end{definition}
When the integral exists, the obivous choice is to split $D$ into rectangular cells and set $x_{i+1}=x_i+\delta x,y_{j+1}=y_j+\delta y$ such that $0<\delta x,\delta y<<\epsilon$.
Then we may fix $y$ first and take $\delta x\to 0$, and then do $\delta y\to 0$.
That is, we split the horizontal region by $\epsilon$-thin stripes and sum over the Riemann integrals in each stripe.
If we do this, then
$$\int_Df\,\mathrm dA=\int_Y\int_{x_y}f(x,y)\,\mathrm dx\,\mathrm dy$$
where $x_y=\{x:(x,y)\in D\}$.
If we do vertical stripes first, then we get stuff like
$$\int_Df\,\mathrm dA=\int_X\int_{y_x}f(x,y)\,\mathrm dy\,\mathrm dx$$
where $y_x=\{y:(x,y)\in D\}$.
In short, we seems to have $\mathrm dA=\mathrm dx\,\mathrm dy=\mathrm dy\,\mathrm dx$
\begin{theorem}[Fubini's Theorem]
    If the integral exists (or under suitable conditions), then
    $$\int_Df\,\mathrm dA=\int_Y\int_{x_y}f(x,y)\,\mathrm dx\,\mathrm dy=\int_X\int_{y_x}f(x,y)\,\mathrm dy\,\mathrm dx$$
\end{theorem}
\begin{example}
    $f(x)=xy^2$ and $D$ is the triangle joining $(0,0),(0,1),(1,0)$.
    Then we have
    $$\int_Df\,\mathrm dA=\int_0^1\int_{0}^{1-y}xy^2\,\mathrm dx\,\mathrm dy=\int_0^1\frac{(1-y)^2y^2}{2}\,\mathrm dy=\frac{1}{60}$$
    If we do $y$ (vertical slices) first,
    $$\int_Df\,\mathrm dA=\int_0^1\int_0^{1-x}xy^2\,\mathrm dy\,\mathrm dx=\int_0^1\frac{x(1-x)^3}{3}\,\mathrm dx=\frac{1}{60}$$
\end{example}
Recall that in the one dimensional case, we can do integrations by some magical substitutions.
Obviously we will wish to extend this technique to integrations over higher dimensions.
\begin{proposition}[Change of Variable]
    Let $x=x(u,v),y=y(u,v)$ be a smooth bijection $D\to D'$ with smooth inverse, then
    $$\iint_D f(x,y)\,\mathrm dx\,\mathrm dy=\iint_{D'}f(x(u,v),y(u,v))|J|\,\mathrm du\,\mathrm dv$$
    where
    $$J=\frac{\partial (x,y)}{\partial (u,v)}=\begin{vmatrix}
        x_u&x_v\\
        y_u&y_v
    \end{vmatrix}$$
    is the Jacobian.
\end{proposition}
\begin{proof}
    Partition $D$ using image of rectangular partition of $D'$.
    Then we have
    $$\int_Af\,\mathrm dA=\lim_{\epsilon\to 0}\sum_{i,j}f(x(u_i,v_j),y(u_i.y_j))\delta A_{ij}^{x,y}$$
    But $\delta A_{ij}^{x,y}\approx |J|\delta A_{ij}^{u,v}$ by considering the area as local parallelograms and expanding the Taylor series to first order.
    So we have this formula.
\end{proof}
\begin{example}
    Consider $x=\rho\cos\phi,y=\rho\sin\phi$ for $\rho\ge 0$, then $|J|=\rho$, hence $\mathrm dx\,\mathrm dy=\rho\,\mathrm d\rho\,\mathrm d\phi$.
    Take $D$ to be the region $x>0,y>0$, then this region is mapped to the region $\phi\in (0,\pi/2)$.
    So let
    $$I=\int_0^\infty e^{-x^2}\,\mathrm dx$$
    then
    $$I^2=\int_0^\infty\int_0^\infty e^{-x^2-y^2}\rho\,\mathrm dx\,\mathrm dy=\int_0^{\pi/2}\int_0^\infty e^{-\rho^2}\rho\,\mathrm d\rho\,\mathrm d\phi=\frac{\pi}{4}\implies I=\frac{\sqrt\pi}{2}$$
\end{example}
\subsection{Integration over Volumes}
For a bounded volume $V$ in $\mathbb R^3$, define sets $V_{ijk}$ having volume $\delta V_{ijk}$ which partition $V$ and each is contained in a ball of radius at most $\epsilon$.
Then we pick some $(x_i,y_j,z_k)$ in each cell $V_{ijk}$ and define the integral over the region $V$ as
$$\lim_{\epsilon\to 0^+}\sum_{i,j,k}f(x_i,y_j.z_k)\delta V_{ijk}$$
If we use a rectangular parallelopiped partition, we find that $\mathrm dV=\mathrm dx\,\mathrm dy\,\mathrm dz$ in any order (by Fubini).
\begin{example}
    1. Consider the domain to be the tetrahedron $V=\{(x,y,z)\in\mathbb R_{\ge 0}^3:x+y+z\le 1\}$.
    So
    $$\int_V\mathrm dV=\int_0^1\int_0^{1-x}\int_0^{1-x-y}\mathrm dz\,\mathrm dy\,\mathrm dx=\frac{1}{6}$$
    2. For a volumn $V$, we define the center of mass $\Delta_{\rm COM}$ by
    $$\Delta_{\rm COM}=\frac{1}{M}\int_V\rho\underline{x}\,\mathrm dV$$
    where $\rho$ is the density, $M=\rho V$ is the mass and the integral is by component.
    Consider the same tetrahedron as above and suppose $\rho=1$.
    Hence $M=1/6$, so
    $$\Delta_{\rm COM}=6\int_V1\begin{pmatrix}
        x\\
        y\\
        z
    \end{pmatrix}\,\mathrm dV=\frac{1}{4}\begin{pmatrix}
        1\\
        1\\
        1
    \end{pmatrix}$$
\end{example}
\begin{proposition}
    Let $\underline{x}=\underline{x}(u,v,w)$ (where $\underline{x}=(x,y,z)$), denote the smooth bijection with smooth inverse which connects the region $V$ in the $xyz$ space and $V'$ in the $uvw$ space, then
    $$\iiint_Vf(x,y,z)\,\mathrm dx\,\mathrm dy\,\mathrm dz=\iiint_{V'}f(x(u,v,w),y(u,v,w),z(u,v,w))|J|\,\mathrm du\,\mathrm dv\,\mathrm dw$$
    where
    $$J=\begin{vmatrix}
        x_u&x_v&x_w\\
        y_u&y_v&y_w\\
        z_u&z_v&z_w
    \end{vmatrix}$$
\end{proposition}
\begin{proof}
    Same (imprecise) idea.
\end{proof}
\begin{example}
    If we use cylindral polars, we find $\mathrm dx\,\mathrm dy\,\mathrm dz=\rho\,\mathrm d\rho\,\mathrm d\phi\,\mathrm dz$, and if we use spherical polars, we have $\mathrm dx\,\mathrm dy\,\mathrm dz=r^2\sin\theta\,\mathrm dr\,\mathrm d\theta\,\mathrm d\phi$.
\end{example}
\begin{example}
    1. Consider a sphere with radius $R$, we want to find its volume.
    If we do it with Cartesians, then
    $$\int_{-R}^R\int_{-\sqrt{R^2-z^2}}^{\sqrt{R^2-z^2}}\int_{-\sqrt{R^2-z^2-y^2}}^{\sqrt{R^2-z^2-y^2}}\mathrm dx\,\mathrm dy\,\mathrm dz=\frac{4}{3}\pi R^3$$
    after a lot of useless effort.\\
    So obviously we choose to use spherical polars, hence the volume is
    $$\int_0^{2\pi}\int_0^\pi\int_0^{R}r^2\sin\theta\,\mathrm dr\,\mathrm d\theta\,\mathrm d\phi=\frac{4}{3}\pi R^2$$
    after minimal effort.\\
    2. A ball of radius $b>0$ with cylinder with radius $a>0$ (and infinite length) with $a>0$ removed.
    So maybe we will use some cylindral polars, so the volume is
    $$\int_0^{2\pi}\int_a^b\int_{-\sqrt{b^2-\rho^2}}^{\sqrt{b^2-\rho^2}}\rho\,\mathrm dz\,\mathrm d\rho\,\mathrm d\phi=\frac{4}{3}\pi(b^2-a^2)^{3/2}$$
    which was easy.
\end{example}


\section{Fundamental class}
Note that if $M$ is a compact manifold, then $H_q(M;R)=0$ for $q\gg 0$, and if $R$ is a PID, then for all $q$, $H_q(M;R)$ is finitely-generated. This follows from:
\begin{claim}
Suppose $X$ admits an open cover $\{U_i\}_{i=1}^n$ such that all intersections are either empty or contractible (this is what you get for a good cover on a manifold). Then $H_q(X;R)=0$ for $q\geq n$, and if $R$ is a PID, then for all $q$, $H_q(X;R)$ is finitely-generated.
\end{claim}
\begin{proof}
Induct. Certainly true for $n=1$. Let $Y=\bigcup^{n-1}_{i=1}U_i$, then this statement is true by induction -- and similarly for $Y\cap U_n$. Now use Mayer-Vietoris. You have $\cdots\to H_q(Y\cap U_n)\to H_q(Y)\oplus H_q(U_n)\to H_q(X)\to H_{q-1}(Y\cap U_n)\to\cdots$. When $q=n-1$, $H_q(Y\cap U_n)$ could be nonzero, and so you might get something nontrivial (???). Also, you'll get a sexseq by unsplicing the lexseq: $0\to H_q(Y)\oplus H_q(U_n)/\text{something}\to H_q(X)\to \text{submodule of }H_{q-1}(Y\cap U_n)\to 0$, where you use $R$ being a PID to conclude that $\text{submodule of }H_{q-1}(Y\cap U_n)$ is finitely generated.
\end{proof}
Let $M$ be an $n$-manifold. We had a map $j:H_n(M)\to \Gamma(M;o_M)$. Here $\Gamma(M;o_M)$ is the collection of compatible elements of $H_n(M,M-x)$ for $x\in M$. This map $j:H_n(M)\to \Gamma(M;o_M)$ sends $c\mapsto(x\mapsto j_x c)$ where $j_x:H_n(M,\emptyset)\to H_n(M,M-x)$. I want to make two refinements.

You can't expect $j$ to be surjective, except maybe when $M$ is compact. Here's why. Let $c\in Z_n(M)$. It's a sum of simplices, and each simplex is compact, and so the union of the images is compact, and hence there's a compact subset $K\subseteq M$ such that $c\in Z_n(K)$. Now if I take $x\not\in K$, then the map $H_n(K)\to H_n(M)$ splits as $H_n(K)\to H_n(M-x)\to H_n(M)$. In the relative homology, $H_n(M,M-x)$, the map $H_n(K)\to H_n(M)\to H_n(M,M-x)$ sends $c$ to zero.
\begin{definition}
Let $\sigma$ be a section of $p:E\to B$ (local system). Then the support of $\sigma$ is defined as $\mathrm{supp}(\sigma)=\overline{\{x\in B|\sigma(x)\neq 0\}}$. The collection of all sections with compact support is $\Gamma_c(B;E)$, and it's a submodule of $\Gamma(B;E)$.
\end{definition}
The first refinement is that $j:H_n(M)\to \Gamma(M;o_M)$ lands in $\Gamma_c(M;o_M)$, because homology is compactly supported.

The second refinement seems a little artificial but is part of the inductive process. Let $A\subseteq M$ be closed. Then you have a restriction map $H_n(M,M-A)\xrightarrow{j_x}H_n(M,M-x)$ for $x\in A$. Thus you get a map $j:H_n(M,M-A)\to \Gamma_c(A;o_M|_{A})$, the latter of which we'll just denote $\Gamma_c(A;o_M)$.
\begin{theorem}
The map $j:H_n(M,M-A)\to \Gamma_c(A;o_M|_{A})$ is an isomorphism and $H_q(M,M-A)=0$ for $q>n$. (If $A=M$ then $j:H_n(M)\to \Gamma_c(M;o_M)$ is an isomorphism.)
\end{theorem}
\begin{proof}
For $X=\RR^n$ and $A=D^n$. Well, $o_{\RR^n}=\RR^n\times H_n(\RR^n,\RR^n-0)$ is trivial (i.e., a product projection), so $\Gamma(D^n;o_{\RR^n})=\Hom_{\mathbf{Top}}(D^n,H_n(\RR^n,\RR^n-0))$ where $H_n(\RR^n,\RR^n-0)$ is discrete, and this is therefore just a map from $\pi_0$ into this, and thus $\Gamma(D^n;o_{\RR^n})=R$ (your coefficient). But also, $H_n(\RR^n,\RR^n-D^n)\cong R$, so you have that $j$ gives $H_n(\RR^n,\RR^n-D^n)\to \Gamma_c(D^n;o_{\RR^n}|_{D^n})$.

Say that this is true for $A,B,A\cap B$ -- we'll prove this for $A\cup B$. Obviously, use Mayer-Vietoris. I have a restriction $\Gamma_c(A\cup B;o_M)\to \Gamma_c(A;o_M)\oplus \Gamma_c(B;o_M)$ that sits in an exact sequence $0\to \Gamma_c(A\cup B;o_M)\xrightarrow{\text{inclusion, determined by }A,B} \Gamma_c(A;o_M)\oplus \Gamma_c(B;o_M)\to \Gamma_c(A\cap B;o_M)$. This is a gluing lemma. We also have a relative Mayer-Vietoris $H_n(M,M-A\cup B)\to H_n(M,M-A)\oplus H_n(M,M-B)\to H_n(M,M-A\cap B)$, so we have:
\begin{align*}
\xymatrix@C=10pt{
	0\ar[r] & \Gamma_c(A\cup B;o_M)\ar[r]^{\text{inclusion }A,B\to A\cup B} & \Gamma_c(A;o_M)\oplus \Gamma_c(B;o_M)\ar[r] & \Gamma_c(A\cap B;o_M)\\
	H_{n+1}(M,M-A\cap B)=0\ar[r]^0 & H_n(M,M-A\cup B)\ar[r]\ar[u] & H_n(M,M-A)\oplus H_n(M,M-B)\ar[r]\ar[u]^{j_\ast}_{\cong} & H_n(M,M-A\cap B)\ar[u]^{j_\ast}_{\cong}
}
\end{align*}
This is a ``local-to-global'' argument. ``I don't feel like going through the point-set topology -- the rest of the proof is just annoyance.'' See Bredon's book for the conclusion of the proof.
\end{proof}
\begin{corollary}
$j:H_n(M)\to \Gamma_c(M;o_M)$ is an isomorphism.
\end{corollary}
\begin{definition}
An $R$-orientation for $M$ is a section $\sigma$ of $\Gamma(M;o_M^\times)$ where $o_M^\times$ is the covering space of $M$ given by the generators (as $R$-modules) of the fibers of $o_M$.
\end{definition}
If $M$ is compact, then $j:H_n(M)\to \Gamma(M;o_M)$, and you get $[M]\leftrightarrow \sigma$. When does that exist?

\underline{Over $\Z$:} $o_M^\times\to M$ is a double cover of $M$ (over every element you have two possible elements given by the two possible orientations ($\pm 1$)). If $M$ is an $n$-manifold and $f:N\to M$ is a covering space, then $N$ is also locally Euclidean. I have the orientation local system to get a pullback local system:
\begin{equation*}
\xymatrix{
	f^\ast o_M=N\times_M o_M\ar[r]\ar[d] & o_M\ar[d]\\
	N\ar[r] & M
}
\end{equation*}
Because $N\to M$ is a covering space, the fibers of $f^\ast o_M$ are the same as the fibers of $o_N$, so actually, $f^\ast o_M\cong o_N$. For example, suppose $N=o_M^\times$. What happens if I consider:
\begin{equation*}
\xymatrix{
	o_N=N\times_M N\ar[r]\ar[d] & N\ar[d]\\
	N\ar[r] & M
}
\end{equation*}
But now, I have the identity $N\to N$ that sits compatibly as:
\begin{equation*}
\xymatrix{
	N\ar[r]^{\mathrm{id}}\ar[d] ^{\mathrm{id}} & N\ar[d]\\
	N\ar[r] & M
}
\end{equation*}
And hence you get $N\to o_N^\times$, which is a section of $o_N^\times\to N$. The conclusion is that $N=o_M^\times$ is canonically oriented (even if $M$ is not oriented!). If $M$ is oriented, then the local system is trivial and you have the trivial double cover.

The overarching conclusion is: if $M$ is an $n$-manifold, then:
\begin{enumerate}
\item $H_q(M)=0$ for $q>n$.
\item If $M$ is compact, then $H_n(M)\xrightarrow{\cong}\Gamma(M,o_M)$.
\item If $M$ is connected and compact, then:
	\begin{enumerate}
	\item if $M$ is oriented with respect to $R$, then $H_n(M)\cong \Gamma(M,o_M)\cong R$.
	\item (I have no idea what was happening here, we didn't reach to a conclusion for a while.) if $M$ is not orientable, then $o_M^\times$ is nontrivial. If $o_M^\times$ has a section, then it's trivial (and so is $o_M$) because if it has a section $\sigma:M\to o^\times_M$, define $M\times R^\times\xrightarrow{\cong} o_M^\times$ by sending $(x,r)\mapsto r\sigma(x)\in o_M^\times$ (and the same thing $M\times R\xrightarrow{\cong} o_M$ for the orientation local system itself). I don't see an argument to conclude that if $M$ is nonorientable, then there aren't any section of $o_M$. In particular, if $R=\Z$, then $H_n(M;\Z)=0$. I'm going to leave this as a statement without proof, unless any of you can help me.
	\end{enumerate}
\end{enumerate}
If a section $\sigma(x)=0$ for some $x$, then $\sigma=0$.
\begin{remark}
Prof. Miller talked with me about this after class. If I recall correctly, one way to think about this is as follows. If you have a local system $p:E\to B$, this can be viewed as a representation of $\pi_1(B)\to R^\times$, and the $\Gamma(B;E)=(E_x)^{\pi_1(B)}$ where $\pi_1(B)$ acts on the fibers by multiplication. Thus $(E_x)^{\pi_1(B)}=R^{\pi_1(B)}$. If $R=\Z$, then $R^\times=\{\pm 1\}$, so $R^{\pi_1(B)}=\{r|ar=r,a\in\pi_1(B)\}$, so that $\Z^{\pi_1(B)}=0$. Hence there are no sections of $o_M$, as desired. For a ring $R$, $o_{M,R}=o_{M,\Z}\otimes R$. Something else for $\Z/2\Z$. A higher homotopy theoretic perspective is that if you have a fibration $E\to B$, then $E=PB\times_{\Omega B}F$ where $F$ is the fiber of the fibration, so that $\Gamma(B;E)=\Map_{\Omega B}(PB,F)=F^{h\Omega B}$. In the case of a covering space you recover what you have above since $\pi_0(\Omega B)=\pi_1(B)$.
\end{remark}


% -*- root: Main.tex -*-
\section{Principle Component Analysis}
$\mathbf{X} \in \mathbb{R}^{D \times N}$. $N$ observations.\\
1. Empirical Mean: $\overline{\mathbf{x}} = \frac{1}{N} \sum_{n=1}^N \mathbf{x}_n$.\\
2. Center Data: $\overline{\mathbf{X}} = \mathbf{X} - [\overline{\mathbf{x}}, \ldots, \overline{\mathbf{x}}] = \mathbf{X} - \mathbf{M}$.\\
3. Cov.: $\boldsymbol{\Sigma} = \frac{1}{N	} \sum_{n=1}^N (\mathbf{x}_n - \overline{\mathbf{x}}) (\mathbf{x}_n - \overline{\mathbf{x}})^\top = \frac{1}{N} \overline{\mathbf{X}}\overline{\mathbf{X}}^\top$.\\
4. Eigenvalue Decomposition: $\boldsymbol{\Sigma} = \mathbf{U} \boldsymbol{\Lambda} \mathbf{U}^\top$.\\
5. Select $K < D$, only keep $\mathbf{U}_K = [u_1,...,u_{K}]$.\\
6. Transform data onto new Basis: $\overline{\mathbf{Z}}_K = \mathbf{U}_K^\top \overline{\mathbf{X}}$.\\
7. Reconstruct to original Basis: $\tilde{\overline{\mathbf{X}}} = \mathbf{U}_k \overline{\mathbf{Z}}_K$.\\
8. Reverse centering: $\tilde{\mathbf{X}} = \tilde{\overline{\mathbf{X}}} + \mathbf{M}$.\\
For compression save $\mathbf{U}_k, \overline{\mathbf{Z}}_K, \overline{\mathbf{x}}$.\\
$\mathbf{U}_k \in \mathbb{R}^{D \times K}, \boldsymbol{\Sigma} \in \mathbb{R}^{D \times D}, \overline{\mathbf{Z}}_K \in \mathbb{R}^{K \times N}, \overline{\mathbf{X}} \in \mathbb{R}^{D \times N}$ \\
$\bullet$ Reconstruction error = sum of discarded eigenvalues. \\
$\bullet$ The transformed dataset $\overline{\mathbf{Z}}_K$ has diagonal covariance matrix.

\subsection*{Iterative View}
Residuals $r_i = x_i - \tilde{x}_i = (\mathbb{1} - uu^T)x_i$\\
Cov. matrix of residuals: $\frac{1}{n} \sum_{i=1}^n r_ir_i^\top = \frac{1}{n} \sum_{i=1}^n (\mathbb{1}-uu^T)x_i x_i^T (\mathbb{1}-uu^T)^T = ... = \Sigma - \lambda uu^T$ \\
1. Find principal eigenvector of $(\Sigma - \lambda u u^T)$\\
2. which is the 2nd principal eigenvector of $\Sigma$\\
3. iterating to get $d$ principal eigenvector of $\Sigma$ \\
$\bullet$ the principal eigenvector of $\Sigma$ points in the direction with largest variance of the data ($argmax_{\|u\|=1} u^\top \Sigma u$).

\subsection*{Power iteration}
find principal eigenvector of A: $v_{t+1} = \frac{Av_t}{||Av_t||}$, $\lim_{t \rightarrow \infty} v_t = u_1$\\