%
% Chapter 1.5
%

\section*{1.5 Limit of a Function}

\subsection*{Limit}

We can make the values of \(f(x)\) arbitrarily close to \(L\) by restricting \(x\) to be sufficiently close to \(a\) but never equal to \(a\).
$$\lim_{x \to a}f(x)=L \text{ means that } f(x) \to L \text{ as } x \to a$$

\subsection*{One-sided Limit}

$$\lim_{x \to a^-}f(x)=L$$
means that the limit of \(f(x)\) as \(x\) approaches \(a\) from the left is equal to \(L\) if we can make the values of \(f(x)\) arbitrarily close to \(L\) by taking \(x\) to be sufficiently close to \(a\) with \(x\) less than \(a\).

$$\lim_{x \to a}f(x)=L \quad \text{if} \quad \lim_{x \to a^-}f(x)=L \quad \text{and} \quad \lim_{x \to a^+}f(x)=L$$

\subsection*{Infinite Limits}

Let \(f\) be a function defined on both side of \(a\), except possibly at \(a\) itself. Then
$$\lim_{x \to a}f(x)=\infty$$
means that the values of \(f(x)\) can be made arbitrarily large by taking \(x\) sufficiently close to \(a\), but not equal to \(a\).
\\\\
Let \(f\) be a function defined on both side of \(a\), except possibly at \(a\) itself. Then
$$\lim_{x \to a}f(x)=-\infty$$
means that the values of \(f(x)\) can be made arbitrarily large negative by taking \(x\) sufficiently close to \(a\), but not equal to \(a\).

\subsection*{Vertical Asymptotes}

The line \(x=a\) is called \textbf{vertical asymptote} of the curve \(y=f(x)\) if at least one of the following statements is true:

$$\lim_{x \to a}f(x)=\infty \quad \lim_{x \to a^-}f(x)=\infty \quad \lim_{x \to a^+}f(x)=\infty$$
$$\lim_{x \to a}f(x)=-\infty \quad \lim_{x \to a^-}f(x)=-\infty \quad \lim_{x \to a^+}f(x)=-\infty$$


\section{Right to Left Computability Class}\label{sec:right-to-left}
After having developed
formulas modifying rightmost bits in \autoref{sec:combining},
the option arises to classify these kind of formulas.
\emph{Hacker's Delight} \cite{Warren:2012:HD:2462741}
therefore introduces the so called ``right to left computability class''.
Every formula in this class shares the property,
that each result bit only depends on input bits
at its position or to the right of it (see \autoref{table:property}).

\begin{table}[H]
\centering
\begin{tabular}{rrl}
\lstinline$x$: & \lstinline$x$$_n\ \ \dots$
    & \fbox{\lstinline$x$$_i$ \lstinline$x$$_{i-1} \dots$\lstinline$x$$_0$}\\
\lstinline$y$: & \lstinline$y$$_n\ \ \dots$
    & \fbox{\lstinline$y$$_i$ \lstinline$y$$_{i-1} \dots$\lstinline$y$$_0$}\\
result \lstinline$r$: & \lstinline$x$$_n\ \ \dots$
    & \fbox{\lstinline$r$$_i$}\lstinline$ r$$_{i-}\dots$\lstinline$r$$_{i-1}$\\
~\\
\multicolumn{3}{c}{\fbox{\lstinline$r$$_i$ only depends on
    \lstinline$x$$_i \dots$\lstinline$x$$_0$ and
    \lstinline$y$$_i \dots$\lstinline$y$$_0$}}\\
\end{tabular}
\caption{Right to left computability property}
\label{table:property}
\end{table}

Notice that, as it is common for a lot of operators,
the result bit \lstinline$r$$_i$ is allowed to depend on
\lstinline$r$$_{i-1}\dots$\lstinline$r$$_0$.
This is already given since
every of these result bits to the right of \lstinline$r$$_i$
only depend on bits of \lstinline$x$ and \lstinline$y$.

Before defining the class,
the previously used set of operators will get slightly modified:
\lstinline$INC$ and \lstinline$DEC$
get exchanged for \lstinline$ADD$ and \lstinline$SUB$.
This will allow for more flexibility while not using
more powerful operations: they can simply be expressed with loops.
The instruction set is therefore:
\[
    \text{\lstinline$NOT$},
    \underbrace{\text{\lstinline$ADD$, \lstinline$SUB$}}
    _{\text{\lstinline$INC$, \lstinline$DEC$}},
    \text{\lstinline$AND$},
    \text{\lstinline$OR$}
\]

It should also be mentioned, that this selection is a rational choice.
Smaller sets would also be possible,
and there are other right to left computable operators
-- they can be combined from the selected ones
(e.g.~shifting left as a sequence of \lstinline$ADD$s).

\begin{quote}
``THEOREM. A function mapping words to words can be implemented
with world-parallel \emph{add}, \emph{subtract}, \emph{or} and \emph{not}
instructions if and only if each bit of the result
depends only on bits at and to the right of each input operand.''
\par\hfill \emph{Hacker's Delight},
page \texttt{12} \cite{Warren:2012:HD:2462741}
\end{quote}

\begin{proof}~
\begin{description}
\item[``$\implies$'':]
In \autoref{sec:rightmost} \lstinline$INC$ and \lstinline$DEC$
where established as right to left operations.
This then also applies to \lstinline$ADD$ and \lstinline$SUB$.
Regarding \lstinline$NOT$, \lstinline$AND$, \lstinline$OR$,
they clearly fulfil the right to left computability property,
since their result bits do not on any adjacent ones at all.

Thereby a formula using only
\lstinline$ADD$, \lstinline$SUB$, \lstinline$AND$,
\lstinline$OR$ and \lstinline$NOT$
only depends on bits at and to the right of each input operand.

\item[``$\impliedby$'':]
The result only depends on bits at and to the right of each input operand.
These can be isolated using \lstinline$AND$ with the argument and a mask
(a constant containing a single \lstinline$1$ at the respective position).

Once every required bit is isolated, the function can be expressed
using only \lstinline$NOT$, \lstinline$OR$ and \lstinline$AND$.
This is true because (as mentioned in \autoref{sec:introduction})
$\{\lnot, \lor, \land\}$ is a complete set of logical operators,
every logical function can be expressed using only its elements.
\end{description}
\end{proof}

The obtained class contains all functions
that can calculate the $n$th result bit
with only knowing the input operands up to their $n$th bit.
This has applications e.g. in bitstreams when working on partial data.


\chapter{Calculus}
\textbf{Definition:} Calculus (from Latin calculus, literally 'small pebble', used for counting and calculations, as on an abacus) is the mathematical study of continuous change. It has two major branches:
\begin{itemize}
	\item \textbf{differential calculus:} concerning rates of change and slopes of curves;
	\item \textbf{integral calculus:} concerning accumulation of quantities and the areas under and between curves
\end{itemize}
These two branches are related to each other by the fundamental theorem of calculus.

\section{Limits}
\subsection{Limits Rules}
If $ \lim_{x \rightarrow c }f(x) = L_1 $ and $ \lim_{x \rightarrow c } g(x) = L_2 $ then:

\begin{itemize}
	\item $ \displaystyle \lim_{x \rightarrow c} \left[ f(x) \pm g(x) \right] = L_1 \pm L_2 $
	\item $ \displaystyle \lim_{x \rightarrow c} \left[ f(x) \cdot g(x) \right] = L_1 \cdot L_2 $
	\item $ \displaystyle \lim_{x \rightarrow c} \frac{f(x)}{g(x)} = \frac{L_1}{L_2} $ \ \ \ \  if $L_2 \ne 0$
	\item $ \displaystyle \lim_{x \rightarrow c} f(x)^n = L_1^n $ \ \ \ \  if $n$ is a positive integer
	\item $ \displaystyle \lim_{x \rightarrow c} f(x)^{\frac{1}{n}} = L_1^{\frac{1}{n}} $ \ \ \ \  if $n$ is a positive integer, and, if $n$ is even, then $L_1>0$
	\item $ \displaystyle \lim_{x \rightarrow c} \frac{f(x)}{g(x)} = \lim_{x \rightarrow c} \frac{f^\prime(x)}{g^\prime(x)}$ \ \ \ \ if $ \lim_{x \rightarrow c } f(x) = \lim_{x \rightarrow c } g(x) = L $ \ \ (L'Hôpital's rule)
\end{itemize}

\subsubsection{Substitution rule}
 $ \displaystyle
\text{Given  } \lim_{x\rightarrow c}f(x)=l \text{   then   } \lim_{x\rightarrow c}g(f(x))=\lim_{y\rightarrow l}g(y)
$

\subsection{Common Limits}
\subsubsection{Simple functions}

\begin{tabularx}{\textwidth}{ l l }
 $ \displaystyle \lim_{x \rightarrow c} a = a $ &
 $ \displaystyle \lim_{x \rightarrow c} x = c $ \\
  $ \displaystyle \lim_{x \rightarrow c} (ax + b) = ac+b $ &
  $ \displaystyle \lim_{x \rightarrow c} x^r = c^r $ \ \ \ if $r$ is a positive integer \\

 $ \displaystyle \lim_{x \rightarrow 0^+} \frac{1}{x^r} = +\infty $ &

 $ \displaystyle \lim_{x \rightarrow 0^-} \frac{1}{x^r} = \begin{cases}
 -\infty & \text{ if $r$ is odd } \\
 +\infty & \text{ if $r$ is even }
 \end{cases}
 $
\end{tabularx}

\subsubsection{Polynomials}
$ \displaystyle \lim_{x\rightarrow\pm\infty}P(x)=\lim_{x\rightarrow\pm\infty}a_nx^x=\pm\infty $

$ \displaystyle \lim_{x\rightarrow\pm\infty}\frac{P(x)}{Q(x)}=\lim_{x\rightarrow\pm\infty}\frac{a_nx^x}{b_mx^m}=\lim_{x\rightarrow\pm\infty}\frac{a_n}{b_m}x^{n-m}=
\begin{cases}
\infty & \text{if } n>m \\
\frac{a_n}{b_m} & \text{if } n=m \\
0 & \text{if } n<m \\
\end{cases} $

\subsubsection{Trigonometric functions}
\begin{tabularx}{\textwidth}{ X X X }

$ \displaystyle \lim_{x\rightarrow0}\frac{\sin(x)}{x}=1 $ &
$ \displaystyle \lim_{x\rightarrow0}\frac{\sin(\alpha x)}{\beta x}=\frac{\alpha}{\beta} $ &
$ \displaystyle  \lim_{x\rightarrow0}\frac{\arcsin(x)}{x}=1 $\\ [1.7ex]

$ \displaystyle \lim_{x\rightarrow0}\frac{1-\cos(x)}{x}=0 $ &
$ \displaystyle \lim_{x\rightarrow0}\frac{1-\cos(x)}{x^2}=\frac{1}{2} $ \\ [1.7ex]

$ \displaystyle \lim_{x\rightarrow0}\frac{\tan(x)}{x}=1 $ &
$ \displaystyle \lim_{x\rightarrow0}\frac{\arctan(x)}{x}=1 $

\end{tabularx}

\subsubsection{Logarithmic and exponential functions}
\begin{tabularx}{\textwidth}{ X X }

$ \displaystyle \lim_{x\rightarrow0}\frac{\ln(x+a)}{x}=a $ &
$ \displaystyle \lim_{x\rightarrow0}\frac{\log_a(x+1)}{x}=\frac{1}{\ln(a)} ~~~ (a>0) $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}x^\alpha= \begin{cases}
+\infty & a>0 \\
0 & a<0
\end{cases} $ &
$ \displaystyle \lim_{x\rightarrow0^+}x^\alpha= \begin{cases}
+\infty & a<0 \\
0 & a>0
\end{cases} $ \\ [2.2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}a^x= \begin{cases}
+\infty & a>1 \\
0 & a<1
\end{cases} $ &
$ \displaystyle \lim_{x\rightarrow-\infty}a^x= \begin{cases}
0 & a>1 \\
+\infty & a<1
\end{cases} $ \\ [2.2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}\log_ax= \begin{cases}
-\infty & a<1 \\
+\infty & a>1
\end{cases} $ &
$ \displaystyle \lim_{x\rightarrow 0^+}\log_ax= \begin{cases}
-\infty & a>1 \\
+\infty & a<1
\end{cases} $ \\ [2.2ex]

$ \displaystyle\lim_{x\rightarrow0}\frac{a^x-1}{x}=ln(a) ~~ \forall a>0 $ &
 $ \displaystyle \lim_{x\rightarrow0}\frac{e^x-1}{x}=1 $ \\[2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}\frac{x^\alpha}{a^x}=0  ~~~~ a>1 $ &
$ \displaystyle \lim_{x\rightarrow-\infty}\frac{e^x}{\left|x\right|^\alpha}=0 ~~~~ a>1 $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}\frac{\ln(x)}{x^\alpha}=0 $ &
$ \displaystyle \lim_{x\rightarrow 0^+}\ln(x)x^\alpha=0 ~~~~ a>0 $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}\frac{\ln(x)}{a^x}=0 ~~~ a>1$ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}\left(1+\frac{1}{x}\right)^x=e $ &
$ \displaystyle \lim_{x\rightarrow+\infty}\left(1-\frac{1}{x}\right)^x=\frac{1}{e} $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow+\infty}\left(\frac{x}{x+k}\right)^x=\frac{1}{e^k} $ &
$ \displaystyle \lim_{x\rightarrow+\infty}\left(\frac{x}{ \sqrt[x]{x!} }\right)=e $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow0}(1+x)^{1/x}={\rm e} $ &
$ \displaystyle \lim_{x\rightarrow+\infty}\left(1+\frac{n}{x}\right)^{mx}={\rm e}^{mn} $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow\infty}\frac{x^p}{a^x}=0~~\mbox{als }|a|>1 $ &
$ \displaystyle \lim_{x\rightarrow0}\frac{(1+x)^\alpha-1}{x}=\alpha ~~~ (a\in\mathbb{R}) $ \\ [2ex]

$ \displaystyle \lim_{x\rightarrow0}\left(a^{1/x}-1\right)=\ln(a) $ &
$ \displaystyle \lim_{x\rightarrow\infty}\sqrt[x]{x}=1 $ \\ [2ex]

\end{tabularx}

\section{Series}
\textbf{Definition:} In mathematics, a series is, roughly speaking, a description of the operation of adding infinitely many quantities, one after the other, to a given starting quantity.

\subsection{Convergence and divergence}
Absolute Convergence: If $\sum\left|s_n\right|$ is convergent.

Conditional Convergence: If $\sum s_n$ is convergent but not absolutely convergent.

If $\sum\limits_n|u_n|$ converges, $\sum\limits_n u_n$ also converges.

If $\lim\limits_{n\rightarrow\infty}u_n\neq0$ then $\sum\limits_n u_n$ is divergent.

\subsection{Positive Series}
Positive Series: If all the terms $ s^n $ are positive.

If $u_n>0~\forall n$ then $\sum\limits_n u_n$ is convergent if
$\sum\limits_n\ln(u_n+1)$ is convergent.

Integral Test: If $ f(n) = s_n $, continuous, positive, decreasing: $\sum s_n$ converges $ \Leftrightarrow \int_{1}^{\infty}f(x)dx $ converges.

Comparison Test: $\sum a_n$ and $\sum b_n$ where $a_k<b_k ~ (\forall k\ge m)
\begin{cases}
\text{If } \sum b_n \text{ converges, so does } \sum a_n \\
\text{If } \sum a_n \text{ diverges, so does } \sum b_n
\end{cases}
$

Limit Comparison Test: $\sum a_n$ and $\sum b_n$ such that $ \lim_{n\rightarrow\infty} \frac{a_n}{b_n}$ exists, $\sum a_n$ converges $\Leftrightarrow$ $\sum b_n$ converges.

Ratio Test: if $\lim_{n\rightarrow \infty}\left|\frac{s_{n+1}}{s_n}\right| = \begin{cases}
<1 \text{ then it is absolutely convergent} \\
1 \text{ then no conclusion} \\
>1 \text{ or } +\infty \text{ then it diverges}
\end{cases}
$

Root Test: if  $\lim_{n\rightarrow \infty} \sqrt[n]{\left|s_n\right|} = \begin{cases}
<1 \text{ then it is absolutely convergent} \\
1 \text{ then no conclusion} \\
>1 \text{ or } +\infty \text{ then it diverges}
\end{cases}
$

\subsection{Alternating Series}
Alternating Series: $ \sum(-1)^{n+1}a_n=a_1-a_2+a_3-a_4+a_5-... $

Leibniz Test: An alternating series of which the absolute values of the terms drop
monotonously to 0 is convergent.

\subsection{Common Series}
\subsubsection{Basic Series}

\begin{tabular}{ l l }
$ \displaystyle \sum_{k=1}^n k = \frac{n(n+1)}{2} $ &
$ \displaystyle \sum_{k=1}^n (2k-1) = n^2 $ \\ [1.5em]
$ \displaystyle \sum_{k=1}^n k^2 = \frac{n(n+1)(2n+1)}{6} $ &
$ \displaystyle \sum_{k=1}^n k^3 = \frac{n^2(n+1)^2}{4} $
\end{tabular}

\subsubsection{Harmonic Series}
\[
\sum_{n=1}^\infty \frac{1}{n^p} ~ \begin{cases}
									\text{is convergent if } p>1 \\
									\text{is divergent if } p\leq1
									\end{cases}
\]

\subsubsection{Geometric Series}
\[
\sum_{0}^{n}q^k=1+q+q^2+...+q^n= \begin{cases}
									n+1 & q=1 \\
									\frac{1-q^{k+1}}{1-q} & q\ne1
								 \end{cases}
~ \text{ then } ~
\sum_{0}^{+\infty}q^k= \begin{cases}
\frac{1}{1-q} & \left|q\right|<1 \\
+\infty & q\ge1 \\
\text{no conclusion} & q\le-1
\end{cases}
\]

\subsubsection{Mengoli Series}
\[
s_n=\sum_{k=2}^{n}\frac{1}{(k-1)k}=1-\frac{1}{n} ~~ \text{ then } ~~  \lim_{n\rightarrow\infty}s_n=1
\]

\section{Landau Symbols}
\subsection{Big-O notation}
The Big-O provides a function that is at most the same order as that of a given function.

If $\exists M>0$ such that $\displaystyle \lim_{x\rightarrow c} \left| \frac{f(x)}{g(x)} \right| < M $ then we say "as $x\rightarrow c, f(x)=O(g(x))$"

\subsection{Little-O notation}
The little-o provides a function that is of lower order of magnitude than a given function.

If $\displaystyle \lim_{x\rightarrow c} \frac{f(x)}{g(x)} = 0 $ then we say "as $x\rightarrow c, f(x)=o(g(x))$"

\section{Asymptotes}
In analytic geometry, an asymptote of a curve is a line such that the distance between the curve and the line approaches zero as one or both of the x or y coordinates tends to infinity.

\subsection{Vertical asymptotes}
The line $x = a$ is a vertical asymptote of the graph of the function $y = f(x)$ if:

 $\displaystyle \lim_{x\rightarrow a^-} f(x) = \pm \infty $ or
 $\displaystyle \lim_{x\rightarrow a^+} f(x) = \pm \infty $

\subsection{Horizontal asymptotes}
The horizontal line $y = c$ is a horizontal asymptote of the function $y = f(x)$ if:

 $\displaystyle \lim_{x\rightarrow -\infty} f(x) = c $ or
 $\displaystyle \lim_{x\rightarrow + \infty} f(x) = c $

\subsection{Oblique asymptotes}
A function $f(x)$ is asymptotic to the straight line $y = mx + n (m \ne 0)$ if:

$\displaystyle \lim_{x\rightarrow +\infty} \left[ f(x) - (mx+n) \right] = 0 $ or
$\displaystyle \lim_{x\rightarrow -\infty} \left[ f(x) - (mx+n) \right] = 0 $

Using the Landau symbols, the formula becomes:

$\displaystyle  f(x) = mx + n + 0(1), x\rightarrow +\infty$

From the last one, it is easy to find that:

$\displaystyle m=\lim_{x\rightarrow +\infty} \frac{f(x)}{x} $  and $\displaystyle q=\lim_{x\rightarrow +\infty} (f(x) - mx) $

\section{Derivatives}
The derivative of a function is the ratio of the difference of function value $ f(x) $ at points $ x+\Delta x $ and $ x $ with $ \Delta x $, when $ \Delta x $ is infinitesimally small. The derivative is the function slope or slope of the tangent line at point x.

\[
f^\prime(x) = \lim_{\Delta x \rightarrow 0} \frac{ f( x + \Delta x ) - f(x) }{\Delta x}
\]

\subsection{Derivatives rules}
\begin{tabular}{ l l }
Sum rule:      & $ \displaystyle (\alpha f(x) \pm \beta g(x) )^\prime = \alpha f^\prime(x) \pm \beta g^\prime(x) $  \\
Product rule:  & $ \displaystyle (f(x) \cdot g(x) )^\prime = f^\prime(x) g(x) + f(x) g^\prime(x) $ \\
Quotient rule: & $ \displaystyle \left( \frac{f(x)}{g(x)} \right)^\prime = \frac{ f^\prime(x) g(x) - f(x) g^\prime(x) }{ g^2(x) } $ \\
Chain rule:    & $ \displaystyle f\left( g(x) \right) ^\prime = f^\prime\left( g(x) \right) \cdot g^\prime(x) $
\end{tabular}

\subsection{Derivative test}

\subsubsection{Second derivative test}
If the function $f$ is twice differentiable at a critical point $x_0$ (i.e. $f^\prime(x_0) = 0$), then:
\begin{itemize}
\item If $ f^{\prime \prime }(x_0)<0 $ then $f$  has a local maximum at $x_0$.
\item If $ f^{\prime \prime }(x_0)>0 $ then $f$  has a local minimum at $x_0$.
\item If $ f^{\prime \prime }(x_0)=0 $ the test is inconclusive.
\end{itemize}

\subsubsection{Concavity test}
A twice-differentiable function $f$ is concave up if $ f^{\prime \prime }(x_0)>0 $ and concave down if $ f^{\prime \prime }(x_0)<0 $.

\subsubsection{Higher order derivative test}
Let $f$ be a real-valued function $n$-times ($n\ge2$) differentiable at a point $x_0$ with
\[
f^\prime(x_0) = ... = f^{(n)}(x_0) = 0 ~~~, ~~~ f^{(n+1)}(x_0) \ne 0
\]
then
\begin{itemize}
	\item If $n$ is odd and  $ f^{(n+1)}(x_0)<0 $ then $f$ has a local maximum at $x_0$.
	\item If $n$ is odd and  $ f^{(n+1)}(x_0)>0 $ then $f$ has a local minimum at $x_0$.
	\item If $n$ is even and  $ f^{(n+1)}(x_0)<0 $ then $x_0$ is a strictly decreasing point of inflection.
	\item If $n$ is even and  $ f^{(n+1)}(x_0)>0 $ then $x_0$ is a strictly increasing point of inflection.
\end{itemize}

\medskip
In addition, when $x_0$ is not a critical point, we have:

Let $f$ be a real-valued function $n$-times ($n\ge3$) differentiable at a point $x_0$ with
\[
f^\prime(x_0)\ne0 ~~~, ~~~ f^{\prime\prime}(x_0) = ... = f^{(n)}(x_0) = 0 ~~~, ~~~ f^{(n+1)}(x_0) \ne 0
\]
then
\begin{itemize}
	\item If $n$ is odd then $x_0$ is not an inflection point.
	\item If $n$ is even and  $ f^{(n+1)}(x_0)<0 $ then $x_0$ is a strictly decreasing point of inflection.
	\item If $n$ is even and  $ f^{(n+1)}(x_0)>0 $ then $x_0$ is a strictly increasing point of inflection.
\end{itemize}

\subsection{Common derivatives}
\subsubsection{Basic Derivatives}
$ \displaystyle f(x)=\alpha ~ \Rightarrow ~ f^\prime(x)=0 $

$ \displaystyle f(x)=x ~ \Rightarrow ~ f^\prime(x)=1 $

$ \displaystyle f(x)=x^n ~ \Rightarrow ~ f^\prime(x)=n x^{n-1} $

\subsubsection{Trigonometric Derivatives}
\begin{tabular}{ l l }
$ \displaystyle f(x)=\sin(x) ~ \Rightarrow ~ f^\prime(x)=\cos(x) $ &
$ \displaystyle f(x)=\arcsin(x) ~ \Rightarrow ~ f^\prime(x)= \frac{1}{ \sqrt{1-x^2} } $ \\ [2ex]
$ \displaystyle f(x)=\cos(x) ~ \Rightarrow ~ f^\prime(x)=-\sin(x) $ &
$ \displaystyle f(x)=\arccos(x) ~ \Rightarrow ~ f^\prime(x)= -\frac{1}{ \sqrt{1-x^2} } $ \\ [2ex]
$ \displaystyle f(x)=\tan(x) ~ \Rightarrow ~ f^\prime(x)=\sec^2(x)= \frac{1}{\cos^2(x)} $ &
$ \displaystyle f(x)=\arctan(x) ~ \Rightarrow ~ f^\prime(x)= \frac{1}{ 1+x^2 } $ \\ [2ex]
$ \displaystyle f(x)=\sec(x) ~ \Rightarrow ~ f^\prime(x)=\sec(x)\tan(x) $ \\ [2ex]
$ \displaystyle f(x)=\cot(x) ~ \Rightarrow ~ f^\prime(x)=-\csc^2(x) $ \\ [2ex]
$ \displaystyle f(x)=\csc(x) ~ \Rightarrow ~ f^\prime(x)=-\csc(x)\cot(x) $
\end{tabular}

\subsubsection{Exponential Derivatives}
\begin{tabular}{ l l }
$ \displaystyle  f(x)=a^x ~ \Rightarrow ~ f^\prime(x)=\ln(a)a^x $ &
$ \displaystyle  f(x)=e^x ~ \Rightarrow ~ f^\prime(x)=e^x $ \\
$ \displaystyle  f(x)=a^{g(x)} ~ \Rightarrow ~ f^\prime(x)=\ln(a)a^{g(x)}g^\prime(x) $ &
$ \displaystyle  f(x)=e^{g(x)} ~ \Rightarrow ~ f^\prime(x)=e^{g(x)}g^\prime(x) $
\end{tabular}

\subsubsection{Logarithm Derivatives}
\begin{tabular}{ l l }
$ \displaystyle  f(x)=\log_a(x) ~ \Rightarrow ~ f^\prime(x)= \frac{1}{ \ln(a)x } $ &
$ \displaystyle  f(x)=\ln(x) ~ \Rightarrow ~ f^\prime(x)= \frac{1}{ x } $ \\ [1.5ex]
$ \displaystyle  f(x)=\log_a(g(x)) ~ \Rightarrow ~ f^\prime(x)= \frac{g^\prime(x)}{ \ln(a)g(x) } $ &
$ \displaystyle  f(x)=\ln(g(x)) ~ \Rightarrow ~ f^\prime(x)= \frac{g^\prime(x)}{ g(x) } $
\end{tabular}

\section{Taylor Series}
A Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point.

\textbf{Taylor's Theorem:}
Let $k \ge 1$ be an integer and let the function $f : \mathbb{R} \rightarrow \mathbb{R}$ be $k$ times differentiable at the point $a \in R$. Then there exists a function $h_k : \mathbb{R} \rightarrow \mathbb{R}$ such that
\[
f(x) = f(a) + f^{\prime}(a)(x-a) + \frac{f^{\prime\prime}(a)}{2!}(x-a)^2 + ... + \frac{f^{k}(a)}{k!}(x-a)^k + h_k ( x ) (x-a)^k
\]
and $\lim_{x \rightarrow a} h_k ( x ) = 0$. This is called the Peano form of the remainder.

The Taylor series of a real or complex-valued function $f(x)$ that is infinitely differentiable at a real or complex number $a$ is the power series:
\[
\sum_{n=0}^{+\infty}\frac{f^{(n)}(a)}{n!}(x-a)^n
\]

\textbf{Maclaurin series:}
When $a = 0$, the series is also called a Maclaurin series.

\textbf{Even/Odd functions and Maclaurin Polynomials:}
\begin{itemize}
	\item If $f$ be an even function, then each of $f$'s Maclaurin polynomials contains only even powers;
	\item If $f$ be an odd function, then each of $f$'s Maclaurin polynomials contains only odd powers.
\end{itemize}

\subsection{Maclaurin series of common functions}

\subsubsection{Basic functions}
$\displaystyle \sqrt{1+x} = 1 + \frac{1}{2}x - \frac{1}{8}x^2 + \frac{1}{16}x^3 + o(x^3)$

\subsubsection{Exponential functions}
$\displaystyle e^x = \sum_{n=0}^{+\infty}\frac{x^n}{n!} = 1 + x + \frac{x^2}{2!} + ... + \frac{x^n}{n!} + o(x^n) $ \ \ \ \ It converges for all $x$

\subsubsection{Logarithm functions}
$\displaystyle \ln(1+x) = \sum_{n=1}^{+\infty}(-1)^{n+1}\frac{x^n}{n} = x - \frac{x^2}{2} + ... + (-1)^{n+1}\frac{x^n}{n} + o(x^n) $ \ \ \ It conv. for $\left|x\right|<1$

\subsubsection{Geometric Series}
$\displaystyle \frac{1}{1+x} = \sum_{n=0}^{+\infty}(-1)^n x^n = 1 - x + x^2 ... + (-1)^n x^n + o(x^n)$

$\displaystyle \frac{1}{1-x} = \sum_{n=0}^{+\infty}x^n $ \ \ \ \ It converges for $\left|x\right|<1$.

$\displaystyle \frac{1}{(1-x)^2} = \sum_{n=1}^{+\infty}nx^{n-1} $ \ \ \ \ It converges for $\left|x\right|<1$.

$\displaystyle \frac{1}{(1-x)^3} = \sum_{n=2}^{+\infty} \frac{(n-1)n}{2} x^{n-2} $ \ \ \ \ It converges for $\left|x\right|<1$.

\subsubsection{Binomial Series}
$\displaystyle (1+x)^\alpha = \sum_{n=0}^{+\infty} \binom{\alpha}{n} x^n $ \ \ It converges for $\left|x\right|<1$ for any real or complex number $\alpha$.

\subsubsection{Trigonometric functions}
\begin{align*}
	\sin(x) &= \sum_{n=0}^{\infty} \frac{(-1)^n}{(2n+1)!}x^{2n+1} &= x - \frac{x^3}{6} + \frac{x^5}{120} - ... ~~~~~~ & \forall x & \\
	\cos(x) &= \sum_{n=0}^{\infty} \frac{(-1)^n}{(2n)!}x^{2n} &= 1 - \frac{x^2}{2} + \frac{x^4}{24} - ... ~~~~~~ & \forall x &\\
	\tan(x) &= \sum_{n=1}^{\infty} \frac{B_{2n}(-4)^n (1-4^n) }{(2n)!}x^{2n-1} &= x + \frac{x^3}{3} + \frac{2x^5}{15} + ... ~~~~~~ & \text{for } |x|<\frac{\pi}{2} &\\
\end{align*}

\subsubsection{Hyperbolic functions}
\begin{align*}
\sinh(x) &= \sum_{n=0}^{\infty} \frac{x^{2n+1}}{(2n+1)!} &= x + \frac{x^3}{3!} + \frac{x^5}{5!} + ... ~~~~~~ & \forall x & \\
\cosh(x) &= \sum_{n=0}^{\infty} \frac{x^{2n}}{(2n)!} &= 1 + \frac{x^2}{2!} + \frac{x^4}{4!} + ... ~~~~~~ & \forall x & \\
\tanh(x) &= \sum_{n=1}^{\infty} \frac{B_{2n} 4^n (4^n-1) }{(2n)!}x^{2n-1} &= x - \frac{x^3}{3} + \frac{2x^5}{15} - \frac{17x^7}{315} + ... ~~~~~~ & \text{for } |x|<\frac{\pi}{2} & \\
\end{align*}

\section{Integral Calculus}
\textbf{Definition:} In mathematics, Integral calculus is a subfield of calculus in which the notion of an integral, its properties and methods of calculation are studied. It concerns accumulation of quantities and the areas under and between curves.

\subsection{Integration rules}
\begin{tabular}{ l l }
	Multiplication by a constant:      & $ \displaystyle \int c f(x)dx = c \int f(x)dx  $  \\
	Sum rule:      & $ \displaystyle \int (f(x) + g(x)) dx = \int f(x)dx + \int g(x)dx $  \\
	Integration by parts:   & $ \displaystyle \int f(x)g^\prime(x)dx = f(x)g(x) + \int f^\prime(x)g(x)dx  $  \\
	Substitution rule: & $ \displaystyle \int f(g(x))g^\prime(x)dx = \int f(y)dy ~~,~~ y=g(x) $  \\
	\end{tabular}

\subsection{Integration of Rational Functions}
A rational function $ \frac{P(x)}{Q(x)}$, where $P(x)$ and $Q(x)$ are both polynomials, can be integrated in four steps:
\begin{enumerate}
\item Reduce the fraction if it is improper (i.e. degree of $P(x)$ is greater than degree of $Q(x)$);
\item Factor $Q(x)$ into linear and/or quadratic (irreducible) factors;
\item Decompose the fraction into a sum of partial fractions;
\item Calculate integrals of each partial fraction.
\end{enumerate}

\subsubsection{Step 1. Reducing an Improper Fraction}
If the fraction is improper, divide the $P(x)$ by $Q(x)$ to obtain
\[\frac{P(x)}{Q(x)}=F(x)+\frac{R(x)}{Q(x)}\]
where $\frac{R(x)}{Q(x)}$ is a proper fraction.

\subsubsection{Step 2. Factoring Q(x) into Linear and/or Quadratic Factors}
Write the denominator $Q(x)$ as
\[
Q(x) = (x-a)^\alpha \cdot\cdot\cdot (x-b)^\beta(x^2+px+q)^\mu \cdot\cdot\cdot (x^2+rx+s)^\nu
\]
where quadratic functions are irreducible, i.e. do not have real roots.

\subsubsection{Step 3. Decomposing the Rational Fraction into a Sum of Partial Fractions}
Write the function as follows:
\[\frac{R(x)}{Q(x)}= \frac{A_1}{(x-a)^{\alpha}} + \frac{A_2}{(x-a)^{\alpha-1}} + ... + \frac{A_\alpha}{(x-a)} + ... + \frac{M_1 x + N_1}{(x^2+rx+s)^{\nu}} + ... + \frac{M_\nu x + N_\nu}{(x^2+rx+s)}\]
Then equate the coefficients of equal powers of $x$ by multiplying both sides of the latter expression by $Q(x)$ and write the system of linear equations in $A_i,B_i,M_i,N_i,...$ .

\subsubsection{Step 4. Integrating partial fractions}
Use the following formulas to evaluate integrals of partial fractions with linear and quadratic denominators:

\begin{enumerate}
	\item $\displaystyle \int \frac{A}{x-a} dx = A \ln \left| x-a \right| +C$

	\item $\displaystyle \int \frac{A}{(x-a)^k} dx = \frac{A}{(1-k)(x-a)^{k-1}} +C$

	\item $\displaystyle \int \frac{1}{(x-b)^2+c^2} dx = \frac{1}{c} \arctan \frac{x-b}{c} + C$

	\item $\displaystyle \int \frac{2x-2b}{(x-b)^2+c^2} dx = \ln \left| (x-b)^2 + c^2 \right| +C$

	\item $\displaystyle \int \frac{1}{(x^2+bx+c)^n} dx = !!TODO!!$

\end{enumerate}

\subsection{Integration of Common Functions - TODO}

\section{Differential calculus}
\textbf{Definition:} In mathematics, differential calculus is a subfield of calculus concerned with the study of the rates at which quantities change.


\section{Transformations and Expectations}

\subsection{Exercise 1}
(a) $F_Y(y) = P(X^3 \leq y) = F_X (y^{\frac{1}{3}})$. Differentiating, we get that
$f_Y(y) = 14y(1 - y^{\frac{1}{3}})$ which integrates to 1 over $(0, 1)$.

(b)  $f_Y(y) = \dv{y} F_X(\frac{y - 3}{4}) = \frac{7}{4} \exp(\frac{-7(y - 3)}{4})$, which again
integrates to 1 over $(3, \infty)$. The sample space of $Y$ consists of $(3, \infty)$ because
$4X + 3$ is monotonically increasing and $4(0) + 3 = 3$.

(c) We need only consider positive square roots, since  $x \in (0, 1)$. Thus,
$f_Y(y) = \frac{1}{2\sqrt{y}}30y (1 - \sqrt{y})^2$, which integrates to 1 over $(0, 1)$.

\subsection{Exercise 2}
(a) $f_Y(y) = \frac{1}{2\sqrt{y}} f_X(\sqrt{y}) = \frac{1}{2\sqrt{y}}$

(b) Plug $x = e^{-y}$ into $f_X(x)$ and multiply by $e^{-y}$ (the negative of the derivative, since $-\log X$
is decreasing).

(c) Plug in  $x = \log y$ and multiply by $\frac{1}{y}$.

\subsection{Exercise 3}
From the definition of $Y$, we can see that the sample space is
$\mathcal{Y} = \big(\frac{n}{n+1}\big)_{\mathbb{N}}$. Solving $y = \frac{x}{x + 1}$ for $y$, we find that
$f_Y(y) = f_X(\frac{y}{1 - y})$.

\subsection{Exercise 4}
(a) Integrating from $-\infty$ to 0 gives $\frac{1}{2}$ and likewise for 0 to  $\infty$, so $f$ is a pdf.

(b) If $t \leq 0$, then we have that $P(X < t) = \frac{1}{2} e^{\lambda t}$. Otherwise,
\begin{align*}
        P(X < t) &= \int_{-\infty}^{t} f(x) = \int_{-\infty}^{0} f(x) + \int_{0}^{t} f(x) \\
                 &= \frac{1}{2} + \frac{1}{2} - \frac{1}{2} e^{-\lambda t} \\
                 &= 1 - \frac{1}{2}e^{-\lambda t}
\end{align*}

(c) If $t \leq 0$, $P(\abs{X} < t)$ is clearly 0. Otherwise,
\begin{align*}
        P(\abs{X} < t) &= P(-t < X < t) = P(X < t) - P(X < -t) \\
                       &= 1 - e^{- \lambda t}
\end{align*}
from the CDFs computed in part (b).

\subsection{Exercise 5}
The sample space $\mathcal{Y}$ is $[0, 1]$. Thus, we need to consider
$P(Y \leq y) = P(X \leq \sin^{-1} \sqrt{y})$ for $y \in [0, 1]$. By symmetry, we can just consider the case
where $\sin^{-1}$ is restricted to $[0, \frac{\pi}{2}]$, as the other three quadrants have the same area.
Therefore,
\begin{align*}
        f_Y(y) &= 4 \dv{y} F_X(\sin^{-1} \sqrt{y}) \\
               &= 4 \dv{y} \frac{\sin^{-1} \sqrt{y}}{2\pi} \\
               &= \frac{1}{\pi \sqrt{y (1 - y)}}
\end{align*}
At $y = 0$ and $y = 1$, the density is infinite/undefined - I'm not really sure how to interpret this.

\subsection{Exercise 11}
(a) We have that
\begin{align*}
        \mathrm{E} [X^2] = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty} x^2 e^{-\frac{x^2}{2}} &= \frac{2}{\sqrt{2\pi}} \int_{0}^{\infty} x^2 e^{-\frac{x^2}{2}} \\
                                                                            &= \frac{2}{\sqrt{2\pi}} \big(\lim_{x \to \infty} -x e^{-\frac{x^2}{2}} + \int_{0}^{\infty} e^{-\frac{x^2}{2}}\big) \\
                                                                            &= 1
\end{align*}
Where we integrated by parts using $dv = xe^{-\frac{x^2}{2}}$ and $u = x$, applied L'H\^opital's rule,
and then compared to the CDF of the normal distribution.

(b) The pdf $f_Y(y)$ is just $f_X(y) + f_X(-y)$, so $f_Y(y) = \frac{2}{\sqrt{2\pi}} e^{-\frac{y^2}{2}}$.
We can then compute $\mathrm{E} [Y]$ as
\begin{align*}
        \mathrm{E} [Y] = \frac{2}{\sqrt{2\pi}} \int_{0}^{\infty} y e^{-\frac{y^2}{2}} &= \frac{2}{\sqrt{2\pi}}
\end{align*}
To compute the variance we need to compute $\mathrm{E} [Y^2]$, which is identical to $\mathrm{E} [X^2]$. Thus,
$\mathrm{Var} [Y] = \mathrm{E} [Y^2] - \mathrm{E} [Y]^2 = 1 - \frac{2}{\pi}$.

\subsection{Exercise 12}
We see that $y = d \tan(x)$. Since $\tan(x)$ is increasing on $(0, \frac{\pi}{2})$, we get that
\begin{align*}
        f_Y(y) &= F_X\big(\tan^{-1} \frac{y}{d}\big) \dv{y} \tan^{-1} \frac{y}{d} \\
               &= \frac{2\tan^{-1} \frac{y}{d}}{\pi d\big(1 + \frac{y^2}{d^2}\big)}
\end{align*}
for $y \in (0, \infty)$. We can compute $\mathrm{E} [Y]$ directly from $f_X(x)$ as
\begin{align*}
        \mathrm{E} [Y] &= \frac{2d}{\pi}\int_{0}^{\frac{\pi }{2}} \tan(x) dx \\
                       &= \infty
\end{align*}
So $\mathrm{E} [Y]$ does not exist.

\subsection{Exercise 13}
A sequence of flips has length $l$ if there are either $l$ heads in a row or $l$ tails in a row, so
$P(X = l) = p^l (1 - p) + (1 - p)^l p = p \text{Geom}(1 - p) + (1 - p) \text{Geom}(p)$. Therefore,
by linearity of expected value, $\mathrm{E} [X] = \frac{p}{1 - p} + \frac{1 - p}{p}$.

\subsection{Exercise 14}
(a) We have that
\begin{align*}
        \int_{0}^{\infty} 1 - F_X(x) dx = \int_{x = 0}^{\infty} \int_{y = x}^{\infty} f_X(y) dy dx
\end{align*}
Since $0 \leq x \leq \infty$ and $x \leq y \leq \infty$, we can change the order of integration by having
$0 \leq y \leq \infty$ on the outside and $0 \leq x \leq y$ on the inside
\begin{align*}
        \int_{x = 0}^{\infty} \int_{y = x}^{\infty} f_X(y) dy dx &= \int_{y = 0}^{\infty} \int_{x = 0}^{y} dx f_X(y) dy \\
                                                                 &= \int_{0}^{\infty} y f_X(y) dy \\
                                                                 &= E[X]
\end{align*}

(b) We can rewrite the expected value as a sum of infinite sums to see that
\begin{align*}
        E[X] &= \sum_{k = 1}^\infty k f(k) = \sum_{k = 1}^\infty f(k) + \sum_{k = 2}^\infty f(k) + ... \\
             &= \sum_{k = 0}^\infty \sum_{j = k + 1}^\infty f(k) = \sum_{k = 0}^\infty (1 - F_X(k))
\end{align*}

\subsection{Exercise 17}
(a) We need to solve $m^3 = 1 - m^3$, since $F_X(x) = x^3$. Solving gives $m = \sqrt[3]{\frac{1}{2}}$.

(b) $f$ is an even function, so $m = 0$.

\subsection{Exercise 18}
We differentiate under the integral sign to get
\begin{align*}
        \dv{a} \mathrm{E} [\abs{X - a}] &= \dv{a} \bigg(\int_{a}^{\infty} (x - a) f_X(x) dx + \int_{-\infty}^{a} (a - x) f_X(x) dx\bigg) \\
                                        &= \int_{a}^{\infty} -f_X(x) dx + \int_{-\infty}^{a} f_X(x) dx \\
                                        &= 2F_X(a) - 1
\end{align*}
Setting the last line equal to 0 yields $a = m$. To see that this is a minimum, we note that $2F_X(a) - 1 \leq 0$
when $a < m$ and $2F_X(a) - 1 \geq 0$ when $a > m$.

\subsection{Exercise 25}
(a)
\begin{align*}
        F_{-X}(x) = P(X \geq -x) &= 1 - F_X(-x) \\
                                 &= \int_{-x}^{\infty} f_X(x) dx \\
                                 &= \int_{-x}^{0} f_X(x) dx + \int_{0}^{\infty} f_X(x) dx \\
                                 &= \int_{0}^{x} f_X(x) dx + \int_{-\infty}^{0} f_X(x) dx \\
                                 &= F_X(x)
\end{align*}

(b)
\begin{align*}
        M_X(t) &= \int_{-\infty}^{\infty} e^{tx} f_X(x) dx \\
               &= \int_{-\infty}^{0} e^{tx} f_X(x) dx + \int_{0}^{\infty} e^{tx} f_X(x) dx \\
               &= \int_{0}^{\infty} e^{-tx} f_X(x) dx + \int_{-\infty}^{0} e^{-tx} f_X(x) dx \\
               &= M_X(-t)
\end{align*}

\subsection{Exercise 29}
(a) For binomial, we have
\begin{align*}
        \mathrm{E} [X(X - 1)] &= \sum_{k = 0}^n k(k - 1) \binom{n}{k} p^k (1 - p)^{n - k} \\
                              &= n (n - 1) p^2 \sum_{k = 2}^n \binom{n - 2}{k - 2} p^(k - 2) (1 - p)^{(n - 2) - (k - 2)} \\
                              &= n (n - 1) p^2
\end{align*}
For Poisson, we have
\begin{align*}
        \mathrm{E} [X(X - 1)] &= \sum_{k = 0}^{\infty} k(k - 1) \frac{\lambda^k e^{-\lambda}}{k!} \\
                              &= \lambda^2 \sum_{k = 2}^{\infty} \frac{\lambda^{k - 2} e^{-\lambda}}{(k - 2)!} \\
                              &= \lambda^2
\end{align*}

(b) To compute the variances, we can use linearity of expectation to see that
$\mathrm{E} [X(X - 1)] + \mathrm{E} [X] - \mathrm{E} [X]^2 = \mathrm{Var} [X]$. So for binomial, we get
\begin{align*}
        \mathrm{Var} [X] &= n (n - 1) p^2 + np - n^2 p^2 = np - np^2 = np(1 - p)
\end{align*}
And for Poisson, we get
\begin{align*}
        \mathrm{Var} [X] &= \lambda^2 + \lambda - \lambda^2 = \lambda
\end{align*}

(c) This one is kind of a pain to typeset; the technique is the same as (a), although more involved.

\subsection{Exercise 31}
No such distribution exists, since $M_X(0) = 0$, which contradicts the fact that $\mathrm{E} [1] = 1$.


\documentclass{memoir}
\usepackage{linalg}

\begin{document}
% Covers 11-20-19 and 11-22-19 (probably)

\section{Positive Operators and Isometries}
\label{sec:positive_operators_and_isometries}
\begin{defn}[Positive Operator]
	An operator \(T \in \mathcal{L}(V)\) is called \textbf{positive} if \(T\) is self-adjoint and
	\begin{align*}
		\langle Tv, v \rangle \geq 0
	\end{align*}
	for all \(v \in V\).
\end{defn}
If \(V\) is a complex vector space, then \(T\) does not need to be self-adjoint (if an inner product is real for all values, then the operator must be self-adjoint; positivity implies real).
\begin{defn}[Square Root]
	An operator \(R\) is called a \textbf{square root} of an operator \(T\) if \(R^2 = T\).
\end{defn}
\begin{lemma}[Properties of Positive Operators]
	Let \(T \in \mathcal{L}(V)\). Then the following are equivalent:
	\begin{itemize}
		\item \(T\) is positive
		\item \(T\) is self-adjoint and all the eigenvalues of \(T\) are nonnegative
		\item \(T\) has a positive square root
		\item \(T\) has a self-adjoint square root
		\item there exists an operator \(R \in \mathcal{L}(V)\) such that \(T = R^{*}R\).
	\end{itemize}
\end{lemma}
\begin{cor}
	Each positive operator on \(V\) has a unique positive square root.
\end{cor}

Using more tools, one can further state that if \(V\) is a complex vector space and \(T\) an invertible operator, then \(T\) has a square root.

\subsection{Isometries}
\label{subsec:isometries}

We can impose one further condition on isomorphisms to strengthen the relation.
\begin{defn}[Isometry]
	An operator \(A \in \mathcal{L}(V)\) is an \textbf{isometry} if
	\begin{align*}
		\|Av\| = \|v\|
	\end{align*}
	for all \(v \in V\).
\end{defn}
Note that we haven't explicitly stated that isometries are isomorphisms. It turns out that this follows directly from the definition above.

\begin{prop}[Characterization of Isometries]
	Let \(A \in \mathcal{L}(V)\). The following are equivalent:
	\begin{itemize}
		\item \(A\) is an isometry
		\item \(\langle Au,Av \rangle = \langle u,v \rangle \) for all \(u,v \in V\)
		\item \(Ae_1,\ldots,Ae_n\) is orthonormal for all choices of orthonormal vectors \(e_1,\ldots,e_n \in V\)
		\item There exists an orthonormal basis \(e_1,\ldots,e_n \in V\) such that \(Se_1,\ldots,Se_n\) is orthonormal
		\item \(S^{*}S = SS^{*}= I\)
		\item \(S^{*}\) is an isometry
		\item \(S\) is invertible and \(S^{-1} = S^{*}\)
	\end{itemize}
\end{prop}
Of course, this gives us that isometries are normal. This in fact gives us one more vital characterzation:
\begin{prop}
	Let \(V\) be a complex inner product space and \(A \in \mathcal{L}(V)\). Then there is an orthonormal basis of \(V\) consisting of eigenvalues of \(A\) with corresponding eigenvalues equal to \(\left| 1 \right| \) if and only if \(A\) is an isometry.
\end{prop}
\end{document}


\documentclass[12pt,a4paper]{article}
\usepackage{amsmath,amsthm,amsfonts,tikz}
\usepackage[euler-digits]{eulervm}
\usepackage{hyperref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newcommand{\iprod}[1]{\langle#1\rangle}
\newcommand{\vecp}[1]{\boldsymbol{p}}
\newcommand{\hatvecp}{\hat{\boldsymbol{p}}}
\newcommand{\Poly}{\mathbb{P}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{Notes on Gauss Quadrature for General Weight Functions}
\author{Bill McLean}
\date{\today}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
\maketitle
\tableofcontents
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Three-term recurrence relation}
Let $\Poly_k$ denote the $(k+1)$-dimensional vector space of real
polynomials with degree at most~$k$, and denote the
infinite-dimensional space of all polynomials by~$\Poly$.  Let $\mu$
be a positive measure on the real line with the property that,
$\int f(x)^2\,d\mu(x)=0$~and $f\in\Poly$ then $f=0$.
We can then define an inner product and norm on~$\Poly$  by
\[
\iprod{f,g}=\int f(x)g(x)\,d\mu(x)\quad\text{and}\quad
\|f\|=\sqrt{\iprod{f,f}}.
\]
Using the Gram--Schmidt procedure to orthogonalise the linearly
independent monomials $1$, $x$, $x^2$, $x^3$, \dots, yields the
polynomial~$p_j(x)$ of degree~$j$ by
\[
p_j(x)=x^j-\sum_{k=0}^{j-1}\frac{\iprod{x^j,p_k}}{\|p_k\|^2}\,p_k(x)
	\quad\text{for $j=0$, $1$, $2$, \dots.}
\]
Therefore, $p_j$ is the unique monic polynomial in~$\Poly_j$ such
that $p_j\perp\Poly_{j-1}$; in particular,
\begin{equation}\label{eq: orthog}
\iprod{p_j,p_k}=0\quad\text{if $j\ne k$.}
\end{equation}

\begin{theorem}\label{thm: 3 term}
The orthogonal polynomials defined above satisfy the three-term
recurrence relation
\begin{equation}\label{eq: 3 term pj}
p_j(x)=(x-a_j)p_{j-1}(x)-b_j^2p_{j-2}(x)
	\quad\text{for $j=1$, $2$, $3$, \dots,}
\end{equation}
with $p_0(x)=1$ and $p_{-1}(x)=0$. The coefficients satisfy
\[
a_j=\frac{\iprod{xp_{j-1},p_{j-1}}}{\|p_{j-1}\|^2}
\quad\text{and}\quad
b_j=\frac{\|p_{j-1}\|}{\|p_{j-2}\|},
\]
except that, by convention, we put $b_1=\sqrt{\int d\nu}=\|p_0\|$.
\end{theorem}
\begin{proof}
Since $p_j(x)-xp_{j-1}(x)$ has degree at most~$j-1$, there are
constants~$c_{jk}$ such that
\[
p_j(x)-xp_{j-1}(x)=\sum_{k=0}^{j-1}c_{jk}p_k(x).
\]
Taking the inner product of both sides with~$p_{j-1}$ and using
the orthogonality property~\eqref{eq: orthog}, we obtain
\[
-\iprod{xp_{j-1},p_{j-1}}=c_{j,j-1}\|p_{j-1}\|^2,
\]
showing that $c_{j,j-1}=-a_j$.  Also, taking the inner product of both
sides with~$p_k$ for $k\le j-3$ gives
\[
c_{jk}\|p_k\|^2=\iprod{p_j-xp_{j-1},p_k}=-\iprod{p_{j-1},xp_k}=0,
\]
so $p_j(x)=(x-a_j)p_{j-1}(x)+c_{j,j-2}p_{j-2}(x)$ and
\[
0=\iprod{p_j,p_{j-2}}=\iprod{xp_{j-1},p_{j-2}}+c_{j,j-2}\|p_{j-2}\|^2.
\]
Finally,
\begin{align*}
\iprod{xp_{j-1},p_{j-2}}&=\iprod{p_{j-1},xp_{j-2}}
	=\iprod{p_{j-1},p_{j-1}}+\iprod{p_{j-1},xp_{j-2}-p_{j-1}}\\
	&=\|p_{j-1}\|^2+0,
\end{align*}
and therefore $c_{j,j-2}=-\|p_{j-1}\|^2/\|p_{j-2}\|^2=-b_j^2$.
\end{proof}

\begin{corollary}
$\|p_j\|=b_1b_2\cdots b_{j+1}$ for $j=2$, $3$, \dots.
\end{corollary}

Recall that the Lagrange interpolation polynomial
\[
\ell_j(x)=\prod_{\substack{k=1\\ k\ne j}}^n\frac{x-x_k}{x_j-x_k}.
\]
has the property that $\ell_j(x_k)=\delta_{jk}$. The zeros of~$p_n$
have the following important properties.

\begin{theorem}
Suppose that $d\mu(x)=w(x)\,dx$ with $w(x)>0$ for $a<x<b$.
(We allow $a=-\infty$ or $b=\infty$.) The orthogonal polynomial has
only real and simple zeros~$x_j$ in the open interval~$(a,b)$, that
is,
\[
a<x_1<x_2<\cdots<x_n<b.
\]
\end{theorem}
\begin{proof}
Suppose for a contradiction that $p_n$ has a zero $\alpha+i\beta$
with~$\beta\ne0$.  Since $p_n$ has real coefficients,
the complex conjugate~$\alpha-i\beta$ is also a zero and thus
\[
p_n(x)=\bigl[x-(\alpha+i\beta)\bigr]\bigl[x-(\alpha-i\beta)\bigr]q(x)
	=[(x-\alpha)^2+\beta^2]q(x)
\]
with $n\ge2$~and $q\in\Poly_{n-2}$. Since $p_n\perp\Poly_{n-2}$,
\[
0=\iprod{p_n,q}=\iprod{[(x-\alpha)^2+\beta^2],q}\\
	=\|(x-\alpha)q\|^2+\beta^2\|q\|^2,
\]
implying that $q=0$ and thus $p_n=0$, a contradiction.

Likewise, suppose for a contradiction that $p_n$ has a multiple real
root~$\alpha$ and write $p_n(x)=(x-\alpha)^2q(x)$ with
$q\in\Poly_{n-2}$.  Then
\[
0=\iprod{p_n,q}=\iprod{(x-\alpha)^2q,q}=\|(x-\alpha)q\|^2,
\]
again implying that $q=0$ and thus $p_n=0$.  We conclude that $p_n$
has only distinct real zeros~$x_j$.

Thus, for $1\le j\le n$,
\[
p_n(x)=\prod_{k=1}^n(x-x_k)=c_j(x-x_j)\ell_j(x)
\quad\text{where}\quad
c_j=\prod_{\substack{k=1\\ k\ne j}}^n(x_j-x_k)\ne0.
\]
Since $\ell_j\in\Poly_{n-1}$ and $p_n\perp\Poly_{n-1}$,
\[
0=\iprod{p_n,\ell_j}=\iprod{c_j(x-x_j)\ell_j,\ell_j}
	=c_j\bigl[(x\ell_j,\ell_j)-x_j\|\ell_j\|^2\bigr],
\]
implying that
\[
x_j=\frac{\iprod{x\ell_j,\ell_j}}{\|\ell_j\|^2}.
\]
By integrating the strict inequalities
\[
a\ell_j(x)^2w(x)<x\ell_j(x)^2w(x)<b\ell_j(x)^2w(x)
	\quad\text{for $a<x<b$,}
\]
we see that $a\|\ell_j\|^2<\iprod{x\ell_j,\ell_j}<b\|\ell_j\|^2$
and therefore $a<x_j<b$, as claimed.
\end{proof}

We denote the ortho\emph{normal} polynomials by
\[
\hat p_j(x)=\frac{p_j(x)}{\|p_j\|},
\]
and find that
\[
b_{j+1}\hat p_j(x)=(x-a_j)\hat p_{j-1}(x)-b_j\hat p_{j-2}(x),
\]
or equivalently,
\begin{equation}\label{eq: 3 term orthog}
b_j\hat p_{j-2}(x)+a_j\hat p_{j-1}(x)+b_{j+1}\hat p_j(x)=xp_{j-1}(x).
\end{equation}

\begin{theorem}[Christoffel--Darboux identity]
\label{thm: Christoffel-Darboux}
The monic orthogonal polynomials satisfy
\[
\frac{1}{b_{n+2}}\sum_{k=0}^n\hat p_k(x)\hat p_k(y)
	=\frac{\hat p_{n+1}(x)\hat p_n(y)-\hat p_n(x)\hat p_{n+1}(y)}{x-y}
	\quad\text{for $x\ne y$.}
\]
\end{theorem}
\begin{proof}
Using \eqref{eq: 3 term orthog}, we see that
\begin{align*}
(x-y)\hat p_{k-1}(x)&\hat p_{k-1}(y)
	=[x\hat p_{k-1}(x)]\hat p_{k-1}(y)
		-\hat p_{k-1}(x)[y\hat p_{k-1}(y)]\\
	&=\bigl[
	b_k\hat p_{k-2}(x)+a_k\hat p_{k-1}(x)+b_{k+1}\hat p_k(x)
	\bigr]\hat p_{k-1}(y)\\
	&\qquad{}-\hat p_{k-1}(x)\bigl[
	b_k\hat p_{k-2}(y)+a_k\hat p_{k-1}(y)+b_{k+1}\hat p_k(y)\bigr]\\
	&=\bigl[b_k\hat p_{k-2}(x)\hat p_{k-1}(y)
		-b_{k+1}\hat p_{k-1}(x)\hat p_k(y)\bigr]\\
	&\qquad{}+\bigl[b_{k+1}\hat p_k(x)\hat p_{k-1}(y)
		-b_k\hat p_{k-1}(x)\hat p_{k-2}(y)\bigr]\\
\end{align*}
and so
\begin{align*}
(x-y)\sum_{k=1}^{n+1}\hat p_{k-1}(x)\hat p_{k-1}(y)
	&=b_1\hat p_{-1}(x)\hat p_0(y)-b_{n+2}\hat p_n(x)\hat p_{n+1}(y)\\
	&\qquad{}+b_{n+2}\hat p_{n+1}(x)\hat p_n(y)
		-b_1\hat p_0(x)\hat p_{-1}(y).
\end{align*}
\end{proof}

\subsection{Legendre polynomials}
The Legendre polynomials have the orthogonality property
\begin{equation}\label{eq: legendre orthog}
\int_{-1}^1 P_k(x)P_l(x)\,dx=\frac{2\delta_{kl}}{2k+1}
\end{equation}
and the 3-term recurrence relation is
\begin{equation}\label{eq: legendre 3 term}
kP_k(x)=(2k-1)xP_{k-1}(x)-(k-1)P_{k-2}(x).
\end{equation}
The normalised Legendre polynomials are
\[
\hat P_k(x)=\frac{P_k(x)}{\|P_k\|}=\biggl(\frac{2k+1}{2}\biggr)^{1/2}
	P_k(x),
\]
and thus
\begin{multline*}
k\biggl(\frac{2}{2k+1}\biggr)^{1/2}\hat P_k(x)
	=(2k-1)\biggl(\frac{2}{2k-1}\biggr)^{1/2}x\hat P_{k-1}(x)\\
	-(k-1)\biggl(\frac{2}{2k-3}\biggr)^{1/2}\hat P_{k-2}(x),
\end{multline*}
or equivalently,
\[
\frac{k\hat P_k(x)}{\sqrt{2k+1}}=\sqrt{2k-1}\,x\hat P_{k-1}(x)
	-\frac{k-1}{\sqrt{2k-3}}\,\hat P_{k-2}(x).
\]
Thus, the Legendre polynomials satisfy
\[
b_{k+1}\hat P_k(x)=(x-a_k)\hat P_{k-1}(x)-b_k\hat P_{k-2}(x)
\]
where
\[
a_k=0\quad\text{and}\quad b_k=\frac{k-1}{\sqrt{(2k-1)(2k-3)}},
\]
except that $b_1=\sqrt{\int_{-1}^1\,dx}=\sqrt{2}$.

\section{Gauss quadrature}
We now consider a quadrature rule
\begin{equation}\label{eq: Qf}
Qf=\sum_{j=1}^n w_jf(x_j)
\end{equation}
and its associated error functional
\[
Ef=\int f(x)\,d\mu(x)-\sum_{j=1}^n w_jf(x_j).
\]

\begin{theorem}\label{thm: Gauss rule}
For each $n\ge1$ there exists a unique quadrature rule~\eqref{eq: Qf}
such that
\begin{equation}\label{eq: dop 2n-1}
Ef=0\quad\text{for all $f\in\Poly_{2n-1}$.}
\end{equation}
This rule has has as its integration points $x_1$, $x_2$, \dots,
$x_n$ the zeros of~$p_n$, that is,
\[
p_n(x_j)=0\quad\text{for $1\le j\le n$,}
\]
and as its weights the numbers
\begin{equation}\label{eq: Gauss weights}
w_j=\iprod{\ell_j,1}=\int\ell_j(x)\,d\mu(x)>0
	\quad\text{for $1\le j\le n$,}
\end{equation}
where $\ell_j$ is the $j$th Lagrange interpolation polynomial:
\[
\ell_j(x)=\prod_{\substack{k=1\\ k\ne j}}\frac{x-x_k}{x_j-x_k}.
\]
\end{theorem}
\begin{proof}
Let $x_1$, $x_2$, \dots, $x_n$ be the zeros of~$p_n$, and define
$w_j=\iprod{\ell_j,1}$.  Given $f\in\Poly_{2n-1}$, let $q$~and $r$
denote the quotient and remainder when $f$ is divided by~$p_n$, that
is,
\[
f(x)=p_n(x)q(x)+r(x)\quad\text{with $r\in\Poly_{n-1}$.}
\]
Since $p_n(x_j)=0$ it follows that $f(x_j)=r(x_j)$ and so
\[
r(x)=\sum_{j=1}^n f(x_j)\ell(x),
\]
and since $q\in\Poly_{n-1}$ and $p_n$ is orthogonal to~$\Poly_{n-1}$,
\begin{multline*}
\int f(x)\,d\mu(x)=\int q(x)p_n(x)\,d\mu(x)+\int r(x)\,d\mu(x)\\
	=\iprod{q,p_n}+\sum_{j=1}^n f(x_j)\int\ell_j(x)\,d\mu(x)
	=0+\sum_{j=1}^n f(x_j)\iprod{\ell_j,1}=Qf,
\end{multline*}
showing that $Ef=0$.

To prove uniqueness, let $Q$ be any quadrature rule~\eqref{eq: Qf}
satisfying \eqref{eq: dop 2n-1}.  Consider the
polynomial~$f\in\Poly_n$ defined by
\[
f(x)=(x-x_1)(x-x_2)\cdots(x-x_n).
\]
If $0\le k\le n-1$ then $x^kf(x)\in\Poly_{2n-1}$ so
\[
\iprod{x^k,f}=\int x^kf(x)\,d\mu(x)=\sum_{j=1}^n w_jx_j^kf(x_j)=0.
\]
Hence, $f\perp\Poly_{n-1}$, and since $f$ is monic it follows that
$f=p_n$ by the uniqueness of the orthogonal polynomials.  Therefore,
the $x_j$ are the zeros of~$p_n$, and moreover
\[
w_j=\sum_{k=1}^nw_k\ell_j(x_k)=\int\ell_k(x)\,d\mu(x)=\iprod{\ell_j,1}
\]
since $\ell_j\in\Poly_{n-1}$.  In fact, $\ell_j^2\in\Poly_{2n-2}$ so
\[
w_j=\sum_{k=1}^nw_k\ell_j(x_k)^2=\int\ell_j(x)^2\,d\mu(x)
	=\|\ell_j\|^2>0,
\]
and the weights satisfy~\eqref{eq: Gauss weights}.
\end{proof}

The quadrature in the theorem above is called the $n$-point Gauss
rule generated by the positive measure~$\mu$.

Defining
\[
T_n=\begin{bmatrix}
a_1&b_2   &       &       &\\
b_2&a_2   &b_3    &       &\\
   &\ddots&\ddots &\ddots &\\
   &      &b_{n-1}&a_{n-1}&b_n\\
   &      &       &b_n    &a_n
\end{bmatrix}
\quad\text{and}\quad
\hatvecp_n(x)=\begin{bmatrix}
	\hat p_0(x)\\ \hat p_1(x)\\ \vdots\\ \hat p_{n-1}(x)
\end{bmatrix},
\]
the equations~\eqref{eq: 3 term orthog} for $0\le j\le n-1$ are
equivalent to
\begin{equation}\label{eq: matrix 3 term orthog}
T_n\hatvecp_n(x)+b_{n+1}\hat p_n(x)\boldsymbol{e}_n=x\hatvecp_n(x).
\end{equation}
The points~$x_j$ and weights~$w_j$ may be computed by solving an
eigenproblem for the \emph{Jacobi matrix}~$T_n$.

\begin{theorem}
The eigenvalues of~$T_n$ are the points $x_1$, $x_2$, \dots, $x_n$
of the associated Gauss quadrature rule.  Moreover, for~$1\le j\le n$,
if
\[
\boldsymbol{v}_j=\begin{bmatrix}v_{1j}\\ v_{2j}\\ \vdots\\ v_{nj}
\end{bmatrix}
\]
denotes the eigenvector of~$T_n$ satisfying
\[
T_n\boldsymbol{v}_j=x_j\boldsymbol{v}_j
\quad\text{and}\quad
v_{1j}=\frac{1}{b_1},
\]
then the weight~$w_j$ corresponding to the $j$th Gauss point~$x_j$
is given by
\[
\frac{1}{w_j}=\sum_{k=1}^n(v_{kj})^2.
\]
\end{theorem}
\begin{proof}
We see at once from~\eqref{eq: matrix 3 term orthog} that
$T_n\hatvecp_n(x)=x\hatvecp_n(x)$ iff $p_n(x_j)=0$, so the
eigenvalues~$x_j$ of~$T_n$ coincide with zeros of~$p_n$ which,
by Theorem~\ref{thm: Gauss rule}, coincide with the Gauss points.
Moreover, $\boldsymbol{v}_j=\hatvecp_n(x_j)$ is the eigenvector
of~$T_n$ that corresponds to the eigenvalue~$x_j$ and is scaled in
such a way that $v_{1k}=\hat p_0(x_j)=1/\|p_0\|=1/b_1$.

Since $p_n$ is monic with zeros $x_1$, $x_2$, \dots, $x_n$,
\[
p_n(x)=\prod_{j=1}^n(x-x_j)
\qquad\text{and thus}\qquad
p_n'(x)=\sum_{j=1}^n\prod_{\substack{k=1\\ k\ne j}}^n(x-x_k).
\]
In particular
\[
p_n'(x_j)=\prod_{\substack{k=1\\ k\ne j}}^n(x_j-x_k),
\]
showing that
\[
p_n(x)=p_n'(x_j)(x-x_j)\ell_j(x)\quad\text{where}\quad
\ell_j(x)=\prod_{\substack{k-1\\ k\ne j}}\frac{x-x_k}{x_j-x_k}.
\]
Hence, $\hat p_n(x)=\hat p_n'(x_j)(x-x_j)\ell_j(x)$ and so
\begin{equation}\label{eq: w_j}
w_j=\iprod{\ell_j,1}=\frac{1}{\hat p_n'(x_j)}
	\int_a^b\frac{\hat p_n(x)}{x-x_j}\,w(x)\,dx
\end{equation}
Putting $y=x_j$ in the Christoffel--Darboux identity of
Theorem~\ref{thm: Christoffel-Darboux} gives
\[
\frac{1}{b_{n+1}}\sum_{k=0}^n\hat p_k(x)\hat p_k(x_j)
	=-\frac{\hat p_n(x)}{x-x_j}\,\hat p_{n+1}(x_j)
\]
and after taking the inner product of both sides with~$p_0(x)=1$
we obtain
\[
\frac{1}{b_{n+2}}\sum_{k=0}^n\iprod{\hat p_k,p_0}\hat p_k(x_j)
	=-\hat p_{n+1}(x_j)\int_a^b\frac{\hat p_n(x)}{x-x_j}\,w(x)\,dx.
\]
Here, $\iprod{\hat p_k,p_0}=0$ for~$k\ge1$ and
$\iprod{\hat p_0,p_0}=\iprod{p_0,p_0}/\|p_0\|=\|p_0\|$, so
by~\eqref{eq: w_j}
\[
\frac{1}{b_{n+2}}=-\hat p_{n+1}(x_j)\hat p_n'(x_j)w_j.
\]
The Christoffel--Darboux identity also gives
\begin{align*}
\frac{1}{b_{n+2}}\sum_{k=0}^n\hat p_k(x_j)^2
&=\lim_{y\to x_j}\frac{1}{b_{n+2}}\sum_{k=0}^n
	\hat p_k(x_j)\hat p_k(y)
=\lim_{y\to x_j}\frac{\hat p_{n+1}(x_j)\hat p_n(y)}{x_j-y}\\
&=-\hat p_{n+1}(x_j)\lim_{y\to x_j}
	\frac{\hat p_n(y)-\hat p_n(x_j)}{y-x_j}
	=-\hat p_{n+1}(x_j)\hat p_n'(x_j)
\end{align*}
and therefore
\[
\frac{1}{w_j}=-b_{n+2}\hat p_{n+1}(x_j)\hat p_n'(x_j)
	=\sum_{k=0}^n\hat p_k(x_j)^2=\sum_{k=1}^n(v_{kj})^2,
\]
as claimed.
\end{proof}

\section{The modified Chebyshev algorithm}

Let $q_0$, $q_1$, $q_2$, \dots be a sequence of monic orthogonal
polynomials generated by a positive measure $\nu$ on the real line.
Thus, $q_k\in\Poly_k$ is a monic polynomial satisfying
\[
\int q_k(x)f(x)\,d\nu(x)=0\quad\text{for all $f\in\Poly_{k-1}$.}
\]
In the usual way, we write the three-term recurrence relation as
\begin{equation}\label{eq: 3 term q}
q_k(x)=(x-\alpha_k)q_{k-1}(x)-\beta_k^2 q_{k-2}(x)
	\quad\text{for $k=1$, $2$, $3$, \dots,}
\end{equation}
with
\[
q_0(x)=1\quad\text{and}\quad q_{-1}(x)=0,
\]
adopting the usual convention that $\beta_1=\sqrt{\int d\nu}$.

Suppose that $p_0$, $p_1$, $p_2$, \dots is another sequence of monic
orthogonal polynomials generated by a positive measure $\mu$ and
having known coefficients $a_k$~and $b_k$ in the associated
three-term recurrence relation~\eqref{eq: 3 term pj}; thus
\[
\int p_k(x)f(x)\,d\mu(x)=0\quad\text{for all $f\in\Poly_{k-1}$.}
\]
Following Gautschi~\cite{Gautschi1982}, we consider the case when
$\alpha_k$~and $\beta_k$ are not known explicitly, but we are able to
find the modified moments
\[
\nu_l=\int\hat p_{l-1}(x)\,d\nu(x)\quad\text{for $1\le l\le 2n$,}
\]
where, as usual, $\hat p_0$, $\hat p_1$, $\hat p_2$, \dots are the
orthonormalised polynomials satisfying
\[
\int\hat p_j(x)\hat p_k(x)\,d\mu(x)=\delta_{jk}.
\]
The next theorem summarises the modified Chebyshev algorithm for
computing $\alpha_k$~and $\beta_k$ for $1\le k\le n$.

\begin{figure}
\caption{The dots correspond to the $\sigma_{lk}$ computed in the
case~$n=5$.  The parallelogram indicates the $\sigma_{lk}$ used in
the expressions for $\alpha_5$~and $\beta_5$.}
\begin{center}
\begin{tikzpicture}[scale=1.0]
\draw[->] (0, -1) -- (0, 6);
\node[above left] at (0, 6) {$k$};
\draw[->] (-1, 0) -- (11, 0);
\node[right] at (11, 0) {$l$};
\foreach \x in {2, 3, ..., 9}
    \draw[fill=blue] (\x, 0) circle(0.10cm);
\foreach \x in {1, 2, ..., 10}
    \draw[fill=blue] (\x, 1) circle(0.10cm);
\foreach \x in {2, 3, ..., 9}
    \draw[fill=red] (\x, 2) circle(0.10cm);
\foreach \x in {3, 4, ..., 8}
    \draw[fill=red] (\x, 3) circle(0.10cm);
\foreach \x in {4, 5, 6, 7}
    \draw[fill=red] (\x, 4) circle(0.10cm);
\foreach \x in {5, 6}
    \draw[fill=red] (\x, 5) circle(0.10cm);
\foreach \y in {1, 2, 3, 4, 5}
    {
    \draw[thin] (-0.075, \y) -- (0, \y);
    \node[left] at (0.0, \y) {$\y$};
    }
\foreach \x in {1, 2, 3, ..., 9, 10}
    \node[below] at (\x, -0.04) {$\x$};
\draw[-] (10, 5) -- (10, 7);
\draw[-] (9, 6) -- (11, 6);
\draw[fill=blue] (9, 6) circle(0.10cm);
\draw[fill=blue] (10, 6) circle(0.10cm);
\draw[fill=blue] (11, 6) circle(0.10cm);
\draw[fill=blue] (10, 5) circle(0.10cm);
\draw[fill=red]  (10, 7) circle(0.10cm);
\node at (9, 6.5) {stencil};
\draw[-] (3.0, 3.5) -- (5.0, 5.5) -- (7.0, 5.5)
      -- (5.0, 3.5) -- (3.0, 3.5);
\end{tikzpicture}
\end{center}
\end{figure}

\begin{theorem}
The coefficients $\alpha_1$, $\alpha_2$, \dots, $\alpha_n$~and
$\beta_1$, $\beta_2$, \dots, $\beta_n$ in the three-term
recurrence relation~\eqref{eq: 3 term q} can be computed by putting
\[
\begin{aligned}
\sigma_{l,0}&=0&&\text{for $2\le l\le 2n-1$},\\
\beta_1&=\sqrt{\nu_1},\\
\sigma_{l,1}&=\frac{\nu_l}{\beta_1}&
	&\text{for $1\le l\le2n$},\\
\alpha_1&=a_1+b_2\,\frac{\sigma_{21}}{\sigma_{11}},
\end{aligned}
\]
and, for $k=1$, $2$, \dots, $n-1$ and $k\le l\le2n-k-1$,
\[
\begin{aligned}
\beta_{k+1}&=b_{k+1}\sqrt{1
		+\frac{b_{k+2}}{b_{k+1}}\,\frac{\sigma_{k+2,k}}{\sigma_{kk}}
		+\frac{a_{k+1}-\alpha_k}{b_{k+1}}\,
			\frac{\sigma_{k+1,k}}{\sigma_{kk}}
		-\frac{\beta_k}{b_{k+1}}\,
			\frac{\sigma_{k+1,k-1}}{\sigma_{kk}}},\\
\sigma_{l+1,k+1}&=\frac{b_{l+2}}{\beta_{k+1}}\,\sigma_{l+2,k}
	+\frac{a_{l+1}-\alpha_k}{\beta_{k+1}}\sigma_{l+1,k}
	+\frac{b_{l+1}}{\beta_{k+1}}\,\sigma_{lk}
	-\frac{\beta_k}{\beta_{k+1}}\,\sigma_{l+1,k-1},\\
\alpha_{k+1}&=a_{k+1}+b_{k+2}\,
	\frac{\sigma_{k+2,k+1}}{\sigma_{k+1,k+1}}
	-\beta_{k+1}\,\frac{\sigma_{k+1,k}}{\sigma_{k+1,k+1}}.
\end{aligned}
\]
\end{theorem}
\begin{proof}
The integrals
\[
\sigma_{lk}=\int\hat p_{l-1}(x)\hat q_{k-1}(x)\,d\nu(x)
\]
satisfy the initial conditions $\sigma_{l,0}=0$~and
\[
\sigma_{l,1}=\frac{\nu_l}{\|q_0\|}
	=\frac{\nu_l}{\beta_1}
\quad\text{with $\beta_1=\sqrt{\nu_1}$.}
\]
In addition, $\sigma_{lk}=0$ if $k>l$.
For $1\le k\le l$,
\begin{align*}
\beta_{k+1}\sigma_{l+1,k+1}&=\int\hat p_l(x)\bigl[
(x-\alpha_k)\hat q_{k-1}(x)-\beta_k\hat q_{k-2}(x)\bigr]\,d\nu(x)\\
	&=\int(x-a_{l+1}+a_{l+1}-\alpha_k)
		\hat p_l(x)\hat q_{k-1}(x)\,d\nu(x)
		-\beta_k\sigma_{l+1,k-1}\\
	&=\int\bigl[(x-a_{l+1})\hat p_l(x)-b_{l+1}\hat p_{l-1}(x)\bigr]
		\hat q_{k-1}(x)\,d\nu(x)\\
	&\qquad{}+(a_{l+1}-\alpha_k)\sigma_{l+1,k}
		+b_{l+1}\sigma_{lk}-\beta_k\sigma_{l+1,k-1}
\end{align*}
so
\begin{equation}\label{eq: beta sigma}
\beta_{k+1}\sigma_{l+1,k+1}=b_{l+2}\sigma_{l+2,k}
	+(a_{l+1}-\alpha_k)\sigma_ { l+1 , k }
		+b_{l+1}\sigma_{lk}-\beta_k\sigma_{l+1,k-1}.
\end{equation}

By Theorem~\ref{thm: 3 term}, we have
$\beta_{k+1}=\|q_k\|/\|q_{k-1}\|$,
and since $q_k(x)-p_k(x)$ has degree at most~$k-1$, the orthogonality
property of the $q_k$ implies that
\[
\int q_k(x)^2\,d\nu(x)=\int p_k(x)q_k(x)\,d\nu(x)
	+\int\bigl[q_k(x)-p_k(x)\bigr]q_k(x)\,d\nu(x)
\]
or equivalently, $\|q_k\|^2=\|p_k\|\|q_k\|\sigma_{k+1,k+1}+0$ and so
\begin{equation}\label{eq: norm pk qk}
\|q_k\|=\|p_k\|\sigma_{k+1,k+1}.
\end{equation}
Thus, for $k\ge1$,
\[
\beta_{k+1}=\frac{\|q_k\|}{\|q_{k-1}\|}
	=\frac{\|p_k\|}{\|p_{k-1}\|}\,
		\frac{\sigma_{k+1,k+1}}{\sigma_{kk}}
	=b_{k+1}\,\frac{\sigma_{k+1,k+1}}{\sigma_{kk}}
\]
and hence, by~\eqref{eq: beta sigma},
\begin{align*}
\beta_{k+1}^2&=\frac{b_{k+1}}{\sigma_{kk}}\biggl(
	b_{k+2}\sigma_{k+2,k}+(a_{k+1}-\alpha_k)\sigma_{k+1,k}
	+b_{k+1}\sigma_{kk}-\beta_k\sigma_{k+1,k-1}\biggr)\\
	&=b_{k+1}^2+\frac{b_{k+1}}{\sigma_{kk}}\biggl(
	b_{k+2}\sigma_{k+2,k}+(a_{k+1}-\alpha_k)\sigma_{k+1,k}
	-\beta_k\sigma_{k+1,k-1}\biggr)\\
	&=b_{k+1}^2\biggl(1
		+\frac{b_{k+2}}{b_{k+1}}\,\frac{\sigma_{k+2,k}}{\sigma_{kk}}
		+\frac{a_{k+1}-\alpha_k}{b_{k+1}}\,
			\frac{\sigma_{k+1,k}}{\sigma_{kk}}
		-\frac{\beta_k}{b_{k+1}}\,
			\frac{\sigma_{k+1,k-1}}{\sigma_{kk}}\biggr).
\end{align*}

Theorem~\ref{thm: 3 term} also gives
$\alpha_{k+1}=\int x\hat q_k(x)^2\,d\nu(x)$ so
\[
\|q_k\|\alpha_{k+1}=\int x\bigl[q_k(x)-p_k(x)\bigr]
	\hat q_k(x)\,d\nu(x)+\int xp_k(x)\hat q_k(x)\,d\nu(x),
\]
with the first term on the right equal to
\begin{align*}
\int\bigl[&q_k(x)-p_k(x)\bigr](x-\alpha_{k+1})
	\hat q_k(x)\,d\nu(x)\\
	&\qquad\qquad\qquad{}
	+\alpha_{k+1}\int
		\bigl[q_k(x)-p_k(x)\bigr]\hat q_k(x)\,d\nu(x)\\
	&=\int\bigl[q_k(x)-p_k(x)\bigr]
	\bigl[\beta_{k+2}\hat q_{k+1}(x)
		+\beta_{k+1}\hat q_{k-1}(x)\bigr]
		\,d\nu(x)+\alpha_{k+1}\times0\\
	&=-\beta_{k+1}\int p_k(x)\hat q_{k-1}(x)\,d\nu(x)
	=-\beta_{k+1}\|p_k\|\sigma_{k+1,k}
\end{align*}
and second equal to
\begin{align*}
\int\bigl[
	&(x-a_{k+1})p_k(x)-b_{k+1}^2p_{k-1}(x)\bigr]\hat q_k(x)\,d\nu(x)\\
	&\qquad{}+\int\bigl[
		a_{k+1}p_k(x)+b_{k+1}^2p_{k-1}(x)\bigr]\hat q_k(x)\,d\nu(x)\\
	&=\int p_{k+1}(x)\hat q_k(x)\,d\nu(x)
		+a_{k+1}\int p_k(x)\hat q_k(x)\,d\nu(x)\\
	&=\|p_{k+1}\|\sigma_{k+2,k+1}+a_{k+1}\|p_k\|\sigma_{k+1,k+1}.
\end{align*}
Recalling \eqref{eq: norm pk qk}, we see that
\begin{multline*}
\|p_k\|\sigma_{k+1,k+1}\alpha_{k+1}
	=a_{k+1}\|p_k\|\sigma_{k+1,k+1}+\|p_{k+1}\|\sigma_{k+2,k+1}\\
		-\beta_{k+1}\|p_k\|\sigma_{k+1,k}
\end{multline*}
and therefore, since $\|p_{k+1}\|/\|p_k\|=b_{k+2}$,
\[
\alpha_{k+1}=a_{k+1}+b_{k+2}\,
	\frac{\sigma_{k+2,k+1}}{\sigma_{k+1,k+1}}
	-\beta_{k+1}\,\frac{\sigma_{k+1,k}}{\sigma_{k+1,k+1}}.
\]
\end{proof}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{A log weight}
We now consider the example
\[
d\nu(x)=x^\rho\log x^{-1}\,dx\quad\text{for $0<x<1$,}
\]
with $\rho>-1$.  As our known family of orthogonal polynomials we
use shifted Legendre polynomials.  Recalling that the standard
Legendre polynomials satisfy
\[
\int_{-1}^1P_j(y)P_k(y)\,dy=\frac{2}{2l+1},
\]
we see that the polynomials
\[
\hat p_l(x)=\sqrt{2l+1}\,P_l(2x-1)
\]
satisfy
\[
\int_0^1\hat p_j(x)\hat p_k(x)\,dx=\delta_{jk}.
\]
Thus
\[
\frac{\nu_l}{\sqrt{2l-1}}=\int_0^1 P_{l-1}(2x-1)x^\rho\log x^{-1}\,dx.
\]
Following \cite{Gautschi1979}, we make the substitution $y=2x-1$
and obtain
\begin{multline*}
\frac{\nu_l}{\sqrt{2l-1}}=\frac{\log 2}{2^{\rho+1}}\int_{-1}^1
	P_{l-1}(y)(y+1)^\rho\,dy\\
	-\frac{1}{2^{\rho+1}}\int_{-1}^1
		P_{l-1}(y)(y+1)^\rho\log(y+1)\,dy.
\end{multline*}
It is known that
\[
\int_{-1}^1 P_{l-1}(y)(y+1)^\rho\,dy
	=\frac{2^{\rho+1}\Gamma(\rho+1)^2}%
{\Gamma(\rho+1+l)\Gamma(\rho+2-l)},
\]
and by logarithmic differentiation of this identity with respect
to~$\rho$, we find that
\begin{multline*}
\int_{-1}^1 P_{l-1}(y)(y+1)^\rho\log(y+1)\,dy
	=\frac{2^{\rho+1}\Gamma(\rho+1)^2}%
{\Gamma(\rho+1+l)\Gamma(\rho+2-l)}\\
	\times\bigl[\log2+2\psi(\rho+1)-\psi(\rho+1+l)-\psi(\rho+2-l)
	\bigr],
\end{multline*}
where $\psi(x)=\Gamma'(x)/\Gamma(x)$ denotes the digamma function.
Thus,
\begin{multline}\label{eq: mu l}
\frac{\nu_l}{\sqrt{2l-1}}
	=\frac{\Gamma(\rho+1)^2}{\Gamma(\rho+1+l)\Gamma(\rho+2-l)} \\
	\times\bigl[\psi(\rho+1+l)+\psi(\rho+2-l)-2\psi(\rho+1)\bigr].
\end{multline}
Using the reflection formulae
\[
\Gamma(1-z)\Gamma(z)=\frac{\pi}{\sin\pi z}
\quad\text{and}\quad
\psi(1-z)=\psi(z)+\pi\cot\pi z,
\]
we find that
\[
\lim_{z\to m}\frac{1}{\Gamma(-z)}=0
\quad\text{and}\quad
\lim_{z\to m}\frac{\psi(-z)}{\Gamma(-z)}=(-1)^{m+1}m!
\]
for $m\in\{0,1,2,\dots\}$.  Thus, if $\rho=r\in\{0,1,2,\dots\}$
then
\[
\frac{\nu_l}{\sqrt{2l-1}}=(-1)^{l-1-r}\,\frac{(r!)^2(l-r-2)!}{(l+r)!}
	\quad\text{for $l\in\{r+2, r+3, r+4, \dots\}$.}
\]
Otherwise, we can use the functional identities
\[
\Gamma(z+1)=z\Gamma(z)\quad\text{and}\quad
\psi(z+1)=\psi(z)+\frac{1}{z}
\]
to simplify \eqref{eq: mu l}.  In fact,
\[
\Gamma(\rho+1)=\rho(\rho-1)\cdots(\rho+2-l)\Gamma(\rho+2-l)
\]
and
\[
\Gamma(\rho+1+l)=(\rho+l)(\rho+l-1)\cdots(\rho+2)(\rho+1)
	\Gamma(\rho+1),
\]
so
\begin{align*}
\frac{\Gamma(\rho+1)^2}{\Gamma(\rho+1+l)\Gamma(\rho+2-l)}
	&=\frac{\rho(\rho-1)\cdots(\rho+2-l)}%
{(\rho+l)(\rho+l-1)\cdots(\rho+2)(\rho+1)}\\
	&=\frac{1}{\rho+1}\prod_{j=1}^{l-1}\frac{\rho+1-j}{\rho+1+j}.
\end{align*}
Moreover,
\[
\psi(\rho+1+l)=\psi(\rho+1)+\frac{1}{\rho+1}+\frac{1}{\rho+2}
	+\cdots+\frac{1}{\rho+l}
\]
and
\[
\psi(\rho+2-l)=\psi(\rho+1)-\frac{1}{\rho}-\frac{1}{\rho-1}
	-\cdots-\frac{1}{\rho-(l-2)},
\]
so
\[
\psi(\rho+1+l)+\psi(\rho+2-l)-2\psi(\rho+1)
	=\frac{1}{\rho+1}+\sum_{j=1}^{l-1}\biggl(\frac{1}{\rho+1+j}
		-\frac{1}{\rho+1-j}\biggr).
\]
Hence,
\begin{equation}\label{eq: nul}
\frac{\nu_l}{\sqrt{2l-1}}=\frac{1}{\rho+1}\biggl[\frac{1}{\rho+1}
	+\sum_{j=1}^{l-1}\biggl(\frac{1}{\rho+1+j}
		-\frac{1}{\rho+1-j}\biggr)\biggr]
	\prod_{k=1}^{l-1}\frac{\rho+1-k}{\rho+1+k}.
\end{equation}

Choose a non-negative integer~$r$ such that $|\rho-r|$ is as small as
possible, and define
\[
B_l=\prod_{\substack{k=1\\ k\ne r+1}}^{l-1}\frac{\rho+1-k}{\rho+1+k}
\]
so that
\[
\prod_{k=1}^{l-1}\frac{\rho+1+k}{\rho+1-k}
	=\frac{\rho-r}{\rho+r+2}\,B_l\quad\text{for $l\ge r+2$.}
\]
We can rewrite \eqref{eq: nul} as
\[
\frac{\nu_l}{\sqrt{2l-1}}=\frac{B_l}{1+\rho}\biggl[
	\frac{1}{1+\rho}+\sum_{j=1}^{l-1}
	\biggl(\frac{1}{\rho+1+j}-\frac{1}{\rho+1-j}\biggr)
		\biggr]\quad\text{for $1\le l\le r+1$,}
\]
with
\begin{multline*}
\frac{\nu_l}{\sqrt{2l-1}}=\frac{B_l}{1+\rho}\,\frac{\rho-r}{\rho+r+2}
	\biggl[\frac{1}{1+\rho}
	+\sum_{\substack{j=1\\ j\ne r+1}}^{l-1}\biggl(
	\frac{1}{\rho+1+j}-\frac{1}{\rho+1-j}\biggr)\biggr]\\
+\frac{B_l}{1+\rho}\,\frac{1}{\rho+r+2}
	\biggl(\frac{\rho-r}{\rho+r+2}-1\biggr)
		\quad\text{for $l\ge r+2$.}
\end{multline*}

If $\rho=r$ then we can use the formula~\eqref{eq: nul} to compute
$\nu_l$ for $1\le l\le r+1$.  Next, for $l=r+2$,
\[
\frac{\nu_{r+2}}{\sqrt{2r+3}}
	=-\,\frac{(r!)^2}{(2r+2)!}=\frac{-1}{2(r+1) }
	\prod_{j=1}^r\frac{j}{2(2j+1)},
\]
and after that we have the recursion
\[
\frac{\nu_{l+1}}{\sqrt{2l+1}}=-\frac{l-r-1}{l+r+1}\,
	\frac{\nu_l}{\sqrt{2l-1}}
	\quad\text{for $l\ge r+2$,}
\]
or equivalently,
\[
\nu_{l+1}=-\nu_l\times\frac{l-r-1}{l+r+1}\,\sqrt{\frac{2l+1}{2l-1}}
	\quad\text{for $l\ge r+2$.}
\]

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{plain}
\bibliography{notes_refs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
\endinput


%  -----------------------------------------------------------------------------
%  Author         : Bimalka Piyaruwan Thalagala
%  GitHub         : https://github.com/bimalka98
%  Date Created   : 01.09.2020
%  Last Modified  : 18.02.2020
%  -----------------------------------------------------------------------------

\documentclass[a4paper,11pt]{article}%,twocolumn
\input{settings/packages}
\input{settings/page}
\input{settings/macros}

\begin{document}

\input{content/title_page}
\tableofcontents

\begin{center}
	\textbf{\textit{* PDF is clickable}}
\end{center}
\vspace{1cm}
\hrule

%%-----------------------------------------------------------------------

\section{Computer Vision: Detecting Corners/Features of an Image}

\textit{\large \textbf{Related Concepts in Linear Algebra:} Eigenvalues, Eigenvectors, Characteristic Polynomial, Quadratic Forms, Principal Axis Theorem, Orthogonal matrix, Diagonalization}

\subsection{Problem Identification}

In computer vision field finding corners/features in an image($I(x, y)$) is a key operation which is done prior to many advanced applications such as feature mapping, panorama stitching and 3D reconstruction. A corner can be identified as a point in an image whose neighborhood is locally unique to the given image. When comparing this neighborhood with any of the surrounding neighborhoods, significant difference can be observed between them.\\

This difference can be quantified mathematically using the following function evaluated over a region called a window($W$). Here the most basic Error function is used to make the explanation easy, and this may vary in different implementation of corner detection algorithms.

\begin{equation}
	E(u,v) = \sum_{(x,y) \in W}[I(x+u, y+v) – I(x, y)]^2
\end{equation}

This function can be further simplified using Taylor series and can be rearranged to get an expression in the \textbf{\textit{Quadratic form associated with $M$}} where the $M$ is called the \textit{\textbf{Second Moment Matrix}} which consists of summation of quadratic forms of gradients of the image over a window centered at the point of interest. Here gradient of the image along x axis is $I_{x}$ while $I_{y}$ is the gradient along y axis.

\begin{equation}
	 \begin{split}
		 	E(u,v) &= \sum_{(x,y) \in W}[I_{x}^{2}.u^{2}+2.I_{x}.I_{y}.u.v + I_{y}^{2}.v^{2}]\\
		 	&= \begin{bmatrix} u & v \end{bmatrix}
		 	 \begin{bmatrix}
		 	 	\sum_{(x,y) \in W}[I_x ^2 ]& \sum_{(x,y) \in W}[I_x I_y]  \\ \sum_{(x,y) \in W}[I_x I_y]& \sum_{(x,y) \in W}[I_y ^2 ]
	 	 	\end{bmatrix}
		 	\begin{bmatrix} u \\ v \end{bmatrix}\\
		 	&=\begin{bmatrix} u & v \end{bmatrix}M\begin{bmatrix} u \\ v \end{bmatrix}\\
		 	E(\underline{x})&= \underline{x}^\top M \underline{x}
	 \end{split}
  \label{euv}
 \end{equation}

The plot of the above function looks like follows for a given point where the $\sum_{(x,y) \in W}[I_x ^2 ]= 105983042.0$, $\sum_{(x,y) \in W}[I_x I_y]=13718014.0$ and $\sum_{(x,y) \in W}[I_y ^2 ]=105671174.0$\cite{cv}. (\textit{These values correspond to an actual corner in the Figure \ref{corners}}). If the point of interest is a corner it will give a very large Error ($E(u,v)$) value even for a very low ($u,v$) shift in any direction. This can be intuitively recognized as a steep surface(more narrower bowl shape) as depicted in Figure \ref{euvsurf}.

\begin{figure}[!h]
	\centering
	\subfigure[Surface Plot of $E(u,v)$]
	{ \includegraphics[scale=0.67]{figures/euvsurf}
\label{euvsurf}
}\hfill
	\subfigure[Contour Plot(Horizontal Slices) of $E(u,v)$]
	{ \includegraphics[scale=0.67]{figures/euvcontour}
\label{euvcontour}
}
	\caption{Error Function E(u,v) }

\end{figure}

\subsection{Solution through concepts of Linear Algebra}

This narrowness can be mathematically quantified by examining a horizontal slice of the surface plot in figure \ref{euvsurf}. Consider one particular contour in Figure \ref{euvcontour}, then along that contour, function value is a constant i.e $E(\underline{x}) = constant = \underline{x}^\top M \underline{x}$ and then the above Expression \ref{euv} represents equation of an ellipse. \textit{If the surface is narrow, the radii of the ellipse must be small for a given $E(u,v)$ value.} That is function value must increase rapidly as we moves outward from the point of interest. Lengths of the major and minor axes(radii) of the ellipse are given by $\lambda_{min}^{-\frac{1}{2}}$ and $\lambda_{max}^{-\frac{1}{2}}$ respectively, where the $\lambda_{min}$ and $\lambda_{max}$ are the \textbf{\textit{eigenvalues associated with the second moment matrix $M$}}. We can prove this as follows\cite{corner}.\\

Since $M$ is a symmetric matrix, an \textbf{\textit{Orthogonal($P^\top=P^{-1}$) matrix}} $P$ can be found such that $\underline{x} = P\underline{y}$  which transforms $E(\underline{x}) = \underline{x}^\top M \underline{x}$ into the form  $E(\underline{y}) = \underline{y}^\top D \underline{y}$ where $D$ is a diagonal matrix consists of the eigenvalues of the $M$. The columns of matrix $P$ consists of the \textbf{\textit{eigenvectors}} associated with the eigenvalues of the second moment matrix $M$.

\begin{equation}
	\begin{split}
E(\underline{x}) &= \underline{x}^\top M \underline{x}\\
&= (P\underline{y})^\top M (P\underline{y})\\
&= (\underline{y}^\top P^\top) M (P\underline{y})\\
&= \underline{y}^\top (P^\top M P)\underline{y}\\
&= \underline{y}^\top D \underline{y}\\
E({y_{1},y_{2}})&= \begin{bmatrix} y_{1} & y_{2} \end{bmatrix}
\begin{bmatrix}
	\lambda_{min} & 0  \\ 0 & \lambda_{max}
\end{bmatrix}
\begin{bmatrix} y_{1} \\ y_{2} \end{bmatrix}\\
	 \end{split}
 \label{orthdiag}
\end{equation}

Comparing result of Expression \ref{orthdiag} with the standard form of an ellipse,

\begin{equation}
	\begin{split}
		{\frac{x^2}{a^2} + \frac{y^2}{b^2}} &= {\begin{bmatrix} x & y \end{bmatrix}
		\begin{bmatrix} 1/a^2 & 0 \\ 0 & 1/b^2 \end{bmatrix}
		\begin{bmatrix} x \\ y \end{bmatrix}} \\
 &= {\begin{bmatrix} y_{1} & y_{2} \end{bmatrix}
		\begin{bmatrix}\lambda_{min} & 0  \\ 0 & \lambda_{max} \end{bmatrix}
		\begin{bmatrix} y_{1} \\ y_{2} \end{bmatrix}}\\
	\end{split}
\end{equation}

Therefore, $1/a^2 = \lambda_{min}$ and $1/b^2 = \lambda_{max}$. Which yields that
$a = \lambda_{min}^{-\frac{1}{2}}$ and $b = \lambda_{max}^{-\frac{1}{2}}$.\\

Therefore to find corners in an image all we have to do is find the \textbf{\textit{eigenvalues}} associated with each second moment matrix for each pixel$(x,y)$ in an image. According to the magnitudes of eigenvalues following classification can be done.

\begin{enumerate}[a.)]
	\item  \textbf{At a Corner:} $E(u,v)$ increases rapidly in any direction and therefore \textit{both eigenvalues are large}.
	\item 	\textbf{At an Edge:}  $E(u,v)$ increases rapidly only in one direction and therefore \textit{one eigenvalue is relatively larger than the other}.
	\item \textbf{At a flat area:} $E(u,v)$ does not increase rapidly and therefore \textit{both eigenvalues are small}.
\end{enumerate}

\begin{figure}[!h]
\centering
\includegraphics[scale= 0.3]{figures/corner}
\caption{Detecting corners in an image, Step by Step approach\cite{cv}}
\label{corners}
\end{figure}

Calculating eigenvalues is computationally expensive therefore what is known as Corner Response Function(R) is used when implementing algorithms. Here $\alpha$ $\in$ [0.04,0.06] and \textbf{At a Corner:} R $\gg$ 0, \textbf{At an Edge:} R $<$ 0, \textbf{At a flat area:} $|R|$ is small. Therefore corners can be extracted by applying an appropriate threshold value for R.

\begin{equation}
	\begin{split}
R &= det(M) - \alpha* trace(M)^2\\
 & = \lambda_{min}*\lambda_{max} - \alpha*(\lambda_{max} + \lambda_{min})^2
	\end{split}
\end{equation}

\pagebreak

\section{Robotic Systems: Coordinate frame Transformation}

\textit{\large \textbf{Related Concepts in Linear Algebra:} Linear Transformations, Matrix Transformations, Change of basis, Matrix Multiplication}

\subsection{Problem Identification}

When robot arms are used in industrial activities, robot engineer needs to make sure the \textbf{\textit{end-effector}}(\textit{gripper or any other tool attached at the end of the arm}) of the robot arm is at the exact location in the exact orientation at the operation. Otherwise the required behavior of the arm can not be obtained. Problem is usually robot takes the measurements using sensors or cameras placed somewhere else in the system and therefore these measurements(\textbf{\textit{position vectors}}) are in with respect to those sensor's or camera's coordinate frame while the robot arm operates with respect to some other(Robot base) coordinate frame. Flowing figure illustrates this situation.\\

\begin{figure}[!h]
	\centering
	\subfigure[Robotic System]
	{ \includegraphics[scale=0.67]{figures/env}
	}\hfill
	\subfigure[Coordinate Systems]
	{ \includegraphics[scale=0.67]{figures/difcord}
	}
	\caption{Use of Different Coordinate systems in an Industrial Robotic System\cite{pics}}
\end{figure}

Therefore to make use of the information gathered by the sensors/cameras first we need to \textbf{\textit{transform}} position vectors gathered by camera with respect to its coordinate frame, in to robot arm's base coordinate frame. Then we can use inverse kinematics to calculate the required rotation angles and translations that we should provide to the arm to get the required behavior.

\subsection{Solution through concepts of Linear Algebra}

In order to fully describe an object in the 3D world we need its \textit{\textbf{Rotation and Translation}} with respect to some reference(\textit{in this case the robot arm}) coordinate frame(say $\alpha$). For that the object of interest(\textit{in this case the sensor/camera}) must be given its own coordinate frame(say $\beta$) too. The mathematical way to represent differences between the reference frame and the object's frame is called the \textbf{\textit{Homogeneous Transformation Matrix}}\cite{robotic} in Robotics and it is denoted as,

\begin{center}
	$ H^{\alpha}_{\beta} $ : read as transformation of frame $\beta$ with respect to the frame $\alpha$
\end{center}

This 	$ H^{\alpha}_{\beta} $ can be mathematically represented as a $4 \times 4$ matrix which consists of $3 \times 3$ Rotation matrix and a $3 \times 1$ Translation matrix with last row as $[0~0~0~1]$ to make it  $4 \times 4$ square matrix to make the computation easy.

\begin{equation}
	\begin{split}
		 H^{\alpha}_{\beta} &=
			\begin{bmatrix}
				 r_{11} & r_{12} & r_{13} & d_1 \\
				 r_{21} & r_{22} & r_{23} & d_2 \\
				 r_{31} & r_{32} & r_{33} & d_3 \\
				 0 & 0 & 0 &1 \\
			\end{bmatrix}
	\end{split}
\end{equation}

There are  \textbf{\textit{6 Principal Movements}} as, rotations about 3 axes(X,Y,Z) and translations along 3 axes(X,Y,Z). Correct combination(\textit{obtain by multiplying required principal matrices like $Rot(e_i,\theta_m)*Trans(e_j,d_k)$}) of these 6 principal movements  can be used to get any of the advanced transformations, from the reference frame to object's frame. Those basic matrices are as follows where $e_i$ refers to the elements in standard \textbf{\textit{basis}}( $e_1 = [1 ~0~ 0]^\top$, $e_2 = [0 ~1~ 0]^\top$ and $e_3 = [0 ~0~ 1]^\top$) while $\theta_i$ indicates how many degrees the object has rotated about the $e_i$ basis vector. Additionally $d_i$ indicates the displacement along the direction of $e_i$ basis vector(axis)\cite{tfs1}. \\

\begin{center}
	\begin{tabular}{| c || c|}
		\hline
 \textbf{Principal Rotations about axes(X,Y,Z)\cite{tfs2}} &	\textbf{Principal Translations along axes(X,Y,Z)\cite{tfs2}}\\\hline
 		&\\
		$Rot(e_1,\theta_1) =
\begin{bmatrix}
	1 & 0 & 0 & 0 \\
	0 & \cos(\theta_1) & -\sin(\theta_1) & 0 \\
	0 & \sin(\theta_1) & \cos(\theta_1) & 0 \\
	0 & 0 & 0 &1 \\
\end{bmatrix}$ &
		$Trans(e_1,d_1)=
\begin{bmatrix}
	1 & 0 &0  & d_1 \\
	0 & 1 & 0 & 0 \\
	0 & 0 & 1 & 0 \\
	0 & 0 &0  & 1 \\
\end{bmatrix}$
\\[1cm]
%--------------------------------------------------------------------------
$Rot(e_2,\theta_1) =
\begin{bmatrix}
\cos(\theta_1)	& 0 & \sin(\theta_1)  & 0 \\
0	& 1 & 0 & 0 \\
-\sin(\theta_1)	& 0 & \cos(\theta_1) & 0 \\
0 & 0 & 0 &1 \\
\end{bmatrix}$ &
		$Trans(e_2,d_2) =
\begin{bmatrix}
	1 & 0 &0  & 0 \\
	0 & 1 & 0 & d_2 \\
	0 & 0 & 1 & 0 \\
	0 & 0 & 0 & 1 \\
\end{bmatrix}$
\\[1cm]
%--------------------------------------------------------------------------
$Rot(e_3,\theta_1) =
\begin{bmatrix}
	\cos(\theta_1) & -\sin(\theta_1)&0  & 0 \\
	\sin(\theta_1) & \cos(\theta_1)   &0  & 0 \\
	0&0  &1  & 0 \\
	0 & 0 & 0 &1 \\
\end{bmatrix}$ &
		$Trans(e_3,d_3) =
\begin{bmatrix}
	1 & 0 &0  & 0 \\
	0 & 1 & 0 & 0 \\
	0 & 0 & 1 & d_3 \\
	0 & 0 & 0 & 1 \\
\end{bmatrix}$\\
		&\\\hline\hline
\end{tabular}
\end{center}

Once we construct the \textbf{\textit{Homogeneous Transformation Matrix which transforms robot base frame  into camera frame}}, we just have to multiply the position vectors obtained w.r.t camera frame by this transformation matrix to get the corresponding position vectors in the robot base frame. Let  $^1u = [u_1 ~u_2~ u_3~1]^\top$ be a  position vector(homogeneous) of an object in the camera frame and  $H^{0}_{1}$ be the homogeneous transformation matrix, then the position vector of the object in the robot base frame is\cite{tfs3}, say $^0u$

\begin{equation}
	\begin{split}
		^0u &= H^{0}_{1}* ^1u\\
		^0u &=
		{\begin{bmatrix}
			r_{11} & r_{12} & r_{13} & d_1 \\
			r_{21} & r_{22} & r_{23} & d_2 \\
			r_{31} & r_{32} & r_{33} & d_3 \\
			0 & 0 & 0 &1 \\
		\end{bmatrix}} * \begin{bmatrix} u_1 \\u_2\\ u_3\\1 \end{bmatrix}
	\end{split}
\end{equation}

Moreover, since the Homogeneous Transformation Matrix is invertible, following expression is also valid which makes it possible to move between coordinate systems easily.
\begin{equation}
{H^{0}_{1}}^{-1} = H^{1}_{0}
\end{equation}

\vfill
\vspace{1cm}
\hrule

\scriptsize
\begin{flushleft}
	\bibliographystyle{plain}
\bibliography{refer}
\end{flushleft}
\end{document}


\section{Eigenvectors, Eigenvalues and Diagonal Matrices}
This is the first step into the wonderful land of diagonalisation of endomorphisms.
Consider a vector space $V$ over $F$ with $\dim V=n<\infty$ and let $\alpha:V\to V$ be an endomorphism.
The general problem is whether we can find a basis $B$ of $V$ such that $[\alpha]_B$ is in a nice enough form.
In other words, by our change-of-basis formula, we want to know when can a matrix be conjugate to another matrix in a nice form.
\begin{definition}
    1. $\alpha\in L(V)=L(V,V)$ is diagonalisable if there exists a basis $B$ of $V$ such that $[\alpha]_B$ is diagonal, i.e. $([\alpha]_B)_{ij}=0$ for $i\neq j$.\\
    2. $\alpha\in L(V)$ is triangulable if there exists a basis $B$ of $V$ such that $[\alpha]_B$ is (upper) triangular.
\end{definition}
\begin{remark}
    A matrix is diagonalisable (resp. triangulable) iff it is conjugate to a diagonal (resp. triangular) matrix.
\end{remark}
\begin{definition}
    1. $\lambda\in F$ is an eigenvalue of $\alpha$ if $\alpha(v)=\lambda v$ for some $v\neq 0$.\\
    2. $v\in V$ is an eigenvector of $\alpha$ if $v\neq 0$ and there exists some $\lambda\in F$ such that $\alpha(v)=\lambda v$.\\
    3. $V_\lambda=\{v\in V:\alpha(v)=\lambda v\}\le V$ is called the eigenspace of $\alpha$ associated to $\lambda$.
\end{definition}
\begin{lemma}
    If $\alpha\in L(V)$ and $\lambda\in F$, then $\lambda$ is an eigenvalue iff $\det(\alpha-\lambda\operatorname{id}_V)=0$.
\end{lemma}
\begin{proof}
    Follows from the fact that matrices with nonzero determinant have zero kernel.
\end{proof}
\begin{remark}
    If $\alpha(v_j)=\lambda v_j$ for $v_j\neq 0$, then completing it into a basis $B=\{v_1,\ldots,v_j,\ldots,v_n\}$ of $V$ gives $([\alpha]_B)_{ij}=\lambda\delta_{ij}$.
\end{remark}
Recall that for a field $F$, a polynomail in $F$ is $f(t)=a_nt^n+\cdots+a_0\in F[t]$ with $a_i\in F$.
Let $\deg f$ be the largest $m$ such that $a_m\neq 0$, then we know that $\deg(f+g)\le\max\{\deg f,\deg g\}$ and $\deg(fg)=\deg(f)+\deg(g)$.
We say $\lambda$ is a root of $f$ iff $f(\lambda)=0$, and $g(t)$ divides $f(t)$ if there is some $q(t)\in F[t]$ such that $f(t)=g(t)q(t)$.
\begin{lemma}
    If $\lambda$ is a root of $f$, then $x-\lambda$ divides $f$.
\end{lemma}
\begin{proof}
    Write $f(t)=f(t)-f(\lambda)$ and factrorise.
\end{proof}
\begin{remark}
    We say $\lambda$ is a root of multiplicity $k$ if $(t-\lambda)^k$ divides $f$ but $(t-\lambda)^{k+1}$ does not.
\end{remark}
\begin{example}
    $f(t)=(t-1)^2(t-2)^3$ has roots $1$ with multiplicity $2$ and $2$ with multiplicity $3$.
\end{example}
\begin{corollary}
    A polynomial of degree $n$ has at most $n$ roots, counted with multiplicity.
\end{corollary}
\begin{proof}
    Induction.
\end{proof}
\begin{corollary}
    For polynomials $f_1,f_2$ of degree less than $n$ with $f_1(t_i)=f_2(t_i)$ for distinct $t_1,\ldots,t_n$, then $f_1=f_2$.
\end{corollary}
\begin{proof}
    $\deg(f_1-f_2)<n$.
\end{proof}
\begin{theorem}[Fundamental Theorem of Algebra]
    Any polynomial $f\in \mathbb C[t]$ of positive degree has a root.
\end{theorem}
\begin{proof}
    Omitted.
\end{proof}
Consequently, $f$ has exactly $\deg f$ many roots counted with multplicity.
This means that any $f\in\mathbb C[t]$ can be written as
$$f(t)=c\prod_{i=1}^n(t-\lambda_i)^{\alpha_i},c,\lambda_i\in\mathbb C,\alpha_i\in\mathbb N, \sum_{i=1}^n\alpha_i=\deg f$$
\begin{definition}
    For $\alpha\in L(V)$, the characteristic polynomial of $\alpha$ is $\chi_\alpha(t)=\det(\alpha-t\operatorname{id}_V)\in F[t]$.
\end{definition}
\begin{remark}
    Conjugate matrices then have the same characteristic polynomial.
\end{remark}
\begin{theorem}
    $\alpha\in L(V)$ is triangulable iff
    $$\chi_\alpha(t)=c\prod_{i=1}^n(t-\lambda_i)$$
    for some $c,\lambda_i\in F$.
\end{theorem}
Consequently, any matrix in $\mathbb C$ is triangulable.
\begin{proof}
    The ``only if'' part is trivial.
    For the ``if'' direction, we do induction on $n=\dim V$.
    The $n=1$ case is trivial.
    For $n>1$, there is $\lambda$ such that $\chi_\alpha(\lambda)=0$ by assumption.
    Let $\{v_1,\ldots,v_k\}$ be a basis of $U=V_\lambda$ and extend it to a basis $B=\{v_1,\ldots,v_n\}$ of $V$.
    We then have
    $$[\alpha]_B=\left( \begin{array}{c|c}
        I_k&\ast\\ \hline
        0&C
    \end{array} \right)$$
    So the induced endomorphism $\bar\alpha:V/U\to V/U$ has matrix $C$ under the basis $\{v_{k+1}+U,\ldots,v_n+U\}$.
    Then by the induction hypothesis, we can choose another set of basis $\{\tilde{v}_{k+1}+U,\ldots,\tilde{v}_n+U\}$ so that $C$ is triangular.
    Hence $\alpha$ is triangular under the basis $\{v_1,\ldots,v_k,\tilde{v}_{k+1},\ldots,\tilde{v}_n\}$.
    This completes the proof.
\end{proof}
\begin{lemma}
    Suppose $V$ is a vector space over $F=\mathbb R$ or $\mathbb C$ such that $\dim V=n<\infty$ and suppose $\alpha\in L(V)$ is an endomorphism with matrix $A$.
    Say $\chi_\alpha(t)=(-1)^nt^n+c_{n-1}t^{n-1}+\cdots+c_0$, then $c_0=\det A$, $c_{n-1}=(-1)^{n-1}\operatorname{tr}A$.
\end{lemma}
\begin{proof}
    $\det A=\chi_A(0)=c_0$.
    For $c_{n-1}$, note that the statement is true for triangular $A$, so we are done by the preceding theorem.
\end{proof}


\section{Countability}
We want to find a way to describe the sizes of infinite sets.
For example, $\mathbb N$ ``looks smaller'' than $\mathbb Z,\mathbb Q, \mathbb R$, but is that the case?
\begin{definition}
    A set $A$ is called countable if either $A$ is finite or there exists a bijection $A\to\mathbb N$.
\end{definition}
Equivalently, $A$ is countable iff we can list the element of $A$ as $\{a_1,a_2,a_3,\ldots\}$ (which might terminate if $A$ is finite).
\begin{example}
    1. Every finite set is countable.\\
    2. $\mathbb N$ is countable.\\
    3. $\mathbb Z$ is countable.
    Consider the listing $\{0,1,-1,2,-2,3,-3,\ldots\}$.
    Or, written in formula
    $$a_n=
    \begin{cases}
        n/2\text{, if $n$ is even}\\
        (1-n)/2\text{, if $n$ is odd}
    \end{cases}$$
\end{example}
A natural question to ask, following the last example, is whether all sets are countable.
For example, is $\mathbb Q$ countable?
How about $\mathbb R$?
\begin{proposition}
    Let $A$ be a set, then $A$ is countable iff there is an injection $f:A\to\mathbb N$
\end{proposition}
\begin{proof}
    If $A$ is countable, it is obvious that there is an injection.\\
    Conversely, it is done if $A$ is finite, so we can henceforth assume that $A$ is infinite.
    Note that $f(A)=\operatorname{Im}f$ is a subset of $\mathbb N$ and $f$ is a bijection $A\to f(A)$.
    So it is enough to show that $f(A)$ is countable.
    We can order $f(A)$ like we did in $\mathbb N$, then define $b_n$ recursively by $b_1=\min f(A)$ and $b_{n+1}=\min (f(A)\setminus\{b_i:1\le i\le n\})$.
    So the sequence $\{b_n\}_{n\in\mathbb N}$ lists $f(A)$, hence $f(A)$ is countable, so $A$ is countable.
\end{proof}
\begin{corollary}
    Every subset of a countable is countable.
\end{corollary}
\begin{remark}
    Consider the set $\{n/(n+1):n\in\mathbb N\}\cup\{1\}\subset \mathbb R$, then this set is countable, but we cannot hit everything by writing every element in increasing order.
\end{remark}
\begin{theorem}
    $\mathbb N\times\mathbb N$ is countable.
\end{theorem}
\begin{proof}
    Define the listing $a_1,a_2,\ldots\in\mathbb N\times\mathbb N$ by $a_1=(1,1)$ and if we have $a_n=(p,q)$, then
    $$a_{n+1}=\begin{cases}
        (p-1,q+1)\text{, if $p>1$}\\
        (p+q,1)\text{, otherwise}
    \end{cases}$$
    This lists all points in $\mathbb N\times\mathbb N$.
    All $(x,y)\in\mathbb N\times\mathbb N$ are hit by induction on $x+y$.
\end{proof}
\begin{proof}[Alternative proof]
    Consider the map $f:\mathbb N\times\mathbb N\to\mathbb N$ by $(a,b)\mapsto 2^a3^b$ is an injection.
\end{proof}
More generally, the same shows
\begin{corollary}\label{union_countable}
    Let $\{A_i\}_{i\in\mathbb N}$ be a collection of countable sets, then $\bigcup_{i\in \mathbb N}A_i$ is countable.
\end{corollary}
\begin{proof}
    Each $A_i$ is countable, so we can list $A_i$ as $\{a_{i1},a_{i2},\ldots\}$ (which might terminate if $A_i$ is finite).
    Now consider the function $f:\bigcup_{i\in \mathbb N}A_i\to\mathbb N$ by $x\mapsto 2^i3^j$ where $x=a_{ij}$ such that $i$ is the least such that $x\in A_i$ and $j$ is the least such that $x=a_{ij}$.
    This is an injection.
\end{proof}
\begin{example}
    1. $\mathbb Q$ is countable.
    Indeed,
    $$\mathbb Q=\bigcup_{n\in\mathbb N}\frac{1}{n}\mathbb Z$$
    So we are done by the preceding corollary.
    Alternatively, there is an obvious injection from $\mathbb Q$ to $\mathbb Z\times\mathbb Z$.\\
    2. The set $\mathbb A$ of all algebraic numbers is countable.
    Indeed, each polynomial has only finitely many roots, so it is enough to show that there are couontably many integer polynomials, then the claim is proved by Corollary \ref{union_countable}.
    But again by this corollary it is enough to show that there are only countably many integer polynomials of degree $d$ for any $d\in\mathbb N$.
    However this set injects to $\mathbb Z^{d+1}$ by $a_0+a_1x+\cdots+a_dx^d\mapsto (a_0,a_1,\ldots,a_d)$, so it is countable.
\end{example}
We have got many many countable sets in our stock, so the question is then whether all sets are countable.
\begin{theorem}
    $\mathbb R$ is uncountable (that is, not countable).
\end{theorem}
\begin{proof}
    It suffices to show that $(0,1)$ is uncountable.
    So given any sequence $r_1,r_2,\ldots$ of $(0,1)$, we shall show that there is some $s\in (0,1)$ that is not of the form $r_i$.
    We write each number in $\{r_i:i\in\mathbb N\}$ as decimals.
    That is, $r_i=0.r_{i1}r_{i2}\ldots$.
    For each $n\in\mathbb N$, choose a digit $s_n\in\{5,6\}\setminus\{r_{nn}\}\neq\varnothing$,
    \footnote{There is nothing special about $5,6$, we just need to get rid of the case of $9999\ldots$}
    then $0.s_1s_2\ldots\in (0,1)$ is not of the form $a_i$.
\end{proof}
The above proof is called Cantor's Diagonal Argument.
\begin{remark}
    $\mathbb A\cap \mathbb R$ is countable but $\mathbb R$ is not, so there exists transcendental numbers.
    In fact, ``most'' reals are transcendental, as the set $\mathbb R\setminus\mathbb A$ is uncountable.
\end{remark}
\begin{theorem}
    $2^{\mathbb N}$ is uncountable.
\end{theorem}
\begin{proof}
    Suppose $2^{\mathbb N}=\{s_1,s_2,\ldots\}$.
    Consider the set $S\subset\mathbb N$ by $n\in S\iff n\notin s_n$, that is $\{n\in\mathbb N:n\notin s_n\}$, so $S\neq s_i$ for any $i$, contradiction.
\end{proof}
\begin{remark}
    This really is exactly the proof idea of $\mathbb R$ being uncountable.
    Alternatively, we can actually inject $(0,1)$ to $2^\mathbb N$ by writing in base $2$.
\end{remark}
The proof above can be actually extended to the following:
\begin{theorem}
    Let $X$ be a set, then there is no bijection $X\to 2^X$.
\end{theorem}
For example, there is no bijection between $\mathbb R$ and $2^{\mathbb R}$.
\begin{proof}
    Let $f:X\to 2^X$ be a function.
    Consider the set $S=\{x\in X:x\notin f(x)\}$.
    But $S\notin f(X)$ since for any $x\in X,S\neq f(x)$ (as $x\in S\iff x\notin f(x)$), so $f$ is not surjective.
\end{proof}
Note that the set $S=\{x\in X:x\notin f(x)\}$ is very similar to Russel's Paradox.\\
Now we want to explore more countability arguments.
\begin{example}
    Any collection $\{A_i\}_{i\in I}$ of pairwise disjoint intervals is countable.
    The first proof that we can do is to inject the set into the rational numbers by choosing one from each interval.
    The second proof is by observing that the numbers of intervals in the family with length at least $1/n$ is countable, then we can write our family as a countable union of countable sets, hence is countable.
\end{example}
There are generally two sorts of arguments to show a set is uncountable, either copying the diagonal argument or inject an uncountable set to it.
To show it is countable, either we can list it (which however often get fiddly) or we can inject it into a countable set.
Another way is to use the fact that a countable union of countable sets is countable.\\
Now consider $A,B$ nonempty.
Intuitively, we think of $A$ bijecting with $B$ as saying that the size of $A$ is the size of $B$.
Similarly, we think of $A$ injecting into $B$ as $A$ has at most as large as $B$, and $A$ surjecting to $B$ as $A$ has at least as large as $B$.
For these to make sense, we certainly want that $A$ injects into $B$ if and only if $B$ surjects into $A$.
Indeed, given $f:A\to B$ injective, then consider $a\in A$, then $g:B\to A$ by
$$b\mapsto\begin{cases}
    f^{-1}(b)\text{, if $b\in f(A)$}\\
    a\text{, otherwise}
\end{cases}$$
Then $g$ is surjective.
Conversely, given $g:B\to A$ surjective, consider $f:A\to B$ with $a\mapsto a'$ by choosing an $a'\in g^{-1}(\{a\})$.
\footnote{Does it depend on the Axiom of Choice?}
Also it is intuitive that if $A$ injects into $B$ and $B$ injects into $A$, then $A$ bijects with $B$.
\begin{theorem}[Schr\"oder–Bernstein]
    Let $f:A\to B$ and $g:B\to A$ be injective, then there is a bijection $h:A\to B$
\end{theorem}
\begin{proof}
    For $a\in A$, we write $g^{-1}(a)$ for the unique (if exists) point in $B$ such that $g(g^{-1}(a))=a$.
    Define similarly $f^{-1}(b)$ for $b\in B$.
    The ``ancestors'' of $a\in A$ consists of
    $$g^{-1}(a), f^{-1}(g^{-1}(a)),g^{-1}(f^{-1}(g^{-1}(a))),f^{-1}(g^{-1}(f^{-1}(g^{-1}(a)))),\ldots$$
    which may or may not terminate.
    Similarly for $b\in B$.
    Let $A_0$ be the set of $a\in A$ such that the ancestor sequence of $a$ that terminates in even time, that is it has even length (so it includes those $a\notin g(B)$ since $0$ is even).
    And $A_1$ be the set of $a\in A$ such that the ancestor sequence of $a$ that terminates in odd time, $A_\infty$ be those whose ancestor sequence does not terminate.
    Similarly construct $B_0,B_1,B_\infty$.
    So $A_0,A_1,A_\infty$ partitions $A$ and $B_0,B_1,B_\infty$ partitions $B$.
    By definition, $f|_{A_0}$ is a bijection $A_0\to B_1$ (as every $b\in B_1$ is in the image of $f|_{A_0}$).
    And simiarly, $g|_{B_0}$ is a bijection $B_0\to A_1$.
    As for infinity cases, $f|_{A_\infty}$ is a bijection $A_{\infty}\to B_\infty$, so the function $h$ can be defined by
    $$h(a)=
    \begin{cases}
        f(a)\text{, if $a\notin A_1$}\\
        g^{-1}(a)\text{, if $a\in A_1$}
    \end{cases}$$
    Then $h:A\to B$ is well-defined and bijective.
\end{proof}
\begin{example}
    $[0,1]$ and $[0,1]\cup [2,3]$ biject.
    Indeed, $x\mapsto x$ is an injection from $[0,1]\to [0,1]\cup [2,3]$.
    Also $x\mapsto x/3$ injects $[0,1]\cup[2,3]$ to $[0,1]$
\end{example}
Now, is it true that for any sets $A,B$, either $A$ injects into $B$ or vice versa?
The answer is yes, but very hard and way beyond the scope of this course.\\
Now given $\mathbb N$, we can construct a strictly increasing sequence (in terms of sizes) $\mathbb N,2^{\mathbb N}, 2^{2^{\mathbb N}},\ldots$, but does every set injects into one of them?
The answer is obviously no, since we can take $X=\mathbb N\cup 2^{\mathbb N}\cup 2^{2^{\mathbb N}}\cup\cdots$.
Now this is definitely not the biggest set either, since we can now take again $X,2^X, 2^{2^X},\ldots$, which is again beaten by the union $X'$ of all of them.
We can then construct a sequence $X,X',X'',\ldots$, which is again beaten by $X\cup X'\cup X''\cup\cdots$.
And we can do the same thing again and again and again and this never ends... (but the course does here).