\section{Spans, Linear Independence and Steinitz Exchange Lemma}
In this section, we shall characterise the properties of the dimension and basis of a vector space.
\begin{definition}[Span of a Family of Vectors]
    Let $V$ be a vector space over $F$ and $S\subset V$.
    We define the span of $S$ to be
    $$\langle S\rangle=\operatorname{span}(S)=\left\{\sum_{i=1}^n\lambda_is_i:n\in\mathbb N,\lambda_i\in F,s_i\in S\right\}$$
\end{definition}
That is, $\langle S\rangle$ consists of all possible (finite) linear combination of elements of $S$.
By convention, we say $\langle \varnothing\rangle=\{0\}$.
Note also that the span of $S$ is essentially the minimal subspace of $V$ containing $S$.
\begin{example}
    1. Take $V=\mathbb R^3$ and
    $$S=\left\{\begin{pmatrix}
        1\\
        0\\
        0
    \end{pmatrix},\begin{pmatrix}
        0\\
        1\\
        2\\
    \end{pmatrix},\begin{pmatrix}
        3\\
        -2\\
        -4
    \end{pmatrix}\right\}$$
    then
    $$\langle S\rangle=\left\{\begin{pmatrix}
        a\\
        b\\
        2b
    \end{pmatrix}:a,b\in\mathbb R\right\}$$
    2. Take $V=\mathbb R^n$ and let $e_i$ be the vector in $V$ that only has $1$ at the $i^{th}$ entry and zero elsewhere, then $\langle \{e_i\}_{i=1}^n\rangle=V$.\\
    3. Let $V=\mathbb R^X$ and $S_x:X\to\mathbb R$ be such that $S_x(y)=1_{x=y}$.
    Then $\langle \{S_x\}_{x\in\mathbb R}\rangle$ are the set of functions $f\in\mathbb R^X$ that has finite support.
\end{example}
\begin{definition}
    Let $V$ be a vector space over $F$ and $S\subset V$.
    We say $S$ spans $V$ if $\langle S\rangle =V$.
\end{definition}
\begin{example}
    Take $V=\mathbb R^2$, then any set of two non-parallel vectors would span $V$.
\end{example}
\begin{definition}
    A vector space $V$ over a field $F$ is finite dimensional if there is a finite $S\subset V$ that spans $V$.
\end{definition}
\begin{example}
    The space $V=\mathbb P[x]$ be the set of polynomials in $\mathbb R$ and $V_n=\mathbb P_n[x]$ be the set of real polynomials with degree at most $n$.
    Then $V_n=\langle\{1,x,\ldots,x^n\}\rangle$ is finite dimensional, but $V$ is not finite dimensional as any finite set of polynomials must be contained in $V_n$ where $n$ is the maximal degree of polynomials in that set.
\end{example}
As $\mathbb N$ is well-ordered, there must be a minimum number of vectors that can possibly span $V$.
We then focus on how to capture this minimality.
\begin{definition}[(Linear) Independence]
    Let $V$ be a vector space over $F$.
    We say $\{v_1,\ldots,v_n\}\subset V$ are (linearly) independent (or is a free family) if for any $\lambda_1,\ldots,\lambda_n\in F$
    $$\sum_{i=1}^n\lambda_iv_i=0\implies\forall i,\lambda_i=0$$
    On the other hand, this set is not linearly independent if there exists $\lambda_1,\ldots,\lambda_n\in F$ not all zero such that $\sum_{i=1}^n\lambda_iv_i=0$.
\end{definition}
\begin{example}
    Let $V=\mathbb R^3$ and
    $$v_1=(1,0,0)^\top,v_2=(0,1,0)^\top,v_3=(1,1,0)^\top,v_4=(0,1,1)^\top$$
    Then $\{v_1,v_2\}$ is linearly independent.
    Note that $v_3\in\langle\{v_1,v_2\}\rangle$, so $\{v_1,v_2,v_3\}$ is not linearly independent.
    On the other hand, $v_4\notin\langle\{v_1,v_2\}\rangle$, which as one can verify means that $\{v_1,v_2,v_4\}$ is linearly independent.
\end{example}
\begin{remark}
    If the family $\{v_i\}_{1\le i\le n}$ is linearly independent, then none of $v_i$ is zero.
\end{remark}
\begin{definition}[Basis]
    A subset $S\subset V$ is a basis if it is linearly independent and $\langle S\rangle=V$.
\end{definition}
\begin{remark}
    When $S$ spans $V$, we say that $S$ is a generating family of $V$.
    So a basis is just a linearly independent generating family.
\end{remark}
\begin{example}
    1. Take $V=\mathbb R^n$, then the family $\{e_i\}_{1\le i\le n}$ where $e_i$ is the vector having $1$ at $i^{th}$ entry and zero otherwise is a basis.\\
    2. Take $V=\mathbb C$ over $\mathbb C$, then $\{a\}$ is a basis for any $a\neq 0$.\\
    3. Take also $V=\mathbb C$ but over $\mathbb R$, then $\{1,i\}$ is a basis.\\
    4. Take $V=\mathbb P[x]$ be the set of polynomials in $\mathbb R$ and $S=\{x^n:n\ge 0\}$.
    Then $S$ is a basis.
    Worth noting that $|S|=\infty$ in this case.
\end{example}
\begin{lemma}
    If $V$ is a vector space over $F$, then $\{v_1,\ldots,v_n\}$ is a basis of $V$ if and only if for any vector $v\in V$, there is a unique decomposition
    $$v=\sum_{i=1}^n\lambda_iv_i$$
\end{lemma}
\begin{remark}
    If the conditions are true, then the tuple $(\lambda_1,\ldots,\lambda_n)$ (ordered via the ordering one chose on $v_i$) is called the coordinate of $v$ in the basis $(v_i)$.
\end{remark}
\begin{proof}
    Trivial.
\end{proof}
\begin{lemma}
    If $S$ is a finite set that spans $V$, then a subset of $S$ is a basis of $V$.
\end{lemma}
\begin{proof}
    If $S$ is independent, then we are done.
    Otherwise, there is some $\lambda\neq 0$ and $\lambda_w$ such that there is $v\in S$ with
    $$\lambda v+\sum_{w\in S\setminus\{v\}}\lambda_ww=0\implies v=\frac{1}{\lambda}\sum_{w\in S\setminus\{v\}}\lambda_ww\in\langle S\setminus\{v\}\rangle$$
    Therefore $S\setminus\{v\}$ also spans $V$.
    We can repeat this process and, by the well-ordering of $\mathbb N$, will reach a basis.
\end{proof}
\begin{theorem}[Steinitz Exchange Lemma]\label{steinitz}
    Let $V$ be a finite dimensional vector space over $F$, $\{v_1,\ldots,v_m\}\subset V$ linearly independent, $\{w_1,\ldots,w_n\}\subset V$ a generating set, then:\\
    1. $m\le n$.\\
    2. Up to relabeling, $\{v_1,\ldots,v_m,w_{m+1},\ldots,w_n\}$ spans $V$.
\end{theorem}
\begin{proof}
    Suppose $\{v_1,\ldots,v_l,w_{l+1},\ldots,w_n\}$ spans $V$ for some $l<m$, then
    $$\exists\alpha_i,\beta_i\in F, v_{l+1}=\sum_{i\le l}\alpha_iv_i+\sum_{i>l}\beta_iw_i$$
    But $\{v_i\}$ is linearly independent, so one of the $\beta_i$ is nonzero.
    By relabelling $\beta_{l+1}\neq 0$, then $w_{l+1}\in\langle\{v_1,\ldots,v_l,v_{l+1},w_{l+2}\ldots,w_n\}\rangle$, therefore the set of vectors $\{v_1,\ldots,v_l,v_{l+1},w_{l+2}\ldots,w_n\}$ also spans $V$.
    The theorem is then obvious by induction.
\end{proof}
\begin{corollary}\label{dim_well_defined}
    Let $V$ be a finite dimensional vector space, then any two bases of $V$ have the same cardinality.
\end{corollary}
\begin{proof}
    Immediate.
\end{proof}
This corollary allows us to give a proper definition of the dimension of a vector space.
Before we step right into that, another corollary of Theorem \ref{steinitz} can help us to capture important properties of a finite dimensional vector space that will come in handy in further discussions of basis.
\begin{corollary}
    Let $V$ be a vector space with $\dim V=n$, then:\\
    1. Any independent set of vectors has size at most $n$.
    The size is exactly $n$ iff this set is a basis.\\
    2. Any spanning set has size at least $n$.
    The size is exactly $n$ iff this set is a basis.
\end{corollary}
\begin{proof}
    Obvious.
\end{proof}


\subsection{Cosine functions as activation function}
Let us use a simple example to motivate the spectral Barron space. Consider a bounded domain $\Omega\subset \mathbb
R^d$ and a real function $u\in L^1(\Omega)$.
Recall the Fourier transform of $u\in L^1(\mathbb{R})$ in Definition~\ref{def:fourier1} and \ref{def:fourier2}.
This gives the following integral representation of $u$ in terms of the cosine function
\begin{equation}
 \label{eq:reint}
u(x)=Re\int_{\mathbb{R}^d} e^{2\pi i\omega\cdot x} \hat u(\omega)d\omega
= \int_{\mathbb{R}^d}\cos (2\pi (\omega\cdot x + b(\omega))) |\hat u(\omega)|d\omega,
\end{equation}
where $ \hat u(\omega)= e^{2\pi ib(\omega)}|\hat u(\omega)|$. Let
\begin{equation}
 \label{eq:2}
g(x, \omega) = \cos(2\pi (\omega\cdot x + b(\omega)))\quad \mbox{ and }\quad
\rho(\omega)= |\hat u(\omega)| .
\end{equation}
Thus,
\begin{equation}
\label{int-rep}
u(x)= \int_{\mathbb{R}^d}g(x,\omega) \rho(\omega)d\omega,
\end{equation}
If
$$
\int_{\mathbb R^d} |\hat u(\omega)|d\omega <\infty,
$$
then $\|\rho\|_{L^1}<\infty$. By applying the Lemma \ref{lem:sample},
there exist $\omega_i\in \mathbb R^d$
such that
\begin{equation}
  \label{eq:3}
\|u-u_N\|_{0,\Omega}\le N^{-1/2}\|\hat u\|_{L^1(\mathbb R^d)}.
\end{equation}
where
\begin{equation}\label{cosfn}
u_N(x) = {\|\hat u\|_{L^1(\mathbb R^d)}\over N} \sum_{i=1}^N \cos (2\pi(\omega_i\cdot x + b(\omega_i)))
\end{equation}
More generally, we consider the approximation property in $H^m$-norm.
By \eqref{eq:reint},
\begin{equation}
\partial^\alpha u(x)= \int_{\mathbb{R}^d} \cos^{|\alpha|}(2\pi (\omega\cdot x + b(\omega)))\omega^\alpha |\hat u(\omega)|d\omega,  \quad \forall\ |\alpha|\le m.
\end{equation}
For any positive integer $m$, let
\begin{equation} \label{eq:gm}
g_m(x,\omega)= {\cos (2\pi (\omega\cdot x + b(\omega)))\over  (1+ \|\omega\|)^m}\quad \mbox{and}\quad \rho_m(\omega)= (1+ \|\omega\|)^m|\hat u(\omega) |,
\end{equation}
where
$$
\| \rho_m\|_{L^1(\mathbb R^d)}=\int_{\mathbb R^d} (1+ \|\omega\|)^m|\hat u(\omega) | d\omega<\infty.
$$
Then, $\displaystyle u(x)=\int_{\mathbb R^d} g_m(x,\omega)\rho_m d\omega = \| \rho_m\|_{L^1(\mathbb R^d)}\mathbb{E}g_m(x,\omega)$. Define
\begin{equation}
u_N(x) = {\|\rho_m\|_{L^1(\mathbb R^d)}\over N} \sum_{i=1}^N g_m(x,\omega_i)
= {\|\rho_m\|_{L^1(\mathbb R^d)}\over N} \sum_{i=1}^N {\cos (2\pi(\omega_i\cdot x + b(\omega_i)))\over (1+\|\omega_i\|)^m}.
\end{equation}
It holds that
$$
\partial^\alpha (u(x) - u_N(x))={\|\rho_m\|_{L^1(\mathbb R^d)}\over N}\sum_{i=1}^N \mathbb{E} \partial^\alpha (g_m(x,\omega) -  g_m(x,\omega_i)).
$$
By Lemma \ref{MC},
\begin{align}
\mathbb{E}_N \sum_{|\alpha|\le m}\|\partial^\alpha (u(x) - u_N(x))\|_{0, \Omega}^2
&\le
\|\rho_m \|_{L^1(\mathbb R^d)}^2\mathbb{E}_N \sum_{|\alpha|\le m}\frac{1}{N^2}\sum_{i=1}^N \left (\mathbb{E} \partial^\alpha (g_m(x,\omega) -  g_m(x,\omega_i))\right )^2
\\
&\le
{\|\rho_m \|_{L^1(\mathbb R^d)}^2\over N}  \sum_{|\alpha|\le m}\mathbb{E} \left (\partial^\alpha g_m(x,\omega)\right)^2
\end{align}
Note that the definitions of $g_m$ and $\rho_m$ in \eqref{eq:gm} guarantee that
$$
|\partial^\alpha g_m(x,\omega)|\le 1.
$$
Thus,
$$
\mathbb{E}_N \sum_{|\alpha|\le m}\|\partial^\alpha (u(x) - u_N(x))\|_{0, \Omega}^2 \lesssim {\|\rho_m \|_{L^1(\mathbb R^d)}^2\over N}.
$$
This implies that there exist
$\omega_i\in \mathbb R^d$ such that
\begin{equation}
\label{cosHm}
\|u-u_N\|_{H^m(\Omega)}\lesssim N^{-1/2}\int_{\mathbb{R}^d} (1+ \|\omega\|)^m|\hat u(\omega) | d\omega.
\end{equation}

Given $v\in L^2(\Omega)$,   consider all the possible extension $v_E:
\mathbb{R}^d \mapsto \mathbb{R}$ with $v_E |_{\Omega} = v$ and define
the spectral  Barron norm for any $s\ge 1$:
	\begin{equation}\label{barron-norm0}
	\|v\|_{B^{s}(\Omega)} = \inf_{v_E |_{\Omega} = v} \int_{\mathbb{R}^d}(1+\|\omega\|)^s|\hat{v}_E(\omega)|d\omega
	\end{equation}
and  spectral  Barron space
\begin{equation}
  \label{Barron}
	B^{s}(\Omega) = \{v\in L^2(\Omega): \|v\|_{B^{s}(\Omega)}<\infty\}.
\end{equation}

In summary, we have
\begin{equation}
\|u-u_N\|_{H^m(\Omega)}\lesssim N^{-1/2}  \|u\|_{B^{m}(\Omega)},
\end{equation}
where $u_N$ is defined in \eqref{cosfn}.

Specifically, we will consider the problem of approximating a function with bounded Barron norm \eqref{barron-norm0} in the Sobolev space $H^m(\Omega)$. Our first step will be to prove a lemma showing that the Sobolev norm is bounded by the Barron norm.

\begin{lemma}\label{smoothness-lemma}
 Let $m \geq 0$ be an integer and $\Omega\subset \mathbb{R}^d$ a bounded domain. Then for any Schwartz function $v$, we have
 \begin{equation}\label{embend}
 \|v\|_{W^{m,\infty}(\Omega)} \lesssim \|v\|_{{B}^m(\Omega)} \lesssim  \|v\|_{H^{m + {d\over 2}+\epsilon}(\Omega)},
 \end{equation}
 where $\epsilon$ is positive.
\end{lemma}
\begin{proof}
Recall the inverse Fourier transform in Definition \ref{def:fourier1}
$$
v(x)=\int \hat{v}(\omega) e^{2 \pi i\omega \cdot x} d \omega.
$$
For any $|\alpha|\le m$,
$$
|\partial^\alpha v|=|(2\pi)^{|\alpha|}\int \hat{v}(\omega) \omega^\alpha e^{2 \pi i\omega \cdot x} d \omega|\le |(2\pi)^{|\alpha|}\int \hat{v}(\omega) \|\omega\|^\alpha e^{2 \pi i\omega \cdot x} d \omega| \lesssim \|v\|_{B^m(\Omega)},
$$
which proves $ \|v\|_{W^{m,\infty}(\Omega)} \lesssim \|v\|_{{B}^m(\Omega)}$.

\iffalse
 Let $\chi$ be a Schwartz function satisfying $\chi(x) = 1$ for $x\in \Omega$. Such a function exists because $\Omega$ is bounded. Let $\alpha$ be any multi-index with $|\alpha|\leq m$. Then we have
 \begin{equation}
  \|D^\alpha v\|_{L^2(\Omega)} \leq \|\chi D^\alpha u\|_{L^2(\mathbb{R}^d)} \leq \|\hat\chi * \widehat{D^\alpha v}\|_{L^2(\mathbb{R}^d)}
 \end{equation}
 Now we use Young's inequality to obtain
 \begin{equation}
  \|\hat\chi * \widehat{D^\alpha v}\|_{L^2(\mathbb{R}^d)} \leq \|\hat\chi\|_{L^2(\mathbb{R}^d)}\|\widehat{D^\alpha v}\|_{L^1(\mathbb{R}^d)} \leq \|\hat\chi\|_{L^2(\mathbb{R}^d)}\|v\|_{\mathcal{B}^m(\Omega)}.
 \end{equation}
 Combining this over all multi-indices $\alpha$, we get
 \begin{equation}
  \|u\|_{H^m(\Omega)} \lesssim \|v\|_{\mathcal{B}^m(\Omega)},
 \end{equation}
 as desired. \fi

 A version of the second inequality
  in \eqref{embend} and its proof
can be found in \cite{barron1993universal}. Below is a proof, by
definition and Cauchy-Schwarz
  inequality,
\begin{align}
\|v\|_{B^m(\Omega)} =& \inf_{v_E |_{\Omega} = v} \left(\int_{\mathbb{R}^d}(1+\|\omega\|)^m|\hat{v}_E(\omega)|d\omega \right)^2
\\
\le &  \int_{\mathbb{R}^d}(1+\|\omega\|)^{-d - 2\epsilon}d\omega  \inf_{v_E |_{\Omega} = v}
\int_{\mathbb{R}^d}(1+\|\omega\|)^{d + 2m + 2\epsilon} |\hat{v}_E(\omega)|^2d\omega
\\
\lesssim &  \inf_{v_E |_{\Omega} = v}
\int_{\mathbb{R}^d}(1+\|\omega\|)^{d + 2m + 2\epsilon} |\hat{v}_E(\omega)|^2d\omega
\lesssim \|v\|_{H^{m + {d\over 2}+\epsilon}(\Omega)}.
\end{align}

\end{proof}


\newpage
\chapter*{Summary of Probability Theory I}
\addcontentsline{toc}{chapter}{Summary of Probability Theory I}

{\scriptsize

\begin{framed}
\begin{tabular}{rcl}
SET SUMMARY\\ \\
$\{a_1,a_2,\dots,a_n\}$& $-$& a set containing the elements, $a_1,a_2,\dots,a_n$.\\
$a\in A$ &$-$& $a$ is an element of the set $A$.\\
$A\subseteq B$ &$-$& the set $A$ is a subset of $B$.\\
$A\cup B$ &$-$& ``union'', meaning the set of all elements which are in $A$ or $B$, \\
& & \ \ or both.\\
$A\cap B$ &$-$& ``intersection'', meaning the set of all elements in both $A$ and $B$.\\
$\{\}$ or $\emptyset$ &$-$& empty set.\\
$\Omega $ &$-$& universal set.  \\
$A^c $ &$-$& the complement of $A$, meaning the set of all elements in $\Omega$,\\
& & \ \ the universal set, which are not in $A$.\\
\end{tabular}
\end{framed}

\begin{framed}
EXPERIMENT SUMMARY\\

\begin{tabular}{rcl}
Experiment &$-$& an activity producing distinct outcomes.\\
$\Omega$ &$-$& set of all outcomes of the experiment.\\
$\omega$ &$-$& an individual outcome in $\Omega$, called a simple event.\\
$A\subseteq \Omega$ &$-$& a subset $A$ of $\Omega$ is an event.\\
Trial &$-$& one performance of an experiment resulting in 1 outcome.\\
\end{tabular}
\end{framed}

\begin{framed}
PROBABILITY SUMMARY

\medskip

Axioms:
\begin{enumerate}
\item If $A\subseteq \Omega$ then $0\leq P(A)\leq 1$ and $P(\Omega)=1$.
\item If $A$, $B$ are disjoint events, then $P(A\cup B)=P(A)+P(B)$.

[This is true only when $A$ and $B$ are disjoint.]
\item If $A_1,A_2,\dots$ are disjoint then $P(A_1\cup A_2
\cup\dots)=P(A_1)+P(A_2)+\dots$
\end{enumerate}
Rules:
$$P(A^c)\;=\;1-P(A)$$
$$P(A\cup B)\;=\;P(A)+P(B)-P(A\cap B) \qquad [\textrm{always true}]$$
\end{framed}

\newpage
\begin{framed}
CONDITIONAL PROBABILITY SUMMARY\\

$P(A|B)$ means the probability that $A$ occurs given that $B$ has
occurred.

$$P(A|B)\;=\;\frac{P(A\cap B)}{P(B)}\;=\;\frac{P(A)P(B|A)}{P(B)}\quad \textrm{if}\quad P(B)\neq0$$

$$P(B|A)\;=\;\frac{P(A\cap B)}{P(A)}\;=\;\frac{P(B)P(A|B)}{P(A)}\quad \textrm{if}\quad P(A)\neq 0$$

Conditional probabilities obey the 4 axioms of probability.
\end{framed}

\begin{framed}
DISCRETE RANDOM VARIABLE SUMMARY\\

Probability mass function $$f(x)=P(X=x_i)$$
Distribution function $$F(x)=\sum_{x_i\leq x}f(x_i)$$

\begin{center}
{\small
{\renewcommand{\arraystretch}{1.25}
\begin{tabular}{|c|c|p{3.0cm}|p{5.0cm}|}
\multicolumn{1}{c}{\bf Random  Variable} & \multicolumn{1}{c}{\bf Possible  Values}
&\multicolumn{1}{c}{\bf Probabilities} &\multicolumn{1}{c}{\bf Modeled situations} \\\hline
Discrete  uniform&$\{x_1,x_2,\dots,x_k\}$&$P(X=x_i)=\frac{1}{k}$&Situations with $k$ equally likely values.  Parameter: $k$.\\\hline
$\bernoulli(\theta)$&$\{0,1\}$&$P(X=0)=1-\theta$\newline$P(X=1)=\theta$&Situations with only 2 outcomes, coded 1 for success and 0 for failure.\newline Parameter:\newline $\theta=P(\textrm{success}) \in (0, 1)$.\\\hline
Geometric($\theta$)&$\{1,2,3,\dots\}$&$P(X=x)\newline=(1-\theta)^{x-1}\theta$& Situations where you count the number of trials until the first success in a sequence of independent trails with a constant probability of success. \newline Parameter:\newline $\theta=P(\textrm{success}) \in (0, 1)$.\\\hline
Binomial($n,\theta$)&$\{0,1,2,\dots,n\}$&$P(X=x)$\newline $=\displaystyle\binom{n}{x}\theta^x(1-\theta)^{n-x}$&Situations where you count the number of success in $n$ trials where each trial is independent and there is a constant probability of success.\newline Parameters: $n \in \{1,2,\ldots\}$;\newline $\theta=P(\textrm{success}) \in (0, 1)$.\\\hline
Poisson($\lambda$)&$\{0,1,2,\dots\}$&$P(X=x)\newline=\displaystyle \frac{\lambda^xe^{-\lambda}}{x!}$&Situations where you count the number of events in a continuum where the events occur one at a time and are independent of one another.\newline Parameter: $\lambda$= rate $\in (0,\infty)$.\\\hline
\end{tabular}}
}
\end{center}
\end{framed}

\begin{framed}
CONTINUOUS RANDOM VARIABLES: NOTATION\\

$f(x)$: Probability density function (PDF)
\begin{itemize}
\item$f(x)\;\geq\;0$
\item Areas underneath $f(x)$ measure probabilities.
\end{itemize}

$F(x)$: Distribution function (DF)
\begin{itemize}
\item $0\;\leq\;\ F(x)\;\leq \;1$
\item $F(x)\;= \;P(X\leq x)$ is a probability
\item $F^{\prime}(x)\;=\;f(x)$ for every $x$ where $f(x)$ is continuous
\item $F(x)\;=\;\displaystyle\int^x_{-\infty}f(v)dv$
\item $P(a<X\leq b)\;=\;F(b)-F(a)\;=\;\displaystyle \int^b_af(v)dv$
\end{itemize}
\end{framed}

\begin{framed}
\textbf{Expectation} of a function $g(X)$ of a random variable $X$ is defined as:
\[
E(g(X))\; =\;
\begin{cases}
\displaystyle \sum_x g(x) f(x) & \text{if $X$ is a discrete RV}\\[12pt]
\displaystyle \int_{-\infty}^{\infty} g(x) f(x) dx & \text{if $X$ is a continuous RV}
\end{cases}
\]
\begin{center}
Some Common Expectations
{\renewcommand{\arraystretch}{1.75}
%\begin{tabular}{|c|c|p{3.0cm}|p{5.0cm}|}
\begin{tabular}{|c|p{6.0cm}|p{6.0cm}|}
\hline
$g(x)$ & definition & also known as \\
\hline
$x$    & $E(X)$     & Expectation, Population Mean or First Moment of $X$\\
%       &            & or First Moment of $X$\\
\hline
$(x-E(X))^2$ & $V(X):=E((X-E(X))^2)$ \newline $\quad =E(X^2)-(E(X))^2$ & Variance or Population Variance of $X$\\
\hline
$e^{\imath t x}$ & $\phi_X(t) := E\left( e^{\imath t X}\right)$ & Characteristic Function (CF) of $X$\\
\hline
$x^k$  & $E(X^k) =\frac{1}{\imath^k} \left[\frac{d^k \phi_X(t)}{dt^k}\right]_{t=0} $  & $k$-th Moment of $X$\\
\hline
\end{tabular}
}
\end{center}
\end{framed}

\begin{table}[htb]
\centering
{\small
\begin{tabular}{| l | l |}
\hline
Symbol & Meaning\\ \hline
$\BB{1}_{A}(x)$ & Indicator or set membership function that returns $1$ if $x \in A$ and $0$ otherwise\\
$\Rz^d := (-\infty,\infty)^d$ & $d$-dimensional Real Space \\
\hline
\end{tabular}
}
\caption{Symbol Table: Probability and Statistics \label{T:SymbTableProbStats}}
\end{table}

%\section*{Summary of Random Variables}\label{S:SummaryRVs}
\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|c|}%|c}
\hline
Model &PDF or PMF&Mean&Variance\\ \hline%&{\tt MGF}\\
%$\pointmass(\theta)$&$\BB{1}_{\{\theta\}}(x)$&$\theta$&$0$\\%&$e^{\theta t}$\\
$\bernoulli(\theta)$ &$\theta^x(1-\theta)^{1-x} \BB{1}_{\{0,1\}}(x)$&$\theta$&$\theta(1-\theta)$\\%&$\theta e^t+(1-\theta)$\\
$\binomial(n,\theta)$&$(^n_{\theta})\theta^x(1-\theta)^{n-x} \BB{1}_{\{0,1,\ldots,n\}}(x)$&$n \theta$&$n \theta(1-\theta)$\\%&$(\theta e^t+(1-\theta))^n$\\
$\geometric(\theta)$ &$\theta(1-\theta)^{x} \BB{1}_{\Zz_+}(x)$&$ \frac{1}{\theta}-1$&$\frac{1-\theta}{\theta^2}$\\%&$\frac{\theta e^t}{1-(1-\theta)e^t}(t<-log(1-\theta))$\\
$\poisson(\lambda)$&$\frac{\lambda^xe^{-\lambda}}{x!} \BB{1}_{\Zz_+}(x)$&$\lambda$&$\lambda$\\%&$e^{\lambda(e^t-1)}$\\
$\uniform(\theta_1,\theta_2)$&$\BB{1}_{[\theta_1,\theta_2]}(x)/(\theta_2-\theta_1)$&$\frac{\theta_1+\theta_2}{2}$&$\frac{(\theta_2-\theta_1)^2}{12}$\\%&$\frac{e^{\theta_2 t}-e^{\theta_1 t}}{(\theta_2-\theta_1)t}$\\
$\exponential(\lambda)$&$\lambda e^{-\lambda x}$&$\lambda^{-1}$&$\lambda^{-2}$\\%&$\frac{1}{1-\frac{1}{\lambda} t}(t<\lambda)$\\
$\normal(\mu,\sigma^2)$&$\frac{1}{\sigma\sqrt{2\pi}}e^{(x-\mu)^2/(2\sigma^2)}$&$\mu$&$\sigma^2$\\%&$exp\{ut+\frac{\sigma^2t^2}{2}\}$\\
%$\gammA(\alpha,\beta)$&$\frac{\beta^{\alpha}}{\Gamma(\alpha)}{x^{\alpha-1}e^{-\beta x}}$&$\alpha/\beta$&$\alpha/\beta^2$\\%&$(\frac{1}{1-\beta t})^{\alpha}(t<1/\beta)$\\
%$\betA(\alpha,\beta)$ & $\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}$ & $\frac{\alpha}{\alpha+\beta}$ & $\frac{\alpha\beta}{(\alpha+\beta)^2 (\alpha+\beta+1)} $ \\%& $ 1+\Sigma^\infty_{k+1}(\Pi^{k-1}_{r=0}\frac{\alpha+r}{\alpha+\beta+r})\frac{t^k}{k\!}$\\
%$t_v$ & $\frac{\Gamma((v+1)/2)}{\Gamma(v/2)}  \frac{1}{(1+x^2/v)^{(v+1)/2}$ &0 (if $v>1$ ) & $\frac{v}{v-2}$  (if $v>2$) & does not exist \\
% $\chi^2_p$&$\frac{1}{\Gamma(p/2)2^{p/2}}x^{(p-2)-1}e^{-x/2}$&$p$&$2p$\\ \hline%&$(\frac{1}{1-2t})^{p/2}(t<1/2)$\\
\hline
\end{tabular}
\caption{Random Variables with PDF and PMF (using indicator function), Mean and Variance}
%\label{tab:}
\end{table}

\begin{table}[htb]
\centering
{\small
\begin{tabular}{| l | l |}
\hline
Symbol & Meaning\\ \hline
$A = \{ \star, \circ, \bullet \}$& $A$ is a set containing the elements $\star$, $\circ$ and $\bullet$ \\
$\circ \in A$ & $\circ$ belongs to $A$ or $\circ$ is an element of $A$\\
$A \ni \circ$ & $\circ$ belongs to $A$ or $\circ$ is an element of $A$\\
$\odot \notin A$& $\odot$ does not belong to $A$ \\
$\# A$& Size of the set $A$, for e.g.~$\#\{ \star, \circ, \bullet, \odot \}=4$\\
$\Nz$& The set of natural numbers $\{1,2,3,\ldots\}$\\
$\Zz$&The set of integers $\{\ldots,-3,-2,-1,0,1,2,3,\ldots\}$\\
$\Zz_+$& The set of non-negative integers $\{0,1,2,3,\ldots\}$\\
%$\Rz$& The set of real numbers\\
$\emptyset$&Empty set or the collection of nothing or $\{\}$\\
$A \subset B$ & $A$ is a subset of $B$ or $A$ is contained by $B$, e.g.~$A=\{\circ\}, B=\{\bullet\}$\\
$A \supset B$& $A$ is a superset of $B$ or $A$ contains $B$ e.g.~$A=\{\circ, \star,\bullet\}, B=\{\circ, \bullet\}$ \\
$A=B$ & $A$ equals $B$, i.e.~$A \subset B$ and  $B \subset A$\\
$Q \implies R$ & Statement $Q$ implies statement $R$ or If $Q$ then $R$ \\
$Q \iff R$ & $Q \implies R$ and $R \implies Q$ \\
$\{x: x \text{ satisfies property } R \}$& The set of all $x$ such that $x$ satisfies property $R$\\
$A \cup B$& $A$ union $B$, i.e.~$\{x: x\in A \text{ or } x \in B\}$\\
$A \cap B$& $A$ intersection $B$, i.e.~$\{x: x\in A \text{ and } x \in B\}$\\
$A \setminus B $& $A$ minus $B$, i.e.~$\{x: x\in A \text{ and } x \notin B\}$\\
$A:=B$& $A$ is equal to $B$ by definition\\
$A=:B$& $B$ is equal to $A$ by definition\\
$A^c$& $A$ complement, i.e.~$\{x: x\in U, \text{ the universal set, but } x \notin A\}$\\
$A_1 \times A_2 \times \cdots \times A_m$ & The $m$-product set $\{(a_1,a_2,\ldots,a_{m}): a_1 \in A_1, a_2 \in A_2, \ldots, a_{m} \in A_m \}$ \\
$A^m$ & The $m$-product set $\{(a_1,a_2,\ldots,a_{m}): a_1 \in A, a_2 \in A, \ldots, a_{m} \in A \}$ \\
$f := f(x)=y:\Xz \to \Yz$ & A function $f$ from domain $\Xz$ to range $\Yz$ \\
$f^{[-1]}(y)$ & Inverse image of $y$ \\
$f^{[-1]} := f^{[-1]}(y \in \Yz) = X \subset \Xz$ & Inverse of $f$ \\
%$\Zz:=\{\ldots,-2,-1,0,1,2,\ldots\}$ & Integers \\
$a<b$ or $a \leq b$ & $a$ is less than $b$ or $a$ is less than or equal to $b$ \\
$a>b$ or $a \geq b$ & $a$ is greater than $b$ or $a$ is greater than or equal to $b$ \\
$\Qz$ & Rational numbers \\
$(x,y)$ & the open interval $(x,y)$, i.e.~$\{r: x < r < y\}$ \\
$[x,y]$ & the closed interval $(x,y)$, i.e.~$\{r: x \leq r \leq y\}$ \\
$(x,y]$ & the half-open interval $(x,y]$, i.e.~$\{r: x < r \leq y\}$ \\
$[x,y)$ & the half-open interval $[x,y)$, i.e.~$\{r: x \leq r < y\}$ \\
$\Rz := (-\infty,\infty)$ & Real numbers, i.e.~$\{r: -\infty < r <  \infty \}$ \\
$\Rz_+ := [0,\infty)$ & Real numbers, i.e.~$\{r: 0 \leq r <  \infty \}$ \\
$\Rz_{>0} := (0,\infty)$ & Real numbers, i.e.~$\{r: 0 < r <  \infty \}$ \\
\hline
\end{tabular}
}
\caption{Symbol Table: Sets and Numbers \label{T:SummarySymbTableSets}}
\end{table}

\begin{table}[htb]
\centering
{\small
{\renewcommand{\arraystretch}{1.75}
\begin{tabular}{| p{6.0cm}| p{10.0cm}|}
\hline
Symbol & Meaning\\ \hline
$\BB{1}_{A}(x)$ & Indicator or set membership function that returns $1$ if $x \in A$ and $0$ otherwise\\
$\Rz^d := (-\infty,\infty)^d$ & $d$-dimensional Real Space \\
\rv~ & random vector\\
$F_{X,Y}(x,y)$ & Joint distribution function (JDF) of the \rv~ $(X,Y)$\\
$F_{X,Y}(x,y)$ & Joint cumulative distribution function (JCDF) of the \rv~ $(X,Y)$ --- same as JDF \\
$f_{X,Y}(x,y)$ & Joint probability mass function (JPMF) of the discrete \rv~ $(X,Y)$\\
$\mathcal{S}_{X,Y}$\newline$=\{(x_i,y_j): f_{X,Y}(x_i,x_j)>0\}$ & The support set of the discrete \rv~ $(X,Y)$\\
$f_{X,Y}(x,y)$ & Joint probability density function (JPDF) of the continuous \rv~ $(X,Y)$\\
$f_{X}(x)=\int_{-\infty}^{\infty}f_{X,Y}(x,y)dy$ & Marginal probability density/mass function (MPDF/MPMF) of $X$\\
$f_{Y}(y)$\newline$=\int_{-\infty}^{\infty}f_{X,Y}(x,y)dx$ & Marginal probability density/mass function (MPDF/MPMF) of $Y$\\
$E(g(X,Y))$\newline$=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}g(x,y)f_{X,Y}(x,y)dxdy$ & Expectation of a function $g(x,y)$ for continuous \rv \\
$E(g(X,Y))$\newline$=\sum_{(x,y)\in\mathcal{S}_{X,Y}}g(x,y)f_{X,Y}(x,y)$ & Expectation of a function $g(x,y)$ for discrete \rv \\
$E(X^rY^s)$ & Joint moment \\
$\cv(X,Y) = E(XY)-E(X)E(Y)$ & Covariance of $X$ and $Y$, provided $E(X^2)<\infty$ and $E(Y^2)>\infty$\\
$F_{X,Y}(x,y)=F_X(x)F_Y(y)$,\newline for every $(x,y)$ & if and only if $X$ and $Y$ are said to be independent\\
$f_{X,Y}(x,y)=f_X(x)f_Y(y)$,\newline for every $(x,y)$ & if and only if $X$ and $Y$ are said to be independent\\
$F_{X_1,X_2,\ldots,X_n}(x_1,x_2,\ldots,x_n)$ & Joint (cumulative) distribution function (JDF/JCDF) of the discrete or continuous \rv~ $(X_1,X_2,\ldots,X_n)$\\
$f_{X_1,X_2,\ldots,X_n}(x_1,x_2,\ldots,x_n)$ & Joint probability mass/density function (JPMF/JPDF) of the discrete/continuous \rv~ $(X_1,X_2,\ldots,X_n)$\\
$f_{X_1,X_2,\ldots,X_n}(x_1,x_2,\ldots,x_n)$\newline$=\prod_{i=1}^n f_{X_i}(x_i)$,\newline for every $(x_1,x_2,\ldots,x_n)$ & if and only if $X_1,X_2,\ldots,X_n$ are (mutually/jointly) independent\\
\hline
\end{tabular}
}
}
\caption{Symbol Table: Probability and Statistics \label{T:FullSymbTableProbStats}}
\end{table}
}


\section{Compactness}
Recall that a continuous, real-valued function on a closed bounded interval is bounded and attains its bound.
The question is, for which topological space $X$ is it true that every continuous real functions is bounded.
\begin{example}
    1. For finite $X$, every function $X\to\mathbb R$ is bounded.\\
    2. If for all continuous $f:X\to\mathbb R,\exists n\in\mathbb N,\exists A_1,A_2,\ldots,A_n\subset X$ such that $X=\bigcup_iA_i$ and $f$ is bounded on each $A_i$, then $f$ is bounded on $X$.
\end{example}
Note that given continuous $f:X\to\mathbb R$, for $x\in X$, $U_x=f^{-1}((f(x)-1,f(x)+1))$ is open and $f$ is bounded there.
So if there is some finite subset of $\{U_x:x\in X\}$ that still covers $X$, then $f$ must be bounded.
\begin{definition}
    An open cover of a topological space $X$ is a family of open sets $\mathscr U=\{U_i\}_{i\in I}$ in $X$ such that $X=\bigcup_{i\in I}U_i$.\\
    A subcover of $\mathscr U$ is a subset $\mathscr V\subset \mathscr U$ that is also an open cover of $X$.
    $\mathscr V$ is called a finite subcover if it is finite.\\
    $X$ is compact if every open cover of $X$ has a finite subcover.
\end{definition}
\begin{theorem}
    If $X\neq\varnothing$ is compact and $f:X\to\mathbb R$ is continuous, then $f$ is bounded and attains its bound.
\end{theorem}
\begin{proof}
    By continuity of $f$ and compactness of $X$, there is a finite subset of $\{f^{-1}((f(x)-1,f(x)+1)):x\in X\}$ that covers $X$, which means that $f$ is bounded on any set in a finite family, so $f$ is bounded in the union of that family, which is $X$.
    To show that $f$ attains its bound, let $m=\{f(x):x\in X\}$ which exists since $X\neq\varnothing$ and $f$ is bounded.
    Suppose that there is not an $x$ with $f(x)=m$, so for any $x\in X,f(x)>m$ so $\exists m_x$ such that $f(x)>m_x>m$.
    Let $U_x=f^{-1}((m_x,\infty))$ which is open and contains $x$, and $\inf_{U_x}f\ge m_x>x$.
    Note that the family of all $U_x$ is an open cover of $X$, so there is a finite subcover $\{U_x\}_{x\in F}$, so $\forall y\in X,f(y)\ge \min_{x\in F}m_x>m$, contradiction.
\end{proof}
Note that for a subspace $Y\subset X$, $Y$ is compact iff whenever $\mathscr U$ is a family of open set in $X$ whose union contains $Y$, there is a finite subset $\mathscr V\subset \mathscr U$ such that the union of elements in $\mathscr V$ contains $Y$.
\begin{theorem}\label{01compact}
    $[0,1]$ is compact.
\end{theorem}
\begin{proof}
    Let $\mathscr U$ be a set of open sets in $\mathbb R$ thar contains $[0,1]$, assume that there does not exist finite subcover that contains $[0,1]$, then if $0\le a<b\le 1$ and $[a,b]$ cannot be covered by any finite $\mathscr V\subset\mathscr U$, then let $C=(a+b)/2$, then one of $[a,c],[c,b]$ cannot be covered by finite $\mathscr V\subset\mathscr U$.\\
    Therefore, inductively we can find intervals $I_n=[a_n,b_n]$ such that $$I_0=[0,1],I_{n+1}\subset I_n,|b_n-a_n|=1/2^n$$
    thus $a_n\to x,b_n=a_n+(b_n-a_n)\to x$ for some $x\in [0,1]$.
    Now there is $U\subset \mathscr U$ such that $x\in U$, but then there is some $\epsilon$ with $(x-\epsilon,x+\epsilon)\subset U$, therefore for sufficiently large $n$, $I_n\subset U$, which is a contradiction.
\end{proof}
\begin{proposition}\label{compact_haus_closed}
    Let $X$ be a topological space and $Y\subset X$ a subspace, then\\
    1. If $X$ is compact and $Y$ is closed in $X$, then $Y$ is compact.\\
    2. If $X$ is Hausdorff and $Y$ is compact, then $Y$ is closed.
\end{proposition}
\begin{proof}
    1. Let $\mathscr U$ covers $Y$, then $\mathscr U\cup \{X\setminus Y\}$ is an open cover of $X$.
    Since $X$ is compact, there is a finite subcover $\mathscr V\subset \mathscr U\cup \{X\setminus Y\}$ that covers $X$, hence $\mathscr V\setminus\{X\setminus Y\}\subset \mathscr U$ is finite and covers $Y$.\\
    2. We want to show that its complement is open.
    Indeed, for any $x\in X\setminus Y$, and for any $y\in Y$, there are disjoint $U_y,V_y$ such that $x\in U_y,y\in V_y$, then $\{V_y\}_{y\in Y}$ is an open cover of $Y$, thus there is some finite set $F\subset Y$ such that $Y\subset\bigcup_{y\in F}V_y$, so $U=\cap_{y\in F}U_y$ is open, and by definition it is disjoint from $Y$, hence $x\in U\subset X\setminus Y$, which shows that $X\setminus Y$ is open.
\end{proof}
\begin{proposition}
    If $X$ is compact and $f:X\to Y$ is continuous, then $f(X)$ is compact.
\end{proposition}
\begin{proof}
    For any open $\{U_i\}_{i\in I}$ that covers $f(X)$, $\{f^{-1}(U_i)\}_{i\in I}$ is an open cover of $X$, therefore there is some finite $F\subset I$ such that $\{f^{-1}(U_i)\}_{i\in F}$ covers $X$, hence $\{U_i\}_{i\in F}$ covers $f(X)$.
\end{proof}
\begin{remark}
    1. Compactness is a topological property.\\
    2. Let $f:X\to Y$ and $A\subset X$.
    Suppose $A$ is compact, then $f(A)$ is compact.
\end{remark}
\begin{example}
    For $a<b$, $[a,b]=f([0,1])$ where $f(x)=(b-a)x+a$ which is continuous, thus every closed bounded interval is compact.
\end{example}
\begin{corollary}
    If $X$ is compact and $R$ is an equivalence relation on $X$, then $X/R$ is compact.
\end{corollary}
\begin{proof}
    The quotient map is continuous and surjective.
\end{proof}
\begin{theorem}[Topological Inverse Function Theorem]
    If $f:X\to Y$ is a continuous bijection and $X$ is compact and $Y$ is Hausdorff, then $f$ is a homeomorphism.
\end{theorem}
\begin{proof}
    It suffices to check that $f$ is an open map, which, since $f$ is a bijection, is equivalent to say that $f$ is a closed map.\\
    Fix any closed $V\subset X$, then $V$ is compact since $X$ is compact, thus $f(V)$ is compact since $f$ is continuous, therefore $f(V)$ is closed since $Y$ is hausfdorff.
    The result follows.
\end{proof}
\begin{example}
    Consider $f:\mathbb R\to S^1$ by $f(t)=e^{2\pi it}$ induces a continuous bijection $\tilde{f}:\mathbb R/\mathbb Z\to S^1$.
    Now $\mathbb R/\mathbb Z=q([0,1])$ (where $q$ is the quotient map) is compact and $S^1$ is Hausdorff since it is a metric space, therefore $\tilde{f}$ is a homeomorphism.
\end{example}
\begin{theorem}[Tychonorff's Theorem on Finite Products
    \footnote{It works for arbitrary products, but that case is much much harder}
    ]\label{tycho_finite}
    Finite products of compact spaces are compact.
\end{theorem}
\begin{proof}
    It suffices to show for $2$.\\
    Assume $X,Y$ are compact.
    Fix $x\in X$, $\exists W_y\in\mathscr U$ with $x,y\in W_y$, so there is some $U_y$ open in $X$ and $V_y$ open in $Y$ such that $(x,y)\in U_y\times V_y\subset W_y$, so there is a finite $F_Y\subset Y$ with $\bigcup_{y\in F_Y}V_y=Y$.
    Let $T_x=\bigcup_{y\in F_Y}U_y$ is open and contains $x$ and note that $T_x\times Y\subset \bigcup_{y\in F_Y}W_y$.
    But then $\{T_x\}_{x\in X}$ covers $X$, so there is a finite $F_X\subset X$ such that $\{T_x\}_{x\in F_X}$ covers $X$, hence
    $$X\times Y\subset \bigcup_{x\in F_X}T_x\times Y=\bigcup_{x\in F_X}\bigcup_{y\in F_Y}W_y$$
    The last term is the required finite subcover.
\end{proof}
\begin{theorem}[Heine-Borel Theorem]
    A subset $K\subset\mathbb R^n$ if and only if it is closed and bounded.
\end{theorem}
\begin{proof}
    If $K$ is compact, note that $f:\mathbb R^n\to\mathbb R$ by $x\mapsto \|x\|$, thus it is bounded.
    $K$ is also closed by Proposition \ref{compact_haus_closed}.\\
    Conversely, if $K$ is closed and bounded, there is some $M>0$ such that $K\subset [-M,M]^n$ which is compact by Theorem \ref{01compact} and \ref{tycho_finite}.
    Since $K$ is closed in $[-M,M]^n$, it is compact by Proposition \ref{compact_haus_closed}.
\end{proof}
\begin{definition}
    Given an open set $U\subset\mathbb R^n$, a sequence of functions $f_k:U\to\mathbb R$ converges locally uniformly on $U$ to some function $f:U\to\mathbb R$ if $\forall x\in U, \exists r>0, D_r(x)\subset U$ and $f_k\to f$ uniformly on $D_r(x)$.
\end{definition}
Thus this happens if and only if $f_n\to f$ uniformly on any compact subset of $U$.
\begin{definition}
    A topological space $X$ is called sequentially compact if and only if every sequence in $X$ has a convergent subsequence.
\end{definition}
\begin{example}
    Any closed bounded subset of $\mathbb R^n$ is sequentially compact by Bolzano-Weierstrass.
\end{example}
\begin{definition}
    Fix a metric space $(M,d)$.
    For $\epsilon>0$ and $F\subset M$.
    We say $F$ is an $\epsilon$-net for $M$ if $\forall x\in M,\exists y\in F,d(x,y)\le \epsilon$.
    That is,
    $$M=\bigcup_{y\in F} B_\epsilon(y)$$
    We say $M$ is totally bounded if for any $\epsilon>0$, there is a finite $\epsilon$-net for $M$.
\end{definition}
Note that any compact space is totally bounded, but the converse is not true by taking $[0,1)$, but the only thing missing here is completeness.
\begin{theorem}
    The followings are equivalent:\\
    (1) $M$ is compact.\\
    (2) $M$ is sequentially compact.\\
    (3) $M$ is totally bounded and complete.
\end{theorem}
\begin{proof}
    $1\implies 2$: Let $(x_n)$ be a sequence in $M$, so for $n\in\mathbb N$, let $A_n=\{x_k:k>n\}$.
    We shall show that $\bigcap_{n\in\mathbb N}\bar{A}$ is nonempty.
    Assume not, then
    $$\bigcup_{n\in\mathbb N}M\setminus\bar{A}=M$$
    But $M$ is compact and all $M\setminus\bar{A}$ are closed, there is finite subcover, hence there is some $N\in\mathbb N$ such that $\bigcup_{n\le N}M\setminus\bar{A}=M$.
    Also $A_m\supset A_n,\forall m\le n$, so we have the complement of the closure of $A_{N}$ would be $M$, so that closure is empty, contradiction.\\
    So we can fix $x\in \bigcap_{n\in\mathbb N}\bar{A}$.
    It is then trivial to construct a subsequence of $x_n$ that converges to $x$.\\
    $2\implies 3$: $M$ is complete since a Cauchy sequence with convergent subsequence is convergent.
    To see it is totally bounded, assume it is not, then there is some $\epsilon>0$ such that every $\epsilon$-net is infinite.
    Pick $x_1\in M$, then if we have already picked $x_1,\ldots,x_n$, we can pick $x_{n+1}\notin \bigcup_{k=1}^nB_\epsilon(x_k)$, which we can do since $M$ has no finite $\epsilon$-net.
    But this $(x_n)$ does not have any Cauchy subsequence, so it has no converging subsequence.\\
    $3\implies 1$: Assume $M$ is not compact, so there is an open cover $\mathscr U$ without any finite subcover.
    We say $A\subset M$ is ``bad'' if there is no finite subcover of $A$ in $\mathscr U$.
    So $M$ is bad but $\varnothing$ is not.
    Note if $A=\bigcup_{i=1}^nB_i$ is bad, then there is some $i$ such that $B_i$ is bad.\\
    Next, we want to show that if $A$ is bad and $\epsilon>0$, then $\exists B\subset A$ such that $B$ is bad and $\operatorname{diam}B=\sup_{x,y\in B}d(x,y)<\epsilon$.
    Indeed, since $M$ is bounded, we have a finite $\epsilon/2$-net $F$, that is,
    $$\bigcup_{x\in F}B_{\epsilon/2}(x)=M\implies \bigcup_{x\in F}(B_{\epsilon/2}(x)\cap A)=A$$
    But this would mean that there is some $x\in F$ such that $B_{\epsilon/2}(x)\cap A$ is bad, and by triangle inequality its diameter is less than $\epsilon$.
    Using this we can construct a sequence $M\supset A_1\supset A_2\supset\cdots$ such that $A_n$ is bad for any $n$ and $\operatorname{diam}A<1/n$.
    So we can pick $x_n\in A_n$, then $x_n$ is Cauchy, thus it tends to a limit $x\in M$ by completeness, so there is some $U\in\mathscr U$ such that $x\in U$, so $\exists r>0$ such that $D_r(x)\subset U$, which provides a finite subcover for $A_n$ where $n$ is large enough.
\end{proof}
\begin{remark}
    We have a new proof of Bolzano-Weierstrass now!
    We can also have a new proof of Theorem \ref{tycho_finite} for metric spaces.\\
    However, the equivalence of sequentially compactness and compactness fails in both directions in general topological spaces.
\end{remark}


\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}

\begin{document}

\section{Formuals}

Permutations, $k$ objects from a sequence of $n$, or $_{n}P_{k}$.

\begin{equation}
   _{n}P_{k} = \frac{n!}{(n - k)!} = n(n-1)(n-2)\ldots\ (n-k+1)
\end{equation}

Permutations with repetition, same as Permutation except you can choose the same object multiple times.

\begin{equation}
   _{n}P^r_{k} = k^n
\end{equation}

Combinations, $k$ objects from a set $n$ to form a new set, or $\binom{n}{k}$ (pronounced $n$ choose $k$).

\begin{equation}
   \binom{n}{k} = \frac{n!}{k!(n-k)!}
\end{equation}

Combinations with repetition, $k$ objects from a set $n$ to form a new \textit{multi}set, or $\left(\binom{n}{k}\right)$.

\begin{equation}
   \left(\binom{n}{k}\right) = \binom{n + k - 1}{k}
\end{equation}

Combination identities,

\begin{gather}
   \binom{n}{k} = \binom{n}{n - k} \\
   \binom{n}{k} = \binom{n - 1}{k} + \binom{n - 1}{k - 1} \\
   \binom{n}{m} \binom{m}{k} = \binom{n}{k} \binom{n - k}{m - k}
\end{gather}

Sums,

\begin{gather}
   \sum^n_{i=1} i = \frac{n(n+1)}{2} \\
   \sum^n_{i=1} i^2 = \frac{n(n+1)(2n+1)}{6} \\
   \sum^n_{i=1} i^3 = \frac{n^2(n+1)^2}{4} \\
   \sum^n_{k=0} \binom{n}{k} = 2^n
\end{gather}

Logarithm identities,

\begin{gather}
   \log_b x = \frac{\log_a x}{\log_a b} \\
   \log x^y = y \log x \\
   \log(xy) = \log x + \log y
\end{gather}

Trigonometry identities,

\begin{gather}
   \tan x = \frac{\sin x}{\cos x} \\
   \sin^2 x + \cos^2 x = 1 \\
   \sin x = \cos (\frac{\pi}{2} - x) \\
   \sin (x \pm y) = \sin x \cos y \pm \cos x \sin y \\
   \pi = 4 \arctan(1) \\
   e^{ix} = \cos x + i \sin x
\end{gather}

\end{document}


\section{Real inner product spaces}

\begin{outcome}
  \begin{enumerate}
  \item Check whether an operation is an inner product.
  \item Give an example of a vector space on which more than one inner
    product can be defined.
  \item Calculate the inner product of vectors in various examples
    of inner product spaces.
  \item Calculate the norm of a vector and the angle between two vectors.
  \item Use the Cauchy-Schwarz inequality and the triangle inequality
    to reason about the size of inner products and norms of vectors.
  \end{enumerate}
\end{outcome}

In this section, we fix $K$ to be the field of real numbers.

\begin{definition}{Real inner product space}{real-inner-product-space}
  A \textbf{(real) inner product space}%
  \index{inner product space!real}%
  \index{real inner product space} is a real vector space $V$ equipped
  with an operation that assigns to any pair of vectors
  $\vect{u},\vect{v}\in V$ a real number $\iprod{\vect{u},\vect{v}}$,
  called the \textbf{inner product}%
  \index{inner product}%
  \index{multiplication!inner product} of $\vect{u}$ and
  $\vect{v}$. This operation must satisfy the following properties:
  \begin{enumerate}
  \item Symmetry:%
    \index{symmetry!of inner product}
    $\iprod{\vect{u},\vect{v}} = \iprod{\vect{v},\vect{u}}$.
  \item Linearity:%
    \index{linearity!of inner product}
    $\iprod{\vect{u},k\vect{v}+\ell\vect{w}}
    = k\iprod{\vect{u},\vect{v}}+\ell\iprod{\vect{u},\vect{w}}$.
  \item The positive definite property:%
    \index{positive definite property}
    $\iprod{\vect{u},\vect{u}} \geq 0$, and moreover,
    $\iprod{\vect{u},\vect{u}}=0$ if and only if
    $\vect{u} = \vect{0}$.
  \end{enumerate}
\end{definition}

\begin{example}{$\R^n$ with the usual dot product}{rn-with-dot-product}
  $V=\R^n$, with the usual dot product
  \begin{equation*}
    \iprod{\vect{u},\vect{v}} = \vect{u}\dotprod\vect{v} =
    u_1v_1 + \ldots + u_nv_n
  \end{equation*}
  is an inner product space.%
  \index{inner product space!Rn@$R^n$}
\end{example}

\begin{proof}
  The relevant properties were shown in
  Proposition~\ref{prop:properties-dot-product}.
\end{proof}

There exist other inner product operations on $\R^n$ besides the dot
product. The following is an example.

\begin{example}{$\R^2$ with a non-standard inner product}{rn-with-inner-product}
  Let $V=\R^2$, and consider the matrix
  $A=\begin{mymatrix}{cc} 1 & 1 \\ 1 & 2 \end{mymatrix}$.
  Define an operation $\iprod{-,-}$ by
  \begin{equation*}
    \iprod{\vect{u},\vect{v}} = \vect{u}^T A \vect{v} =
    \begin{mymatrix}{cc} u_1 & u_2 \end{mymatrix}
    \begin{mymatrix}{cc} 1 & 1 \\ 1 & 2 \end{mymatrix}
    \begin{mymatrix}{c} v_1 \\ v_2 \end{mymatrix}
    = u_1v_1 + u_1v_2 + u_2v_1 + 2u_2v_2.
  \end{equation*}
  Then $\R^2$, with this operation, is an inner product space.
\end{example}

\begin{proof}
  We must show the three properties are satisfied. For symmetry, note
  that a scalar is equal to its own transpose. Therefore
  \begin{equation*}
    \iprod{\vect{u},\vect{v}}
    ~=~ \vect{u}^T A \vect{v}
    ~=~ (\vect{u}^T A \vect{v})^T
    ~=~ \vect{v}^T A^T \vect{u}.
  \end{equation*}
  But this is equal to $\iprod{\vect{v},\vect{u}}$ because $A$ is a
  symmetric matrix, i.e., $A^T=A$.
  For linearity, note that, by properties of matrix multiplication,
  \begin{equation*}
    \iprod{\vect{u},k\vect{v}+\ell\vect{w}}
    ~=~ \vect{u}^T A(k\vect{v}+\ell\vect{w})
    ~=~ k\vect{u}^T A\vect{v} + \ell\vect{u}^T A\vect{w}
    ~=~ k\iprod{\vect{u},\vect{v}} + \ell\iprod{\vect{u},\vect{w}}.
  \end{equation*}
  For the positive definite property, we have
  \begin{equation*}
    \iprod{\vect{u},\vect{u}}
    ~=~ u_1^2 + 2u_1u_2 + 2u_2^2
    ~=~ (u_1^2 + 2u_1u_2 + u_2^2) + u_2^2
    ~=~ (u_1+u_2)^2 + u_2^2
    ~\geq~ 0,
  \end{equation*}
  because the sum of two squares is always non-negative. Moreover, if
  equality holds in the last equation, then we must have $u_1+u_2=0$
  and $u_2=0$, and this is only possible if $\vect{u}=\vect{0}$.
\end{proof}

\begin{example}{Continuous functions on an interval}{continuous-interval}
  Let $a<b$ be real numbers, and let
  \begin{equation*}
    [a,b] = \set{x \mid a \leq x \leq b}
  \end{equation*}
  be the closed interval. Let $V=C[a,b]$%
  \index{Cab@$C[a,b]$} be the vector space of all continuous
  functions%
  \index{continuous function}%
  \index{function!continuous}%
  \index{inner product space!of continuous functions}
  $f:[a,b]\to\R$. Given two functions $f,g\in C[a,b]$, define
  \begin{equation*}
    \iprod{f,g} = \int_{a}^{b} f(x)g(x)\,dx.
  \end{equation*}
  This is an inner product.
\end{example}

\begin{proof}
  This follows from well-known properties of integrals. For symmetry,
  we have
  \begin{equation*}
    \iprod{f,g}
    ~=~ \int_{a}^{b} f(x)g(x)\,dx
    ~=~ \int_{a}^{b} g(x)f(x)\,dx
    ~=~ \iprod{g,f}.
  \end{equation*}
  For linearity, we have
  \begin{equation*}
    \iprod{f,kg+\ell h}
    ~=~ \int_{a}^{b} f(x)(kg(x)+\ell h(x))\,dx
    ~=~ k\int_{a}^{b} f(x)g(x)\,dx+\ell\int_{a}^{b} f(x)h(x)\,dx
    ~=~ k\iprod{f,g}+\ell\iprod{f,h}.
  \end{equation*}
  For the positive definite property, we have
  \begin{equation*}
    \iprod{f,f}
    = \int_{a}^{b} f(x)^2\,dx
    \geq 0.
  \end{equation*}
  This is $\geq 0$ because $a<b$ and the integral over a non-negative
  function is non-negative. Moreover, since $f$ is continuous, we know
  from calculus that the last integral can only be equal to $0$ if $f$
  is the constant zero function.
\end{proof}

\begin{example}{Polynomials}{inner-product-polynomials}
  Let $\Poly$ be the vector space of polynomials (of any degree) with
  real coefficients. Let $a<b$ be real numbers, and consider the inner
  product defined by
  \begin{equation*}
    \iprod{p,q} = \int_{a}^{b} p(x)q(x)\,dx.
  \end{equation*}
  This is an inner product space.%
  \index{inner product space!of polynomials}
\end{example}

\begin{proof}
  The proof is the same as for $C[a,b]$.
\end{proof}

\begin{example}{Real Hilbert space}{hilbert-space}
  Let $\Hilb_{\R}$ be the vector space of all infinite sequences of
  real numbers $a=(a_0,a_1,a_2,\ldots)$ satisfying
  \begin{equation*}
    a_0^2 + a_1^2 + a_2^2 + \ldots < \infty.
  \end{equation*}
  These are called the \textbf{square summable} sequences%
  \index{sequence!square summable}%
  \index{square summable sequence}. (One needs
  to do a little bit of work to show that it is indeed a vector space;
  in particular, to show that the sum of two square summable sequences
  is square summable.) On this space, we can
  define an inner product as follows:
  \begin{equation*}
    \iprod{a,b} = a_0b_0 + a_1b_1 + a_2b_2 + \ldots.
  \end{equation*}
  The details will be worked out in Exercise~\ref{ex:hilbert-space}.
  This inner product space is called \textbf{(real) Hilbert space}%
  \index{Hilbert space}%
  \index{inner product space!Hilbert space}.
\end{example}

We now look at some properties of inner products. The first thing to
note is that while the axioms require linearity in the right
component, symmetry ensures that linearity in the left component also
holds.

\begin{proposition}{Left linearity}{inner-product-left-linearity}
  Let $V$ be an inner product space. Then
  $\iprod{k\vect{u}+\ell\vect{v},\vect{w}}
  = k\iprod{\vect{u},\vect{w}} + \ell\iprod{\vect{v},\vect{w}}$.
\end{proposition}

\begin{proof}
  This follows directly from symmetry. We have
  \begin{equation*}
    \iprod{k\vect{u}+\ell\vect{v},\vect{w}}
    = \iprod{\vect{w},k\vect{u}+\ell\vect{v}}
    = k\iprod{\vect{w},\vect{u}}
    + \ell\iprod{\vect{w},\vect{v}}
    = k\iprod{\vect{u},\vect{w}}
    + \ell\iprod{\vect{v},\vect{w}}.
  \end{equation*}
\end{proof}

Another important property of inner products is the Cauchy-Schwarz
inequality. We have already encountered this in the context of the dot
product in Section~\ref{sec:dot-product}.

\begin{theorem}{Cauchy-Schwarz inequality}{inner-product-cauchy-schwarz}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space. Then%
  \index{Cauchy-Schwarz inequality!in inner product space}%
  \index{inner product space!Cauchy-Schwarz inequality}
  \begin{equation}\label{cauchy-inner}
    \iprod{\vect{u},\vect{v}}^2
    \leq \iprod{\vect{u},\vect{u}} \cdot \iprod{\vect{v},\vect{v}}.
  \end{equation}
\end{theorem}

\begin{proof}
  The proof is almost identical to that of
  Proposition~\ref{prop:cauchy-schwarz-inequality}.  First note that if
  $\vect{u}=\vect{0}$, then both sides of {\eqref{cauchy-inner}} are
  equal to zero, and so there is nothing to show. Therefore, we will
  assume in what follows that $\vect{u}\neq \vect{0}$.  Define a
  function of $t\in \R$ by
  \begin{equation*}
    f(t) = \iprod{t\vect{u}+\vect{v}, t\vect{u}+\vect{v}}.
  \end{equation*}
  Then by the positive definite property, we know that $f(t)\geq 0$
  for all $t\in \R$.  Also from linearity and symmetry, we have
  \begin{eqnarray*}
    f(t) &=& \iprod{t\vect{u}, t\vect{u}+\vect{v}} +
             \iprod{\vect{v}, t\vect{u}+\vect{v}} \\
         &=& t^2\iprod{\vect{u}, \vect{u}}+t\iprod{\vect{u}, \vect{v}} + t\iprod{\vect{v}, \vect{u}}+
             \iprod{\vect{v}, \vect{v}} \\
         &=&t^2\iprod{\vect{u},\vect{u}}+2t\iprod{\vect{u}, \vect{v}}
             + \iprod{\vect{v}, \vect{v}}.
  \end{eqnarray*}
  This means the graph of $y=f(t)$ is a parabola which opens upwards
  and is never negative. It follows that this function has at most one
  root. From the quadratic formula, we know that a quadratic function
  $at^2+bt+c$ has one or zero roots if and only if $b^2-4ac\leq
  0$. Applying this reasoning to the function $f(t)$, we obtain
  \begin{equation*}
    (2\iprod{\vect{u}, \vect{v}})^2
    - 4\iprod{\vect{u},\vect{u}}\iprod{\vect{v},\vect{v}} < 0,
  \end{equation*}
  which is equivalent to
  $\iprod{\vect{u},\vect{v}}^2 \leq \iprod{\vect{u},\vect{u}} \cdot
  \iprod{\vect{v},\vect{v}}$.
\end{proof}

Finally, we can use the inner product to define the norm of a vector.

\begin{definition}{Norm}{inner-product-norm}
  Let $\vect{u}$ be a vector in an inner product space. Then the
  \textbf{norm}%
  \index{norm!in inner product space}%
  \index{vector!norm}%
  \index{inner product space!norm} of $\vect{u}$ is defined as
  \begin{equation*}
    \norm{\vect{u}} = \sqrt{\iprod{\vect{u},\vect{u}}}.
  \end{equation*}
\end{definition}

\begin{example}{Norm in $C[-1,1]$}{norm-in-continuous-functions}
  Calculate the norm of $f(x) = x^2$ in $C[-1,1]$.
\end{example}

\begin{solution}
  In the vector space $C[-1,1]$, the inner product is defined as
  \begin{equation*}
    \iprod{f,g} = \int_{-1}^{1} f(x)g(x)\,dx.
  \end{equation*}
  We therefore have
  \begin{equation*}
    \iprod{f,f}
    = \int_{-1}^{1} f(x)^2\, dx
    = \int_{-1}^{1} x^4\, dx
    = \bigbracket{\frac{1}{5}x^5}_{-1}^{1}
    = \frac{2}{5}.
  \end{equation*}
  Therefore, $\norm{f} = \sqrt{\iprod{f,f}} = \sqrt{\frac{2}{5}}$.
\end{solution}

A vector $\vect{u}$ in an inner product space is called
\textbf{normalized}%
\index{vector!normalized}%
\index{normalized vector} or a \textbf{unit vector}%
\index{vector!unit vector}%
\index{unit vector} if $\norm{\vect{u}} = 1$.
We note that if $\vect{v}$ is any non-zero vector in an inner product
space, then
\begin{equation*}
  \vect{u} = \frac{1}{\norm{\vect{v}}} \vect{v}
\end{equation*}
is normalized.

By the Cauchy-Schwarz inequality, we have the following properties:

\begin{proposition}{Inner product and norm}{inner-product-and-norm}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space. Then
  \begin{equation*}
    \abs{\iprod{\vect{u},\vect{v}}} \leq \norm{\vect{u}}\norm{\vect{v}}.
  \end{equation*}
\end{proposition}

\begin{proof}
  The Cauchy-Schwarz inequality states that
  $\iprod{\vect{u},\vect{v}}^2 \leq
  \norm{\vect{u}}^2\norm{\vect{v}}^2$. The claim follows by taking the
  square root of both sides of the equation.
\end{proof}

\begin{proposition}{Triangle inequality}{inner-product-triangle-inequality}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space. Then
  \begin{equation*}
    \norm{\vect{u}+\vect{v}} \leq \norm{\vect{u}} + \norm{\vect{v}}.
  \end{equation*}
  This is called the \textbf{triangle inequality}%
  \index{triangle inequality!in inner product space}%
  \index{inner product space!triangle inequality}.
\end{proposition}

\begin{proof}
  The proof is essentially the same as that of
  Proposition~\ref{prop:triangle-inequality-dot-product}. We have
  \begin{eqnarray*}
    \norm{\vect{u}+\vect{v}}^2
    &=& \iprod{\vect{u}+\vect{v},\vect{u}+\vect{v}} \\
    &=& \iprod{\vect{u},\vect{u}}+\iprod{\vect{u},\vect{v}}+\iprod{\vect{v},\vect{u}}+\iprod{\vect{v},\vect{v}} \\
    &=&\norm{\vect{u}}^2+2\iprod{\vect{u},\vect{v}}+\norm{\vect{v}}^2 \\
    &\leq& \norm{\vect{u}}^2+2\abs{\iprod{\vect{u},\vect{v}}}+\norm{\vect{v}}^2 \\
    &\leq& \norm{\vect{u}}^2+2\norm{\vect{u}}\norm{\vect{v}}+\norm{\vect{v}}^2 \\
    &=& (\norm{\vect{u}}+\norm{\vect{v}})^2.
  \end{eqnarray*}
  The triangle inequality follows by taking square roots of both sides.
\end{proof}

Generalizing the situation in $\R^n$, we can define the angle
between any two vectors in an inner product space. Note that by the
Cauchy-Schwarz inequality,
\begin{equation*}
  \frac{\abs{\iprod{\vect{u},\vect{v}}}}{\norm{\vect{u}}\norm{\vect{v}}}
  \leq 1,
\end{equation*}
and therefore,
\begin{equation*}
  -1 \leq \frac{\iprod{\vect{u},\vect{v}}}{\norm{\vect{u}}\norm{\vect{v}}}
  \leq 1.
\end{equation*}
This ensures that the following definition is well-defined.

\begin{definition}{Angle between two vectors}{inner-product-angle}
  Let $\vect{u},\vect{v}$ be vectors in an inner product space.
  The \textbf{angle}%
  \index{angle!in inner product space}%
  \index{angle!between vectors} between $\vect{u}$ and
  $\vect{v}$ is, by definition, the unique $\theta$ such that
  $0\leq\theta\leq\pi$ and
  \begin{equation*}
    \cos\theta = \frac{\iprod{\vect{u},\vect{v}}}{\norm{\vect{u}}\norm{\vect{v}}}.
  \end{equation*}
\end{definition}

\begin{example}{Find the angle between two vectors}{inner-product-angle}
  Find the angle between $1$ and $x^2$ in $C[-1,1]$.
\end{example}

\begin{solution}
  We have
  \begin{eqnarray*}
    \iprod{1,1} &=& \int_{-1}^1 1\cdot 1\,dx ~=~ 2, \\
    \iprod{x^2,x^2} &=& \int_{-1}^1 x^2\cdot x^2\,dx ~=~ \frac{2}{5}, \\
    \iprod{1, x^2} &=& \int_{-1}^1 1\cdot x^2\,dx ~=~ \frac{2}{3}.
  \end{eqnarray*}
  Therefore
  \begin{equation*}
    \cos\theta
    ~=~ \frac{\iprod{1,x^2}}{\norm{1}\norm{x^2}}
    ~=~ \frac{\frac{2}{3}}{\sqrt{2}\sqrt{\frac{2}{5}}}
    ~=~ \frac{\sqrt{5}}{3}.
  \end{equation*}
  The angle $\theta$ is $\cos^{-1}(\frac{\sqrt{5}}{3})$, which is
  approximately $0.7297$ radians or $41.81$ degrees.

\end{solution}


\subsubsection{Magnetic Fields}
\noindent
Gauss's Law for magnetism says that that magnetic flux through a closed surface is 0. Rewritten more formally,
\begin{equation*}
	\oint\limits_{S}\oint{\vec{B} \cdot \mathrm{d}\vec{s}} = 0
\end{equation*}
So,
\begin{equation*}
	\iiint\limits_{V}{\nabla \cdot \vec{B}\mathrm{d}V} = \iiint\limits_{V}{0\mathrm{d}V}
\end{equation*}
If we let $V \to 0$,
\begin{equation*}
	\nabla \cdot \vec{B} = 0
\end{equation*}
This means that there are no lone sources or sinks in magnetic fields. You may also hear this law summarized as "There are no magnetic monopoles." This is the 2nd of Maxwell's Equations.


\chapter{Counting and Probability}

\section{Counting}

\begin{enumerate}

\item[C.1{-}1] {How many $k$-substrings does an $n$-string have? (Consider
identical $k$-substrings at different positions to be different.) How many
substrings does an $n$-string have in total?}

\begin{framed}
For every position $i$ of the $n$-string, $i = 1, \dots, n - k + 1$, there is
one $k$-substring the starts at $i$ and ends at $i + k - 1$. Thus, the number of
$k$-substrings in a $n$-string is
\[
  \sum_{i = 1}^{n - k + 1} 1 = n - k + 1.
\]

Thus, the number of substrings (of all sizes) in an $n$-string is
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{n} n - k + 1 &= n^2 + n - \sum_{k = 1}^{n} {k}\\
                             &= n^2 + n - \frac{n (n + 1)}{2}\\
                             &= n (n + 1) - \frac{n (n + 1)}{2}\\
                             &= \frac{n (n + 1)}{2}.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.1{-}2] {An $n$-input, $m$-output \textbf{\emph{boolean function}} is
a function from $\{\texttt{TRUE}, \texttt{FALSE}\}^n$ to
$\{\texttt{TRUE}, \texttt{FALSE}\}^m$. How many $n$-input, $1$-output boolean
functions are there? How many $n$-input, $m$-output boolean functions are
there?}

\begin{framed}
We can view the number of possible inputs of size $n$ as the number of binary
$n$-strings, which is $2^n$.

Now, consider a single-valued function from $\{\texttt{TRUE},
\texttt{FALSE}\}^n$ to $\{\texttt{TRUE}\}$. In this case, the number of
possible functions is the number of possible inputs, which is $2^n$. Since an
$1$-output boolean function has two possible output values, each of the $2^n$
functions we referred in the case of a single-valued function now has two ways
to pick the output value. We can view this number as the number of binary
$2^n$-strings, which is $2^{2^n}$. As for an $n$-output function, each of the
$2^n$ functions we referred in the case of a single-valued function now has
$2^m$ ways to pick the output value. Thus, there are $({2^m})^{2^n}$ of those.
\end{framed}

\item[C.1{-}3] {In how many ways can $n$ professors sit around a circular
conference table? Consider two seatings to be the same if one can be rotated to
form the other.}

\begin{framed}
For two seatings to be different from each other, the ordering of professors in
each seating needs to be different. This number can be viewed as the number
of permutations of a set $n$ elements, which is $n!$. However, note that for
each permutation that starts with professor $k$, $1 \le k \le n$, there are
$n - 1$ other permutations that are just a rotation of it. For instance, the
seatings $\{2, 3, 1\}$ and $\{3, 1, 2\}$ are a rotation of $\{1, 2, 3\}$. Thus,
the number of different seatings can be viewed as fixing the seat of the first
professor and computing the number of permutations of the remaining $n - 1$
professors, which is $(n - 1)!$.
\end{framed}

\item[C.1{-}4] {In how many ways can we choose three distinct numbers from the
set $\{1, 2, \dots, 99\}$ so that their sum is even?}

\begin{framed}
The set has $50$ odd numbers and $49$ even numbers. For the sum be even, we have
to choose three even numbers or one even and two odds. For the case with three
even numbers, there are $49!/(3! \cdot (49 - 3)!) = 18424$ ways of choosing $3$
distincts numbers among the 49 even numbers. As for the case with one even and
two odds, there are 49 ways to choose one even number and
$50!/(2! \cdot (50 - 2)!) = 1225$ ways of choosing $2$ distincts numbers among the
50 odd numbers. Thus, there are $18424 + 49 \cdot 1225 = 78449$ ways to get an
even sum.
\end{framed}

\newpage

\item[C.1{-}5] {Prove the identity
\[
  \binom{n}{k} = \frac{n}{k} \binom{n - 1}{k - 1}
\]
for $0 < k \le n$.
}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \binom{n}{k} &= \frac{n!}{k! \cdot (n - k)!}\\
               &= \frac{n \cdot (n - 1)!}{k \cdot (k - 1)! \cdot (n - k)!}\\
               &= \frac{n}{k} \frac{(n - 1)!}{(k - 1)! \cdot ((n - 1) - (k - 1))!}\\
               &= \frac{n}{k} \binom{n - 1}{k - 1}.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.1{-}6] {Prove the identity
\[
  \binom{n}{k} = \frac{n}{n - k} \binom{n - 1}{k}
\]
for $0 \le k < n$.
}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \binom{n}{k} &= \frac{n!}{k! \cdot (n - k)!}\\
               &= \frac{n \cdot (n - 1)!}{k! \cdot (n - k) \cdot (n - k - 1)!}\\
               &= \frac{n}{n - k} \frac{(n - 1)!}{k! \cdot ((n - 1) - k)!}\\
               &= \frac{n}{n - k} \binom{n - 1}{k}.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.1{-}7] {To choose $k$ objects from $n$, you can make one of the objects
distinguished and consider whether the distinguished object is chosen. Use this
approach to prove that
\[
  \binom{n}{k} = \binom{n - 1}{k} + \binom{n - 1}{k - 1}.
\]
}

\begin{framed}
Let $S = \{s_1, s_2, \dots, s_{n - 1}\}$ and $s_0$ the
distinguished element. To choose $k$ from the $n$ elements, we have to consider
two cases:
\begin{enumerate}
  \item If $s_0$ is selected, it will be necessary to choose the $k - 1$
    remaining elements from $S$. There are $\binom{n - 1}{k - 1}$ combinations.
  \item If $s_0$ is not selected, it will be necessary to choose the $k$
    remaining elements from $S$. There are $\binom{n - 1}{k}$ combinations.
\end{enumerate}
Adding the above together, we have
\begin{equation*}
\begin{aligned}
  \binom{n - 1}{k - 1} + \binom{n - 1}{k}
  &= \frac{(n - 1)!}{(k - 1)! \cdot (n - k)!} + \frac{(n - 1)!}{k! \cdot (n - k - 1)!}\\
  &= \frac{k \cdot (n - 1)!}{k! \cdot (n - k)!} + \frac{(n - k) \cdot (n - 1)!}{k! \cdot (n - k)!}\\
  &= \frac{(k + n - k) \cdot (n - 1)!}{k! \cdot (n - k)!}\\
  &= \frac{n!}{k! \cdot (n - k)!}\\
  &= \binom{n}{k}.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.1{-}8] {Using the result of Exercise C.1-7, make a table for
$n = 0, 1, \dots, 6$ and $0 \le k \le n$ of the binomial coefficients
$\binom{n}{k}$ with $\binom{0}{0}$ at the top, $\binom{1}{0}$ and $\binom{1}{1}$
on the next line, and so forth. Such a table of binomial coefficients is called
\textbf{\emph{Pascal's triangle}}.}

\begin{framed}
The table with binomials
\begin{gather*}
  \binom{0}{0}\\
  \binom{1}{0} \quad \binom{1}{1}\\
  \binom{2}{0} \quad \binom{2}{1} \quad \binom{2}{2}\\
  \binom{3}{0} \quad \binom{3}{1} \quad \binom{3}{2} \quad \binom{3}{3}\\
  \binom{4}{0} \quad \binom{4}{1} \quad \binom{4}{2} \quad \binom{4}{3} \quad \binom{4}{4}\\
  \binom{5}{0} \quad \binom{5}{1} \quad \binom{5}{2} \quad \binom{5}{3} \quad \binom{5}{4} \quad \binom{5}{5}\\
  \binom{6}{0} \quad \binom{6}{1} \quad \binom{6}{2} \quad \binom{6}{3} \quad \binom{6}{4} \quad \binom{6}{5} \quad \binom{6}{6}
\end{gather*}
Using the above table and the result of C.1-7, we have the Pascal's triangle
\begin{gather*}
  1\\
  1 \quad 1\\
  1 \quad 2 \quad 1\\
  1 \quad 3 \quad 3 \quad 1\\
  1 \quad 4 \quad 6 \quad 4 \quad 1\\
  1 \quad 5 \quad 10 \quad 10 \quad 5 \quad 1\\
  1 \quad 6 \quad 15 \quad 20 \quad 15 \quad 6 \quad 1
\end{gather*}
\end{framed}

\item[C.1{-}9] {Prove that
\[
  \sum_{i = 1}^{n} i = \binom{n + 1}{2}.
\]
}

\begin{framed}
We have
\begin{equation*}
\begin{aligned}
  \binom{n + 1}{2} &= \frac{(n + 1)!}{2! \cdot ((n + 1) - 2)!}\\
                   &= \frac{(n + 1) \cdot n \cdot (n - 1)!}{2 \cdot (n - 1)!}\\
                   &= \frac{n (n + 1)}{2}\\
                   &= \sum_{i = 1}^{n} i,
\end{aligned}
\end{equation*}
which also shows that the third Pascal's diagonal has the triangular numbers.
\end{framed}

\newpage

\item[C.1{-}10] {Show that for any integers $n \ge 0$ and $0 \le k \le n$, the
expression $\binom{n}{k}$ achieves its maximum value when $k = \floor{n/2}$ or
$k = \ceil{n/2}$.}

\begin{framed}
It follows from the Pascal's triangle
\begin{gather*}
  1\\
  1 \quad 1\\
  1 \quad 2 \quad 1\\
  1 \quad 3 \quad 3 \quad 1\\
  1 \quad 4 \quad 6 \quad 4 \quad 1\\
  1 \quad 5 \quad 10 \quad 10 \quad 5 \quad 1\\
  1 \quad 6 \quad 15 \quad 20 \quad 15 \quad 6 \quad 1\\
  \quad \quad \vdots \quad \quad
\end{gather*}

We can prove by induction. The base case, which occurs when $n = 0$,
holds since
\[
  \binom{n}{\floor{n/2}} = \binom{n}{\ceil{n/2}} = \binom{0}{0} = 1
\]
is maximum on row 0.  Now, assume it holds for $n$. Then, if $n + 1$ is even,
from Equation (C.3) we have
\begin{equation*}
\begin{aligned}
  \binom{n + 1}{\floor{\frac{n + 1}{2}}} = \binom{n + 1}{\ceil{\frac{n + 1}{2}}}
  &= \binom{n}{\left(\frac{n + 1}{2} - 1\right)} + \binom{n}{\left(\frac{n + 1}{2}\right)}\\
  &= \binom{n}{\left(\frac{n}{2} - \frac{1}{2}\right)} + \binom{n}{\left(\frac{n}{2} + \frac{1}{2}\right)} & \text{(since $n$ is odd)}\\
  &= \binom{n}{\floor{\frac{n}{2}}} + \binom{n}{\ceil{\frac{n}{2}}},
\end{aligned}
\end{equation*}
which shows that is also holds for $n + 1$ since
\[
  \binom{n}{\floor{\frac{n}{2}}} \text{ and } \binom{n}{\ceil{\frac{n}{2}}}
\]
are both maximum on row $n$. The proof is similar when $n + 1$ is odd.
\end{framed}

\newpage

\item[C.1{-}11] {($\star$) Argue that for any integers $n \ge 0$, $j \ge 0$,
$k \ge 0$, and $j + k \le n$,
\[
  \binom{n}{j + k} \le \binom{n}{j} \binom{n - j}{k}.
\]

Provide both an algebraic proof and an argument based on a method for choosing
$j + k$ items out of $n$. Give an example in which equality does not hold.
}

\begin{framed}
For any integers $a \ge 0, b \ge 0,$ and $a \ge b$, we have
\begin{equation*}
\begin{aligned}
  (a + b)! &=   \underbrace{(a + b) \cdot (a + b - 1) \cdot (a + b - 2) \cdots}_\text{$b$ times} a!\\
           &\ge \underbrace{b \cdot (b - 1) \cdot (b - 2) \cdots}_\text{$b$ times} a!\\
           &=   a! \cdot b!.
\end{aligned}
\end{equation*}

Using the above result, we have
\begin{equation*}
\begin{aligned}
  \binom{n}{j} \binom{n - j}{k} &=   \frac{n!}{j! \cdot (n - j)!} \frac{(n - j)!}{k! \cdot ((n - j) - k)!}\\
                                &=   \frac{n!}{j! \cdot k! \cdot ((n - j) - k)!}\\
                                &\ge \frac{n!}{(j + k)! \cdot (n - (j + k))!}\\
                                &=   \binom{n}{j + k}.
\end{aligned}
\end{equation*}

The expression on the left is the number of ways to choose an $(j + k)$-subset
of an $n$-set (which leaves the reamining $n - (j + k)$ elements). Thus, it is
a partition of the original $n$-set into subsets of cardinalities $(j + k)$ and
$n - (j + k)$. The right hand side has two factors: the first binomial
coefficient is the number of ways to choose a $j$-subset of an $n$-set (which
leaves the reamining $n - j$ elements); the second is the number of ways to
choose a $k$-subset from the remaining $n - j$ elements. Thus, it is a partition
of the original $n$-set into subsets of cardinalities $j, k$, and $n - (j + k)$.
Consider now that we choose the $n - (j + k)$ first, leaving behind the
remaining $j + k$ elements. There is precisely one way to choose an
$(j + k)$-subset out of the remaining $j + k$ elements. On the other hand, when
we first choose $j$ and then we choose $k$, if $j < j + k$, there are
\emph{at least} two ways to choose a $j$-subset from the $(j + k)$-subset and
precisely one way to choose a $k$-subset from the remaining $k$ elements. This
notion also applies to the algebraic proof, since
$(j + k)! = j! \cdot k \iff j = 0$ or $k = 0$.
Also note that while the left expression does not count any permutation of the
$(j + k)$-subsets (since it normalizes by $(j + k)!$), the right
expression, despite not counting permutations of each of the subsets
indepentently (since it normalizes by $j! \cdot k!$), it counts permutations of
two subsets together. For instance, let $A = \{a, b\}$. There is only one way to
choose 2 elements from $A$, which is $ab$. However, there are two ways to choose
one element and then another element from $A$, which are $ab$ and $ba$.

\end{framed}

\item[C.1{-}12] {($\star$) Use induction on all integers $k$ such that
$0 \le k \le n/2$ to prove inequality (C.6), and use equation (C.3) to extend it
to all integers $k$ such that $0 \le k \le n$.}

\begin{framed}
Skipped.
\end{framed}

\item[C.1{-}13] {($\star$) Use Stirling's approximation to prove that
\[
  \binom{2n}{n} = \frac{2^{2n}}{\sqrt{\pi n}} (1 + O(1/n)).
\]
}

\begin{framed}
Skipped.
% \begin{equation*}
% \begin{aligned}
%   \binom{2n}{n} &= \frac{(2n)!}{n! \cdot (2n - n)!}\\
%                 &= \frac{(2n)!}{(n!)^2}\\
%                 &= \frac{\sqrt{2 \pi 2n} \left(\frac{2n}{e}\right)^{2n} \left( 1 + \Theta\left(\frac{1}{2n}\right) \right)}
%                         {(\sqrt{2 \pi n} \left(\frac{n}{e}\right)^{n} \left( 1 + \Theta\left(\frac{1}{n}\right) \right))^2}\\
%                 &= \frac{\sqrt{2 \pi 2n} \left(\frac{2n}{e}\right)^{2n} \left( 1 + O\left(\frac{1}{n}\right) \right)}
%                         {2 \pi n \left(\frac{n}{e}\right)^{2n} \left( 1 + O\left(\frac{1}{n}\right) \right)^2}
% \end{aligned}
% \end{equation*}
\end{framed}

\item[C.1{-}14] {($\star$) By differentiating the entropy function $H(\lambda)$,
show that it achieves its maximum value at $\lambda = 1/2$. What is $H(1/2)$?}

\begin{framed}
Skipped.
\end{framed}

\item[C.1{-}15] {($\star$) Show that for any integer $n \ge 0$,
\[
  \sum_{k = 0}^{n} \binom{n}{k} k = n 2^{n - 1}.
\]
}

\begin{framed}
Skipped.
% \begin{equation*}
% \begin{aligned}
%   \sum_{k = 0}^{n} \binom{n}{k} k
%   &= \sum_{k = 1}^{n} \binom{n}{k} k\\
%   &= \sum_{k = 1}^{n} \frac{n!}{k! \cdot (n - k)!}\\
%   &= \sum_{k = 1}^{n} \frac{n \cdot (n - 1)!}{k \cdot (k - 1)! (n - k) \cdot (n - k - 1)!}
% \end{aligned}
% \end{equation*}
\end{framed}

\end{enumerate}

\newpage

\section{Probability}

\begin{enumerate}

\item[C.2{-}1] {Professor Rosencrantz flips a fair coin once. Professor
Guildenstern flips a fair coin twice. What is the probability that Professor
Rosencrantz obtains more heads than Professor Guildenstern?}

\begin{framed}
The sample space $\{\text{H}, \text{T}\}^3$ has size $2^3 = 8$. Since the only
event that satisfies the condition is $\{\text{HTT}\}$, the probability is $1/8$.
\end{framed}

\item[C.2{-}2] {Prove the \textbf{\emph{Boole's inequality}}: For any finite or
countably infinite sequence of events $A_1, A_2, \dots$,
\[
  \text{Pr}\{A_1 \cup A_2 \cup \cdots\} \le \text{Pr}\{A_1\} + \text{Pr}\{A_2\} + \cdots.
\]
}

\begin{framed}
From (C.13) we have
\[
\text{Pr}\{A_1 \cup A_2\} \le \text{Pr}\{A_1\} + \text{Pr}\{A_2\},
\]
which implies
\begin{equation*}
\begin{aligned}
  \text{Pr}\{A_1 \cup A_2 \cup \cdots\} &=   \text{Pr}\{A_1 \cup (A_2 \cup \cdots)\}\\
                                        &\le \text{Pr}\{A_1\} + \text{Pr}\{A_2 \cup (A_3 \cup \cdots)\}\\
                                        &\le \text{Pr}\{A_1\} + \text{Pr}\{A_2\} + \text{Pr}\{A_3 \cup (A_4 \cup \cdots)\}\\
                                        &\le \text{Pr}\{A_1\} + \text{Pr}\{A_2\} + \text{Pr}\{A_3\} \cdots.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.2{-}3] {Suppose we shuffle a deck of 10 cards, each bearing a distinct
number from 1 to 10, to mix the cards thoroughly. We then remove three cards,
one at a time, from the deck. What is the probability that we select the three
cards in sorted (increasing) order?}

\begin{framed}
Let $a < b < c$ denote the number of the three selected cards. There are $3!$
permutations of $\{a, b, c\}$ and $abc$ is the only one which is in sorted
order. Thus, the probability is $1/3! = 1/6$.
\end{framed}

\item[C.2{-}4] {Prove that
\[
  \text{Pr}\{A\;|\;B\} + \text{Pr}\{\overline{A}\;|\;B\} = 1.
\]
}

\begin{framed}
We have
\begin{equation*}
\begin{aligned}
  \text{Pr}\{B\} &= \text{Pr}\{(B \cap A) \cup (B \cap \overline{A})\}\\
                 &= \text{Pr}\{B \cap A\} + \text{Pr}\{B \cap \overline{A}\}\\
                 &= \text{Pr}\{A\} \text{Pr}\{B\;|\;A\} + \text{Pr}\{\overline{A}\} \text{Pr}\{B\;|\;\overline{A}\}.
\end{aligned}
\end{equation*}
Substituting into (C.17) yields
\begin{equation*}
\begin{aligned}
  \text{Pr}\{A\;|\;B\} + \text{Pr}\{\overline{A}\;|\;B\}
  &= \frac{\text{Pr}\{A\} \text{Pr}\{B\;|\;A\}}{\text{Pr}\{B\}} +
     \frac{\text{Pr}\{\overline{A}\} \text{Pr}\{B\;|\;\overline{A}\}}{\text{Pr}\{B\}}\\
  &= \frac{\text{Pr}\{A\} \text{Pr}\{B\;|\;A\} + \text{Pr}\{\overline{A}\} \text{Pr}\{B\;|\;\overline{A}\}}
          {\text{Pr}\{B\}}\\
  &= \frac{\text{Pr}\{A\} \text{Pr}\{B\;|\;A\} + \text{Pr}\{\overline{A}\} \text{Pr}\{B\;|\;\overline{A}\}}
          {\text{Pr}\{A\} \text{Pr}\{B\;|\;A\} + \text{Pr}\{\overline{A}\} \text{Pr}\{B\;|\;\overline{A}\}}\\
  &= 1.
\end{aligned}
\end{equation*}

\end{framed}

\item[C.2{-}5] {Prove that for any collection of events $A_1, A_2, \dots, A_n$,
\[
  \text{Pr}\{A_1 \cap A_2 \cap \cdots \cap A_n\} =
    \text{Pr}\{A_1\} \cdot \text{Pr}\{A_2\;|\;A_1\} \cdot \text{Pr}\{A_3\;|\;A_1 \cap A_2\} \cdots
    \text{Pr}\{A_n\;|\;A_1 \cap A_2 \cap \cdots \cap A_{n - 1}\}.
\]
}

\begin{framed}
It is trivially valid for $n = 1$. As our base case, consider $n = 2$. From (C.16) we have
\[
  \text{Pr}\{A_1 \cap A_2\} = \text{Pr}\{A_1\} \text{Pr}\{A_2\;|\;A_1\}.
\]
Now assume it holds for $n$. For $n + 1$, we have
\begin{equation*}
\begin{aligned}
  \text{Pr}\{A_1 \cap A_2 \cap \cdots \cap A_{n + 1}\}
  &= \text{Pr}\{(A_1 \cap A_2 \cap \cdots \cap A_n) \cap A_{n + 1}\}\\
  &= \text{Pr}\{A_1 \cap A_2 \cap \cdots \cap A_n\} \text{Pr}\{A_{n + 1}\;|\;A_1 \cap A_2 \cap \cdots \cap A_n\}\\
  &= \text{Pr}\{A_1\} \cdot \text{Pr}\{A_2\;|\;A_1\} \cdot \text{Pr}\{A_3\;|\;A_1 \cap A_2\} \cdots
     \text{Pr}\{A_{n + 1}\;|\;A_1 \cap A_2 \cap \cdots \cap A_{n}\}.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.2{-}6] {($\star$) Describe a procedure that takes as input two integers
  $a$ and $b$ such that $0 < a < b$ and, using fair coin flips, produces as
  output heads with probability $a/b$ and tails with probability $(b - a)/b$.
  Give a bound on the expected number of coin flips, which should be $O(1)$.
  (Hint: Represent $a/b$ in binary.)}

\begin{framed}
Consider a continuous uniform probability distribution on
$\interval[open right]{0}{1}$, such that $\text{Pr}\{\interval[open right]{0}{1}\} = 1$. We have
\[
  \text{Pr}\left\{\interval[open right, scaled]{0}{\frac{a}{b}}\right\} = \frac{a}{b},
\]
and
\[
  \text{Pr}\left\{\interval[open right, scaled]{\frac{a}{b}}{1}\right\} = 1 - \frac{a}{b} = \frac{b - a}{b}.
\]

With this notion, we can write a procedure that sorts a real number from
$\interval[open right]{0}{1}$ and return heads if it is lower than $a/b$ or
return tails, otherwise. Using fair coin flips and representing numbers in
binary, for each flip we have a new decimal place from a random number on
$\interval[open right]{0}{1}$ (consider an ``0'' if the coin flip is head and
``1'', otherwise). Then,
\begin{itemize}
\item if the $i$-th flip is 1 and the $i$-th decimal place
of $a/b$ is 0, the sorted number is larger than $a/b$ and we return tails;
\item if the $i$-th flip is 0 and the $i$-th decimal place
of $a/b$ is 1, the sorted number is smaller than $a/b$ and we return head;
\item if the $i$-th flip and the $i$-th decimal place are equal, we sort a new
decimal place.
\end{itemize}

Since we do not know how many decimal places $a/b$ has (if periodic, this number
is infinite), the above procedure does not have a maximum number of iterations.
However, since for each flip we have a probability of 1/2 of returning head or
tails, the probability of terminating at flip $i$, for $i \ge 1$, is
\[
  \underbrace{1/2 \cdot 1/2 \cdots}_\text{$i$ times} = \frac{1}{2^i}.
\]
Thus, by using the notion of expected value and the result (A.8), the expected number of
flips is
\[
  \sum_{i = 1}^{\infty} i \cdot \frac{1}{2^i}
  = \sum_{i = 0}^{\infty} i \cdot \left(\frac{1}{2}\right)^{i}
  = \frac{1/2}{(1 - 1/2)^2} = 2.
\]

\end{framed}

\item[C.2{-}7] {($\star$) Show how to construct a set of $n$ events that are
  paiwise independent but such that no subset of $k > 2$ of them is mutually
  independent.}

\begin{framed}
Skipped.
\end{framed}

\item[C.2{-}8] {($\star$) Two events $A$ and $B$ are \textbf{\emph{conditionally
  independent}}, given $C$, if
  \[
    \text{Pr}\{A \cap B\;|\;C\} = \text{Pr}\{A\;|\;C\} \cdot \text{Pr}\{B\;|\;C\}.
  \]
  Give a simple but nontrivial example of two events that are not independent
  but are conditionally independent given a third event.
}

\begin{framed}
Skipped.
\end{framed}

\newpage

\item[C.2{-}9] {($\star$) You are a contestant in a game show in which a prize
  is hidden behind one of three curtains. You will win the prize if you select
  the correct curtain. After you have picked one curtain but before the curtain
  is lifted, the emcee lifts one of the other curtains, knowing that it will
  reveal an empty stage, and asks if you would like to switch from your current
  selection to the remaining curtain. How would your chances change if you
  switch? (This question is the celebrated \textbf{\emph{Monty Hall problem}},
  named after a game-show host who often presented contestants with just this
  dilemma.)}

\begin{framed}
If you never switch, the only way to win is to choose the right curtain at the
beginning (before the emcee lifts one of the others). In this case, your chance
to win are $1/3$. If you always switch, the only way to loose is to choose the
right curtain at the beginning. In this case, when you choose a curtain without
the prize, the emcee will reveal the other empty curtain and you will
therefore change to the correct one. Thus, your chance to win are $(1
- 1/3) = 2/3$.
\end{framed}

\item[C.2{-}10] {($\star$) A prison warden has randomly picked one prisoner
  among three to go free. The other two will be executed. The guard knows which
  one will go free but is forbidden to give any prisoner information regarding
  his status. Let us call the prisoners $X, Y$, and $Z$. Prisoner $X$ asks the
  guard privately which of $Y$ or $Z$ will be executed, arguing that since he
  already knows that at least one of them must die, the guard won't be revealing
  any information about his own status. The guard tells $X$ that $Y$ is to be
  executed.  Prisoner $X$ feels happier now, since he figures that either he or
  prisoner $Z$ will go free, which means that his probability of going free is
  now $1/2$. Is he right, or are his chances still $1/3$? Explain.}
\begin{framed}
  His chances are still $1/3$. Let $A$ be the event of prisoner $X$ going free
  and $B$ the event that the guard tells $X$ that $Y$ is to be executed. We have
  \[
    \text{Pr}(A\;|\;B) = \frac{\text{Pr}(A) \text{Pr}(B\;|\;A)}{\text{Pr}(B)} = \frac{1/3 \cdot 1/2}{1/2} = \frac{1}{3}.
  \]
\end{framed}

\end{enumerate}

\newpage

\section{Discrete random variables}

\begin{enumerate}

\item[C.3{-}1]{Suppose we roll two ordinary, 6-sided dice. What is the
expectation of the sum of the two values showing? What is the expectation of
the maximum of the two values showing?}

\begin{framed}
There are $36$ elementary events in the sample space. Since they are
ordinary dices, the probability distribution is uniform.

Let $X$ be the random variable of the sum of the two values. The possible
outcomes of $X$ are
\begin{center}
\begin{tabular}{rrrrrrr}
  & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6}\\
  \textbf{1} & 2 & 3 & 4 & 5  & 6  & 7\\
  \textbf{2} & 3 & 4 & 5 & 6  & 7  & 8\\
  \textbf{3} & 4 & 5 & 6 & 7  & 8  & 9\\
  \textbf{4} & 5 & 6 & 7 & 8  & 9  & 10\\
  \textbf{5} & 6 & 7 & 8 & 9  & 10 & 11\\
  \textbf{6} & 7 & 8 & 9 & 10 & 11 & 12
\end{tabular}
\end{center}

Thus, we have
\begin{equation*}
\begin{aligned}
  \text{E}[X] &= \sum_{x = 2}^{12} x \cdot \text{Pr}(X = x)\\
              &= 2 \cdot \frac{1}{36} + 3 \cdot \frac{2}{36} + 4 \cdot \frac{3}{36} + 5 \cdot \frac{4}{36}
               + 6 \cdot \frac{5}{36} + 7 \cdot \frac{6}{36} + 8 \cdot \frac{5}{36} + 9 \cdot \frac{4}{36}
               + 10 \cdot \frac{3}{36} + 11 \cdot \frac{2}{36} + 12 \cdot \frac{1}{36}\\
              &= 7.
\end{aligned}
\end{equation*}

Let $Y$ be the random variable of the maximum of the two values. The possible
outcomes of $Y$ are

\begin{center}
\begin{tabular}{rrrrrrr}
  & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6}\\
  \textbf{1} & 1 & 2 & 3 & 4 & 5 & 6\\
  \textbf{2} & 2 & 2 & 3 & 4 & 5 & 6\\
  \textbf{3} & 3 & 3 & 3 & 4 & 5 & 6\\
  \textbf{4} & 4 & 4 & 4 & 4 & 5 & 6\\
  \textbf{5} & 5 & 5 & 5 & 5 & 5 & 6\\
  \textbf{6} & 6 & 6 & 6 & 6 & 6 & 6
\end{tabular}
\end{center}

Thus, we have
\begin{equation*}
\begin{aligned}
  \text{E}[Y] &= \sum_{x = 1}^{6} x \cdot \text{Pr}(X = x)\\
              &= 1 \cdot \frac{1}{36} + 2 \cdot \frac{3}{36} + 3 \cdot \frac{5}{36} + 4 \cdot \frac{7}{36}
               + 5 \cdot \frac{9}{36} + 6 \cdot \frac{11}{36}\\
              &\approx 4.47.
\end{aligned}
\end{equation*}

\end{framed}

\item[C.3{-}2]{An array $A[1 \dots n]$ contains $n$ distinct numbers that are
randomly ordered, with each permutation of the $n$ numbers being equally
likely. what is the expectation of the index of the maximum element in the
array? What is the expectation of the index of the minimum element in the
array?}

\begin{framed}
Let $X$ and $Y$ be random variables of the index of the maximum and minimum
elements, respectivelly. Since each permutation is equaly likely,
\[
  \text{E}[X] = \text{E}[Y] = \sum_{i = 1}^{n} i \cdot \frac{1}{n} = \frac{1}{n} \frac{n (n + 1)}{2} = \frac{n + 1}{2}.
\]
\end{framed}

\newpage

\item[C.3{-}3]{A carnival game consists of three dice in a cage. A player can
bet a dollar on any of the numbers $1$ through $6$. The cage is shaken, and
the payoff is as follows. If the player's number doesn't appear on any of
the dice, he loses his dollar. Otherwise, if his number appears on exactly
$k$ of the three dice, for $k = 1, 2, 3$, he keeps his dollar and wins $k$
more dollars. What is his expected gain from playing the carnival game
once?}

\begin{framed}
Let $X$ be a random variable of the total gain. The possible outcomes are
$-1, 1, 2, 3$. We have
\begin{equation*}
\begin{aligned}
  & \text{Pr}\{X = -1\} &&= (5/6 \cdot 5/6 \cdot 5/6)         &&= 125/216,\\
  & \text{Pr}\{X = 1\}  &&= (1/6 \cdot 5/6 \cdot 5/6) \cdot 3 &&= 75/216,\\
  & \text{Pr}\{X = 2\}  &&= (1/6 \cdot 1/6 \cdot 5/6) \cdot 3 &&= 15/216,\\
  & \text{Pr}\{X = 3\}  &&= (1/6 \cdot 1/6 \cdot 1/6)         &&= 1/216.
\end{aligned}
\end{equation*}
Thus, we have
\[
  \text{E}[X] = -1 \cdot \frac{125}{216} + 1 \cdot \frac{75}{216} + 2 \cdot \frac{15}{216} + 3 \cdot \frac{1}{216} \approx -0.0787.
\]
\end{framed}

\item[C.3{-}4]{Argue that if $X$ and $Y$ are nonnegative random variables, then
\[
  \text{E}[\max(X, Y)] \le \text{E}[X] + \text{E}[Y].
\]
}

\begin{framed}
The expectation of nonnegative random variables is a summation of nonnegative
numbers. Thus, since $\text{E}[\max(X, Y)]$ is either $\text{E}[X]$ or
$\text{E}[Y]$, it must be equal or lower than $\text{E}[X] + \text{E}[Y]$.
\end{framed}

\item[C.3{-}5]{($\star$) Let $X$ and $Y$ be independent random variables. Prove
that $f(X)$ and $g(Y)$ are independent for any functions $f$ and $g$.}

\begin{framed}
Skipped.

% \begin{equation*}
% \begin{aligned}
%   \text{Pr}\{f(X) = f(x) \text{ and } g(Y) = g(y)\}
%   &= \text{Pr}\{X = x \text{ and } Y = y\}\\
%   &= \text{Pr}\{X = x\} \text{Pr}\{Y = y\}\\
%   &= \text{Pr}\{f(X) = f(x)\} \text{Pr}\{g(Y) = g(y)\}.
% \end{aligned}
% \end{equation*}

% Since $X$ and $Y$ are (discrete) independent random variables, we have
% \begin{equation*}
% \begin{aligned}
% \sum_{x} \sum_{y} \text{Pr}\{X = x \text{ and } Y = y\}
% &= \sum_{x} \sum_{y} \text{Pr}\{X = x\} \text{Pr}\{Y = y\}\\
% &= \sum_{x} \text{Pr}\{X = x\} \sum_{y} \text{Pr}\{Y = y\}.
% \end{aligned}
% \end{equation*}
\end{framed}

\item[C.3{-}6]{($\star$) Let $X$ be a nonnegative random variable, and suppose
that $\text{E}[X]$ is well defined. Prove \textbf{\emph{Markov's inequality}}:
\[
  \text{Pr}\{X \ge t\} \le \text{E}[X]/t
\]
for all $t > 0$.
}

\begin{framed}
We have
\begin{equation*}
\begin{aligned}
  \text{E}[X] &=   \sum_{x}{x \cdot \text{Pr}\{X = x\}}\\
              &\ge \sum_{x \ge t}{x \cdot \text{Pr}\{X = x\}}\\
              &\ge \sum_{x \ge t}{t \cdot \text{Pr}\{X = x\}}\\
              &= t \cdot \sum_{x \ge t}{\text{Pr}\{X = x\}}\\
              &= t \cdot \text{Pr}\{X \ge t\},
\end{aligned}
\end{equation*}
which implies
\[
  \text{Pr}\{X \ge t\} \le \text{E}[X]/t.
\]
\end{framed}

\newpage

\item[C.3{-}7]{($\star$) Let $S$ be a sample space, and let $X$ and $X'$ be
random variables such that $X(s) \ge X'(s)$ for all $s \in S$. Prove that for
any real constant $t$,
\[
  \text{Pr}\{X \ge t\} \ge \text{Pr}\{X' \ge t\}.
\]
}

\begin{framed}
Assuming that the domain of $X$ and $X'$ are the sample space $S$, we have
\begin{equation*}
\begin{aligned}
  \text{Pr}\{X \ge t\} &=   \sum_{s \in S:X(s) \ge t}{\text{Pr}\{X = s\}}\\
                       &=   \sum_{s \in S:X'(s) \ge t}{\text{Pr}\{X' = s\}} + \sum_{s \in S:X(s) \ge t > X'(s)}{\text{Pr}\{X' = s\}}\\
                       &\ge \sum_{s \in S:X'(s) \ge t}{\text{Pr}\{X' = s\}}\\
                       &=   \text{Pr}\{X' \ge t\}.
\end{aligned}
\end{equation*}
\end{framed}

\item[C.3{-}8]{Which is larger: the expectation of the square of a random
variable, or the square of its expectation?}

\begin{framed}
We have from (C.28)
\[
  \text{E}[X^2] = \text{Var}[X] + \text{E}^2[X],
\]
which implies
\[
  \text{E}[X^2] \ge \text{E}^2[X],
\]
since both $\text{Var}[X]$ and $\text{E}^2[X]$ are nonnegative numbers.

\end{framed}

\item[C.3{-}9]{Show that for any random variable X that takes on only the values
0 and 1, we have
\[
  \text{Var}[X] = \text{E}[X] \text{E}[1 - X].
\]
}

\begin{framed}
We have
\[
  \text{E}[X] = 0 \cdot \text{Pr}\{X = 0\} + 1 \cdot \text{Pr}\{X = 1\} = \text{Pr}\{X = 1\},
\]
and
\[
  \text{E}[1 - X] = 1 \cdot \text{Pr}\{X = 0\} + 0 \cdot \text{Pr}\{X = 1\} = \text{Pr}\{X = 0\},
\]
which implies
\begin{equation*}
\begin{aligned}
  \text{Var}[X] &= \text{E}[X^2] - \text{E}^2[X] & \text{(since $X^2 = X$)}\\
                &= \text{E}[X] - \text{E}[X] \text{E}[X]\\
                &= \text{E}[X] (1 - \text{E}[X])\\
                &= \text{E}[X] (1 - \text{Pr}\{X = 1\})\\
                &= \text{E}[X] \text{Pr}\{X = 0\}\\
                &= \text{E}[X] \text{E}[1 - X].
\end{aligned}
\end{equation*}
\end{framed}

\item[C.3{-}10]{Prove that $\text{Var}[aX] = a^2 \text{Var}[X]$ from the
definition (C.27) of variance.}

\begin{framed}
Assuming that $X$ is a random variable and $a$ is a constant, from (C.27) and
(C.22) we have
\begin{equation*}
\begin{aligned}
  \text{Var}[a X] &= \text{E}[(a X - \text{E}[a X])^2]\\
                  &= \text{E}[(a X - a \text{E}[X])^2]\\
                  &= \text{E}[a^2 (X - \text{E}[X])^2]\\
                  &= a^2 \text{E}[(X - \text{E}[X])^2]\\
                  &= a^2 \text{Var}[X].
\end{aligned}
\end{equation*}
\end{framed}

\end{enumerate}


\section{Some Counting and Stirling's Formula}
\subsection{Some Counting}
If we have a finite set $\Omega$, and let $M$ be the number of ways of partition $\Omega$ into $k$ subsets $S_1,S_2,\ldots,S_k$ with $|S_j|=n_j$ with $n_1+n_2+\cdots +n_k=|\Omega|$, then
$$M=\binom{|\Omega|}{n_1,n_2,\ldots,n_k}=\frac{|\Omega|}{n_1!n_2!\cdots n_k!}$$
We suddenly want to count the number of strictly increasing and nondecreasing functions from the set $\{1,2,\ldots,k\}$ to $\{1,2,\ldots,n\}$.
Note that each strictly increasing function in this way are uniquely identified by their image, so the number of such functions equals $\binom{n}{k}$.
\footnote{$\binom{n}{k}=0$ for $k>n$}
To count nondecreasing functions, however, we cannot use this trick.
Nonetheless, we can consider the bijection
$$\{f\text{ nondecreasing}:\{1,2,\ldots,k\}\to\{1,2,\ldots,n\}\}\to$$
$$\{f\text{ strictly increasing}:\{1,2,\ldots,k\}\to\{1,2,\ldots,n+k-1\}\}$$
by assigning a function $f$ in the previous set to the function $g(i)=f(i)+i-1$.
So that number is $\binom{n+k-1}{k}$.
\subsection{Stirling's Formula}
\begin{definition}
    Let $(a_n),(b_n)$ be two positive sequences, we say $a_n\sim b_n$ if $a_n/b_n\to 1$ as $n\to\infty$.
\end{definition}
\begin{theorem}[Stirling]
    $$n!\sim n^n\sqrt{2\pi n}e^{-n}$$
\end{theorem}
\begin{proposition}[Weaker Statement of Stirling's Formula]
    $\log(n!)\sim n\log n$
\end{proposition}
\begin{proof}
    Let $\ell_n=\log(n!)$.
    So we have
    $$\ell(n)=\log 2+\log 3+\cdots+\log n$$
    Note that we have the trivial bound $\log\lfloor x\rfloor\le \log x\le \log\lfloor x+1\rfloor$, hence
    $$\ell_{n-1}\le\int_1^n\log x\,\mathrm dx\le \ell_n$$
    So
    $$n\log(n)-n+1\le \ell_n\le (n+1)\log(n+1)-n$$
    The proposition follows.
\end{proof}
\begin{proof}[Proof of Stirling's Formula]
    Note that
    $$\int_a^bf(x)\,\mathrm dx=\frac{f(a)+f(b)}{2}(b-a)-\frac{1}{2}\int_a^b(x-a)(b-x)f^{\prime\prime}(x)\,\mathrm dx$$
    We take $f=\log$ we have
    $$\int_k^{k+1}\log x\,\mathrm dx=\frac{\log(k)+\log(k+1)}{2}+\frac{1}{2}\int_0^1\frac{x(1-x)}{(x+k)^2}\,\mathrm dx$$
    Summing over $k=1,2,\ldots,n-1$ we have
    $$n\log n-n+1=\log(n!)-\frac{\log n}{2}+\sum_{k=1}^{n-1}a_k$$
    where
    $$a_k=\frac{1}{2}\int_0^1\frac{x(1-x)}{(x+k)^2}\,\mathrm dx$$
    Note that it is easy to see the partial sum of $a_k$ converges, so we define $A=\exp(1-\sum_{k\in\mathbb N}a_k)$ to have
    $$n!=n^n\sqrt{n}e^{-n}A\exp\left(\sum_{k=n}^\infty a_k\right)$$
    Note that the last part goes to $1$ as $n\to\infty$, so it remains to show $A=\sqrt{2\pi}$.\\
    We claim that
    $$2^{-2n}\binom{2n}{n}\sim\frac{1}{\sqrt{\pi n}}$$
    since $2^{-2n}\binom{2n}{n}\sim\sqrt 2/(A\sqrt{n})$, it will prove what we want.
    Consider
    $$I_n=\int_0^{\pi/2}\cos^n\theta\,\mathrm d\theta$$
    It is trivial to see that it equals $I_{2n}=\binom{2n}{n}\pi/2^{2n+1}$ and $I_{2n+1}=2^{2n}\binom{2n}{n}^{-1}/(2n+1)$.
    We want to show that $I_{2n}/I_{2n+1}\to 1$ which will prove the result, but this is obvious since $I_{n+2}/I_n\to 1$ and $I_n$ is decreasing.
\end{proof}


\subsection{Absolute Cartesian Method}

\begin{frame}{Definition}
    \begin{itemize}
        \item Use a fixed reference frame which has $x,y,z$-axes (as opposed to $r,\theta,z$-axes or $\rho,\theta,z$-axes).
        \item In a planar kinematic chain, the position of every link is defined by:
        \begin{itemize}
            \item Joint positions
            \item Center of gravity $g$
            \item Angle $\theta$ with respect to $x$-axis
       \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}
For planar mechanisms, the following relations are considered:
	\begin{itemize}
	    \item[] $\displaystyle (x_A-x_B)^2+(y_A-y_B)^2=AB^2=l_{AB}^2$
	    \item[] $\displaystyle k=\tan{\theta}=\frac{y_B-y_A}{x_B-x_A}$
	    \item[] $\displaystyle\Rightarrow y=ax+b$
	\end{itemize}
where:
	\begin{itemize}
	    \item[-] $A(x_A,y_A)$, $B(x_B,y_B)$ are joint coordinates
	    \item[-] $l_{AB}$ is the length of link $AB$
	    \item[-] $\theta$ is the angle of link $AB$ with respect to $x$-axis
	    \item[-] $a,b$ are coefficients ($a$ is slope of link $AB$, $b$ is intercept)
	\end{itemize}
Solving these equations often yields 2 position coordinates. Depending on positions of other links, choose the correct solution.
\end{frame}


\chapter{Conclusion and Outlook}

\section{Summary}
We defined an algebraic graph structure that expresses the Collatz sequences in the form of a tree. Next, the vertex reachability properties were unveiled by examining the relationship between successive nodes in $H_C$. Moreover, we dealt with graphs that represent other variants of Collatz sequences, for instance $5x+1$ or $181x+1$. The interesting part of both variants is that for these sequences the existence of cycles is known. They serve as the basis for further investigations of the problem.

\section{Further Research}
In subsequent studies, the properties of vertices in $H_C$ might be elaborated upon more closely by taking into account a vertex's label as well as its properties. Moreover we will investigate on the structure of a pruned trees $T_{\ge j}$ including questions on calculating a left-child or right-child within any of these trees.


\subsection{The jackknife}

We have a statistic:

\(S(x_1, x_2,...,x_n)\)

We may want to estimate moments for this statistic, but are unable to do so.

\subsubsection{The jackknife estimator}

The jackknife is an approach for getting moments for statistics.

We start by creating \(n\) statistics each leaving out one observation.

\(\bar S_i(x_1,x_2,...x_{i-1},x_{i+1},...,x_n)\)

We define:

\(\bar S=\dfrac{1}{n}\sum_i\bar S_i\)

\subsubsection{Moments of the jackknife estimator}

We want to know the variance.

\(Var \bar S=\dfrac{n-1}{n}\sum_i(\bar S_i-\bar S)^2\).