\part{Functions}

\chapter{Functions}

\section{Function}

Some types of functions: linear, parabolas, ... \\

The domain of a function $ f $ is the set of all valid input values. The range consists of the set of all output values that can be reached using those domain values. \\

\section{Rational Function}

\begin{itemize}
	\item
	      basic form: $ y = {1 \over x} $ \\

	\item
	      vertical asymptote at $ x = 0 $ \\

	\item
	      horizontal asymptote at $ y = 0 $ \\
\end{itemize}

\begin{figure}
	\centering
	\begin{tikzpicture}[scale=0.9]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-4) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:-0.3] plot (\x,{1 / \x});
		\draw[very thick,color=red,domain=0.3:3] plot (\x,{1 / \x}) node[right] {$ y = {1 \over x} $};
	\end{tikzpicture}
	\caption{rational function}
\end{figure}

\section{Root Function}

\begin{itemize}
	\item
	      basic form: $ y = \sqrt[n]{x} $ \\

	\item
	      $ n $ even: undefined when the root is negative \\

	\item
	      $ n $ odd: $ x \in \mathbb{R} $ \\
\end{itemize}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-4) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=0:3] plot (\x,{sqrt(\x)}) node[right] {$ y = \sqrt[2]{x} $};
	\end{tikzpicture}
	\caption{root function when $ n $ is even}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-4) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:3,samples=100] plot (\x,{\x ^ (1/3)}) node[right] {$ y = \sqrt[3]{x} $};
	\end{tikzpicture}
	\caption{root function when $ n $ is odd}
\end{figure}

\section{Higher-degree of Polynomial Function}

\begin{itemize}
	\item
	      basic form: $ y = x^n $ \\

	\item
	      domain: $ x \in \mathbb{R} $ \\

	\item
	      $ n $ even: both ends of the function tend to $ +\infty $ or both tend to $ -\infty $ \\

	\item
	      $ n $ odd: one end tends to $ +\infty $ while the other tends to $ -\infty $ \\
\end{itemize}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-4) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-2:2] plot (\x,{(\x) ^ 2}) node[right] {$ y = x ^ 2 $};
	\end{tikzpicture}
	\caption{polynomial function when $ n $ is even}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8,yscale=0.2]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-30) -- (0,30) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:3] plot (\x,{(\x) ^ 3}) node[right] {$ y = x ^ 3 $};
	\end{tikzpicture}
	\caption{polynomial function when $ n $ is odd}
\end{figure}

\chapter{Angles, Degrees, and Radians}

\section{Angle}

An angle is created by two rays that intersect at a common endpoint. We use Greek letter $ \theta $ to denote angles. \\

An angle that opens counterclockwise from the x-axis is positive. \\

An angle that opens clockwise from the x-axis is negative. \\

\section{Degree and Radian}

Angles can be measured in 2 ways: \\

\begin{enumerate}
	\item
	      A degree is a measure of the angle formed by $ 1 \over 360 $ of one complete rotation of a circle. \\

	\item
	      A radian is a measure of the angle formed by the arc of a circle whose length is equal to the circle's radius. \\
\end{enumerate}

\begin{theorem}[Degree]
	\begin{align}
		\theta = {s \over r} = {arclength \over radius}
	\end{align}
\end{theorem}

How are radians and degrees related? \\

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\draw[-] (-3,0) -- (3,0);
		\draw[-] (0,-3) -- (0,3);
		\draw (2,0) arc (0:360:2);
		\draw[-,very thick] (0, 0) -- (1.3, 1.5);
		\draw[-,very thick] (0, 0) -- (-0.6, 1.9);
		\draw[-,very thick] (0, 0) -- (-1.9, 0.6) node[left] {$ \approx $ 0.1415 radian};
	\end{tikzpicture}
	\caption{radian}
\end{figure}

\begin{theorem}[Degrees and radians]
	\begin{align}
		 & 180^\circ = \pi \ radians           \\
		 & 1^\circ = {\pi \over 180} \ radians \\
		 & 1 \ radian = {180 \over \pi}^\circ
	\end{align}
\end{theorem}

This relationship provides us with a way to easily convert between the two measures. \\

\begin{exercise}
	Convert from degrees to radians. \\

	(a) $ 30^\circ = $ \\

	(b) $ 220^\circ = $
\end{exercise}

\begin{exercise}
	Convert from radians to degrees. \\

	(a) $ {\pi \over 4} = $ \\

	(b) $ {5\pi \over 6} = $
\end{exercise}

Given any angle $ \theta $, what are these equivalent angles? \\

\begin{theorem}[Equivalent angles]
	\begin{align}
		 & \theta + 2k\pi \ (k \in \mathbb Z)
	\end{align}
\end{theorem}

\chapter{Trigonometric Functions}

\section{Trigonometric Functions}

Let $ O $ be the origin and $ P(x, y) $ be a point on the unit circle so that the radius $ OP $ forms an angle of $ \theta $ radians with respect to the positive x-axis. \\

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\draw[-] (-3,0) -- (3,0);
		\draw[-] (0,-3) -- (0,3);
		\draw (2,0) arc (0:360:2);
		\draw[-,very thick] (0, 0) -- (1.4, 1.4);
		\draw[-,very thick] (0, 0) -- (1.4, 0);
		\draw[-,very thick] (1.4, 0) -- (1.4, 1.4);
	\end{tikzpicture}
	\caption{radian}
\end{figure}

\begin{theorem}[sin / cos]\nonumber
	\begin{align}
		 & x = cos(\theta) \\
		 & y = sin(\theta)
	\end{align}
\end{theorem}

Here are the three most common trigonometric functions and their reciprocals. \\

\begin{figure}[H]
	\centering
	\begin{tikzpicture}
		\begin{axis}[
				xlabel={$ x $}, ylabel={$ y $},
				xmin=-2*pi, xmax=2*pi,
				ymin=-1.5, ymax=1.5,
				xtick={-6.28319, -3.14159, 0, 3.14159, 6.28319},
				xticklabels={$-2\pi$, $-\pi$, $0$, $\pi$, $2\pi$},
				line width=1pt,
				axis lines=center,
			]
			\addplot[smooth,domain=-2*pi:2*pi, red!70]{sin(deg(x))};
		\end{axis}
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		range                    & $ -1 \le sin(\theta) \ge 1 $        \\
		\hline
		doamin                   & $ \theta \in \mathbb R $            \\
		\hline
		$ sin(\theta) = 0 $ when & $ \theta = k\pi,\ k \in \mathbb Z $ \\
		\hline
	\end{tabular}
	\caption{$ y = sin(\theta) $}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}
		\begin{axis}[
				xlabel={$ x $}, ylabel={$ y $},
				xmin=-2*pi, xmax=2*pi,
				ymin=-1.5, ymax=1.5,
				xtick={-6.28319, -3.14159, 0, 3.14159, 6.28319},
				xticklabels={$-2\pi$, $-\pi$, $0$, $\pi$, $2\pi$},
				line width=1pt,
				axis lines=center,
			]
			\addplot[smooth,domain=-2*pi:2*pi, red!70]{cos(deg(x))};
		\end{axis}
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		range                    & $ \theta \in \mathbb R $                           \\
		\hline
		doamin                   & $ -1 \le cos(\theta) \le 1 $                       \\
		\hline
		$ cos(\theta) = 0 $ when & $ \theta = {(2k+1)\pi \over 2},\ k \in \mathbb Z $ \\
		\hline
	\end{tabular}
	\caption{$ y = cos(\theta) $}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\foreach \x / \r in {-4/-2\pi,-3/-\frac{3}{2}\pi,-2/-\pi,-1/-\frac{1}{2}\pi,1/\frac{1}{2}\pi,2/\pi,3/\frac{3}{2}\pi,4/2\pi} \draw (\x,-0.15) -- (\x,+0.15) node[below=7] {$\r$};
		\foreach \y in {-4,-3,-2,-1,1,2,3,4} \draw (-0.15,\y) -- (+0.15,\y) node[left=7] {$\y$};
		\draw[->] (-4.25,0)--(4.25,0) node[right] {$ x $};
		\draw[->] (0,-4.25)--(0,4.25) node[above] {$ y $};
		\foreach \i in {0,...,4} \draw[very thick,color=red] plot [domain={((\i-2)*pi-rad(atan(4))*ceil(\i/4))*(2/pi)}:{((\i-2)*pi+rad(atan(4))*ceil((4-\i)/4))*(2/pi)},smooth] (\x,{tan(\x*pi/2 r)});
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		range                    & $ \theta \in \mathbb R, \theta \ne {(2k+1)\pi \over 2}, k \in \mathbb Z $ \\
		\hline
		doamin                   & $ tan(\theta) \in \mathbb R $                                             \\
		\hline
		$ tan(\theta) = 0 $ when & $ \theta = k\pi,\ k \in \mathbb Z $                                       \\
		\hline
	\end{tabular}
	\caption{$ y = tan(\theta) $}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\foreach \x / \r in {-4/-2\pi,-3/-\frac{3}{2}\pi,-2/-\pi,-1/-\frac{1}{2}\pi,1/\frac{1}{2}\pi,2/\pi,3/\frac{3}{2}\pi,4/2\pi} \draw (\x,-0.15) -- (\x,+0.15) node[below=7] {$\r$};
		\foreach \y in {-4,-3,-2,-1,1,2,3,4} \draw (-0.15,\y) -- (+0.15,\y) node[left=7] {$\y$};
		\draw[->] (-4.25,0)--(4.25,0) node[right] {$ x $};
		\draw[->] (0,-4.25)--(0,4.25) node[above] {$ y $};
		\foreach \i in {0,...,3} \draw[very thick,color=red] plot [domain={((\i-2)*pi+rad(asin(1/4)))*(2/pi)}:{((\i-1)*pi-rad(asin(1/4)))*(2/pi)},smooth] (\x,{cosec(\x*pi/2 r)});
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		range                    & $ \theta \in \mathbb R,\ \theta \ne k\pi,\ k \in \mathbb Z $ \\
		\hline
		doamin                   & $ csc(\theta) \ge 1 $ or $ csc(\theta) \le -1 $              \\
		\hline
		$ csc(\theta) = 0 $ when & $ never $                                                    \\
		\hline
	\end{tabular}
	\caption{$ y = csc(\theta) $}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.9]
		\foreach \x / \r in {-4/-2\pi,-3/-\frac{3}{2}\pi,-2/-\pi,-1/-\frac{1}{2}\pi,1/\frac{1}{2}\pi,2/\pi,3/\frac{3}{2}\pi,4/2\pi} \draw (\x,-0.15) -- (\x,+0.15) node[below=7] {$\r$};
		\foreach \y in {-4,-3,-2,-1,1,2,3,4} \draw (-0.15,\y) -- (+0.15,\y) node[left=7] {$\y$};
		\draw[->] (-4.25,0)--(4.25,0) node[right] {$ x $};
		\draw[->] (0,-4.25)--(0,4.25) node[above] {$ y $};
		\foreach \i in {0,...,4} \draw[very thick,color=red] plot [domain={((\i-2)*pi-rad(acos(1/4))*ceil(\i/4))*(2/pi)}:{((\i-2)*pi+rad(acos(1/4))*ceil((4-\i)/4))*(2/pi)},smooth] (\x,{sec(\x*pi/2 r)});
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		range                    & $ \theta \in \mathbb R,\ \theta \ne {(2k+1)\pi \over 2},\ k \in \mathbb Z $ \\
		\hline
		doamin                   & $ sec(\theta) \ge 1 $ or $ sec(\theta) \le -1 $                             \\
		\hline
		$ sec(\theta) = 0 $ when & $ never $                                                                   \\
		\hline
	\end{tabular}
	\caption{$ y = sec(\theta) $}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8]
		\foreach \x / \r in {-4/-2\pi,-3/-\frac{3}{2}\pi,-2/-\pi,-1/-\frac{1}{2}\pi,1/\frac{1}{2}\pi,2/\pi,3/\frac{3}{2}\pi,4/2\pi} \draw (\x,-0.15) -- (\x,+0.15) node[below=7] {$\r$};
		\foreach \y in {-4,-3,-2,-1,1,2,3,4} \draw (-0.15,\y) -- (+0.15,\y) node[left=7] {$\y$};
		\draw[->] (-4.25,0)--(4.25,0) node[right] {$ x $};
		\draw[->] (0,-4.25)--(0,4.25) node[above] {$ y $};
		\foreach \i in {0,...,3} \draw[very thick,color=red] plot [domain={((\i-2)*pi+rad(atan(1/4)))*(2/pi)}:{((\i-1)*pi-rad(atan(1/4)))*(2/pi)},smooth] (\x,{cot(\x*pi/2 r)});
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		range                    & $ \theta \in \mathbb R,\ \theta \ne k\pi, k \in \mathbb Z $ \\
		\hline
		doamin                   & $ cot(\theta) \in \mathbb R $                               \\
		\hline
		$ cot(\theta) = 0 $ when & $ \theta = {(2k+1)\pi \over 2},\ k \in \mathbb Z $          \\
		\hline
	\end{tabular}
	\caption{$ y = cot(\theta) $}
\end{figure}

\section{Special Triangles}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8]
		\draw (0,0) node[anchor=north]{$ A $}
		-- (4,0) node[anchor=north]{$ C $}
		-- (4,4) node[anchor=south]{$ B $}
		-- cycle;
	\end{tikzpicture}
\end{figure}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8]
		\draw (0,0) node[anchor=north]{$ A $}
		-- (7,0) node[anchor=north]{$ C $}
		-- (7,3) node[anchor=south]{$ B $}
		-- cycle;
	\end{tikzpicture}
\end{figure}

\begin{exercise}
	Evaluate each of the following. \\

	(a) $ sin({\pi \over 4}) = $ \\

	(b) $ cos({\pi \over 4}) = $ \\

	(c) $ csc({\pi \over 4}) = $
\end{exercise}

\begin{exercise}
	Find all values of $ \theta $ satisfying the following. \\

	(a) $ tan(\theta) = {1 \over \sqrt{3}} $ \\
	\\
	\\

	(b) $ sec(\theta) = \sqrt{2} $ \\
	\\
	\\

	(c) $ cot(\theta) = \sqrt{3} $ \\
	\\
	\\
\end{exercise}

\section{Trigonometric Identities}

\begin{theorem}[Trigonometric Identities]
	\begin{align}
		 & sin^2(\theta) + cos^2(\theta) = 1                    \\
		 & sin(-\theta) = -sin(\theta)                          \\
		 & cos(-\theta) = -cos(\theta)                          \\
		 & tan(\theta) = {sin(\theta) \over cos(\theta)}        \\
		 & cot(\theta) = {cos(\theta) \over sin(\theta)}        \\
		 & tan^2(\theta) + 1 = sec^2(\theta)                    \\
		 & cot^2(\theta) + 1 = csc^2(\theta)                    \\
		 & sin(\theta) = cos\left(\theta - {\pi \over 2}\right) \\
		 & sin(a \pm b) = sin(a)cos(b) \pm cos(a)sin(b)         \\
		 & cos(a \pm b) = cos(a)cos(b) \mp sin(a)sin(b)         \\
		 & sin(2\theta) = 2sin(\theta)cos(\theta)               \\
		 & cos(2\theta) = cos^2(\theta) - sin^2(\theta)         \\
		 & sin^2(\theta) = {1 - cos(2\theta) \over 2}           \\
		 & cos^2(\theta) = {1 + cos(2\theta) \over 2}
	\end{align}
\end{theorem}

Reminder: $ trig^n(x) $ is a notation often used to indicate $ (trig(x))^n $. \\

\chapter{Exponential Functions}

\section{Exponential Functions}

Exponential functions are of the form $ y = a^x $, where $ a $ is a positive number and $ x $ is any real number. You might see these sorts of functions when studying population growth, economic growth, global temperature, monetary value, etc. \\

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.6]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-1) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:2] plot (\x,{2 ^ \x}) node[right] {$ y = {a^x} $};
		\draw[very thick,color=red,domain=-2:3] plot (\x,{0.5 ^ \x});
	\end{tikzpicture}
	\caption{exponential function}
\end{figure}

\begin{itemize}
	\item
	      Domain: $ x \in \mathbb R $ \\

	\item
	      Range: $ y > 0 $ \\

	\item
	      The graph $ y = a^x $ always passes through $ (0, 1) $ and $ (1, a) $. \\

	\item
	      If $ a > 1$ then the graph of $ y = a^x $ is increasing. \\

	\item
	      If $ 0 < a < 1 $ then the graph of  $ y = a^x $ is decreasing. \\

	\item
	      $ y = 0 $ is always a horizontal asymptote of $ y = a^x $. \\
\end{itemize}

\section{Exponent Rules}

\begin{theorem}[Exponent Rules]
	\begin{align}
		 & a^{-x} = {1 \over a^x}                    \\
		 & {1 \over a^{-x}} = a^x                    \\
		 & (ab)^x = a^xb^x                           \\
		 & ({a \over b})^x = {a^x \over b^x}         \\
		 & a^{kx} = (a^k)^x = (a^x)^k                \\
		 & a^ma^n = a^{m+n}                          \\
		 & {a^m \over a^n} = a^{m-n}                 \\
		 & a^{1/n} = \sqrt[n]{a}                     \\
		 & a^{m/n} = \sqrt[n]{a^m} = (\sqrt[n]{a})^m
	\end{align}
\end{theorem}

\section{The Base e}

A very special exponential function is $ y = e^x $, where $ e $ is just a content with a non-terminating decimal like $ \pi $. \\

$$
	e = 2.718281845...
$$

What is so special about an exponential function with base $ e $? \\

At any point on the graph, the height of the exponential function is equal to the slope of the tangent line to the graph at that point. \\

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.6]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-3) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:1] plot (\x,{e ^ \x}) node[left] {$ y = {e^x} $};
		\draw[very thick,color=blue,domain=-2:2] plot (\x,{\x + 1});
	\end{tikzpicture}
	\caption{$ y = e^x $}
\end{figure}

\chapter{Logarithmic Functions}

\section{Logarithmic Functions}

Logarithms are the inverse of exponential functions. Let $ a > 0$, then we define a logarithm (log) as follows: \\

\begin{theorem}[Logarithmic Functions]
	\begin{align}
		y   & = log_a(x) \\
		a^y & = x
	\end{align}
\end{theorem}

If no base $ a $ is shown, a base of 10 is assumed. \\

For example: \\
$$
	log(x) = log_{10}(x)
$$

\begin{exercise}
	Evaluate each of the following. \\

	(a) $ log_{2}8 = $ \\

	(b) $ log(100) = $ \\

	(c) $ log_{5}{1 \over 25} = $ \\

	(d) $ log_{8}1 = $
\end{exercise}

Since any positive number to the power of 0 is equal to 1, we have the property that $ log_a(1) $, no matter what the base $ a $ is. \\

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8]
		\draw[->] (-1,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-4) -- (0,4) node[above] {$ y $};
		\draw [very thick,color=red,domain=0.1:4,samples=100] plot (\x,{log2(\x)}) node[right] {$ y = log_2(x) $};
		\draw [very thick,color=red,domain=0.1:4,samples=100] plot (\x,{log10(\x) / log10(0.5)}) node[right] {$ y = log_{1 \over 2}(x) $};
	\end{tikzpicture}
	\caption{logarithmic function}
\end{figure}

\begin{itemize}
	\item
	      Domain: $ 0 < x < \infty $ \\

	\item
	      Range: $ y \in \mathbb R $ \\

	\item
	      $ y = log_a(x) $ always passes through $ (a, 1) $ and $ (1, 0) $. \\

	\item
	      If $ a > 1 $ then the graph of $ y = log_a(x) $ is increasing. \\

	\item
	      If $ 0 < a < 1 $ then the graph of $ y = log_a(x) $ is decreasing. \\

	\item
	      $ x = 0 $ is always a vertical asymptote of $ y = log_a(x) $. \\
\end{itemize}

\section{Logarithm Rules}

\begin{theorem}[Logarithm Rules]
	\begin{align}
		 & log_a(a^x) = x                                      \\
		 & a^{log_a(x)} = x                                    \\
		 & log_a(xy) = log_a(x) + log_a(y)                     \\
		 & log_a\left({x \over y}\right) = log_a(x) - log_a(y) \\
		 & log_a(x^n) = nlog_a(x)
	\end{align}
\end{theorem}

\section{Change of Base Formula}

We can switch between any two bases easily by using the formula: \\

\begin{theorem}[Change of Base Formula]
	\begin{align}
		log_a(x) = {log_b(x) \over log_b(a)}
	\end{align}
\end{theorem}

\begin{exercise}\nonumber
	Proof

	\vspace{5cm}
\end{exercise}

\begin{exercise}\nonumber
	Convert $ log_4(x) $ into a logarithm with each of the following bases. \\

	(a) base 3
	\\
	\\

	(b) base 22
	\\
	\\
\end{exercise}

\section{The Natural Logarithm}

A special logarithm is the natural logarithm, which is the logarithm with a base of $ e $. Rather than write $ log_e(x) $, we typically write $ ln(x) $. \\

The natural logarithm has the exact same properties as any other logarithmic function. \\

\begin{theorem}[Natural Logarithm]
	\begin{align}
		 & ln(e^x) = x   \\
		 & e^{ln(x)} = x
	\end{align}
\end{theorem}

\begin{exercise}\nonumber
	Solve each of the following for $ x $. \\

	(a)
	\begin{align}
		2^x & = 2^{1-x}     \\
		\\
		\\
		\\
	\end{align}
	\\

	(b)
	\begin{align}
		3^{{1 \over 2} + 10} & = 27   \\
		\\
		\\
		\\
		\\
	\end{align}
	\\

	(c)
	\begin{align}
		2^x     & = 10                   \\
		\\
		\\
		\\
		\\
	\end{align}
	\\

	(d)
	\begin{align}
		log(x) - 1   & = log(x - 1)                           \\
		\\
		\\
		\\
		\\
		\\
		\\
		\\
	\end{align}
	\\

	(e)
	\begin{align}
		log_2(x) + log_2(x^2) & = 6 \\
		\\
		\\
		\\
		\\
		\\
	\end{align}
	\\

	(f)
	\begin{align}
		log_2(x^4) + log_2(x^2) & = 6     \\
		\\
		\\
		\\
		\\
		\\
	\end{align}
\end{exercise}


\subsection{Topological spaces}\label{subsec:topological_spaces}

\begin{definition}\label{def:topological_space}\mcite[11]{Engelking1989}
  Let \( X \) be any set and \( \mscrT \subseteq \pow(X) \) be a family of subsets of \( X \). \( \mscrT \) is called a \term{topology} on \( X \) and the tuple \( (X, \mscrT) \) is said to be a \term{topological space} if the following axioms are satisfied:
  \begin{thmenum}
    \thmitem[def:topological_space/O1]{O1} \( \varnothing, X \in T \)
    \thmitem[def:topological_space/O2]{O2} \( U, V \in T \implies U \cap V \in T \)
    \thmitem[def:topological_space/O3]{O3} \( \mscrT' \subseteq T \implies \bigcap \mscrT' \in T \)
  \end{thmenum}

  If the topology is obvious from the context, we say that \( X \) is a topological space.

  Elements of the set \( X \) are called \term{points} of the topological space, elements of \( \mscrT \) are called \term{open sets} and set-theoretic complements of open sets are called \term{closed sets}.

  If \( x \in U \in T \), we say that \( U \) is a \term{neighborhood} of \( x \). Note that some authors (e.g. \cite[38]{Kelley1955}) alternatively define neighborhoods as arbitrary sets that contain an open set that contains \( x \). For simplicity, we define the subfamily
  \begin{equation*}
    \mscrT(x) \coloneqq \{ U \in T \colon x \in U \}.
  \end{equation*}

  We say that \( U \) is a \term{punctured neighborhood} of \( x \) if \( U \cup \{ x \} \) is an open set and, consequently, a neighborhood of \( x \).

  Dually, we can define the family \( \mscrF \) of closed sets, where
  \begin{thmenum}
    \thmitem[def:topological_space/F1]{F1} \( \varnothing, X \in F \)
    \thmitem[def:topological_space/F2]{F2} \( U, V \in F \implies U \cup V \in F \)
    \thmitem[def:topological_space/F3]{F3} \( \mscrF' \subseteq F \implies \bigcup \mscrF' \in F \)
  \end{thmenum}

  If \( (X, \mscrT) \) is a topological space, we denote the corresponding family of closed sets by
  \begin{equation*}
    \mscrF_\mscrT \coloneqq \{ X \setminus U \colon U \in T \}.
  \end{equation*}
\end{definition}

\begin{definition}\label{def:standard_topologies}
  On a space \( X \), we can explicitly define the following standard topologies:
  \begin{thmenum}
    \thmitem{def:standard_topologies/discrete} The \term{discrete topology} \( \mscrT \coloneqq \pow(X) \).
    \thmitem{def:standard_topologies/indiscrete} The \term{indiscrete topology} \( \mscrT \coloneqq \{ \varnothing, X \} \).
    \thmitem{def:standard_topologies/co_cardinal} For any \hyperref[def:cardinal]{cardinal} \( \xi \), the \term{co-\( \xi \) topology} \( \mscrT \coloneqq \{ A \subseteq X \colon \card A < \xi \} \) and, in particular, \term{cofinite} (\( \xi = \aleph_0 \)) and \term{cocountable} (\( \xi = c \)) topologies.
  \end{thmenum}

  For a deeper connection between discrete and indiscrete topologies, see \fullref{ex:def:category_adjunction/set_top}.
\end{definition}

\begin{proposition}\label{thm:set_open_iff_neighborhood_is_contained}
  A set \( A \) is open if and only if every point of \( A \) has a neighborhood \( U \) such that \( U \subseteq A \).
\end{proposition}
\begin{proof}
  This holds vacuously for empty sets. Assume that \( A \subseteq X \) is nonempty.

  \SufficiencySubProof Assume that \( A \) is open and let \( x_0 \in A \). Then \( A \) is a neighborhood of \( x_0 \) and the theorem holds trivially.
  \NecessitySubProof Assume that every point \( x \in A \) has a neighborhood \( U_x \) such that \( U_x \subseteq A \). Take the union
  \begin{equation*}
    B \coloneqq \cup_{x \in A} U_x.
  \end{equation*}

  Obviously \( B \subseteq A \). Aiming at a contradiction, suppose that \( y_0 \in A \setminus B \). Then \( y_0 \) has a neighborhood \( U_{y_0} \) such that \( U_{y_0} \setminus B \) is nonempty. But this is impossible by the definition of \( B \). The obtained contradiction proves \( B = A \).
\end{proof}

\begin{remark}\label{rem:abritrary_family_to_topology}
  It is sometimes easier to define a topology \( \mscrT \) via a subset of \( \mscrT \). We will gradually construct a topology from a bare family of sets in \( X \). First, we will give two definitions for a base, one on which does not require an existing topology.
\end{remark}

\begin{definition}\label{def:topological_base}\mcite[12]{Engelking1989}
  Fix a topological space \( (X, \mscrT) \). We say that the family \( \mscrB \subseteq T \) is a \term{base} for the topology \( \mscrT \) if \( \mscrB \) satisfies any of the equivalent conditions:
  \begin{thmenum}
    \thmitem{def:topological_base/union} Every open set \( U \in T \) is the union \( U = \bigcup \mscrB' \) of some subset \( \mscrB' = \mscrB \)
    \thmitem{def:topological_base/subset} For any point \( x \in X \) and for any neighborhood \( U \) of \( x \) there exists a set \( V \in B \) in the base such that \( x \in V \subseteq U \)
  \end{thmenum}
\end{definition}
\begin{proof}
  \ImplicationSubProof{def:topological_base/union}{def:topological_base/subset} Fix a point \( x \in X \) and a neighborhood \( U \in T \) of \( x \). Let \( \mscrB' \) be a subfamily of \( \mscrB \) such that
  \begin{equation*}
    U = \bigcup \mscrB'.
  \end{equation*}

  Then \( x \in V \) for at least one \( V \in B' \).

  \ImplicationSubProof{def:topological_base/subset}{def:topological_base/union} Fix an open set \( U \in T \). Then for every \( x \in U \), there exists a set \( V_x \in B \) such that \( x \in V_x \subseteq U \). We have
  \begin{equation*}
    \bigcup_{x \in U} V_x \subseteq U \subseteq \bigcup_{x \in U} V_x,
  \end{equation*}
  thus
  \begin{equation*}
    U = \bigcup_{x \in U} V_x.
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:topological_base_axioms}\mcite[12]{Engelking1989}
  Let \( X \) be an arbitrary set and let \( \mscrB \) be a family of subset that satisfies
  \begin{thmenum}
    \thmitem[thm:topological_base_axioms/B1]{B1} \( \bigcup \mscrB = X \)
    \thmitem[thm:topological_base_axioms/B2]{B2} \( \forall U, V \in B, \forall x \in U \cap V, \exists W \in B: x \in W \subseteq U \cap V \)
  \end{thmenum}

  Then the family
  \begin{balign}\label{thm:topological_base_axioms/topology}
    \mscrT \coloneqq \left\{ \bigcup \mscrB' \colon \mscrB' \subseteq B \right\}
  \end{balign}
  is a topology on \( X \). Furthermore, \( \mscrB \) is a \hyperref[def:topological_base]{base} of \( \mscrT \).

  In particular, the base on any topology satisfies \fullref{thm:topological_base_axioms/B1} -- \fullref{thm:topological_base_axioms/B2}.
\end{proposition}
\begin{proof}
  We will first prove that \( \mscrT \) is indeed a topology.

  \begin{refenum}
    \refitem{def:topological_space/O1} \( \varnothing = \bigcup \varnothing \in \tau \) and \( X = \bigcup \mscrB \in T \) (by \fullref{thm:topological_base_axioms/B1})

    \refitem{def:topological_space/O3} Fix \( \mscrT' = \{ U_\alpha \colon \alpha \in A \} \subseteq T \). By \fullref{def:topological_base/union}, every set \( U_\alpha \) has a corresponding subfamily \( \mscrB_\alpha \) of \( \mscrB \) such that \( U_\alpha = \bigcup \mscrB_\alpha \).

    Define \( \mscrB' \coloneqq \bigcup_{\alpha \in A} \mscrB_\alpha \). Obviously \( \mscrB' \subseteq B \) and thus, by \fullref{thm:topological_base_axioms/B1}, \( \bigcup \mscrB \in T \).

    \refitem{def:topological_space/O2} Fix \( U, V \in T \) and families \( \mscrB_U, \mscrB_V \subseteq B \) such that \( U = \bigcup \mscrB_U \) and \( V = \bigcup \mscrB_V \).

    Fix arbitrary \( U' \in B_U \) and \( V' \in B_V \). We will show that \( U' \cap V' \in \tau \).

    By \fullref{thm:topological_base_axioms/B2}, for every \( x \in U' \cap V' \) there exists a neighborhood \( W_x \) of \( x \) such that \( W \subseteq U' \cap V' \).

    The family \( \mscrB_{U',V'} \coloneqq \{ W_x \colon x \in U' \cap V' \} \) is a subfamily of \( \mscrB \) and thus \( U' \cap V' = \bigcup \mscrB_{U',V'} \in T \).

    Hence, by \fullref{def:topological_space/O3}, \( U \cap V \in \tau \).

    Now, for any \( U \in T \), by \fullref{thm:topological_base_axioms/topology}, there exists a subfamily \( \mscrB' \subseteq B \) such that
    \begin{equation*}
      U = \bigcup \mscrB'.
    \end{equation*}

    Hence, \( \mscrB \) is a base for \( \mscrT \).
  \end{refenum}
\end{proof}

\begin{definition}\label{def:topological_space_weight}
  We define the \term{weight} of \( (X, \mscrT) \) as the cardinal
  \begin{equation*}
    w((X, \mscrT)) \coloneqq \min \{ \abs{\mscrB} \colon \mscrB \text{ is a base for } \mscrT \}.
  \end{equation*}

  We simply write \( w(X) \) when the topology is clear from the context.

  Spaces for which \( w(X) \leq \aleph_0 \) are said to be \term{second-countable}.
\end{definition}
\begin{proof}
  The definition is correct because cardinals are well-ordered when regarded as initial ordinals.
\end{proof}

\begin{definition}\label{def:topological_subbase}\mcite[12]{Engelking1989}
  Fix a topological space \( (X, \mscrT) \). We say that the family \( \mscrP \subseteq T \) is a \term{subbase} for the topology \( \mscrT \) if the family
  \begin{equation*}
    \mscrB \coloneqq \left\{ \bigcap P' \colon P' \text{ is a nonempty \hyperref[def:set_finiteness]{finite} subset of } P \right\}
  \end{equation*}
  of finite intersections of \( \mscrP \) is a \hyperref[def:topological_base]{base} of \( \mscrT \).
\end{definition}

\begin{proposition}\label{thm:subbase_from_arbitrary_family}
  Fix a set \( X \) and a family of subsets \( \mscrP \subseteq \pow(X) \). The family \( \mscrP' \coloneqq \mscrP \cup X \) is then a \hyperref[def:topological_subbase]{subbase} of some topology on \( X \).
\end{proposition}

\begin{definition}\label{def:topological_local_base}\mcite[12]{Engelking1989}
  Fix a topological space \( (X, \mscrT) \) and a point \( x \in X \). We say that the family \( \mscrB(x) \subseteq T \) is a \term{local base} for \( \mscrT \) at \( x \) if every neighborhood of \( x \) contains a set from \( \mscrB(x) \).

  Given a base \( \mscrB \), unless explicitly noted, we consider the subfamily \( \mscrB(x) \) of all members of \( \mscrB \) containing \( x \).

  The indexed family of local bases \( \{ \mscrB(x) \colon x \in X \} \) is called a \term{neighborhood system} of \( \mscrT \).
\end{definition}

\begin{proposition}\label{thm:neighborhood_iff_union_in_topological_local_base}
  Analogously to \fullref{def:topological_base/union}, a set \( A \) containing \( x \) is a neighborhood of \( x \) if and only if \( A \) is a union of elements of the local \hyperref[def:topological_local_base]{base} \( \mscrB(x) \).
\end{proposition}
\begin{proof}
  Analogous to the proof of the equivalence in \fullref{def:topological_base}.
\end{proof}

\begin{proposition}\label{thm:topological_local_base_axioms}\mcite[13]{Engelking1989}
  Let \( X \) be an arbitrary set and let \( \{ \mscrB(x) \subseteq \pow(X) \colon x \in X \} \) be an indexed family of families of subsets of \( X \) that satisfies
  \begin{thmenum}
    \thmitem[thm:topological_local_base_axioms/BP1]{BP1} For every \( x \in X \), \( \mscrB(x) \neq \varnothing \) and \( x \in U \) for every \( U \in B(x) \).
    \thmitem[thm:topological_local_base_axioms/BP2]{BP2} For every \( x \in X \) and for all \( U, V \in B(x) \), \( \exists W \in B(x): W \subseteq U \cap V \).
    \thmitem[thm:topological_local_base_axioms/BP3]{BP3} For all \( x, y \in X \), \( x \in U \in B(y) \) implies that there exists \( V \in B(x) \) such that \( U \subseteq V \).
  \end{thmenum}

  Then the family
  \begin{equation*}
    \mscrB \coloneqq \bigcup_{x \in X} \mscrB(x)
  \end{equation*}
  is the \hyperref[thm:topological_base_axioms]{base} of some topology \( \mscrT \) on \( X \). Furthermore, \( \{ \mscrB(x) \subseteq \pow(X) \colon x \in X \} \) is a \hyperref[def:topological_local_base]{neighborhood system} for \( (X, \mscrT) \).

  In particular, the local base on any topology satisfies \fullref{thm:topological_local_base_axioms/BP1} -- \fullref{thm:topological_local_base_axioms/BP3}.
\end{proposition}

\begin{definition}\label{def:topological_space_character}
  We define the \term{character} of the point \( x \in X \) as the cardinal
  \begin{equation*}
    \chi(x) \coloneqq \min \{ \card \mscrB(x) \colon \mscrB(x) \text{ is a local base for } \mscrT \text{ at } x \}.
  \end{equation*}

  We define the \term{character} of of \( (X, \mscrT) \) as
  \begin{equation*}
    \chi((X, \mscrT)) \coloneqq \sup \{ \chi(x) \colon x \in X \}.
  \end{equation*}

  We simply write \( \chi(X) \) when the topology is clear from the context.

  Spaces for which \( \chi(X) \leq \aleph_0 \) are said to be \term{first-countable}.
\end{definition}
\begin{proof}
  The definition is correct because cardinals are well-ordered when regarded as initial ordinals.
\end{proof}

\begin{definition}\label{def:topological_local_subbase}
  Combining \fullref{def:topological_subbase} and \fullref{def:topological_local_base}, we define a \term{local subbase} for \( \mscrT \) at \( x \) to be a family \( \mscrP(x) \subseteq T \) such that every neighborhood \( U \) of \( x \) contains a finite intersection of sets from \( \mscrP(x) \).

  Given a subbase \( \mscrP \), unless explicitly noted, we consider the subfamily \( \mscrP(x) \) of all members of \( \mscrP \) containing \( x \).
\end{definition}

\begin{definition}\label{def:topological_closure_operator}\mcite[33]{Engelking1989}
  Let \( (X, \mscrT) \) be a topological space. Define the \term{closure operator}
  \begin{balign*}
     & \cl: \pow(X) \to \pow(X)                                           \\
     & \cl(A) \coloneqq \bigcap \{ F : F \in F_\mscrT, A \subseteq F \}.
  \end{balign*}
\end{definition}

\begin{proposition}\label{thm:def:topological_closure_operator/properties}
  The closure \hyperref[def:topological_closure_operator]{operator} has the following basic properties
  \begin{thmenum}
    \thmitem{thm:def:topological_closure_operator/properties/closed} The set \( A \) is closed if and only if \( A = \cl A \).
    \thmitem{thm:def:topological_closure_operator/properties/neighborhood_intersection} For any \( x \in X \), \( x \in \cl A \) if and only if every neighborhood of \( x \) intersects \( A \).
    \thmitem{thm:def:topological_closure_operator/properties/monotone} \( \cl \) is \hyperref[def:partially_ordered_set/homomorphism]{monotone}, i.e. if \( A \subseteq B \), then \( \cl(A) \subseteq \cl(B) \).
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:topological_closure_operator/properties/closed} The condition \( A = \cl{A} \) is equivalent to \( A \) being a closed superset of itself, which is equivalent to \( A \) being closed.

  \SubProofOf{thm:def:topological_closure_operator/properties/neighborhood_intersection} Note that this proof relies on \fullref{def:topological_boundary}, however we do not use this property when defining the boundary.

  \SufficiencySubProof Fix \( x \in \cl{A} \) and let \( U \) be a neighborhood of \( x \). If \( x \in A \), then obviously \( x \in U \cap A \neq \varnothing \). If \( x \not\in A \), then \( U \cap A \neq \varnothing \) by \fullref{def:topological_boundary/neighborhoods}. In both cases, we obtain \( U \cap A \neq \varnothing \), which proves the statement.

  \NecessitySubProof Fix \( x \in X \) and assume that every neighborhood of \( x \) intersects \( A \). Since the case \( x \in A \) is trivial, suppose that \( x \not\in A \). By \fullref{thm:set_open_iff_neighborhood_is_contained}, every neighborhood \( U \) of \( x \) does not entirely belong to \( A \). By \fullref{def:topological_boundary/neighborhoods}, \( x \in \fr A \subseteq \cl A \).

  \SubProofOf{thm:def:topological_closure_operator/properties/monotone} If \( A \subseteq B \), every closed superset of \( B \) is also a closed superset of \( A \).
\end{proof}

\begin{proposition}\label{thm:topological_closure_operator_axioms}\mcite[14]{Engelking1989}
  Let \( X \) be an arbitrary set and let \( \cl: \pow(X) \to \pow(X) \) be a function that satisfies
  \begin{thmenum}
    \thmitem[thm:topological_closure_operator_axioms/CO1]{CO1} \( \cl(\varnothing) = \varnothing \)
    \thmitem[thm:topological_closure_operator_axioms/CO2]{CO2} \( \forall A \in \pow(X), A \subseteq \cl(A) \)
    \thmitem[thm:topological_closure_operator_axioms/CO3]{CO3} \( \forall A, B \in \pow(X), \cl(A \cup B) = \cl(A) \cup \cl(B) \)
    \thmitem[thm:topological_closure_operator_axioms/CO4]{CO4} \( \forall A \in \pow(X), \cl(\cl(A)) = \cl(A) \)
  \end{thmenum}

  Then the family
  \begin{equation*}
    \mscrT \coloneqq \{ X \setminus F \colon F = \cl(F) \}
  \end{equation*}
  is a topology on \( X \). Furthermore, \( \cl = \cl^{\mscrT} \), where \( \cl^{\mscrT} \) is the closure \hyperref[def:topological_closure_operator]{operator} on \( (X, \mscrT) \).

  In particular, the closure operator on any topology satisfies \fullref{thm:topological_closure_operator_axioms/CO1} -- \fullref{thm:topological_closure_operator_axioms/CO4}.
\end{proposition}

\begin{definition}\label{def:interior_operator}\mcite[15]{Engelking1989}
  Let \( (X, \mscrT) \) be a topological space. Define the \term{interior operator}
  \begin{balign*}
     & \Int: \pow(X) \to \pow(X)                                     \\
     & \Int(A) \coloneqq \bigcup \{ U : U \in T, U \subseteq A \}.
  \end{balign*}
\end{definition}

\begin{proposition}\label{thm:def:interior_operator/properties}
  The interior \hyperref[def:interior_operator]{operator} has the following basic properties
  \begin{thmenum}
    \thmitem{thm:def:interior_operator/properties/open} A set \( A \) is a topological space is open if and only if \( A = \int A \).
    \thmitem{thm:def:interior_operator/properties/monotone} \( \int \) is \hyperref[def:partially_ordered_set/homomorphism]{monotone}, i.e. if \( A \subseteq B \), then \( \Int(A) \subseteq \Int(B) \).
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:interior_operator/properties/open} Follows from \fullref{thm:def:topological_closure_operator/properties/closed} and \fullref{thm:interior_closure_complement}.
  \SubProofOf{thm:def:interior_operator/properties/monotone} Follows from \fullref{thm:def:topological_closure_operator/properties/monotone} and \fullref{thm:interior_closure_complement}.
\end{proof}

\begin{proposition}\label{thm:interior_closure_complement} For every set \( A \subseteq X \) we have
  \begin{itemize}
    \item \( X \setminus \Int(A) = \cl(X \setminus A) \)
    \item \( X \setminus \cl(A) = \Int(X \setminus A) \)
  \end{itemize}
\end{proposition}
\begin{proof}
  Any open subset \( U \subseteq A \) is a closed superset of \( X \setminus A \). A point belongs to \( \Int(A) \) if it belongs to at least one open subset of \( A \), which happens if and only if it belongs to at least one closed superset of \( X \setminus A \). Therefore,
  \begin{balign*}
    X \setminus \Int(A)
     & =
    X \setminus \bigcup \{ U : U \in T, U \subseteq A \}
    =                                            \\ &=
    X \setminus \bigcup \{ F : F \in F_\mscrT, X \setminus A \subseteq F \}
    \reloset {X \setminus (X \setminus A) = A} = \\ &=
    \bigcup \{ F : F \in F_\mscrT, F \subseteq A \}.
    =                                            \\ &=
    \cl(A).
  \end{balign*}

  The other equality is obtained by noting that \( X \setminus \cl(A) = X \setminus (X \setminus \Int(A)) = \Int(A) \).
\end{proof}

\begin{proposition}\label{thm:interior_operator_axioms}
  Let \( X \) be an arbitrary set and let \( \int: \pow(X) \to \pow(X) \) be a function that satisfies
  \begin{thmenum}
    \thmitem[thm:interior_operator_axioms/IO1]{IO1} \( \Int(X) = X \)
    \thmitem[thm:interior_operator_axioms/IO2]{IO2} \( \forall A \in \pow(X), \Int(A) \subseteq A \)
    \thmitem[thm:interior_operator_axioms/IO3]{IO3} \( \forall A, B \in \pow(X), \Int(A \cap B) = \Int(A) \cap \Int(B) \)
    \thmitem[thm:interior_operator_axioms/IO4]{IO4} \( \forall A \in \pow(X), \Int(\Int(A)) = \Int(A) \)
  \end{thmenum}

  Then the family
  \begin{equation*}
    \mscrT \coloneqq \{ U \colon U = \Int(U) \}
  \end{equation*}
  is a topology on \( X \). Furthermore, \( \int = \int_\mscrT \), where \( \int_\mscrT \) is the interior \hyperref[def:interior_operator]{operator} on \( (X, \mscrT) \).

  In particular, the interior operator on any topology satisfies \fullref{thm:interior_operator_axioms/IO1} -- \fullref{thm:interior_operator_axioms/IO4}.
\end{proposition}

\begin{definition}\label{def:topological_boundary}
  For a subset \( A \) of a topological space we define its \term{boundary} \( \fr(A) \) equivalently as
  \begin{thmenum}
    \thmitem{def:topological_boundary/closure} \( \fr(A) \coloneqq \cl(A) \setminus \Int(A) \)
    \thmitem{def:topological_boundary/neighborhoods} \( \fr(A) \) is the set of all points \( x \in X \) such that every neighborhood of \( x \) intersects both \( A \) and \( X \setminus A \).
  \end{thmenum}
\end{definition}
\begin{proof}
  The equivalence of the definitions is trivial when \( \fr(A) = \varnothing \). We assume that \( \fr(A) \neq \varnothing \).

  \ImplicationSubProof{def:topological_boundary/closure}{def:topological_boundary/neighborhoods} Let \( x \in \cl(A) \setminus \Int(A) \).

  Aiming for a contradiction, suppose that there is a neighborhood \( U \) of \( x \) that does not intersect \( A \). Then \( U \subseteq X \setminus A \). Hence, \( A \subseteq X \setminus U \). Since \( X \setminus U \) is closed, it follows that \( \cl(A) \subseteq X \setminus U \) as the intersection of all closed supersets of \( A \). But \( x \not\in X \setminus U \), therefore \( x \not\in \cl(A) \), which contradicts our choice of \( x \in \cl(A) \).

  This proves that every neighborhood of \( x \) intersects \( A \).

  By passing to complements, we can reuse this to prove that every neighborhood of \( x \) intersects \( X \setminus A \) using \fullref{thm:interior_closure_complement}.

  \ImplicationSubProof{def:topological_boundary/neighborhoods}{def:topological_boundary/closure} Suppose that every neighborhood of \( x \in \fr(A) \) intersects both \( A \) and \( X \setminus A \). Therefore, no neighborhood of \( x \) is contained in neither \( A \) not \( X \setminus A \) and \( x \) belongs to neither \( \Int(A) \) nor \( \Int(X \setminus A) \). Hence,
  \begin{equation*}
    x \in (X \setminus \Int(X \setminus A)) \setminus \Int(A) \reloset {\ref{thm:interior_closure_complement}} = \cl(A) \setminus \Int(A).
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:def:topological_boundary/properties}
  The \hyperref[def:topological_boundary]{topological boundary} has the following basic properties
  \begin{thmenum}
    \thmitem{thm:def:topological_boundary/properties/closed} \( \fr(A) \) is a closed set.
    \thmitem{thm:def:topological_boundary/properties/not_open} If \( \fr(A) \) is not empty, it is not an open set.
    \thmitem{thm:def:topological_boundary/properties/complement} \( \fr(A) = \fr(X \setminus A) \).
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:topological_boundary/properties/closed} Note that
  \begin{equation*}
    \fr(A) = \cl(A) \setminus \Int(A) = \cl(A) \cap (X \setminus \Int(A)),
  \end{equation*}
  which is the intersection of two closed sets. Hence, \( \fr(A) \) is itself a closed set.

  \SubProofOf{thm:def:topological_boundary/properties/not_open} Note that \( \fr(A) \) is either empty or is not open because \fullref{def:topological_boundary/neighborhoods} is incompatible with \fullref{thm:set_open_iff_neighborhood_is_contained}.

  \SubProofOf{thm:def:topological_boundary/properties/complement} By \fullref{thm:interior_closure_complement},
  \begin{balign*}
    \fr(A)
     & =
    \cl(A) \setminus \Int(A)
    =                                                  \\ &=
    \cl(A) \cap (X \setminus \Int(A))
    \reloset {\ref{thm:interior_closure_complement}} = \\ &=
    (X \setminus \Int(X \setminus A)) \cap \cl(X \setminus A)
    =                                                  \\ &=
    \cl(X \setminus A) \setminus \Int(X \setminus A)
    =                                                  \\ &=
    \fr(X \setminus A).
  \end{balign*}
\end{proof}

\begin{definition}\label{def:topological_derived_set}\mcite[24]{Engelking1989}
  Let \( (X, \mscrT) \) be a topological space.

  \begin{thmenum}
    \thmitem{def:topological_derived_set/cluster_point} We say that the point \( x_0 \in X \) is a \term{cluster point} or an \term{accumulation point} of the set \( A \subseteq X \) if \( x \in \cl(A \setminus \{ x \}) \). It is not necessary for \( x_0 \) to belong to \( A \).

    \thmitem{def:topological_derived_set/derived_set} The set of all cluster points of \( A \) is called the \term{derived set} of \( A \) and is denoted by \( \derived(A) \).

    \thmitem{def:topological_derived_set/perfect_set} If a set equals its derived set, we call it a \term{perfect set}.

    \thmitem{def:topological_derived_set/isolated_point} Points in \( A \setminus \derived(A) \) are said to be \term{isolated points} of \( A \).

    \thmitem{def:topological_derived_set/discrete_set} If \( \derived(A) = \varnothing \), that is, if \( A \) consists of only discrete points, we say that \( A \) is a \term{discrete set}.
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:def:derived_set/properties}
  \hyperref[def:topological_derived_set]{Derived sets} have the following basic properties
  \begin{thmenum}
    \thmitem{thm:def:derived_set/properties/cluster_via_neighborhoods} \( x \) is a cluster point of \( A \) if and only if every neighborhood of \( x \) intersects \( A \setminus \{ x \} \)
    \thmitem{thm:def:derived_set/properties/isolated_via_neighborhoods} \( x \) is an isolated point of \( A \) if and only if there exists a neighborhood of \( x \) that does not intersect \( A \setminus \{ x \} \)
    \thmitem{thm:def:derived_set/properties/closed} \( \derived(A) \) is a closed set.
    \thmitem{thm:def:derived_set/properties/closure} \( A \cup \derived(A) = \cl(A) \).
    \thmitem{thm:def:derived_set/properties/closed_iff_contains_all_cluster_points} A set is closed if and only if it contains all of its cluster points. Compare this result to \fullref{thm:limit_point_iff_in_closure}.
    \thmitem{thm:def:derived_set/properties/closed_iff_only_isolated_and_cluster_points} A set if closed if and only if every point is either a cluster point or an isolated point.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:derived_set/properties/cluster_via_neighborhoods} If every neighborhood \( U \) of \( x \in A \) intersects \( A \setminus \{ x \} \), by \fullref{thm:def:topological_closure_operator/properties/neighborhood_intersection}, \( x \in \cl(A \setminus \{ x \}) \) and \( x \) is therefore a cluster point.

  \SufficiencySubProofOf{thm:def:derived_set/properties/isolated_via_neighborhoods} Dual to \fullref{thm:def:derived_set/properties/cluster_via_neighborhoods}.

  \SubProofOf{thm:def:derived_set/properties/closed} Consider the complement of \( \derived(A) \). If it is empty, \( \derived(A) \) is trivially closed. Otherwise, let \( x \in X \setminus \derived(A) \).

  \begin{itemize}
    \item If \( x \) is an isolated point of \( A \), by \fullref{thm:def:derived_set/properties/isolated_via_neighborhoods} there exists a neighborhood of \( x \) that does not intersect \( A \setminus \{ x \} \).
    \item If \( x \) is not a point of \( A \), aiming at a contradiction, assume that every neighborhood of \( x \) intersects \( A \). Then, by \fullref{def:topological_boundary/neighborhoods}, \( x \in \fr(A) \). But \( \fr(A) \subseteq \cl(A) \) and \( \cl(A) = \cl(A \setminus \{ x \}) \) because \( x \) does not belong to \( A \). Therefore, \( x \) is a cluster point of \( A \). This contradicts our assumption that \( x \not\in \derived(A) \), hence we can conclude that there exists a neighborhood of \( X \) that does not intersect \( A = A \setminus \{ x \} \).
  \end{itemize}

  In both cases, \fullref{thm:set_open_iff_neighborhood_is_contained} allows us to conclude that \( X \setminus \derived(A) \) is open and, hence, \( \derived(A) \) is closed.

  \SubProofOf{thm:def:derived_set/properties/closure} Clearly \( A \subseteq \cl(A) \). Also
  \begin{equation*}
    \derived(A) \subseteq \bigcup_{x \in X} \cl(A \setminus \{ x \}) \subseteq \cl(A).
  \end{equation*}

  Now we will prove the reverse inclusion. Let \( x \in \cl(A) \). Then either \( x \in A \) or \( x \in \fr(A) \). Assume the latter. By \fullref{def:topological_boundary/neighborhoods}, every neighborhood \( U \) of \( x \) has points both in \( A \) and outside \( A \), therefore \( U \cap (A \setminus \{ x \}) \) is nonempty. By \fullref{thm:def:topological_closure_operator/properties/neighborhood_intersection}, \( x \in \cl(A \setminus \{ x \}) \), that is, \( x \in \derived(A) \).

  \SubProofOf{thm:def:derived_set/properties/closed_iff_contains_all_cluster_points}
  If \( A \) is closed, by \fullref{thm:def:derived_set/properties/closure},
  \begin{equation*}
    A \cup \derived(A) = \cl(A) = A,
  \end{equation*}
  hence \( \derived(A) \subseteq A \).

  \NecessitySubProofOf{thm:def:derived_set/properties/isolated_via_neighborhoods} Dual to \fullref{thm:def:derived_set/properties/cluster_via_neighborhoods}.

  \SubProofOf{thm:def:derived_set/properties/closed} Consider the complement of \( \derived(A) \). If it is empty, \( \derived(A) \) is trivially closed. Otherwise, let \( x \in X \setminus \derived(A) \).

  \begin{itemize}
    \item If \( x \) is an isolated point of \( A \), by \fullref{thm:def:derived_set/properties/isolated_via_neighborhoods} there exists a neighborhood of \( x \) that does not intersect \( A \setminus \{ x \} \).
    \item If \( x \) is not a point of \( A \), aiming at a contradiction, assume that every neighborhood of \( x \) intersects \( A \). Then, by \fullref{def:topological_boundary/neighborhoods}, \( x \in \fr(A) \). But \( \fr(A) \subseteq \cl(A) \) and \( \cl(A) = \cl(A \setminus \{ x \}) \) because \( x \) does not belong to \( A \). Therefore, \( x \) is a cluster point of \( A \). This contradicts our assumption that \( x \not\in \derived(A) \), hence we can conclude that there exists a neighborhood of \( X \) that does not intersect \( A = A \setminus \{ x \} \).
  \end{itemize}

  In both cases, \fullref{thm:set_open_iff_neighborhood_is_contained} allows us to conclude that \( X \setminus \derived(A) \) is open and, hence, \( \derived(A) \) is closed.

  \SubProofOf{thm:def:derived_set/properties/closure} Clearly \( A \subseteq \cl(A) \). Also
  \begin{equation*}
    \derived(A) \subseteq \bigcup_{x \in X} \cl(A \setminus \{ x \}) \subseteq \cl(A).
  \end{equation*}

  Now we will prove the reverse inclusion. Let \( x \in \cl(A) \). Then either \( x \in A \) or \( x \in \fr(A) \). Assume the latter. By \fullref{def:topological_boundary/neighborhoods}, every neighborhood \( U \) of \( x \) has points both in \( A \) and outside \( A \), therefore \( U \cap (A \setminus \{ x \}) \) is nonempty. By \fullref{thm:def:topological_closure_operator/properties/neighborhood_intersection}, \( x \in \cl(A \setminus \{ x \}) \), that is, \( x \in \derived(A) \).

  \SubProofOf{thm:def:derived_set/properties/closed_iff_contains_all_cluster_points} Assume that \( \derived(A) \subseteq A \) and, aiming at a contradiction, suppose that \( A \) is not closed. Fix a point \( x \in \cl(A) \setminus A \). By \fullref{thm:def:derived_set/properties/closure}, this is a cluster point. By \fullref{thm:def:derived_set/properties/cluster_via_neighborhoods}, every for neighborhood \( U \) of \( x \) the intersection \( U \cap (A \setminus \{ x \}) \subseteq U \cap A \) is nonempty. Since this holds for arbitrary neighborhoods, by \fullref{thm:def:topological_closure_operator/properties/neighborhood_intersection}, \( A \) is closed.

  \SufficiencySubProofOf{thm:def:derived_set/properties/closed_iff_only_isolated_and_cluster_points}
  Special case of \fullref{thm:def:derived_set/properties/closed_iff_contains_all_cluster_points}.
  \NecessitySubProofOf{thm:def:derived_set/properties/closed_iff_only_isolated_and_cluster_points} We already know from \fullref{thm:def:derived_set/properties/closed_iff_contains_all_cluster_points} that it is sufficient for \( \derived(A) \) to belong to \( A \) for \( A \) to be closed. But \( A \setminus \derived(A) \) consists of all isolated points, therefore every point in \( A \) is either a cluster point or an isolated point.
\end{proof}

\begin{definition}\label{def:topologically_dense_set}\mcite[25]{Engelking1989}
  Let \( (X, \mscrT) \) be a topological space and \( A \subseteq X \) be any set. We say that \( A \) is

  \begin{thmenum}
    \thmitem{def:topologically_dense_set/dense} \term{dense} in \( X \) if \( \cl{A} = X \) (if \( X \) is assumed from the context, we simply say that \( A \) is dense).

    \thmitem{def:topologically_dense_set/codense} \term{codense} in \( X \) if \( X \setminus A \) is dense, i.e. \( \cl(X \setminus A) = X \).

    \thmitem{def:topologically_dense_set/nowhere_dense} \term{nowhere dense} in \( X \) if \( \cl(A) \) is codense, i.e. \( X = \cl(X \setminus \cl A) \reloset {\ref{thm:interior_closure_complement}} = \cl(\Int(X \setminus A)) \).

    \thmitem{def:topologically_dense_set/dense_in_itself} \term{dense in itself} if \( A \subseteq \derived(A) \), i.e. if \( A \) has no isolated points.
  \end{thmenum}

  We define the \term{density} \( d(X) \) of \( X \) to be the minimum \hyperref[def:cardinal]{cardinality} of all dense sets. If \( d(X) \leq \aleph_0 \), we say that the space is \term{separable}.
\end{definition}

\begin{proposition}\label{thm:def:dense_set/properties}
  \hyperref[def:topologically_dense_set/dense]{Dense sets} have the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:dense_set/properties/open_intersection}\mcite[prop. 1.3.5]{Engelking1989} The set \( A \) is dense if and only if every nonempty open set intersects \( A \).
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:dense_set/properties/open_intersection} Special case of \fullref{thm:def:topological_closure_operator/properties/neighborhood_intersection}.
\end{proof}

\begin{proposition}\label{thm:def:nowhere_dense/properties}
  \hyperref[def:topologically_dense_set/nowhere_dense]{Nowhere dense sets} have the following basic properties:
  \begin{thmenum}
    \thmitem{thm:def:nowhere_dense/properties/empty_interior} Nowhere dense sets have an empty interior
    \thmitem{thm:def:nowhere_dense/properties/contained_in_boundary} Nowhere dense sets are entirely contained in their boundaries.
    \thmitem{thm:def:nowhere_dense/properties/interior_of_closure} The set \( A \) is nowhere dense if and only if \( \Int(\cl(A)) = \varnothing \).
    \thmitem{thm:def:nowhere_dense/properties/closure_contains_no_open_set} The set is nowhere dense if and only if its closure does not contain a nonempty open set.
    \thmitem{thm:def:nowhere_dense/properties/open_subset}\mcite[prop. 1.3.5]{Engelking1989}The set \( A \) is nowhere dense if and only if every open set contains a nonempty open subset disjoint from \( A \).
    \thmitem{thm:def:nowhere_dense/properties/subset} A subset of a nowhere dense set is nowhere dense.
    \thmitem{thm:def:nowhere_dense/properties/homeomorphism} The \hyperref[def:homeomorphism]{homeomorphic} image of a nowhere dense set is nowhere dense.
    \thmitem{thm:def:nowhere_dense/properties/complement_dense} A set is closed and nowhere dense if and only if its complement is open and dense.
  \end{thmenum}
\end{proposition}
\begin{proof}
  \SubProofOf{thm:def:nowhere_dense/properties/interior_of_closure} Follows directly from \fullref{thm:interior_closure_complement}.
  \SubProofOf{thm:def:nowhere_dense/properties/empty_interior} Follows from \fullref{thm:def:nowhere_dense/properties/interior_of_closure} because \( \Int(A) \subseteq \Int(\cl(A)) = \varnothing \).

  \SubProofOf{thm:def:nowhere_dense/properties/contained_in_boundary} Follows from \fullref{thm:def:nowhere_dense/properties/empty_interior} and \fullref{def:topological_boundary/closure}.

  \SubProofOf{thm:def:nowhere_dense/properties/closure_contains_no_open_set} By \fullref{thm:def:dense_set/properties/open_intersection}, \( A \) is nowhere dense if and only if every nonempty open set intersects \( X \setminus \cl(A) \reloset {\ref{thm:interior_closure_complement}} = \Int(X \setminus A) \). By \fullref{thm:set_open_iff_neighborhood_is_contained}, the last condition is equivalent to every nonempty open set having a nonempty open subset in \( \Int(X \setminus A) = X \setminus \cl(A) \), which in turn implies \fullref{thm:def:nowhere_dense/properties/closure_contains_no_open_set}.

  \SubProofOf{thm:def:nowhere_dense/properties/subset} Let \( A \) be a nowhere dense set and let \( B \subseteq A \). Then
  \begin{equation*}
    \Int(\cl(B))
    \reloset {\ref{thm:def:interior_operator/properties/monotone}} \subseteq
    \Int(\cl(A))
    \reloset {\ref{thm:def:nowhere_dense/properties/interior_of_closure}} =
    \varnothing,
  \end{equation*}
  therefore \( B \) is also nowhere dense.

  \SubProofOf{thm:def:nowhere_dense/properties/homeomorphism} Let \( f: X \to Y \) be a homeomorphic embedding (not necessarily surjective) and let \( A \subseteq X \) be a nowhere dense set. Let \( V \) be an open set in \( Y \). Then \( f^{-1}(V) \) is open in \( X \) and, by \fullref{thm:def:nowhere_dense/properties/open_subset}, there exists an open subset \( U \subseteq f^{-1}(V) \) that is disjoint from \( A \). Therefore, \( f(U) \subseteq f(f^{-1}(V)) \reloset {\ref{thm:function_image_preimage_composition/image_of_preimage}} \subseteq V \). Furthermore, \( f(U) \) is open and \( f(U) \cap f(A) \reloset {\ref{thm:def:function_image/properties/intersection}} = f(U \cap A) = f(\varnothing) = \varnothing \), therefore \( f(A) \) is nowhere dense.

  \SubProofOf{thm:def:nowhere_dense/properties/complement_dense} If \( A \) is an open dense set, then \( X \setminus A \) is closed and
  \begin{equation*}
    \cl(X \setminus \cl(X \setminus A))
    =
    \cl(X \setminus (X \setminus A))
    =
    \cl(A)
    =
    X,
  \end{equation*}
  therefore \( X \setminus A \) is nowhere dense.
\end{proof}

\begin{definition}\label{def:borel_algebra}
  Fix a topological space \( X \) and \( \mscrF \subseteq \pow(X) \). Denote by \( \mscrF_\delta \) the family of all countable intersections of elements of \( \mscrF \) and by \( \mscrF_\sigma \) the family of all countable unions of elements of \( \mscrF \).

  The family \( F_\delta \) is the family of countable unions of closed sets and \( G_\sigma \) is the family of countable intersections of open sets.
\end{definition}


\section{Hierarchical Basis and Hierarchical Subspace}
\label{sec:22hierSubspaces}

\minitoc{67mm}{5}

\noindent
The dimension of the nodal space $\ns{\*l}$ is given by
\begin{equation}
  \label{eq:dimensionFG}
  \dim \ns{\*l}
  = \setsize{\fgset{\*l}}
  = \prod_{t=1}^d (2^{l_t} + 1).
\end{equation}
If we choose the same level $n \in \natz$ in all dimensions,
then the dimension of $\ns{n,d}$ and the
number of grid points grow at least as fast as
$2^{nd} = (\ms{n}^{-1})^d$.
This exponential dependency between $\dim \ns{n,d}$ and $d$ is known as the
\term{curse of dimensionality} \cite{Bellman61Adaptive}.
The curse makes interpolation on $\ns{\*l}$ computationally infeasible
for dimensionalities $d > 4$,
as we would have to calculate and store
$\dim(\ns{\*l})$-many coefficients $\interpcoeff{\*l,\*i}$.%

\subsection{Hierarchical Splitting in the Univariate Case}
\label{sec:221hierUV}

\paragraph{Hierarchical subspaces}

In order to reduce the computational effort,
we first split $\ns{\*l}$ into smaller subspaces and then identify
subspaces that we can omit at the cost of a slightly larger error.
In the univariate case, the key observation is that a grid point of a
level $l$ can be written as a grid point of a higher level~$l'$:
\begin{equation}
  \label{eq:rewriteGridPoint}
  \gp{l,i} = \gp{l',i'},\quad
  l' \ge l,\quad
  i' = 2^{l'-l} i.
\end{equation}
Conversely, this implies that every grid point $\gp{l,i}$ of level $l \ge 1$
and index $i \ge 1$ can be uniquely written
as a grid point of a coarser level $l'$ (or $l' = l$) and an odd index $i'$:
\begin{equation}
  \gp{l,i} = \gp{l',i'},\quad
  l' = l - \bracket*{\log_2(\xor(i, i-1) + 1) - 1},\quad
  i' = 2^{l'-l} i,
\end{equation}
where $\xor$ is the bitwise ``exclusive or'' function.
The term in square brackets is the exponent of the
highest power of two that divides $i$.
The two boundary points zero and one are obtained by
inserting an additional level $l' = 0$ with indices $i' \in \{0, 1\}$.
As shown in \cref{fig:pointSplittingUniform},
this implies that $\fgset{l}$ decomposes into
\begin{equation}
  \fgset{l}
  = \bigdotcup_{l'=0}^l \{\gp{l',i'} \mid i' \in \hiset{l'}\},\quad
  \hiset{l'} \ceq
  \begin{cases}
    \{i' = 0, \dotsc, 2^{l'} \mid \text{$i'$ odd}\},&l' > 0,\\
    \{0, 1\},&l' = 0,
  \end{cases}
\end{equation}
where $\dotcup$ indicates the disjoint union.
We call the spaces spanned by the basis functions that correspond to the
index sets $\hiset{l'}$ \term{hierarchical subspaces} $\hs{l'}$:
\begin{equation}
  \hs{l'}
  \ceq \spn\{\basis{l',i'} \mid i' \in \hiset{l'}\}.
\end{equation}
The corresponding basis functions
$\basis{l',i'}$, $l' = 0, \dotsc, l$, $i' \in \hiset{l'}$,
are called \term{hierarchical basis functions.}
The hierarchical hat function basis is shown in \cref{fig:hierarchicalHat}.

\begin{SCfigure}
  \includegraphics{pointSplitting_1}%
  \caption[%
    Decomposition of the set of univariate grid points%
  ]{%
    The set of grid points $\fgset{l}$ of level $l = 4$ \emph{(top)}
    decomposes into hierarchical grids of level $l' \le l$,
    whose grid points $\gp{l',i'}$ have odd indices $i' \in \hiset{l'}$
    ($\gp{0,0}$ being the only exception).%
  }%
  \label{fig:pointSplittingUniform}%
\end{SCfigure}

\begin{figure}
  \subcaptionbox{%
    Basis functions $\bspl{l',i'}{1}$ ($l' \le l$, $i' \in \hiset{l'}$)
    and grid points $\gp{l',i'}$ \emph{(dots).}
    The domain is the unit interval $\clint{0, 1}$.%
  }[72mm]{%
    \includegraphics{hierarchicalBasis_2}%
  }%
  \hfill%
  \subcaptionbox{%
    Piecewise linear interpolant $\fgintp{l}$
    of some function data $\objfun(\gp{\*l,\*i})$
    as a linear combination of hierarchical hat functions \emph{(stacked).}
    The two boundary functions are combined to a single function
    \emph{\textcolor{C0}{(blue)}} for simplicity.%
  }[72mm]{%
    \includegraphics{interpolant_2}%
  }%
  \caption[%
    Univariate hierarchical hat functions%
  ]{%
    Univariate hierarchical hat functions up to level $l = 3$.%
  }%
  \label{fig:hierarchicalHat}%
\end{figure}

\paragraph{Hierarchical splitting}

For the hat function basis $\bspl{l,i}{1}$ and other basis types,
we can prove that the corresponding nodal space
decomposes into the direct sum of all
hierarchical subspaces of coarser levels or the same level, i.e.,
\begin{equation}
  \label{eq:hierSplittingUV}
  \ns{l}
  \overset{?}{=} \bigoplus_{l'=0}^l \hs{l'}.
\end{equation}
We call this relation \term{hierarchical splitting.}
Here, the direct sum $\oplus$ is
the vector space sum that additionally indicates
that the dimension of the sum $\sum_{l'=0}^l \hs{l'}$ is the sum
of the dimensions of the summands $\hs{l'}$
(analogously to
$\setsize{\fgset{l}}
= \sum_{l'=0}^l \setsize{\{\gp{l',i'} \mid i' \in \hiset{l'}\}}$,
where $\fgset{l}$ is the disjoint union of the sets
$\{\gp{l',i'} \mid i' \in \hiset{l'}\}$).
In general, \eqref{eq:hierSplittingUV} may not be true,
depending on the type of basis functions.
The following lemma provides a characterization
that can be used to prove \eqref{eq:hierSplittingUV} for hat functions.

\vspace*{\fill}
\pagebreak

\begin{lemma}[univariate hierarchical splitting characterization]
  \label{lemma:hierSplittingUV}
  \Cref{eq:hierSplittingUV} is equivalent to the satisfaction of
  both of the following conditions:
  \begin{itemize}
    \item
    The hierarchical subspaces $\hs{l'}$ ($l' \le l$)
    are subspaces of $\ns{l}$.

    \item
    The hierarchical functions
    $\basis{l',i'}$ ($l' \le l$, $i' \in \hiset{l'}$)
    are linearly independent.
  \end{itemize}
\end{lemma}
\begin{proof}
  The first condition is equivalent to $\sum_{l'=0}^l \hs{l'} \subset \ns{l}$.
  The second condition is equivalent to
  $\dim \sum_{l'=0}^l \hs{l'} = \sum_{l'=0}^l \dim \hs{l'}$,
  i.e., to the directness of the sum.
  Therefore, the logical conjunction of both is equivalent to
  $\bigoplus_{l'=0}^l \hs{l'} \subset \ns{l}$.
  If the sum is direct,
  the dimension of the sum is equal to $2 + \sum_{l'=1}^l 2^{l'-1} = 2^l + 1$
  (due to $\dim \hs{l'} = \setsize{\hiset{l'}} = 2^{l'-1}$ for $l' > 0$ and
  $\dim \hs{l'} = 2$ for $l' = 0$),
  which is also the dimension of $\ns{l}$.
  The only subspace of $\ns{l}$ that has the same
  dimension as $\ns{l}$ is $\ns{l}$ itself,
  so we infer $\bigoplus_{l'=0}^l \hs{l'} = \ns{l}$.
\end{proof}
\begin{corollary}[univariate hierarchical splitting for hat functions]
  \label{cor:hierSplittingHatUV}
  The hierarchical splitting \eqref{eq:hierSplittingUV}
  holds for the hat function basis.
\end{corollary}
\begin{proof}
  The first condition of \cref{lemma:hierSplittingUV}
  is satisfied as piecewise linear splines of level~$l'$
  are also piecewise linear splines of higher levels $l \ge l'$.
  We can prove the linear independence for the second condition by induction
  over $l$:
  If a linear combination of $\bspl{l',i'}{1}$
  ($l' \le l$, $i' \in \hiset{l'}$)
  vanishes everywhere, then the coefficients of level $l$ must be zero,
  as otherwise the basis functions $\bspl{l,i'}{1}$ ($i' \in \hiset{l}$) would
  introduce kinks at $\gp{l,i'}$, which the zero function does not have.
  This means that we have a zero linear combination of $\bspl{l',i'}{1}$ for
  $l' \le l - 1$, $i' \in \hiset{l'}$,
  and by the induction hypothesis, the other coefficients also vanish.
\end{proof}

\subsection{Hierarchical Splitting in the Multivariate Case}
\label{sec:222hierMV}

Multivariate hierarchical subspaces are defined analogously
to the univariate case:
\begin{equation}
  \hs{\*l}
  \ceq \spn\{\basis{\*l,\*i} \mid \*i \in \hiset{\*l}\},\quad
  \hiset{\*l}
  \ceq \hiset{l_1} \times \dotsb \times \hiset{l_d},\quad
  \*l \in \natz^d.
\end{equation}
The univariate hierarchical splitting \eqref{eq:hierSplittingUV}
can now be generalized to
\begin{equation}
  \label{eq:hierSplittingMV}
  \ns{\*l}
  \overset{?}{=} \bigoplus_{\*l'=\*0}^\*l \hs{\*l'}.
\end{equation}
Again, this relation does not hold in general.
We use a multivariate counterpart of \thmref{lemma:hierSplittingUV}
to prove that \eqref{eq:hierSplittingMV} holds if
the corresponding univariate relation \eqref{eq:hierSplittingUV}
holds for all dimensions:

\begin{lemma}[multivariate hierarchical splitting characterization]
  \label{lemma:hierSplittingMV}
  \Cref{eq:hierSplittingMV} is equivalent to the satisfaction of
  both of the following conditions:
  \begin{itemize}
    \item
    The hierarchical subspaces $\hs{\*l'}$ ($\*l' \le \*l$)
    are subspaces of $\ns{\*l}$.

    \item
    The basis functions
    $\basis{\*l',\*i'}$ ($\*l' \le \*l$, $\*i' \in \hiset{\*l'}$)
    are linearly independent.
  \end{itemize}
\end{lemma}

\vspace*{0pt plus 0.3fill}

\begin{proof}
  If the sum is direct, then its dimension is given by
  \begin{equation}
    \hspace*{-5mm}
    \dim \sum_{\*l'=\*0}^\*l \hs{\*l'}
    = \sum_{l_1'=0}^{l_1} \dotsb \sum_{l_d'=0}^{l_d}
    \prod_{t=1}^d \dim \hs{l_t'}
    = \prod_{t=1}^d \sum_{l_t'=0}^{l_t} \dim \hs{l_t'}
    = \prod_{t=1}^d (2^{l_t} + 1)
    = \dim \ns{\*l}
    \hspace*{-5mm}
  \end{equation}
  using \eqref{eq:dimensionFG}.
  The rest is analogous to the proof of \cref{lemma:hierSplittingUV}.
\end{proof}

\vspace*{0pt plus 1fill}

\begin{proposition}[from univariate to multivariate splitting]
  \label{prop:splittingUVToMV}
  If univariate splitting \eqref{eq:hierSplittingUV}
  holds for every dimension,
  then the multivariate splitting \eqref{eq:hierSplittingMV} holds as well.
\end{proposition}

\vspace*{0pt plus 0.3fill}

\begin{proof}
  We check the two conditions of \cref{lemma:hierSplittingMV}
  given the two univariate conditions of \cref{lemma:hierSplittingUV}:
  \begin{enumerate}
    \item
    The hierarchical basis functions $\basis{\*l',\*i'}$
    of $\hs{\*l'}$ ($\*l' \le \*l$, $\*i' \in \hiset{\*l'}$)
    are tensor products of functions $\basis{l_t',i_t'}$.
    According to the first condition of \cref{lemma:hierSplittingUV},
    each $\basis{l_t',i_t'}$ can be written as a linear combination of
    the nodal basis $\basis{l_t,i_t}$ ($i_t = 0, \dotsc, 2^{l_t}$).
    We can expand the tensor product to a linear combination
    of tensor products of the univariate nodal basis functions.
    Therefore, $\basis{\*l',\*i'}$ is a linear combination of
    multivariate nodal functions, i.e., $\basis{\*l',\*i'} \in \ns{\*l}$.
    As this is true for all $\*i' \in \hiset{\*l'}$, we obtain
    $\hs{\*l'} \subset \ns{\*l}$.

    \item
    The linear independence of the hierarchical functions $\basis{\*l',\*i'}$
    ($\*l' \le \*l$, $\*i' \in \hiset{\*l'}$) can be shown completely
    analogously to the proof of
    \thmref{lemma:tensorProductLinearIndependence}.
  \end{enumerate}
  According to \cref{lemma:hierSplittingMV},
  the multivariate splitting \eqref{eq:hierSplittingMV} holds.
\end{proof}

\vspace*{0pt plus 1fill}

A direct consequence of \cref{prop:splittingUVToMV} is that
the hierarchical splitting holds for the hierarchical hat function basis.

\pagebreak

\begin{corollary}[multivariate hierarchical splitting for hat functions]
  \label{cor:hierSplittingHatMV}
  The multivariate hierarchical splitting \eqref{eq:hierSplittingMV}
  holds for the hat function basis.
\end{corollary}
\begin{proof}
  Follows directly by applying
  \thmref{cor:hierSplittingHatUV} to \cref{prop:splittingUVToMV}.
\end{proof}


\subsection{Autoregressive models (AR)}

\subsubsection{AR(\(1\))}

Our basic model was:

\(x_t=\alpha + \epsilon_t \)

We add an autoregressive component by adding a lagged observation.

\(x_t=\alpha + \beta x_{t-1}+\epsilon_t\)

\subsubsection{AR(\(p\))}

AR(\(p\)) has \(p\) previous dependent variables.

\(x_t=\alpha + \sum_{i=1}^p\beta_ix_{t-i}\)

\subsubsection{Propagation of shocks}

A shock bumps up the output variable, which bumps up output variables forever, at a decreasing rate.


\chapter{}

\begin{ex}
	Let \(X_{1}, \cdots, X_{n}\) be a random sample from the a.c. distribution
	\[
	f(x \mid \theta)=\theta x^{\theta-1} 1_{\{0 \leqslant x \leqslant 1\}}, \quad 0<\theta<\infty .
	\]
	Find the method of moments estimator of \(\theta\).
\end{ex}

\begin{solution}
	For the sample \(\{X_i\}_{i=1}^n\),
	\[
		\begin{aligned}
			\bar{X}=\mu_1=E(X_i)&=\int_0^1xf(x)\der x=\int_0^1\theta x^\theta\der x\\
			&=\frac{\theta}{\theta+1}x^{\theta+1}\big|_0^1=\frac{\theta}{\theta+1}.
		\end{aligned}
	\]
	Then, the method of moments estimator for $\theta$ is
	\[
		\hat{\mu}_1=\frac{1}{n}\sum_{i=1}^nX_i=\frac{\theta}{\theta+1}
	\]
	\[
		\hat{\theta}=\frac{\hat{\mu}_1}{1-\hat{\mu}_1}.
	\]
\end{solution}

\begin{ex}
	Let \(X_{1}, \cdots, X_{n} \stackrel{\text { i.i.d. }}{\sim} \operatorname{Geo}(p), 0<p<1\), i.e.,
	\[
	\mathbb{P}\left(X_{1}=k \mid p\right)=(1-p)^{k-1} p, \quad k \in \mathbb{N} .
	\]
	What is the method of moments estimator of \(p\) ?
\end{ex}

\begin{solution}
	For the sample \(\{X_i\}_{i=1}^n\),
	\[
		\begin{aligned}
			\bar{X}=\mu_1=E(X_i)&=\sum_{k=1}^\infty k\cdot (1-p)^{k-1}p=\frac{1}{p}
		\end{aligned}
	\]
	So, the method of moments estimator of $p$ is
	\[
		\hat{\mu}_1=\frac{1}{n}\sum_{i=1}^nX_i=\frac{1}{p}
	\]
	\[
		\hat{p}=\frac{1}{\hat{\mu}_1}.
	\]
\end{solution}

\begin{ex}
	Assume \(X \sim B(n, p)\), where both \(n \in \mathbb{N}\) and \(p \in(0,1)\) are unknown. Given a random sample of \(N\) observations of \(X\), compute the method of moments estimator of \(n\) and \(p\).
\end{ex}

\begin{solution}
	For the sample \(\{X_i\}_{i=1}^N\),
	\[
		\bar{X}=\mu_1=E(X_i)=\sum_{k=0}^nk\cdot \binom{n}{k}p^k(1-p)^{n-k}=np,
	\]
	\[
		Var(X_i)=E(X_i^2)-(E(X_i))^2=\mu_2-\mu^2=np(1-p),
	\]
	Now, solve these two equations:
	\[
		\left\{\begin{array}{l}
			\hat{\mu}_1=np\\
			\hat{\mu}_2-\hat{\mu}_1^2=np(1-p)
		\end{array}\right.
	\]
	We can get that
	\[
		\hat{p}=1-\frac{\hat{\mu}_2-\hat{\mu}_1^2}{\hat{\mu}_1},\qquad\hat{n}=\frac{\hat{\mu}_1^2}{\hat{\mu}_1-\hat{\mu}_2+\hat{\mu}_1^2}.
	\]
\end{solution}

\begin{ex}
	Let \(X_{1}, \cdots, X_{n}\) be a random sample from the density
	\[
	f(x \mid \theta)=\theta x^{-2}, \quad 0<\theta \leqslant x<\infty .
	\]
	What can you say about the existence of the method of moments estimator of \(\theta\) ?
\end{ex}

\begin{solution}
	\[
		E(X_i)=\int_{\theta}^\infty x\cdot \theta x^{-2}\der x=\theta \ln x \big|_\theta^\infty=\theta(\ln\infty-\ln\theta)=\infty,
	\]
	So, the method of moments estimator for \(\theta\) does not exist.
\end{solution}

\begin{ex}
	Show that each of the following is an exponential family and describe the natural parameter space \(\mathcal{E}\) of the associated canonical exponential family.
	\begin{enumerate}[(a)]
		\item \(\{\Gamma(\alpha, \lambda)\}_{\alpha, \lambda>0}\), i.e.,
		\[
		f(x \mid \alpha, \lambda)=\frac{\lambda^{\alpha}}{\Gamma(\alpha)} x^{\alpha-1} e^{-\lambda x} 1_{\{x>0\}} ;
		\]
		\item \(\{\operatorname{Beta}(a, b)\}_{a, b>0}\), i.e.,
		\[
		f(x \mid a, b)=\frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} x^{a-1}(1-x)^{b-1} 1_{\{0<x<1\}} ;
		\]
		\item \(\{\operatorname{Poi}(\lambda)\}_{\lambda>0}\);
		\item \(\{\operatorname{NegBin}(r, p)\}_{0 \leqslant p \leqslant 1}(r\) being known), i.e.,
		\[
		f(x \mid p)=\left(\begin{array}{c}
		r+x-1 \\
		x
		\end{array}\right) p^{r}(1-p)^{x}, \quad x \in \mathbb{N} \cup\{0\} .
		\]
	\end{enumerate}
\end{ex}

\begin{solution}
	\begin{enumerate}[(a)]
		\item
		\[
			\begin{aligned}
				f(x|\alpha, \lambda)&=1_{\{x>0\}}\exp(\alpha\ln(\lambda)-\ln(\Gamma(\alpha))+(\alpha-1)\ln(x)-\lambda x)\\
				&=1_{\{x>0\}}\exp((\alpha-1)\ln(x)-\lambda x+\alpha\ln(\lambda)-\ln(\Gamma(\alpha))),
			\end{aligned}
		\]
		So, the nature parameter is $(\eta_1,\eta_2)=(\alpha-1, -\lambda)$. And the parameter space \(\mathcal{E}\) is \((-1,\infty)\times(-\infty,0)\).
		\item
		\[
			\begin{aligned}
				f(x|a,b)=1_{\{0<x<1\}}&\exp((a-1)\ln(x)+(b-1)\ln(1-x)\\&+\ln(\Gamma(a+b))-\ln(\Gamma(a))-\ln(\Gamma(b))),
			\end{aligned}
		\]
		So, the nature parameter is $(\eta_1,\eta_2)=(a-1, b-1)$. And the parameter space \(\mathcal{E}\) is \((-1,\infty)\times(-1,\infty)\).
		\item When \(k\in\mathbb{N}\),
		\[
			f(x)=P(X=k)=\frac{e^{-\lambda}\lambda^k}{k!}
		\]
		\[
			\begin{aligned}
				P(X=k)=&1/k!\cdot\exp(-\lambda+k\ln(\lambda)),
			\end{aligned}
		\]
		So, the nature parameter is $(\eta)=(\ln(\lambda))$. And the parameter space \(\mathcal{E}\) is \((-\infty,\infty)\).
		\item When \(x\in\mathbb{N}\),
		\[
			\begin{aligned}
				f(x|p)&=\binom{r+x-1}{x}\exp\left(r\ln(p)+x\ln(1-p)\right),
			\end{aligned}
		\]
		So, the nature parameter is $(\eta)=(\ln(1-p))$. And the parameter space \(\mathcal{E}\) is \((-\infty,0]\).
	\end{enumerate}
\end{solution}

\begin{ex}
	In \(n\) independent trials with \(k+1\) possible outcomes, let the probability of the \(i\)-th outcome be \(p_{i}\) in each trial. If \(X_{i}\) denotes the number of trials resulting in outcome \(i, i=0,1, \ldots, k\), then the joint distribution of \(X\) is the multinomial distribution, i.e.,
	\[
		f( x \mid \mathbf{p}):=\mathbb{P}\left(X_{0}=x_{0}, X_{1}=x_{1}, \ldots, X_{k}=x_{k} \mid \mathbf{p}\right)=\frac{n !}{x_{0} ! x_{1} ! \ldots x_{k} !} p_{0}^{x_{0}} p_{1}^{x_{1}} \ldots p_{k}^{x_{k}}.
	\]
	\begin{enumerate}[(a)]
		\item Verify that \(\{f( x \mid \mathbf{p})\}_{\mathbf{p} \in \Pi_{k}}\) is an exponential family, where \(\Pi_{k}\) is the \(k\)-dimensional simplex in \(\mathbb{R}^{k+1}\). Determine \(\mathcal{E}\).
		\item Using results for canonical exponential families, show that
		\[
			\mathbb{E} X_{i}=n p_{i}, \quad \operatorname{Cov}\left(X_{j}, X_{i}\right)=\left\{\begin{array}{cc}
			n p_{j}\left(1-p_{j}\right), & k=j \\
			-n p_{j} p_{k}, & k \neq j
		\end{array}\right.
		\]
	\end{enumerate}
\end{ex}

\begin{solution}
	\begin{enumerate}[(a)]
		\item We know for every $p_i$, \(0\leqslant p_i\leqslant 1\).
		\[
			f(x|\mathbf{p})=\frac{n !}{x_{0} ! x_{1} ! \ldots x_{k} !} \exp\left(x_0\ln (p_0)+\cdots+x_k\ln (p_k)\right).
		\]
		So, the nature parameter is $(\eta_0,\cdots,\eta_k)=(\ln(p_0),\cdots,\ln(p_k))$. And the parameter space \(\mathcal{E}\) is \((-\infty,0]\times\cdots\times(-\infty,0]\).
		\item Noting this $p_{k}=1-\sum_{i=0}^{k-1}p_i$, we can parameterize the distribution as:
		\[
			\begin{aligned}
				f(x|\mathbf{p})&=\frac{n!}{x_0!x_1!\cdots x_k!}\exp\left\{\sum_{i=0}^{k-1}\ln(p_i)x_i+\left(n-\sum_{i=0}^{k-1}x_i\right)\ln\left(1-\sum_{i=0}^{k-1}p_i\right)\right\}\\
				&=\frac{n!}{x_0!x_1!\cdots x_k!}\exp\left\{\sum_{i=0}^{k-1}\ln\left(\frac{p_i}{1-\sum_{i=0}^{k-1}p_i}\right)x_i+n\ln\left(1-\sum_{i=0}^{k-1}p_i\right)\right\}.
			\end{aligned}
		\]
		Let $\eta_i=\ln\left(\frac{p_i}{1-\sum_{i=0}^{k-1}p_i}\right)$, then we know $p_i=\frac{e^{\eta_i}}{\sum_{i=0}^ke^{\eta_i}}$. So,
		\[
			\begin{aligned}
				A(\eta)=n\ln\left(\sum_{i=0}^ke^{\eta_i}\right).
			\end{aligned}
		\]
		So,
		\[
			\mathbb{E}X_i=A_{\eta_i}'(\eta)=n\frac{e^{\eta_i}}{\sum_{i=0}^ke^{\eta_i}}=np_i.
		\]
		When $i=j$,
		\[
			\text{Cov}(X_j,X_i)=A_{\eta_i}''(\eta)=n\frac{e^{\eta_i}(\sum_{i=0}^k)e^{\eta_i}-e^{\eta_i}e^{\eta_i}}{\sum_{i=0}^ke^{\eta_i}}=np_i(1-p_i).
		\]
		When $i\not=j$,
		\[
			\text{Cov}(X_j,X_i)=\frac{A(\eta)}{\partial\eta_i\,\partial\eta_j}=ne^{\eta_i}\frac{-e^{\eta_j}}{(\sum_{i=0}^ke^{\eta_i})^2}=-np_ip_j.
		\]
	\end{enumerate}
\end{solution}

\begin{ex}
	The Inverse Gaussian density, \(I G(\mu, \lambda)\), is given by
	\[
		f(x \mid \mu, \lambda)=\left(\frac{\lambda}{2 \pi}\right)^{1 / 2} x^{-3 / 2} \exp \left\{\frac{-\lambda(x-\mu)^{2}}{2 \mu^{2} x}\right\}, \quad x>0, \quad \mu>0, \quad \lambda>0.
	\]
	\begin{enumerate}[(a)]
		\item Show that this is an exponential family generated by \(T(X)=-\frac{1}{2}\left(X, X^{-1}\right)^{T}\) and \(h(x)=\) \((2 \pi)^{-1 / 2} x^{-3 / 2}\).
		\item Show that the canonical parameters \(\eta_{1}, \eta_{2}\) are given by \(\eta_{1}=\mu^{-2} \lambda, \eta_{2}=\lambda\), and that \(A\left(\eta_{1}, \eta_{2}\right)=-\left(\frac{1}{2} \log \left(\eta_{2}\right)+\sqrt{\eta_{1} \eta_{2}}\right), \mathcal{E}=[0, \infty) \times(0, \infty)\).
		\item Find the moment generating function of \(T\) and show that \(\mathbb{E}(X)=\mu, \operatorname{Var}(X)=\mu^{3} \lambda^{-1}\), \(\mathbb{E}\left(X^{-1}\right)=\mu^{-1}+\lambda^{-1}, \operatorname{Var}\left(X^{-1}\right)=(\lambda \mu)^{-1}+2 \lambda^{-2} .\)
	\end{enumerate}
\end{ex}

\begin{solution}
	\begin{enumerate}[(a)]
		\item \[
			\begin{aligned}
				f(x|\mu,\lambda)&=(2 \pi)^{-1 / 2} x^{-3 / 2}\exp\left\{-\lambda(x^2-2\mu x+\mu^2)\cdot(2\mu^2x)^{-1}+\frac{1}{2}\ln(\lambda)\right\}\\
				&=(2 \pi)^{-1 / 2} x^{-3 / 2}\exp\left\{\frac{-\lambda}{2\mu^2}x + \frac{\lambda}{\mu}-\frac{\lambda}{2}x^{-1}+\frac{1}{2}\ln(\lambda)\right\}
			\end{aligned}
		\]
		So, $T(X)=-\frac{1}{2}(X,X^{-1})$, \(h(x)=\) \((2 \pi)^{-1 / 2} x^{-3 / 2}\).
		\item $B((\lambda, \mu))=-(1/2\ln(\lambda)+\lambda/\mu)$, and $\eta=(\eta_1,\eta_2)=(\lambda/\mu^2, \lambda)$. So,
		\[
			A(\eta)=-(1/2\ln(\eta_2)+(\eta_1\eta_2)^{1/2}).
		\]
		\item The moment-generating function of $T(X)$ is
		\[
			M(s)=\exp\left(A(s+\eta)-A(\eta)\right).
		\]
		And
		\[
			\mathbb{E}(-1/2X)=A_{\eta_1}'(\eta)=-(1/2(\eta_1\eta_2)^{-1/2}\eta_2)=-1/2\mu,\quad \mathbb{E}X=\mu.
		\]
		\[
			\text{Var}(-1/2X)=A_{\eta_1}''(\eta)=1/4(\eta_1\eta_2)^{-3/2}\eta_2^2=1/4\mu^3\lambda^{-1},\quad \text{Var}(X)=\mu^3\lambda^{-1}.
		\]
		\[
			\mathbb{E}(-1/2X^{-1})=A_{\eta_2}'(\eta)=-(1/2\eta_2^{-1}+1/2(\eta_1\eta_2)^{-1/2}\eta_1)=-1/2(\mu^{-1}+\lambda^{-1}),
		\]
		\[
			\mathbb{E}(X^{-1})=\mu^{-1}+\lambda^{-1}.
		\]
		\[
			\text{Var}(-1/2X^{-1})=A_{\eta_2}''(\eta)=1/2\eta_2^{-2}+1/4(\eta_1\eta_2)^{-3/2}\eta_1^2=1/2\lambda^{-2}+1/4(\lambda^{-1}\mu^{-1}),
		\]
		\[
			\text{Var}(X^{-1})=2\lambda^{-2}+(\lambda\mu)^{-1}.
		\]
	\end{enumerate}
\end{solution}

\begin{ex}
	For each of the following families: \((i)\) verify that it is an exponential family; \((i i)\) describe the curve in which the parameter vector \(\theta\) lies (with respect to the natural parametrization).
	\begin{enumerate}[(a)]
		\item \(\left\{\mathcal{N}\left(\theta, a \theta^{2}\right)\right\}_{\theta \in \mathbb{R}}, a\) is known;
		\item \(\{\Gamma(\alpha, \alpha)\}_{\alpha>0} ;\)
		\item \(\{f(x \mid \theta)\}_{\theta \in \mathbb{R}}\), where \(f(x \mid \theta)=C \exp \left\{-(x-\theta)^{4}\right\}, x \in \mathbb{R}\), and \(C\) is a normalizing constant.
	\end{enumerate}
\end{ex}

\begin{solution}
	\begin{enumerate}[(a)]
		\item
		\begin{enumerate}[(a)]
			\item[(i)] The density function of normal distribution can be write as:
			\[
				\begin{aligned}
					f(x|\theta)&={\frac {1}{{\sqrt {2\pi a}\theta}}}\;\exp\left(-{\frac {\left(x-\theta \right)^{2}}{2a\theta^2}}\right)\\
					&=e^{-\frac{1}{2a}}\exp\left(-\frac{1}{2a\theta^2}x^2+\frac{1}{a\theta}x-\ln(\sqrt{2\pi a}\theta)\right).
				\end{aligned}
			\]
			$h(x)=e^{-\frac{1}{2a}}1_{-\infty<x<\infty}$, $\eta=\left(-\frac{1}{2a\theta^2}, \frac{1}{a\theta}\right)$, $T(X)=(x^2, x)$.
			\item[(ii)]
			Let $u=-\frac{1}{2a\theta^2}$, $v=\frac{1}{a\theta}$, then
			\[
				u=-\frac{a}{2}v^2.
			\]
			It is a parabola.
		\end{enumerate}
		\item
		\begin{enumerate}[(a)]
			\item[(i)]
			\[
				\begin{aligned}
					f(x \mid \alpha, \alpha)&=\frac{\alpha^{\alpha}}{\Gamma(\alpha)} x^{\alpha-1} e^{-\alpha x} 1_{\{x>0\}}\\
					&=\exp\left(-\alpha x+(\alpha-1)\ln(x)+\alpha\ln(\alpha)-\ln(\Gamma(\alpha))\right)1_{\{x>0\}}
				\end{aligned}
			\]
			$h(x)=1_{\{x>0\}}$, $T(X)=(x, \ln(x))$, $\eta=(-\alpha,\alpha-1)$.
			\item[(ii)]
			Let $u=-\alpha$, $v=\alpha-1$, then
			\[
				v=-u-1,\quad u<0.
			\]
			It is a line.
		\end{enumerate}
		\item
		\begin{enumerate}[(a)]
			\item[(i)]
			\[
				\begin{aligned}
					f(x|\theta)&=C\exp(-(x-\theta)^4)\\
					&=C\exp\left(-x^4+4\theta x^3-6\theta^2x^2+4\theta^3x-\theta^4\right)
				\end{aligned}
			\]
			$h(x)=C\exp(-x^4)1_{\{-\infty<x<\infty\}}$, $T(X)=(4x^3, -6x^2, 4x)$, $\eta=(\theta, \theta^2, \theta^3)$.
			\item[(ii)] It is a spiral. In Mathematica, we can get the plot of the curve like this:
\begin{minted}[breaklines]{mathematica}
ParametricPlot3D[{t, t^2, t^3}, {t, 0, 1}, {ViewPoint -> {1.3, -2.4, 2.}}]
\end{minted}
		\includegraphics{1.8.c.eps}
	\end{enumerate}
	\end{enumerate}
\end{solution}

\begin{ex}
	(In this problem, we generalize the proposition shown in class)

	Let \(\{f( x \mid \theta)\}_{\theta \in \Theta}\) be a \(k\) parameter exponential family generated by \((T, h)\). Show that the distribution for \(T(\mathbf{X})\) is also a \(k\)-parameter exponential family. Hint: let \(G_{*}(d  x)=h( x) G(d  x)\) and define the induced probability measure
	\[
		\mathbb{P}_{T}(B):=\mathbb{P}(T(\mathbf{X}) \in B)=\int_{\mathbb{R}^{n}} 1_{B}(T( x)) \exp \{\langle W(\theta), T( x)\rangle-B(\theta)\} G_{*}(d  x),
	\]
	for \(B \in \mathcal{B}\left(\mathbb{R}^{k}\right)\). Now recall the following fundamental result on changes of measure, established in the course Probability theory \(I\).

	{\bfseries Theorem. (Meerschaert and Scheffler (2001), \(p .4)\)} If \(\mu(d x)\) is a measure on \(\mathcal{B}\left(\mathbb{R}^{d}\right)\) and if \(T: \mathbb{R}^{d} \rightarrow \mathbb{R}^{m}, f: \mathbb{R}^{m} \rightarrow \mathbb{R}^{n}\) are Borel measurable, then
	\[
		\int_{\mathbb{R}^{d}} f(T(\boldsymbol{x})) \mu(d \boldsymbol{x})=\int_{\mathbb{R}^{m}} f(\boldsymbol{y})(T \mu)(d \boldsymbol{y}),
	\]
	where we define the measure \((T \mu)(B)=\mu\left(T^{-1}(B)\right), B \in \mathcal{B}\left(\mathbb{R}^{m}\right)\).
\end{ex}

\begin{solution}
	Knowing that $f(x|\theta)$ is generated by $(T,h)$, i.e.
	\[
		f(x|\theta)=h(x)\exp\left(\eta^T(\theta)T(x)-B(\theta)\right),
	\]
	And we suppose $T(x)$ is a one-to-one function. Then the p.d.f. for $T(x)$ is
	\[
		\begin{aligned}
			\mathbb{P}_{T}(B)&=\int_{\mathbb{R}^{n}} 1_{B}(T( x)) \exp \{\langle W(\theta), T( x)\rangle-B(\theta)\} G_{*}(\der  x)\\
			&=\int_{\mathbb{R}^m}1_{B}(t) \exp \{\langle W(\theta), t\rangle-B(\theta)\}( TG_{*})(\der t)\\
			&=\int_{\mathbb{R}^m}1_{B}(t) \exp \{\langle W(\theta), t\rangle-B(\theta)\}G_*(\der T^{-1}(t))\\
			&=\int_{\mathbb{R}^m}1_{B}(t) \exp \{\langle W(\theta), t\rangle-B(\theta)\}\frac{h(T^{-1}(t))}{\|T'(T^{-1}(t))\|}G(\der t),
		\end{aligned}
	\]
	where $\|\cdot\|$ denotes the Jacobian matrix. So, the distribution of $T(x)$ is also a $k$-parameter exponential family.
\end{solution}

\begin{ex}
	Let \(\mathcal{E} \neq \emptyset\) be the natural parameter set of some canonical \(k\)-parameter exponential family, \(k \in \mathbb{N}\). Show that \(\operatorname{int}(\mathcal{E}) \neq \emptyset\) if and only if \(\mathcal{E}\) is not contained in any \((k-1)\)-dimensional hyperplane (suggestion: for the converse, set \(k=3\) to help visualize the problem and use the fact that \(\mathcal{E}\) is convex).
\end{ex}

\begin{solution}
	% For $k=3$, if $\mathcal{E}$ is contained in a $2$ dimension plane,
	For $k=3$, it is clear, but how to solve $k>3$?
\end{solution}

\begin{ex}
	Let \(\{f(\cdot \mid \boldsymbol{\eta})\}_{\boldsymbol{\eta} \in \mathcal{E}}\) be an a.c. canonical \(k\)-parameter exponential family. Fix an event \(A\).
	\begin{enumerate}[(a)]
		\item Prove that if \(\mathbb{P}_{\eta_{0}}(A)=0\) for some \(\eta_{0} \in \mathcal{E}\), then \(\mathbb{P}_{\eta}(A)=0\) for all \(\eta \in \mathcal{E}\).
		\item Conclude that if \(\mathbb{P}_{\eta_{0}}(A)<1\) for some \(\eta_{0} \in \mathcal{E}\), then \(\mathbb{P}_{\eta}(A)<1\) for all \(\eta \in \mathcal{E}\).
	\end{enumerate}
\end{ex}

\begin{solution}
	\begin{enumerate}[(a)]
		\item \[\mathbb{P}_{\eta_0}(A)=\int1_{A}(x)h(x)\exp\left\{\eta_0T(x)-A(\eta_0)\right\}\mu(\der x)=0\]
		If $\mathbf{P}_{\eta_0}(A)=0$, because $\exp\left\{\eta_0T(x)-A(\eta_0)\right\}$ is all nonzero. So, $h(x)=0$, that means $\forall \eta$, $\mathbf{P}_{\eta}(A)=0$.
		\item If there is some $\eta_1$, such that $\mathbb{P}_{\eta_1}(A)=1$, then $A=\mathcal{S}$, which means event $A$ is the whole space. But it is impossible because $\mathbb{P}_{\eta_0}(A)<1$.
	\end{enumerate}
\end{solution}

\begin{ex}
	Determine the natural parameter space \(\mathcal{E}\) of the a.c. one-parameter canonical exponential family \(\{f(. \mid \boldsymbol{\eta})\}_{\boldsymbol{\eta}}\) when \(T(x)=x\) and
	\begin{enumerate}[(a)]
		\item \(h(x)=e^{-|x|}\);
		\item \(h(x)=\frac{e^{-|x|}}{1+x^{2}}\).
	\end{enumerate}
	Conclude that the natural parameter space does not have to be an open set.
\end{ex}

\begin{solution}
	\begin{enumerate}[(a)]
		\item \begin{align*}
			&\int h(x)\exp(<T(x), \eta>)G(\der x)\\
			=&{}\int e^{-|x|}e^{\eta x}\der x\\
			=&{}\int_{-\infty}^0 e^{(\eta+1) x}\der x+ \int_{0}^\infty e^{(\eta-1) x}\der x\\
			=&{}\frac{1}{\eta+1}e^{(\eta+1) x}\big|_{-\infty}^0+\frac{1}{\eta-1}e^{(\eta-1)x}\big|_0^{\infty}
		\end{align*}
		$\eta+1>0$, $\eta-1<0\Rightarrow -1<\eta<1$. In addition, $\eta$ can be $1$ and $-1$. So, $-1\leqslant\eta\leqslant1$.
		\item \begin{align*}
			&\int h(x)\exp(<T(x), \eta>)G(\der x)\\
			=&{}\int \frac{e^{-|x|}}{1+x^2}e^{\eta x}\der x\\
			=&{}\int_{-\infty}^0 \frac{e^{(\eta+1) x}}{1+x^2}\der x+ \int_{0}^\infty \frac{e^{(\eta-1) x}}{1+x^2}\der x,
		\end{align*}
		If $\eta+1>0$, $\eta-1<0$. So, the natural parameter space is also $[-1,1]$.
	\end{enumerate}
\end{solution}

\begin{ex}
	Consider a classical linear regression, i.e., assume we have a sample \(Y_{1}, \ldots, Y_{n}\) of independent random variables such that, for \(i=1, \ldots, n, Y_{i} \sim \mathcal{N}\left(\mu_{i}, \sigma^{2}\right), \mu_{i}=\beta_{1}+\beta_{2} z_{i}, \sigma^{2}>0, \beta_{1} \in \mathbb{R}\), \(\beta_{2} \in \mathbb{R}\). Show that this forms a canonical exponential family with natural parameter space \(\mathcal{E}=\mathbb{R} \times \mathbb{R} \times(-\infty, 0)\) up to a permutation of the latter three components.
\end{ex}

\begin{solution}
	Knowing that $\mu_i=\beta_1+\beta_2+z_i$,
	\begin{align*}
		f(y_i|\mu_i,\sigma)&=\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(y_i-(\beta_1+\beta_2z_i))^2}{2\sigma^2}\right)\\
		&=\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{1}{2\sigma^2}y_i^2+\frac{\beta_1}{\sigma^2}y_i+\frac{\beta_2}{\sigma^2}z_iy_i-\frac{(\beta_1+\beta_2z_i)^2}{\sigma^2}\right).
	\end{align*}
	$\eta=(\frac{\beta_1}{\sigma^2}, \frac{\beta_2}{\sigma^2}, -\frac{1}{2\sigma^2})$, $\mathcal{E}=\mathbb{R}\times\mathbb{R}\times(-\infty,0)$.
\end{solution}


The challenge in any text-based modeling approach is the representation of data. A computer cannot comprehend words like humans, thus it needs a numeric approach for text analysis. This leads to the idea of using ids to map words~\parencite{nlpfundamentals}. For example, consider the sentence: “John went to the park.”. We can easily construct a basic dictionary for the words in this sentence and map them to a unique id as such:

$\{"john" : 0, "went" : 1, "to" : 2, "the" : 3, "park" : 4\}$

\subsubsection{One-Hot Encoding}
This process could also occur for categories that need to be numerically represented, such as publications. However, this map tells the model that “park” is more important, or weighted more heavily, than “john”. If the model calculates averages, it will find the average of the words “went” and “the” to equal “to”. These types of relationships are incorrect and would inhibit the model’s effectiveness. To mitigate this problem, researchers developed the concept of one-hot encoding, which binarizes the process of mapping items to their numeric representations~\parencite{harris_harris_2015}. A matrix is generated with dimensions (length of items in entry X length of total dictionary) for each example. Each row of the matrix has a single one and all other entries are zero to represent the single word. Using the above example, our sentence representation would become:

\[\begin{bmatrix}
1 & 0 & 0 & 0 & 0\\
0 & 1 & 0 & 0 & 0\\
0 & 0 & 1 & 0 & 0\\
0 & 0 & 0 & 1 & 0\\
0 & 0 & 0 & 0 & 1\\
\end{bmatrix}\]

Although this solves the initial problem, there are some disadvantages to this approach. Firstly, the creation of such sparse matrices for each entry in a large dataset is inefficient for storage purposes. It is also plagued by the “curse of dimensionality” as each category appends an entire dimension to the matrix~\parencite{bellman_1954}. The addition of items to the dictionary leads to changing the representations for all previous words.

This approach also lacks interpretability. As the items are fed into the model, the one-hot encoded matrix tells the user nothing about the relationship between words. It also breaks down word order, which is required for models such as \acrshort{bert}.

\subsubsection{Embedding Vectors}
These problems led to the development of vectorized word representations (embeddings). Word embeddings map each word to a unique n-sized vector of a single dimension. The embeddings are not directly interpretable by humans, as they capture latent qualities of the word, however embeddings offer a significant advantage over alternatives in their ability to offer comprehensible representations. Similar words will be found to be spatially contiguous. The relationship between words can also be calculated directly through cosine similarity~\parencite{dot_products_2008}. The most famous example of this type of interpretability is the mathematical relationship \textbf{king - man + woman = queen} which can be observed in \Cref{fig:king_man_woman_queen_pic}.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth]{fig/king-man-woman-queen.png}
\caption{Spatial representation of king - man + woman = queen~\textcite{king_man_woman_queen}}
\label{fig:king_man_woman_queen_pic}
\end{figure}

This allows for each example in the dataset to be represented as a unidimensional list of word ids, which can be mapped to their corresponding vectors when being processed by the model. The data storage advantages are obvious, as each entry becomes easily stored as a single dense list instead of a highly-sparse matrix. Cecause of both the data-related and interpretability advantages offered by embeddings, they were utilized for both of the tested models.


\chapter{Introduction to Polynomials}

Watch Khan Academy's \textbf{Polynomials intro} video at \url{https://youtu.be/Vm7H0VTlIco}\index{polynomial}

A \emph{monomial}\index{monomial} is the product of a number and a variable raised to a non-negative (but possibly zero) integer power. Here are some monomials:
\begin{multicols}{4}
  \begin{equation*}
    3 x^2
  \end{equation*}

  \begin{equation*}
    -2 x^{15}
  \end{equation*}

  \begin{equation*}
    \pi x^2
  \end{equation*}

  \begin{equation*}
    (3.33)x^{100}
  \end{equation*}

  \begin{equation*}
    7x
  \end{equation*}

  \begin{equation*}
    3
  \end{equation*}

  \begin{equation*}
    -\frac{2}{3}x^{12}
  \end{equation*}

  \begin{equation*}
    0
  \end{equation*}

\end{multicols}

The exponent is called the \emph{degree} of the monomial\index{monomial!degree}. Examples: $3x^{17}$
has degree 17, $-7x$ has degree 1, and $3.2$ has degree 0 (because you can think of it as $(3.2)x^0$).\index{degree!polynomial}

The number in the product is called the \emph{coefficient}\index{monomial!coefficient}.  Example: $3x^{17}$ has a coefficient of 3, $-2x$ has a coefficient of -2, and $(3.4)x^{1000}$ has a coefficient of 3.4.\index{coefficient!polynomial}

A \emph{polynomial} \index{polynomial!definition of} is the sum of one or more monomials.  Here are some polynomials:
\begin{multicols}{3}
  \begin{equation*}
    4 x^2 + 9x + 3.9
  \end{equation*}

  \begin{equation*}
    -2 x^{10} + (3.4)x - 45x^{900} - 1
  \end{equation*}

  \begin{equation*}
    \pi x^2 + \pi x + \pi
  \end{equation*}

  \begin{equation*}
    3.3
  \end{equation*}

  \begin{equation*}
   7x + 2
  \end{equation*}

  \begin{equation*}
    3x^{20}
  \end{equation*}
\end{multicols}
We say that each monomial is a \emph{term} of the polynomial.

$x^{-5} + 12$ is \emph{not} a polynomial because the first term has a negative exponent.

$x^{2} - 32x^{\frac{1}{2}} + x$ is \emph{not} a polynomial because the second term has a non-integer exponent.

$\frac{x + 2}{x^2 + x + 5}$ is \emph{not} a polynomial because it is not just a sum of monomials.

\begin{Exercise}[title={Identifying Polynomials}, label=findpolynomials]
    Circle only the polynomials.
\begin{multicols}{3}
  \begin{equation*}
    -2 x^3 + \frac{1}{2}x + 3.9
  \end{equation*}

  \begin{equation*}
    2 x^{-10} + 4x - 1
  \end{equation*}

  \begin{equation*}
    (4.5)x^2 + \pi x
  \end{equation*}

  \begin{equation*}
    x^{\frac{2}{3}}
  \end{equation*}

  \begin{equation*}
   7
  \end{equation*}

  \begin{equation*}
    3x^{20} + 2x^{19} -5 x^{18}
  \end{equation*}
\end{multicols}
\end{Exercise}

\begin{Answer}[ref=findpolynomials]
\begin{multicols}{3}
  \begin{equation*}
    \boxed{-2 x^3 + \frac{1}{2}x + 3.9}
  \end{equation*}

  \begin{equation*}
    2 x^{-10} + 4x - 1
  \end{equation*}

  \begin{equation*}
    \boxed{(4.5)x^2 + \pi x}
  \end{equation*}

  \begin{equation*}
    x^{\frac{2}{3}}
  \end{equation*}

  \begin{equation*}
   \boxed{7}
  \end{equation*}

  \begin{equation*}
    \boxed{3x^{20} + 2x^{19} -5 x^{18}}
  \end{equation*}
\end{multicols}

\end{Answer}

We typically write a polynomial starting at the term with the highest
degree and proceed in decreasing order to the term with the lowest
degree:
\begin{equation*}
2 x^9 - 3x^7 + \frac{3}{4}x^3 + x^2 + \pi x -9.3
\end{equation*}
This is known as \emph{the standard form}.  The first term of the
standard form is called \emph{the leading term}, and we often call the
coefficient of the leading term \emph{the leading coefficient}.  We
sometimes speak of the degree of the polynomial, which is just the
degree of the leading term.\index{standard form!polynomial}

\begin{Exercise}[title={Standard of a Polynomial}, label=polynomialstandardform]
  Write $21x^2 - x^3 + \pi - 1000x$ in standard form. What is the degree of this polynomial? What is its leading coefficient?
\end{Exercise}
\begin{Answer}[ref=polynomialstandardform]
  Standard form would be $-x^3 + 21x^2 - 1000x + \pi$. The degree is 3. The leading coefficient is $-1$
\end{Answer}

\begin{Exercise}[title={Evaluate a Polynomial}, label=evaluatepolynomial]
  Let $y = x^3 - 3x^2 + 10x - 12$. What is $y$ when $x$ is $4$?
\end{Exercise}
\begin{Answer}[ref=evaluatepolynomial]
  $4^3 - (3)(4^2) + (10)(4) - 12 = 64 - 48 + 40 - 12$. So $y = 44$
\end{Answer}

I would be remiss in my duties if I didn't mention one more thing
about polynomials: mathematians have defined a polynomial to be a sum
of a \emph{finite} number of monomials.

It is certainly possible to have a sum of an infinite number of monomials
like this:
\begin{equation*}
1 + \frac{1}{2}x + \frac{1}{4}x^2 + \frac{1}{8}x^3 + \frac{1}{16}x^4 + \ldots
\end{equation*}
This is an example of an \emph{infinite series}; we don't consider
them polynomials. Infinite series are interesting and useful, but I
will not discuss them much until later in the course.