\section{Introduction}
A \textbf{cryptosystem} consists of a family of \textit{enciphering transformations} $f$, each corresponding to a choice of parameters $p$, from a set $P$ of all possible plaintext message units to a set $C$ of all possible ciphertext message units. The transformation requires:
\begin{itemize}
	\item An algorithm, which is the same for the whole family and supposedly publicly known;
	\item An enciphering key $K_E$, the value of parameters $p$;
	\item A deciphering key $K_D$.
\end{itemize}
The deciphering key is needed in order to compute $f^{-1}$, i.e.\ decipher the message. The transformation uses the same algorithm as encrypting, except with a different key.

A \textbf{public key} or \textbf{asymmetric cryptosystem}, by definition, has the property that someone who knows only to encipher cannot use the encipher key to find the deciphering key without a prohibitively lengthy computation: the function $f : P \rightarrow C$ is easy to compute once $K_E$ is known, but it is very hard in practice to compute the inverse function $f^{-1} : C \rightarrow P$.

In mathematical terms, the inverse computation should be \textbf{infeasible}, namely so computationally intensive that it is impossible to evaluate it in any reasonable time period.

This implies that $f$ is not invertible without some additional information, therefore $f$ is an \textit{one-way function}. Most public key algorithms are in fact based on number-theoretic functions, whose goal is usually not to have a compact mathematical description between input and output.

A public key cryptosystem, therefore, involves not only one key but a pair: one is public and can be publicly shared, and the other is private and its discover by a third part would compromise the security of the system.

The main point of this scheme is that it is not necessary that the key possessed by the person who encrypts the message is secret. Receivers can only decrypt using their secret key.

Information needed to send secret messages, therefore, can be made public without anyone being able to read the content. This makes communication possible within two parts which have never interacted before, an useful feature among modern systems with huge workloads.

\subsection{Cyclic groups and generators}
\textbf{Cyclic groups} are a way to generalize public key algorithms within groups not necessarily constrained by a prime number. They have a key role in cryptography, since they allow many useful one-way functions which can be used to encrypt information: in fact, cryptographic functions which are hard to break should be defined within finite groups.

A \textbf{group} is a set of elements $G$ together with an operation $\circ$ which combines two elements of $G$. A group has the following properties:
\begin{enumerate}
	\item The group operation $\circ$ is \textit{closed}: for all $a, b \in G$, it holds that $a \circ b = c \in G$;
	\item The group operation is \textit{associative}: $a \circ (b \circ c) = (a \circ b) \circ c$ for all $a, b, c \in G$;
	\item There is an element $1 \in G$ called the \textit{neutral element} (identity) such that $a \circ 1 = 1 \circ a = a$ for all $a \in G$;
	\item For each $a \in G$ there exists an element $a^{-1}$ called the \textit{inverse}, such that $a \circ a^{-1} = a^{-1} \circ a = 1$;
	\item A group $G$ is \textit{abelian} (or \textit{commutative}) if, furthermore, $a \circ b = b \circ a$ for all $a, b \in G$.
\end{enumerate}
Cryptography majorly relies on \textit{multiplicative groups}, where the operation $\circ$ denotes multiplication.

Algorithms related to discrete logarithm problem, for example, concern the group $\mathbb{Z}^*_n$, consisting in the set of all integers $i = 0, 1, \dots, n - 1$ for which $\gcd(i, n) = 1$. $\mathbb{Z}^*_n$ forms an abelian group under multiplication modulo $n$, where the identity element is $e = 1$.

Since cryptography concerns finite structures, groups also have to respect the property to have a finite number of elements, defining the \textit{cardinality} or \textit{order} of the group $G$ by $|G|$.

Every element of a group $G$ has also an order, which is the smallest positive integer $k$ such that:
$$a^k = a \circ a \circ \dots \circ a = 1$$
The $\circ$ operation is applied $k$ times, obtaining the identity element of $G$.

Example: the order of the element $a = 3$ in the group $\mathbb{Z}^*_{11}$ is 5, since $a^5 = 1 \mod 11$.

The powers of $a$ run through the same finite sequence of remainders indefinitely, adopting a cyclic behavior. This allows to introduce cyclic groups.

A group $G$ which contains an element $\alpha$ with maximum order $|G|$ is said to be \textbf{cyclic}. Elements with maximum order are called primitive or \textbf{generators}, since every element $a$ of $G$ can be written as a power $\alpha^i = a$ of this element for some $i$, so that $\alpha$ generates the entire group.

For every prime $p$, $(\mathbb{Z}^*_p, \cdot)$ is an abelian finite cyclic group.

Having a finite cyclic group $G$, some interesting properties hold:
\begin{enumerate}
	\item For every $a \in G$, it holds that:
	\begin{enumerate}
		\item $a^{|G|} = 1$;
		\item The order of $a$ divides $|G|$;
	\end{enumerate}
	\item The number of primitive elements of $G$ is $\phi(G)$, where $\phi$ defines the Euler function (number of positive integers up to $G$ which are coprime to it);
	\item if $|G|$ is prime, then all elements $a \neq 1$ are primitive.
\end{enumerate}
Those concepts have relevancy in cryptography: prime fields are widely used for building discrete logarithm cryptosystems, using the fact that in a cyclic group \textit{only element orders which divide the group cardinality} exist.

\subsubsection{Subgroups}
\textbf{Subgroups} are subsets of cyclic groups which are groups themselves, therefore respect all the previously stated properties.

Let $(G, \circ)$ be a cyclic group. Then, every element $a \in G$ with $ord(a) = s$ is the primitive element of a cyclic subgroup with $s$ elements.

As previously stated, the generator can be non-unique. An important special case are subgroups of prime order: with cardinality $q$, all elements $e \neq 1$ have order $q$ as well.

If $H$ is a subgroup of $G$, then $|H|$ divides $|G|$.

To obtain a construction method for subgroups from a given finite cyclic group, only the cardinality $n$ and a primitive element are needed: then, $\alpha^{n/k}$ is computed to obtain a generator $\alpha$ of the subgroups with $k$ elements.

This follows knowing that every integer $k$ which divides the cardinality $n$ there exists exactly one cyclic subgroup $H$ of $G$ of order $k$, consisting in the elements $a \in G$ satisfying $a^k = 1$.

Seminar - Algebraic Methods and Algorithms in Cryptology  /chapters/introduction.tex


\subsection{Unary operators}

A unary operator takes one input and returns another.

Only negation, \(\neg \) is of interest.

The following statements are equivalent:

\begin{itemize}
\item \(T\)
\item \(\neg F\)
\end{itemize}


\chapter{Applications Beyond Analysis and Resynthesis}

\subsection{Discussion of the Stop Function}\label{section:audioEffects:stopFunction}
In this section, the stop function of the detection stage is revisted.

    \subsubsection{Residual Power Threshold}
    Perhaps the most striaghtforward stop function is checking power of the residual signal, $x_r$. Given a threshold $t$:
    \begin{align}\label{eq:audioEffects:stopFunction:powerThreshold}
        s(y, t) =
            \begin{cases}
                0, & \text{if } t \geq \sum_{N = 0}^{N-1} y[n]^2 \\
                1, & \text{otherwise}
            \end{cases}
    \end{align}
    When the periodicity transform is performed short-time on a signal, that is, in frames, it is typically best to set $t$ equal to some fractional amount of the input vector $x$ so as to allow the threshold for frame $i$, $t_i$, to be dynamic:
    \begin{align*}
        t_i = t \sum_{N = 0}^{N-1} x[n]^2
    \end{align*}
    where $0 \leq t < 1$. There are of course many variations of the above thresholding formulation in Equation \eqref{eq:audioEffects:stopFunction:powerThreshold} such as $||x||_p$.

    If one uses the simple threshold stop function, it is important to recognize that it is possible to continue finding periodic data in a residual that carries no meaningful information as to the ``essence'' of the input signal (refer to the paragraph beginning this section). Although these ``phantom'' periodicities may help the convex program to better minimize the reconstruction error, they often add no value and can often distort the useful periodicities already detected.

        As an example, let $x$ be the combination of three sinusoids with periods 17, 21, and 50, plus zero-mean Gaussian noise with a SNR of -18 dB:
        \begin{figure}[h]
            \centering
            % \includegraphics[width=0.75\textwidth]{PATH/TO/MY/FIG}
            \caption{[A signal $x$ that is the sum of three equal-amplitude sinusoids with $p = \{17, 21, 50\}$ plus zero-mean Gaussain noise.]A signal $x$ that is the sum of three equal-amplitude sinusoids with $p = \{17, 21, 50\}$ plus zero-mean Gaussain noise.}
            \label{fig:audioEffects:stopFunction:residual:sinesPlusNoise}
        \end{figure}
        where $N$ is the length of the signal and is equal to 1000, and $G(n)$ is the Gaussian noise at sample $n$. What periods are found at various values of $t$? Below is the output of the convex program using the projection in Equation \eqref{eq:intro:sethares} and different values of $t$. We also set $t$ directly to simplify:
        \begin{figure}[h]
            \centering
            % \includegraphics[width=0.75\textwidth]{PATH/TO/MY/FIG}
            \caption[A figure showing the effect of different threshold values for the stop function.\index{Values of $t$}]{Results of period detection using various values of $t$. \emph{Top Panel:} $t = 0.2$. Note that not enough data is captured and only two periods are found. \emph{Middle Panel:} $t = 0.1$ Since $t$ is exactly equal to the SNR of the signal, we find that the three periods we expect are found before the algorithm terminates. \emph{Bottom Panel:} $t = 0.05$ Although we find the three periods we expect, the algorithm also finds periods in the residual noise. This has the further effect of distorting the already detected periodic waveforms as one will see in Section \ref{ch3:recoveringWaveforms}}
            \label{fig:audioEffects:stopFunction:thresholdEffect}
        \end{figure}
        Notice that in the bottom panel, $t$ is set too low and we therefore see many periods which, while they do ``exist'', do not necessarly add information but are rather imparted by the projection process and numerical happenstance.

        In a close-to-ideal world (one in which we admit noise), one would simply set $t$ to be the level of the noise floor in the signal. In this way, one would only capture the periodic components while leaving the noise (relatively) untouched. This, of course, is not possible in real data and is especially prone to error when non-integer periods are present. See Section \ref{ch3:nonintegerPeriods} for a longer discussion.

    \subsubsection{Sum of the Power of the Periodic Components}
    A novel way to approch the problem and one that keeps in mind the qualitative criteria of ``listenability'' is to check whether or not the sum of the powers of the detected periodic waveforms exceeds the power of the original signal. For convenience, we define the power to be $E(x) \equiv \sum_{n = 0}^{N - 1} |x[n]|^2$:
    \begin{align}
        s(x, X_Q) = \begin{cases}
            0, & \text{if } E(x) \leq \sum_{x_q \in X_Q} E(x_q) \\
            1, & \text{otherwise}
        \end{cases}
    \end{align}
    where $X_Q$ is the set of projections of the best periods acquired through projection of $x_r$ onto $P_p$. Note that these projections are also of length $N$. This often results in a less-than-perfect reconstruction from a residual minimization standpoint but in practice has shown better listenability of the components themselves.

    \subsubsection{Decrease in the Total Power of Periodic Components}
    An interesting phenomenon was observed in processing real data, often data that contains noise and/or non-integer periodicities whereby the sums of the derived periodic components \emph{decreases} as more iterations are performed.

    \subsubsection{Lack of Significant Periods}
    If the input to the ML estimator is $x = \mathcal{N}(0, \sigma^2)$, the resulting powers of the periods, normalized as Equation \eqref{eq:intro:sethares:periodicNormGamma}, we see that there are no significant peaks in the periodogram:
    \begin{figure}[h]
        \centering
        % \includegraphics[width=0.75\textwidth]{PATH/TO/MY/FIG}
        \caption[Periodogram of zero-mean Gaussian noise for $2 \leq p \leq 500$ for $N = 1000$]
        {Periodogram of zero-mean Gaussian noise for $2 \leq p \leq 500$ for $N = 1000$. Notice the conspicuous lack of significant periods for the ML estimator to choose from.}
        \label{fig:audioEffects:gaussianPeriodogram}
    \end{figure}

    We can measure this by defining the ``flatness'' of the periodogram to be:
    \begin{align} \label{eq:audioEffects:flatness}
        P_{\text{flatness}}(P) &= \frac
            {
                % e^{\big( \frac{1}{N} \sum_{n=0}^{N-1} \ln P[n] \big)}
                \sqrt[N]{\Pi_{n=0}^{N-1} P_n}
                % \big( \Pi_{n=0}^{N-1} P_n \big)^{\frac{1}{N}}
            }
            {
                \frac{1}{N} \sum_{n=0}^{N-1} P_n
            }
    \end{align}
    where $N$ is the number of periods we search for (i.e. $N = P_{max} - P_{min}$) and $t$ is again a threshold, all typically converted to decibels. Notice that this is identical to the spectral flatness measure that is common in measuring the Fourier Transform of a discrete time signal. It proves useful in this regard as well, as a simple but effective measure of determining whether or not there remain any more significant periods in the data.

    Suppose $x = S_1 + \mathcal{N}(0, \sigma^2)$ and $||S_1||_2 \ll ||\mathcal{N}(0, \sigma^2)||_2$. The resulting periodogram will be very similar to that of pure zero-mean Gaussain noise yet we know there is a periodic signal in $x$. What ought to be the interpretation in this case? In practice, it was found that in this case, the periodic signal $S_1$ is so insignificant so as to have no meaningful value as to the constitution of $x$. In other words, in such a case nothing of value is lost by disregarding $S_1$ given its overall weakness in the signal.

\subsection{Time Stretching}

\subsection{Monaural Source Separation}


\chapter{Unit 4}
\section{First Fundamental Theorem of Calculus}
Let $f$ be a function defined $\forall x\in[a,b]$. Let $F$ be the antiderivative
of $f$ on $[a,b]$. It can be said that:
\begin{equation}
  \int_{a}^{b} f(x) dx = F(b) - F(a)
\end{equation}

\subsection{The Mean Value Theorem}
If $f$ is continuous on the closed interval $[a,b]$, then there exists a number
$x=c$ such that:
\begin{equation}
  \int_{a}^{b} f(x)dx = f(c)(b-a)
\end{equation}

\section{Second Fundamental Theorem of Calculus}
\begin{equation}
  \frac{d}{dx}\left[\int_a^x f(t) dt\right]=f(x)
\end{equation}

\begin{equation}
  \frac{d}{dx}\left[ \int_a^{u(x)} f(t) dt\right] = f(u(x))\frac{du}{dx}
\end{equation}

\section{Sigma Limit Process}
Let $c_i = a+i(\Delta{x})$, and $\Delta{x}=\frac{b-a}{n}$.

\begin{equation}
  \int_a^b f(x) dx = \lim_{\Delta x\to 0} \sum_{i=1}^{n}f(c_i)\Delta{x_i}
\end{equation}

\section{Logarithmic \& Exponential Functions}
\begin{equation}
  \int \frac{du}{u} = \ln|u| + C
\end{equation}

\begin{equation}
  \int e^u dx = du * e^u + C
\end{equation}

\section{Differential Equations}
\begin{equation}
  y=Ce^{kt}
\end{equation}

\section{Inverse Trigonometric Functions}
\begin{align}
  \int \frac{1}{a^2+u^2} du = \frac{1}{a}\arctan{\frac{u}{a}} + C \\
  \int \frac{1}{\sqrt{a^2-u^2}} du = \arcsin{\frac{u}{a}} + C &&
  \int \frac{1}{u\sqrt{u^2-a^2}} du = \frac{1}{a}\arcsec{\frac{|u|}{a}} + C
\end{align}


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Power Series and Convergence}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to power series and their convergence properties.
\end{abstract}
\maketitle

\begin{exercise}
Compute the radius of convergence of the power series below.
\[ \sum_{n = 1}^\infty \frac{(x-3)^n}{2^n + 1} \]
\[ R = \answer{2} \]
\begin{hint}
When computing a limit like
\[ \lim_{n \rightarrow \infty} \frac{2^{n} + 1}{2^{n+1} + 1}, \]
remember the effect of orders of growth: $1 <\!\!< 2^n$, and so from the perspective of the limit, the $+1$'s in both the numerator and denominator are both negligible.
\end{hint}
\end{exercise}

\begin{exercise}
Compute the radius of convergence of the power series below.
\[ \sum_{n = 1}^\infty \frac{(-3)^n (\ln n) (x-3)^n}{4^n} \]
\[ R = \answer{\frac{4}{3}} \]
\begin{hint}
When computing a limit like
\[ \lim_{n \rightarrow \infty} (\ln n)^{1/n}, \]
remember that we already know $n^{1/n} \rightarrow 1$ and that $\ln n <\!\!<n$ as well, so we expect $(\ln n)^{1/n} \rightarrow 1$ as well.
\end{hint}
\end{exercise}

\begin{exercise}
Compute the radius of convergence of the power series below.
\[ \sum_{n = 1}^\infty \frac{(-2)^n n! (x-3)^n}{3^n+n^2} \]
\[ R = \answer{0} \]
\end{exercise}

\begin{exercise}
Compute the interval of convergence for the power series below.
\[ \sum_{n=1}^\infty \frac{(-2)^n (x-3)^n}{\ln n} \]
The left endpoint of the interval is $\answer{5/2}$; it \wordChoice{\choice{is}\choice[correct]{is not}} included in the interval of convergence.
The right endpoint of the interval is $\answer{7/2}$; it is \wordChoice{\choice[correct]{is}\choice{is not}} included in the interval of convergence.
\end{exercise}

\begin{exercise}
Reindex the series below:
\[ \sum_{n=1}^\infty \frac{x^{2n+1}}{n + 2} = \sum_{n = 0} \frac{x^{\answer{2n+3}}}{\answer{n+3}} \]
\end{exercise}

\begin{exercise}
Differentiate the series below term-by-term:
\[ \frac{d}{dx} \sum_{n=0}^\infty \frac{n}{n+1} x^n = \sum_{n=1} \answer{\frac{n^2}{n+1}} x^{\answer{n-1}}. \]
(Note that the $n=0$ term goes away because the derivative of a constant is zero.)
\end{exercise}

\begin{exercise}
Integrate the series below term-by-term:
\[ \int_0^x \left[ \sum_{n=0}^\infty \frac{x^n}{(2n+3)^2} \right]  = \sum_{n=0} \answer{\frac{1}{(n+1)(2n+3)^2}} x^{\answer{n+1}}. \]
\end{exercise}

\begin{exercise}
For each step below, apply a term-by-term operation, a multiplication by a monomial, or a substitution to derive a new series formula from a known formula.
\begin{itemize}
\item Use the formula \[ \frac{1}{1-x} = \sum_{n=0}^\infty x^n \]
to derive a summation formula for $1/(1+x^2)$, i.e.,
\[ \frac{1}{1+x^2} = \sum_{n=0}^\infty \answer{(-1)^n} x^{\answer{2n}}. \]
\item Use the formula you derived above to develop a power series expansion for arctangent:
\[ \arctan x = \sum_{n=0}^\infty \answer{\frac{(-1)^n}{(2n+1)}} x^{\answer{2n+1}}. \]
\item The radius of convergence of this last series is $R = \answer{1}$.
\end{itemize}
\end{exercise}

\begin{exercise}
For each step below, apply a term-by-term operation, a multiplication by a monomial, or a substitution to derive a new series formula from a known formula.
\begin{itemize}
\item Use the formula \[ \frac{1}{1-x}   = \sum_{n=0}^\infty x^n \] to determine the sum of the series below for $-1 < x < 1$:
\[ \sum_{n = 0}^\infty \frac{x^{n+1}}{n+1} = \answer{- \ln |1-x|}. \]
(Don't forget absolute values if you need them.)
\item Use the formula you just derived to determine the sum of the series
\[ \sum_{n=0}^\infty \frac{x^{2n+3}}{n+1} = \answer{- x \ln |1-x^2|.} \]
\item Use the formula you just derived to determine the sum of the series
\[ \sum_{n=0}^\infty \frac{2^{2n} x^{2n+3}}{n+1} = \answer{- \frac{1}{4} x \ln |1-4x^2|.} \]
\end{itemize}
\end{exercise}

\section*{Sample Quiz Questions}
\begin{question}%%%%%[PowerSerInterval001]

Find the full interval of convergence for the power series \[\sum_{m=2}^{\infty} \frac{(-3)^mm^2(x - 5)^m }{\ln m}.\]
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice[correct]{\(\displaystyle \left(\frac{14}{3},\frac{16}{3}\right)\)}
\choice{\(\displaystyle \left[\frac{14}{3},\frac{16}{3}\right)\)}
\choice{\(\displaystyle \left(2,8\right]\)}
\choice{\(\displaystyle \left[2,8\right]\)}
\choice{\(\displaystyle \left(-\infty,\infty\right)\)}
\end{multiplechoice}
\begin{feedback}
First observe that
\[\begin{aligned}
\frac{1}{R} & = \lim_{m\rightarrow \infty} \left| \frac{\frac{(-3)^{(m+1)}{(m+1)}^2 }{\ln {(m+1)}} }{\frac{(-3)^mm^2 }{\ln m}} \right| \\
& = \lim_{m\rightarrow \infty} \left| \frac{-3{(m+1)}^2\ln m }{m^2\ln {(m+1)} } \right| \\
& = 3
\end{aligned}\]
because \[\lim_{m \rightarrow \infty} \frac{(m+1)^2 \ln m}{m^2 \ln (m+1)} = \lim_{m \rightarrow \infty} \frac{(m+1)^2}{m^2} \lim_{m \rightarrow \infty} \frac{\ln m}{\ln (m+1)} = 1\] by virtue of l'Hospital's rule applied to both limits on the right-hand side.
\begin{hint}
This means that the radius equals \(1/3\). At the endpoint \(x=14/3\), the series equals \[\sum_{m=2}^{\infty} \frac{m^2 }{\ln m },\] which diverges by the \(n\)-th term divergence test because \(\lim_{m \rightarrow \infty} m^{2} / \ln m = \infty \neq 0\). At the endpoint \(x=16/3\), the series equals \[\sum_{m=2}^{\infty} (-1)^m\frac{m^2 }{\ln m },\] which diverges for the same reason as the other endpoint, i.e., the terms do not go to zero.
\end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[PowerSerInterval003]

Find the full interval of convergence for the power series \[\sum_{m=2}^{\infty} \frac{(-4)^m(\ln m)(x - 1)^m }{(-2)^mm}.\]
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle \left(\frac{1}{2},\frac{3}{2}\right)\)}
\choice[correct]{\(\displaystyle \left[\frac{1}{2},\frac{3}{2}\right)\)}
\choice{\(\displaystyle \left(-1,3\right]\)}
\choice{\(\displaystyle \left[-1,3\right]\)}
\choice{\(\displaystyle \left(-\infty,\infty\right)\)}
\end{multiplechoice}
\begin{feedback}
First observe that
\[\begin{aligned}
\frac{1}{R} & = \lim_{m\rightarrow \infty} \left| \frac{\frac{(-4)^{(m+1)}(\ln {(m+1)}) }{(-2)^{(m+1)}{(m+1)}} }{\frac{(-4)^m(\ln m) }{(-2)^mm}} \right| \\
& = \lim_{m\rightarrow \infty} \left| \frac{-4(\ln {(m+1)})m }{-2(\ln m){(m+1)} } \right| \\
& = 2
\end{aligned}\]
because \[\lim_{m \rightarrow \infty} \frac{m\ln (m+1)}{(m+1) \ln m} = \lim_{m \rightarrow \infty} \frac{m}{m+1} \lim_{m \rightarrow \infty} \frac{\ln (m+1)}{\ln m} = 1\] by virtue of l'Hospital's rule applied to both limits on the right-hand side. \begin{hint}
This means that the radius equals \(1/2\). At the endpoint \(x=3/2\), the series equals \[\sum_{m=2}^{\infty} \frac{(\ln m) }{m },\] which diverges by direct comparison to the harmonic series, i.e., the \(p\)-series with \(p = 1\). At the endpoint \(x=1/2\), the series equals \[\sum_{m=2}^{\infty} (-1)^m\frac{(\ln m) }{m },\] which converges by the alternating series test because the sign of the terms alternates and \(\ln m / m\) decreases to zero as \(m \rightarrow \infty\). \end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[PowerSerInterval004]

Find the full interval of convergence for the power series \[\sum_{m=2}^{\infty} \frac{(-2)^m(\ln m)(x + 4)^m }{6^mm}.\]
(Hints will not be revealed until after you choose a response.)
\begin{multiplechoice}
\choice{\(\displaystyle \left(-\frac{13}{3},-\frac{11}{3}\right)\)}
\choice{\(\displaystyle \left[-\frac{13}{3},-\frac{11}{3}\right)\)}
\choice[correct]{\(\displaystyle \left(-7,-1\right]\)}
\choice{\(\displaystyle \left[-7,-1\right]\)}
\choice{\(\displaystyle \left(-\infty,\infty\right)\)}
\end{multiplechoice}
\begin{feedback}
First observe that
\[\begin{aligned}
\frac{1}{R} & = \lim_{m\rightarrow \infty} \left| \frac{\frac{(-2)^{(m+1)}(\ln {(m+1)}) }{6^{(m+1)}{(m+1)}} }{\frac{(-2)^m(\ln m) }{6^mm}} \right| \\
& = \lim_{m\rightarrow \infty} \left| \frac{-2(\ln {(m+1)})m }{6(\ln m){(m+1)} } \right| \\
& = \frac{1}{3}
\end{aligned}\]
because \[\lim_{m \rightarrow \infty} \frac{m\ln (m+1)}{(m+1) \ln m} = \lim_{m \rightarrow \infty} \frac{m}{m+1} \lim_{m \rightarrow \infty} \frac{\ln (m+1)}{\ln m} = 1\] by virtue of l'Hospital's rule applied to both limits on the right-hand side. \begin{hint}
This means that the radius equals \(3\). At the endpoint \(x=-7\), the series equals \[\sum_{m=2}^{\infty} \frac{(\ln m) }{m },\] which diverges by direct comparison to the harmonic series, i.e., the \(p\)-series with \(p = 1\). At the endpoint \(x=-1\), the series equals \[\sum_{m=2}^{\infty} (-1)^m\frac{(\ln m) }{m },\] which converges by the alternating series test because the sign of the terms alternates and \(\ln m / m\) decreases to zero as \(m \rightarrow \infty\). \end{hint}
\end{feedback}

\end{question}

\begin{question}%%%%%[PowerSerInterval005]

Find the full interval of convergence for the power series \[\sum_{m=1}^{\infty} \frac{\sqrt[3]{m}(x - 2)^m }{m!}.\]
\begin{multiplechoice}
\choice{\(\displaystyle \left(1,3\right)\)}
\choice{\(\displaystyle \left[1,3\right)\)}
\choice{\(\displaystyle \left(1,3\right]\)}
\choice{\(\displaystyle \left[1,3\right]\)}
\choice[correct]{\(\displaystyle \left(-\infty,\infty\right)\)}
\end{multiplechoice}
\begin{feedback}
First observe that
\[\begin{aligned}
\frac{1}{R} & = \lim_{m\rightarrow \infty} \left| \frac{\frac{\sqrt[3]{{(m+1)}} }{{(m+1)}!} }{\frac{\sqrt[3]{m} }{m!}} \right| \\
& = \lim_{m\rightarrow \infty} \left| \frac{\sqrt[3]{{(m+1)}} }{(m+1)\sqrt[3]{m} } \right| \\
& = 0
\end{aligned}\]
because \(m+1\) in the denominator tends to \(\infty\) and \[\lim_{m \rightarrow \infty} \frac{\sqrt[3]{m+1}}{\sqrt[3]{m}} = \lim_{m \rightarrow \infty} \left( 1 + m^{-1} \right)^{1/3} = \left( 1 + \lim_{m \rightarrow \infty} m^{-1} \right)^{1/3} = 1.\]
This means that the radius is infinite and the interval of convergence is \((-\infty,\infty)\).
\end{feedback}

\end{question}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2015C.03]

For which values of \(x\) does the series \(\displaystyle \sum_{n=1}^\infty \frac{(-1)^{n+1}(x-1)^n}{n 4^n}\) converge?
\begin{multiplechoice}
\choice{\(-3 < x < 5\)}
\choice{\(-3 \leq x < 5\)}
\choice[correct]{\(-3 < x \leq 5\)}
\choice{\(-5 < x \leq 3\)}
\choice{\(-5 \leq x < 3\)}
\choice{\(-5 \leq x \leq 3\)}
\end{multiplechoice}

\end{question}

\begin{question}%%%%%[2016C.11]

Find the interval of convergence of the power series below.
\[ \sum_{n=1}^\infty \frac{(4x-1)^n}{n^\frac{3}{4} (n^2+2)} \]
\begin{multiplechoice}
\choice{\(\displaystyle \left( 0 , \frac{1}{2} \right]\)}
\choice[correct]{\(\displaystyle \left[ 0 , \frac{1}{2} \right]\)}
\choice{\(\displaystyle \left( 0 , \frac{1}{2} \right)\)}
\choice{\(\displaystyle \left[ 0 , \frac{1}{2} \right)\)}
\choice{\(\displaystyle \left( -\frac{1}{2} , 0\right]\)}
\choice{\(\left( - \infty, \infty \right)\)}
\end{multiplechoice}

\end{question}

\begin{question}%%%%%[2017C.13]

Find the interval of convergence of the power series \(\displaystyle \sum_{n=2}^\infty \frac{2^n (x+5)^n}{\sqrt[3]{n}}\).
\begin{multiplechoice}
\choice{\(\displaystyle \left[ -\frac{11}{2}, -\frac{9}{2} \right]\)}
\choice[correct]{\(\displaystyle \left[ -\frac{11}{2}, -\frac{9}{2} \right)\)}
\choice{\(\displaystyle \left( -\frac{11}{2}, -\frac{9}{2} \right)\)}
\choice{\(\displaystyle \left[ \frac{9}{2}, \frac{11}{2} \right)\)}
\choice{\(\displaystyle \left( \frac{9}{2}, \frac{11}{2} \right)\)}
\choice{\(\displaystyle \left[ \frac{9}{2}, \frac{11}{2} \right]\)}
\end{multiplechoice}

\end{question}

\end{document}


\documentclass[../main.tex]{subfiles}
\begin{document}
\subsection{Generic greedy algorithm}
The greedy algorithm is an algorithmic paradigm. It does not always provide a solution in general, not least an optimal solution. As such the greedy algorithm is a description of a problem solving heuristic.  It says that when trying to solve a problem we should choose the local optimum at each iteration in the hope of finding a global optimum. In this section, we will introduce the generic procedure that makes the greedy algorithm and showing how it can be modified in a variety of ways, including for compatibility with matroids.

\begin{algorithm}[H]
\caption{Greedy algorithm\cite{jungnickel}}\label{greedy}
Let $(E,\mathcal{S})$ be an independence system and $\omega:E \longrightarrow \mathbb{R^+}$
\begin{algorithmic}[1]
\Procedure{GREEDY}{$E, \mathcal{S}, \omega,T$}
\State order the elements of $E$ according to their weight
\State $E = \{ e_1,...,e_m\}$ with $\omega(e_1) \geq \omega(e_2) \geq ... \geq \omega(e_m)$
\State $T\gets \emptyset$
\For{$k=1$ to $m$}
	\If{$T \cup \{e_k\} \in \mathcal{S}$}
	 	\State append $e_k$ to $T$
	\EndIf
\EndFor
\EndProcedure
\end{algorithmic}
\end{algorithm}

We can see in the above pseudocode (taken from \cite{jungnickel}) that this is a sequential iterative algorithm. Our data is kept in a list sorted by weight: heaviest to lightest. We then select the heaviest entry in our list at each iteration and append it to our initial empty variable $T.$ By the end of the process we should have joined a certain number of the elements to our variable $T.$ Where $T$ at the time of termination is the greedy algorithm solution.\\ This is how the algorithm looks in the most abstract sense, later we will see it used as a way of solving the \textit{minimal spanning tree} problem mentioned in \textit{definition 3.11}.
\subsection{Modified greedy for Matroids}
Here, we will see how this algorithm corresponds to matroids. The greedy algorithm above, constructed a maximal weight element from our list called $T.$ Depending on the structure of our system and the termination clause this element's cardinality can differ. It is our hope to show that when the greedy algorithm is applied to a matroid we get an optimal solution, meaning that we generate a base of the matroid. However, for now it is enough to show that the greedy algorithm always produces a solution in general. This is illustrated by the below pseudocode, which was adapted from Oxley's\cite{ox_book} description.

\begin{algorithm}[H]
\caption{Matroid greedy algorithm}\label{matroid_ver}
The \textit{greedy algorithm} for the pair $(\mathcal{I}, \omega)$ is as follows:
\begin{algorithmic}[1]
\Procedure{GREEDY}{$E, \mathcal{I}, \omega$}
\State Set $T \gets \emptyset$ and $E = \{e_1,...,e_m\}$
\While{$\exists e \in E \setminus T$ such that $T \cup \{e\} \in \mathcal{I}$}
\State Choose such an element $e_{max}$ of maximum weight,
\State let $T = T \cup \{e_{max}\}$
\EndWhile
\State\Return $T$
\EndProcedure
\end{algorithmic}
\end{algorithm}

The procedure described above is as follows: Initialise the solution variable $T$ as the emptyset (the empty set is always independent). We want to build the base of the matroid: $B_G.$ At each iteration select the heaviest weighted element possible that is not already contained in $T$ such that $T \cup \{e\} \in \mathcal{I}$ where $e$ is the selected element, we want to repeat this procedure as many times as possible until there are no such elements left to choose. Then we return $T,$ as $T$ is now equal to the solution of the greedy algorithm, i.e $B_G$ (the base generated by the greedy algorithm).

\begin{rem}
It is interesting to note that in this way we should find a maximal member $B_{max}$ of $\mathcal{B}$ (the collection of bases of a matroid) and we will prove that this is certainly the case later in \textit{theorem 5.2}. But if we negate this process in the following way we can use this exact procedure to find a minimal member $B_{min}$ of $\mathcal{B}.$ This works as follows:\\
Let $\omega : E \longrightarrow \mathbb{R^{-}}$ be the weight function, we want to find an independent set $A$ whose weight is maximal, where
\begin{equation}
\omega(A) := \sum_{e \in A} |\omega (e)|
\end{equation}
Due to the fact that our weights are now negative real numbers, finding the maximal element at each iteration corresponds to finding the value with the minimal absolute value. And so through completing the greedy algorithm process we should successfully find our minimal element $B_{min}$ of $\mathcal{B}.$
\end{rem}
\end{document}


\chapter[Finite elements for parabolic problems]{Finite elements for \\
parabolic problems in 1D}

We now consider a more general parabolic PDE with mixed boundary conditions,
\begin{equation}\label{eq: parabolic ivp 1d}
\begin{aligned}
u_t+\mathcal{L}u&=f(x,t)&&\text{for $0<x<L$ and $0<t<T$,}\\
u&=\gamma_0(t)&&\text{at $x=0$, for $0<t<T$,}\\
au'&=\gamma_L(t)&&\text{at $x=L$, for $0<t<T$,}\\
u&=u_0(x)&&\text{for $0<x<L$ when $t=0$,}
\end{aligned}
\end{equation}
where $\mathcal{L}u=-\bigl(a(x)u'\bigr)'+c(x)u$ as before
in~\eqref{eq: L self-adjoint}.  The identity~\eqref{eq: Lu v by parts} implies
that, for any test function~$v(x)$,
\begin{equation}\label{eq: parabolic 1d weak}
\int_0^Lu_tv\,dx+\int_0^L\bigl(a(x)u_xv_x+c(x)uv\bigr)\,dx
	=\gamma_L(t)v(L)+\int_0^Lf(x,t)v\,dx
\quad\text{provided $v(0)=0$.}
\end{equation}
We will use this relation to formulate a semidiscrete finite element
solution~$u_h(x,t)\approx u(x,t)$ and then apply finite difference
approximations in time to derive some fully-discrete schemes.

\section{Semidiscrete method}\label{sec: semidiscrete parabolic FEM}
Let $V_h$ denote the space of continuous, piecewise-linear functions for a
given mesh~\eqref{eq: 1d nodes} on the spatial interval~$[0,L]$.  We define the
solution set (now dependent on~$t$) and test space by
\[
S_h(t)=\{\,v\in V_h:v(0)=\gamma_0(t)\,\}
\quad\text{and}\quad
T_h=\{\,v\in V_h:v(0)=0\,\}.
\]
We also choose $u_{0h}\in V_h$ such that $u_0\approx u_{0h}$; the simplest
choice would be the interpolant $u_{0h}=\mathcal{Q}_1u_0$. The semidiscrete
finite element solution~$u_h(x,t)$ is then defined for~$0\le t\le T$ by
requiring that $u_h(\cdot,t)\in S_h(t)$ and
\begin{equation}\label{eq: semidiscrete fem 1d}
\int_0^L(u_h)_tv\,dx+\int_0^L\bigl(a(x)(u_h)_xv_x+c(x)u_hv\bigr)\,dx
    =\gamma_L(t)v(L)+\int_0^Lf(x,t)v\,dx
\quad\text{for all $v\in T_h$,}
\end{equation}
with $u_h(0)=u_{0h}$. The stiffness matrix~$\boldsymbol{A}$ and mass
matrix~$\boldsymbol{C}$ are the same as in the stationary problem
(section~\ref{sec: self-adjoint 1d}), but the nodal
values of the solution are now time-dependent, with
\[
u_h(x,t)=\sum_{q=0}^P U_q(t)\chi_q(x)\quad\text{for $0\le x\le L$,}\quad
\text{where $U_q(t)=u_h(x_q,t)$.}
\]
Likewise, the load vector is time-dependent in general, since
\[
f_p(t)=\int_0^Lf(x,t)\chi_p(x)\,dx.
\]
Noting
\[
\int_0^L(u_h)_t\chi_p\,dx=\int_0^L\sum_{q=0}^P
    \frac{dU_q}{dt}\,\chi_q(x)\chi_p(x)\,dx=\sum_{q=0}^P m_{pq}\,\frac{dU_q}{dt}
    \quad\text{where $m_{pq}=\int_0^L\chi_q\chi_p\,dx$,}
\]
we see by choosing $v=\chi_p$ in~\eqref{eq: semidiscrete fem 1d} that
\[
\sum_{q=0}^P\biggl(m_{pq}\,\frac{dU_q}{dt}+a_{pq}U_q+c_{pq}U_q\biggr)
    =\gamma_L(t)\chi_p(L)+f_p(t),
\]
with $U_q(0)=u_{0h}(x_q)$.  Since $U_0(t)=u_h(x_0,t)=\gamma_0(t)$,
\[
\sum_{q=1}^P\biggl(m_{pq}\,\frac{dU_q}{dt}+a_{pq}U_k+c_{pq}U_q\biggr)
    =f_p(t)+g_p(t)
\]
where
\[
g_p(t)=\gamma_L(t)\delta_{pP}-m_{p0}\gamma_0'(t)-(a_{p0}+c_{p0})\gamma_0(t),
\]
and so the nodal values satisfy the $P\times P$ system of linear ODEs,
\begin{equation}\label{eq: semidiscrete fem 1d ODE}
\boldsymbol{M}\,\frac{d\boldsymbol{U}}{dt}
+(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{U}=\boldsymbol{f}(t)
    +\boldsymbol{g}(t)\quad\text{for $0\le t\le T$,}
    \quad\text{with $\boldsymbol{U}(0)=\boldsymbol{U}_0$,}
\end{equation}
where
\[
\boldsymbol{M}=[m_{pq}]_{p,q=1}^P,\qquad
\boldsymbol{A}=[a_{pq}]_{p,q=1}^P,\qquad
\boldsymbol{C}=[c_{pq}]_{p,q=1}^P,
\]
and
\[
\boldsymbol{f}(t)=[f_p(t)]_{p=1}^P,\qquad
\boldsymbol{g}(t)=[g_p(t)]_{p=1}^P,\qquad
\boldsymbol{U}_0=[u_{0h}(x_p)]_{p=1}^P.
\]
By using an energy argument, we can show that $u_h$ is stable.

\begin{theorem}
Assume that $c(x)\ge0$.  Then, the semidiscrete finite element
method~\eqref{eq: semidiscrete fem 1d} has a unique
solution~$u_h(\cdot,t)\in S_h(t)$ for~$0\le t\le T$.  Furthermore, the
method is stable in the sense that
\[
\|(u_h-\eta)(\cdot,t)\|\le\|(u-\eta)(\cdot,0)\|
	+2\int_0^t\|(f-\eta_s-\mathcal{L}\eta)(\cdot,s)\|\,ds
	\quad\text{for $0\le t\le T$,}
\]
where $\eta$ is any function satisfying
\[
\eta(\cdot,t)\in S_h(t),\qquad \eta(0,t)=\gamma_0(t),\qquad
	a(L)\eta_x(L,t)=\gamma_L(t).
\]
For example, we my choose
\[
\eta(x,t)=\gamma_0(t)+\frac{\gamma_L(t)}{a(L)}\,x.
\]
\end{theorem}
\begin{proof}
The existence and uniqueness of $u_h$ follows from a standard result for
systems of ODEs.  To establish the stability estimate, we first
use the energy inner product~\eqref{eq: energy iprod} to write
\eqref{eq: semidiscrete fem 1d} as
\[
\iprod{(u_h)_t,v}+\iprod{u_h,v}_{\mathcal{L}}=\gamma_L(t)v(L)
	+\iprod{f(\cdot,t),v}\quad\text{for all $v\in T_h$.}
\]
Thus, the difference
\[
w(x,t)=u_h(x,t)-\eta(x,t)
\]
satisfies
\[
w(\cdot,t)\in S_h(t)\quad\text{and}\quad
w(0,t)=u_h(0,t)-\gamma_0(t)=0,
\]
implying that $w(\cdot,t)\in T_h$ for each~$t\in[0,T]$, and we see by
choosing $v(x)=w(x,t)$ that
\[
\iprod{(u_h)_t,w(\cdot,t)}+\iprod{u_h,w(\cdot,t)}_{\mathcal{L}}
	=\gamma_L(t)w(L,t)+\iprod{f(\cdot,t),w(\cdot,t)}.
\]
Since $u_h=w+\eta$,
\[
\iprod{w_t+\eta_t,w}+\iprod{w+\eta,w}_{\mathcal{L}}
	=\gamma_L(t)w(L,t)+\iprod{f,w},
\]
and integration by parts gives
\[
\iprod{\eta,w}_{\mathcal{L}}=[a\eta_xw]_0^L+\iprod{\mathcal{L}\eta,w}
	=\gamma_L(t)w(L,t)+\iprod{\mathcal{L}\eta,w}.
\]
Therefore,
\[
\iprod{w_t,w}+\|w\|_{\mathcal{L}}^2=\iprod{g,w}
\quad\text{where}\quad g(x,t)=f(x,t)-\eta_t(x,t)-(\mathcal{L}\eta)(x,t).
\]
Observe that
\[
\iprod{w_t,w}=\frac{1}{2}\,\frac{d}{dt}\|w(\cdot,t)\|^2,
\]
so after integrating in time to arrive at the estimate
\[
\|w(\cdot,t)\|^2+2\int_0^t\|w(\cdot,s)\|_{\mathcal{L}}^2\,ds
\le\|w_0\|^2+2\int_0^t\iprod{g(\cdot,s),w(\cdot,s)}\,ds
\quad\text{for $0\le t\le T$,}
\]
where
\[
w_0(x)=w(x,0)=(u-\eta)(x,0).
\]
Since $\|w(\cdot,t)\|$ is a continuous function of~$t$, there is a
$t^*\in[0,t]$ such that
\[
\|w(\cdot,t^*)\|=\max_{0\le s\le t}\|w(\cdot,s)\|.
\]
Therefore, using the Cauchy--Schwarz inequality,
\begin{multline*}
\|w(\cdot,t)\|^2\le\|w(\cdot,t)\|^2+2\int_0^t\|w(\cdot,s)\|_{\mathcal{L}}^2\,ds
\le\|w(\cdot,t^*)\|^2+2\int_0^{t^*}\|w(\cdot,s)\|_{\mathcal{L}}^2\,ds\\
\le\|w_0\|^2+2\int_0^{t^*}\|g(\cdot,s)\|\|w(\cdot,s)\|\,ds
\le\|w(\cdot,t^*)\|\biggl(\|w_0\|+2\int_0^t\|g(\cdot,s)\|\,ds\biggr),
\end{multline*}
so
\[
\|w(\cdot,t)\|\le\|w_0\|+2\int_0^t\|g(\cdot,s)\|\,ds,
\]
which gives the desired estimate.
\end{proof}
\begin{corollary}
\begin{multline*}
\|u_h(\cdot,t)\|\le\|u_{0h}\|+2\int_0^t\|f(\cdot,s)\|\,ds\\
	+C\biggl(|\gamma_0(t)|+|\gamma_L(t)|
	+\int_0^t\bigl(|\gamma_0(s)|+|\gamma_0'(s)|
	+|\gamma_L(s)|+|\gamma_L'(s)|\bigr)\,ds\biggr).
\end{multline*}
\end{corollary}

\section{Time stepping}\label{sec: time stepping}
We can use finite differences for the time integration, putting
\begin{equation}\label{eq: uniform tn}
t_n=n\,\Delta t\quad\text{for $0\le n\le N$,}
    \quad\text{where $\Delta t=\frac{T}{N}$,}
\end{equation}
and seeking $U^n_p\approx u_h(x_p,t_n)\approx u(x_p,t_n)$.

\begin{example}
The \emph{forward Euler method} based on~\eqref{eq: semidiscrete fem 1d ODE} is
\[
\boldsymbol{M}\,\frac{\boldsymbol{U}^{n+1}-\boldsymbol{U}^n}{\Delta t}
+(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{U}^n=\boldsymbol{f}^n
    +\boldsymbol{g}^n\quad\text{for $0\le n\le N-1$,}
    \quad\text{with $\boldsymbol{U}^0=\boldsymbol{U}_0$,}
\]
where $\boldsymbol{U}^n=[U^n_p]_{p=1}^P$,
$\boldsymbol{f}^n=\boldsymbol{f}(t_n)$~and
$\boldsymbol{g}^n=\boldsymbol{g}(t_n)$.  Notice that this method is not
actually explicit, due to the presence of the matrix~$\boldsymbol{M}$: at the
$n$th time step we have to solve the linear system
\[
\boldsymbol{M}\boldsymbol{U}^{n+1}=\bigl(\boldsymbol{M}
-\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)\boldsymbol{U}^n
    +\Delta t\,\bigl(\boldsymbol{f}^n+\boldsymbol{g}^n\bigr).
\]
\end{example}

\begin{example}
The \emph{backward Euler method} based on~\eqref{eq: semidiscrete fem 1d ODE} is
\[
\boldsymbol{M}\,\frac{\boldsymbol{U}^{n}-\boldsymbol{U}^{n-1}}{\Delta t}
+(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{U}^n=\boldsymbol{f}^n
    +\boldsymbol{g}^n\quad\text{for $1\le n\le N$,}
    \quad\text{with $\boldsymbol{U}^0=\boldsymbol{U}_0$,}
\]
which requires that we solve the linear systems
\[
\bigl(\boldsymbol{M}+\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{U}^n=\boldsymbol{M}\boldsymbol{U}^{n-1}
    +\Delta t\,\bigl(\boldsymbol{f}^n+\boldsymbol{g}^n\bigr).
\]
\end{example}

\begin{example}
The \emph{Crank--Nicolson method} based on~\eqref{eq: semidiscrete fem 1d ODE}
is
\[
\boldsymbol{M}\,\frac{\boldsymbol{U}^{n}-\boldsymbol{U}^{n-1}}{\Delta t}
+(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{U}^{n-1/2}=\boldsymbol{f}^{n-1/2}
    +\boldsymbol{g}^{n-1/2}\quad\text{for $1\le n\le N$,}
    \quad\text{with $\boldsymbol{U}^0=\boldsymbol{U}_0$,}
\]
which requires that we solve the linear systems
\[
\bigl(\boldsymbol{M}+\tfrac12\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{U}^n
=\bigl(\boldsymbol{M}-\tfrac12\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{U}^{n-1}
    +\Delta t\,\bigl(\boldsymbol{f}^{n-1/2}+\boldsymbol{g}^{n-1/2}\bigr).
\]
\end{example}

\section{Discrete separation of variables}\label{sec: discrete separation}
Consider the special case when $f(x,t)\equiv0$, $\gamma_0(t)\equiv0$~and
$\gamma_L(t)\equiv0$.  Let $(\phi_k,\lambda_k)$ be the $k$th eigenpair for the
operator $\mathcal{L}$, so that
\[
\mathcal{L}\phi_k=\lambda_k\phi_k\quad\text{for $0<x<L$,}\quad
\text{with $\phi_k(0)=0=a\phi_k'(L)$.}
\]
For the special choice of initial condition
\[
u_0(x)=\phi_k(x),
\]
the solution of the continuous problem $u_t+\mathcal{L}u=0$ is simply
\begin{equation}\label{eq: u0 eigenfunction}
u(x,t)=e^{-\lambda_kt}\phi_k(x).
\end{equation}
We shall assume that $c(x)\ge0$ for $0<x<L$, which ensures that $\lambda_k>0$
for all~$k$; see exercise~\ref{ex: quadratic form}.  Thus, the
factor~$e^{-\lambda_kt}$ decreases monotonically to zero as~$t\to\infty$.

Each of the matrices $\boldsymbol{A}$, $\boldsymbol{C}$~and $\boldsymbol{M}$ is
real and symmetric, and in addition $\boldsymbol{M}$ is positive-definite, so
there exist \emph{generalised eigenpairs} $(\boldsymbol{\Phi}_k,\Lambda_k)$
such that
\[
(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{\Phi}_k
    =\Lambda_k\boldsymbol{M}\boldsymbol{\Phi}_k,\qquad
\boldsymbol{\Phi}_j^T(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{\Phi}_k
    =\Lambda_k\delta_{jk},\qquad
\boldsymbol{\Phi}_j^T\boldsymbol{M}\boldsymbol{\Phi}_k=\delta_{jk}
\]
for $j$, $k\in\{1,2,\ldots,P\}$.  Our assumptions ensure that
$\boldsymbol{A}+\boldsymbol{C}$ is positive-definite, so $\Lambda_k>0$
for~$1\le k\le2P$.

Suppose we choose discrete initial data
\[
\boldsymbol{U}_0=\boldsymbol{\Phi}_k.
\]
For moderate values of~$k$, we expect
$\boldsymbol{\Phi_k}\approx[\phi_k(x_p)]_{p=1}^P$, assuming we label the
$\lambda_k$~and $\Lambda_k$ so that
\[
0<\lambda_1\le\lambda_2\le\lambda_3\le\cdots
\quad\text{and}\quad
0<\Lambda_1\le\Lambda_2\le\cdots\le\Lambda_P,
\]
and make appropriate choices for the corresponding eigenfunctions and
eigenvectors. In our special case, the semidiscrete initial-value
problem~\eqref{eq: semidiscrete fem 1d ODE} simplifies to
\[
\boldsymbol{M}\,\frac{d\boldsymbol{U}}{dt}
+(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{U}=\boldsymbol{0}
    \quad\text{for $0\le t\le T$,}
    \quad\text{with $\boldsymbol{U}(0)=\boldsymbol{\Phi}_k$,}
\]
which has the solution
\begin{equation}
\boldsymbol{U}(t)=e^{-\Lambda_kt}\boldsymbol{\Phi}_k,
\end{equation}
which again decreases monotonically to zero as~$t\to\infty$.

Let us compare theses continuous-time solutions with the discrete-time
solutions for the three time-stepping schemes described in
section~\ref{sec: time stepping}.

\begin{example}
The forward Euler method in this special case is
\[
\boldsymbol{M}\boldsymbol{U}^{n+1}=\bigl(\boldsymbol{M}
-\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)\boldsymbol{U}^n,
\]
and since
\[
\bigl(\boldsymbol{M}
-\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)\boldsymbol{\Phi}_k
    =\boldsymbol{M}\Phi_k-\Delta t\,\Lambda_k\boldsymbol{M}\boldsymbol{\Phi}_k
    =\boldsymbol{M}(1-\Delta t\,\Lambda_k)\boldsymbol{\Phi}_k
\]
we see by induction on $n$ that
\[
\boldsymbol{U}^n=(1-\Delta t\,\Lambda_k)^n\boldsymbol{\Phi}_k.
\]
If $\Delta t\,\Lambda_k<1$ then the factor~$(1-\Delta t\,\Lambda_k)^n$
decreases monotonically to zero as~$t_n\to\infty$, so we have the correct
qualitative behaviour of the solution, and furthermore
\[
e^{-\Lambda_kt_n}=e^{-\Lambda_k n\,\Delta t}
    =\bigl(e^{-\Delta t\Lambda_k}\bigr)^n\approx(1-\Delta t\,\Lambda_k)^n
    \quad\text{if $\Delta t\,\Lambda_k$ is small, for $n$ fixed.}
\]
If $1<\Delta t\,\Lambda_k<2$ then $(1-\Delta t\,\Lambda_k)^n$ tends to zero,
but with an oscillating sign, as $t_n\to\infty$, which is the wrong qualitative
behaviour. If $\Delta t\,\Lambda_k>2$, then $(1-\Delta t\,\Lambda_k)^n$ not
only oscillates but is also unbounded.
\end{example}

\begin{example}
The \emph{backward Euler method} in this special case is
\[
\bigl(\boldsymbol{M}+\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{U}^n=\boldsymbol{M}\boldsymbol{U}^{n-1},
\]
and since
\[
\bigl(\boldsymbol{M} +\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)^{-1}
\boldsymbol{M}\boldsymbol{\Phi}_k
    =(1+\Delta t\,\Lambda_k)^{-1}\boldsymbol{\Phi_k}
\]
we see by induction on $n$ that
\[
\boldsymbol{U}^n=(1+\Delta t\,\Lambda_k)^{-n}\boldsymbol{\Phi}_k.
\]
The factor~$(1+\Delta t\,\Lambda_k)^{-n}$ always decreases monotonically to
zero as~$t_n\to\infty$, so we have the correct qualitative behaviour no matter
how large $\Delta t$.  Furthermore,
\[
e^{-\Lambda_kt_n}=\bigl(e^{-\Delta t\Lambda_k}\bigr)^n
    \approx(1+\Delta t\,\Lambda_k)^{-n}
    \quad\text{if $\Delta t\,\Lambda_k$ is small, for $n$ fixed.}
\]
\end{example}

\begin{example}
The \emph{Crank--Nicolson method} in this special case is
\[
\bigl(\boldsymbol{M}+\tfrac12\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{U}^n
=\bigl(\boldsymbol{M}-\tfrac12\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{U}^{n-1},
\]
and since
\[
\bigl(\boldsymbol{M}+\tfrac12\Delta t\,
    (\boldsymbol{A}+\boldsymbol{C})\bigr)^{-1}
\bigl(\boldsymbol{M}-\tfrac12\Delta t\,(\boldsymbol{A}+\boldsymbol{C})\bigr)
    \boldsymbol{\Phi_k}
    =\frac{(1-\tfrac12\Delta t\,\Lambda_k)^{-1}}%
{(1+\tfrac12\Delta t\,\Lambda_k)^{-1}}\,\boldsymbol{\Phi_k},
\]
we see by induction on $n$ that
\[
\boldsymbol{U}^n=\biggl(
    \frac{1-\tfrac12\Delta t\,\Lambda_k}{1+\tfrac12\Delta t\,\Lambda_k}\biggr)^n
    \boldsymbol{\Phi}_k.
\]
Thus, the correct qualitative behaviour occurs when $\Delta t\,\Lambda_k<2$.
If $\Delta t\,\Lambda_k>2$, then $\boldsymbol{U}^n$ still tends to zero
as~$t_n\to\infty$, but with a sign oscillation.  Also, rate of decay becomes
slower and slower with increasing values of~$\Delta t\,\Lambda_k$.
\end{example}

For general initial data~$u_0$, we know from section~\ref{sec: separation}
that the continuous solution is
\[
u(x,t)=e^{-t\mathcal{L}}u_0(x)
    =\sum_{k=1}^\infty\widehat{(u_0)}_k e^{-\lambda_kt}\phi_k(x)
\quad\text{where}\quad
\widehat{(u_0)}_k=\frac{\langle u_0,\phi_k\rangle}{\|\phi_k\|^2}.
\]
Similarly, we claim that for a general discrete
data~$\boldsymbol{U}_0$, the semidiscrete finite element solution has the form
\begin{equation}\label{eq: semidiscrete FEM Fourier}
\boldsymbol{U}(t)=e^{-t\boldsymbol{M}^{-1}(\boldsymbol{A}+\boldsymbol{C})}
    \boldsymbol{U}_0
    =\sum_{k=1}^P\widehat{(\boldsymbol{U}_0)}_k e^{-\Lambda_kt}
    \boldsymbol{\Phi}_k
\quad\text{where}\quad
\widehat{(\boldsymbol{U}_0)}_k
=\frac{\langle\boldsymbol{U}_0,\boldsymbol{\Phi}_k\rangle_{\boldsymbol{M}}}%
{\|\boldsymbol{\Phi_k}\|_{\boldsymbol{M}}^2}.
\end{equation}
Here, $\langle\boldsymbol{V},\boldsymbol{W}\rangle_{\boldsymbol{M}}
=\boldsymbol{V}^T\boldsymbol{M}\boldsymbol{W}$ and
$\|\boldsymbol{V}\|_{\boldsymbol{M}}
=\sqrt{\boldsymbol{V}^T\boldsymbol{M}\boldsymbol{V}}$ are the inner product and
norm induced by the positive-definite matrix~$\boldsymbol{M}$.  Indeed,
$\boldsymbol{U}(t)=e^{-t\boldsymbol{M}^{-1}(\boldsymbol{A}+\boldsymbol{C})}
\boldsymbol{U}_0$ because
\[
\frac{d\boldsymbol{U}}{dt}+\boldsymbol{M}^{-1}(\boldsymbol{A}+\boldsymbol{C})
\boldsymbol{U}=\boldmath{0},
\]
and we have
\[
\boldsymbol{U}_0=\sum_{k=1}^P\widehat{(\boldsymbol{U}_0)}_k\boldsymbol{\Phi}_k,
\]
because the eigenvectors~$\boldsymbol{\Phi}_k$ form an
$\boldsymbol{M}$-orthogonal basis for~$\mathbb{R}^P$.  The
representation~\eqref{eq: semidiscrete FEM Fourier} then follows because
$\boldsymbol{M}^{-1}(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{\Phi}_k
=\Lambda_k\boldsymbol{\Phi}_k$.

\begin{example}\label{example: forward Euler FEM}
For general discrete initial data~$\boldsymbol{U}_0$, the forward Euler method
gives
\[
\boldsymbol{U}^n=\sum_{k=1}^P\widehat{(\boldsymbol{U}_0)}_k
    (1-\Delta t\,\Lambda_k)^n\Phi_k,
\]
and therefore
\[
\|\boldsymbol{U}^n\|_{\boldsymbol{M}}^2
    =\sum_{k=1}^P\bigl|\widehat{(\boldsymbol{U}_0)}_k\bigr|^2
    |1-\Delta t\,\Lambda_k|^{2n}\|\Phi_k\|_{\boldsymbol{M}}^2.
\]
Since
\[
\|\boldsymbol{U}_0\|_{\boldsymbol{M}}^2
    =\sum_{k=1}^P\bigl|\widehat{(\boldsymbol{U}_0)}_k\bigr|^2
    \|\Phi_k\|_{\boldsymbol{M}}^2,
\]
it follows that
\[
\|\boldsymbol{U}^n\|_{\boldsymbol{M}}
    \le\Bigl(\max_{1\le k\le P}|1-\Delta t\,\Lambda_k|^n\Bigr)
    \|\boldsymbol{U}_0\|_{\boldsymbol{M}}
	=(|1-\Delta t\,\Lambda_P)^n\|\boldsymbol{U}_0\|_{\boldsymbol{M}}.
\]
Hence, if $\Delta t\le 1/\Lambda_P$ then $\|\boldsymbol{U}^n\|_{\boldsymbol{M}}
\le\|\boldsymbol{U}_0\|_{\boldsymbol{M}}$ and the forward Euler method is
stable in the $\boldsymbol{M}$-norm.
\end{example}

\begin{example}
A similar argument shows that the backward Euler method gives
\[
\|\boldsymbol{U}^n\|_{\boldsymbol{M}}
    \le\Bigl(\max_{1\le k\le P}(1+\Delta t\,\Lambda_k)^{-n}\Bigr)
    \|\boldsymbol{U}_0\|_{\boldsymbol{M}}
	=(1+\Delta t\,\Lambda_1)^{-n}\|\boldsymbol{U}_0\|_{\boldsymbol{M}},
\]
and hence we have stability in the $\boldsymbol{M}$-norm for any~$\Delta t$.
\end{example}

\begin{example}
For the Crank--Nicolson method,
\[
\|\boldsymbol{U}^n\|_{\boldsymbol{M}}
    \le\biggl(\max_{1\le k\le P}\biggl|
    \frac{1-\tfrac12\Delta t\,\Lambda_k}{1+\tfrac12\Delta t\,\Lambda_k}
    \biggr|^n\biggr)\|\boldsymbol{U}_0\|_{\boldsymbol{M}}
=\biggl|\frac{1-\tfrac12\Delta t\,\Lambda_1}{1+\tfrac12\Delta t\,\Lambda_1}
	\biggr|^n\|\boldsymbol{U}_0\|_{\boldsymbol{M}},
\]
and we again have stability in the $\boldsymbol{M}$-norm for any~$\Delta t$.
\end{example}

\begin{Exercises}

\exercise\label{ex: quadratic form}
Suppose that $\boldsymbol{A}\in\mathbb{R}^{N\times N}$ is symmetric and
positive-semidefinite, and let
$\|\boldsymbol{v}\|=\sqrt{\boldsymbol{v}^T\boldsymbol{v}}$ denote the
Euclidean norm in~$\boldsymbol{R}^N$.
\begin{description}
\item{(i)}
Show that the minimum eigenvalue of~$\boldsymbol{A}$ equals the minimum value
of the \emph{Rayleigh quotient}:
\[
\lambda_{\min}(\boldsymbol{A})
    =\min_{\boldsymbol{0}\ne\boldsymbol{v}\in\mathbb{R}^N}
    \frac{\boldsymbol{v}^T\boldsymbol{A}\boldsymbol{v}}{\|\boldsymbol{v}\|^2}.
\]
Hint: choose an orthonormal basis for~$\mathbb{R}^N$ consisting of eigenvectors
of~$\boldsymbol{A}$.
\item{(ii)} Show likewise for the maximum eigenvalue that
\[
\lambda_{\max}(\boldsymbol{A})
    =\max_{\boldsymbol{0}\ne\boldsymbol{v}\in\mathbb{R}^N}
    \frac{\boldsymbol{v}^T\boldsymbol{A}\boldsymbol{v}}{\|\boldsymbol{v}\|^2}.
\]
\item{(iii)} How does this result generalise if we replace the Euclidean norm
with the norm~$\|\boldsymbol{v}\|_{\boldsymbol{M}}=\sqrt{\boldsymbol{v}^T
\boldsymbol{M}\boldsymbol{v}}$ induced by a symmetric and strictly
positive-definite matrix~$\boldsymbol{M}\in\mathbb{R}^{N\times N}$?
\end{description}

\exercise\label{ex: polynomial derivative}
Let $f(\xi)=a_1+a_2\xi+a_3\xi^2+\cdots+a_{r+1}\xi^r$ be any real polynomial of
degree at most~$r$, and form the associated (column) vector of coefficients
$\boldsymbol{a}=[a_i]_{i=1}^{r+1}$.
\begin{description}
\item{(i)} Find the matrix~$\boldsymbol{B}$ such that
$\int_0^1|f(\xi)|^2\,d\xi=\boldsymbol{a}^T\boldsymbol{B}\boldsymbol{a}$.
\item{(ii)} Prove that $\boldsymbol{B}$ is symmetric and (strictly)
positive-definite.
\item{(iii)} Find the matrix~$\boldsymbol{D}$ such that
$\int_0^1|f'(\xi)|^2\,d\xi=\boldsymbol{a}^T\boldsymbol{D}\boldsymbol{a}$.
\item{(iv)} Prove that $\boldsymbol{D}$ is symmetric and positive-semidefinite.
\item{(v)} With the help of exercise~\ref{ex: quadratic form}, find a
constant~$C_r$ (depending only on~$r$) such that
\[
\int_0^1|f'(\xi)|^2\,d\xi\le C_r\int_0^1|f(\xi)|^2\,d\xi.
\]
\item{(vi)} Compute the numerical value of $C_r$ for $r\in\{0,1,2,3\}$.
\end{description}
\begin{ans}
(i) $\boldsymbol{B}=[b_{ij}]_{i,j=1}^{r+1}$ where $b_{ij}=1/(i+j-1)$\quad
(iii) $\boldsymbol{D}=[d_{ij}]_{i,j=1}^{r+1}$ where
\[
d_{ij}=\begin{cases}
    0&\text{if $i=1$ or $j=1$,}\\
    (i-1)(j-1)/(i+j-3)&\text{otherwise.}
\end{cases}
\]
(iv) $C_r=\lambda_{\max}(\boldsymbol{D})=\max_{\boldsymbol{a}\ne\boldsymbol{0}}
\boldsymbol{a}^T\boldsymbol{D}\boldsymbol{a}/
\boldsymbol{a}^T\boldsymbol{B}\boldsymbol{a}$
(vi)
\begin{center}
\renewcommand{\arraystretch}{1.25}
\begin{tabular}{r|r}
$r$&$C_r$\\
\hline
   0&     0.00\\
   1&    12.00\\
   2&    60.00\\
   3&   170.12
\end{tabular}
\end{center}
\end{ans}

\exercise
Let $C_r$ be the constant from part (v) of
exercise~\ref{ex: polynomial derivative}, and recall the notation used in
section~\ref{sec: accuracy of interpolation}.
\begin{description}
\item{(i)} Show that
\[
\int_{x_{p-1}}^{x_p}|f(x)|^2\,dx=h_p\int_0^1|\hat f(\xi)|^2\,d\xi
\quad\text{and}\quad
\int_{x_{p-1}}^{x_p}|f'(x)|^2\,dx=h_p^{-1}\int_0^1|\hat f'(\xi)|^2\,d\xi.
\]
\item{(ii)} Deduce that if $f\in\mathbb{P}_r$ then
\[
\int_{x_{p-1}}^{x_p}|f'(x)|^2\,dx
    \le C_rh_p^{-2}\int_{x_{p-1}}^{x_p}|f(x)|^2\,dx.
\]
\item{(iii)} Hence prove the \emph{inverse inequality}: if $v$ is a continuous
piecewise polynomial of degree at most~$r$, then
\[
\int_0^L|v'(x)|^2\,dx\le C_rh_{\min}^{-2}\int_0^L|v(x)|^2\,dx
\quad\text{where $h_{\min}=\min_{1\le p\le P}h_p$.}
\]
\end{description}

\exercise
Recall the notation from sections
\ref{sec: semidiscrete parabolic FEM}--\ref{sec: discrete separation}.
\begin{description}
\item{(i)}
Show that for any $v$, $w\in T_h$,
\[
\int_0^L\bigl(a(x)v'w'+c(x)vw\bigr)\,dx
    =\boldsymbol{W}^\top(\boldsymbol{A}+\boldsymbol{C})\boldsymbol{V}
\quad\text{and}\quad
\int_0^Lv(x)w(x)\,dx=\boldsymbol{W}^T\boldsymbol{M}\boldsymbol{V},
\]
where $V_p=v(x_p)$~and $W_p=w(x_p)$.
\item{(ii)}
Hence prove that
\[
\Lambda_{\max}\le C_1h_{\min}^{-2}a_{\max}+c_{\max}\quad\text{where}\quad
a_{\max}=\max_{0\le x\le L}a(x)\quad\text{and}\quad
c_{\max}=\max_{0\le x\le L}c(x).
\]
\item{(iii)}
What does the upper bound in part~(ii) imply about the stability of the forward
Euler method?
\end{description}
\begin{ans}
(iii) We saw in example~\ref{example: forward Euler FEM} that if $\Delta
t>1/\Lambda_P$ then the forward Euler method has at least one unstable mode.
Since $\Lambda_P=O(h_{\min}^{-2})$, to ensure stability we must expect to need
$\Delta t$ of order~$h_{\min}^2$.  This conclusion is consistent with the
stability condition of Theorem~\ref{thm: explicit Euler stability} for the
finite difference case.
\end{ans}

\end{Exercises}


\section{Limits}

\subsection{Delta-Epsilon Definition on R}

The limit of $f$ as $x$ approaches $a$ is $L$ if given a positive real number
$\epsilon$, there exists a corresponding $\delta$ so that numbers selected with distance from $a$
less than $\delta$ and $>0$ it is guaranteed that outputs under $f$ are within distance $\epsilon$ from $L$.

\begin{mybox}{Formal definition}
    $\lim_{x\to a}f(x)=L$ if $\forall \epsilon >0$, $\exists\delta > 0$ such that $0< |x-a| < \delta \implies |f(x)-L|<\epsilon$
\end{mybox}

\begin{center}
    \includegraphics[scale=0.5]{figures/Screen Shot 2021-05-12 at 3.29.55 PM.png}
\end{center}

\subsection{Definition of Limit on R2}

If $f:\R^2\to \R$ and $P_0=(x_0,y_0)$, then $\lim _{(x, y) \rightarrow\left(x_{0}, y_{0}\right)} f(x, y)=L$
if given any small positive number $\epsilon$, $\delta>0$ is guaranteed so when there is a point $C\neq P_0$ within circle of radius $\delta$ about $P_0$, $f(x,y)$ lies within $\epsilon$ of $L$.

\begin{mybox}{Formal definition}
\[\lim _{(x, y) \rightarrow\left(x_{0}, y_{0}\right)} f(x, y)=\lim _{\tb{x} \rightarrow P_{0}} f(x, y)=L \text { if } \forall \varepsilon>0 \;\exists \delta>0\]
such that
\[0<\sqrt{\left(x-x_{0}\right)^{2}+\left(y-y_{0}\right)^{2}}<\delta \Longrightarrow|f(x, y)-L|<\varepsilon\]
\end{mybox}

\begin{mybox}{Continuous functions}
    \begin{itemize}
        \item Polynomials: $f(x,y)=x^2y+xy^2$
        \item Exponentials: $f(x,y)=e^{xy}$
        \item Trigonometric: $f(x,y)=\sin(x+y)$
        \item Compositions of any continuous functions: $f(x,y)=\cos(e^{x^2y-xy^2})$
        \item Sums, differences, products of continuous functions: $f(x, y)=x^{2} y-x y^{2}+e^{x y} \sin (x y)$
        \item Quotients of continuous functions (domain of quotient does not include zeros of denominator): $f(x, y)=\frac{x^{2 y}-x y^{2}}{\sin (x y)}$
    \end{itemize}
\end{mybox}

\subsection{Computational Techniques}

Can use continuity to find limit (plug in point). Can also use conjugate multiplication to simplify and compute. Cancelling terms works because a limit approaches a value instead of equaling it.
Can also squeeze a function between 2 continuous ones to find limit.

Let $\lim _{(x, y) \rightarrow(0,0)} \frac{x^{2}}{\sqrt{x^{2}+y^{2}}}$ exist.
Know that $0\leq \lim _{(x, y) \rightarrow(0,0)} \frac{x^{2}}{\sqrt{x^{2}+y^{2}}}$.
Because $y^2>0$, $\frac{x^{2}}{\sqrt{x^{2}+y^{2}}} \leq \frac{x^{2}+y^{2}}{\sqrt{x^{2}+y^{2}}}$.
Thus, $\lim _{(x, y) \rightarrow(0,0)} 0 \leq \lim _{(x, y) \rightarrow(0,0)} \frac{x^{2}}{\sqrt{x^{2}+y^{2}}} \leq \lim _{(x, y) \rightarrow(0,0)}\left(x^{2}+y^{2}\right)^{1 / 2}$
so the limit is 0.

Other strategies: If a term $k(x,y)\geq 0\;\forall\; (x,y)\in \R^2$, then can simply perform $\pm 1$ to denominator for squeeze theorem proofs.

\begin{mybox}{Common inequalities}
\begin{itemize}
    \item AM-GM inequality: $\frac{x+y}{2}\geq \sqrt{xy}\implies (x-y)^2\geq 0$
    \item Triangle inequality: $|x+y|\leq |x|+|y|$ and $|x-y|\geq |x|-|y|$
    \item $|e^x-1|\leq |x|e^{|x|}$
    \item $(x\pm y)^2\geq 0$
\end{itemize}
\end{mybox}

Proof that $\frac{\sin x}{x}=1$:

\begin{center}
    \includegraphics[scale=0.4]{figures/Screen Shot 2021-04-21 at 10.59.12 AM.png}
\end{center}

It is evident that:
\begin{gather}
    \frac{1}{2}\sin x\leq \frac{1}{2}x\leq \frac{1}{2}\tan x\\
    \sin x\leq x \leq \tan x\\
    1\leq \frac{x}{\sin x}\leq \frac{1}{\cos x}\\
    1\geq \frac{\sin x}{x}\geq \cos x
\end{gather}

Applying the squeeze theorem to 4:

\begin{gather*}
    \lim_{x\to 0}1\geq \lim_{x\to 0}\frac{\sin x}{x}\geq \lim_{x\to 0} \cos x\\
    1\geq \frac{\sin x}{x}\geq 1
\end{gather*}

Thus, $\lim_{x\to 0}\frac{\sin x}{x}=1$.

\section{Derivatives}

\subsection{Limit Definition}

A function $f:\R^m \to \R^n$ is differentiable at $P_0$ if there is a linear transformation from $n\times m$ matrix $Df(P_0)$ satisfying

\[\lim_{\tb{x}\to P_0}\frac{\big\vert \big\vert f \big( \tb{x} \big)-\big(f(P_0)+\mathrm{D}f(P_0)(\tb{x}-P_0) \big) \big\vert \big\vert}{\vert \vert \tb{x}-P_0 \vert \vert}=0\]

If the function is differentiable at this point, then $Df(P_0)$ is the derivative of $f$ at $P_0$. Called Jacobian matrix of $f$ at $P_0$.

Can adapt this to single-variable calculus:

\[\displaystyle \lim_{x\to a} \frac{\vert f(x)-\big(f(a)+(m_a)(x-a)\big) \vert}{\vert x-a \vert}=0\]

Simplifying and reconfiguring to limit definition of a derivative:

\begin{align*}
    \lim_{x\to a} \frac{\vert f(x)-\big(f(a)+(m_a)(x-a)\big) \vert}{\vert x-a \vert}&=0\\
    \lim_{x\to a}\left\vert \frac{f(x)-\big(f(a)+(m_a)(x-a)\big)}{x-a} \right\vert&=0\\
    \left \vert \lim_{x\to a} \frac{f(x)-\big(f(a)+(m_a)(x-a)\big)}{x-a} \right \vert&=0\\
    \lim_{x\to a} \frac{f(x)-\big(f(a)+(m_a)(x-a)\big)}{x-a}&=0 \, \mbox{since}\, \vert 0 \vert=0\\
    \lim_{x\to a} \left( \frac{f(x)-f(a)}{x-a}-m_a\right)&=0\\
    \lim_{x\to a}\frac{f(x)-f(a)}{x-a}&=m_a
\end{align*}

Equivalent to the case of a $1\times 1$ matrix $Df(P_0)$ where the single entry is $f'(a)$.
Also $m_a=f'(a)\implies f(x)=f(a)+f'(a)(x-a)$ is the linear approximation.

As $x$-values approach $a$, $f(x)-(f(a)+f'(a)(x-a))$ approaches 0 faster. Thus, $\frac{f(x)-\big(f(a)+f'(a)(x-a)\big)}{x-a}$ approaches 0.

\subsection{Multivariable Application}

Approximating plane of some $f:\R^2\to \R$ is $L_{P_0}(x,y)=f(a,b)+f_x(a,b)(x-a)+f_y(a,b)(y-b)$, i.e. the plane spanned by
$\langle 1,0,f_x(a,b)\rangle$ and $\langle 0,1,f_y(a,b)\rangle$ passing through $(a,b,f(a,b))$.

Is identical to $\begin{bmatrix}f_x(P_0)&f_y(P_0)\end{bmatrix}\begin{bmatrix}x-a\\y-b\end{bmatrix}$. The matrix is the matrix of partial derivatives.

Such a function is differentiable if:

\[\displaystyle \lim_{(x,y)\to(a,b)}\frac{f(x,y)-\left(f(a,b)+\Big(\begin{matrix} f_x(a,b)&f_y(a,b) \end{matrix} \Big)\left(\begin{matrix}x-a\\y-b\\ \end{matrix} \right) \right)}{\sqrt{(x-a)^2+(y-b)^2}}=0\]

Numerator is linear approximation and denominator is distance to point $(a,b)$. Thus:

\begin{mybox}{Differentiability}
\[f(x,y) \mbox{ is differentiable at the point }P_0\mbox{ if}\displaystyle \lim_{(x,y)\to P_0}\frac{|f(x,y)-L_{P_0}(x,y)|}{||(x,y)-P_0||}=0\]
\end{mybox}

Geometrically, if a circle about $P_0$ is drawn and radius is collapsed, distance between $f$ and approximating plane will become 0 much faster, so fraction becomes 0.

If there is a radius $\delta > 0$ on the disk of radius $\delta$ centered at $P_0$, $f_x(x,y)$ and $f_y(x,y)$ are continuous at every point on the disk, then $f(x,y)$ is continuous at every point on the disk.
Differentiability can not be established by the existence of partial derivatives at a point, it is sufficient to demonstrate continuous partials
at a point.

The criteria for differentiability are that $f_x,f_y$ exist and $f$ is locally linear at $P_0$. The tangent plane exists here. However,
it is sufficient to demonstrate differentiability by showing that both $f_x$ and $f_y$ are continuous on an open disk $D$ to conclude that $f(x,y)$
is differentiable on $D$. Alternatively, the above limit definition can be used.

If a function $f:\R^m\to \R^n$ exists, then it is made up of $n$ component functions on $\R^m$. The Jacobian matrix is as follows:

\[Df\left( x_1,\ldots x_m\right) =\begin{bmatrix} \nabla f_{1}\left(x_1,\ldots x_m\right) \\ \vdots \\ \nabla f_{n}\left(x_1,\ldots x_m\right) \end{bmatrix}\]

The number of rows $n$ depends on components (codomain) of the output space whereas the domain determines columns ($m$). Thus, linear approximation to a function $f:\R^m\to \R^n$ at $P_0$ is

\[L_{P_0}(\tb{x})=f(P_0)+\Big(\mbox{matrix of partial derivatives at}\,P_0\Big)(\tb{x}-P_0)\]

In parametric equations (paths), let $\tb{c}(t)=\big(x(t), y(t), z(t) \big)$ have continuous partials.
Thus, the matrix of partials $D\tb c(t)=\begin{bmatrix}x'(t)\\y'(t)\\z'(t)\end{bmatrix}$. Is the vertical velocity vector.
If the approximation $L_{t_0}(t)$ is found, it yields:

\begin{align*}
    L_{t_0}(t)&=\tb c(t_0)+
    \left(
    \begin{matrix}
    x'(t_0)\\
    y'(t_0)\\
    z'(t_0)\\
    \end{matrix}
    \right)
    (t-t_0)\\
    &=\tb{c} (t_0)+(t-t_0)\tb c\,'(t_0)
\end{align*}

Also, note that matrix of partials for a scalar valued function $f:\R^n\to \R$ is simply the gradient; there is 1 row and $n$ columns in the gradient vector.

\section{Derivative Rules}

\begin{mybox}{General rules}
If $f,g:\R^m\to \R^n$ are differentiable at $P_0\in \R^m$ for $c\in \R$:
\begin{itemize}
    \item $h(\tb{x})=cf(\tb{x})\implies \mathrm{D}h(P_0)=cDf(P_0)$
    \item $h(\tb{x})=f(\tb{x})+g(\tb{x})\implies \mathrm{D}h(P_0)=\mathrm{D}f(P_0)+\mathrm{D}g(P_0)$ (domain = codomain; sum of two $n\times m$ matrices)
\end{itemize}

If $f,g:\R^n\to \R$ are differentiable at $P_0\in \R^n$:
\begin{itemize}
    \item $h(\tb{x})=f(\tb{x})g(\tb{x})\implies \mathrm{D}h(P_0)=f(P_0)\mathrm{D}g(P_0)+g(P_0)\mathrm{D}f(P_0)$ (no commutativity as matrix times scalar DNE)
\end{itemize}

If $f,g:\R^n\to \R$ are both differentiable at $P_0\in \R^n$ and $g(P_0)\neq 0$:
\begin{itemize}
    \item $\mathrm{D}\left(\frac{f}{g}\right)(P_0)=\frac{g(P_0)\mathrm{D}f(P_0)-f(P_0)\mathrm{D}g(P_0)}{g(P_0)^2}$
\end{itemize}
\end{mybox}

\subsection{Chain Rule}

\begin{mybox}{Chain rule}
If $g:\R^m\to \R^p$ and $f:\R^p\to \R^n$ and $g$ is differentiable at $P_0\in \R^m$ and the same for $f$ at $g(P_0)$:
\begin{itemize}
    \item $\mathrm{D}\Big(f\circ g\Big)(P_0)=\mathrm{D}f\big(g(P_0)\big)\mathrm{D}g(P_0)$
\end{itemize}
\end{mybox}

Thus, derivative of composites is a matrix product. $\mathrm{D}f(g(P_0))\to n\times p$ and $\mathrm{D}g(P_0)\to p\times m$
so $\mathrm{D}(f\circ g)(P_0)\to n\times m$.

If $\tb{c}:\R\to \R^n$ is a path differentiable at $t_0\in \R$ and $f:\R^n\to \R$ is differentiable at $\tb{c}(t_0)$, then:

\[\frac{d}{dt}f(\tb{c}(t_0))=\mathrm{D}f(\tb{c}(t_0))D\tb{c}(t_0)=\left[ \nabla f\left( x_{1},x_{2},\ldots x_{n}\right) \right]\begin{bmatrix} \vdots \\ \tb{c}\,'(t_0) \\ \vdots \end{bmatrix}=\nabla f(\tb{c}\,'(t_0))\cdot \tb{c}\,'(t_0)\]

Thus, derivative of scalar valued function and path is dot product of gradient and velocity.

\section{Gradient}

\subsection{Directional Derivative}

Directional derivative of a scalar function $f:\R^n\to \R$ at some $P_0\in\R^n$ in the direction of $\tb{v}\in\R^n$ is given by

\[\boxed{f_{\tb{v}}\left(P_{0}\right)=\nabla f\left(P_{0}\right) \cdot \frac{\tb{v}}{\|\tb{v}\|}}\]

Maximizing the directional derivative involves the dot product. Let $f:\R^n\to \R$ and $P_0\in\R^n$ where $\nabla f(P_0)$ is defined.
Also, let $\tb{u}$ be a unit vector such that $f_{\tb{u}}(P_0)$ is maximized. The directional derivative is

\[f_{\tb u}(P_0)=\nabla f(P_0)\cdot \tb u=\|\nabla f(P_0)\|\cos \theta\]

Since $\cos\theta \in [-1,1]$, the directional derivative is maximized at $\cos\theta = 1\implies \tb{u}||\nabla f(P_0)||$.

\begin{mybox}{Gradient fact}
The gradient of a function at $P_0$ points in the direction of steepest ascent.
\end{mybox}

For a more general case given $f:\R^n\to \R$ and $P_0\in\R^n$ where $\nabla f_{\tb{v}}(P_0)$ is maximized and $\nabla f(P_0)$ exists:
\begin{align*}
    \nabla f_{\tb{v}}(P_0)&=\nabla f(P_0) \cdot \frac{\tb{v}}{||\tb{v}||}\\
    &=\nabla f(P_0) \cdot \frac{\nabla f(P_0)}{||\nabla f(P_0)||}\\
    &=\frac{\nabla f(P_0)\cdot \nabla f(P_0)}{||\nabla f(P_0)||}\\
    &=||\nabla f(P_0)||
\end{align*}

Thus, the length of the gradient vector tells how steep the ascent is in the direction of the steepest ascent.

\subsection{Gradient Fields}

A image in the codomain $\R^n$ where arrows originating at each point point in the steepest direction $\forall\; P\in \R^n$ where the partials exist.
Are always perpendicular to level curves of $f$.

Let $f:\R^n\to \R$ and $P\in \R^n$ a point where partials of $f$ exist. Considering the level set $f(\tb{x})=f(P)$,
all values map to same codomain value as $P$. Also, let $\tb{c}:\R^n\to \R$ have an image entirely within this level set $f(\tb{x})=f(P)$
and $\tb{c}(t_0)=P$. Then, the proof must show that $\nabla f(P)\cdot \tb c\,'(t_0)=0$.

Note that since $h(t)=f(\tb{c}(t))$, $h'(t_0)=\nabla f \Big(\tb c (t_0) \Big)\cdot \tb c\,'(t_0)=\nabla f (P)\cdot  \tb c\,'(t_0)$.
Since the image of $\tb{c}(t)$ lies within $f(\tb{x})=f(P)$, $f$ is constant on the image of $\tb{c}$ (every point on $\tb{c}$ maps to $f(P)$ under $f$).
Thus, $h(t)=f(\tb{c}(t))\implies h'(t)=0\;\forall\;t$.

\subsection{Sphere}

Sphere is not a function as a point $P=(x,y)$ correspond to multiple $z$ values.
Can be viewed as a level surface in $\R^3$ for some $f:\R^3\to \R$. The function is $f(x,y,z)=x^2+y^2+z^2$
and the level surface sphere is $x^2+y^2+z^2=R^2$ where $R$ is the radius.
Given the point $(0,0,R)$, the gradient is upward along the $z$-axis. All curves passing through this point
have a tangent vector perpendicular to $\nabla f(0,0,R)$ since gradients are perpendicular to level sets.
It can be said that any vector perpendicular to $\nabla f(0,0,R)$ is a tangent to at least 1 curve in the sphere passing through $(0,0,R)$.
Thus, the perpendicular space to $\nabla f(0,0,R)$ forms a plane of tangent vectors at $(0,0,R)$ and is a tangent plane.
An expression for the tangent plane is

\begin{align*}
    \nabla f (0,0,R)\cdot(x-0, y-0, z-R)&=0\\
    f_x(0,0,R)x+f_y(0,0,R)y+f_z(0,0,R)&=0\\
    2(0)x+2(0)y+2(R)(z-R)&=0\\
    2Rz-2R^2&=0\\
    z&=R\\
\end{align*}

Expression for the tangent plane to a level set of some $f:\R^3\to \R$:

\begin{align*}
    \nabla f (P_0)\cdot(x-x_0, y-y_0,z- z_0)&=0\\
    f_x(P_0)(x-x_0)+f_y(P_0)(y-y_0)+f_z(P_0)(z-z_0)&=0\\
    f_x(P_0)x+f_y(P_0)y+f_z(P_0)z&=\nabla f(P_0)\cdot P_0\\
\end{align*}
$$\boxed{f_x(P_0)x+f_y(P_0)y+f_z(P_0)z=\nabla f(P_0)\cdot P_0}$$

\section{Implicit Function Theorem}

\subsection{Single variable}

Is the explanation behind single-variable implicit differentiation.
If $F:\R^2\to \R$ is of class $C^1$ and there is a point $\tb{x}_0=(a,b)\in\R^2$ such that
$F(a,b)=0$ and $F_y(a,b)\neq 0$, then it is guaranteed that
\begin{itemize}
    \item There is some small $\delta>0$ and some small $\epsilon>0$ so that for $(x,y)\in\R^2$ with $|x-a|<\delta$ and $|y-b|<\epsilon$ there is a unique function $g:\R\to\R$ satisfying
    $F(x,g(x))=0$ and any point on the level set $F(x,y)=0$ satisfying these 2 conditions on coordinates will have property $y=g(x)$. To summarize, near the point $(a,b)$ points on $F(x,y)=0$ lie on the graph of a unique function $y=g(x)$.
    \item $g$ is of class $C^1$; differentiable so $g'(x)$ is continuous and $g'(x)=-\frac{F_x}{F_y}$
\end{itemize}

\begin{center}
    \includegraphics[scale=0.4]{figures/Screen Shot 2021-05-10 at 7.08.59 AM.png}
\end{center}

\begin{mybox}{Special case}
    Given that $F:\R^{n+1}\to \R$ has continuous partials, let $(\tb x, z)$ denote the points in $\R^{n+1}$
    where $\tb x\in \R^n$ and $z\in \R$. Assume that $(\tb x_0,z_0)$ satisfies

    \[F(\tb x_0,z)=0\;\text{and}\;\frac{\partial F}{\partial z}(\tb x_0,z)\neq 0\]

    Then, $\tb x_0$ and $z_0$ are contained in a neighborhood such that there exists a unique $z=g(\tb x)$ defined for $\tb x$ and $z$ in these neighborhoods such that

    \[F(\tb x, g(\tb x))=0\]

    It can also be said that $z=g(\tb x)$ is continuously differentiable with derivative

    \[\mathbf{D} g(\mathbf{x})=-\left.\frac{1}{\frac{\partial F}{\partial z}(\mathbf{x}, z)} \mathbf{D}_{\mathbf{x}} F(\mathbf{x}, z)\right|_{z=g(\mathbf{x})}\]

    $\tb D_{\tb x}F$ denotes the partial of $F$ with respect to $\tb x$ so that $\mathbf{D}_{\mathbf{x}} F=\left[\partial F / \partial x_{1}, \ldots, \partial F / \partial x_{n}\right]$.
    Thus,

    \[\frac{\partial g}{\partial x_{i}}=-\frac{\partial F / \partial x_{i}}{\partial F / \partial z}, \quad i=1, \ldots, n\]
\end{mybox}

\subsection{Multivariable definition}

If $F:\R^3\to \R$ is of class $C^1$ and there exists $\tb{x}_0=(a,b,c)\in\R^3$ such that
$F(a,b,c)=0$ and $F_z(a,b,c)\neq 0$ then is is guaranteed that
\begin{itemize}
    \item There is some small $\delta > 0$ and $\epsilon > 0$ so that for $(x,y,z)\in\R^3$ with $\sqrt{(x-a)^2+(y-b)^2}<\delta$ and $|z-c|<\epsilon$ there is a unique function $g:\R^2\to\R$
    that satisfies $F(x,y,g(x,y))=0$ and any point satisfying these inequalities on $x,y,z$-coordinates that lies on $F(x,y,z)=0$ will have the property $z=g(x,y)$, so near the point $(a,b,c)$ on the level set $F(x,y,z)=0$ points can be seen as lying on $z=g(x,y)$.
    \item $g$ is of class $C^1$ so it is differentiable and both partials exist and are continuous, and $g_x(x,y)=-\frac{F_x}{F_z},g_y(x,y)=-\frac{F_y}{F_z}$.
\end{itemize}

The IFT can justify the existence of a tangent plane to a surface.

Let $F:\R^3\to \R$ be of class $C^1$ and the surface $S=\{(x,y,z)\in\R\;|\;F(x,y,z)=c\}$ for some $c\in\R$ be the level set of $F$.
The tangent plane is given by $F_x(P)x+F_y(P)y+F_z(P)z=\nabla F(P)\cdot \tb P$ for $P=(x_0,y_0,z_0)$. The plane is not defined when
$\nabla F(P)=\tb 0$.

Applying the IFT, this plane is tangent to $S$. If $\nabla F(P)\neq \tb 0$ then at least one of $F_x,F_y,F_z\neq 0$ at $P$.
Without loss of generality, let $F_z\neq 0$. Then, let $f(x,y,z)=F(x,y,z)-c$. Since $f_z(P)=F_z(P)\neq 0$, applying the IFT to $f$ makes the conclusion that
near $P$ $z$ is a differentiable function of $x,y$ so that there is a unique function $g:\R^2\to \R$ such that $z=g(x,y)$. The tangent plane to $g$ at $P$ is
\begin{align*}
    z &=g\left(x_{0}, y_{0}\right)+g_{x}\left(x_{0}, y_{0}\right)\left(x-x_{0}\right)+g_{y}\left(x_{0}, y_{0}\right)\left(y-y_{0}\right) \\
    &=z_{0}+g_{x}\left(x_{0}, y_{0}\right)\left(x-x_{0}\right)+g_{v}\left(x_{0}, y_{0}\right)\left(y-y_{0}\right)
\end{align*}

It can then be shown that this plane is the same as given by the gradient approximation:

It follows that $f_x(P)=F_x(P)$ and $f_y(P)=F_y(P)$. Then,

\begin{align*}
    z&=z_0+g_x(x_0,y_0)(x-x_0)+g_y(x_0,y_0)(y-y_0)\\
    z&=z_0-\frac{F_x(P)}{F_z(P)}(x-x_0)-\frac{F_y(P)}{F_z(P)}(y-y_0)\\
    F_z(P)z&=F_z(P)z_0-F_x(P)x+F_x(P)x_0-F_y(P)y+F_yy_0\\
    F_x(P)x+F_y(P)y+F_z(P)z&=F_z(P)z_0+F_x(P)x_0+F_y(P)y_0\\
    F_x(P)x+F_y(P)y+F_z(P)z&=\nabla F(P)\cdot P
\end{align*}

\begin{mybox}{General implicit theorem}
    If $\mathrm{det}(A)\neq 0$ where $A$ is the $m\times m$ matrix

    \[
        \left[\begin{array}{ccc}
            \frac{\partial F_{1}}{\partial z_{1}} & \cdots & \frac{\partial F_{1}}{\partial z_{m}} \\
            \vdots & \vdots \\
            \frac{\partial F_{m}}{\partial z_{1}} & \cdots & \frac{\partial F_{m}}{\partial z_{m}}
        \end{array}\right]
    \]

    then near the point $(\tb x_0,\tb z_0)$, the following system

    \[\begin{array}{c}
        F_{1}\left(x_{1}, \ldots, x_{n}, z_{1}, \ldots, z_{m}\right)=0 \\
        F_{2}\left(x_{1}, \ldots, x_{n}, z_{1}, \ldots, z_{m}\right)=0 \\
        \vdots \\
        F_{m}\left(x_{1}, \ldots, x_{n}, z_{1}, \ldots, z_{m}\right)=0 .
    \end{array}\]

    defines smooth and unique functions such that

    \[z_{i}=k_{i}\left(x_{1}, \ldots, x_{n}\right) \quad(i=1, \ldots, m)\]

    where the derivatives are calculared with implicit differentiation.
\end{mybox}

\subsection{IFT for vector-valued functions}

Let $F:\R^{m+n}\to \R^n$ be of class $C^1$, then
\[F(x_1, x_2, x_3,...,x_m, z_1, z_2, ..., z_n)=\Big(F_1(\mbox{input vector}), F_2(\mbox{input vector}), F_3(\mbox{input vector}),...,F_n(\mbox{input vector})\Big)\]

The inputs to $F$ are the $m$-dimensional vector $\tb x_0=(x_1,x_2,\ldots,x_m)$ and the $n$-dimensional vector $\tb z_0=(z_1,z_2,\ldots, z_n)$.
The conditions for IFT are met when
\begin{itemize}
    \item $F(\tb x_0,\tb z_0)=\tb 0$
    \item $\mbox{Det}\left(
        \begin{matrix}
        \frac{\partial F_1}{\partial z_1}&\dots&\frac{\partial F_1}{\partial z_n}\\
        \vdots&\ddots&\vdots\\
        \frac{\partial F_n}{\partial z_1}&\dots&\frac{\partial F_n}{\partial z_n}
        \end{matrix}
        \right)\neq 0$
\end{itemize}

Then there is a unique $C^1$ function $g:\R^m\to \R^n$ such that for all points satisfying $F(\tb v)=\tb 0$ that are sufficiently near $P=(\tb x_0,\tb z_0)$ we can conclude $\tb z=g(\tb x)$.

\begin{mybox}{Inverse function theorem}
    This case is in the attempt to solve the system

    \[\left.\begin{array}{c}
        f_{1}\left(x_{1}, \ldots, x_{n}\right)=y_{1} \\
        \ldots \\
        f_{n}\left(x_{1}, \ldots, x_{n}\right)=y_{n}
    \end{array}\right\}\]

    Solving this is the same as inverting the equations of this system.
    The condition is that in a neighborhood of some $\tb x_0$, $\mathrm{det}(A)\neq 0$
    where $A$ is the determinant of $\tb Df(\tb x_0)$ and $f=(f_1,\ldots,f_n)$. $\mathrm{det}(A)$
    (the Jacobian determinant) is given by

    \[\left.\frac{\partial\left(f_{1}, \ldots, f_{n}\right)}{\partial\left(x_{1}, \ldots, x_{n}\right)}\right|_{\mathbf{x}=\mathbf{x}_{0}}=J(f)\left(\mathbf{x}_{0}\right)=\left|\begin{array}{ccc}
        \frac{\partial f_{1}}{\partial x_{1}}\left(\mathbf{x}_{0}\right) & \cdots & \frac{\partial f_{1}}{\partial x_{n}}\left(\mathbf{x}_{0}\right) \\
        \vdots & & \vdots \\
        \frac{\partial f_{n}}{\partial x_{1}}\left(\mathbf{x}_{0}\right) & \cdots & \frac{\partial f_{n}}{\partial x_{n}}\left(\mathbf{x}_{0}\right)
    \end{array}\right|\]

    If this determinant is not 0, then the system can be solved for $\tb x=g(\tb y)$ for $\tb x$ near $\tb x_0$ and $\tb y$ near $\tb y_0$ and $g$ has continuous partials.
\end{mybox}

This is reasonable because a linear approximation acts in an invertible way locally, so both the derivative and function can be said to be invertible.

\subsection{Example problem}

Given a nonlinear system of equations:
\begin{align*}
    xu+yvu^2&=2\\
    xu^3+y^2v^4&=2\\
\end{align*}

Implicit differentiation is done by treating $u$ and $v$ as functions of $x,y$. In this case,
can find how a small change in $x$ affects $u$ when all variables are 1 (i.e. $(x,y,u,v)=(1,1,1,1)$)).
First equation differentiation gives
\begin{align*}
    \frac{\partial }{\partial x}\Big(xu+yvu^2\Big)&=\frac{\partial }{\partial x}
    \Big(2\Big)\\
    u+x\frac{\partial u}{\partial x}+yv\left(2u\frac{\partial u}{\partial x}\right)+yu^2\frac{\partial v}{\partial x}&=0\\
    (x+2uvy)\frac{\partial u}{\partial x}+yu^2\frac{\partial v}{\partial x}&=-u
\end{align*}

Second equation gives
\begin{align*}
    \frac{\partial }{\partial x}\Big(xu^3+y^2v^4\Big)&=\frac{\partial}{\partial x}(2)\\
    x\Big(3u^2\frac{\partial u}{\partial x}\Big)+u^3+y^24v^3\frac{\partial v}{\partial x}&=0\\
    3xu^2\frac{\partial u}{\partial x}+y^24v^3\frac{\partial v}{\partial x}&=-u^3\\
\end{align*}

Plugging in $(1,1,1,1)$ gives
\begin{align*}
    3u_x+v_x&=-1\\
    3u_x+4v_x&=-1\\
\end{align*}

Thus, $u_x=-\frac{1}{3}$. In this problem, $F$ is defined as $F(x,y,u,v)=(xu+yvu^2-2,xu^3+y^2v^4-2)$. Or,
\begin{align*}
    F_1(x,y,u,v)&=xu+yvu^2-2\\
    F_2(x,y,u,v)&=xu^3+y^2v^4-2
\end{align*}

The matrix to be checked for the condition is
\[
    \begin{bmatrix}
    \frac{\partial F_1}{\partial u}&\frac{\partial F_1}{\partial v}\\
    \frac{\partial F_2}{\partial u}&\frac{\partial F_2}{\partial v}\\
    \end{bmatrix}
    =
    \begin{bmatrix}
    x+2yvu&yu^2\\
    3xu^2&4y^2v^3\\
    \end{bmatrix}
    \implies
    \begin{bmatrix}
        3&1\\
        3&4
    \end{bmatrix}
\]

Since the determinant is 9, the IFT application is valid.


\chapter{Matrix Group}
\section{Linear Groups}

\begin{defn}[normal subgroup]
A \textbf{normal subgroup} $N$ of a given group $G$ is a subgroup which left and right cosets $gN$ and $Ng$ are the same: $gN = Ng$, i.e., $$gNg^{-1} = N$$ for every $g\in G.$ We denote it as $N \triangleleft G$.
\end{defn}
\begin{prop}[why normal?] The quotient group $G/N$ (read $G$ mod $N$) is well-defined if and only if $N$ is normal of G.
\end{prop}
\begin{proof}
Whatever we take, the equivalence class must be the same. If $N$ is normal and letting $x\sim \tilde x$, i.e., $x^{-1}\tilde x \in N$, $$\tilde xN \subseteq (xN)N = xN$$ and \textit{vice versa}. If $\bar x = \bar{\tilde x} $ if $x\sim \tilde x$ and $\overline{xy} = \bar x \bar y,$ we have $$gN = (1g)N = NgN = hNgN = (hg)N$$ for every $h\in N$, hence $N = g^{-1} hgN$ so that $N$ is normal: letting $n = g^{-1}hg\tilde{n}$, we have $g^{-1}hg=n\tilde{n}^{-1}\in N$ for every $h\in N$ whence $g^{-1}Ng \subseteq N.$
\end{proof}
\begin{defn}[general linear group and special linear group] The \textbf{general linear group} of a given vector space $V$ is a (multiplicative) group of automorphism on $V$; that is, $\operatorname{GL}(V) = \mathfrak L(V,V)^\times.$ And the \textbf{special linear group} of $V$ is the subgroup of $\operatorname{GL}(V)$ which is consisted by linear transformations whose determinant are all 1.

We denote $\operatorname{GL}(n, F) = \operatorname{GL}(F^n),$ and $\operatorname{SL}(n, F) = \operatorname{SL}(F^n).$
\end{defn}
\begin{theorem}[first isomorphism theorem]
For any homomorphism $\varphi:~G\to H$ for two groups $G$ and $H$, $$G/\ker \varphi \approx \operatorname{im} \varphi.$$
\end{theorem}
\begin{proof}
\hfill
\begin{center}
\leavevmode
\xy
\xymatrix {
G \ar@{->}[rr]^{\varphi} \ar@{->}[dr] &&\operatorname{im}\varphi\\ & G/\operatorname{ker}\varphi \ar@{->}[ur]_{\approx} &\\
}
\endxy
\end{center}
\end{proof}
\begin{prop}[GL and SL] $$\operatorname{GL}(V)/\operatorname{SL}(V) \approx F^\times.$$
\end{prop}

\begin{defn}[center] The \textbf{center} $Z(G)$ of a group $G$ is the subgroup of elements which satisfy the `commutative law', i.e., $$Z(G) = \{z\in G: \quad zg = gz \textrm{ for every }g\in G\}.$$ $Z$ for \textit{zentrum}, which means `center' in German.
\end{defn}
\begin{prop}[normality of the center]
$$Z(G)\triangleleft G.$$
\end{prop}
\begin{proof}
Trivially, $$gZ(G) = \{gz:~z\in Z(G)\} = \{zg:~z \in Z(G)\} = Z(G)g.$$
\end{proof}
\begin{ex}
What are the centers of (a) $\operatorname{GL}(n, F)$ and (b) $\operatorname{SL}(n, F)$?

Answer: (a) $0\ne cI$'s, (b) $\alpha I$'s where $\alpha^n = 1.$
\end{ex}
\begin{proof}
(a) is just all. Let $AZ = ZA$ for all invertible $A.$ Then, especially, for all \textit{elementary matrices}, $EZ = ZE.$ Note that multiplying $E$ left is the same with elementary \textit{row} operating, while multiplying right is for elementary \textit{column} operating. (Especially, for $E_{i + cj}$'s.) Hence we obtain that $Z$ is diagonal. Instead a more detailed explanation, we see an example: $$
\begin{pmatrix}
1 & 3 \\
0 & 1
\end{pmatrix}
\begin{pmatrix}
a & *_1 \\
*_2 & b
\end{pmatrix}
=
\begin{pmatrix}
a + 3*_2 & *_1 + 3b \\
*_2 & b
\end{pmatrix},$$
$$\begin{pmatrix}
a & *_1 \\
*_2 & b
\end{pmatrix}
\begin{pmatrix}
1 & 3 \\
0 & 1
\end{pmatrix}
=
\begin{pmatrix}
a  & *_1 + 3a \\
*_2 & b + 3*_2
\end{pmatrix},$$
hence $*_1$ and $*_2$ are zero. Similar details says that $Z$ must be diagonal, for bigger matrices.

Now, the proof is done: since $E_{i\leftrightarrow j}$ is an elementary matrix, $Z_{ii} = Z_{jj}$, for every $i$ and $j$ pair. Therefore $Z$ is a `nonzero'(since $Z$ is invertible!) multiple of $I.$

Not so surprisingly, the proof works on any \textit{ring} with 1; if we modify `nonzero' to `invertible', that is, $c\in R^{\times}.$
\end{proof}
\begin{add}[divide by center?] Dividing by center means to ignore the difference due to the elements of $Z.$ Since $$Z(G) = \{z\in G:\quad z = gzg^{-1}\textrm{ for every }g\in G\},$$ we have some `morphisms' $\varphi_g:~a \mapsto b = gzg^{-1}$ and \textit{their group} $$\operatorname{Inn}(G) = \{ \varphi_g:\quad g \in G\}.$$ We call this group the \textbf{inner automorphism group} of $G$.

We want to show that $G/Z(G) \approx \operatorname{Inn}(G).$ The idea is easy: use the homomorphism $\varphi_\bullet$ above: $$\varphi_\bullet:\quad G \to \operatorname{Inn}(G).$$ The kernel of this homomorphism is just the center of $G$, since the (multiplicational) identity of $\operatorname{Inn}(G)$ is the identity function $\operatorname{id}_G$ and, from $$\varphi_z = z\bullet z^{-1} = \operatorname{id}_G, \qquad \forall g\in G,$$i.e., $$\varphi_z (g) = zgz^{-1} = g, \qquad \forall g\in G,$$ we get $zg = gz$ whence $z \in Z(G).$ Therefore $\ker (\varphi_\bullet) = Z(G)$, and by the first isomorphism theorem, we obtain $$G/Z(G) \approx \operatorname{Inn}(G).$$
\end{add}

\begin{defn}[PGL and PSL]
The \textbf{projective general linear group} is defined by $$\operatorname{PGL}(V) = \operatorname{GL}(V)/Z(\operatorname{GL}(V)).$$
The \textbf{projective special linear group} is defined by $$\operatorname{PSL}(V) = \operatorname{SL}(V)/Z(\operatorname{SL}(V)).$$
\end{defn}

Projective geometry is difficult...

\section{Orthogonal group}
\begin{defn}[orthogonal transformation]
For an \textit{inner product space} $(V,~\langle \bullet , \bullet \rangle)$ (or even just a quadratic space with non-degenerate symmetric bilinear form), an \textbf{orthogonal transformation} of $V$ is an invertible linear transformation which preserves the given inner product, that is, such $A\in \operatorname{GL}(V)$: $$\langle v, ~w \rangle = \langle Av,~ Aw \rangle.$$ The group of such transformations is called the \textbf{orthogonal group} $\operatorname{O}(V)$ of $V$, and also denote $\operatorname{O}(n, F) = \operatorname{O}(F^n)$ and $\mathrm O(n) = \mathrm O(n, \mathbb R)$. $F^n$ is considered with \textit{dot product.}

Similarly, $\operatorname{SO}(V) = \{T\in \operatorname{O}(V):~ \operatorname{det}T = 1\}$, and analogous definitions for $\operatorname{SO}(n,F)$ and $\operatorname{SO}(n).$ Obviously, it is called the \textbf{special orthogonal group} of $V$.
\end{defn}
\begin{defn}[unitary group]
If we give a \textit{hermitian form} $(V, ~ \langle \bullet, \bullet \rangle )$ rather than an inner product, where the given field is `trivially' the field $\mathbb C$ of complex number, we define analogously \textbf{unitary group} $\operatorname{U}(n)$ as we defined the orthogonal group:$$\langle v, ~w \rangle = \langle Av,~ Aw \rangle, \qquad A \in \operatorname{GL}(n,~\mathbb C).$$ We \textit{already know} what is $\operatorname{SU}(n)$ and how to call it \textsf{:D}.
\end{defn}

\begin{prop}
\hfill
\begin{itemize}
    \item $\operatorname{SO}(V) \triangleleft\operatorname{O}(V) \triangleleft \operatorname{GL}(V);$
    \item $\operatorname{SO}(V) \triangleleft\operatorname{SL}(V) \triangleleft \operatorname{GL}(V);$
    \item $\operatorname{O}(n, F) = \{ A\in \mathfrak M_{n,n}(F)^\times:~ A^{-1}=A^\mathsf T\},$ if the inner product is a standard one, so-called \textit{dot product}. (\textit{Canonically isomorphic!})
\end{itemize}
Also the followings hold: for $\operatorname{O}(n, F)$,  every element is a matrix with pairwise orthonormal columns (or rows).
\end{prop}
Good, well, why it is called `orthogonal'? It is because these preserves the `angle' of two vectors, especially the \textit{orthogonality}. Then, \textit{what} is orthogonal? Which matrices are orthogonal?
\begin{prop}
\hfill

In $\mathbb R^2$, $\operatorname{O}(2)$ consists of rotations and reflections. And the group of rotations is just $\operatorname{SO}(2).$
\begin{proof}From
$$A=\begin{pmatrix}a&b\\c&d\end{pmatrix}, \qquad AA^\mathsf T = \begin{pmatrix}a&b\\c&d\end{pmatrix}\begin{pmatrix}a&c\\b&d\end{pmatrix} = \begin{pmatrix}a^2+b^2&ac+bd\\ac+bd&c^2+d^2\end{pmatrix} = I,$$
we obtain $a^2 + b^2 = 1 = c^2 + d^2$ and $ac+bd = 0$. Solutions for the first equality are just sines and cosines, namely: $$a = \cos x, ~~b = \sin x, \quad c = \cos y,~~d = \sin y.$$ (Orders of sine and cosine do not have to consider; since there is an inversion $\theta~\mapsto~\frac \pi 2 - \theta$.) Evaluating to another equality, $$\cos x \cos y + \sin x \sin y = \cos(y-x) = 0,\qquad y-x = \frac{2k-1}{2}\pi.$$ Hence $y = x + \frac{2k-1}{2}\pi.$ Substituting it, we get $$c = - \sin x \sin \left(\frac{2k-1}{2}\pi\right) = \mp \sin x,\qquad d = \cos x \sin \left(\frac{2k-1}{2}\pi\right) = \pm \cos x.$$ Due to a \textit{custom} in math and other sciences, we use $\theta = -x$ and finally get $$A=\begin{pmatrix}\cos \theta &-\sin \theta \\ \pm \sin \theta & \pm \cos \theta \end{pmatrix}.$$

If the signa of second row are pluses, $$A_+ = \begin{pmatrix}\cos \theta &-\sin \theta \\\sin \theta & \cos \theta \end{pmatrix} = R_\theta,$$ where $R_\theta$ is the \textbf{rotation matrix} of angle $\theta$. Since $ \operatorname{det}R_\theta = 1$, $R_\theta \in \operatorname{SO}(2).$

If the signa of second row are minuses, $$A_- = \begin{pmatrix}\cos \theta &-\sin \theta \\-\sin \theta & -\cos \theta \end{pmatrix}  = \begin{pmatrix}1&0\\0&-1\end{pmatrix} \begin{pmatrix}\cos \theta &-\sin \theta \\\sin \theta & \cos \theta \end{pmatrix} =\begin{pmatrix}1&0\\0&-1\end{pmatrix} R_\theta = S_{-\theta/2},$$ where $S_\varphi$ is a \textbf{reflection matrix} w.r.t. a line $\theta = \varphi$ in polar coordinate system. (Draw it $\sim \!.$) Note that $\operatorname{det}S_\varphi = -1$.
\end{proof}
\end{prop}

How about 3-dimensional space? We consider a rotation on a line, the \textit{axis}. For example, there are `basic' three rotations: $$\displaystyle {\begin{alignedat}{1}R_{x}(\theta )&={\begin{pmatrix}1&0&0\\0&\cos \theta &-\sin \theta \\[3pt]0&\sin \theta &\cos \theta \\[3pt]\end{pmatrix},}\\[6pt]R_{y}(\theta )&={\begin{pmatrix}\cos \theta &0&\sin \theta \\[3pt]0&1&0\\[3pt]-\sin \theta &0&\cos \theta \\\end{pmatrix},}\\[6pt]R_{z}(\theta )&={\begin{pmatrix}\cos \theta &-\sin \theta &0\\[3pt]\sin \theta &\cos \theta &0\\[3pt]0&0&1\\\end{pmatrix}.}\end{alignedat}}$$ Surprisingly, they are almost \textit{all}, i.e., the following holds. (Details are omitted.)

\begin{theorem}[decomposition of rotation]
For every `rotation' $R\in\operatorname{SO}(3)$,
$$R=R_{z}(\alpha )\,R_{y}(\beta )\,R_{x}(\gamma )\,$$
 where \textit{Tait-Bryan angles} of $R$ are $\alpha$, $\beta$, $\gamma$, about axes $z$, $y$, $x$ respectively.
\end{theorem}

Following our knowledge, there efinition for \textit{arbitrary rotation} is quite obvious, and only acceptable:

\begin{defn}[rotation] \textbf{Rotation} is an element of SO.\end{defn}

We must figure out the following definitions.
\begin{defn}[PGO, PSO, PGU, PSU]Projective (general) orthogonal group $\operatorname{PGO}(V)$ and projective special orthogonal group $\operatorname{PSO}(V).$ Similarly for U's...
\end{defn}
\begin{ex}
Calculate them! What is $Z(\operatorname{O}(V))$ and $Z(\operatorname{SO}(V))$?
\end{ex}
\begin{proof}
Same with \textbf{Example 1.1.} A difference is that $\det Z = \pm 1$ in $\operatorname{O}(V).$ Another one is for $\operatorname{SO}$: for odd-dimensional $V$, $Z(\operatorname{SO}(V))=\{I\}$ (a trivial group) since $\operatorname{det}\pm I = \pm 1$; while $Z(\operatorname{SO}(V))=\{\pm I\}$ for even-dimensional $V$ since $\operatorname{det}\pm I = 1.$
\end{proof}
\begin{coro}
\emph{PSO $\approx$ SO} for odd-dimensional vector space $V$.
\end{coro}
\begin{prop}
$$\operatorname{PSU}(2)\approx \operatorname{SO}(3), \qquad \operatorname{SU}(2) \dhxrightarrow{\text{double}} \operatorname{SO}(3),$$ where $\dhxrightarrow{\text{double}}$ means that there is a double covering.
\end{prop}

``The shortest path between two truths in the real domain passes through the complex domain.'' ---Jacques Hadamard.

\begin{proof}
$Z(\operatorname{SU}(2)) = \{\pm I\}$? Trivial. Then it suffices to show that $\operatorname{PSU}(2)\approx \operatorname{SO}(3)$. A transformation $A$ of $\operatorname{PSU}(2)$ satisfies \textsf{(U)} $AA^\dagger = I$ by the definition. Use same method as \textbf{Proposition 1.5.}: let $$A=\begin{pmatrix}a&b\\c&d\end{pmatrix},$$ then $$AA^\dagger = \begin{pmatrix}a&b\\c&d\end{pmatrix}\begin{pmatrix}\bar a&\bar c\\\bar b&\bar d\end{pmatrix} = \begin{pmatrix}a\bar a+b\bar b&a\bar c+b\bar d\\\bar a c+\bar b d&c\bar c + d\bar d\end{pmatrix}=I$$ whence $$|a|^2 + |b|^2 = 1 = |c|^2 + |d|^2, \qquad a\bar c + b\bar d = 0.$$ The first equality gives us $$a = e^{i\varphi_1}\cos x,~~b = e^{i\varphi_2}\sin x,~~c = e^{i\varphi_3}\cos y,~~d = e^{i\varphi_4}\sin y,$$ and the second equality gives $$\cos x \cos y + e^{i(-\varphi_1+\varphi_2+\varphi_3-\varphi_4)}\sin x \sin y = 0,$$ since $\overline{e^{i\theta}} = e^{-i\theta}$ for real $\theta$. If $-\varphi_1+\varphi_2+\varphi_3-\varphi_4 \ne 0,$ the equality must not hold unless $b=d=0$, which leads to a contradiction. Also, since \textsf{(S)} $\operatorname{det}A = 1$, $$ad-bc = e^{i(\varphi_1 + \varphi_4)}(\cos x \sin y - \sin x \cos y) = e^{i(\varphi_1 + \varphi_4)} \sin(y-x) = 1$$ whence $\varphi_1 + \varphi_4 =\varphi_2 + \varphi_3 = k\pi$ and $y-x = \frac{2k-1}{2} \pi.$ Therefore $$A = \begin{pmatrix}e^{i\varphi_1}\cos x&e^{i\varphi_2}\sin x \\ \mp e^{- i\varphi_2}\sin x&\pm e^{-i\varphi_1}\cos x \end{pmatrix}.$$ Finally, \textsf{(P)} ignore one signum of them, then we have $$A = \begin{pmatrix}e^{i\varphi_1}\cos \theta & - e^{i\varphi_2}\sin \theta \\  e^{- i\varphi_2}\sin \theta& e^{-i\varphi_1}\cos \theta \end{pmatrix}.$$ Hence, for example, there is an `isomorphism' $$\begin{pmatrix}e^{i\varphi_1}\cos \theta & - e^{i\varphi_2}\sin \theta \\  e^{- i\varphi_2}\sin \theta& e^{-i\varphi_1}\cos \theta \end{pmatrix} \leftrightarrow (\theta, \varphi_1, \varphi_2) \leftrightarrow R_z(\theta)R_y(\varphi_1)R_x(\varphi_2),$$ since $R_\bullet (\alpha + \beta) = R_\bullet (\alpha)R_\bullet (\beta).$ Therefore $\operatorname{PSU}(2)\approx \operatorname{SO}(3).$
\end{proof}
\section{SO(1,1)}
\begin{defn}[indefinite orthogonal group] Consider the Euclidean space only, i.e.,   $F=\mathbb R$.
The \textbf{indefinite orthogonal group} $\operatorname{O}(p, q)$ is something like O, but the inner product is not provided while the following bilinear form is given: $$\langle v, w\rangle = v^\mathsf T \operatorname{diag}(\underbrace{1, \cdots, 1}_{p}, \underbrace{-1, \cdots, -1}_{q}) w,$$
 for $(p+q)$-dimensional vectors $v$ and $w$. For instance, $\operatorname{O}(n) = \operatorname{O}(n,0) = \operatorname{O}(0,n).$
And SO$(p,q)$ is ...
\end{defn}
We are interested in O(1,1) and O(1,3) in particular.
\begin{prop}
$\operatorname{SO}(1,1)$ can be represented by a hyperbolae $x^2 - y^2 = 1$, hence 2 connected curves. $\operatorname{SO}^+$ is the `connected' component of this group which contains the identity $I$, $$\operatorname{SO}^+ = \left\{ \begin{pmatrix}\cosh \theta & \sinh \theta \\ \sinh \theta & \cosh \theta\end{pmatrix}:~~ \theta\in\mathbb R \right\}.$$ In fact, we call the connected component of a given `topological' group that contains the identity element the \textbf{identity component} of given group.
\end{prop}
\begin{center}

\begin{tikzpicture}
\draw[->] (-3,0) -- (3,0) node[right] {$x$};
\draw[->] (0,-3) -- (0,3) node[above] {$y$};
\draw[dashed] (-3,-3) -- (3,3);
\draw[dashed] (3,-3) -- (-3,3);
\draw[scale=1,domain=-3:3,smooth,variable=\y,blue,thick] plot ({(\y*\y + 1)^(1/2)},{\y});
\node at (2.3,1.5) {$\operatorname{SO}^{+}$};
\draw[scale=1,domain=-3:3,smooth,variable=\y,blue,thick] plot ({-(\y*\y + 1)^(1/2)},{\y});
\node at (-2.5,1.5) {$-\operatorname{SO}^{+}$};
\node at (0.2,-0.5) {O};
\end{tikzpicture}
\end{center}
\begin{proof}
Completely same process. Note that if $A\in\operatorname{SO}(1,1)$, $$v^\mathsf T \begin{pmatrix}1&0\\0&-1\end{pmatrix}w = \langle v,w\rangle = \langle Av,Aw\rangle =v^\mathsf T A^\mathsf T \begin{pmatrix}1&0\\0&-1\end{pmatrix}Aw,$$ hence $$\begin{pmatrix}1&0\\0&-1\end{pmatrix} = A^\mathsf T \begin{pmatrix}1&0\\0&-1\end{pmatrix} A.$$ Then we have $$\operatorname{SO}(1,1) = \left\{ \pm \begin{pmatrix}\cosh \theta & \sinh \theta \\ \sinh \theta & \cosh \theta\end{pmatrix}:~~ \theta\in\mathbb R \right\}.$$ We \textit{can(?)} represent it as a parametrized hyperbola: $$\pm \begin{pmatrix}\cosh \theta & \sinh \theta \\ \sinh \theta & \cosh \theta\end{pmatrix} \longleftrightarrow \pm\begin{pmatrix}\cosh \theta\\ \sinh \theta\end{pmatrix},$$ then $\operatorname{SO}^+$ and $-\operatorname{SO}^+$ are connected components of $\operatorname{SO}(1,1).$
\end{proof}
What does `connected' means? Detail definition is in \textit{topology}: it cannot separated by some open sets.

We will stop our work here about groups for the time being. If we learn \textit{topology} or \textit{Lie group theory}, it will continue...


\subsection{Structural supply and demand functions}

Supply:

\(Q_s=\alpha_1 + \beta_1P+\gamma_1I + \epsilon_1\)

Demand:

\(Q_d=\alpha_2 + \beta_2P+\gamma_2I + \epsilon_2\)

Can't estimate because the equations are simulataneous.

To estimate, \(cov(P, \epsilon_1)\) needs to be \(0\), but what is it?

\(cov(P, \epsilon_1)= E[(P-E[P])(\epsilon_1-E[\epsilon_1])]\)
\(cov(P, \epsilon_1)= E[(P-E[P])(\epsilon_1-E[\epsilon_1])]\)