\documentclass{article}

\usepackage{amsmath}
\usepackage{amssymb}

\usepackage{comment}
\usepackage{dirtytalk}

\usepackage{cleveref}

\bibliographystyle{ieeetr}

\begin{document}

\section{The Abstract Koopman Operator} \label{sec:Abstract}

    Koopman operators consist on a dimension lifting technique that allows us to model a Finite Dimensional Nonlinear Dynamical System through an Infinite Dimensional Linear Dynamical System.

    Koopman operators provide a convenient framework where traditional, more mature and optimized Linear System Identification and Control tools can be applied to a nonlinear system. In contrast with conventional linearization methods that depend on local derivatives, the Koopman operator provides an exact description (at least in theory) of the system's dynamics and, more importantly from the Applied Controls point of view, it is easily adaptable to a fully data-driven pipeline, provided we have some way of probing the internal states of the system (gray-box framework).

    We start by stating the abstract problem, partially borrowing the definition from Mezić et Al.\cite{Koopman_Basics}\cite{Applied_Koopmanism}. Consider a continuous-time dynamical system such as the one given in \cref{eq:CT_Dyn} on the state space $M$, where $\mathbf{x}$ is a coordinate vector of the state, and $\textbf{F}$ is (assumed to be) a non-linear vector-valued smooth function of the same dimension as its argument.

    \begin{equation} \label{eq:CT_Dyn}
        \mathbf{\dot{x}} = \mathbf{F}\left( \mathbf{x} \right) , \mathbf{x} \in M
    \end{equation}

    Let $S^t(\mathbf{x}_0)$ denote the position the position at time $t$ of the trajectory of the system in \cref{eq:CT_Dyn} that starts at $t=0$ from the state $\mathbf{x}_0$, and let $g:M \mapsto \mathbb{C}$ an \textit{arbitrary} (at least for now), complex-valued function. We call $g$ an observable of the system in \cref{eq:CT_Dyn}, whose value observed over a trajectory starting from $\mathbf{x}_0$ at $t=0$ changes with time according to the flow, as described in \cref{eq:obs_traj}.

    \begin{equation} \label{eq:obs_traj}
        g\left(t,\mathbf{x}_0\right) = g \left(\mathbf{S}^t\left(\mathbf{x}_0\right)\right)
    \end{equation}

    The space $\mathcal{G}$ of all observables such as $g$ is a linear functional space, and thus we can define a family of linear operators $U^t:\mathcal{G} \mapsto \mathcal{G}$ with $t \in \left[0,\infty\right)$ through \cref{eq:Koopman_CT}.

    \begin{equation} \label{eq:Koopman_CT}
        U^t g\left(\mathbf{x}_0\right) = g \left(S^t \left(\mathbf{x}_0\right)\right)
    \end{equation}

    Thus, for a fixed $t$, $U^t$ maps the vector-valued observable $g(\mathbf{x}_0)$ into $g\left(t,\mathbf{x}_0\right)$. We will call the family of all operators $U^t$, indexed by time $t$, the \textbf{Koopman Semigroup} given by the generator $U$ of the continuous-time system in \cref{eq:CT_Dyn}.

    In Discrete Time, the definition is even simpler. Let \cref{eq:DT_Dyn} describe a discrete-time dynamical system with $\mathbf{z} \in M$ and $\mathbf{T}:M \mapsto M$. Then, the associated Koopman operator $U$ is defined by \cref{eq:Koopman_DT}.

    \begin{equation} \label{eq:DT_Dyn}
        \mathbf{z}_{k+1} = \mathbf{T}\left(\mathbf{z}\right) , \left\{\mathbf{z}_k,\mathbf{z}_{k+1}\right\} \subset M
    \end{equation}

    \begin{equation} \label{eq:Koopman_DT}
        U g\left(\mathbf{z}\right) = g \circ \mathbf{T} \left(\mathbf{z}\right)
    \end{equation}

    At this point, it is worth noting that the discrete-time Koopman operator is also linear; i.e., $U\left(c_1 \mathbf{f}_1\left(\mathbf{z}\right) + c_2 \mathbf{f}_2\left(\mathbf{z}\right)\right) =
    c_1 \mathbf{f}_1\left(\mathbf{T}\left(\mathbf{z}\right)\right) + c_2 \mathbf{f}_2\left(\mathbf{T}\left(\mathbf{z}\right)\right) =
    c_1 U\mathbf{f}_1\left(\mathbf{z}\right) + c_2 U\mathbf{f}_2\left(\mathbf{z}\right)$. This linearity allows us to derive \cref{eq:DT_Koopman_Concat}, which implies that the dynamics of the observables of a system over a trajectory $S^k\left(\mathbf{z}_0\right)$ can be entirely described by the consecutive application of the discrete-time Koopman operator without having to \say{observe} the system twice for a single trajectory.

    \begin{equation} \label{eq:DT_Koopman_Concat}
        g \circ \mathbf{T}\left(\mathbf{T}\left(\mathbf{z}_0\right)\right) = U g\left(\mathbf{T}\left(\mathbf{z}_0\right)\right) = U g \circ \mathbf{T} \left(\mathbf{z}_0\right) = U^2 g\left(\mathbf{z}_0\right) , \mathbf{z}_0 \in M
    \end{equation}

\begin{comment}
    By taking advantage of this linearity, we can now make use of eigenfunction decomposition in order to obtain further information from the defined operator. We call $\phi : M \mapsto \mathbb{C}$ an eigenfunction of the Koopman operator $U$, associated to an eigenvalue $\lambda \in \mathbb{C}$.
\end{comment}

\section{The Koopman Operator for Data-driven Real System Identification}

    \subsection{Applied problem definition}

        After properly defining the abstract the Koopman operator, we now want to establish a practical framework in order to make use of its properties for System Identification and Control applications. We start by redefining $M$ as a real vector-space, and we follow the numerical scheme proposed by Mauroy-Goncalves\cite{Goncalves} as a data-driven Identification method for dynamical systems.

        We consider the Real Dynamical System described in \cref{eq:CT_Dyn_Real}, where $n$ is the dimension of the system.

        \begin{equation} \label{eq:CT_Dyn_Real}
            \dot{\mathbf{x}} = \mathbf{F}\left(\mathbf{x}\right), \mathbf{x} \in \mathbb{R}^n
        \end{equation}

        The vector field $\mathbf{F}$ is w.l.o.g. of the form described in \cref{eq:F_Decomp}, where the $k$ vectors $\mathbf{w}_i = \left( w_i^1 \cdots w_i^n \right)^T \in \mathbb{C}^n$ are unknown coefficients to be identified, whereas the $k$ library functions $h_i : \mathbb{R}^n \mapsto \mathbb{C}$ are assumed to be known; note that some coefficients might be zero.

        \begin{equation} \label{eq:F_Decomp}
            \mathbf{F}(\mathbf{x}) = \sum_{i = 1}^{N_F} \mathbf{w}_i h_i \left( \mathbf{x} \right)
        \end{equation}

        Our first goal is to identify the vector field F (i.e., the $N_F$ coefficient vectors $\mathbf{w}_i$) from snapshot measurements of the system trajectories. We consider $K$ snapshot pairs $\left(\mathbf{x}_k,\mathbf{y}_k\right)$ obtained from noisy measurements yielding \cref{eq:x_noisy,eq:y_noisy}, where $\epsilon$ is the state-dependent measurement noise, and \cref{eq:snap}, which is associated to the dynamical system described in \cref{eq:CT_Dyn}.

        \begin{align}
            \mathbf{x}_k &= \bar{\mathbf{x}}_k + \epsilon\left(\mathbf{x}_k\right) \label{eq:x_noisy} \\
            \mathbf{y}_k &= \bar{\mathbf{y}}_k + \epsilon\left(\mathbf{y}_k\right) \label{eq:y_noisy} \\
            \mathbf{y}_k &= \mathbf{S}^{T_s}\left(\mathbf{x}_k\right) \label{eq:snap}
        \end{align}

        We assume that the measurement noise is Gaussian and proportional to the state value; i.e., $\epsilon\left(\mathbf{x}\right) = \mathbf{x} \cdot \mathbf{\eta} , \eta \sim \mathcal{N}\left(0,\sigma_\text{meas}\right)$. We also assume all snapshot pairs lie in a compact set $X \subset \mathbb{R}^n$ and are obtained with the same sampling time $T_s$, but we are not compelled to extracting all samples from the same trajectory (i.e., associated to the same starting point $\mathbf{z}_0$).

    \subsection{Applied Koopman operator}

        As we stated in \cref{sec:Abstract}, we can alternatively describe the behavior of the system  through a collection of \textit{observables} $g : \mathbb{R}^n \mapsto \mathbb{C}$ in a lifted space instead of using the original system states. Provided that the functions are continuously differentiable, their dynamics in the lifted space are given by \cref{eq:obs_dyn}, where $\dot{g}$ denotes $\frac{\partial \left(f \circ \mathbf{S}^t\right)}{\partial t}$ and $\nabla$ denotes the gradient.

        \begin{equation} \label{eq:obs_dyn}
            \dot{g} = \left(\mathbf{F} \cdot \nabla\right) g , g \in \mathcal{G}
        \end{equation}

        In contrast to \cref{eq:CT_Dyn_Real}, the dynamics in \cref{eq:obs_dyn} are now infinite dimensional, but linear. Not only that, but we can also recover the semigroup-driven dynamics described in \cref{eq:Koopman_CT}. That means that while the original state-space flow induced by \cref{eq:CT_Dyn_Real} is given by the nonlinear flow map $S^t$, the flow induced by \cref{eq:obs_dyn} in the lifted space $\mathcal{G}$ is given by the linear semigroup of Koopman operators $U^t$ behaving just as in \cref{eq:obs_traj}.

        Under appropriate conditions (see \cite{Goncalves}), the semigroup of Koopman operators is strongly continuous and generated by the operator $L$ in \ref{eq:infinitesimal_gen}, which we will call the infinitesimal generator of the Koopman operator; we denote its domain by $\mathcal{D} \left(L\right)$.

        \begin{equation} \label{eq:infinitesimal_gen}
            L = \mathbf{F} \cdot \nabla \implies U^t = e^{Lt}
        \end{equation}

        We will later exploit this continuity to manipulate the inherent sample time of the desired discrete Koopman operator, since this versatility turns out to be really helpful when controlling a real system.

    \subsection{Linear identification in the Lifted Space}

        The semigroup of Koopman operators $U^t$ that describes the flow $S^t$ is unique for each system. It is by exploiting this equivalence relation that we can set ourselves to recover the system dynamics entirely from the Lifted Space. We divide the identification into three main steps:

        \begin{itemize}
            \item \textit{Data lifting:} the snapshot pairs $\left(\mathbf{x}_k,\mathbf{y}_k\right)$ described in \cref{eq:snap} are lifted to the space of observables by constructing new pairs of the form\\$\left(\mathbf{g}\left(\mathbf{x}_k\right),\mathbf{g}\left(\mathbf{y}_k\right)\right)$ for some $\mathbf{g} = \left\{g_j\right\}_{j=1}^{N} \subset \mathcal{G}$; each \textit{basis function} $g_j$ is assumed to be continuously differentiable. \Cref{eq:Koopman_snap} follows from \cref{eq:x_noisy,eq:y_noisy,eq:snap}.

            \begin{align} \label{eq:Koopman_snap}
                \mathbf{g} \left(\mathbf{y}_k\right) &= \mathbf{g} \left(S^{T_s} \left(\mathbf{x}_k - \epsilon\left(\mathbf{x}_k\right)\right) + \epsilon\left(\mathbf{y}_k\right)\right)
                \\
                & \approx U^{T_s} \mathbf{g}\left(\mathbf{x}_k\right) + \mathcal{O} \left(\left|\left|\epsilon\right|\right|\right)
            \end{align}

            \item \textit{Identification of the Koopman operator:} A truncated finite-dimensional projection of the Koopman operator is obtained through a classical linear identification method, similar to a component of the Extended DMD algorithm\cite{EDMD}, which in turn yields an approximation of the infinitesimal generator from \ref{eq:infinitesimal_gen}.

            \item \textit{Identification of the vector field:} Using \ref{eq:infinitesimal_gen}, we can recover the original system's vector field $F$.
        \end{itemize}

        The specifics of each step will be discussed in \cref{sec:algorithm}.

\section{Lifting Algorithm} \label{sec:algorithm}

    \subsection{First Step: Data lifting}

        At this point, a choice has to be made: so far, we have treated the library of observables as an infinite collection with all possible functions of the form $g:M \mapsto \mathbb{C}$; nevertheless, in practice such a collection is unfeasible, which is why we have to analyze the Koopman operator over a truncated realization of the theoretical functional space.

        % Add proper references to real world applications and other benchmarks
        The selection of the observable functions can be completely arbitrary (see \dots) or even physically-informed (see \dots). Nevertheless, for the sake of generality, we will present the algorithm using a sequence of monomials. These are not only congruent with the classical theory of Taylor expansion of arbitrary functions, but also facilitate scalability both in order and dimension while still providing a numerically trackable way of reaching a (theoretical) complete basis.

        Let $\mathcal{F}_N \subset \mathcal{F}$ be the linear subspace of observables generated by the basis $\left\{g_j\right\}_{j=1}^{N} = \left\{p_j\right\}_{j=1}^{N}$ of all polynomials of the form $p_j = x_1^{s_{j_1}} \cdots x_n^{s_{j_n}}$ with total degree lesser or equal than $m$, so that $N = \frac{(n+m)!}{n!m!}$.

        For each snapshot pair $\left(\mathbf{x}_k,\mathbf{y}_k\right) \in \mathbb{R}^{n \times 2} , k \in \left\{1,\dots,K\right\}$, we construct a new pair $\left(\mathbf{p}\left(\mathbf{x}_k\right),\mathbf{p}\left(\mathbf{y}_k\right)\right) \in \mathbb{R}^{N \times 2}$, where $\mathbf{p}\left(\mathbf{x}\right) = \left(p_1\left(\mathbf{x}\right) , \dots p_N\left(\mathbf{x}\right)\right)^T$ denotes the vector of basis monomials under the above constraints. From now on, we will also use the $\mathbb{R}^{K \times N}$ matrices in \cref{eq:snap_matrix}.

        \begin{equation} \label{eq:snap_matrix}
            \mathbf{P_x} =
            \begin{pmatrix}
                \mathbf{p}\left(\mathbf{x}_1\right)^T \\
                \vdots \\
                \mathbf{p}\left(\mathbf{x}_K\right)^T
            \end{pmatrix}
            \qquad
            \mathbf{P_y} =
            \begin{pmatrix}
                \mathbf{p}\left(\mathbf{y}_1\right)^T \\
                \vdots \\
                \mathbf{p}\left(\mathbf{y}_K\right)^T
            \end{pmatrix}
        \end{equation}

    \subsection{Second Step: Identification of the Koopman operator}

        Now, we proceed with the identification of the Koopman operator $U^t , t = T_s$; specifically, we will identify the finite-rank truncation $U_N: \mathcal{G}_N \mapsto \mathcal{G}_N$ of the form $U_N = P_N \left.U^{T_s}\right|_{\mathcal{G}_N}$, where $P_N : L\left(\mathcal{G}\right) \mapsto L\left(\mathcal{G}_N\right)$ is the projection operator onto the subspace $L\left(\mathcal{G}_N\right)$ of Linear Functionals and where $\left.U^{T_s}\right|_{\mathcal{G}_N} : \mathcal{G}_N \mapsto \mathcal{G}$ is the restriction of the Infinite-Dimensional Koopman operator to the truncated subspace of observables $\mathcal{G}_N$.

        Consider equation \cref{eq:trunc_dyn}, where we exploit the linearity of the Koopman operator in the vector space to describe the dynamics of the polynomial observables through a pair of coefficient vectors $\left(\mathbf{a},\mathbf{b}\right) \in \mathbb{R}^{N \times 2}$ (arbitrary as long as they comply with the aforementioned expression), we can define a matrix $\overline{\mathbf{U}}_N \in \mathbb{R}^{N \times N}$ that satisfies \cref{eq:UN_Matrix}.

        \begin{equation} \label{eq:trunc_dyn}
            g = \mathbf{a}^T \mathbf{p} , \quad U_N g = \mathbf{b}^T \mathbf{p}
        \end{equation}

        \begin{equation} \label{eq:UN_Matrix}
            \overline{\mathbf{U}}_N \mathbf{a} = \mathbf{b}
        \end{equation}

        The matrix $\overline{\mathbf{U}}_N$ is a representation of the projected Koopman operator $U_N$, providing an approximate finite-dimensional linear description of the nonlinear system in the truncated space of observables. As previously mentioned, this approximation does not depend on local linearization techniques, which makes it valid globally (granted sufficient trajectories covering all present dynamics were fed to the algorithm).

        \Cref{eq:Koopman_to_Matrix_1} follows from \cref{eq:trunc_dyn,eq:UN_Matrix} and, since this holds for all coefficient vectors $\mathbf{a}$, we get \cref{eq:Koopman_to_Matrix_2}, where the operator $U_N$ acts on each component of the polynomial vector $\mathbf{p}$. This implies that the $j$-th column of the resulting matrix is associated to the projection onto $\mathcal{F}_N$ of the image of the basis function $p_j$ through the finite-rank Koopman operator $U_N^{T_s}$.

        \begin{equation} \label{eq:Koopman_to_Matrix_1}
            U_N g = U_N \left(\mathbf{a}^T \mathbf{p}\right) = \left(\overline{\mathbf{U}}_N \mathbf{a}\right)^T \mathbf{p} = \mathbf{p}^T \overline{\mathbf{U}}_N \mathbf{a}
        \end{equation}

        \begin{equation} \label{eq:Koopman_to_Matrix_2}
            U_N \mathbf{p}^T = \mathbf{p}^T \overline{\mathbf{U}}_N
        \end{equation}

        We now get the least-squares fit for the orthogonal projection of an observable by solving the minimization problem in \cref{eq:PN_Min_Prob}, which yields the expression presented in \cref{eq:PN_Min_Sol} where $\mathbf{P}^\dagger$ denotes the Moore-Penrose pseudo-inverse of $\mathbf{P}$.

        \begin{equation} \label{eq:PN_Min_Prob}
            P_N g = \text{arg min}_{\tilde{g} \in \text{span}\left\{p_1 , \dots , p_N\right\}} \sum_{k=1}^K \left|\tilde{g}\left(\mathbf{x}_k\right) - g\left(\mathbf{x}_k\right)\right|^2
        \end{equation}

        \begin{equation} \label{eq:PN_Min_Sol}
            P_N g = \mathbf{p}^T \mathbf{P_x}^\dagger
                \begin{pmatrix}
                    g\left(\mathbf{x}_1\right) \\
                    \vdots \\
                    g\left(\mathbf{x}_K\right)
                \end{pmatrix}
        \end{equation}

        % Replace Goncalves cite with a more detailed demonstration of the measurement noise effect
        The above means that we can get the least-squares coefficients associated to the projection of the $j$-th polynomial observable by setting $g=U^{T_s} p_j$ and using \cref{eq:PN_Min_j}, under the assumption that the measurement noise $\left|\left|\epsilon\right|\right|$ is small enough \cite{Goncalves}.

        \begin{equation} \label{eq:PN_Min_j}
            P_N \left(U^{T_s} p_j\right) = \mathbf{p}^T \mathbf{P_x}^\dagger
                \begin{pmatrix}
                    U^{T_s} p_j\left(\mathbf{x}_1\right) \\
                    \vdots \\
                    U^{T_s} p_j\left(\mathbf{x}_K\right)
                \end{pmatrix}
            \approx \mathbf{p}^T \mathbf{P_x}^\dagger
                \begin{pmatrix}
                    p_j\left(\mathbf{y}_1\right) \\
                    \vdots \\
                    p_j\left(\mathbf{y}_K\right)
                \end{pmatrix}
        \end{equation}

        By concatenating the above expression for all polynomial observables we get \cref{eq:UN_Final}, which corresponds to the least-squares approximation of the matrix representation of the truncated Koopman operator indexed by the sampling time $T_s$.

        \begin{equation} \label{eq:UN_Final}
            \overline{\mathbf{U}}_N \approx \mathbf{P}_x^\dagger \mathbf{P}_y
        \end{equation}

    \bibliography{main.bib}

\end{document}


\subsubsection{Quadratic Factors}
\noindent
If a quadratic doesn't have real roots, then we have a quadratic factor. Here, we'll assume that the quadratic factor isn't repeated.
So, $Q(x) = R(x)(ax^2+bx+c)$, $b^2-4ac < 0$, and $R(x)$ is not evenly divisible by $ax^2+bx+c$.
In this case, we say
\begin{equation*}
	\frac{P(x)}{R(x)(ax^2+bx+c)} = \left(\text{Decomposition of }R(x)\right)+\frac{A_1x+B_1}{ax^2+bx+c}.
\end{equation*}
We then solve for the constants in the numerator, possibly having to solve a system of equations or using previous results and less convenient values for $x$.

\begin{example}
	Find the partial fraction decomposition of the following expression:
	\begin{equation*}
		\frac{6x^2+21x+11}{x^3+5x^2+3x+15}.
	\end{equation*}
\end{example}
\noindent
Factoring,
\begin{equation*}
	x^2+5x^2+3x+15 = (x+5)(x^2+3).
\end{equation*}
So,
\begin{equation*}
	\frac{6x^2+21x+11}{x^3+5x^2+3x+15} = \frac{A_1}{x+5}+\frac{A_2x+B_2}{x^2+3}.
\end{equation*}
Multiplying each side by the denominator,
\begin{equation*}
	6x^2+21x+11 = A_1(x^2+3)+(A_2x+B_2)(x+5).
\end{equation*}
At $x=-5$,
\begin{equation*}
	56 = 28A_1 \implies A_1 = 2.
\end{equation*}
Now we'll use the previous result and another value for $x$. We can use $x=0$ to not have to worry about the $A_2$ term.
At $x=0$,
\begin{equation*}
	11 = 2(3) + (B_2)(5) \implies B_2 = 1.
\end{equation*}
Now we'll use the previous 2 results to find $A_2$. $x=1$ is a good choice to keep the numbers small.
At $x=1$,
\begin{equation*}
	38 = 2(1+3)+(A_2+1)(6) \implies A_2 = 4.
\end{equation*}
So,
\begin{equation*}
	\frac{6x^2+21x+11}{x^3+5x^2+3x+15} = \frac{2}{x+5}+\frac{4x+1}{x^2+3}.
\end{equation*}


\lab{Algorithms}{Canonical Transformations and the QR Decomposition}{Canonical Transformations and the QR Decomposition}
\label{Ch:Canonical Transformations}

\objective{Use orthogonal transformations to perform QR decomposition.}

\section*{Orthogonal transformations}
Recall that a matrix $Q$ is \emph{unitary} if $Q^* Q = I$ or for real matrices, $Q^T Q = I$ (since the conjugate of a real number is itself). We like unitary transformations because they're very numerically stable. The number $\kappa(A) = \norm{A} \norm{A^{-1}}$ is called the \emph{condition number} of $A$. We'll discuss condition number more in Lab \ref{Ch:Norms and Geometry}; for now, all you need to know is that if $\kappa(A)$ is small, then problems involving $A$ are less susceptible to numerical errors. For induced matrix norms (which include most of the matrix norms we would ever care about),  it holds that $\norm{Q}=1$ when $Q$ is unitary. The inequality $\norm{AB} \leq \norm{A} \norm{B}$ also holds for these norms. It follows that $\kappa(A) = \norm{A} \norm{A^{-1}} \geq \norm{A A^{-1}} = \norm{I} = 1$. Note that if $Q$ is unitary, $Q^{-1} = Q^*$ and $Q^*$ is also unitary, so $\kappa(Q) = \norm{Q} \norm{Q^*} = 1$. This means that orthogonal matrices have the smallest possible condition number, which is great!

Any unitary matrix $Q$ can be described as a reflection, a rotation, or some combination of the two. If $det(Q) = 1$, then $Q$ is a rotation; if $det(Q) = -1$, then $Q$  is the composition of a reflection and a rotation.  Let's explore these two types of unitary transformations and some of their applications. We will focus on the real case to simplify matters.

\section*{Householder reflections}
A Householder reflection is a linear transformation $P: \mathbb{R}^n \rightarrow \mathbb{R}^n$ that reflects a vector $x$ about a hyperplane. See figure \ref{fig:Householder reflector}. Recall that a hyperplane can be defined by a unit vector $v$ which is orthogonal to the hyperplane. As shown in the figure, $x - \langle v,x \rangle v$ is the projection of $x$ onto the hyperplane defined by $v$. (You should verify this geometrically.) However, to reflect \emph{across} the hyperplane, we must move twice as far; that is, $Px = x - 2\langle v,x \rangle v$. This can be written $Px = x - 2v(v^\ast x)$, so $P$ has matrix representation $P = I - 2vv^\ast$. Note that $P^\ast P = I$; thus $P$ is orthogonal.

\begin{figure}
	\centering
	\includegraphics[width= \textwidth]{fig1}
	\caption{Householder reflector}
	\label{fig:Householder reflector}
\end{figure}

\subsection*{Householder triangularization}
Consider the problem of computing the $QR$ decomposition of a matrix $A$. You've already learned the Gram-Schmidt and the Modified Gram-Schmidt algorithms for this problem. The $QR$ decomposition can also be computed using Householder triangularization. Gram-Schmidt and Modified Gram-Schmidt \emph{orthogonalize} $A$ by a series of \emph{triangular} transformations. Conversely, the Householder method \emph{triangularizes} $A$ by a series of \emph{orthogonal} transformations.

Let's demonstrate this method on a $4 \times 3$ matrix $A$. First we find a orthogonal transformation $Q_1$ that maps the first column of A into the range of $e_1$.

\def\mc#1{\multicolumn{1}{c|}{#1}}
\begin{equation*}
\begin{pmatrix}
\ast & \ast & \ast \\
\ast & \ast & \ast \\
\ast & \ast & \ast \\
\ast & \ast & \ast
\end{pmatrix}
\underrightarrow{Q_1}
\begin{pmatrix}

\ast & \ast & \ast & \\ \cline{2-3}
\mc{0} & \ast & \mc{\ast}& \\
\mc{0} & \ast & \mc{\ast} & \\
\mc{0}& \ast & \mc{\ast} & \\ \cline{2-3}
\end{pmatrix}
\end{equation*}
Let $A_2$ be the boxed submatrix of $A$. Now find an orthogonal transformation $Q_2$ that maps the first column of $A_2$ into the range of $e_2$.

\begin{equation*}
\begin{pmatrix}
\ast & \ast \\
\ast & \ast \\
\ast & \ast
\end{pmatrix}
\underrightarrow{Q_2}
\begin{pmatrix}
\ast & \ast \\
0 & \ast \\
0 & \ast
\end{pmatrix}
\end{equation*}
Similarly, $ \begin{pmatrix} \ast \\ \ast \end{pmatrix} \underrightarrow{Q_3} \begin{pmatrix} \ast \\ 0 \end{pmatrix} $. (Technically $Q_2$ and $Q_3$ act on the whole matrix and not just on the submatrices, so that $Q_i: \mathbb{R}^n \rightarrow \mathbb{R}^n$ for all $i$. $Q_2$ leaves the first row alone, and $Q_3$ leaves the first two rows alone.) Then $Q_3 Q_2 Q_1 A =$

\begin{equation*}
Q_3 Q_2 Q_1
\begin{pmatrix}
\ast & \ast & \ast \\
\ast & \ast & \ast \\
\ast & \ast & \ast \\
\ast & \ast & \ast \\
\end{pmatrix}
= Q_3 Q_2
\begin{pmatrix}
\ast & \ast & \ast \\
0 & \ast & \ast \\
0 & \ast & \ast \\
0 & \ast & \ast \\
\end{pmatrix}
= Q_3
\begin{pmatrix}
\ast & \ast & \ast \\
0 & \ast & \ast \\
0 & 0 & \ast \\
0 & 0 & \ast \\
\end{pmatrix}
=
\begin{pmatrix}
\ast & \ast & \ast \\
0 & \ast & \ast \\
0 & 0 & \ast \\
0 & 0 & 0 \\
\end{pmatrix}
\end{equation*}

We've accomplished our goal, which was to triangularize $A$ using orthogonal transformations. But now, how do we find the $Q_i$ that do what we want? Using Householder reflections. (Surprise!)

For example, to find $Q_1$, we choose the right hyperplane to reflect $x$ into the range of $e_1$. It turns out there are two hyperplanes that will work, as shown in figure \ref{fig:two reflectors}. (In the complex case, there are infinitely many such hyperplanes.) Between the two, the one that reflects $x$ further will be more numerically stable. This is the hyperplane perpendicular to \textbf{$v = sign(x_1)\norm{x}e_1 + x$}. The whole process is summarized in Algorithm \ref{Alg:Householder triangularization}.

\begin{figure}
	\centering
	\includegraphics[width= \textwidth]{fig2}
	\caption{two reflectors}
	\label{fig:two reflectors}
\end{figure}

\begin{pseudo}{Householder triangularization}{A}
\label{Alg:Householder triangularization}
m,n \GETS size(A)\\
\FOR k \GETS 1 \TO n-1 \DO
\BEGIN
   x = A_{k:m,k}\\
   v_k = sign(x_1)\norm{x}_2 e_1 + x\\
   v_k = v_k / \norm{v_k}_2\\
   P_k = eye(m,m)\\
   P_k[k:m,k:m] = P_k[k:m,k:m] - 2 v v^T\\
   A = sp.dot(P_k,A);
\END
\end{pseudo}

This algorithm returns upper triangular $R$. You can find $Q$ s.t. $QR = A$ by multiplying the $P_k$ together appropriately.

\begin{problem}
\label{prob:HouseholderQR}
Write a script using Householder reflections to find the QR decomposition of a matrix A.
\end{problem}

\subsection*{Stability of the Householder QR algorithm}

Try the following in Python.

\begin{lstlisting}
In [1]:  import scipy as sp
In [2]:  import numpy.linalg as la
In [3]:  import my_householder
In [4]:  Q,X = la.qr(sp.rand(50,50)) #create a random orthogonal matrix:
In [5]:  R = sp.triu(sp.rand(50,50)) # create a random upper triangular matrix
In [6]:   A = sp.dot(Q,R) #Q and R are the exact QR decomposition of A
# use your Householder QR script to estimate Q and R:
In [7]:   Q1,R1 = my_householder.qr(A)
#now check the relative errors of Q1 and R1
In [8]:  la.norm(Q1-Q)/la.norm(Q)
Out [8]:  0.282842955725
In [9]:  la.norm(R1-R)/la.norm(R)
Out[9]:  0.0428922016647
\end{lstlisting}
This is terrible! Python works in $16$ decimal points of precision. But $Q_1$ and $R_1$ are only accurate to $0$ and $1$ decimal points, respectively. We've lost $16$ decimal points of precision!

Don't lose hope. Check how close the product $Q_1 R_1$ is to $A$.
\begin{lstlisting}
In [10]:  A1 = sp.dot(Q1,R1)
In [11]:  la.norm(A1-A)/la.norm(A)
Out[11]:  9.73996046986e-16
\end{lstlisting}
We've now recovered $15$ digits of accuracy. The errors in $Q_1$ and $R_1$ were somehow ``correlated," so that they canceled out in the product. The errors in $Q_1$ and $R_1$ are called \emph{forward errors}. The error in $A_1$ is the \emph{backward error}. The Householder $QR$ algorithm is a backward stable algorithm.

Householder QR factorization is more numerically stable than Gram-Schmidt or even Modified Gram-Schmidt (MGS). However, MGS is still useful for some types of iterative methods, because it finds the orthogonal basis one vector at a time instead of all at once (for example see Lab \ref{Ch:EigSolve}).

\subsection*{Upper Hessenberg Form}

%I'm not sure about this math. Is this true in the real case?
An upper Hessenberg matrix is a square matrix with zeros below the first subdiagonal. Every  $n \times n$ matrix $A$ can be written $A = Q^THQ$ where $Q$ is orthogonal and $H$ is an upper Hessenberg matrix, called the Hessenberg form of $A$. Note the similarity of this decomposition to the Schur decomposition in Lab \ref{Ch:Jordan}.

The Hessenberg decomposition can be computed using Householder reflections, in a process very similar to Householder triangularization. Let's demonstrate this process on a $5 \times 5$ matrix $A$. Note that $A=Q^THQ$ is equivalent to $QAQ^T = H$; thus our strategy is to multiply $A$ on the right and left by a series of orthogonal matrices until it is in Hessenberg form. If we try the same $Q_1$ as in the first step of the Householder algorithm, then with $Q_1 A$ we introduce zeros in the first column of $A$. However, since we now have to multiply $Q_1 A$ on the left by $Q_1^T$, all those zeros are destroyed, as demonstrated below. (Although this process may seem futile now, it actually does tend to decrease the size of the subdiagonal entries. If we repeat it over and over again, the subdiagonal entries will often converge to zero. That's the idea behind the $QR$ algorithm in Lab \ref{Ch:EigSolve}.)
\[
\begin{array}{ccccc}
\begin{pmatrix}
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
\end{pmatrix}
&\underrightarrow{Q_1 \cdot }&
\begin{pmatrix}
* & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
\end{pmatrix}
&\underrightarrow{\cdot Q_1^T }&
\begin{pmatrix}
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
\end{pmatrix}
\\
A & & Q_1A & & Q_1 A Q_1^T
  \end{array}
\]
Instead, let's try starting with a different $Q_1$ that leaves the \emph{first} row alone and reflects the \emph{rest} of the rows into the range of $e_2$. This means that $Q_1^T$ leaves the first column alone.
\[
\begin{array}{ccccc}
\begin{pmatrix}
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
* & * & * & * & *\\
\end{pmatrix}
&\underrightarrow{Q_1 \cdot }&
\begin{pmatrix}
* & * & * & * & *\\
* & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
\end{pmatrix}
&\underrightarrow{\cdot Q_1^T }&
\begin{pmatrix}
* & * & * & * & *\\
* & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
0 & * & * & * & *\\
\end{pmatrix}
\\
A & & Q_1A & & Q_1 A Q_1^T
  \end{array}
\]
We now iterate through the matrix until we obtain
\begin{equation*}
Q_3 Q_2 Q_1 A Q_1^T Q_2 ^T Q_3^T =
\begin{pmatrix}
* & * & * & * & *\\
* & * & * & * & *\\
0 & * & * & * & *\\
0 & 0 & * & * & *\\
0 & 0 & 0 & * & *\\
\end{pmatrix}
\end{equation*}

\begin{problem}
Write a script that transfers an input matrix to upper Hessenberg form. (Hint: You only need to modify your code code from problem \ref{prob:HouseholderQR} slightly.) We will use this technique in the eigenvalue lab later.
\end{problem}

\section*{Givens rotations}

The matrix $\begin{pmatrix} cos(\theta) & -sin(\theta) \\ sin(\theta) & cos(\theta) \end{pmatrix}$ rotates a vector counterclockwise by $\theta$. Given a vector $x = \begin{pmatrix} a \\ b \end{pmatrix}$, we can rotate $x$ into the range of $e_1$ by choosing the correct $\theta$. The problem is equivalent to solving for $c = cos(\theta)$ and $s = sin(\theta)$ in the system
\begin{equation}
\label{eq:Givens rotation system}
\begin{pmatrix} c & -s \\ s & c \end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix}
= \begin{pmatrix}r\\0\end{pmatrix}
\end{equation}
 In fact, it's not necessary to compute $\theta$; we solve for $c$ and $s$ directly. An obvious solution is $c = \frac{a}{\sqrt{a^2 + b^2}}$ and $s = \frac{b}{\sqrt{a^2+b^2}}$.

\subsection*{Givens triangularization}

Like Householder, the Givens $QR$ algorithm, \emph{triangularizes} a matrix $A$ using \emph{orthogonal transformations}. The Householder $QR$ algorithm worked one column at a time; Givens works one element at a time. Each nonzero below the main diagonal can be zeroed out with a Givens rotation. If we solve for $c$ and $s$ as above, the matrix
\begin{equation*}
G(i,j,c,s) =
\begin{pmatrix}
1   & \cdots & 0 & \cdots & 0 & \cdots & 0 \\
 \vdots & \ddots & \vdots &  & \vdots & & \vdots \\
0   & \cdots &    c   & \cdots &    -s   & \cdots &    0   \\
 \vdots &        & \vdots & \ddots & \vdots &        & \vdots \\
 0   & \cdots &   s   & \cdots &    c   & \cdots &    0   \\
\vdots &        & \vdots &        & \vdots & \ddots & \vdots \\
0   & \cdots &    0   & \cdots &    0   & \cdots &    1
\end{pmatrix}
\end{equation*}
where
\[ \begin{array}{ll}
g_{i\, i} = c  & g_{j\, i}= -s   \\
g_{j\, j} = c &  g_{i\, j}= s  \\
\end{array}\]
rotates by $\theta$ in the $i,j$ plane. Also, it's easy to check that $G(i,j,c,s)$ is orthogonal. Left-multiplying $A$ by $G(i,j,c,s)$ zeroes out $A_{i\, j}$. For example,

\[
\begin{array}{ccccccc}
\begin{pmatrix}
*&*&*\\
*&*&*\\
*&*&*
\end{pmatrix}
&
\underrightarrow{G(3,1)}
&\begin{pmatrix}
*&*&*\\
*&*&*\\
0&*&*
\end{pmatrix}
&
\underrightarrow{G(2,1)}
&\begin{pmatrix}
*&*&*\\
0&*&*\\
0&*&*
\end{pmatrix}
&
\underrightarrow{G(3,2)}
&\begin{pmatrix}
*&*&*\\
0&*&*\\
0&0&*
\end{pmatrix}
\end{array}
\]

The Givens QR method is often slower than Householder, because it works one element at a time. But it is faster than Householder when $A$ is sparse, and it is very parallelizable.

\begin{problem}
Write a script that uses Givens rotations to do $QR$ decomposition. By performing successive Givens rotations, triangularize $A$ to find $R$. You can find $Q$ using the chain of rotations.

To find each rotation, you will have to solve \eqref{eq:Givens rotation system} for $c$ and $s$. Let $b$ be the element you want to zero out, say $A_{i\,j}$. Then $a$ is in the same \emph{column} as $b$, and in the \emph{row} you want to rotate into (it's like we're squishing the whole vector into $a$'s spot.) You can choose $a$ to be the diagonal element above $b$.
\end{problem}

\begin{problem}
Compare the MGS, Householder, and Givens algorithms for $QR$ decomposition on various matrices. Try different sizes and different levels of sparsity. Which is the fastest? Which is the most stable?
\end{problem}

%Sources: http://www.cs.unc.edu/~krishnas/eigen/node5.html
% http://en.wikipedia.org/wiki/Givens_rotation
%http://en.wikipedia.org/wiki/QR_decomposition
%	Note the Operation count: Householder is 2/3 n^3, MGS is 2 n^3
%http://en.wikipedia.org/wiki/QR_algorithm
%Applied Numerical methods using MATLAB by Yang has some code written for this
%http://www.math.kent.edu/~reichel/courses/intr.num.comp.2/lecture21/evmeth.pdf
%	These are eigenvalue algorithms explained carefully
%http://en.wikipedia.org/wiki/Householder_transformation
%Numerical Linear Algebra, by Lloyd N. Trefethen and David Bau III, Chapters 10 and 16


\section{Simplifying expression for \texorpdfstring{$w_p(\phi)$}{}}

The sine function is a phase shift of the cosine function, so
\begin{equation}
    \sin(2k)=\sin(2\phi+\pi/2)=\cos(2\phi).
\end{equation}
It is also given that
\begin{equation}
    2\sin^2(2k)=1-\cos(2k).
\end{equation}
Because $\cos(2k)=-\sin(2\phi)$ by a similar argument,
\begin{equation}
    2\sin^2(2k)=1+\sin(2\phi).
\end{equation}
Finally
\begin{align}
    w_p(\phi)&=\frac{1}{\underbrace{\sin(2\phi+\pi/2)}_{\cos(2\phi)}}[1\underbrace{-\frac{a_Tr_0^2}{\mu}}_{-\frac{4}{\beta^2(3\pi+8)}}(3\phi+2)]+\frac{2}{\beta^2(3\pi+8)}\underbrace{(2\sin^2(\phi+\pi/4)+3)}_{\sin(2\phi)+4}\\
    &=\frac{1}{\cos(2\phi)}\left[1-\frac{4}{\beta^2(3\pi+8)}(3\phi+2)\right]+\frac{2}{\beta^2(3\pi+8)}(\sin(2\phi)+4)
\end{align}

\section{Numerical evaluation of \texorpdfstring{$y(\phi_0)=0$}{}}

\textbf{\textit{Notation:} Let $\phi_0$ satisfy $y(\phi_0)=0,w_p(\phi_0)=1$ and $\phi_\mathrm{min}$ minimize $w_p(\phi)$.}

To evaluate $y_p(\phi)=0$, we first define \verb|phi = (1:1:1000)*pi/4/1000|.
Using this vector to formulate \verb|y_p(Beta)|, where \verb|Beta| is a variable constant, we have a vector of length 1000 representing $y_p(\phi)$.
Note that

\begin{equation}
    y_p(\phi)=\frac{\sqrt{\varphi(\phi)}}{\beta^2(3\pi+8)\sin(2\phi+pi/2)}\sqrt{\frac{\mu}{r_0}}\cot(\phi+\pi/4)
\end{equation}

$\varphi(\phi)$ was defined in Eq. 18-20 of the presentation, and the above expression comes from Eq. 21 of the presentation. This implies that $\sin(2\phi+\pi/2)\neq 0$, and further that $y_p$ is only defined for $\varphi(\phi)\geq 0$. Thus, the vector \verb|y_p(Beta)| must be shortened to reflect this and exclude complex numbers.
From calculations, we find that the real representation of $y_p$ is \verb|y_p(Beta)(1:548)| using $a_T=0.2,r_0=1,\mu=1$ with $\beta\approx 1.07$.
To find the section of this vector that is $\in \mathbb{R}$, we have called a function \verb|realBreakpoint(vector)| in Figure \ref{loop1}.
Sorting this vector and determining the corresponding $\phi_0$ solves the problem, where we expect \verb|y_minValue = 0| and \verb|phi_0| to be the corresponding value of $\phi$:

\begin{verbatim}
    [y_values index_vector] = sort(y_p(Beta)(1:548));
    y_minValue = y_values(1);
    phi_0 = index_vector(1);
\end{verbatim}

This approach can be extended for a changing parameter $\beta$.
Since $\delta=\frac{1}{\beta^2}\in (0.001,1)$, $\beta=\sqrt{\frac{1}{\delta}}\in (1,\sqrt{1000})$.
In Octave/MATLAB, we can implement it as

\begin{verbatim}
    delta = (1:1:1000)/1000;
    Beta_vec = sqrt(1./delta);
\end{verbatim}

Mathematically, we say that
\begin{equation}
\vec\beta=\begin{bmatrix}\beta_1\\\vdots\\\beta_n\end{bmatrix}\text{ and }\vec{\delta}=\begin{bmatrix}1/\beta_1^2\\\vdots\\1/\beta_n^2\end{bmatrix}
\label{paravectors}
\end{equation}
where we choose $n=1000$.
Thus, by iterating through the values of \verb|Beta_vec|, we can generate a corresponding vector \verb|phi_m1| to represent how $\phi_0$ changes with respect to $\delta$:

\begin{figure}[H]
    \begin{verbatim}
        phi_m1 = zeros(1,n);
        w_1 = zeros(1,n);
        w_min = zeros(1,n);
        y_0 = zeros(1,n);

        for i = 1:n
            y_real = y_p(Beta_vec(i))(1:realBreakpoint(y_p(Beta_vec(i))));
            [yvals idx] = sort(y_real);
            if (length(idx) > 1)
                w_1(i) = w_p(Beta_vec(i))(idx(2));
                phi_m1(i) = phi(idx(2));
                y_0(i) = yvals(2);
            endif
        end
    \end{verbatim}
    \caption{Numerically finding $\vec\phi_0$ over $\vec{\delta}$}\label{loop1}
\end{figure}

In Figure \ref{loop1} above, the vector \verb|y_0| is updated with the value $y_p(\phi_{0,i})$ in each iteration
for each value of $\beta$, and is expected to be 0. The vector \verb|w_1| is updated with $w_p(\phi_{m,i})$, which is expected to be unity. As was done with \verb|y_p(Beta)|,
we have defined a vector \verb|w_p(Beta)| to represent $w_p(\phi)$ where \verb|Beta| is a constant that can be varied.
Thus, we should expect $w_p(\vec\phi_\mathrm{min})=\vec{1}$, $y_p(\vec\phi_0)=\vec{0}\in \mathbb{R}^n$, and the existence of $\vec\phi_m=\begin{bmatrix}\phi_{m,1}\\\vdots\\\phi_{m,n}\end{bmatrix}\in \mathbb{R}^n$ at the end of the loop.
The second index of \verb|y_p(Beta_vec(i))| is accessed to find $\phi_{0,i}$ within the loop because $\phi=0$ always satisfies $y_p(\phi)=0$, and we want to find the second such value.
The conditional check is to account for cases where the domain of $y_p(\phi)\in \mathbb{R}$ is very small (i.e. \verb|length(y_p(Beta_vec(i))| is 1), so we are only able to find $\phi_{0,i}=0\implies y(\phi_0)=0$.

The results of $\phi_0$ vs. $\delta$ are expressed in Figure \ref{yminplot} below. $y_p(\phi_0)\approx 0$ and $w_p(\phi_\mathrm{min})\approx 1$ within numerical error, which verifies the validity of these results.

\begin{figure}[H]
    \centering
    \includegraphics{plots/phi_delta_y.pdf}
    \caption{A plot of these results}\label{yminplot}
\end{figure}

\section{Numerical minimization of \texorpdfstring{$w_p(\phi)$}{}}

We use a similar approach as before to minimize $w_p(\phi)$ numerically.
In Figure \ref{loop1}, \verb|w_1(i)| was updated
to reflect the value of $w_p(\phi_{\mathrm{min},i})$ where $\phi=\phi_{\mathrm{min},i}$ minimized $y_p(\phi)$ for corresponding values of $\beta_i,\delta_i$
from Eq. \ref{paravectors}. The following code defines a vector \verb|phi_m2| of length $n=1000$ and populates it with the value of $\phi_{\mathrm{min},i}$ that minimizes $w_p(\phi)$ for $\beta_i$.
The corresponding minimum values are stored in another vector \verb|w_min|. Note that the first index of the sorted vector \verb|w_vals| is accessed, since we are looking for the absolute minimum.

\begin{figure}[H]
    \begin{verbatim}
        phi_m2 = zeros(1,n);
        w_min = zeros(1,n);
        for i = 1:n
            [w_vals idx] = sort(w_p(Beta_vec(i)));
            phi_m2(i) = phi(idx(1));
            w_min(i) = w_vals(1);
        end
    \end{verbatim}
    \caption{Iteratively finding $\vec\phi_\mathrm{min}$ for various $\beta$}
\end{figure}

\begin{figure}[H]
    \includegraphics{plots/phi_delta_min.pdf}
    \caption{Values of $\phi_\mathrm{min}$ that minimize $w_p(\phi)$ for various $\beta$}\label{wminplot}
\end{figure}

Figure \ref{wminplot}, showing $\phi_\mathrm{min}$ (representing \verb|phi_m2|) vs. $\delta$, summarizes these results.


\section*{Matrix Reconstruction}
\subsection*{Alternating Least Squares}
Beyond SVD: unobserved entries!
$f(\mathbf{U},v_i)=\sum_{(i,j)\in I} (a_{i,j} - \langle \mathbf{u}_j, \mathbf{v}_i \rangle)^2$,
Fix one, alternate other: \\
$\mathbf{U} \gets \argmin_\mathbf{U} f(\mathbf{U},\mathbf{V})$, $\mathbf{V} \gets \argmin_\mathbf{V} f(\mathbf{U},\mathbf{V})$ \\
Can decompose (solve independently) \\
$f(\mathbf{U},v_i)=\sum_i \left[ \sum_{(i,j)\in I} (a_{i,j} - \langle \mathbf{u}_j, \mathbf{v}_i \rangle)^2 \right]$ \\
Can add regularization $\mu (||U||_F^2 + ||V||_F^2 )\quad \mu >0$ \\
Upd: $u_i = \left( \sum_{(i,j)\in \mathcal{I}} v_jv_j^\top + I_k\lambda\right)^{-1} \left(\sum_{(i,j)\in \mathcal{I}} a_{ij}v_j \right)$

\subsection*{SVD Thresholding}
$\mathbf{B}^{*}=\mathit{shrink}_\tau(\mathbf{A}):=\argmin_{\mathbf{B}}{\{\|\mathbf{A-B}\|^2_F + \tau\|\mathbf{B}\|_{*}\}}$\\
then with SVD holds $\mathbf{B^*=UD_\tau V^T, D_\tau} =$ \\
$\mathit{diag}(\max\{0,\sigma_i - \tau\}),\Pi(\mathbf{X}) = x_{ij} \text{ if } (i,j) \in \mathcal{I} \text{ el. } 0$ \\
Iteration: $\mathbf{B}_{t+1}=\mathbf{B}_t + \eta_t \Pi(\mathbf{A} - \mathit{shrink}_\tau(\mathbf{B}_t))$


\section{More on Bilinear Forms}
Guess what?
We are back to bilinear forms again!
But this time, we are interested in bilinear forms $\phi:V\times V\to F$ for a finite dimensional vector space $V$ over $F$.
For a basis $B$ of $V$, we write $[\phi]_B=[\phi]_{B,B}$.
\begin{lemma}
    Let $B,B'$ be bases of $V$ and $P=[\operatorname{id}]_{B',B}$, then $[\phi]_{B'}=P^\top[\phi]_BP$.
\end{lemma}
\begin{proof}
    Proposition \ref{bilinear_change_of_basis}.
\end{proof}
\begin{definition}
    Matrices $A,B\in M_n(F)$ are congruent if there is invertible $P$ such that $A=P^\top BP$.
\end{definition}
\begin{remark}
    Easily congruence is an equivalence relation.
\end{remark}
\begin{definition}
    A bilineart form $\phi$ in $V$ is symmetric if $\phi(u,v)=\phi(v,u)$ for any $u,v\in V$.
\end{definition}
\begin{remark}
    $\phi$ is symmetric iff $[\phi]_B$ is symmetric in some basis $B$.\\
    If we want to diagonalise $\phi$, then it has to be symmetric by the preceding lemma.
\end{remark}
\begin{definition}
    A map $Q:V\to F$ is a quadratic form iff there exists a bilinear form $\phi:V\times V\to F$ such that $Q(u)=\phi(u,u)$.
\end{definition}
\begin{remark}
    If $B=\{e_i\}$ and $A=[\phi]_B=(\phi(e_i,e_j))_{i,j}$, then for $u=\sum_ix_ie_i$ we have
    $$Q(u)=\phi\left( \sum_{i=1}^nx_ie_i,\sum_{i=1}^nx_ie_i \right)=\sum_{i=1}^n\sum_{j=1}^nx_ix_j\phi(e_i,e_j)=x^\top Ax$$
    where $x=(x_1,\ldots,x_n)^\top$.
    Also observe that $x^\top Ax=x^\top Sx$ where $S=(A+A^\top)/2$ is symmetric.
\end{remark}
\begin{proposition}
    If $Q:V\times V\to F$ is a quadratic form, then there exists a unique symmetric bilinear form $\phi:V\times V\to F$ such that $Q(u)=\phi(u,u)$ for all $u\in V$.
\end{proposition}
\begin{proof}
    We know that $Q(u)=\psi(u,u)$ for some bilinear $\psi$.
    Take $\phi=(\psi+\psi^\top)/2$ where $\psi^\top(u,v)=\psi(v,u)$ works.
    To see it is unique, just observe that for a symmetric $\phi$,
    $$Q(u+v)=\phi(u+v,u+v)=\phi(u,u)+2\phi(u,v)+\phi(v,v)$$
    which implies that necessarily $\phi(u,v)=(Q(u+v)-Q(u)-Q(v))/2$ (known as the polarisation identity), so in particular $\phi$ is uniquely determined.
\end{proof}
\begin{theorem}[Diagonalisation of Bilinear Forms]\label{bilinear_diag}
    Let $\phi:V\times V\to F$ be a symmetric bilinear form and $\dim V<\infty$, then there exists a basis of $V$ such that $[\phi]_B$ is diagonal.
\end{theorem}
\begin{proof}
    We proceed by induction on $n=\dim V$.
    If $\phi(u,u)=0$ for any $u\in V$, then $\phi$ is identically zero by the polarisation identity.
    Otherwise, we can always find $u\in V\setminus\{0\}$ such that $\phi(u,u)\neq 0$.
    Write $u=e_1$ and define
    $$U=\langle \{e_i\}\rangle^\perp=\{v\in V:\phi(e_1,v)=0\}=\ker \phi(e_1,\cdot)$$
    So $\dim U=n-1$ and $U+e_1=U\oplus e_1$.
    Pick a basis $\{e_2,\ldots,e_n\}$ of $U$ such that $\phi|_U$ is diagonal in this basis, which is possible by induction hypothesis.
    Then $\phi$ is diagonal in $\{e_1,\ldots,e_n\}$.
\end{proof}
\begin{example}
    Take $V=\mathbb R^3$ and $Q(x)=x_1^2+x_2^2+2x_3^3+2x_1x_2+2x_1x_3-2x_2x_3$.
    By inspection if we take
    $$A=\begin{pmatrix}
        1&1&1\\
        1&1&-1\\
        1&-1&2
    \end{pmatrix}$$
    then $Q(x)=x^\top Ax$.
    Of course, we can follow the algorithm illustrated in the proof.
    We can alternatively complete the square to get
    \begin{align*}
        Q(x)&=x_1^2+x_2^2+2x_3^3+2x_1x_2+2x_1x_3-2x_2x_3\\
        &=(x_1+x_2+x_3)^2+(x_3-2x_2)^2-(2x)^2
    \end{align*}
    then under the new basis $x_1+x_2+x_3,x_3-2x_2,2x_2$, $A$ has the matrix
    $$A'=\begin{pmatrix}
        1&0&0\\
        0&1&0\\
        0&0&-1
    \end{pmatrix}$$
    More concretely, we have $A'=P^\top AP$ where
    $$P=\begin{pmatrix}
        1&1&1\\
        0&-2&1\\
        0&-2&0
    \end{pmatrix}^{-1}$$
\end{example}


\subsection{Volume Between Surfaces}
\noindent
Similar to how we could find the area between two curves in single-variable calculus, we can find the volume between two surfaces.
\begin{equation*}
	V_{\text{bwtn}} = \iint\limits_{D}{f\mathrm{d}A} - \iint\limits_{D}{g\mathrm{d}A} = \iint\limits_{D}{(f-g)\mathrm{d}A}
\end{equation*}

\noindent
If $g$ is below $f$ in $D_1 \subset D$ but is above $f$ in $D_2 \subset D$, where $D_1 \cup D_2 = D \text{ and } D_1 \cap D_2 = \emptyset$, we can still find the volume between the surfaces by splitting $D$ into $D_1$ and $D_2$.
\begin{equation*}
	V_{\text{bwtn}} = \iint\limits_{D}{\abs{f-g}\mathrm{d}A} = \iint\limits_{D_1}{(f-g)\mathrm{d}A} + \iint\limits_{D_2}{(g-f)\mathrm{d}A}
\end{equation*}


\lab{Inverse Problems}{Inverse Problems}
\label{lab:inverse_problems}

An important concept in mathematics is the idea of a well posed problem.
The concept initially came from Jacques Hadamard.
A mathematical problem is \textit{well posed} if
\begin{enumerate}
	\item a solution exists,
	\item that solution is unique, and
	\item the solution is continuously dependent on the data in the problem.
\label{inverse_problems:continuous_dependence}
\end{enumerate}
A problem that is not well posed is \textit{ill posed}.
Notice that a problem may be well posed, and yet still possess the property that small changes in the data result in larger changes in the solution; in this case the problem is said to be \text{ill conditioned}, and has a large \text{condition number}.

Note that for a physical phenomena, a well posed mathematical model would seem to be a necessary requirement!
However, there are important examples of mathematical problems that are ill posed.
For example, consider the process of differentiation.
Given a function $u$ together with its derivative $u'$, let $\tilde{u}(t) = u(t) +  \epsilon \sin(\epsilon^{-2}t)$ for some small $\epsilon > 0$.
Then note that
\begin{align*}
	\|u-\tilde{u}\|_{\infty} &= \epsilon,
\end{align*}
while
\begin{align*}
	\|u'-\tilde{u}'\|_{\infty} &= \epsilon^{-1}.
\end{align*}
Since a small change in the data leads to an arbitrarily large change in the output, differentiation is an ill posed problem.
And we haven't even mentioned numerically approximating a derivative!

For an example of an ill posed problem from PDEs, consider the backwards heat equation with zero Dirichlet conditions:
\begin{align}
\begin{split}
	&{} u_t = -u_{xx}, \quad (x,t) \in (0,L)\times (0,\infty),\\
	&{} u(0,t) = u(L,t) = 0, \quad t \in (0,\infty),\\
	&{} u(x,0) = f(x), \quad x \in (0,L).
\end{split}
\end{align}
For the initial data $f(x)$ the unique\footnote{See \textit{Partial Differential Equations} by Lawrence C. Evans, chapter 2.3, for a proof of uniqueness.} solution is $u(x,t) = 0.$
Given the initial data $f(x) = \frac{1}{n}\sin ( \frac{n \pi x}{L})$, one can check that there is a unique solution $u(x,t) = \frac{1}{n}\sin ( \frac{n \pi x}{L})\exp ( (\frac{n \pi }{L})^2 t)$.
Thus, on a finite interval $[0,T]$, as $n \to \infty$ we see that a small difference in the initial data results in an arbitrarily large difference in the solution.

\section*{Inverse Problems}
As implied by the name, inverse problems come in pairs.
For example, differentiation and integration are inverse problems.
The easier problem (in this case integration) is often called the direct problem.
The direct problem is usually studied first historically.

Given a physical system, together with initial data (the ``cause"), the direct problem will usually predict the future state of the physical system (the ``effect"); see Figure \ref{fig:cause_and_effect}.
Inverse problems often turn this on its head - given the current state of a physical system at time $T$, what was the physical state at time $t = 0$?

Alternatively, suppose we measure the current state of the system, and we then measure the state at some future time.
An important inverse problem is to determine an appropriate mathematical model that can describe the evolution of the system.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{cause_and_effect.pdf}
\caption{Cause and effect within a given physical system.}
\label{fig:cause_and_effect}
\end{figure}

\section*{Another look at heat flow through a rod}
Consider the following ordinary differential equation, together with natural boundary conditions at the ends of the interval\footnote{This example of an ill-posed problem is given in \textit{Inverse Problems in the Mathematical Sciences} by Charles W Groetsch.}:
\begin{align}
\begin{cases}
	-(au')' = f, & x \in (0,1),\\
	a(0)u'(0) = c_0, & a(1)u'(1) = c_1.
\end{cases} \label{inverse_problems:heat_flow}
\end{align}
This BVP can, for example, be used to describe the flow of heat through a rod.
The boundary conditions would correspond to specifying the heat flux through the ends of the rod.
The function $f(x)$ would then represent external heat sources along the rod, and $a(x)$ the density of the rod at each point.

Typically, the density $a(x)$ would be specified, along with any heat sources $f(x)$, and the (direct) problem is to solve for the steady-state heat distribution $u(x)$.
Here we shake things up a bit: suppose the heat sources $f$ are given, and we can measure the heat distribution $u(x)$.
Can we find the density of the rod?
This is an example of a \textit{parameter estimation problem}.

Let us consider a numerical method for solving \eqref{inverse_problems:heat_flow} for the density $a(x)$.
Subdivide $[0,1]$ into $N$ equal subintervals, and let $x_j = jh$, $j = 0, \ldots,N$, where $h = 1/N$.
Let $\phi_j(x)$ be the tent functions (used earlier in the finite element lab), given by
\begin{align*}
	\phi_j(x) = \begin{cases}
(x - x_{j-1})/h  &  x \in [x_{j-1},x_j],\\
 (x_{j+1} - x)/h  &  x \in [x_{j},x_{j+1}],\\
0 & \text{ otherwise.}
\end{cases}
\end{align*}
We look for an approximation $a^h(x)$ of the form
\begin{align}
	a^h &= \sum_{j=0}^N \alpha_j \phi_j, \quad \alpha_j=a(x_j).
\label{inverse_problems:approximate}
\end{align}
Integrating \eqref{inverse_problems:heat_flow} from $0$ to $x$, we obtain
\begin{align}
\begin{split}
&{} \int_0^x -(au')'\, ds = \int_0^x f(s)\, ds,\\
&{} -[a(x)u'(x) - c_0] = \int_0^x f(s)\, ds,\\
&{} u'(x) = \frac{c_0 - \int_0^x f(s)\, ds}{a(x)}.
\end{split}
\end{align}
Thus for each $x_j$
\begin{align*}
	u'(x_j) &= \frac{c_0 - \int_0^{x_j} f(s)\, ds}{a(x_j)},\\
	&= \frac{c_0 - \int_0^{x_j} f(s)\, ds}{\alpha_j}.
\end{align*}
The coefficients $\alpha_j$ in \eqref{inverse_problems:approximate} can now be approximated by minimizing
\begin{align*}
	\sum_{j=0}^N \left( \frac{c_0 - \int_0^{x_j} f(s)\, ds}{\alpha_j} - u'(x_j)  \right)^2.
\end{align*}
%For $c_0 = 3/8,$ $c_1 = 5/4$, $u(x) = x^2 + x/2 + 5/16$, and
%\begin{align*}
%	f &= \begin{cases}
%		-6x^2 + 3x - 1 & x \leq 1/2,\\
%		-1 & 1/2 < x \leq 1,
%	\end{cases}
%\end{align*}
%we partially implement this method with the following code:
%
%\begin{lstlisting}
%import numpy as np
%from scipy.optimize import minimize
%import matplotlib.pyplot as plt
%
%def f(x):
%	out = -np.ones(x.shape)
%	m = np.where(x<.5)
%	out[m] = -6*x[m]**2. + 3.*x[m] - 1.
%	return out
%
%def u(x):
%	return (x+1./4)**2. + 1./4
%
%def integral_of_f(x):
%	# out =  \int_0^x f(s) ds
%	return out
%
%def derivative_of_u(x):
%	# out = u'(x)
%	return out
%
%x = np.linspace(0,1,11)
%F, u_p = integral_of_f(x), derivative_of_u(x)
%
%def sum_of_squares(alpha):
%	pass
%
%guess = (1./4)*(3-x)
%sol = minimize(sum_of_squares,guess)
%
%plt.plot(x,sol.x,'-ob',linewidth=2)
%plt.show()
%\end{lstlisting}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{density_a.pdf}
\caption{The solution $a(x)$ to Problem \ref{prob:inverse}}
\label{fig:inverse_problems:num1}
\end{figure}

\begin{problem}
Solve \eqref{inverse_problems:heat_flow} for $a(x)$ using the following conditions:

\noindent $c_0 = 3/8,$ $c_1 = 5/4$, $u(x) = x^2 + x/2 + 5/16$, $x_j=.1j$ for $j=0,1,\dots,10$, and
\begin{align*}
	f &= \begin{cases}
		-6x^2 + 3x - 1 & x \leq 1/2,\\
		-1 & 1/2 < x \leq 1,
	\end{cases}
\end{align*}
Produce the plot shown in Figure \ref{fig:inverse_problems:num1}.

Hint: use the \li{minimize} function in \li{scipy.optimize} and some initial guess to find the $a_j$.
\label{prob:inverse}
\end{problem}

\begin{problem}
	Find the density function $a(x)$ satisfying
	\begin{align}
	\begin{cases}
		-(au')' = -1, & x \in (0,1),\\
		a(0)u'(0) = 1, & a(1)u'(1) = 2.
	\end{cases} \label{inverse_problems:ill_posed}
	\end{align}
	where $u(x) = x + 1 + \epsilon \sin(\epsilon^{-2}x)$.
Using several values of $\epsilon  > 0.66049142$, plot the corresponding density $a(x)$ for $x$ in \li{np.linspace(0,1,11)} to demonstrate that the problem is ill-posed.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{ill_posed_density_a.pdf}
\caption{The density function $a(x)$ satisfying \eqref{inverse_problems:ill_posed} for $\epsilon = .8$.}
\label{fig:inverse_problems:exercise1}
\end{figure}


\section{Statistical models}
\[E, \{ P_\theta \} _{\theta \in \Theta }\]
$E$ is a sample space for $X$ i.e. a set that contains all possible outcomes of $X$\\
$\displaystyle \{ \mathbb {P_\theta }\} _{\theta \in \Theta }$ is a family of probability distributions on $E$.\\
$\Theta$ is a parameter set, i.e. a set consisting of some possible values of $\Theta$.\\
$\theta$ is the true parameter and unknown. In a parametric model we assume that $\Theta \subset \mathbb{R}^d,$ for some $d \geq 1$.
\subsection{Identifiability}
\begin{align*}
&\theta \neq \theta' \Rightarrow \mathbb{P}_{\theta} \neq \mathbb{P}_{\theta'}\\
&\mathbb{P}_{\theta} = \mathbb{P}_{\theta'} \Rightarrow \theta = \theta'
\end{align*}
A Model is well specified if:
\begin{align*}
&\exists \theta \ s.t.\ \mathbb{P} =\mathbb{P}_{\theta }
\end{align*}


\section{Hitting and Absorption Probabilities}
\begin{definition}
    Let $(X_n)$ be a Markov chain.\\
    Then hitting time $H^A$ of a set $A\subset I$ is the random variable $\Omega\to\{0,1,\ldots\}\cup\{\infty\}$ given by $H^A(\omega)=\inf\{n\ge 0:X_n(\omega)\in A\}$ where $\inf\varnothing=+\infty$ by convention.\\
    The hitting probability of $A$ is
    $$h_i^A=\mathbb P_i[H^A<\infty]=\mathbb P_i[\text{hit $A$}]$$
    If $A$ is a closed class, $h_i^A$ is called the absorption probability.\\
    The mean hitting time is the expected time to reach $A$.
    $$k_i^A=\mathbb E_i[H^A]=\mathbb E[\text{time to hit $A$}]$$
\end{definition}
\begin{example}
    Consider the Markov chain with transition matrix
    $$\begin{pmatrix}
        1&&&\\
        1/2&&1/2&\\
        &1/2&&1/2\\
        &&&1
    \end{pmatrix}$$
    Observe that $\{1\},\{4\}$ are absorbing classes.
    Starting from $2$, we want to know the probability of absorption in $\{4\}$ and the average time before the chain is absorbed in $\{1\}$ and $\{4\}$.\\
    Let $h_i=h_i^{\{4\}}$ and $k_i=k_i^{\{1,4\}}$.\\
    Obviously $h_1=0$ and $h_4=1$.
    We have $h_2=h_1/2+h_3/2=h_3/2$ and $h_3=h_2/2+h_4/2=h_2/2+1/2$, solving which gives $h_2=1/3$.\\
    For $k$'s, we have $k_1=k_4=0$ and $k_2=1+k_1/2+k_3/2=1+k_3/2$, $k_3=1+k_2/2+k_4/2=1+k_2/2$.
    Solve for $k_2$ gives $k_2=2$.
\end{example}
We want to systematise our computation.
This inspires the following theorem:
\begin{theorem}
    The vector $h^A$ of hitting probabilities is the minimal nonnegative solution to the system
    $$\begin{cases}
        h_i^A=1\text{, for $i\in A$}\\
        h_i^A=\sum_{j\in I}p_{ij}h_j^A\text{, otherwise}
    \end{cases}$$
    We need it to be minimal in the sense that if $x=(x_i)_{i\in A}$ is another nonnegatve solution, then $x_i\ge h_i^A$ for all $i\in I$.
\end{theorem}
\begin{proof}
    $h^A$ is obviously a solution to the system.
    Indeed, if $X_0=i\in A$ it is obvious.
    Otherwise $X_0=i\notin A$, then by the Markov property,
    $$\mathbb P_i[H^A<\infty|X_1=j]=\mathbb P_j[H^A<\infty]=h_j^A$$
    So
    \begin{align*}
        h_i^A&=\mathbb P_i[H^A<\infty]\\
        &=\sum_{j\in I}\mathbb P_i[H^A<\infty|X_1=j]\mathbb P_i[X_1=j]\\
        &=\sum_{j\in I}h_j^Ap_{ij}
    \end{align*}
    To see it is minimal, let $x$ be any nonnegative solution to the system.
    If $i\in A$ we clearly have $x_i=1\ge 1=h_i^A$.
    For $i\notin A$, then
    \begin{align*}
        x_i&=\sum_{j\in I}p_{ij}x_j=\sum_{j\in A}p_{ij}x_j+\sum_{j\notin A}p_{ij}x_j\\
        &=\sum_{j\in A}p_{ij}+\sum_{j\notin A}p_{ij}\left( \sum_{k\in A}p_{jk}+\sum_{k\notin A}p_{jk}x_k \right)\\
        &=\mathbb P_i[X_1\in A]+\mathbb P_i[X_1\notin A,X_2\in A]+\sum_{j\notin A,k\notin A}p_{ij}p_{jk}x_k\\
        &=\cdots\\
        &=\sum_{s=1}^n\mathbb P[X_t\notin A\text{ for $t<s$},X_s\in A]+\sum_{j_1,\ldots,j_n\notin A}p_{ij_1}p_{j_1j_2}\cdots p_{j_{n-1}}p_{j_n}x_{j_n}\\
        &=\mathbb P_i[H^A\le n]+\sum_{j_1,\ldots,j_n\notin A}p_{ij_1}p_{j_1j_2}\cdots p_{j_{n-1}}p_{j_n}x_{j_n}\\
        &\ge\mathbb P_i[H^A\le n]
    \end{align*}
    So $x_i\ge \mathbb P_i[H^A\ge n]$ for all $n$, therefore
    $$x_i\ge\lim_{n\to\infty}P_i[H^A\le n]=\mathbb P_i[H^A<\infty]=h_i^A$$
    Therefore $h^A$ is minimal.\\
    To finish off the proof, observe that the system must has a solution since $h^A$ is one, and the solution obtained in this way must be $h^A$ since the minimality condition guaranteed uniqueness.
\end{proof}
\begin{example}
    Consider our previous example with transition matrix
    $$\begin{pmatrix}
        1&&&\\
        1/2&&1/2&\\
        &1/2&&1/2\\
        &&&1
    \end{pmatrix}$$
    Then the system in the above theorem is
    $$\begin{cases}
        h_1=h_1\\
        h_4=1\\
        h_2=h_1/2+h_3/2\\
        h_3=h_2/2+h_4/2
    \end{cases}$$
    which is what we obtained eariler except that it does not determine $h_1$.
    By the minimality condition we can then choose $h_1=0$ and get the results we got earlier.
\end{example}
\begin{example}[Gambler's Ruin]
    Consider the Markov chain with $\mathbb N\cup\{0\}$ many states where for any $i$, $p_{i,i+1}=p$ and $p_{i+1,i}=q=1-p$ for some fixed $p\in(0,1)$ and $p_{0,0}=1$.
    We want to know the absorption probability $h_i=h_i^{\{0\}}$.
    We can use this to model a casino, where a gambler with initial fortune $i$ wants to know the probability of him going broke.
    By the theorem, the system we need is
    $$\begin{cases}
        h_0=1\\
        h_i=ph_{i+1}+qh_{i-1},i>0
    \end{cases}$$
    Assuming $p\neq q$, then one can easily get the general solution to be $h_i=A+B(q/p)^i$ for constants $A,B$.
    For $p<q$ (which is the actual situation in most casino), then $0\le h_i\le 1$ gives $B=0$ and $A=1$ which means $h_i=1$ for all $i$.
    If $p>q$, then $h_0=1$ gives $B=1-A$, so
    $$h_i=\left( \frac{q}{p} \right)^i+A\left( 1-\left( \frac{q}{p} \right)^i \right)$$
    As $h_i\ge 0$ for all $i$, we have $A\ge 0$.
    The minimality condition then gives $A=0$, so $h_i=(q/p)^i$.\\
    If $p=q=1/2$, then the general solution for the recursion is $h_i=A+Bi$ for constants $A,B$.
    We can put in the initial conditions as usual and obtain $h_i=1$ for all $i$.
\end{example}
\begin{example}[Birth and Death Chain]
    Again take the state space as the nonnegative integers.
    For $i>0$, we set $p_{i,i+1}=p_1$ and $p_{i,i-1}=q_1=1-p_1$ where $p_i\in (0,1)$ for all $i$.
    Set $p_{0,0}=1$ as usual.
    We again want to know the absorbing probability $h_i=h_i^{\{0\}}$ which can be interpreted as the extinction probability.
    Now the theorem told us
    $$\begin{cases}
        h_0=1\\
        h_i=p_ih_{i+1}+q_ih_{i-1},i>0
    \end{cases}$$
    Consider $u_i=h_{i-1}-h_i$, then for $i>0$,
    $$p_iu_{i+1}-q_iu_i=p_ih_i-p_ih_{i+1}-q_ih_{i-1}+q_ih_i=(p_i+q_i-1)h_i=0$$
    So we have
    $$u_{i+1}=\frac{q_i}{p_i}u_i=\left( \frac{q_iq_{i-1}\cdots q_1}{p_ip_{i-1}\cdots p_1} \right)u_1=\gamma_iu_1,\gamma_i=\frac{q_iq_{i-1}\cdots q_1}{p_ip_{i-1}\cdots p_1}$$
    Hence we have
    $$h_i=1-(h_0-h_i)=1-A(\gamma_0+\cdots +\gamma_{i-1}),\gamma_0=1,A=u_1$$
    If $\sum_i\gamma_i=\infty$, then $h_i\in [0,1]$ gives $A=0$, so $h_i=1$ for all $i$.
    Otherwise $\gamma=\sum_i\gamma_i\in\mathbb R_+$, so by minimality $A=\gamma^{-1}$.
    Hence,
    $$h_i=\frac{\sum_{j=i}^\infty\gamma_j}{\sum_{j=0}^\infty\gamma_j}$$
    In particular, for any $i$, we always have $h_i\in (0,1)$, so the population survives or dies both with positive probability.
\end{example}
We've investigated a lot on hitting probability, but what about mean hitting time?
\begin{theorem}
    The vector of mean hitting times $k^A=(k_i^A)_{i\in I}$ is the minimal solution to
    $$\begin{cases}
        k_i^A=0,i\in A\\
        k_i^A=1+\sum_{j\notin A}p_{ij}k_j^A,i\notin A
    \end{cases}$$
\end{theorem}
The proof is quite analogous to the previous theorem on hitting time.
\begin{proof}
    The same idea as in hitting time works to show that $k^A$ is indeed a solution.
    If $X_0=i\in A$ then obvously $k_i^A=0$.
    Otherwise $X_0=i\notin A$, so by the Markov property
    $$\mathbb E_i[H^A|X_1=j]=1+\mathbb E_j[H^A]=1+k_j^A$$
    Therefore
    $$k_i^A=\mathbb E_i[H^A]=\sum_{j\in I}\mathbb E_i[H^A|X_1=j]\mathbb P_i[X_1=j]=\sum_{j\in I}(1+k_j^A)p_{ij}=1+\sum_{j\in I}k_j^Ap_{ij}$$
    To show this solution is minimal, assume $x$ is any other nonegative solution, then $x_i=0\ge 0=k_i^A$ for any $i\in A$.
    For $i\notin A$,
    \begin{align*}
        x_i&=1+\sum_{j\notin A}p_{ij}x_j\\
        &=1+\sum_{j\notin A}p_{ij}\left( 1+\sum_{k\notin A}p_{jk}x_k \right)\\
        &=\mathbb P_i[H^A\ge 1]+\mathbb P_i[H^A\ge 2]+\sum_{j\notin A,k\notin A}p_{ij}p_{jk}x_k\\
        &=\cdots\\
        &=\sum_{s=1}^n\mathbb P_i[H^A\ge s]+\sum_{j_1,\ldots,j_n\notin A}p_{ij_1}\cdots p_{j_{n-1}j_n}x_{j_n}\\
        &\ge\sum_{s=1}^n\mathbb P_i[H^A\ge s]\ge\sum_{s=1}^\infty\mathbb P_i[H^A\ge s]\\
        &=\mathbb E_i[H^A]=k_i^A
    \end{align*}
    which shows the minimality.
    The proof can be finished off in the same way as we did for the previous theorem.
\end{proof}


\subsection{Faraday's Law of Induction \& Ampere's Law}
\noindent
Faraday’s Law of Induction quantifies the idea that changing magnetic flux in a coil induces a current in the coil. More precisely, it induces a voltage, the potential function of electric field $\left(\vec{E} = \nabla V\right)$, and this field will oppose the magnetic field that induced it. More formally,
\begin{equation*}
	-\frac{\partial}{\partial t}\iint\limits_{S}{\vec{B} \cdot \mathrm{d}\vec{s}} = \oint\limits_{C}{\vec{E} \cdot \mathrm{d}\vec{r}}
\end{equation*}
\begin{equation*}
	-\iint\limits_{S}{\frac{\partial}{\partial t}\vec{B} \cdot \mathrm{d}\vec{s}} = \iint\limits_{S}{\nabla \times \vec{E} \cdot \mathrm{d}\vec{s}}
\end{equation*}
by Stokes's Theorem.\\
As $S$ collapses to a point,
\begin{equation*}
	\nabla \times \vec{E} = -\frac{\partial}{\partial t}\vec{B}.
\end{equation*}
This is Faraday's Law. It is the 3rd of Maxwell's Laws.\\

\noindent
The final of Maxwell's Equations is Ampere's Law (with Maxwell's correction). It says that
\begin{equation*}
	\nabla \times \vec{B} = \mu_0\epsilon_0\frac{\partial\vec{e}}{\partial t} + \mu_0J
\end{equation*}
where $J$ is the current density and $\mu_0$ is the permeability of free space. Using Maxwell's equations and some basic properties of waves, we can derive the speed of light as $c = \frac{1}{\sqrt{\mu_0\epsilon_0}}$.


\section{Preprocessing of the Samples}
\label{sec:Preprocessing}
In order to make samples more comparable, all samples are preprocessed. First, the multi-channel samples are panned to mono. Since there are no spatial features used in the project, no stereo information is needed. To remove the stereo information the arithmetic mean in between the left and right channel is calculated. This is the mid signal of the stereo signal
\[
    x_\text{mid}[k] = \frac{1}{2} \left( x_\text{left}[k] + x_\text{right}[k] \right).
\]
Second, silence at the beginning and end of each sample is cut off. All signal below a threshold level at -75~dBFS is detected as silence
\[
    x_\text{sil}[k] =
    \begin{cases}
          0 & \text{if} \quad 20\cdot \lg(x_\text{mid}[a]) < -75~\forall~a \leq k\\
          0 & \text{if} \quad 20\cdot \lg(x_\text{mid}[a]) < -75~\forall~a \geq k\\
          x_\text{mid}[k] & \text{else}
    \end{cases}
\]
Lastly, all samples were amplified to have an root mean square (RMS) of 1 in order to normalize the energy of the samples
\[
    x_\text{pre}[k] = \frac{x_\text{sil}[k]}{\sqrt{\overline{x^2}}},
\]
$\overline{x^2}$ is the mean of the squared signal.


\subsection{Isomorphisms}

The focus of this section is on linear transformations which are both one to one and onto. When this is the case, we call the transformation an isomorphism.

\begin{definition}{Isomorphism}{isomorphism}
Let $V$ and $W$ be two vector spaces and let $T: V \to W$ be a linear transformation.
Then $T$ is called an \textbf{isomorphism}
\index{isomorphism}if the following two conditions are satisfied.

\begin{itemize}
\item $T$ is one to one.

\item $T$ is onto.
\end{itemize}
\end{definition}

\begin{definition}{Isomorphic}{isomorphic}
Let $V$ and $W$ be two vector spaces and let $T: V \to W$ be a linear transformation.
Then if $T$ is an isomorphism, we say that $V$ and $W$ are \textbf{isomorphic.}
\index{isomorphic}
\end{definition}

Consider the following example of an isomorphism.

\begin{example}{Isomorphism}{isomorphism}
Let $T:\Mat_{2,2}\to\R^4$ be defined by
\[
T\begin{mymatrix}{cc} a & b \\ c & d \end{mymatrix}
=\begin{mymatrix}{c} a\\ b\\ c \\ d \end{mymatrix}
\mbox{ for all }
\begin{mymatrix}{cc} a & b \\ c & d \end{mymatrix}
\in\Mat_{2,2}. \]
Show that $T$ is an isomorphism.
\end{example}

\begin{solution}
Notice that if we can prove $T$ is an isomorphism, it will mean that $\Mat_{2,2}$ and $\R^4$ are isomorphic.
It remains to prove that
\begin{enumerate}
\item $T$ is a linear transformation;
\item $T$ is one-to-one;
\item $T$ is onto.
\end{enumerate}

\textbf{$T$ is linear:}
Let $k,p$ be scalars.
\begin{eqnarray*}
T \paren{k \begin{mymatrix}{cc} a_1 & b_1 \\ c_1 & d_1 \end{mymatrix} +  p \begin{mymatrix}{cc} a_2 & b_2 \\ c_2 & d_2 \end{mymatrix}}
&=& T \paren{\begin{mymatrix}{cc} k a_1 & k b_1 \\ k c_1 & k d_1 \end{mymatrix} +  \begin{mymatrix}{cc} p a_2 & p b_2 \\ p c_2 & p d_2 \end{mymatrix}} \\
&=& T \paren{\begin{mymatrix}{cc} k a_1 + p a_2 & k b_1 + p b_2 \\ k c_1 + p c_2& k d_1 + p d_2 \end{mymatrix}} \\
&=& \begin{mymatrix}{c} k a_1 + p a_2 \\ k b_1 + p b_2 \\ k c_1 + p c_2 \\ k d_1 + p d_2 \end{mymatrix}  \\
&=& \begin{mymatrix}{c} k a_1 \\ k b_1 \\ k c_1 \\ k d_1 \end{mymatrix} + \begin{mymatrix}{c} p a_2 \\ p b_2 \\ p c_2 \\ p d_2 \end{mymatrix} \\
&=& k \begin{mymatrix}{c} a_1 \\  b_1 \\  c_1 \\  d_1 \end{mymatrix} + p \begin{mymatrix}{c}  a_2 \\  b_2 \\  c_2 \\  d_2 \end{mymatrix} \\
&=& k T \paren{\begin{mymatrix}{cc} a_1 & b_1 \\ c_1 & d_1 \end{mymatrix}} +  p T \paren{\begin{mymatrix}{cc} a_2 & b_2 \\ c_2 & d_2 \end{mymatrix}}
\end{eqnarray*}

Therefore $T$ is linear.

\textbf{$T$ is one-to-one:}
By Lemma~\ref{lem:one-to-one-abstract} we need to show that if $T(A) = 0$ then $A = 0$ for some matrix $A \in \Mat_{2,2}$.
\[
T\begin{mymatrix}{cc} a & b \\ c & d \end{mymatrix}
= \begin{mymatrix}{c} a\\ b\\ c \\ d \end{mymatrix}
= \begin{mymatrix}{c} 0 \\ 0 \\ 0 \\ 0 \end{mymatrix}
\]

This clearly only occurs when $a=b=c=d=0$ which means that
\[
A = \begin{mymatrix}{cc} a & b \\ c & d \end{mymatrix} = \begin{mymatrix}{cc} 0 & 0 \\ 0 & 0 \end{mymatrix} = 0
\]

Hence $T$ is one-to-one.

\textbf{$T$ is onto:}
Let
\[ \vect{x}=\begin{mymatrix}{c} x_1\\x_2\\x_3\\x_4 \end{mymatrix}\in\R^4,\]
and
define matrix $A\in\Mat_{2,2}$ as follows:
\[ A=\begin{mymatrix}{cc} x_1 & x_2 \\ x_3 & x_4 \end{mymatrix}.\]

Then $T(A)=\vect{x}$, and therefore $T$ is onto.

Since $T$ is a linear transformation which is one-to-one and onto, $T$ is an isomorphism. Hence $\Mat_{2,2}$ and $\R^4$ are isomorphic.
\end{solution}

An important property of isomorphisms is that the inverse of an isomorphism
is itself an isomorphism and the composition of isomorphisms is an
isomorphism. We first recall the definition of composition.

\begin{definition}{Composition of transformations}{composite-transformation}
Let $V, W, Z$ be vector spaces and suppose $T: V \to W$ and $S: W \to Z$ are linear transformations. Then the composite of $S$ and $T$ is
\[
S \circ T: V \to Z
\]
and is defined by
\[
(S \circ T) (\vect{v}) = S(T(\vect{v})) \mbox{ for all } \vect{v} \in V
\]
\end{definition}

Consider now the following proposition.

\begin{proposition}{Composite and inverse isomorphism}{composite-inverse}
Let $T:V\rightarrow W$ be an isomorphism. Then $T^{-1}:W\rightarrow V$ is
also an isomorphism. Also if $T:V\rightarrow W$ is an isomorphism and if $
S:W\rightarrow Z$ is an isomorphism for the vector spaces $V,W,Z$, then $
S\circ T$ defined by $(S\circ T) (v) = S(
T(v)) $ is also an isomorphism.
\end{proposition}

\begin{proof}
Consider the first claim. Since $T$ is onto, a typical
vector in $W$ is of the form $T(\vect{v})$ where $\vect{v} \in V$. Consider then for $a,b$
scalars,
\begin{equation*}
T^{-1}(aT(\vect{v}_{1})+bT(\vect{v}_{2}))
\end{equation*}
where $\vect{v}_{1}, \vect{v}_2 \in V$. Consider if this is equal to
\begin{equation*}
aT^{-1}(T(\vect{v}_{1})) +bT^{-1}(T(\vect{v}_{2})) =a\vect{v}_{1}+b\vect{v}_{2}?
\end{equation*}
Since $T$ is one to one, this will be so if
\begin{equation*}
T(a\vect{v}_{1}+b\vect{v}_{2}) =T(T^{-1}(aT(\vect{v}_{1})+bT(\vect{v}_{2}))
) =aT(\vect{v}_{1})+bT(\vect{v}_{2})
\end{equation*}
However, the above statement is just the condition that $T$ is a linear map.
Thus $T^{-1}$ is indeed a linear map. If $\vect{v} \in V$ is given, then $
\vect{v}=T^{-1}(T(\vect{v})) $ and so $T^{-1}$ is onto. If $T^{-1}(\vect{v})=\vect{0}$, then
\begin{equation*}
\vect{v}=T(T^{-1}(\vect{v})) =T(\vect{0})=\vect{0}
\end{equation*}
and so  $T^{-1}$ is one to one.

Next suppose $T$ and $S$ are as described. Why is $S\circ T$ a linear map?
Let for $a,b$ scalars,
\begin{eqnarray*}
S\circ T(a\vect{v}_{1}+b\vect{v}_{2}) &\equiv &S(T(
a\vect{v}_{1}+b\vect{v}_{2})) =S(aT(\vect{v}_{1})+bT(\vect{v}_{2})) \\
&=&aS(T(\vect{v}_{1})) +bS(T(\vect{v}_{2})) \equiv a(S\circ
T) (\vect{v}_{1}) +b(S\circ T) (\vect{v}_{2})
\end{eqnarray*}
Hence $S\circ T$ is a linear map. If $(S\circ T) (\vect{v})
=0$, then $S(T(\vect{v})) =\vect{0}$ and it follows that $T(\vect{v})=\vect{0}$ and hence by this lemma again, $\vect{v}=\vect{0}$. Thus $S\circ
T $ is one to one. It remains to verify that it is onto. Let $\vect{z}\in Z$. Then
since $S$ is onto, there exists $\vect{w}\in W$ such that $S(\vect{w})=\vect{z}$. Also, since $T$
is onto, there exists $\vect{v}\in V$ such that $T(\vect{v})=\vect{w}$. It follows that $S(
T(\vect{v})) =\vect{z}$ and so $S\circ T$ is also onto.
\end{proof}

Suppose we say that two vector spaces $V$ and $W$ are related if there exists an isomorphism of one to the other, written as $V\iso W$.
Then the above proposition suggests that $\iso$ is an equivalence relation. That is: $\iso$
satisfies the following conditions:

\begin{itemize}
\item $V\iso V$

\item If $V\iso W$, it follows that $W\iso V$

\item If $V\iso W$ and $W\iso Z$, then $V\iso Z$
\end{itemize}

We leave the proof of these to the reader.

The following fundamental lemma describes the relation between bases and
isomorphisms.

\begin{lemma}{Bases and isomorphisms}{bases-isomorphism}
Let $T:V\rightarrow W$ be a
\index{isomorphism!invertible matrices}
\index{invertible matrices!isomorphism} linear map where $V,W$ are vector spaces.  Then a linear transformation $T$ which is one to one has the property that
if $\set{\vect{u}_{1},\ldots,\vect{u}_{k}} $ is linearly
independent, then so is $\set{T(\vect{u}_{1}),\ldots,T(\vect{u}_{k})}$.
 More generally, $T$ is an
isomorphism if and only if whenever $\set{
\vect{v}_{1},\ldots,\vect{v}_{n}} $ is a basis for $V$, it follows
that $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ is a basis for $W$.
\end{lemma}

\begin{proof}
First suppose that $T$ is a linear map and is one to one
and $\set{\vect{u}_{1},\ldots,\vect{u}_{k}} $ is linearly
independent. It is required to show that $\set{T(\vect{u}_{1}),\ldots,T(\vect{u}_{k})} $ is also linearly independent. Suppose then that
\begin{equation*}
\sum_{i=1}^{k}c_{i}T(\vect{u}_{i})=\vect{0}
\end{equation*}
Then, since $T$ is linear,
\begin{equation*}
T\paren{\sum_{i=1}^{n}c_{i}\vect{u}_{i}} =\vect{0}
\end{equation*}
Since $T$ is one to one, it follows that
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{u}_{i}=0
\end{equation*}
Now the fact that $\set{\vect{u}_{1},\ldots,\vect{u}_{n}} $ is
linearly independent implies that each $c_{i}=0$. Hence $\set{T(\vect{u}
_{1}),\ldots,T(\vect{u}_{n})} $ is linearly independent.

Now suppose that $T$ is an isomorphism and $\set{\vect{v}_{1},\ldots,\vect{
v}_{n}} $ is a basis for $V$. It was just shown that $\set{T(\vect{v}
_{1}),\ldots,T(\vect{v}_{n})} $ is linearly independent. It remains to
verify that the span of $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $
is all of $W$. This is where $T$ is onto is used. If $\vect{w}\in W$, there
exists $\vect{v}\in V$ such that $T(\vect{v})=\vect{w}$. Since $\set{\vect{v}
_{1},\ldots,\vect{v}_{n}} $ is a basis, it follows that there exists
scalars $\set{c_{i}} _{i=1}^{n}$ such that
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\vect{v}.
\end{equation*}
Hence,
\begin{equation*}
\vect{w}=T(\vect{v})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}}
=\sum_{i=1}^{n}c_{i}T\vect{v}_{i}
\end{equation*}
which shows that the span of these vectors $\set{T(\vect{v}_{1}),\ldots,T
(\vect{v}_{n})} $ is all of $W$ showing that this set of vectors is a
basis for $W$.

Next suppose that $T$ is a linear map which takes a basis to a basis. Then
for $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $ a basis for $V$, it
follows $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ is a basis for $
W$. Then if $w\in W$, there exist scalars $c_{i}$ such that $
w=\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} $
showing that $T$ is onto. If $T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} =0$
then $\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=\vect{0}$ and since the vectors $\set{T(\vect{v}
_{1}),\ldots,T(\vect{v}_{n})} $ are linearly independent, it follows
that each $c_{i}=0$. Since $\sum_{i=1}^{n}c_{i}\vect{v}_{i}$ is a typical vector in
$V$, this has shown that if $T(\vect{v})=0$ then $\vect{v}=\vect{0}$ and so $T$ is also one to one.
Thus $T$ is an isomorphism.
\end{proof}

The following theorem illustrates a very useful idea for defining an
isomorphism. Basically, if you know what it does to a basis, then you can
construct the isomorphism.

\begin{theorem}{Isomorphic vector spaces}{isomorphic-vector-spaces}
Suppose $V$ and $W$ are two vector spaces. Then the two vector spaces are isomorphic if and only
if they have the same dimension. In the case that the two vector spaces have
the same dimension, then for
\index{isomorphism!equivalence} a linear transformation $T:V\rightarrow W$, the
following are equivalent.

\begin{enumerate}
\item $T$ is one to one.

\item $T$ is onto.

\item $T$ is an isomorphism.
\end{enumerate}
\end{theorem}

\begin{proof}Suppose first these two vector spaces have the same
dimension. Let a basis for $V$ be $\set{
\vect{v}_{1},\ldots,\vect{v}_{n}} $ and let a basis for $W$ be $
\set{\vect{w}_{1},\ldots,\vect{w}_{n}}$. Now define $T$ as
follows.
\begin{equation*}
T(\vect{v}_{i})=\vect{w}_{i}
\end{equation*}
for $\sum_{i=1}^{n}c_{i}\vect{v}_{i}$ an arbitrary vector of $V$,%
\begin{equation*}
T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} = \sum_{i=1}^{n}c_{i}T
(\vect{v}_{i})=\sum_{i=1}^{n}c_{i}\vect{w}_{i}.
\end{equation*}
It is necessary to verify that this is well-defined. Suppose then that
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\sum_{i=1}^{n}\hat{c}_{i}\vect{v}_{i}
\end{equation*}
Then
\begin{equation*}
\sum_{i=1}^{n}(c_{i}-\hat{c}_{i}) \vect{v}_{i}=0
\end{equation*}
and since $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $ is a basis, $
c_{i}=\hat{c}_{i}$ for each $i$. Hence
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{w}_{i}=\sum_{i=1}^{n}\hat{c}_{i}\vect{w}_{i}
\end{equation*}
and so the mapping is well-defined. Also if $a,b$ are scalars,
\begin{eqnarray*}
T\paren{a\sum_{i=1}^{n}c_{i}\vect{v}_{i}+b\sum_{i=1}^{n}\hat{c}_{i}\vect{v}
_{i}} &=&T(\sum_{i=1}^{n}(ac_{i}+b\hat{c}_{i}) \vect{v
}_{i}) =\sum_{i=1}^{n}(ac_{i}+b\hat{c}_{i}) \vect{w}_{i} \\
&=&a\sum_{i=1}^{n}c_{i}\vect{w}_{i}+b\sum_{i=1}^{n}\hat{c}_{i}\vect{w}_{i} \\
&=&aT\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} +bT\paren{\sum_{i=1}^{n}
\hat{c}_{i}\vect{v}_{i}}
\end{eqnarray*}
Thus $T$ is a linear map.

Now if
\begin{equation*}
T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} =\sum_{i=1}^{n}c_{i}\vect{w}
_{i}=\vect{0},
\end{equation*}
then since the $\set{\vect{w}_{1},\ldots,\vect{w}_{n}} $ are
independent, each $c_{i}=0$ and so $\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\vect{0}$
also. Hence $T$ is one to one. If $\sum_{i=1}^{n}c_{i}\vect{w}_{i}$ is a
vector in $W$, then it equals
\begin{equation*}
\sum_{i=1}^{n}c_{i}T\vect{v}_{i}=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}}
\end{equation*}
showing that $T$ is also onto. Hence $T$ is an isomorphism and so $V$ and $W$
are isomorphic.

Next suppose these two vector spaces are isomorphic. Let $T$ be the name of
the isomorphism. Then for $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $
a basis for $V$, it follows that a basis for $W$
is $\set{T\vect{v}_{1},\ldots,T\vect{v}_{n}} $ showing that the two
vector spaces have the same dimension.

Now suppose the two vector spaces have the same dimension.

First consider the claim that $1.)\Rightarrow 2.)$. If $T$ is one to one,
then if $\set{\vect{v}_{1},\ldots,\vect{v}
_{n}} $ is a basis for $V$, then $\set{T(\vect{v}_{1}),\ldots,T(\vect{v
}_{n})} $ is linearly independent. If it is not a basis, then it must
fail to span $W$. But then there would exist $\vect{w}\notin \sspan
\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ and it follows that $
\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n}),\vect{w}} $ would be
linearly independent which is impossible because there exists a basis for $W$
of $n$ vectors. Hence
\[
\sspan\set{T(\vect{v}_{1}),\ldots,T(\vect{v}
_{n})} =W
\]
and so $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $
is a basis. Hence, if $\vect{w}\in W$, there exist scalars $c_{i}$ such that
\begin{equation*}
\vect{w}=\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}
_{i}}
\end{equation*}
showing that $T$ is onto. This shows that $1.)\Rightarrow 2.)$.

Next consider the claim that $2.)\Rightarrow 3.)$. Since $2.)$ holds, it
follows that $T$ is onto. It remains to verify that $T$ is one to one. Since
$T$ is onto, there exists a basis of the form $\set{T(\vect{v}_{i}),\ldots,T
(\vect{v}_{n})}$. If $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $
is linearly independent, then this set of vectors must also be a basis for $
V $ because if not, there would exist $\vect{u}\notin \sspan\set{\vect{
v}_{1},\ldots,\vect{v}_{n}} $ so $\set{\vect{v}_{1},\ldots,\vect{v}
_{n},\vect{u}} $ would be a linearly independent set which is
impossible because by assumption, there exists a basis which has $n$
vectors. So why is$\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $
linearly independent? Suppose
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\vect{0}
\end{equation*}
Then
\begin{equation*}
\sum_{i=1}^{n}c_{i}T\vect{v}_{i}=\vect{0}
\end{equation*}
Hence each $c_{i}=0$ and so, as just discussed, $\set{\vect{v}_{1},\ldots,
\vect{v}_{n}} $ is a basis for $V$. Now it follows that a typical
vector in $V$ is of the form $\sum_{i=1}^{n}c_{i}\vect{v}_{i}$. If $T\paren{
\sum_{i=1}^{n}c_{i}\vect{v}_{i}} =\vect{0}$, it follows that
\begin{equation*}
\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=\vect{0}
\end{equation*}
and so, since $\set{T(\vect{v}_{i}),\ldots,T(\vect{v}_{n})} $ is
independent, it follows each $c_{i}=0$ and hence $\sum_{i=1}^{n}c_{i}\vect{v}
_{i}=\vect{0}$. Thus $T$ is one to one as well as onto and so it is an
isomorphism.

If $T$ is an isomorphism, it is both one to one and onto by definition so $
3.)$ implies both $1.)$ and $2.)$.
\end{proof}

Note the interesting way of defining a linear transformation in the first
part of the argument by describing what it does to a basis and then
``extending it linearly''.

Consider the following example.

\begin{example}{}{}
Let $V=\R^{3}$ and let $W$ denote the polynomials of degree at most
2. Show that these two vector spaces are isomorphic.
\end{example}

\begin{solution}
First, observe that a basis for $W$ is $\set{1,x,x^{2}} $ and a basis for $V$
is $\set{\vect{e}_{1},\vect{e}_{2},\vect{e}_{3}}$. Since these two
have the same dimension, the two are
isomorphic. An example of an isomorphism is this:\
\begin{equation*}
T(\vect{e}_{1})=1,T(\vect{e}_{2})=x,T(\vect{e}_{3})=x^{2}
\end{equation*}
and extend $T$ linearly as in the above proof. Thus
\begin{equation*}
T(a,b,c) =a+bx+cx^{2}
\end{equation*}
\end{solution}


\subsection{Boundary Points, Open \& Closed Sets}
\noindent
Given some set $\Omega \subset \mathbb{R}^n$, $x$ is an interior point to $\Omega$ if $\exists \delta \mid N(x, \delta) \subset \Omega$. That is, $x$ is an interior point to $\Omega$ if you can draw a circle of non-zero radius around $x$ such that the entire circle is inside of $\Omega$.\\
All points that are not interior points are boundary points. Formally, $x$ is a boundary point of $\Omega$ if $\forall \delta \mid N(x,\delta) \not\subset \Omega$.\\
Using our definitions of interior and boundary points, we can define and open set as one that doesn't contain any of its boundary points and a closed set as one that contains all of its boundary point. Note that a set that contains some of its boundary points is neither open nor closed.


\subsection{Inequality constraints}

linear programming means of the form
max \(c^Tx\)
st. \(Ax<=b\)
\(x>=0\)
this is the canonical form

\subsubsection{Lagrangians with inequality constraints}

We can add constraints to an optimisation problem. These constraints can be equality constraints or inequality constraints. We can write constrained optimisation problem as:

Minimise \(f(x)\) subject to

\(g_i(x)\le 0\) for \(i=1,...,m\)

\(h_i(x)=0\) for \(i=1,…,p\)

We write the Lagrangian as:

\(\mathcal{L}(x, \lambda, \nu )=f(x)+\sum_{i=1}^m\lambda_i g_i(x)+\sum_{i=1}^p\nu_ih_i(x)\)

If we try and solve this like a standard Lagrangian, then all of the inequality constraints will instead by equality constraints.

\subsubsection{Affinity of the Lagrangian}

The Lagrangian function is affine with respect to \(\lambda \) and \(\nu \).

\(\mathcal{L}(x, \lambda, \nu )=f(x)+\sum_{i=1}^m\lambda_i g_i(x)+\sum_{i=1}^p\nu_ih_i(x)\)

\(\mathcal{L}_{\lambda_i}(x, \lambda, \nu )=g_i(x)\)

\(\mathcal{L}_{\nu_i}(x, \lambda, \nu )=h_i(x)\)

As the partial differential is constant, the partial differential is an affine function.


\documentclass{ximera}
\input{../preamble}
\title{Exercises: Cumulative}
%%%%%\author{Philip T. Gressman}

\begin{document}
\begin{abstract}
Exercises relating to various topics we have studied.
\end{abstract}
\maketitle

\begin{exercise}
The series
\[ \sum_{n=1}^\infty \left( e^{ \frac{1}{n}} - 1 \right) \]
is \wordChoice{\choice{convergent}\choice[correct]{divergent}} by limit comparison to the $p$-series
\[ \sum_{n=1}^\infty \frac{1}{n^p} \]
with $p = \answer{1}$.  Likewise, the
series
\[ \sum_{n=1}^\infty \left( e^{ \frac{1}{n}} - 1 - \frac{1}{n} \right) \]
is \wordChoice{\choice[correct]{convergent}\choice{divergent}} by limit comparison to the $p$-series
with $p = \answer{2}$.
\begin{hint}
Use a Maclaurin series to determine the dominant behavior of
\[ e^{\frac{1}{n}} - 1 \]
as $n \rightarrow \infty$.
\end{hint}
\end{exercise}

\begin{exercise}
Fill in the blank below with an appropriate constant to make the series absolutely convergent:
\[ \sum_{n=1}^\infty \left[ \ln \left( 1 + \frac{1}{n} \right) - \frac{\answer{1}}{n} \right] \]
\begin{hint}
Use the Maclaurin series for $\ln (1 + x)$ and substitute $x = 1/n$.
\end{hint}
\end{exercise}

\begin{exercise}
Determine whether the series below converges absolutely, conditionally, or diverges.
\[ \sum_{n=1}^\infty n \sin \frac{1}{n} \]
\begin{multipleChoice}
\choice{Absolute}
\choice{Conditional}
\choice[correct]{Diverge}
\end{multipleChoice}
\begin{hint}
Show that the terms do not go to zero as $n \rightarrow \infty$.
\end{hint}
\end{exercise}

\begin{exercise}
Determine whether the series below converges absolutely, conditionally, or diverges.
\[ \sum_{n=1}^\infty \frac{1}{n} \sin \frac{1}{n} \]
\begin{multipleChoice}
\choice[correct]{Absolute}
\choice{Conditional}
\choice{Diverge}
\end{multipleChoice}
\begin{hint}
Use the Maclaurin series for $\sin x$ and evaluate at $x = 1/n$. Find the dominant term.
\end{hint}
\end{exercise}

\begin{exercise}
Determine whether the series below converges absolutely, conditionally, or diverges.
\[ \sum_{n=1}^\infty \frac{1}{n} \ln \left(2 + \frac{1}{n} \right) \]
\begin{multipleChoice}
\choice{Absolute}
\choice{Conditional}
\choice[correct]{Diverge}
\end{multipleChoice}
\begin{hint}
Try limit comparison with the harmonic series.
\end{hint}
\end{exercise}
\begin{exercise}
Determine whether the series below converges absolutely, conditionally, or diverges.
\[ \sum_{n=1}^\infty \frac{1}{n} \ln \left(1 + \frac{1}{n} \right) \]
\begin{multipleChoice}
\choice[correct]{Absolute}
\choice{Conditional}
\choice{Diverge}
\end{multipleChoice}
\begin{hint}
Do a Taylor expansion of $\ln (1 + x)$ and evaluate at $x = 1/n$.
\end{hint}
\end{exercise}

\begin{exercise}
Compute the sum of the series below.
\[ \sum_{n=0}^\infty \frac{(-1)^n}{2n+1} \frac{1}{3^n} \]
\begin{itemize}
\item We know the series
\[ \sum_{n=0}^\infty \frac{(-1)^n x^{2n+1}}{2n+1} = \answer{\arctan x} \]
for $-1 < x < 1$.
\item This means
\[ \sum_{n=0}^\infty \frac{(-1)^n x^{2n}}{2n+1} = \answer{\frac{\arctan x}{x}}. \]
\item We conclude
\[  \sum_{n=0}^\infty \frac{(-1)^n}{2n+1} \frac{1}{3^n} = \answer{\frac{\pi \sqrt{3}}{6}}. \]
\end{itemize}
\end{exercise}

\begin{exercise}
Compute the sum of the series below.
\[ \sum_{n=0}^\infty n 3^{-n} = \answer{\frac{3}{4}}. \]
\begin{hint}
\[ \frac{1}{1-x} = \sum_{n=0}^\infty x^n \ \ \text{ when } -1 < x < 1. \]
\end{hint}
\begin{hint}
\[ \frac{1}{(1-x)^2} = \sum_{n=0}^\infty n x^{n-1} \ \ \text{ when } -1 < x < 1. \]
\end{hint}
\end{exercise}

\begin{exercise}
Compute the sum of the series below.
\[ \sum_{n=1}^\infty \frac{3}{n(n+1)} = \answer{3}. \]
\begin{hint}
You don't need Taylor series for this one.
\end{hint}
\begin{hint}
It's a telescoping series
\end{hint}
\end{exercise}

\section*{Sample Exam Questions}

\begin{question}%%%%%[2015C.02]

If it converges, find the sum of the series \(\displaystyle \sum_{n=0}^\infty \frac{(-1)^n \pi^{2n}}{3^{2n} (2n)!}\). \offline{If the series diverges, explain why.}
\begin{multiplechoice}
\choice{\(\ln 2\)}
\choice{\(\ln 3 - \ln 2\)}
\choice{\(e^{-2}\)}
\choice[correct]{\(\displaystyle \frac{1}{2}\)}
\choice{\(\displaystyle \frac{2}{e}\)}
\choice{diverges}
\end{multiplechoice}
\begin{feedback}
We recognize the Taylor series for cosine:
\[ \cos x = \sum_{n=0}^\infty \frac{(-1)^n \pi^{2n}}{3^{2n}(2n)!} \]
The series in question is exactly
\[ \sum_{n=0}^\infty \frac{(-1)^n (\pi/3)^{2n}}{(2n)!} = \cos \frac{\pi}{3} = \frac{1}{2}. \]
\end{feedback}

\end{question}

\begin{question}%%%%%[2015C.05]

What is the limit of the sequence \(\displaystyle \left\{ n^2 \left( 1 - \cos \frac{1}{n} \right) \right\}\)?
\begin{multiplechoice}
\choice{\(1\)}
\choice{\(-1\)}
\choice{\(\displaystyle \frac{\sqrt{3}}{2}\)}
\choice[correct]{\(\displaystyle \frac{1}{2}\)}
\choice{\(\displaystyle -\frac{\sqrt{3}}{2}\)}
\choice{diverges}
\end{multiplechoice}

\end{question}

\begin{question}%%%%%[2017C.10]

Find the limit of the sequence
\[ a_n = \left\{ n \left[ \ln (n+3) - \ln n \right] \right\}. \]
\begin{multiplechoice}
\choice{\(0\)}
\choice{\(1\)}
\choice{\(\ln 3\)}
\choice[correct]{\(3\)}
\choice{\(\infty\)}
\choice{the limit does not exist}
\end{multiplechoice}

\end{question}

\end{document}


\section{Analysis of Unconstrained Stochastic Gradient Descent.}
First, let's examine the case with $P = 0$ and let's make no assumptions about strong convexity. Assume $\|G(x, \xi)\| \leq M$ for all $x$ and $\xi$. Let $x_*$ denote any optimal solution of \eqref{eq:origpro}. Then we have
\begin{eqnarray}
\mathbb{E}[\|x_{k+1}-x_*\|^2]&=&\mathbb{E}[\|x_{k}-\gamma_k G(x_k,\xi_k)-x_*\|^2]  \nonumber \\
&=&\mathbb{E}[\|x_{k}-x_*\|^2]- 2\gamma_k\mathbb{E}[\langle G(x_k,\xi_k), x_k-x_*\rangle ]+ \gamma_k^2 \mathbb{E}[\|\gamma_k G(x_k,\xi_k)\|^2] \nonumber \\
&\leq&\mathbb{E}[\|x_{k}-x_*\|^2]- 2\gamma_k\mathbb{E}[\langle G(x_k,\xi_k), x_k-x_*\rangle ]+ \gamma_k^2 M^2 \nonumber \\
&=&\mathbb{E}[\|x_{k}-x_*\|^2]- 2\gamma_k\mathbb{E}[\langle \nabla f(x_k), x_k-x_*\rangle ]+ \gamma_k^2 M^2 \label{eq:eqnproof5} \\
&\leq& \mathbb{E}[\|x_{k}-x_*\|^2]- 2\gamma_k\mathbb{E}[f(x_k)-f(x_*)]+ \gamma_k^2 M^2 \label{eq:eqnproof6}
\end{eqnarray}
\eqref{eq:eqnproof5} follows because
\begin{eqnarray}
\mathbb{E}[\langle G(x_k,\xi_k), x_k-x_*\rangle ]&=&\mathbb{E}_{\xi_0,\ldots,\xi_{k-1}}[\mathbb{E}_{\xi_k}[\langle G(x_k,\xi_k), x_k-x_*\rangle\ |\ \xi_0,\ldots,\xi_{k-1} ]] \nonumber\\
&=&\mathbb{E}_{\xi_0,\ldots,\xi_{k-1}}[\langle \nabla f(x_k), x_k-x_*\rangle\ |\ \xi_0,\ldots,\xi_{k-1} ] \\
&=&\mathbb{E}[\langle \nabla f(x_k), x_k-x_*\rangle]
\end{eqnarray}
by the law of iterated expectation. \eqref{eq:eqnproof6} is a consequence of the inequality
\begin{equation}
\langle \nabla f(x_k), x_k-x_*\rangle \geq f(x_k)-f(x_*)
\end{equation}
which holds because $f$ is convex.

Arranging the bound: by telescoping $k$ from $0$ to $n$,
\begin{equation}
2\gamma_k\mathbb{E}[f(x_k)-f(x_*)] \leq -\mathbb{E}[\|x_{k+1}-x_*\|^2-\|x_{k}-x_*\|^2] + \gamma_k^2 M^2,
\end{equation}
we have since $\|x_{n+1}-x_*\|^2\geq 0$
\begin{equation}
2\sum_{k=0}^n \gamma_k\mathbb{E}[f(x_k)-f(x_*)] \leq -\mathbb{E}[\|x_{n+1}-x_*\|^2-\|x_{0}-x_*\|^2] + M^2 \sum_{k=0}^{n} \gamma_k^2\leq D^2+M^2  \sum_{k=0}^{n} \gamma_k^2.
\end{equation}
Dividing by the sum of the $\gamma_k$, we have for any $n$
\begin{equation}
\frac{1}{\sum_{k=0}^n \gamma_k}\sum_{k=0}^n \gamma_k\mathbb{E}[f(x_k)-f(x_*)] \leq \frac{D^2+M^2  \sum_{k=0}^{n} \gamma_k^2}{2\sum_{k=0}^n \gamma_k}.
\end{equation}
where $D=\|x_0-x_*\|$. Let $\bar{x}:=(\sum_{k=0}^n \gamma_k)^{-1}\sum_{k=0}^n \gamma_k x_k$. Then, by convexity (Jensen's inequality), we have
\begin{equation}
\mathbb{E}[f(\bar{x})-f(x_*)] \leq \frac{D^2+M^2  \sum_{k=0}^{n} \gamma_k^2}{2\sum_{k=0}^n \gamma_k}.
\end{equation}
This is precisely the bound rate of convergence for deterministic subgradient descent.


\subsection{The Model}

The  model  used  by  both  Hudzovic\cite{ref:hudzovic}  and Sani\cite{ref:sani}
approximate  the  step  response of a plant by using a series  of  PT1  elements
multiplied  together with varying time constants $T_k$ to form  a  PTn  element,
$G(s)$. This is defined as:

\begin{equation}
    G_n(s,r) = y_0 + K_s \prod_{k=1}^{n} \frac{1}{1+s \cdot T_k(r)}
    \label{eq:ptn}
\end{equation}

where the scale factor, $K_s$, is defined as:

\begin{equation}
    K_s = \frac{xa(\infty)}{xeo}
\end{equation}

The  transfer function in equation \ref{eq:ptn} serves as a basis to  model  the
step response of many systems.

Rather than individually having to find the time constants  $T_1\ldots  T_n$  --
the effort of which would greatly increase with the order $n$ -- the two methods
of  P.  Hudzovic  and  L.  Sani instead calculate these constants using a common
function $T_k(r)$.

The  approach  proposed  by  P.  Hudzovic\cite{ref:hudzovic}  for  $T_k(r)$  is:

\begin{equation}
    T_k(r) = \frac{T}{1 - (k-1)r}
    \label{eq:hudzovic}
\end{equation}

where  the  constant  $r$  must  be   confined  to  the  interval  $0  \le  r  <
\frac{1}{n-1}$.

The   approach   proposed   by   L.   Sani\cite{ref:sani}   for   $T_k(r)$   is:

\begin{equation}
    T_k(r) = T \cdot r^{k-1}
    \label{eq:sani}
\end{equation}

where  the  constant $r$ must be confined to  the  interval  $0  \leq  r  <  1$.

As  can  be  seen,  in both cases, the  problem  has  been  reduced  to  finding
appropriate values for  $n$,  $T$  and  $r$  such  that  the  step  response  of
$G_n(s,r)$ approximates the data acquired from the plant as closely as possible.


\subsection{Convergence of Gradient Descent method}
Now we are ready to study the rate of convergence of unconstrained minimization schemes.
For the optimization problem \eqref{optmodel},
\begin{equation}\label{key}
\min_{x\in \mathbb{R}^n} f(x).
\end{equation}
We assume that $f(x)$ is convex.
Then we say that $x^*$ is a minimizer if
$$
f(x^*) = \min_{x \in \mathbb{R}^n} f(x).
$$
%By Theorem \ref{th:critical},
For minimizer $x^*$, we have
\begin{equation}\label{key}
\nabla f(x^*) = 0.
\end{equation}
We have the next two properties of the minimizer for convex functions:
\begin{enumerate}
	\item If $f(x) \ge c_0$, for some $c_0 \in \mathbb{R}$, then we have
	\begin{equation}\label{key}
	\mathop{\arg\min} f \neq \emptyset.
	\end{equation}
	\item If $f(x)$ is $\lambda$-strongly convex, then $f(x)$ has a unique minimizer, namely, there
	exists a unique $x^*\in \mathbb{R}^n$ such that
	\begin{equation}\label{key}
	f(x^*) = \min_{x\in \mathbb{R}^n }f(x).
	\end{equation}
\end{enumerate}

To investigate the convergence of gradient descent method, let us recall the gradient
descent method:
\begin{algorithm}
	\caption{ FGD}
	\label{alg:FGD}
	{\bf For}: $t = 1, 2, \cdots$
	\begin{equation}\label{equ:fgd-iteration}
	x_{t+1} =  x_{t} - \eta_t \nabla f(x_t),
	\end{equation}
	where $\eta_t$ is the stepsize / learning rate.
\end{algorithm}

We have the next theorem about the convergence of gradient descent method under the Assumption
\ref{ass:GD}.

\begin{theorem}
	For Gradient Descent Algorithm \ref{alg:FGD}, if $f(x)$ satisfies Assumption \ref{ass:GD}, then
	\begin{equation}
	\|x_t - x^*\|^2 \le  \alpha^t \|x_0 - x^*\|^2
	\end{equation}
	if $0<\eta_t <\frac{2\lambda}{L^2}$ and $\alpha < 1$.

	Particularly, if $\eta_t = \frac{\lambda}{L^2}$, then
	\begin{equation}
	\|x_t - x^*\|^2 \le  \left(1 - \frac{\lambda^2}{L^2}\right)^t \|x_0 - x^*\|^2.
	\end{equation}
\end{theorem}

\begin{proof}
	Note that
	\begin{equation}
	x_{t+1} - x =  x_{t} - \eta_t \nabla f(x_t)  - x.
	\end{equation}
	By taking $L^2$ norm for both sides, we get
	\begin{equation}
	\|x_{t+1} - x \|^2 = \|x_{t} - \eta_t \nabla f(x_t) - x \|^2.
	\end{equation}
	Let $x = x^*$. It holds that
	\begin{equation}
	\begin{aligned}
	\|x_{t+1} - x^* \|^2 &=  \| x_{t} - \eta_t \nabla f(x_t) - x^* \|^2 \\
	&= \|x_t-x^*\|^2 - 2\eta_t \nabla f(x_t)^\top (x_t - x^*) + \eta_t^2 \|\nabla f(x_t) - \nabla f(x^*)\|^2 \qquad \mbox{\scriptsize (by $\nabla f(x^*)=0$)}\\
	&\le \|x_t - x^*\|^2 - 2\eta_t \lambda \|x_t - x^*\|^2 + \eta_t ^2 L^2 \|x_t - x^*\|^2  \quad \mbox{\scriptsize (by $\lambda$- strongly convex \eqref{strongConvIneq} and Lipschitz)}\\
	&\le (1 - 2\eta_t \lambda + \eta_t^2 L^2) \|x_t - x^*\|^2
	=\alpha \|x_t - x^*\|^2,
	\end{aligned}
	\end{equation}
	where
	$$
	\alpha = \left(L^2 (\eta_t  -{\lambda\over L^2})^2 + 1-{\lambda^2\over L^2}\right)<1\  \mbox{if } 0< \eta_t<\frac{2\lambda}{L^2}.
	$$
	Particularly, if $\eta_t =\frac{\lambda}{L^2}$,
	$$
	\alpha=1-{\lambda^2\over L^2},
	$$
	which finishes the proof.
\end{proof}
This means that if the learning rate is chosen appropriatly, $\{x_t\}_{t=1}^\infty$ from the gradient descent method will converge to the minimizer $x^*$ of the function.

There are some issues on Gradient Descent method:
\begin{itemize}
\item $\nabla f(x_{t})$ is very expensive to compute.
\item Gradient Descent method does not yield generalization accuracy.
\end{itemize}
The stochastic gradient descent (SGD) method in the next section will focus on these two issues.


\subsection{Unitary groups \(U(n, F)\)}

\subsubsection{Metric preserving transformations for sesquilinear forms}

For bilinear forms, the transformations which preserved metrics were:

\(P^T=P^{-1}\)

For sesquilinear they are different:

\(u^*Mv\)

\((Pu)^*M(Pv)\)

\(u^*P^*MPv\)

So we want the matrices where:

\(P^*MP=M\)

\subsubsection{The unitary group}

The unitary group is where \(M=I\)

\(P^*P=I\)

\(P^*=P^{-1}\)

We refer to these using \(U\) instead of \(P\).

\(U^*=U^{-1}\)

\subsubsection{Parameters of the unitary group}

The unitary group depends on the dimension of the vector space, and the underlying field. So we can have:

\begin{itemize}
\item \(U(n, R)\); and
\item \(U(n, C)\).
\end{itemize}

\subsubsection{We generally refer only to the complex}

For the \(U(n, R)\) we have:

\(U^*=U^{-1}\)

\(U^T=U^{-1}\)

This is the condition for the orthogonal group, and so we would instead write \(O(n)\).

As a result, \(U(n)\) refers to \(U(n,C)\).
\subsubsection{\(U(1)\): The circle group}