\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{bm}
\usepackage[margin=1in]{geometry}
\usepackage{parskip}
\usepackage{algorithm}
\usepackage{algpseudocode}

\usepackage[backend=biber,style=numeric,sorting=ynt,bibencoding=utf8]{biblatex}
\addbibresource{sources.bib}

\renewcommand\aa{\bm{\theta}}
\renewcommand\t{\vartheta}
\newcommand\R{\bm{R}}
\newcommand\q{\bm{q}}
\newcommand\I{\bm{I}}
\renewcommand\u{\bm{u}}
\renewcommand\v{\bm{v}}
\renewcommand\skew[1]{[#1]_{\times}}
\newcommand\Real{\mathbb{R}}
\newcommand\norm[1]{\Vert#1\Vert}
\newcommand\vecop[1]{\text{vec} \left( #1\right)}

\DeclareMathOperator{\arctantwo}{arctan2}

\begin{document}

\title{Overview of 3D orientation transformations with Jacobians}
\author{Christian V. Berg}
\maketitle
\date

Many of the following transformations are functions of matricies. Since we are
also interested in their first order derivative, we need an efficient way
of ordering their partial derivates. In the following derivations, we use the
\textit{Magnus-Neudecker} layout convention as presented in
\cite{magnus_matrix_1985}. A practical introduction can be found in
\cite{fackler_notes_2005}. We will be using the product rule in particular: Given
functions $f(X):\Real^{n \times m} \to \Real^{p \times q}$ and
$g(X):\Real^{n \times m} \to \Real^{q \times k}$ then
\begin{align}
  \frac{\partial \vecop{f(X) g(X)}}{\partial \vecop{X}} =
    \left(g(X)^T \otimes \bm{I}_k \right) \frac{\partial \vecop{f(X)}}{\partial \vecop{X}}
    + \left(\bm{I}_p \otimes f(X) \right) \frac{\partial \vecop{g(X)}}{\partial \vecop{X}}
\end{align}

where $\otimes$ is the Kronecker product. You will also find some references to the Lie group $SO(3)$ and its exponential map.
Though Lie theory is not needed to understand what follows, an excellent introduction
to the topic and its applications in robotics can be found in \cite{sola_micro_2019}.

\tableofcontents

\section{Notations}
\subsection{Symbols}
\begin{tabular}{l p{15cm}}
  $\t \in \Real$ & Angle \\
  $\aa \in \Real^3$ & Axis-angle such that $\aa = \t \, \u$, $\t \in \Real$ and a unit vector $\u \in S^2$ \\
  $\q \in S^3$ & Unit quaternion such that $\q = \begin{bmatrix} w \\ \v \end{bmatrix}$ \\
  $\R \in SO(3)$ & Rotation matrix \\
  $\I \in \Real^{3\times3}$ & Identity matrix \\
  $\bm{0} \in \Real^{3}$ & Zero vector \\
  $\bm{1} \in \Real^{3}$ & Vector of ones \\
   $e_i \in \Real^3$ & $i^{th}$ standard orthonormal base vector of $\Real^3$, i.e.
     $e_1=\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix},
     e_2 =\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix},
       e_3 =\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$ \\ [10pt] \\
   $E_i = \skew{e_i}\in \Real^{3 \times 3}$ & $i^{th}$ generator of $SO(3)$ \\ [10pt] \\
\end{tabular}

\subsection{Operators}
\begin{tabular}{l p{15cm}}
  $ (.)^T : \Real^{m \times n} \to \Real^{n \times m}$ & Matrix transpose \\
  $tr(.) : \Real^{n \times n} \to \Real$ & Matrix trace \\
  $\skew{.} : \Real^3 \to  \Real^{3\times 3}$ & Skew symmetric matrix operator such that
  $\begin{bmatrix} x \\ y \\ z \end{bmatrix}_\times =
  \begin{bmatrix}
    0& - z & y \\
    z & 0 & -x \\
    -y & x & 0
  \end{bmatrix}$ \\ [10pt] \\
  $(.)^\vee : \Real^{3 \times 3} \to \Real^{3}$ & Vee operator such that
  $\begin{bmatrix}
    a_1& - z & y \\
    z & a_2 & -x \\
    -y & x & a_3
  \end{bmatrix}^\vee = \begin{bmatrix} x \\ y \\ z \end{bmatrix}$ \newline Notice that $\left(\skew{\bm{d}}\right)^\vee = \bm{d}$ \\
    $\vecop{.} : \Real^{m \times n} \to \Real^{m\,n}$ & Vectorization operator of a matrix such that
   $\vecop{\begin{bmatrix}
    a_1 & a_3 \\
    a_2 & a_4
    \end{bmatrix}} = \begin{bmatrix} a_1 \\ a_2 \\ a_3 \\ a_4 \end{bmatrix}$ \\ [10pt] \\
\end{tabular}

\section{Rotation matrix and axis-angle}
\subsection{Rotation matrix from axis-angle}
This transformation is provided by Rodrigues' rotation formula, and gives a closed form expression of the exponential map of $SO(3)$:
\begin{align}
  \R = \exp(\skew{\aa}) = \I + \frac{\sin \t}{\t} \skew{\aa} + \frac{1 - \cos \t}{\t^2} \skew{\aa}^2
\end{align}

First order Taylor approximation at $\aa = \bm{0}$
\begin{align}
  \R = \exp(\skew{\aa}) = \I + \skew{\aa} + O(\skew{\aa}^2) \approx \I + \skew{\aa}
\end{align}

\subsection{Rotation matrix from axis-angle Jacobian}
Since the expression is fairly long, we will define 2 intermediate functions:
\begin{align}
  k_1(\t) = \frac{\sin{\t}}{\t} \;\;&,\;\;
  \frac{\partial k_1(\t)}{\partial \t} = \frac{\t\cos{\t} - \sin{\t}}{\t^2} \\
  k_2(\t) = \frac{1-\cos{\t}}{\t^2} \;\;&,\;\;
  \frac{\partial k_2(\t)}{\partial \t} = \frac{\t\sin{\t} - 2\left(1-\cos{\t} \right)}{\t^3}
\end{align}

For both functions, we have
\begin{align}
  \frac{\partial k_i(\t)}{\partial \aa} &=
    \frac{\partial k_i(\t)}{\partial \t} \frac{\t}{\partial \aa} =
    \frac{\partial k_i(\t)}{\partial \t} \frac{\sqrt{\aa^T\aa}}{\partial \aa} =
    \frac{\partial k_i(\t)}{\partial \t} \frac{\aa^T}{\t}
\end{align}

Using the Kronecker product $\otimes$, it then follows
\begin{align}
  \frac{\partial \vecop{k_1(\t) \skew{\aa}}}{\partial \aa}
  &= \left( \I \otimes k_1(\t)\I \right) \frac{\partial \vecop{\skew{\aa}}}{\partial \aa}
    + \left(\skew{\aa}^T \otimes \I \right) \frac{\partial \vecop{k_1(\t)\;\I}}{\partial \aa} \\
  &= k_1(\t) \frac{\partial \vecop{\skew{\aa}}}{\partial \aa}
    - \left(\skew{\aa} \otimes \I \right) \vecop{\I} \frac{\partial k_1(\t)}{\partial \t} \frac{\aa^T}{\t}
\end{align}

where we used $\skew{.}^T = - \skew{.}$. Next, we also have
\begin{align}
  \frac{\partial \vecop{k_2(\t) \skew{\aa}^2}}{\partial \aa}
  &= \left( \I \otimes k_2(\t)\I \right) \frac{\partial \vecop{\skew{\aa}^2}}{\partial \aa}
    + \left({\skew{\aa}^2}^T \otimes \I \right) \frac{\partial \vecop{k_2(\t)\;\I}}{\partial \aa} \\
  &= k_2(\t) \frac{\partial \vecop{\skew{\aa}^2}}{\partial \aa}
    + \left({\skew{\aa}^2} \otimes \I \right) \vecop{\I} \frac{\partial k_2(\t)}{\partial \t} \frac{\aa^T}{\t}
\end{align}

where we used ${\skew{.}^2}^T = \skew{.}^2$. Putting it all together, we get:

\begin{align}
  \frac{\partial \vecop{\R}}{\partial \aa} =
    \frac{\partial \vecop{k_1(\t) \skew{\aa}}}{\partial \aa}
    + \frac{\partial \vecop{k_2(\t) \skew{\aa}^2}}{\partial \aa}
\end{align}

When $\t = 0$, we reuse the exponential map of $SO(3)$ to deduce that
\begin{align}
  \left. \frac{\partial \vecop{\R}}{\partial \aa} \right \vert_{\aa = \bm{0}} =
  \left. \frac{\partial \vecop{\exp(\skew{\aa})}}{\partial \aa_i} \right \vert_{\aa = \bm{0}} \approx
  \left. \frac{\partial \vecop{\I + \skew{\aa}}}{\partial \aa_i} \right \vert_{\aa = \bm{0}} =
  \frac{\partial \vecop{\skew{\aa}}}{\partial \aa}
\end{align}

\subsection{Axis-angle from rotation matrix}
We begin by retrieving the angle $\t$:
\begin{align}
  tr(\R)
    &= tr(\I) + \frac{\sin \t }{\t} tr(\skew{\aa}) + \frac{1 - \cos \t}{\t^2} tr(\skew{\aa}^2) \\
    &= 3 - 2 (1 - \cos \t) \\
  \t &=  \arccos \frac{tr(\R) - 1}{2}
\end{align}

where we used $tr(\skew{\aa}^2) =  - 2 \aa^T \aa = -2\t^2$ and $tr(\skew{.}) = 0$.

Next, and assuming $\t \neq 0 \iff tr(\R) \neq 3$ and $\t \neq \pi \iff tr(\R) \neq -1$ then
\begin{align}
  \R - \R^T
    &= \frac{\sin{\t}}{\t} \left(\skew{\aa} - \skew{\aa}^T\right)
    + \frac{1 - \cos \t}{\t^2} \left( \skew{\aa}^2 - \skew{\aa}^{2^T} \right) \\
    &= \frac{2\, \sin \t}{\t} \skew{\aa} \\
  \aa &= \frac{\t \, \left(\R - \R^T \right)^\vee }{2\, \sin \t}
\end{align}

When $\t$ is close to $0 \iff tr(\R)$ close to $3$, we can reuse the first
order Taylor approximation from before:

\begin{align}
  \aa \approx \left( \R - \I \right)^\vee
\end{align}

When $\t = \pi \iff tr(\R) = -1$
\begin{align}
  \R = \I + \frac{2}{\t^2} \skew{\aa}^2
  &= \I + 2\skew{\u}^2 \\
  &= \I + 2\, (\u \u^T - \I) \\
  \u \u^T = \frac{\R + \I }{2}
\end{align}

where we used $\skew{\u}^2 =  \u \u^T - \u^T \u \I$ and $\u^T \u = 1$ by definition.
Since $\u \u^T$ has the form:
\begin{align}
  \u \u^T =
  \begin{bmatrix}
    \u_x^2 & \u_x \u_y & \u_x \u_z \\
    \u_y \u_x & \u_y^2 & \u_y \u_z \\
    \u_z \u_x & \u_z \u_y & \u_z^2
  \end{bmatrix}
\end{align}

we know that the largest element of the diagonal corresponds to the most
significant axis of rotation. Since $\t=\pi$ then the largest diagonal element
is always different from zero. Therefore, we take the square root of the largest
diagonal element, then use it to divide its corresponding column vector in
order to retrieve $\u$. Note that we cannot deduce the sign of $\u$, only the
relative "sign-ness" of each element.

\begin{align}
  \u = \pm
    \frac{col_i \left[ \frac{\R + \I}{2} \right] }
         {\sqrt{diag_i \left[ \frac{\R + \I }{2} \right] }}
  \;,\; i = argmax \;\; diag \left( \frac{\R + \I }{2} \right)
\end{align}

Finally
\begin{align}
  \aa = \pm \pi \u
\end{align}

\subsection{Axis-angle from rotation matrix Jacobian}
Once again we will define an intermediate function and variable:
\begin{align}
  k_3(\t) = \frac{\t}{2\sin{\t}} \;\;&,\;\;
  \frac{\partial k_3(\t)}{\partial \t} =
    \left(1 - \frac{\t}{\tan{\t}} \right) \frac{1}{2\sin{\t}} \\
  x = \frac{tr(\R) - 1}{2} \;\;&,\;\; \t = \arccos(x)
\end{align}

Consequently, we have
\begin{align}
  \frac{\partial k_3(\t)}{\partial tr(\R)}
  &=\frac{\partial k_3(\t)}{\partial \t}
    \frac{\partial \t}{\partial x}
    \frac{\partial x}{\partial tr(\R)} \\
  &=-\frac{\partial k_3(\t)}{\partial \t}
    \frac{1}{2\sqrt{1-\t^2}}
\end{align}

Assuming $\t \neq 0 \iff tr(\R) \neq 3$ and $\t \neq \pi \iff tr(\R) \neq -1$, then
\begin{align}
  \frac{\partial \vecop{\aa}}{\partial \vecop{\R}} =
    \big({(\R - \R^T)^\vee}^T \otimes \I \big)
    \vecop{\I}
    \frac{\partial k_3(\t)}{\partial tr(\R)}
    \frac{\partial tr(\R)}{\partial \vecop{\R}} +
    k_3(\t) \frac{\partial \left( \R - \R^T \right)^\vee}{\partial \vecop{\R}}
\end{align}

When $\t = 0 \iff tr(\R) = 3$, then
\begin{align}
  \left. \frac{\partial \aa}{\partial \vecop{\R}} \right \vert_{\aa = \bm{0}} \approx
    \frac{\partial \left(\R - \I \right)^\vee}{\partial \vecop{\R}}
\end{align}

\section{Rotation matrix and quaternion}
\subsection{Rotation matrix from quaternion}
Assuming unit quaternions, we have
\begin{align}
  \R = \I + 2 \; w \; \skew{\v} + 2\; \skew{\v}^2
\end{align}

\subsection{Rotation matrix from quaternion Jacobian}
\begin{align}
  \frac{\partial \vecop{\R}}{\partial w} &= 2\, \vecop{\skew{\v}} \\
  \frac{\partial \vecop{\R}}{\partial \v} &=
  2w \frac{\partial\vecop{\skew{\v}}}{\partial \v} +
  2 \frac{\partial\vecop{\skew{\v}^2}}{\partial \v} \\
  \frac{\partial \vecop{\R}}{\partial \q} &=
    \begin{bmatrix}
      \frac{\partial \vecop{\R}}{\partial w} &
      \frac{\partial \vecop{\R}}{\partial \v}
    \end{bmatrix}
\end{align}

\subsection{Quaternion from rotation matrix}
We begin by retrieving the scale component of the quaternion:
\begin{align}
  tr(\R)
    &= tr(\I) + 2\; w \; tr(\skew{\v}) + 2 \; tr(\skew{\v}^2) \\
    &= 3 - 4 \v^t \v = 4\;w^2 - 1 \\
  w &= \pm \frac{ \sqrt{ tr(\R) + 1} }{2}
\end{align}

Assuming $w \neq 0 \iff tr(\R) \neq -1$
\begin{align}
  \R - \R^T
    &= 2\; w \; \left( \skew{\v} - \skew{\v}^T \right) + 2 \; \left(\skew{\v}^2 - {\skew{\v}^2}^T \right) \\
    &= 4\; w \; \skew{\v} \\
  \v &= \frac{\left( \R -\R^T \right)^\vee}{4\;w}
\end{align}

When $w = 0$, then
\begin{align}
  \R &= \I + 2 \skew{\v}^2  = \I + 2 \left ( \v \v^T - \I \right) \\
  \v \v^T &= \frac{\R + \I}{2}
\end{align}

and we can reuse the same method as for axis-angle when $\t = \pi$.

\subsection{Quaternion from rotation matrix Jacobian}
As before, we define
\begin{align}
  k_4(w) = \frac{1}{4w} \;\;,\;\; \frac{\partial k_4(w)}{\partial w} = -\frac{1}{4w^2}
\end{align}

Assuming $w \neq 0 \iff tr(\R) \neq -1$
\begin{align}
  \frac{\partial w}{\partial \vecop{\R}}
    &=\frac{\partial w}{\partial tr(\R)} \frac{\partial tr(\R)}{\partial \vecop{\R}}
    = \frac{1}{4\sqrt{tr(\R) + 1}} \frac{\partial tr(\R)}{\partial \R} \\
  \frac{\partial \v}{\partial \vecop{\R}}
    &= \big({\left(\R - \R^T\right)^\vee}^T \otimes \I \big)\frac{\partial k_4(w)}{\partial w}\frac{\partial w}{\partial \vecop{\R}}
    + k_4(w) \frac{\partial \left(\R - \R^T\right)^\vee }{\partial \vecop{\R}}
\end{align}

\section{Axis-angle and quaternion}
\subsection{Quaternion from axis-angle}
\begin{align}
  \q =
    \begin{bmatrix}
       \cos{ \frac{\t}{2} } \\[5pt]
       \frac{\sin{ \frac{\t}{2} }}{\t} \aa
    \end{bmatrix}
\end{align}

When $\t$ is close to $0$, then the second order Taylor approximation is
\begin{align}
  \q \approx
    \begin{bmatrix}
      1 - \frac{\t^2}{8} \\[5pt]
      \left( \frac{1}{2} - \frac{\t^2}{48} \right) \aa
    \end{bmatrix}
\end{align}

\subsection{Quaternion from axis-angle Jacobian}
\begin{align}
  \frac{\partial w}{\partial \aa}
  &=  \frac{\partial w}{\partial \t} \frac{\partial \t}{\partial \aa}  \\
  &=-\frac{\sin{\frac{\t}{2}}}{2} \frac{\aa^T}{\t}  \\
     \frac{\partial \v}{\partial \aa} &=
  \aa \frac{\partial \frac{\sin{ \frac{\t}{2} }}{\t}}{\partial \aa} +
  \frac{\sin{ \frac{\t}{2} }}{\t} \frac{\partial \aa}{\partial \aa}\\
  &= \aa \frac{\partial \frac{\sin{ \frac{\t}{2} }}{\t}}{\partial \t} \frac{\partial \t}{\partial \aa}  +
  \frac{\sin{ \frac{\t}{2} }}{\t} \I \\
  &=
     \aa \frac{\t \, \cos{\frac{\t}{2}} - 2\,\sin{\frac{\t}{2}}}{2\,\t^2} \frac{\aa^T}{\t}
    + \frac{\sin{\frac{\t}{2}}}{\t} \I
\end{align}

When $\t$ is close to $0$, then
\begin{align}
  \left. \frac{\partial w}{\partial \aa} \right \vert_{\aa = 0} &\approx - \frac{1}{4} \aa^T \\
  \left. \frac{\partial w}{\partial \aa} \right \vert_{\aa = 0} &\approx - \frac{1}{24} \aa \aa^T + \left(\frac{1}{2} - \frac{\t^2}{48} \right) \I
\end{align}

and finally

\begin{align}
  \frac{\partial \q}{\partial \aa} =
    \begin{bmatrix}
      \frac{\partial w}{\partial \aa} \\[5pt]
      \frac{\partial \v}{\partial \aa}
    \end{bmatrix}
\end{align}

\subsection{Axis-angle from quaternion}
Using $y=\sqrt{\v^T\v}$
\begin{align}
  \aa = 2 \frac{\arctantwo \left(y, w \right)}{y} \v
\end{align}

When $w$ approaches $1$ from the left $\iff \t$ close to $0$, we can reuse the second order Taylor approximation from before to deduce
\begin{align}
  \aa \approx \frac{6\v}{w+2}
\end{align}

\subsection{Axis-angle from quaternion Jacobian}
We define an intermediate function
\begin{align}
  k_5(y,x) &= 2 \frac{\arctantwo \left(y, w \right)}{y} \\
  \frac{\partial k_5(y,x)}{\partial x} &= \frac{-2}{y^2 + w^2} = -2 \\
  \frac{\partial k_5(y,x)}{\partial y} &= 2 \frac{ \frac{w\,y}{y^2 + w^2} - \arctantwo \left(y, w \right)}{y^2} = 2 \frac{ w\,y - \arctantwo \left(y, w \right)}{y^2}
\end{align}
where we used $y^2 + w^2 = \q^T \q = 1$ since we are assuming unit quaternions. It then follows:
\begin{align}
  \frac{\partial \aa}{\partial w} &= \frac{\partial k_5(y,w)}{\partial w} \v \\
  \frac{\partial \aa}{\partial \v}
  &= \v \frac{\partial k_5(y,w)}{\partial y} \frac{\partial y}{\partial \v} + k_5(y,w) \I
  =  \v \frac{\partial k_5(y,w)}{\partial y} \frac{\v^T}{y} + k_5(y,w) \I
\end{align}

When $w$ approach $1$ from the left $\iff \t$ close to $0$, then
\begin{align}
  \left. \frac{\partial \aa}{\partial w} \right \vert_{w = 1^-} &\approx - \frac{6}{\left( w + 2 \right)^2} \v \\
  \left. \frac{\partial \aa}{\partial \v} \right \vert_{w = 1^-} &\approx \frac{6}{w + 2} \I
\end{align}

and finally

\begin{align}
  \frac{\partial \aa}{\partial \q} =
    \begin{bmatrix}
      \frac{\partial \aa}{\partial w} &
      \frac{\partial \aa}{\partial \v}
    \end{bmatrix}
\end{align}

\section{Euler angles and rotation matrix}

This section is largely based on \cite{eberly_euler_nodate}, which details how
one can retrieve the Euler angles of a given sequence of rotation from a rotation matrix.

The following subsection presents the pseudocode to retrieve the wanted Euler angles, but
unlike \cite{eberly_euler_nodate}, the pseudocode is generalized for the intrinsic order of rotation.

We define

\begin{tabular}{l p{15cm}}
  $\R_i(\t_k)$ & The rotation matrix about the axis $e_i$ by an angle $\t_k$ \\
  $s_k$ & Sine of the angle $\t_k$, i.e. $s_k = \sin \t_k$\\
  $c_k$ & Cosine of the angle $\t_k$, i.e. $c_k = \cos \t_k$\\
  $M$ & The "sign" matrix used to correct the sign of the sine and cosine element of the rotation matrix \\
\end{tabular}

Using this, a rotation matrix can be written as $\R = \R_a(\t_1) \; \R_b(\t_2) \; \R_c(\t_3)$.
For example, the intrinsic sequences of rotation \textit{x-y-z} by angles $(\t_x, \t_y, \t_z)$ is written
$\R = \R_1(\t_x) \; \R_2(\t_y) \; \R_3(\t_z)$.

A particular matrix element is denoted by a lower case letter and the indicies in subscript, i.e.
\begin{align}
  \R = \begin{bmatrix}
    r_{1,1} & r_{1,2} & r_{1,3} \\
    r_{2,1} & r_{2,2} & r_{2,3} \\
    r_{3,1} & r_{3,2} & r_{3,3}
  \end{bmatrix} \text{       }
\end{align}

The both algorithms rely on a sign matrix given by
\begin{align}
  M = \begin{bmatrix}
    m_{1,1} & m_{1,2} & m_{1,3} \\
    m_{2,1} & m_{2,2} & m_{2,3} \\
    m_{3,1} & m_{3,2} & m_{3,3}
  \end{bmatrix}
  = \begin{bmatrix}
    1 & -1 & 1 \\
    1 & 1 & -1 \\
    -1 & 1 & 1
  \end{bmatrix} =
  \I + \skew{\bm{1}}
\end{align}

\subsection{Tait-Bryan angles from rotation matrix}

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Function{ToTaitBryan}{$\R,a,b,c$}\Comment{Corresponds to the intrinsic rotation sequence $e_a-e_b-e_c$}
  \State $M = \I + \skew{\bm{1}}$\Comment{Sign matrix}
  \State $s_2 = r_{a,c}$
  \If{$s_2 < +1$}
    \If{$s_2 > -1$}
      \State $\t_2 = \arcsin(m_{a,c} \; s2)$
      \State $\t_1 = \arctantwo(m_{b,c} \; r_{b,c}, m_{c,c} \; r_{c,c})$
      \State $\t_3 = \arctantwo(m_{a,b} \; r_{a,b}, m_{a,a} \; r_{a,a})$
    \Else\Comment{Gimbal lock: $s_2 = -1$}
      \State $\t_2 = - m_{a,c} \; \frac{\pi}{2}$
      \State $\t_1 = \arctantwo(m_{b,a} \; r_{b,a}, m_{b,b} \; r_{b,b})$
      \State $\t_3 = 0$
    \EndIf
  \Else\Comment{Gimbal lock: $s_2 = 1$}
    \State $\t_2 = m_{a,c} \; \frac{\pi}{2}$
    \State $\t_1 = \arctantwo(m_{b,a} \; r_{b,a}, m_{b,b} \; r_{b,b})$
    \State $\t_3 = 0$
  \EndIf
  \State \textbf{return} $\t_1, \t_2, \t_3$
\EndFunction
\end{algorithmic}
\end{algorithm}

\subsection{Proper Euler angles from rotation matrix}

\begin{algorithm}[H]
\begin{algorithmic}[1]
  \Function{ToProperEuler}{$\R,a,b$}\Comment{Corresponds to the intrinsic rotation sequence $e_a-e_b-e_a'$}
  \State $M = \I + \skew{\bm{1}}$\Comment{Sign matrix}
  \State $c = 6 - a -b$\Comment{$c$ is the "unused" axis}
  \State $c_2 = r_{a,a}$
  \If{$c_2 < +1$}
    \If{$c_2 > -1$}
      \State $\t_2 = \arccos(m_{a,a} \; c2)$
      \State $\t_1 = \arctantwo(m_{b,a} \; r_{b,a}, m_{c,a} \; r_{c,a})$
      \State $\t_3 = \arctantwo(m_{a,b} \; r_{a,b}, m_{a,c} \; r_{a,c})$
    \Else\Comment{Gimbal lock: $c_2 = -1$}
      \State $\t_2 = \pi$
      \State $\t_1 = - \arctantwo(m_{b,c} \; r_{b,c}, m_{b,b} \; r_{b,b})$
      \State $\t_3 = 0$
    \EndIf
  \Else\Comment{Gimbal lock: $c_2 = 1$}
    \State $\t_2 = 0$
    \State $\t_1 = \arctantwo(m_{b,c} \; r_{b,c}, m_{b,b} \; r_{b,b})$
    \State $\t_3 = 0$
  \EndIf
  \State \textbf{return} $\t_1, \t_2, \t_3$
\EndFunction
\end{algorithmic}
\end{algorithm}

\newpage

\section{Common matrix derivatives}

Using $e'_i$ as $i^{th}$ standard orthonormal base vector of $\Real^9$, then

\begin{align}
  \forall X \in \Real^{3 \times 3} \;\; \text{ then } \;\; \frac{\partial \vecop{X^T}}{\partial \vecop{X}}
  = \begin{bmatrix} e'_1 &   e'_4 &   e'_7 &   e'_2 &   e'_5 &   e'_8 &   e'_3 &   e'_6 &   e'_9  \end{bmatrix} = J_T
\end{align}

\begin{align}
  \forall X \in \Real^{n \times n} \;\; \text{ then } \;\; \frac{\partial \vecop{tr(X)}}{\partial \vecop{X}}
  = \vecop{\I_n}
\end{align}

\begin{align}
  \forall \bm{v} \in \Real^{3} \;\; \text{ then } \;\; \frac{\partial \skew{\bm{v}}}{\partial \bm{v}}
  = \begin{bmatrix} \vecop{E_1} &  \vecop{E_2} & \vecop{E_3} \end{bmatrix} = J_\times
\end{align}

\begin{align}
  \forall X \in \Real^{3 \times 3} \;\; \text{ then } \;\; \frac{\partial X^\vee}{\partial \vecop{X}} =
  \begin{bmatrix}
    0 & 0 & 0 & 0 & 0 & 0 & 0 & -1 & 0 \\
    0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\
    0 & 0 & 0 & 0 & -1 & 0 & 0 & 0 & 0
  \end{bmatrix} = J_\vee
\end{align}

\begin{align}
  \forall X \in \Real^{3 \times 3} \;\; \text{ then } \;\; \frac{\partial (X - X^T)^\vee}{\partial \vecop{X}}
  = \frac{\partial (X - X^T)^\vee}{\partial \vecop{X - X^T}}
  \frac{\partial \vecop{X - X^T}}{\partial \vecop{X}} = J_\vee \; (\I_9 - J_T)
\end{align}

\printbibliography

\end{document}


%!TEX root = index.tex
\section{Symmetries \& Polynomials}

\epigraph{Out of nothing I have created a strange new universe.}{Janos Bolyai}

We need one last definition, that of a normal subgroup, to understand the Galois' ideas using the language of group theory.

\subsection{Cycle decomposition and Normality}
Every permutation naturally breaks up as cycles. For example $ (3 \: 4 \: 1 \: 5 \: 2) \in S_5$ breaks up as,
\begin{align*}
  (3 \: 4 \: 1 \: 5 \: 2) \rightsquigarrow 1 \mapsto 3 \mapsto 1 \mbox{ and } 2 \mapsto 4 \mapsto 5 \mapsto 2
\end{align*}
We say that $ (3 \: 4 \: 1 \: 5 \: 2)$ has a \textbf{cycle decomposition} $[1 \: 3] [2 \: 4 \: 5]$ and \textbf{cycle type} $2 + 3$. The order of the cycles and of the elements within the cycles is not relevant. We can rewrite the above permutation as having a cycle decomposition $[4 \: 5 \: 2][1 \: 3] $ and cycle type $3 + 2$.

{The cycle decomposition is the more common way of writing permutations, but it is harder to multiply two permutations when they're written in the cycle notation and needs some getting used to.}

\begin{questions}
  \item Determine cycle decompositions and cycle types of all the elements of $ S_3$.
  \item
  \label{ques:s4}
  What are the possible cycle types for elements in $ S_4$? How many elements are there in each cycle type (this is a long problem, patience is the key here).
\end{questions}

\noindent A subgroup $ G \subseteq S_n$ is called \textbf{normal} if it satisfies the following property:

if it contains one element of a certain cycle type then it contains \emph{all} the elements of that cycle type.

\begin{questions}[resume]
  \item Determine which of the subgroups of $ S_3$ are normal.\footnote{Just so that we're all on the same page $ S_3$ has 6 subgroups: $ \{ e \}$, $ \{ e, (1 \: 3 \: 2) \}$, $ \{ e,  (3 \: 2 \: 1) \}$, $ \{ e, (2 \: 1 \: 3) \}$, $ \{ e,  (2 \: 3 \: 1),  (3 \: 1 \: 2) \}$, and $ S_3$ itself.}

  \item
  \begin{enumerate}
    \item Find the cycle type of $ e \in S_n$.
    \item What other elements of $ S_n$ have the same cycle type as $ e$?
    \item Argue that the trivial group is a normal subgroup of $ S_n$. (Also note that $ S_n$ itself is a normal subgroup.)
  \end{enumerate}
\end{questions}

\newpage
\subsection{Normal subgroups of $ S_n$}
Suppose a permutation $ \sigma \in S_n$ has a cycle type $ a_1 + a_2 + \cdots + a_k$ then we say that $ \sigma$ is an \textbf{even permutation} if $ (a_1 - 1) + (a_2 - 1) + \cdots + (a_k - 1)$ is even, \textbf{odd} otherwise. This is also called the \textbf{parity} of the permutation.

For example, $ \sigma = (3 \: 4 \: 1 \: 5 \: 2)$ has cycle type $ 2 + 3$ and $ (2 - 1) + (3 - 1) = 1 + 2 = 3$ is odd, hence the parity of $(3 \: 4 \: 1 \: 5 \: 2)$ is an odd permutation.

\begin{questions}[resume]
  \item
  \begin{enumerate}
    \item Determine which permutations of $ S_3$ are even and which ones are odd.
    \item Determine which cycle types of $ S_4$ correspond to even permutations and which ones to odd permutations, hence count the number of even and odd permutations in $ S_4$.
    \item What is the parity of the identity $ e = (1 \: 2 \: \cdots \: n) \in S_n$?
  \end{enumerate}
\end{questions}
As it turns out the subset containing all the even permutations forms a normal subgroup of $ S_n$, denoted $A_n$, called the \textbf{alternating group} and has size $ n!/2$. (We'll assume this fact.)

\begin{questions}[resume]
  \item
  \begin{enumerate}[resume]
      \item What is the subgroup $ A_3$?
      \item What is the subgroup $ A_4$? Have you encountered this subgroup before?
  \end{enumerate}
\end{questions}

$ A_n$ is the only true friend $ S_n$ has.
\begin{thm}
  Every $ S_n$, for $ n>4$, has exactly 3 normal subgroups $ \{e \}$, $ A_n$ and $ S_n$
\end{thm}
There is no deep reason why this theorem is true, it is simply a matter a computing the normal subgroups carefully. This is a recurring phenomenon in group theory, seemingly elementary mathematical objects have very structured symmetric groups, and these then give rise to very beautiful and deep mathematics.

\newpage
\subsection{Normal subgroups of $ S_4$}
By direct computations we can find all the normal subgroups of $ S_4$. We'll need the following theorem about subgroups.
\begin{thm}
  The size of a subgroup divides the size of the total group.
\end{thm}

\begin{questions}[resume]
  \item \begin{enumerate}
    \item Verify the above theorem for $ S_3$.
    \item Verify the above theorem for $ \{ e \}, A_4, S_4$, as subgroups of $ S_4$.
    \item Verify the above theorem of the dihedral groups $ D_{2n}$ which are subgroups of $ S_n$.
  \end{enumerate}
\end{questions}

We already know 3 subgroups of $ S_4$: $ \{ e \}, A_4, S_4$. Turns out there is exactly one more called the \textbf{Klein 4-group}, denoted $ K_4$.
\begin{questions}[resume]
  \item \begin{enumerate}
    \item What are the possible sizes of subgroups of $ S_4$?
    \item Show that $ S_4$ has exactly 4 normal subgroups: $ \{ e \}, K_4, A_4, S_4$.

     (You'll need to use the number of elements in each cycle type of $ S_4$ that you've already computed in \ref{ques:s4} Remember that if a normal subgroup contains one element of a certain cycle type then it must contain all the elements with that cycle type.)
  \end{enumerate}
\end{questions}

This is the fortunate accident that allows us to solve the quartic. This does not occur for any other $ S_n$.

\newpage
\subsection{The Cubic and the Quartic}
We'll now return to polynomials and understand our methods using the language of symmetric groups.

The symmetry groups $ S_n$ naturally acts on a set of $ n$ variables, but more importantly it also acts on the set of all polynomials in $ n$ variables. For example, the permutation $ (1 \: 3 \: 2)$ sends $ \beta_1 \beta_2^2 +  \beta_3 $ to $ \beta_1 \beta_3^2 +  \beta_2 $. Let us go back to the cubic and quartic and analyze them using this new language.\\

Recall that we had the following intermediate variables for the cubic
  \begin{align*}
    \sqrt{\Delta} = (\beta_1 -\beta_2)(\beta_2 - \beta_3)(\beta_3 - \beta_1) \mbox{ and } -\sqrt{\Delta} = (\beta_2 -\beta_1)(\beta_3 - \beta_2)(\beta_1 - \beta_3)
  \end{align*}

\begin{questions}[resume]
  \item \begin{enumerate}
  \label{ques:cubic}
    \item Identify the \emph{subgroup} of $ S_3$ that fixes \emph{both} $\sqrt{\Delta} $ and $-\sqrt{\Delta}$.
    \item What is the subgroup of $ S_3$ that fixes all the symmetric variables corresponding to the coefficients: $ \beta_1 \beta_2 \beta_3$, $\beta_1 \beta_2 + \beta_2 \beta_3 + \beta_3 \beta_1$, and $\beta_1 + \beta_2 + \beta_3$.
    \item What is the subgroup of $ S_3$ that fixes the individual variables $\beta_1 , \beta_2,  \beta_3 $.
  \end{enumerate}
  This problem generalizes to all $ n$.
\end{questions}

Similarly we had the following intermediate variables for the quartic
  \begin{align*}
    \lambda_1 = \gamma_1 \gamma_2 + \gamma_3 \gamma_4
    \mbox{ and } \lambda_2 = \gamma_1 \gamma_3 + \gamma_2 \gamma_4
    \mbox{ and } \lambda_3 = \gamma_1 \gamma_4 + \gamma_2 \gamma_3
  \end{align*}

\begin{questions}[resume]
  \item
  \label{ques:quartic}
Identify the \emph{subgroup} of $ S_4$ that fixes \emph{all} the elements $ \lambda_1, \lambda_2, \lambda_3$.
\end{questions}

Notice that all the groups you've computed above are normal subgroups.

\newpage
\noindent \textbf{Galois correspondence: } Galois' theorem states that there is a one-to-one correspondence between (sequence of) \emph{intermediate variables}\footnote{We need the concept of a \emph{field} to make this more precise.} in the roots of polynomials of degree $ n$ and (sequence of) normal subgroups of $ S_n$. As mentioned earlier a {general} polynomial can be solved by radicals only if there exists (sequence of) intermediate variables of lower degrees.

For any $ S_n$ the correspondence is
\begin{align*}
  \{ e \} & \leftrightarrow  \mbox{ roots }   \\
  A_n & \leftrightarrow \{ +\sqrt{\Delta}, -\sqrt{\Delta} \} \\
  S_n & \leftrightarrow \mbox{ coefficients }
\end{align*}

For $ S_3$ this correspondence is enough as we've already seen that $ \mu_1^3, \mu_2^3$ could be expressed in terms of $ \pm \sqrt{\Delta} $, and it is possible to recover the roots from these.\\

For $ S_n, n \ge 4$ knowing $ \{ +\sqrt{\Delta}, -\sqrt{\Delta} \} $ is not enough to recover the roots.\\

However we get lucky for $ S_4$, as $ S_4$ has a special normal subgroup $ K_4$ and the variables $ \{ \lambda_1, \lambda_2, \lambda_3 \}$ corresponding to this normal subgroup do indeed satisfy a lower degree (cubic) polynomial and the roots can be recovered from these.
\begin{align*}
  K_4 & \leftrightarrow \{ \lambda_1, \lambda_2, \lambda_3 \}
\end{align*}

But because $ S_5$ does not have this accidental normal subgroup, there are not enough intermediate variables which could allow us to find a general formula using radicals. This then is the underlying reason why a quintic polynomial cannot be solved using radicals: as $ S_5$ has very few normal subgroups!!!\footnote{A more precise statement is that $ A_5$ has no normal subgroups.}\\

\noindent \textbf{For further reading:} If you're interested in learning more about this you should start by some group theory in more details. One of my favorite algebra book for beginners is \emph{Algebra}, by Michael Artin.


\subsection{Fields}\label{subsec:fields}

\begin{definition}\label{def:field}
  As mentioned in \fullref{def:semiring/field}, fields are commutative division rings.
\end{definition}

\begin{proposition}\label{thm:ideals_of_field}
  The only \hyperref[def:semiring_ideal]{ideals} of a field are \( \{ 0 \} \) and \( \BbbK \).
\end{proposition}

\begin{theorem}\label{thm:ring_of_integers_module_prime_is_field}
  The ring \( \BbbZ_n \) (see \fullref{def:ring_of_integers_modulo}) of integers modulo \( n \) is a field if \( n \) is a prime \hyperref[def:prime_number]{number}.
\end{theorem}
\begin{proof}
  We only need to show that \( \BbbZ_n \) has a multiplicative inverse for any nonzero element.

  Fix \( x \in \BbbZ_n \). If \( y \) is a multiplicative inverse of \( x \), we should have
  \begin{equation*}
    xy \cong 1 \pmod n,
  \end{equation*}
  which is the same as
  \begin{equation*}
    n \mid (xy - 1).
  \end{equation*}

  \Fullref{thm:bezout_identity} gives us integers \( a, b \in \BbbZ \) such that
  \begin{equation*}
    ax + bn = \gcd(x, n) = 1,
  \end{equation*}
  which is the same as
  \begin{equation*}
    -bn = xa - 1.
  \end{equation*}

  Define \( y \coloneqq \rem(a, n) \). This is the multiplicative inverse of \( x \).
\end{proof}

\begin{definition}\label{def:field_extension}
  If \( \Bbbk \) and \( \BbbK \) are fields and \( \Bbbk \) is a unital \hyperref[def:first_order_substructure]{subring} of \( \BbbK \), we say that \( \Bbbk \) is a \term{subfield} of \( \BbbK \) and that \( \BbbK \) is a \term{field extension} of \( \Bbbk \). If \( \BbbK = \Bbbk \), we say that \( \BbbK \) is a \term{trivial field extension} of \( \Bbbk \).

  Field extension are also denoted as \( \BbbK / \Bbbk \) to highlight the roles of \( \BbbK \) and \( \Bbbk \). This is not a quotient ring, but simply a notation. See \fullref{def:galois_group}.

  We define the following
  \begin{thmenum}
    \thmitem{def:field_extension/dimension} The extension \( \BbbK \) is a vector space over \( \Bbbk \). We denote the dimension of this vector space by
    \begin{equation*}
      [\BbbK : \Bbbk].
    \end{equation*}

    We call \( [\BbbK : \Bbbk] \) the dimension of \( \BbbK \) over \( \Bbbk \) and if \( [\BbbK : \Bbbk] \) is finite, we say that \( \BbbK \) is a \term{finite extension} of \( \Bbbk \).

    \thmitem{def:field_extension/generated_extension} If \( x_1, \ldots, x_n \) are members of \( \BbbK \), we will use the following
    \begin{itemize}
      \item The ring \( \Bbbk[x_1, \ldots, x_n] \) obtained by evaluating \hyperref[thm:polynomial_ring_universal_property]{polynomials}.
      \item The field \( \Bbbk(x_1, \ldots, x_n) \) obtained by evaluating rational algebraic \hyperref[def:rational_algebraic_function]{functions}.
    \end{itemize}
  \end{thmenum}
\end{definition}

\begin{remark}\label{rem:adjoint_extension_field}
  Any field of rational algebraic \hyperref[def:rational_algebraic_function]{functions} over \( \BbbK \) is always a field extension of \( \BbbK \). We say that the field \( \BbbK(X) \) is obtained from \( \BbbK \) by \term{adjoining} a new element \( X \). Although formally \( X \) is a polynomial, we regard it as a symbol in the sense of \fullref{def:formal_language}.
\end{remark}

\begin{definition}\label{def:galois_group}\mcite[124]{Knapp2016BasicAlgebra}
  Let \( \BbbK \) be a field \hyperref[def:field_extension]{extension} of \( \Bbbk \). The group \( \op{Gal}(\BbbK / \Bbbk) \) of automorphisms of \( \BbbK \) that leave \( \Bbbk \) fixed is called the \term{Galois group} of the field extension \( \BbbK / \Bbbk \).
\end{definition}

\begin{example}\label{thm:galois_group_complex_over_real}
  The Galois \hyperref[def:galois_group]{group} \( \op{Gal}(\BbbC / \BbbR) \) is the group of all \( \BbbR \)-linear functions \( \varphi: \BbbC \to \BbbC \) such that
  \begin{equation*}
    \varphi(\BbbR) = \BbbR.
  \end{equation*}

  The only such functions are rotations and axial symmetries. No nontrivial rotations of the complex plane leave \( \BbbR \) intact and the only nontrivial axial symmetry that fixes \( \BbbR \) is \( a + bi \mapsto a - bi \). Hence,
  \begin{equation*}
    \op{Gal}(\BbbC / \BbbR) \cong \BbbZ_2.
  \end{equation*}
\end{example}

\begin{definition}\label{def:transcendetal_element}\mcite[454]{Knapp2016BasicAlgebra}
  We say that the element \( a \in \BbbK \) of the field extension \( \BbbK \) of \( \Bbbk \) is \term{transcendental} over \( \BbbK \) if any of the equivalent conditions hold:
  \begin{thmenum}
    \thmitem{def:transcendetal_element/evaluation} The evaluation \hyperref[thm:polynomial_ring_universal_property]{map} \( \Phi_a: \Bbbk[X] \to \Bbbk[a] \) is injective.

    \thmitem{def:transcendetal_element/polynomial} There exists no polynomial \( p(X) \in \Bbbk[X] \) such that \( p(a) = 0 \).
  \end{thmenum}

  If \( a \) is not transcendental, we say that is is \term{algebraic}.
\end{definition}

\begin{definition}\label{def:algebraic_extension}\mcite[456]{Knapp2016BasicAlgebra}
  We say that the field extension \( \BbbK \) of \( \Bbbk \) is an \term{algebraic extension} if every element of \( \BbbK \) is algebraic over \( \Bbbk \).
\end{definition}

\begin{proposition}\label{thm:field_elements_are_algebraic}
  Every field is an \hyperref[def:algebraic_extension]{algebraic} of itself.
\end{proposition}
\begin{proof}
  If \( a \in \BbbK \), then \( \BbbK[a] = \Bbbk \) because every polynomial evaluates to some real element, depending on \( a \), and the constant polynomials already take all possible values. Thus, \( \Phi_a \) is not injective.

  Since \( a \in \BbbK \) was arbitrary, we conclude that all elements from a field are algebraic over \( \BbbK \).
\end{proof}

\begin{theorem}\label{thm:algebraic_extension_always_exists}\mcite[485]{Knapp2016BasicAlgebra}
  If \( p(X) \) is a prime polynomial over the field \( \BbbK \), there exists an algebraic extension of \( \BbbK = \Bbbk[u] \), where \( u \in \BbbK \) is a root of \( p(X) \).
\end{theorem}
\begin{proof}
  Since \( p(X) \) is prime, \( \braket{p(X)} \) is a nontrivial prime ideal. By \fullref{thm:prime_ideals_are_maximal_in_pid}, \( \braket{p(X)} \) is maximal and by \fullref{def:maximal_ring_ideal}, the quotient \( \BbbK \coloneqq \Bbbk / \braket{p(X)} \) is a field. It is an extension field of \( \Bbbk \).

  Define
  \begin{equation*}
    u \coloneqq X + \braket{p(X)}.
  \end{equation*}

  Then
  \begin{equation*}
    p(u) = p(X + \braket{p(X)}) = p(X) + \braket{p(X)} = \braket{p(X)},
  \end{equation*}
  hence \( u \) is a root of \( p(X) \) in \( \BbbK \).

  Thus, \( u \) is algebraic over \( \Bbbk \) and \( \BbbK \) is an algebraic extension of \( \Bbbk \).

  It remains to show that \( \BbbK = \Bbbk[u] \). First, take a coset \( q(X) + \braket{p(X)} \) in \( \BbbK \). We have
  \begin{equation*}
    q(X) + \braket{p(X)} = (q(X) - X) + (X + \braket{p(X)}) = (q(X) - X) + u,
  \end{equation*}
  hence this belongs in \( \Bbbk[u] \). Conversely, evaluating a polynomial \( q(X) \in \Bbbk[X] \) at \( u \) gives us
  \begin{equation*}
    q(u) = q(X + \braket{p(X)}) = q(X) + \braket{p(X)},
  \end{equation*}
  which is a coset of \( \BbbK \).
\end{proof}

\begin{proposition}\label{thm:finite_field_extensions_are_algebraic}
  Finite field extensions are algebraic.
\end{proposition}
\begin{proof}
  Fix a finite field extension \( \BbbK / \Bbbk \) and denote by \( n \) the dimension \( [\BbbK : \Bbbk] \). Assume that \( a \in \BbbK \) is transcendental. Then the evaluation map
  \begin{equation*}
    \Phi_a: \Bbbk[X] \to \Bbbk[a]
  \end{equation*}
  is injective. But \( \Bbbk[X] \) has a countably infinite monomial basis, so \( \Bbbk[a] \) must also have a countable basis consisting of \( 1, a, a^2, \ldots \). But \( \Bbbk[a] \) is a subspace of \( \Bbbk \), which is finite dimensional over \( \Bbbk \).

  The obtained contradiction proves the theorem.
\end{proof}

\begin{theorem}\label{thm:e_is_transcendental}\label{thm:eulers_constant_is_transcendental}
  \hyperref[def:exponential_function]{Euler's constant} \( e \) is transcendental over \( \BbbQ \).
\end{theorem}

\begin{theorem}\label{thm:pi_is_transcendental}\mcite[454]{Knapp2016BasicAlgebra}
  The number \( \pi \) (see \fullref{def:pi}) is transcendental over \( \BbbQ \).
\end{theorem}

\begin{example}\label{ex:polynomials_over_pi}
  \Fullref{thm:pi_is_transcendental} implies that the polynomials \( \BbbQ[X] \) can be embedded into \( \BbbR \) via \( \Phi_\pi: \BbbQ[X] \to \BbbR \). We can identify a polynomial
  \begin{equation*}
    p(X) = \sum_{i=0}^n a_i X^i
  \end{equation*}
  with rational coefficients with the number
  \begin{equation*}
    p(\pi) = \sum_{i=0}^n a_i \pi^i.
  \end{equation*}
\end{example}

\begin{definition}\label{def:algebraically_closed_field}\mcite[prop. 9.20]{Knapp2016BasicAlgebra}
  We say that the field \( \BbbK \) is algebraically closed if any of the equivalent conditions are satisfied:
  \begin{thmenum}
    \thmitem{def:algebraically_closed_field/trivial_algebraic_extensions} \( \BbbK \) has no nontrivial algebraic \hyperref[def:algebraic_extension]{extensions}.
    \thmitem{def:algebraically_closed_field/linear_irreducible_polynomials} Every irreducible polynomial in \( \Bbbk[X] \) is linear.
    \thmitem{def:algebraically_closed_field/at_least_one_root} Every nonconstant polynomial in \( \Bbbk[X] \) has at least one root in \( \Bbbk \).
    \thmitem{def:algebraically_closed_field/factorization} Every polynomial in \( \Bbbk[X] \) \hyperref[def:factorization_in_ring]{factors} into a product of linear polynomials.
    \thmitem{def:algebraically_closed_field/exactly_n_roots} Every polynomial in \( \Bbbk[X] \) of degree \( n \) has exactly \( n \) roots in \( \Bbbk \).
  \end{thmenum}
\end{definition}
\begin{proof}
  \ImplicationSubProof{def:algebraically_closed_field/trivial_algebraic_extensions}{def:algebraically_closed_field/linear_irreducible_polynomials} Let \( p(X) \) be an irreducible polynomial in \( \Bbbk[X] \). By \fullref{thm:ufd_prime_iff_irreducible}, \( p(X) \) is prime. By \fullref{thm:algebraic_extension_always_exists}, there exists an algebraic extension \( \BbbK \) of \( \Bbbk \) such that the prime polynomial \( p(X) \) has a root in \( \BbbK \). But \( \Bbbk \) has no nontrivial algebraic extensions, hence \( F = G \) and \( p(X) \) has a root \( u \in F \).

  If \( p(X) \) is not linear, we can divide \( p(X) \) by \( (X - u) \) to obtain a lower-degree non-constant polynomial. Hence, \( p(X) \) is linear.

  \ImplicationSubProof{def:algebraically_closed_field/linear_irreducible_polynomials}{def:algebraically_closed_field/factorization} With induction on the polynomial degree, we split a polynomial \( p(X) \) into a product of linear polynomials.

  This is obvious for \( \deg p = 1 \). Assume that the statement holds for polynomial of degree strictly less than \( n \) and let \( p(X) \) be a polynomial of degree \( n \). By \fullref{def:factorization_in_ring}, it is \hyperref[def:irreducible_ring_element]{reducible}, that is, there exist non-invertible (that is, non-constant) polynomials \( r_1(X) \) and \( r_2(X) \), such that
  \begin{equation*}
    p(X) = r_1(X) r_2(X).
  \end{equation*}

  Since both \( r_1(X) \) and \( r_2(X) \) are non-constant, they have a positive degree less than \( n \). Hence, the induction conjecture holds for them and both can be factored into linear polynomials. Therefore, their product \( p(X) \) can also be factored into linear polynomials.

  This completes the proof.

  \ImplicationSubProof{def:algebraically_closed_field/at_least_one_root}{def:algebraically_closed_field/factorization} Suppose that \( u_1 \) is a root of \( p(X) \). \Fullref{thm:polynomial_root_iff_divisible} tells us that \( p(X) \) is divisible by \( (X - u_1) \). Using induction on the degree of \( p(X) \), we can factor \( p(X) \) into
  \begin{equation*}
    p(X) = a (X - u_1) (X - u_2) \cdots (X - u_n),
  \end{equation*}
  where \( a \in F \).

  \ImplicationSubProof{def:algebraically_closed_field/factorization}{def:algebraically_closed_field/exactly_n_roots} Follows from \fullref{thm:polynomial_root_iff_divisible} Follows from \fullref{thm:polynomial_root_iff_divisible} by induction on the polynomial degree. Note that the number of roots is bounded by \( n \) (see \fullref{thm:integral_domain_polynomial_root_limit}).

  \ImplicationSubProof{def:algebraically_closed_field/exactly_n_roots}{def:algebraically_closed_field/trivial_algebraic_extensions} By \fullref{thm:integral_domain_polynomial_root_limit}, if \( p(X) \) has degree \( n \) and exactly \( n \) roots, then it has no more roots. Hence, all roots of \( p(X) \) are already in the field \( \Bbbk \) and \( \Bbbk \) is the only algebraic extension of itself.
\end{proof}

\begin{proposition}\label{thm:no_finite_extensions_of_closed_fields}
  There exist no nontrivial finite extensions of an algebraically closed field.
\end{proposition}
\begin{proof}
  Let \( \BbbK \) is a finite extension of the algebraically closed field \( \Bbbk \). By \fullref{thm:finite_field_extensions_are_algebraic}, \( \BbbK \) is an algebraic extension. But every element of \( \Bbbk \) is already algebraic over \( \Bbbk \), therefore \( \BbbK \subseteq \Bbbk \).

  We conclude that \( \Bbbk = \BbbK \), hence the only finite extension of an algebraically closed field is trivial.
\end{proof}

\begin{definition}\label{def:splitting_field}\mcite[458]{Knapp2016BasicAlgebra}
  We say that a polynomial \( p(X) \in \BbbK[X] \) over the field \( \BbbK \) \term{splits} if \( p(X) \) can be \hyperref[def:factorization_in_ring]{factored} into a product of linear polynomials in \( \BbbK[X] \).

  A \term{splitting field} of \( p(X) \) over \( \Bbbk \) is a field extension \( \BbbK / \Bbbk \) such that
  \begin{itemize}
    \item \( p(X) \) splits over \( \BbbK \).
    \item \( \BbbK \) is \hyperref[def:generated_ring_ideal]{generated} by \( \Bbbk \) and the roots of \( p(X) \) over \( \BbbK \).
  \end{itemize}
\end{definition}

\begin{proposition}\label{thm:splitting_field_existence}\mcite[thm. 9.12]{Knapp2016BasicAlgebra}
  A splitting field exists for every polynomial \( p(X) \in \BbbK[X] \).
\end{proposition}

\medskip

\begin{theorem}\label{thm:galois_field_existence}\mcite[thm. 9.14]{Knapp2016BasicAlgebra}
  Fix a prime \hyperref[def:prime_number]{number} \( p \) and a positive integer \( n \). Then there exists up to an isomorphism a unique field with \( p^n \) elements. Furthermore, this is a splitting field for \( X^{p^n} - X \) for the field \( \BbbZ_p \).

  We call this field the \term{Galois field} of \( p^n \) elements over the prime field \( \BbbZ_p \) and denote it by
  \( \BbbF_{p^n} \). We identify \( \BbbZ_p \) with \( \BbbF_p \).
\end{theorem}

\begin{theorem}\label{thm:f2_is_boolean_algebra}
  The Galois field \( \BbbF_2 \) is a \hyperref[def:boolean_algebra]{Boolean algebra} with joins and meets induced by the ordering (see \fullref{thm:binary_lattice_operations}) and complements given by \( \neg x \coloneqq x \mapsto x \oplus 1 \).

  More concretely:
  \begin{itemize}
    \item The top element is \( \sup \{ 0, 1 \} = 1 \)
    \item The bottom element is \( \inf \{ 0, 1 \} = 0 \)
    \item Joins are given by \( \inf \{ x, y \} = xy \)
    \item Meets are given by multiplication \( \sup \{ x, y \} = x \oplus y \oplus (x \odot y) = x \oplus y \oplus \inf \{ x, y \} \)
  \end{itemize}
\end{theorem}
\begin{proof}
  Addition and multiplication in \( \BbbF_2 \) works as usual integer \hyperref[def:set_of_integers]{arithmetic}, except that \( 1 \oplus 1 = 0 \):
  \begin{equation*}
    \begin{tabular}{c c | c c}
      \( x \) & \( y \) & \( x \oplus y \) & \( x \odot y \) \\
      \hline
      \( 0 \) & \( 0 \) & \( 0 \)          & \( 0 \)         \\
      \( 0 \) & \( 1 \) & \( 1 \)          & \( 0 \)         \\
      \( 1 \) & \( 0 \) & \( 1 \)          & \( 0 \)         \\
      \( 1 \) & \( 1 \) & \( 0 \)          & \( 1 \)
    \end{tabular}
  \end{equation*}

  Evidently \( 1 \) is a top element and \( 0 \) is a bottom element, thus \( \BbbF_2 \) is a \hyperref[def:semilattice/lattice]{lattice}.

  Distributivity of multiplication over addition is inherited from \( \BbbZ \), however unlike in \( \BbbZ \), addition distributes over multiplication:
  \begin{equation*}
    \begin{tabular}{c c c | c c}
      \( x \) & \( y \) & \( z \) & \(x \odot (y \oplus z) \) & \( (x \odot y) \oplus (x \odot z) \) \\
      \hline
      \( 0 \) & \( 0 \) & \( 0 \) & \( 0 \)                   & \( 0 \)                              \\
      \( 0 \) & \( 0 \) & \( 1 \) & \( 0 \)                   & \( 0 \)                              \\
      \( 0 \) & \( 1 \) & \( 0 \) & \( 0 \)                   & \( 0 \)                              \\
      \( 0 \) & \( 1 \) & \( 1 \) & \( 0 \)                   & \( 0 \)                              \\
      \( 1 \) & \( 0 \) & \( 0 \) & \( 0 \)                   & \( 0 \)                              \\
      \( 1 \) & \( 0 \) & \( 1 \) & \( 1 \)                   & \( 1 \)                              \\
      \( 1 \) & \( 1 \) & \( 0 \) & \( 1 \)                   & \( 1 \)                              \\
      \( 1 \) & \( 1 \) & \( 1 \) & \( 1 \)                   & \( 1 \)
    \end{tabular}
  \end{equation*}

  Thus, \( \BbbF_2 \) is a distributive \hyperref[def:semilattice/distributive_lattice]{lattice}.

  It is also evident that the complementation \( \neg x = x \mapsto x \oplus 1 \) gives the desired result:
  \begin{itemize}
    \item \( \inf \{ x, \neg x \} = x \odot \neg x = x \odot (x \oplus 1) = (x \odot x) \oplus (x \odot 1) = 0 \) since \( x \odot x = x \odot 1 \).
    \item \( \sup \{ x, \neg x \} = x \oplus \neg x \oplus (x \odot \neg x) = x \oplus \neg x \oplus 0 = x \oplus (x \oplus 1) = 1 \).
  \end{itemize}

  Therefore, \( (\BbbF_2, 1, 0, \inf, \sup, \neg x ) \) is a Boolean algebra.
\end{proof}

\begin{proposition}
  For any function \( f: \BbbF_n \to \BbbF_n \) over any Galois field \( \BbbF_n \), there exists a unique polynomial \( p(X) \in \BbbF_n[X] \) of degree \( n - 1 \) such that the corresponding function \( p(x) \) agrees with \( f(x) \) on all of \( \BbbF_n \).
\end{proposition}
\begin{proof}
  We simply use \fullref{thm:lagrange_interpolation} on all points of the field.
\end{proof}

\begin{algorithm}\label{alg:finite_field_polynomial_reduction}
  Let \( \BbbF_p \) be the \hyperref[thm:galois_field_existence]{Galois field} for some prime number \( p \). Consider the nonzero polynomial
  \begin{equation*}
    f(X_1, \ldots, X_n) \coloneqq \sum_{k_1=0}^{m_1} \cdots \sum_{k_n=0}^{m_n} a_{k_1,\ldots,k_n} X_1^{k_1} \cdots X_n^{k_n}.
  \end{equation*}

  We will build a polynomial \( \hat f(X_1, \ldots, X_n) \) of degree at most \( n - 1 \) that corresponds to the same function. In terms of the \hyperref[thm:polynomial_ring_universal_property]{evaluation homomorphism}
  \begin{equation*}
    \Phi: \BbbF_p[X_1, \ldots, X_n] \to \fun(\BbbF_p^n, \BbbF_p),
  \end{equation*}
  this means that
  \begin{equation*}
    \Phi(f) = \Phi(\hat f).
  \end{equation*}

  This can be achieved by grouping some of the coefficients. The univariate case is more understandable, so we will instead initially consider the polynomial
  \begin{equation*}
    g(X) \coloneqq \sum_{k=0}^m a_k X^k.
  \end{equation*}

  We will now use (and later prove) that the following two univariate polynomials evaluate to the same function:
  \begin{equation}\label{eq:alg:finite_field_polynomial_reduction/reduction}
    \Phi(X^s) = \Phi(X^{\rho(s)}),
  \end{equation}
  where
  \begin{align*}
    &\rho: \{ 1, 2, 3, \ldots \} \to \{ 1, 2, \ldots, p - 1 \} \\
    &\rho(s) \coloneqq \begin{cases}
      p - 1,          &(p - 1) \mid s \\
      \rem(s, p - 1), &\T{times}.
    \end{cases}
  \end{align*}

  Obviously \( \rho(s) = s \) for \( s < p \). It follows from the linearity of \( \Phi \) that
  \begin{equation*}
    \hat g(X)
    \coloneqq
    \sum_{k=0}^m a_k X^{r(k)}
    =
    \sum_{j=0}^{p-1} \left( \sum_{k \in r^{-1}(j)} a_k \right) X^j
  \end{equation*}
  is the desired polynomial.

  In the multivariate case, the reduced polynomial is
  \begin{equation*}
    \hat f(X_1, \ldots, X_n) \coloneqq \sum_{j_1=0}^{p-1} \cdots \sum_{j_n=0}^{p-1} \left( \sum_{k_1 \in r^{-1}(j_1)} \cdots \sum_{k_n \in r^{-1}(j_n)} a_{k_1,\ldots,k_n} \right) X_1^{k_1} \cdots X_n^{k_n}.
  \end{equation*}

  Correctness follows, again, from the linearity of \( \Phi \).
\end{algorithm}
\begin{proof}[Proof of \eqref{eq:alg:finite_field_polynomial_reduction/reduction}]
  By \fullref{thm:fermats_little_theorem},
  \begin{equation}\label{eq:alg:finite_field_polynomial_reduction/fermat}
    \Phi(X^p) = \Phi(X).
  \end{equation}

  We now use induction on \( q \coloneqq \quot(s, p) \).
  \begin{itemize}
    \item If \( q = 0 \), then \( s < p \) and no reduction is necessary.
    \item If \( q > 0 \), let \( s = (p - 1) q + r \). We have two cases
    \begin{itemize}
      \item If \( (p - 1) \mid s \), i.e. if \( r = 0 \), then
      \begin{balign*}
        \Phi(X^s)
        &=
        \Phi(X^{(p-1)q})
        = \\ &=
        \Phi(X^{(p-1)(q-1) + (p-1)})
        = \\ &=
        \Phi(X^{(p-1)(q-1)}) \Phi(X^{p-1})
        \reloset {\T{ind.}} = \\ &=
        \Phi(X^{p-1}) \Phi(X^{p-1})
        = \\ &=
        \Phi(X^{2(p-1)})
        = \\ &=
        \Phi(X^{p+(p-2)})
        = \\ &=
        \Phi(X^p) \Phi(X^{p-2})
        \reloset {\eqref{eq:alg:finite_field_polynomial_reduction/fermat}} = \\ &=
        \Phi(X^{p-1}).
      \end{balign*}

      \item If \( r > 0 \), we have
      \begin{balign*}
        \Phi(X^s)
        &=
        \Phi(X^{(p-1)q+r})
        = \\ &=
        \Phi(X^{[(p-1)(q-1) + r] + (p-1)})
        = \\ &=
        \Phi(X^{(p-1)(q-1)+r}) \Phi(X^{p-1})
        \reloset {\T{ind.}} = \\ &=
        \Phi(X^r) \Phi(X^{p-1})
        = \\ &=
        \Phi(X^{r-1}) \Phi(X^p)
        \reloset {\eqref{eq:alg:finite_field_polynomial_reduction/fermat}} = \\ &=
        \Phi(X^{r-1}) \Phi(X)
        = \\ &=
        \Phi(X^r).
      \end{balign*}
    \end{itemize}
  \end{itemize}
\end{proof}


\documentclass[main.tex]{subfiles}
\begin{document}

\section{November exercises}

\subsection{Exercise 4}

% \marginpar{Tuesday\\ 2020-10-6, \\ compiled \\ \today}

After being given a probability distribution \(\mathbb{P}(x)\), we define the \emph{characteristic function} \(\phi \) as its Fourier transform, which can also be expressed as the expectation value of \(\exp(- i \vec{k} \cdot \vec{x})\):
%
\begin{align}
\phi (\vec{k}) = \int \dd[n]{x} \exp(- i \vec{k} \cdot \vec{x}) \mathbb{P}(x)
= \mathbb{E} \qty[ \exp(- i \vec{k} \cdot \vec{x})]
\,.
\end{align}

\begin{claim}
A multivariate normal distribution
%
\begin{align}
\mathcal{N}(\vec{x} | \vec{\mu}, C)
&= \frac{1}{(2\pi )^{n/2} \sqrt{\det C}} \eval{\exp(- \frac{1}{2} \vec{y}^{\top} C^{-1} \vec{y})}_{\vec{y} = \vec{x} - \vec{\mu}}
\,,
\end{align}
%
has a characteristic function equal to
%
\begin{align}
\phi (\vec{k}) = \exp(- i \vec{\mu}\cdot \vec{k} - \frac{1}{2} \vec{k}^{\top} C \vec{k})
\,.
\end{align}
\end{claim}

\begin{proof}[Proof: completing the square]
The integral we need to compute is given, absorbing the normalization into a factor \(N\), by
%
\begin{align}
\phi (\vec{k}) = N \int \dd[n]{x} \eval{\exp(- i \vec{k} \cdot \vec{x} - \frac{1}{2} \vec{y}^{\top} C^{-1} \vec{y})}_{\vec{y} = \vec{x} - \vec{\mu}}
\,.
\end{align}

The only integrals we really know how to do are Gaussian ones, so we want to rewrite the argument of the exponential so that it is a quadratic form. The manipulation goes as follows, considering the opposite of the argument the exponential in order to have less minus signs and defining the symmetric matrix \(V = C^{-1}\):
%
\begin{align}
i \vec{k} \cdot \vec{x} + \frac{1}{2} \vec{y}^{\top} V \vec{y}
&= i \vec{k} \cdot \vec{x} +
\frac{1}{2} \vec{x}^{\top} V \vec{x}
- \vec{x}^{\top} V \vec{\mu }
+ \frac{1}{2} \vec{\mu}^{\top} V \vec{\mu}  \\
&= \frac{1}{2} \vec{x}^{\top} V \vec{x}
+ \vec{x}^{\top}\qty(
    i \vec{k} - V \vec{\mu}
)
+ \frac{1}{2} \vec{\mu}^{\top} V \vec{\mu}  \\
\begin{split}
&= \underbrace{\frac{1}{2} \qty(\vec{x} + V^{-1} (i \vec{k} - V \vec{\mu}))^{\top} V \qty(\vec{x} + V^{-1} (i \vec{k} - V \vec{\mu}))}_{\Circled{1}} +\\
&\phantom{=}\ \underbrace{- \frac{1}{2} \qty(i \vec{k} - V \vec{\mu})^{\top} V^{-1} \qty(i \vec{k} - V \vec{\mu}) + \frac{1}{2} \vec{\mu}^{\top} V \vec{\mu}}_{\Circled{2}}
\,,
\end{split}
\end{align}
%
which we can now integrate, since it is now a quadratic form in terms of a shifted variable, \(\vec{x} + \vec{p}\), where the constant (with respect to \(\vec{x}\)) vector \(\vec{p}\) is given by \(V^{-1}(i \vec{k} - V \vec{\mu} )\).\footnote{
In the last step we applied the matrix square completion formula: for a  symmetric matrix \(A\) and vectors \(\vec{x}\), \(\vec{b}\) we have
%
\begin{align} \label{eq:square-completion}
&\frac{1}{2} \qty(\vec{x} + A^{-1} \vec{b})^{\top} A \qty(\vec{x} + A^{-1} \vec{b}) - \frac{1}{2} \vec{b}^{\top} A^{-1} \vec{b}  =\\
&= \frac{1}{2} \qty[ \vec{x}^{\top} A \vec{x}
+  \vec{x}^{\top} A A^{-1} \vec{b}
+  \qty(A^{-1} \vec{b})^{\top} A \vec{x}
+  \qty( A^{-1} \vec{b})^{\top} A A^{-1} \vec{b}
-  \vec{b}^{\top} A^{-1} \vec{b}]  \\
&=\frac{1}{2} \qty[ \vec{x}^{\top} A \vec{x}
+  \vec{x}^{\top} \vec{b}
+  \vec{b}^{\top} (A^{-1})^{\top} A \vec{x}
+  \vec{b}^{\top} (A^{-1})^{\top}  \vec{b}
-  \vec{b}^{\top} A^{-1} \vec{b}]  \\
&= \frac{1}{2} \vec{x}^{\top} A \vec{x}
+ \vec{b}^{\top} \vec{x}
\,,
\end{align}
%
which we used with \(\vec{b} = i \vec{k} - V \vec{\mu}\).}

Now, shifting the integral from one in  \(\dd[n]{x}\) to one in \(\dd[n]{(x + p)}\) does not change the measure, since the Jacobian of a shift  is the identity.
Then, we have
%
\begin{align}
\phi (\vec{k}) &= N \int \dd[n]{(x+p)} \exp(- \Circled{1} - \Circled{2})  \\
&= N \sqrt{\frac{(2\pi )^{n}}{\det V}} \exp(- \Circled{2})  \\
&= \underbrace{\frac{1}{\sqrt{\det V \det C}}}_{= 1} \exp(- \Circled{2})
\,,
\end{align}
%
since the determinant of the inverse is the inverse of the determinant.

Now, we only need to simplify \(\Circled{2}\):
%
\begin{align}
\Circled{2} &= - \frac{1}{2} \qty[
    - \vec{k}^{\top} V^{-1} \vec{k}
    - 2i \vec{\mu}^{\top} V V^{-1} \vec{k}
    + \vec{\mu}^{\top} V V^{-1} V \vec{\mu}
]
+ \frac{1}{2} \vec{\mu}^{\top} V \vec{\mu}  \\
&= \frac{1}{2} \vec{k}^{\top} C \vec{k} + i \vec{\mu}^{\top} \vec{k}
\,,
\end{align}
%
inserting which into the exponent yields the desired result.
\end{proof}

\begin{proof}[Proof: by diagonalization]
We now follow a different approach: the covariance matrix \(C\) is symmetric, so we will always be able to find an orthogonal matrix \(O\) (satisfying \(O^{\top} = O^{-1}\)) such that \(C = O^{\top} D O\), where \(D\) is diagonal.
We will then also have \(V = C^{-1} = O^{\top} D^{-1} O\).
Let us denote the eigenvalues of \(D\) as \(\lambda _i\), and the eigenvalues of \(D^{-1}\) as \(d_i = \lambda _i^{-1}\).

Defining \(\vec{z} = O \vec{x}\), \(\vec{m} = O \vec{\mu}\), \(\vec{u} = O \vec{k}\) the negative of the argument of the integral becomes:
%
\begin{align}
 i \vec{k} \cdot \vec{x}
 + \frac{1}{2}
\qty(\vec{x} - \vec{\mu})^{\top} C^{-1}
\qty(\vec{x} - \vec{\mu}) &=
 i \vec{u} \cdot \vec{z}
+ \frac{1}{2} (\vec{z} - \vec{m})^{\top} D^{-1} (\vec{z} - \vec{m})  \\
&=
i \vec{u} \cdot \vec{z}
+ \frac{1}{2} \sum _{i} d_i (z_i - m_i)^2   \\
&= \sum _{i} \qty[i u_i z_i + \frac{d_i}{2} \qty(z_i^2 + m_i^2 - 2 m_i z_i)]  \\
&= \sum _{i} \qty[z_i^2 \frac{d_i}{2} + z_i \qty(i u_i - m_i d_i) + \frac{d_i}{2} m_i^2]
\,.
\end{align}

With this, and since by \(\det O = 1\) we have \(\dd[n]{z} = \dd[n]{x}\), we can decompose our Gaussian integral into a product of Gaussian integrals:
%
\begin{align}
\phi (\vec{k}) &=
N \int \dd[n]{x} \exp(-  i \vec{k} \cdot \vec{x}
 - \frac{1}{2}
\qty(\vec{x} - \vec{\mu})^{\top} C^{-1}
\qty(\vec{x} - \vec{\mu}))  \\
&= N \int \dd[n]{z} \exp(- \sum _{i} \qty[z_i^2 \frac{d_i}{2} + z_i \qty(i u_i - m_i d_i) + \frac{d_i}{2} m_i^2])  \\
&= N \prod_i \int \dd{z_i}
\exp(- z_i^2 \frac{d_i}{2} - z_i \qty(i u_i - m_i d_i) - \frac{d_i}{2} m_i^2)  \\
&= N \prod_i \sqrt{\frac{2\pi }{d_i}} \exp(\frac{(i u_i - m_i d_i)^2}{2 d_i} - \frac{d_im_i^2}{2}) \\
&= \frac{1}{\sqrt{\det C \det V}} \prod_i \exp(\frac{- u_i^2 + m_i^2 d_i^2 - 2 i u_i m_i d_i }{2 d_i} - \frac{d_i m_i^2}{2}) \\
&= \exp(\sum _{i} \qty[- \frac{u_i^2}{2d_i} - i u_i m_i])  \\
&= \exp(- \frac{1}{2} \vec{u}^{\top} C \vec{u} - i \vec{u} \cdot \vec{m})  \\
&= \exp(- \frac{1}{2} \vec{k}^{\top} C \vec{k} - i \vec{k} \cdot \vec{\mu })
\,,
\end{align}
%
where we have used the expression for the single-variable Gaussian integral:
%
\begin{align} \label{eq:single-variable-gaussian-integral}
\int \dd{z} \exp(- a z^2 + bz + c) = \sqrt{ \frac{\pi}{a}} \exp( \frac{b^2}{4 a} + c)
\,,
\end{align}
%
which comes from the one-variable completion of the square:
%
\begin{align}
-az^2 + bz + c = -a \qty(z - \frac{b}{2a})^2 + \frac{b^2}{4a} + c
\,.
\end{align}

Also, we used the fact that orthogonal transformation do not change fully-contracted objects, such as scalar products or bilinear forms.
\end{proof}

\subsection{Exercise 5}

We can calculate the moments of a distribution through its characteristic function:
%
\begin{align}
\mathbb{E} \qty[x_\alpha^{n_\alpha } \dots x_\beta^{n_\beta }]
= \eval{\frac{\partial^{n_\alpha \dots n_\beta } \phi (\vec{k})}{\partial(- i k_\alpha )^{n_\alpha } \dots \partial(-i k_\beta )^{n_\beta }}}_{\vec{k} = 0}
\,.
\end{align}

In the multivariate Gaussian case we can then calculate the mean (component by component) as
%
\begin{align}
\mathbb{E}(x_\alpha ) &= \eval{\pdv{\phi (\vec{k})}{(-i k_\alpha )}}_{\vec{k} = 0}  \\
&= \eval{\pdv{}{(-i k_\alpha )}}_{\vec{k} = 0}
\exp(- \frac{1}{2} \vec{k}^{\top} C \vec{k} - i \vec{k} \cdot \vec{\mu })  \\
&= \eval{\qty[- i \sum_\beta  k_\beta  C_{\beta  \alpha } + \mu _\alpha ] \exp(- \frac{1}{2} \vec{k}^{\top} C \vec{k} - i \vec{k} \cdot \vec{\mu })}_{\vec{k} = 0}  \\
&= \mu _\alpha
\,,
\end{align}
%
where we used the fact that the differentiation of a symmetric bilinear form is as follows:
%
\begin{align}
\pdv{}{k_\alpha } \qty(\sum _{\beta \gamma } k_\beta k_\gamma C_{\beta \gamma }) = 2 \sum _{\beta \gamma } \delta_{\beta \alpha } k_\gamma C_{\beta \gamma } = 2 \sum_{\gamma } k_\gamma C_{\alpha \gamma }
\,.
\end{align}

The covariance matrix can be computed by linearity as
%
\begin{align}
\widetilde{C}_{\alpha \beta } = \mathbb{E}\qty[\qty(x_ \alpha - \mathbb{E}(x_\alpha )) \qty(x_\beta - \mathbb{E}(x_\beta ))]
= \mathbb{E}\qty[x_\alpha  x_\beta ] - \mu _\alpha \mu _\beta
\,,
\end{align}
%
the first term of which reads as follows:
%
\begin{align}
\mathbb{E}[x_\alpha x_\beta ] &= \eval{\pdv[2]{\phi (\vec{k})}{(-i k_\beta  )}{(-i k_\alpha  )}}_{\vec{k} = 0}  \\
&= \eval{\pdv{}{(-i k_\beta )}}_{\vec{k} = 0} \qty[-i \sum _{\beta } k_\beta C_{\beta \alpha } + \mu _\alpha ] \exp(- \frac{1}{2} \vec{k}^{\top} C \vec{k} - i \vec{k} \cdot \vec{\mu })  \\
&= C_{\alpha \beta } + \mu _\alpha \mu _\beta
\,,
\end{align}
%
therefore, as expected, \(\widetilde{C}_{\alpha \beta }\) is indeed \(C_{\alpha \beta }\).

\subsection{Exercise 6}

\begin{claim}
The characteristic function of a multivariate Gaussian is, up to normalization, a multivariate Gaussian.
\end{claim}

\begin{proof}
The characteristic function is the exponential of (minus)
%
\begin{align}
\frac{1}{2} \vec{k}^{\top} C \vec{k} + i \vec{k} \cdot \vec{\mu}
&=  \frac{1}{2} \qty(\vec{k} + i C^{-1} \vec{\mu })^{\top} C
\qty(\vec{k} + iC^{-1}\vec{\mu }) + \frac{1}{2} \vec{\mu }^{\top} C^{-1} \vec{\mu }
\,,
\end{align}
%
which means that the characteristic function is in the form
%
\begin{align}
\phi (\vec{k}) = \const \times \exp(- \frac{1}{2} (\vec{k} -\vec{m})^{\top} C (\vec{k} - \vec{m}) )
\,,
\end{align}
%
a multivariate normal with mean \(\vec{m} = -i C^{-1} \vec{\mu}\) and covariance matrix \(C^{-1}\), the inverse of the covariance matrix of the corresponding MVN.
\end{proof}

\end{document}


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{divider}
\section*{\hspace*{-1.6cm} divider}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Find dividers of an integer, closest from the square root of the integer.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[N,M] = divider(N1)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
        {\ty divider} find two integers {\ty N} and {\ty M} such that {\ty
        M*N=N1}, with {\ty M} and {\ty N} as close as possible from {\ty
        sqrt(N1)}.\\
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Examples}
\begin{verbatim}
         N1=256; [N,M]=divider(N1); [N,M]
         ans =
               16    16
         N1=258; [N,M]=divider(N1); [N,M]
         ans =
               6     43
\end{verbatim}


\documentclass{memoir}
\usepackage{notestemplate}

%\logo{~/School-Work/Auxiliary-Files/resources/png/logo.png}
%\institute{Rice University}
%\faculty{Faculty of Whatever Sciences}
%\department{Department of Mathematics}
%\title{Class Notes}
%\subtitle{Based on MATH xxx}
%\author{\textit{Author}\\Gabriel \textsc{Gress}}
%\supervisor{Linus \textsc{Torvalds}}
%\context{Well, I was bored...}
%\date{\today}

%\makeindex

\begin{document}

% \maketitle

% Notes taken on

\begin{defn}[Gaussian Integers]
	The Gaussian integers are elements of the quadratic integer ring \(\Z[i]\). Elements of the ring are the complex numbers \(a+bi \in \C\) with \(a,b \in \Z\). The field norm \(N\) maps
	\begin{align*}
		a+bi \mapsto a^2+b^2
	\end{align*}
	and hence the units\(u\) are given by
	\begin{align*}
		N(a+bi) = a^2+b^2 = \pm 1 \implies u \in \left\{ \pm 1, \pm i \right\}
	\end{align*}
\end{defn}

In general, let \(\mathcal{O}\) be a quadratic integer ring and \(N\) the associated field norm. The multiplicity of the norm gives us a natural irreducibility condition.

\begin{hw}
	Let \(\alpha  \in \mathcal{O}\) be an element such that \(N(\alpha ) = \pm p\) for a prime \(p \in \Z\). Then \(\alpha \) is irreducible in \(\mathcal{O}\).
\end{hw}

Let \(\pi \in \mathcal{O}\) be a prime element. Observe that \((\pi ) \cap \Z\) is a prime ideal in \(\Z\). Because \(N(\pi )\) is a nonzero integer, we have that \((\pi )\cap \Z = p\Z\) for some integer prime \(p\). Hence \(\pi \mid p\) in \(\mathcal{O}\)-- this hints that we can find the prime elements of \(\mathcal{O}\) by determing how primes in \(\Z\) factor as elements of \(\mathcal{O}\).\\

Suppose \(\pi \mid p\) in \(\mathcal{O}\). Then
\begin{align*}
	N(\pi )N(\pi ') = N(p) = p^2 \implies N(\pi ) = \pm p^2 \text{ or } N(\pi ) = \pm p
\end{align*}
If the first holds, then \(\pi '\) is a unit and \(p = \pi \) is irreducible in \(\Z[i]\). If the second holds, then \(\pi '\) is also irreducible and \(p = \pi \pi '\) is the product of precisely two irreducibles.\\

Now returning to the case of the Gaussian integers, it follows that \(p\) factors in \(\Z[i]\) into precisely two irreducibles if and only if \(p = a^2+b^2\) for \(a,b \in \Z\). Otherwise, it remains irreducible in \(\Z[i]\). If \(p=a^2+b^2\) then the irreducible elements in \(\Z[i]\) are \(a\pm bi\).

\begin{exmp}[Factoring an even prime]
	Clearly \(2 = 1^2+1^2\) and so we get a factorization
	\begin{align*}
		2 = (1+i)(1-i) = -i(1+i)^2
	\end{align*}
	In fact, \((1+i)\) and \((1-i)\) are associates-- this is the only situation in which conjugate irreducibles can be associates.
\end{exmp}
Since \(a^2 \equiv 0 \pmod{4}\) or \(a^2 \equiv 1\pmod{4}\) for any integer \(a \in \Z\), an odd prime in \(\Z\) that satisfies \(p = a^2+b^2\) must be congruent to \(1 \pmod{ 4}\). Hence, if \(p\) is a prime of \(\Z\) and \(p \equiv 3 \pmod{4}\) then \(p\) is irreducible in \(\Z[i]\). In fact, in the first case with \(p \equiv 1 \pmod{4}\), \(p\) must factor into two distinct irreducibles \((a+bi)(a-bi)\).

\begin{lemma}
	The prime number \(p \in \Z\) divides an integer of the form \(n^2+1\) if and only if \(p = 2\) or \(p \equiv 1 \pmod{ 4}\).
\end{lemma}

The Gaussian integers admit a nice Euclidean algorithm that is key to factoring primes further.
\begin{thm}
	The Gaussian integers \(\Z[i]\) form a Euclidean Domain.
\end{thm}
\begin{proof}% Find an alternate proof?
	We will show that \(f(\alpha) = N(\alpha)\) suffices. Observe that
	\begin{align*}
		\alpha = \beta \rho + \theta \iff\\
		\frac{\alpha}{\beta = \rho + \frac{\theta}{\beta}}\iff\\
		\frac{\alpha}{\beta}-\rho = \frac{\theta}{\beta} \iff\\
		\left| \frac{\alpha}{\beta} - \rho \right| < 1
	\end{align*}
	Of course, this is the distance between \(\frac{\alpha}{\beta}\) and \(\rho\). But there always exists a lattice point within distance \(1\) of any \(\C\), and so therefore the statement holds.
\end{proof}

\begin{thm}[Fermat's Theorem on sum of squares]
	The prime \(p\) can be written as \(p = a^2+b^2\) for \(a,b \in \Z\) if and only if \(p = 2\) or \(p \cong 1 \pmod{4}\). The representation of \(p\) as the sum of two squares is unique.
\end{thm}
This is the last key piece we need to classify all prime elements of Gaussian integers.
\begin{thm}[Prime elements of Gaussian integers]
	All Gaussian primes take on the form:
	\begin{itemize}
		\item \(\varepsilon(1+i)\)
		\item \(\varepsilon q\), where \(q\) is prime and \(q \equiv 3 \pmod{4}\)
		\item \(\pi\) where \(N(\pi)\) is a prime with \(N(\pi ) \equiv 1 \pmod{4}\)
	\end{itemize}
	where \(\varepsilon\) is a unit.
\end{thm}
Why is it important to classify all prime elements of \(\Z[i]\)? We will see shortly that this allows us to approach number theory questions by factoring primes in \(\Z[i]\).

\begin{prop}[Disjoint Partitions of Fields]
	Let \(F\) be a field. Then we can partition \(F\) into disjoint sets by taking all sets of the form
	\begin{align*}
		\left\{ a, -a, a^{-1}, (-a)^{-1} \right\}
	\end{align*}
	where \(a\in F\) is non-zero. The union of all sets of this form with \(\left\{ 0 \right\} \) forms a partition of \(F\).
\end{prop}

\begin{thm}[Two Squares Theorem]
Consider the equation \(x^2+y^2 = n\), and let \(n = 2^{\alpha}p_1^{\beta_1}\ldots p_r^{\beta_r}q_1^{\gamma_1}\ldots q_s^{\gamma_s}\) be the Gaussian factorization of \(n\). Then, \(x^2+y^2 = n\) is solvable in \(\Z\) if and only if all \(\gamma_j\) are even. Furthermore, the number of solutions is
\begin{align*}
	4 \prod_{j=1}^{r} (\beta_j + 1)
\end{align*}
\end{thm}
\begin{proof}
	First, we write \(n = x^2+y^2 = (x+yi)(x-yi)\). Using the Gaussian factorization, we rewrite
	\begin{align*}
		n = 2^{\alpha}p_1^{\beta_1}\cdot \ldots\cdot q_1^{\gamma_1}\cdot \ldots = (-i)^{\alpha}(1+i)^{2\alpha}\pi_1^{\beta_1}\overline{\pi_1}^{\beta_1}\cdot \ldots\cdot q_1^{\gamma_1}
	\end{align*}
	Now observe that
	\begin{align*}
		(x+yi)\mid n \implies x+yi = \varepsilon (1+i)^{\alpha'}\pi_1^{\beta_1'}\overline{\pi_1}^{\beta_1''}\cdot \ldots\cdot q_1^{\gamma_1'}\cdot \ldots \\
		\implies x-yi = \overline{\varepsilon}(1-i)^{\alpha'}\overline{\pi_1}^{\beta_1'}\pi_1^{\beta_1'}\cdot \ldots\cdot q_1^{\gamma_1}
	\end{align*}
	Then because
	\begin{align*}
		n = (x+yi)(x-yi)
	\end{align*}
	We have
	\begin{align*}
		  2\alpha = \alpha' + \alpha' \iff\alpha' = \alpha\\
		\beta_1 = \beta_1' + \beta_1'' \iff \beta_1' = 0,1,\ldots,\beta_1; \beta_1'' = \beta_1-\beta_1' \\
		\gamma_1 = \gamma_1' + \gamma_1' \iff \gamma_1 \text{ even}, \gamma_1' = \frac{\gamma_1}{2}\\
		(-i)^{\alpha} = \varepsilon \overline{\varepsilon}(-i)^{\alpha} \iff 1 = \varepsilon \overline{\varepsilon} \text{ which always holds}
	\end{align*}
	Thus, the equation is always solvable if all the \(\gamma\) are even. Looking at the above, the number of solutions will be
	\begin{align*}
		1 \cdot (\beta_j+1) \cdot 1 \cdot 4 = 4 \prod_{j=1}^{r} (\beta_j + 1)
	\end{align*}
\end{proof}

%\subsection{Fermat's Last Theorem}
%\label{subsec:fermat_s_last_theorem}
%\begin{thm}[Fermat's Last Theorem]
%	Let \(n\geq 3\). Does \(x^{n}+y^{n}=z^{n}\) have positive integer solutions?
%\end{thm}
%It is clear that if it is true for \(n  = 4\), \(n= p\) prime, then it holds, as of course it will hold for any multiples. We can rewrite this as
%\begin{align*}
%	x^{p} = z^{p}-y^{p}
%\end{align*}
%\(y\) is a parameter, so the roots are
%\begin{align*}
%	z^{p} = y^{p} \implies z = (y^{p})^{\frac{1}{p}} = z, z\rho, z\rho^2,\ldots,z\rho^{p-1} \text{ where } \rho = \cos \frac{\pi}{p} + i \sin^2 \frac{2\pi}{p}
%\end{align*}
%So we can rewrite this as
%\begin{align*}
%	x^{p} = z^{p}-y^{p} = (z-y)(z-\rho y)\ldots(z-\rho y^{p-1})
%\end{align*}
%Observe that each prime must be a \(p\)-th power as they cannot share factors. Let
%\begin{align*}
%	H_p = \left\{ a_0 + a_1\rho + \ldots + a_{p-2}\rho^{p-2} \right\}, a_j \in \Z \\
%	\rho^{p-1}+\rho^{p-2} + \ldots + \rho + 1 = 0
%\end{align*}
%If the factors on the RHS are pairwise coprime, then \(z-y = \varepsilon_0 \theta_0^{p}\), \(z-y\rho = e_1 \theta_1^{p}\)
%BUT the units are non-trivial for these types of coefficients, AND we don't have UFT.\\
%
%Then Kummer did a different direction. Note that ~"if there is a gcd then it is UFT" (not exactly, but dw about it). Then consider for \(a,b \in \Z\)
%\begin{align*}
%	\textrm{gcd}(a,b) = d \implies d = au + bv
%\end{align*}
%Consider the set
%\begin{align*}
%	\left\{ak + bl \mid k,l \in \Z \right\} = \left\{dn \mid n \in \Z \right\}
%\end{align*}
%Now consider
%\begin{align*}
%	\left\{ak + bl \mid k,l \in H_p \right\}
%\end{align*}
%If \( \exists \textrm{gcd}(a,b) \implies\) this set is the set of multiples of gcd. So for "ideal numbers" UFT holds and the proof holds. So for "ideal numbers" UFT holds and the proof holds.

% \printindex
\end{document}


\chapter{Parametric Assumptions}

This chapter covers the implementation of our approach for
analyzing parametric assumptions.
We support all assumptions satisfying the following restrictions:
\begin{enumerate}
\item The group setting models a symmetric leveled $k$-linear map.
  Note that if $k=2$, there are no intermediate levels and
  the source (resp. target) groups is the sole group on level $1$ (resp. $2$).
  For the non-leveled $k$-linear case, the intermediate groups
  are irrelevant if the input and challenge are all located on
  level $1$ and level $k$ (the target group).
\item The problem is a computational problem or a decisional
 problem. Decisional problems must be real-or-random and
 the challenge must be in the target group.
\item The input given to the adversary must be a monomial.
\item The challenge must be a monomial.\footnote{
    Note that everything generalizes to polynomials $C$ since
    the method can be applied to each monomial in $C$.}
\end{enumerate}%
%
\newcommand{\range}[1]{r_{#1} \in [\alpha_{#1},\beta_{#1}]}%
\renewcommand{\brack}[1]{[#1]}%
%
\label{assumption_def}%
For a a symmetric leveled $k$-linear map,
  random variables $\vec{X}$ and range limits $\vec{l}$,
  the adversary input is of the following form
\begin{align*}
  I_1 :={}& \forall \range{1,1},\ldots,\range{1,w_1}:\, \vec{X}^{\vec{f_{1}}}
    & \text{ in group }\lambda_1\\
  \ldots & \\
  I_n :={}& \forall \range{n,1},\ldots,\range{n,w_n}:\, \vec{X}^{\vec{f_{n}}}
    & \text{ in group }\lambda_n
\end{align*}
  and the challenge is of the form
  $C := \vec{X}^{\vec{g}} \text{ in group }\lambda$.
We assume the following:
\begin{description}
\item[WF1] $f_{j,i} \in \mathbb{Z}[k,\vec{l},r_{j,1},\ldots,r_{j,w_j}]$,
          no overshadowing of range indices, and
          $\alpha_{j,i},\beta_{j,i} \in \mathbb{Z}[\vec{l}]$
\item[WF2] $g_i \in \mathbb{Z}[k,\vec{l}]$
\item[WF3] $\lambda_i$ is either a positive integer or
  of the form $k - i$ for a positive integer $i$
\item[WF4] $\lambda=k$ for decisional problems
\end{description}
%
Note that we always assume that $\alpha_i \leq \beta_i$ since
  we assume that all ranges are non-empty and increasing.
We also assume that $k > i$ for all such levels occuring in the
  assumption.


\chapter{Demosaicing}
\label{chap:appendix_demosaicing}
In this chapter we describe a method allows to address the problem of demosaicing a mosaiced image. It is based on a convex energy that is optimized by using a primal-dual solver. \\ \\
However, please notice that this method has not been implemented while writing this thesis. This method, as well as all the derivations below were derived during a semester project in the class of $\textit{Convex Optimization}$ held by Prof. Dr. P. Favaro in 2015. \\ \\
The complete report as well as the corresponding source code to run this solver can be found at: $\text{\url{https://github.com/simplay/convex_opt}}$.

\section{Problem Statement}
\label{sec:demosaicing_problem_statement}
In this work we address the problem of demosaicing by formulating it as a convex optimization problem. Given a mosaiced image $g$ depicting a raw camera production, we want to find an optimal demosaiced image $u$. For describing the optimality property we take into account a cost function that describes the color smoothness and also that the resulting image $u$ should be close to the given raw input image $g$. \\ \\
More precisely, let $g$ denote the bayer filter camera raw input image. Then we want to solve for $u=(u_r, u_g, u_g)$ (RGB image) minimzating the following energy term (cost function):
\begin{align}
	E(u_c) = \norm{\nabla u_c}_2 + \frac{\lambda}{2} \norm{u_c - g}^2_{\Omega_{c}}
\label{eq:basis_cost_demosaicing}
\end{align}
with the measure
\begin{equation}
	\norm{u_C - g}^2_{\Omega_{C}} = \sum_x \sum_y \Omega_{C}(x,y)\norm{u_{c}(x,y) - g(x,y)}^2
\label{eq:measure}
\end{equation}
where C denotes the three different color channels. $\Omega_{C}$ is defined such that $\Omega_{C}(x,y) = 1$ if the pixel value at $(x,y)$ is \emph{valid}$\footnote{this means that the pixel at location (x,y) is valid for the bayer color mask C}$ and $\Omega_{C}(x,y) = 0$ when the data is missing. \\ \\
The cost function from Equation $\ref{eq:basis_cost_demosaicing}$ consists of a smoothness term, $\norm{\nabla u_c}_2$ and $\norm{u_c - g}^{2}_{\Omega_{c}}$. The first term ensures a smooth color transition between colors in a l2 norm sense. The second term ensures that the reconstructed images does not deviate too much from the given input, i.e. de demosaiced image should resemble to the provided mosaiced raw camera image. This similarity term is further parameterized by a regularization term $\lambda$, indicating how strong the output should match the given input according to the formulated measure in Equation $\ref{eq:measure}$. In summary, larger values for $\lambda$ weight the similarity of the input and output image more, and contrarely, lower values weight the color smoothness term more. \\ \\
Hereby, minimizating the cost function from Equation $\ref{eq:basis_cost_demosaicing}$ leads to an optimal demosaiced image $u$.
Mathematically we want to solve for
\begin{equation}
	\widetilde{u} = \argmin_{u_c} E(u_c)
\label{eq:our_general_cost_function}
\end{equation}
We can further simplify the cost function stated in Equation $\ref{eq:basis_cost_demosaicing}$ relying on the following observation: Since the function $\Omega_{C}$ is only true for pixels that correspond to the color channel C in the bayer mask, we see that $\Omega_{C}(x,y)\norm{u_{c}(x,y) - g(x,y)}$ is only not equal zero if the pixel at location $(x,y)$ belongs to the color channel $C$. Therefore we are allowed to solve the stated optimization problem from Equation $\ref{eq:basis_cost_demosaicing}$ for each color channel separately. \\ \\
According to this insight we are supposed to minimize the following three independent$\footnote{Independent in the sense that we are allowed to solve for each color channel separately}$ convex problems:
\begin{align}
	\widetilde{u}_R = \argmin_{u_R} \norm{\nabla u_R}_2 + \frac{\lambda}{2} \norm{u_R - g}^2_{\Omega_{R}} \nonumber \\
	\widetilde{u}_G = \argmin_{u_G} \norm{\nabla u_G}_2 + \frac{\lambda}{2} \norm{u_G - g}^2_{\Omega_{G}}\nonumber \\
	\widetilde{u}_B = \argmin_{u_B} \norm{\nabla u_R}_2 + \frac{\lambda}{2} \norm{u_B - g}^2_{\Omega_{B}}
\label{eq:our_convex_probelm}
\end{align}
Where we still rely on the measure defined in Equation $\ref{eq:measure}$ but C was replayed by the appropriate color channel$\footnote{C stands for either the color channel R, G or B.}$. We notice that the equations in Equation $\ref{eq:our_convex_probelm}$ tell us that we have to solve three different energies similar to the one formulated in Equation $\ref{eq:our_general_cost_function}$. \\ \\
In the next section we will describe how to solve the stated minimization problems from Equation $\ref{eq:our_convex_probelm}$ numerically. \\ \\

\section{Primal-Dual Form}
\label{sec:primal_dual_form}
In this section we derive the primal-dual form of the stated convex demosaicing problem. But first off, let us consider an initial problem of the form
\begin{align}
	\min_{x \in X} F(K x) + G(x)
\label{eq:initial_primal}
\end{align}
where $F$, $G$ are convex functions and $K$ denotes a linear operator. The primal-dual formulation for Equation $\ref{eq:initial_primal}$ is given by
\begin{align}
	\min_{x \in X} \max_{y \in Y} < Kx, y > - F^*(y) + G(x)
\label{eq:initial_primal_dual}
\end{align}
For a given mosaiced RGB image $u_{RGB}$ encoded as a 3 dimensional $M \times N$ matrix, i.e. a tensor of dimension $M \times N \times 3$. As mentioned in the problem statement we can solve three independent convex problems in order to solve the problem of demosaicing a RGB image. Therefore let in the following $u$ define stand for one particular color channel of the given color image $u_{RGB}$.
\begin{equation}
\min_{u \in U} \norm{\nabla u} + \frac{\lambda}{2} \norm{u - g}^2_{\Omega}
\label{eq:initial_energy}
\end{equation}
where $\norm{u - g}^2_{\Omega}$ is defined as in Equation $\ref{eq:measure}$ and $g$ is the corresponding color channel of the mosaiced image described in the problem statement. \\ \\
We observe that Equation $\ref{eq:initial_energy}$ has the same structure as the initial problem stated in Equation $\ref{eq:initial_primal}$. This allows us to formulate the primal-dual form of Equation $\ref{eq:initial_energy}$ which will look like the following:
\begin{align}
	\min_{u \in U} \max_{y \in Y} < Kx, y > - F^*(y) + G(x)
\label{eq:initial_primal_dual}
\end{align}
Where $K$, $F$ and $G$ are defined as:
\begin{align}
	K &= \nabla \nonumber \\
	F &= \norm{\cdot}_2 \nonumber \\
	G &= \norm{u - g}^2_{\Omega}
\label{eq:def_kfg}
\end{align}
Note that $F*$ denotes the convex conjugate form of $F$. The convex conjugate of $F$ has an explicit identity that can be computed using the Legendre-Fenchel-Transform.
\begin{align}
	F^*(y) &= (\norm{\cdot}_2)^*(y) \nonumber \\
		  &= \sup_x x^T y - \twonorm{x} \nonumber \\
		  &= \sup_x x^T y - \max_{\twonorm{z} \leq 1} x^T z \nonumber \\
		  &= \sup_x \min_{\twonorm{z} \leq 1} x^T(y-z) \nonumber \\
		  &= \begin{cases}
   				0  			& \text{if} \twonorm{y} \leq 1 \\
   				\infty      & \text{otherwise}
  			 \end{cases} \nonumber \\
  		  &= \delta(y)
\label{eq:legendre_fenchel_transform_f}
\end{align}
The first equality is simply the definition of $F$. The second equality is using the so called Legendre-Fenchel transformation,
\begin{equation}
	(\norm{\cdot}_2)^*(y) = \sup_x x^T y - \twonorm{x} \nonumber
\end{equation}.
In the third equality I make use of the Cauchy-Schwarz inequality,
\begin{equation}
	\twonorm{x} = \max_{\twonorm{z} \leq 1} x^T z
\end{equation}
Plugging Equation $\ref{eq:legendre_fenchel_transform_f}$ and the definitions in from Equation $\ref{eq:def_kfg}$ into the primal-dual Equation $\ref{eq:initial_primal_dual}$ we conclude the following final primal-dual formulation:
\begin{equation}
\min_{u \in U} \max_{y \in Y} <\nabla u, y> - \delta(y) + \frac{\lambda}{2}\norm{u - g}^2_{\Omega}
\label{eq:final_primal_dual}
\end{equation}

\section{Primal-Dual steps}
In this section I will present an iterative solver for our stated primal-dual formulation. \\ \\
In the following I am going to rely on an algorithm formulated by A.Chambolle and T.Pocke which allows to solve primal-formulations as ours stated in Equation $\ref{eq:final_primal_dual}$. \\ \\
They stated an iterative algorithm that has the following update steps:
\begin{align}
	y^{n+1} &= \prox_{\sigma F^*}(y^n + \sigma K \bar{x}^n) \nonumber \\
	x^{n+1} &= \prox_{\tau G}(x^n - \tau K^* y^{n+1}) \\
	\bar{x}^{n+1} &= x^{n+1} + \theta(x^{n+1} - x^n)
\label{eq:update_rules_plain}
\end{align}
with $\theta \in (0, 1]$ and the constraint $\tau \sigma \norm{K}^2 < 1$. Note that stated constraint is important in order to guarantee convergence of their algorithm. \\ \\
Hereby $prox(\cdot)$ denotes the proximity operator and is defined as
\begin{equation}
	\prox_{\lambda F}(z) = \arg \min_x \frac{1}{2} \twonorm{x - z}^2 + \lambda F(x)
\end{equation}
In the following we will derive explicit identities for the update rules in Equation $\ref{eq:update_rules_plain}$that can be numerically solved. Our goal is to find an expression for the proximity operator.

\subsection{Update for $y^{n+1}$}
\label{sec:pd_yn}
In this subsection we derive an identity for the $y^{n+1}$ update rule from Equation $\ref{eq:update_rules_plain}$. The key idea is to use the so called Moreau's Identity:
\begin{equation}
	\prox_{\lambda F^*}(z) = z - \lambda \cdot \prox_{F/ \lambda}(z / \lambda)
\label{eq:moreau}
\end{equation}
Next, we apply the Moreau's identity to the proximity operator of the Legendre-Fenchel transformation.
\begin{align}
	\prox_{\lambda F^*}(y^n + \sigma K \bar{x}^{n})
	&= (y^n + \sigma K \bar{x}^{n}) - \sigma \prox_{\frac{F}{\sigma} } \left(\frac{y^n + \sigma K \bar{x}^{n} }{\sigma} \right) \nonumber \\
	&= (y^n + \sigma K \bar{x}^{n}) - \sigma \left( \frac{y^n + \sigma K \bar{x}^{n} }{\sigma} \max{0, 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n} }}} \right) \nonumber \\
	&= (y^n + \sigma K \bar{x}^{n}) - \left( y^n + \sigma K \bar{x}^{n} \right) \max{\left(0, 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n} }}\right)}
\label{eq:y_n_1_expression}
\end{align}
For the first equality we use the definition of Equation $\ref{eq:moreau}$ and for the second equality we used the fact (proven during class) that
\begin{align}
	\prox_{\frac{\twonorm{\cdot}}{\sigma}}(\frac{x}{\sigma}) = \frac{x}{\sigma} \max{\left(0, 1-\frac{1}{\twonorm{x}}\right)}
\end{align}
To simplify our derivation even and also get rid of the proximity operator we next make a case distinction for $\twonorm{y^n + \sigma K \bar{x}^{n}}$.
\begin{itemize}
	\item If $\twonorm{y^n + \sigma K \bar{x}^{n}} \geq 1$
		then
		\begin{align}
			0 \leq 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}} \leq 1
		\end{align}

		Therefore $\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}}$ is smaller than one and thus

		\begin{align}
			\max{\left(0, 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}}\right)}
			&= 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}}
		\end{align}

		This insight can directly be used for the maximum expression in Equation $\ref{eq:y_n_1_expression}$ and we hence obtain:

		\begin{align}
			\prox_{\lambda F^*}(y^n + \sigma K \bar{x}^{n})
			&= (y^n + \sigma K \bar{x}^{n}) - \left( y^n + \sigma K \bar{x}^{n} \right) \max{\left(0, 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n} }}\right)} \nonumber \\
			&= (y^n + \sigma K \bar{x}^{n}) - \left( y^n + \sigma K \bar{x}^{n} \right) \left( 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}} \right)\nonumber \\
			&= (y^n + \sigma K \bar{x}^{n}) -(y^n + \sigma K \bar{x}^{n}) +\frac{y^n + \sigma K \bar{x}^{n}}{\twonorm{y^n + \sigma K \bar{x}^{n}}} \nonumber \\
			&= \frac{y^n + \sigma K \bar{x}^{n}}{\twonorm{y^n + \sigma K \bar{x}^{n}}}
		\end{align}

	\item If $\twonorm{y^n + \sigma K \bar{x}^{n}} < 1$
		then
		\begin{align}
			1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}} < 0
		\end{align}
		thus we conclude
		\begin{align}
			\max{\left(0, 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n}}}\right)}
			&= 0
		\end{align}
		which offers us the following new identity for Equation $\ref{eq:y_n_1_expression}$:

		\begin{align}
			\prox_{\lambda F^*}(y^n + \sigma K \bar{x}^{n})
			&= (y^n + \sigma K \bar{x}^{n}) - \left( y^n + \sigma K \bar{x}^{n} \right) \max{\left(0, 1-\frac{1}{\twonorm{y^n + \sigma K \bar{x}^{n} }}\right)} \nonumber \\
			&= (y^n + \sigma K \bar{x}^{n}) - \left( y^n + \sigma K \bar{x}^{n} \right) 0 \nonumber \\
			&= y^n + \sigma K \bar{x}^{n}
		\end{align}
\end{itemize}
By using the results from the case distinction from above we can simplify Equation $\ref{eq:y_n_1_expression}$ even further to:
\begin{equation}
	\prox_{\lambda F^*}(y^n + \sigma K \bar{x}^{n}) = \frac{y^n + \sigma K \bar{x}^{n}}{\max{\left(1,\twonorm{y^n + \sigma K \bar{x}^{n}} \right)}}
\label{eq:y_p_1_we_proxy}
\end{equation}
Finally, the only left step to do is to plug in the definition of $K$ into Equation $\ref{eq:y_p_1_we_proxy}$ which gives us then the final update rule for $y_{n+1}$ when relying on the update rule from Equation $\ref{eq:y_n_1_expression}$:
\begin{align}
	y_{n+1} = \frac{y^n + \sigma \nabla \bar{x}^{n}}{\max{\left(1,\twonorm{y^n + \sigma \nabla \bar{x}^{n}} \right)}}
\label{eq:update_rule_y_n_p_1}
\end{align}

\subsection{Update for $x^{n+1}$}
\label{sec:pd_xn}
\begin{align}
x^{n+1} &= \prox_{\tau G}(x^n - \tau K^* y^{n+1}) \\
		&= \prox_{\tau \frac{\lambda}{2} \norm{u - g}_{\Omega}^2 }(x^n - \tau \nabla^* y^{n+1}) \\
	    &= \arg \min_{z} \frac{1}{2} \twonorm{\left(x^n - \tau \nabla^* y^{n+1} \right) - z}^2 + \tau \frac{\lambda}{2}\norm{z - g}_{\Omega}^2 \\
	    &= \arg \min_{z} E(z)
\label{eq:energy_x_p_1}
\end{align}
To simplify the following derivations, let us define the following substitution:
\begin{align}
	m := \left(x^n - \tau \nabla^* y^{n+1} \right)
\end{align}
We can solve for $x^{n+1}$ by finding the zeros of the partial derivative of $E(z)$ from Equation $\ref{eq:energy_x_p_1}$. Let us start with the partial derivative along $z$ of $E(z)$ from Equation $\ref{eq:energy_x_p_1}$:
\begin{align}
	\partial_{z} E(z)
	&= \partial_{z} \left( \frac{1}{2} \twonorm{m - z}^2 + \tau \frac{\lambda}{2}\norm{z - g}_{\Omega}^2 \right) \nonumber \\
	&= \frac{1}{2} \partial_{z} \left[ \left( m - z \right)^{T}\left( m - z \right) + \tau \lambda \Omega \left( z -g \right)^{T}\left( z -g \right) \right] \nonumber \\
	&= \frac{1}{2} \partial_{z} \left[ m^{T}m -2m^{T} z + z^{T} z + \tau \lambda \Omega \left( z^{T}z -2z^{T} g + g^{T} g\right) \right] \nonumber \\
	&= \frac{1}{2} \left[ -2m + 2z + \tau \lambda \Omega \left( 2 z -2g \right) \right] \nonumber \\
	&= \left[ -m + z + \tau \lambda \Omega z - \tau \lambda \Omega g \right]	 \nonumber \\
	&= \left[ \left(1+\tau \lambda \Omega \right)z-m - \tau \lambda \Omega g \right]	 \nonumber \\
\label{eq:derivative_x_n_p_1}
\end{align}
Next, let us set the finding from Equation $\ref{eq:derivative_x_n_p_1}$ to zero and solve for $z$:
\begin{align}
	\partial_{z} E(z)
	&= 0 \nonumber \\
	&\Leftrightarrow \left(1+\tau \lambda \Omega \right)z-m - \tau \lambda \Omega g = 0 \nonumber \\
	&\Rightarrow z = \left(m +  \tau \lambda \Omega g \right) \left( 1+\tau \lambda \Omega\right)^{-1} \nonumber \\
	&\Rightarrow z = \frac{m +  \tau \lambda \Omega g}{1+\tau \lambda \Omega} \nonumber \\
\label{eq:zeros_ez}
\end{align}
Note that the division $(1+\tau \lambda \Omega)$ denotes a component-wise division, since $\Omega$ is applied component-wise to the elements of $g$. In addition, 1 and $\Omega$ are representing matrices here (of same dimension as $g$ and $z$ ($u$ respectively).
By plugging the definition $m$ into Equation $\ref{eq:zeros_ez}$ and using the fact, that $z$ corresponds to $x^{n+1}$ we can conclude:
\begin{align}
	x^{n+1}
	&= \frac{x^n - \tau \nabla^* y^{n+1} +  \tau \lambda \Omega g}{1+\tau \lambda \Omega} \nonumber \\
	&= \frac{x^n + \tau div(y^{n+1}) +  \tau \lambda \Omega g}{1+\tau \lambda \Omega}
\label{eq:update_x_n_p_1}
\end{align}
In the last step we used the well known fact, that
\begin{align}
	\nabla^* (v) = -div(v)
\end{align}
for any vector-field $v$ of the form
\begin{align}
	v = \nabla u
\end{align}
In the next section I explain how I used the derived update rules in my actual implementation and what parameter values I have used.

\section{Implementation}
In this section I how I have actually Implemented the so far described dual-primal solver for demosaicing a raw image. One important note in advance. In the discrete case, the following holds true
\begin{align}
	\nabla^* (y^{n+1})
	&= \nabla^{T} (y^{n+1}) \nonumber \\
	&= div(y^{n+1})
\end{align}
So we have to omit a minus one factor. This affects the update rule for $x^{n+1}$ derived previously. \\ \\
In the previous section we have defined explicit update rules. Aggregating all finding, mainly those from Equation $\ref{eq:update_x_n_p_1}$ and Equation $\ref{eq:update_rule_y_n_p_1}$, and plugging them into Equation $\ref{eq:update_rules_plain}$ we get our update rules
\begin{align}
	y^{n+1} &= \frac{y^n + \sigma \nabla \bar{x}^{n}}{\max{\left(1,\twonorm{y^n + \sigma \nabla \bar{x}^{n}} \right)}} \nonumber \\
	x^{n+1} &= \frac{x^n - \tau div(y^{n+1}) +  \tau \lambda \Omega g}{1+\tau \lambda \Omega} \\
	\bar{x}^{n+1} &= x^{n+1} + \theta(x^{n+1} - x^n)
\label{eq:final_update_rules_plain}
\end{align}
I initialized $x_n$ with the mosaiced image $g$, $y^{n}$ with a zeros \footnote{a tensor of dimension $M \times N \times 2$ filled with zeros, where $(M \times N)$ denotes the dimension of one color channel of $g$.} and $\bar{x}^{n}$ also with the mosaiced image $g$. \\ \\
For computing $\nabla$ I used a forward difference approximation scheme. For computing the divergence operator of the vector-field $y^{n+1}$ I used backward difference approximation scheme. The reason for using a backward difference using a backward difference is to shift back gradients (remember, the divergence is applied to $y^{n+1}$ which is the result of a forward difference. Otherwise, when not altering between a forward-and backward difference we would end up with shifted gradients. \\ \\
For computing the divergence, I relied on its mathematical definition. For a given vector-field $v = (v_x, v_y)$ the divergence is defined as the following:
\begin{align}
	div(v) = \partial_x v_x + \partial_y v_y
\end{align}
Since in our case we have $v = y^{n+1}$ and $y^{n+1}$ a vector valued function of the form $y^{n+1} = (y_{x}^{n+1}, (y_{y}^{n+1})$ it follows:
\begin{align}
	div(y^{n+1})
	&= div((y_{x}^{n+1}) + div((y_{y}^{n+1}) \\
	&= \left( \partial_x y_{x}^{n+1} + \partial_y y_{x}^{n+1} \right) + \left( \partial_x y_{y}^{n+1} + \partial_y y_{y}^{n+1} \right)
\end{align}
Initially, I used the following parameter setting:
\begin{align}
	\lambda &= 1000 \\
	\theta &= 0.5 \\
	\tau &= 2*10^{-3} \\
	\sigma &= \frac{1}{\tau * \sqrt{\norm{K}}}
\label{eq:parameter_set_up}
\end{align}
With $\sqrt{\norm{K}} = \sqrt{4}$, a strong upper bound for the function $K$\footnote{For further information about this upper bound please have a look at Chambolle, Antonin: An algorithm for total variation minimisation and application. In Journal of Mathematical imaging and vision 20, 2004.}. \\ \\
For consistency, a named all functions in my Matlab code the same as in this report. Furthermore I used a fixed number of iterations for computing my iterative demosaiced images. The final algorithms I have to perform is the following: For each color-channel $C \in \{R,G,B\}$ Do Loop until $\norm{\bar{x}^{n+1} - \bar{x}^{n}}$ is small enough do: use parameter setup as defined in Equation $\ref{eq:parameter_set_up}$ and then solve the update rules from Equation $\ref{eq:final_update_rules_plain}$. Finally, merge all color iterative color channel solutions to a color image. \\ \\
When computing the gradient and divergence finite approximation schemes, I used a zero padding boundary condition. Since I also tried out this kind of boundary condition in the first report, comparing the results produced in this project which those from the first project is valid. \\ \\
One last comment: From the definition of the update rules, we see that the value of $\lambda$ directly affects the parameters $\tau$ and $\sigma$. Thus, when changing the value of $\lambda$ we also would have to find new best $\tau$ and $\sigma$ parameters. Hence, changing $\lambda$ also affects the convergence behaviour of the primal dual method.