\documentclass{article}
\title{Diagonal projections and rank histograms for measuring the distance between estimated copulas and observed data}
\author{Ma\"{e}l Forcier}
\date{May 2017}

\usepackage[utf8]{inputenc}
\usepackage{mathtools}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amsthm}
 \usepackage{relsize}
\usepackage{stmaryrd}
\usepackage{dsfont}
\renewcommand\qedsymbol{$\blacksquare$}

\begin{document}
   \maketitle
   \section{Introduction}
  	To create scenarios for renewable energies production, we need to be able to measure the dependance through space and time or between different sources. To focus on the correlation, we use copulas. Giving a set of datas, different methods exist to choose the best copulas and parameters that will best fit and model our datas. Nevertheless, we need tools to verify and measure how each of the copulas fit our observed data. Moreover, we need a tool that focus on the tails where the dependance matters a lot for us. I will briefly present in section 1, the loglikelihood which is a classical way to measure the overall fit of a distribution. Then, I will describe more precisely a way to measure how a copula fit the datas in the tail using rank histogram on diagonal projections.

   \section{Loglikelihood}
   Present briefly the loglikelihood. Refer to a good paper explaining loglikelihood ?

   \section{Solving the problem of unreproducibility}
	Each day j, we look at a set of data including for example a description of the state of the system early in the morning and the observations of the 90 previous days. With this set of data, we are able to choose a distribution represented by its cumulative density function \begin{math}F_j\end{math} with a set of parameters \begin{math} \theta_j \end{math} that we hope will predict the best what will happen on day j. Some techniques are presented in \cite{vineconstruction} or \cite{fourcopulas} but we will not focus on them in this article. We now want to verify if this distribution was well chosen. \newline
	\newline
   Unfortunately, we only observe what happens on the day j once. Let us note \begin{math}O_j \end{math} the day j observation. \begin{math}O_j \end{math} is not sufficient to check if this random variable follow the distribution \begin{math} F_j \end{math}. Moreover, each day is different and for instance another day \begin{math}i\neq j\end{math}  will give a different distribution \begin{math} F_i \end{math} with different parameters \begin{math} \theta_i \end{math}. Thus, it is impossible to verify if each distribution is correct each day. Nevertheless, their are techniques to verify if our procedure is valid and if the estimation of distributions makes sense. \newline
   \newline

   Let us define \begin{math}U_j = F_{j}(O_{j})\end{math}. It is easy to prove that \begin{math} U_j\end{math} should have a uniform distribution. (This classical result is often used to generate a random variable X following cumulative density function F with uniform random variable U : \begin{math}X = F^{-1}(X)\end{math}.)
   As all \begin{math}U_j\end{math} are computed independantly, all the \begin{math}U_j\end{math} must be independant and distributed uniformly.\newline
   \newline
   We now have a set of observation \begin{math} \textbf{U} = (U_1,..U_n)\end{math} that should be independant and uniformly distributed on [0,1]. We can now compute the extent to which it follows a uniform distribution for example by computing the Earth Mover Distance between the empirical distribution of U and the uniform distribution or by using rank histograms.

	\section{Rank histograms}

   Imagine we have our problem in 1 dimension.
   Give an example in dimension 1.
   Present rank histograms refers to \cite{hamill2000}
   \newline
   \newline
   As one can see in \cite{hamill2000}, the biggest problem with rank histograms is that they are primarly useful only in one dimension. So we have to make projection. Projecting on the marginals is useless because what we care about is the dependance. Thus, we will project on the diagonals. This way we can measure the fit of the copulas in the corner that we are interested in.

	\section{Earth Mover Distance}
	 \newtheorem{definition}{Definition}
	 \begin{definition}
	 The \textbf{Earth Mover Distance} (EMD) between two histograms $P=((x_i,p_i))_i$ and $Q=((y_j,q_j))_j$ is :
	 \begin{equation*}
	 EMD(P,Q) = \frac{\min\limits_{(f_{i,j}) \in F} \sum_{i,j} f_{i,j} d_{i,j}}{ \min(\sum_i p_i, \sum_i q_i)}
	 \end{equation*}
	 \newline
	 where \begin{math}F=\{(f_{i,j})| f_{i,j}\geq 0,\sum\limits_{i} f_{i,j} \leq P_i, \sum\limits_{j} f_{i,j} \leq Q_j, \sum\limits_{i,j} f_{i,j} = \min(\sum\limits_i p_i, \sum\limits_i q_i)\} \end{math}\newline
	 and $d_{i,j}$ is the distance between $x_i$ and $y_j$.
	\end{definition}

	In our problem, we look at histograms of distribution with real density function. So, with probability 1 we will not have the same result twice. Thus, the weight of our histograms will just be 1 for each value : $\forall i, q_i =1, \forall j, p_j=1$.\newline
	Moreover, we will only consider vectors with same dimension n. We can now simplify the notation and define the EMD between two vectors :

	\begin{definition}
	 The \textbf{Earth Mover Distance} (EMD) between two vectors $\textbf{u}=(u_1,...,u_n)$ and $\textbf{v}=(v_1,...,v_n)$ of dimension n is :
	 \begin{equation*}
	 EMD(\textbf{u},\textbf{v}) = \frac{1}{n}\min\limits_{(f_{i,j}) \in F} \sum_{i=1}^n \sum_{j=1}^n f_{i,j} d_{i,j}
	 \end{equation*}
	 \newline
	 where \begin{math}F=\{(f_{i,j})| f_{i,j}\geq 0,\sum\limits_{i=1}^n f_{i,j} \leq 1, \sum\limits_{j=1}^n f_{i,j} \leq 1, \sum\limits_{i=1}^n \sum\limits_{j=1}^n f_{i,j} = n\}\end{math}
	 \newline
	 and $d_{i,j}=|u_i-v_j|$
	\end{definition}

	Solving this linear program is possible but there is a faster way to compute the EMD thanks to the following property :

	\newtheorem{property}{Property}
	\begin{property}
	For any vectors \textbf{u} and \textbf{v} of dimension n :
	\begin{equation*}
	EMD(\textbf{u},\textbf{v}) = \frac{1}{n} \sum_{i=1}^n |\tilde{u}_i - \tilde{v}_i| =  \frac{1}{n}||\tilde{\textbf{u}}-\tilde{\textbf{v}}||_1
	\end{equation*}
	where $\tilde{\textbf{x}}$ is the sorted vector of $\textbf{x}$ :
	\newline
	$\{x_1,...,x_n\}=\{\tilde{x}_1,...,\tilde{x}_n\}$ and $\forall i \leq j, \tilde{x}_i \leq \tilde{x}_j$\newline
	\end{property}

	\textbf{Demonstration :}\newline
	\newline
	First we have :
	\begin{multline*}
	\begin{split}
	F	&= \{(f_{i,j})| f_{i,j}\geq 0,\sum\limits_{i=1}^n f_{i,j} \leq 1, \sum\limits_{j=1}^n f_{i,j} \leq 1, \sum\limits_{i=1}^n \sum\limits_{j=1}^n f_{i,j} = n\} \newline \\
		&= \{(f_{i,j})| f_{i,j}\geq 0,\sum\limits_{i=1}^n f_{i,j} = 1, \sum\limits_{j=1}^n f_{i,j} = 1 \}
	\end{split}
	\end{multline*}
	The way $\supset$ is trivial.\newline
	\newline
	Let be $(f_{i,j})_{i,j} \in F$ and let us suppose that $\exists i_0, \sum\limits_{i=1}^n f_{i_0,j} < 1 : \sum\limits_{i=1}^n f_{i_0,j} = 1-\epsilon $.\newline
	Thus, $\sum\limits_{i=1}^n \sum\limits_{j=1}^n f_{i,j} \leq 1-\epsilon +\sum\limits_{i=1,i\neq i_0}^n 1= n-\epsilon < n $\newline
	$(f_{i,j})_{i,j} \notin F$ : Contradiction \newline
	\newline
	We will now demonstrate that it exists $f_{i,j}$ integers that solve the minimum problem. This a classical demonstration using several theorems. For any precision, see the chapter 2 of \cite{gaubert}.\newline

	Let us define $M = (M_{k,(i,j)}) \in \mathcal{M}_{2n,n^2}(\mathbb{R})$ with :\newline
	\[
   		M_{k,(i,j)} =  \begin{cases}
        1  & \quad \text{if } k=i \\
    	-1 & \quad \text{if } k=j+n\\
    	0 & \quad \text{else}\\
  	\end{cases}
  	\]
	Because $1\leq i,j \leq n$ the cases are incompatible.\newline
	\newline
	We now have :\newline
	\begin{equation*}
	F = \{(f_{i,j})| f_{i,j}\geq 0, Mf = B \}
	\end{equation*}
	where $B =
 	\begin{pmatrix}
 	 1  \\
 	 \vdots \\
  	1 \\
  	-1 \\
  	\vdots  \\
  	-1
 	\end{pmatrix}$\newline
 	\newline
 	The coefficient of M are just -1,0 and 1 and
	M has just once 1 and once -1 on each of its column. \newline
	Thanks to the Poincaré lemma (see \cite{gaubert} chapter 2), we can say that M is totally unimodulary. Because B has integer coefficient, we can say that all extreme points of F have integer coefficients.\newline
	So, we finally have it exists $f_{i,j}$ integers that solve the minimization problem.\newline
	\newline
	Let $(f_{i,j})_{i,j}$ solve the mnimization problem with integer coefficients.\newline
	$\forall i, \forall j, 0\leq f_{i,j} \leq 1$ and $f_{i,j}\in \mathbb{N} \Rightarrow f_{i,j} =0$ or $ f_{i,j} =1$ \newline
	$\forall i, \sum\limits_{j=1}^n f_{i,j} = 1 \Rightarrow \exists ! j_0,  f_{i,j_0} =1$\newline
	$\forall j, \sum\limits_{i=1}^n f_{i,j} = 1 \Rightarrow \exists ! i_0,  f_{i_0,j} =1$\newline
	\newline
	We now have :
	\begin{equation*}
	\exists \sigma \in \mathfrak{S}_n, f_{i,j} =  \begin{cases}
        1  & \quad \text{if } j=\sigma(i) \\
    	0 & \quad \text{else}\\
  	\end{cases}
	\end{equation*}
	So,
	\begin{equation*}
	EMD(\mathbf{u},\mathbf{v}) = \min\limits_{\sigma \in \mathfrak{S}_n} \sum_{i=1}^n d_{i,\sigma{i}} = \min\limits_{\sigma \in \mathfrak{S}_n} \sum_{i=1}^n |u_i-v_{\sigma (i)}|
	\newline
	\newline
	\end{equation*}

	We now have to prove that this min is reached when $(u_i)_i$ is sorted in the same order than $(v_{\sigma (i)})_i$. Since the indexes have symetric roles and are just notations indicating coefficients, we can consider that u and v are sorted :\newline
  $\forall i \leq j, u_i \leq u_j, v_i \leq v_j$\newline
  With this notation, we need to prove that this minimum is reached for $\sigma = Id$.\newline
  \newline
  Suppose that $\sigma \neq Id$ reaches the minimum, so $supp(\sigma)\neq \emptyset$ and $supp \sigma$ contains at least two elements. Let us define:

  \begin{multline*}
	\begin{split}
	q	&=  \max supp(\sigma)\\
	p	&=  \sigma^{-1}(q) \\
	r 	&= \sigma (q)
	\end{split}
	\end{multline*}

	We have $q\leq p$ and $r\leq p$, so $u_q\leq u_p$ and $v_r\leq v_p$.

	\begin{itemize}
	\item Case 1 : $u_p \leq u_q \leq v_r \leq v_q$
	\end{itemize}

	\begin{multline*}
	\begin{split}
	|u_p-v_q| + |u_q-v_r|	&=  v_q-u_p+v_r-u_q\\
		&=  v_r-u_p+v_q-u_q \\
	 	&= |u_p-v_r|+|u_q-v_q|
	\end{split}
	\end{multline*}

	\begin{figure}
      \includegraphics[width=0.5\textwidth]{demo_sort1.png}
    \caption{Graphic representation of case 1 : \emph{The upper black line represents \textbf{v} and the lower one \textbf{u}. The red points are from left to right $v_r$ and $v_q$ and The green points are from left to right $u_p$ and $u_q$. The blue lines represent what the cost when using $\sigma$ and the green lines  the cost when using $\tilde{\sigma}$. } }
\end{figure}

\begin{itemize}
	\item Case 2 : $u_p\leq v_r \leq u_q \leq v_q$
	\end{itemize}

	\begin{multline*}
	\begin{split}
	|u_p-v_q| + |u_q-v_r|	&\leq  |u_p-v_q|\\
		&=  v_q-u_p \\
	 	&= v_q-u_q+u_q-u_p\\
	 	&\leq v_q-u_q+v_r-u_p\\
	 	&= |u_q-v_q|+|u_p-v_r|
	\end{split}
	\end{multline*}

\begin{figure}

    \includegraphics[width=0.5\textwidth]{demo_sort2.png}
     \caption{Graphic representation of case 2 : \emph{The upper black line represents \textbf{v} and the lower one \textbf{u}. The red points are from left to right $v_r$ and $v_q$ and The green points are from left to right $u_p$ and $u_q$. The blue lines represent what the cost when using $\sigma$ and the green lines  the cost when using $\tilde{\sigma}$. } }
\end{figure}

\begin{itemize}
	\item Case 3 : $u_p\leq v_r \leq v_q \leq u_q$
	\end{itemize}

\begin{figure}

    \includegraphics[width=0.5\textwidth]{demo_sort3.png}
     \caption{Graphic representation of case 3 : \emph{The upper black line represents \textbf{v} and the lower one \textbf{u}. The red points are from left to right $v_r$ and $v_q$ and The green points are from left to right $u_p$ and $u_q$. The blue lines represent what the cost when using $\sigma$ and the green lines  the cost when using $\tilde{\sigma}$. } }
\end{figure}

	\begin{multline*}
	\begin{split}
	|u_p-v_q| + |u_q-v_r|	&=  v_q-u_p+u_q-v_r\\
		&=  u_q-u_p+v_q-v_r \\
	 	&\leq u_q-u_p\\
	 	&= u_q-v_q+v_q-u_p\\
	 	&\leq u_q-v_q+v_r-u_p\\
	 	&= |u_q-v_q|+|u_p-v_r|
	\end{split}
	\end{multline*}

	\begin{itemize}
	\item Case 4 : $v_r\leq v_q \leq u_p \leq u_q$\newline
	Same as case 1 by inversing the symetric roles of \textbf{u} and \textbf{v}.

	\item Case 5 : $v_r\leq u_p \leq v_q \leq u_q$\newline
	Same as case 2 by inversing the symetric roles of \textbf{u} and \textbf{v}.

	\item Case 6 : $v_r\leq u_p \leq u_q \leq v_q$\newline
	Same as case 3 by inversing the symetric roles of \textbf{u} and \textbf{v}.
	\end{itemize}

In all this cases, we have :
\begin{equation*}
	|u_p-v_q| + |u_q-v_r| \leq |u_q-v_q|+|u_p-v_r|
\end{equation*}
Let us define $\tilde{\sigma} = \sigma \circ (p q)$ :\newline
\begin{equation*}
 	\forall i \notin \{p,q\}, \tilde{\sigma} (i)=\sigma (i)
\end{equation*}
\begin{equation*}
 	\tilde{\sigma} (p)=r
\end{equation*}
\begin{equation*}
 	\tilde{\sigma} (q)=q
\end{equation*}

Then  we have

\begin{multline*}
	\begin{split}
	\sum_{i=1}^n |u_i-v_{\sigma (i)}|	&= \sum_{i\notin \{p,q\}} |u_i-v_{\sigma (i)}|+|u_p-v_q|+|u_q-v_r|\\
	 	&\leq \sum_{i\notin \{p,q\}} |u_i-v_{\sigma (i)}|+|u_q-v_q|+|u_p-v_r|\\
	 	&=\sum_{i=1}^n |u_i-v_{\tilde{\sigma} (i)}|
	\end{split}
	\end{multline*}

So, $\tilde{\sigma}$ reaches the minimum too and $supp(\tilde{\sigma}) = supp(\sigma) \setminus \{\max supp(\sigma) \}$. By doing this operation many times, we can remove all the elements of $supp(\sigma)$. So Id reaches the minimum. $\Box$

   \section{Projection on diagonal}
   Because we are very interested in extreme events, we will focus on tails of the multivariate distribution. To study their dependance, it is interesting to consider the corners of the space of copulas which is an hypercube.
   \subsection{Corner}

	\begin{definition}
		A \textbf{corner} of an hypercube \begin{math} [0,1]^d \end{math} is a point \begin{math} \textbf{a}=(a_1,...,a_d) \in \{0,1\}^d \end{math} : \begin{equation*}
		\forall i \in  \llbracket 1,d \rrbracket, a_{i} = 0 \text{ or } a_{i}=1
		\end{equation*}
	\end{definition}
	So there is \begin{math} 2^d\end{math} corners in a hypercube of dimension d.\newline

	\subsection{Diagonal}

	\begin{definition}
		A \textbf{diagonal} \begin{math} \Delta \end{math} is a segment which links to opposite corner \textbf{a} and \textbf{b} :
		\begin{equation*}
			\Delta =[\textbf{a},\textbf{b}]\text{ where }\newline
			\forall i \in \llbracket 1,d \rrbracket , a_i = 0 \iff b_i=1
		\end{equation*}
			Alternatively :
		\begin{equation*}
			\Delta =\{(1-\lambda)\textbf{a} + \lambda \textbf{b}, \lambda \in [0,1]\}, \text{ where }\newline
			\forall i \in \llbracket 1,d \rrbracket , a_i = b_i\text{ mod 2 }
		\end{equation*}
	Because one diagonal can be written [\textbf{a},\textbf{b}] or [\textbf{b},\textbf{a}], we will always consider \begin{math} a_1 =0 \end{math} so that each diagonal has a unique way notation.\newline
	\newline
	We can also define the \textbf{direction} of a diagonal as the vector :

	\begin{equation*}
		U_\Delta =\frac{1}{\sqrt{d}}(\textbf{b}-\textbf{a})
	\end{equation*}
	\end{definition}

   \subsection{Projection}

	\begin{definition}
	The \textbf{matrix of projection} on the linear space will be :
	\begin{equation*}
		M_\Delta = U_\Delta U_\Delta^\top
	\end{equation*}

	Finally, the \textbf{projection on the diagonal} which is an affine space is the function \begin{math} P_\Delta \end{math} such that :
	\begin{equation}
		P_\Delta(X) = M_\Delta(X-C)+C \text{ where } C=(\frac{1}{2},...,\frac{1}{2})
	\end{equation}

	\end{definition}

	So there are \begin{math} 2^{d-1}\end{math} diagonals, directions and matrix of projection in an hypercube of dimension d.\newline
	\newline
	The division by \begin{math} \sqrt{d} \end{math} in the definition of direction permits to have a unit vector.\newline
	\newline
	M is indeed a matrix thanks to the order of the factors (and not a scalar product as \begin{math} U^\top U \end{math}).\newline
	\newline
	One should not confuse the matrix of projection on the linear space and the traditional affine projection on the diagonal. That is why we need to translate everything with the center of the hypercube C.
	\newline
	\begin{figure}

    \includegraphics[width=0.7\textwidth]{Diagonals3d.png}
    \caption{Examples of diagonals, \emph{The blue one is [(0,0,0),(1,1,1)], the yellow one is [(0,1,0),(1,0,1)], the green one is [(0,1,1),(1,0,0)],  and the red one is [(0,0,1),(1,1,0)] }}
\end{figure}

\begin{figure}

    \includegraphics[width=0.7\textwidth]{proj2Ddiag0.png}
    \caption{Projection of 30 points on the [(0,0),(1,1)] diagonal \emph{The diagonal is in blue, the initial points are green and their projections are red.}}
\end{figure}

	\begin{figure}

    \includegraphics[width=0.7\textwidth]{proj3d.png}
    \caption{Projection of 20 points on the [(0,0,0),(1,1,1)] diagonal \emph{The diagonal is in blue, the initial points are green and their projections are red.}}
\end{figure}

	  \begin{figure}

    \includegraphics[width=0.7\textwidth]{proj3Ddiag2.png}
    \caption{Projection of 20 points on the [(1,0,0),(0,1,1)] diagonal \emph{The diagonal is in blue, the initial points are green and their projections are red.}}
\end{figure}

	\subsection{Distribution on the diagonal}
	We now want to study the distribution of the points projected on the diagonal to compare it to a uniform distribution. Since the diagonal is a segment, each point x of the diagonal can be described by only one scalar number $\lambda$ : $x = (1-\lambda)a+\lambda b $ (cf Definition of the diagonal). \newline
	$\lambda$ can be understood as the normalised distance between a and x :
	\begin{equation*}
		\| x-a \|	= \| (1-\lambda)a+\lambda b\|
					= \lambda \| a - b \|
					= \lambda \sqrt{d}
	\end{equation*}
Where $\| . \|$ is a norm in our space.\newline
\newline
But $\lambda$ can be easily evaluated by taking the first coordonates of x :
	\begin{equation*}
		x_1 = (1-\lambda)a_1 + \lambda b_1 = (1-\lambda) *0 + \lambda *1 = \lambda
	\end{equation*}
This equality is possible thanks to our useful convention $(a_1,b_1)= (0,1)$.\newline
\newline
We now have a unique number that should be uniformly distributed on [0,1].

	  \section{Our algorithm}

	  For each day j do :

\begin{itemize}

\item Thanks to all day \begin{math}i \leq j-1\end{math}, fit a parametric distribution model with copula \begin{math} C_j \end{math}

\item Generate n realizations U of the random variable with the copula dependance and uniform marginals :\newline

	Generate \begin{math} \textbf{U}=(U_1,...,U_n) \end{math} with each \begin{math} U_i = (U_{i,1},...,U_{i,d}) \in [0,1]^d \end{math} \newline
\newline
	 where \begin{math} \forall i \in \llbracket 1,n \rrbracket , \newline \mathbb{P} (U_{i,1} \leq u_1,..., U_{i,d} \leq u_d )= C(u_1,...,u_d) \text{ and } \newline
	( \forall j \in \llbracket 1,d \rrbracket, U_{i,j} \end{math} is uniformly distributed on [0,1].) \newline
	For example, n=10000  \newline

\item For each diagonal $\Delta$ :
\item
Project all the \begin{math} U_i \end{math} on the diagonal
\newline
	Define \begin{math} \textbf{V}_{\Delta} =(V_{\Delta ,1},...,V_{\Delta ,n})=(P_\Delta(U_1),...,P_\Delta(U_n)) \end{math}

\item Define the empirical distribution on this diagonal : \newline
\newline
	\begin{math} F_\Delta (X) = \frac{1}{n} \sum_{k=1}^n \mathds{1}_{X \leq V_{\Delta ,k}} \end{math}
	\newline
	\newline
	where \begin{math} a \leq b \iff \forall i \in \llbracket 1,d \rrbracket,  a_i \leq b_i  \end{math}

\item Observe with the data what happened on day j \newline
		Call this observation $O_j = (O_{j,1},...,O_{j,d})$

\item Pass it in the copula space \newline
		Define \begin{math} Q_j = (F_1 (O_{j,1}),...,F_d (O_{j,d})) \end{math} \newline
		where the $F_i$ are the cumulative density functions of the marginals estimated with another method.

\item Project $Q_j$ on the diagonal \newline
		Define $R_{\Delta ,j} = P_\Delta (Q_j)$

\item Either define $S_{\Delta,j} = F_\Delta (R_{\Delta,j})  $ \newline
	and compute the distance between the empirical distribution of the $S_{\Delta}=(S_{\Delta,i})_{i \in days}$  and the uniform distribution on [0,1].

\item Or make a rank histogram with the $R_{\Delta}=(R_{\Delta,i})_{i \in days}$
	\begin{math} F_\Delta^{-1} (P_j) \end{math}

	\section{Test code}

	In this section, I will explain how we computed this algorithm with different parameters. They are arguments of many test functions I wrote and are just strings defining options :\newline

	source : the type of power source we want it can be 'solar' or 'wind'\newline

	datatype : if the datas is power ('actuals'), errors ('errors), a normal distributed sample ('normal-sample') or a uniformly distributed sample ('uniform-sample'), the two last ones are options to make verifications. \newline

	segment_marginals : the way you segment the datas to fit the marginals, you can either take only the date at the hour of your dps ('hour') or fit this marginal with the datas of the whole day ('anytime'). Note : this is not the way the datas are segmented to fit the copula.

	kind : which projection you want it can be on a diagonal ('diagonal'), a marginal ('marginal') or can even compose with the kendall function ('kendal'). \newline

	index : the index of the diagonal or the marginal you want to project with. Diagonals are indexed in the order of diag. list of diags. Marginals are index as the coordonates. index does not matter for kendall function.\newline

	method : way you choose the data to fit the distributions, you can either fit copulas and marginals with the datas of the whole year and check the observation then ('wholeyear'), or you

\end{itemize}

   \begin{thebibliography}{9}

	\bibitem{hamill2000}
  	Thomas M. Hamill,
 	 \emph{Interpretation of Rank Histograms for Verifying Ensemble Forecasts},
  	2000.

	\bibitem{vineconstruction}
  	Kjersti Aas, Claudia Czado, Arnoldo Frigessi, Henrik Bakken,
 	 \emph{Pair-copula construction of multiple dependence},
  	2007.

  	\bibitem{fourcopulas}
  	Kjersti Aas,
 	 \emph{Modeling the dependance structure of financial assets : A survey of four copulas},
  	2004.

  	\bibitem{gaubert}
  	Stéphane Gaubert, Frédéric Bonnans,
  	\emph{Recherche opérationnelle : aspects mathématiques et applications},
  	Editions de l'École polytechnique,
  	2016

	\end{thebibliography}
\end{document}


\chapter{Text-Structure and Math}
In this very first chapter, we clarify how to add your math within the text in a proper way. Below you will find some small samples of the book \emph{''Single Channel Phase-Aware Signal Processing in Speech Communication: Theory and Practice''} \cite{MowlaeePejman2016}, may this example arouses your interest to dig more into the \emph{Signal Processing} and it's related topic \emph{Speech Processing}.

\begin{mdframed}
	\begin{lstlisting}[caption={Adding section and subsection}]
		\section{Phase Estimation Fundamentals}
		\subsection{Background and Fundamentals}
		The problem of interest in many signal processing.......
	\end{lstlisting}
\end{mdframed}

\section{Phase Estimation Fundamentals}
\subsection{Background and Fundamentals}
The problem of interest in many signal processing applications including radar, spectrum estimation and signal enhancement, is to detect a signal of interest in a noisy observation. The signal of interest is often represented as a sum of sinusoids characterized by their amplitude, frequency and phase parameters. Since these parameter triplet suffices to describe the signal, the problem degenerates to the detection and estimation of the sinusoidal parameters. This topic has been widely addressed in the literature of signal detection \cite{VanTrees1968} and estimation \cite{Kay1993}. While many previous studies have been focused on deriving estimators for amplitude and frequency of sinusoids in noise (see e.g. \cite{Stoica2005} for an overview), the issue of phase estimation has been less addressed. Reliable phase estimation for practical applications has not been adequately addressed, in particular for signal enhancement.
\begin{mdframed}
	\begin{lstlisting}[caption={Add citations into your text}]
	......the sinusoidal parameters. This topic has been widely addressed in the
	literature of signal detection \cite{VanTrees 1968} and estimation
	\cite{Kay 1993}.
	\end{lstlisting}
	Do not forget to add these specific bibliography fields into your \emph{.bib} file!
\end{mdframed}
\section{Key Examples: Phase Estimation Problem}\label{ch3:PE1}
\subsection{Example 1: discrete-time sinusoid}~\\
\noindent To reveal the phase structure of one sinusoid's frequency response we consider the following real-valued sequence
\begin{equation}\label{eq:c3.1}
x(n)=\cos(\omega_0n+\phi),
\end{equation}
with $\omega_0$ as frequency and $\phi$ as phase shift. Application of the \index{discrete-time Fourier transform}discrete-time Fourier transform (\gls{DTFT}), defined as
\begin{equation}\label{eq:c3.2}
X(e^{j\omega})=\text{\gls{DTFT}T}\left(x(n)\right)=\sum_{n=-\infty}^{\infty}x(n)e^{-j\omega n},
\end{equation}
yields the following frequency domain representation of the sequence $x(n)$
\begin{equation}\label{eq:c3.3}
X(e^{j\omega})=\pi e^{j\phi}\delta(\omega-\omega_0)+\pi e^{-j\phi}\delta(\omega+\omega_0),
\end{equation}
with $\delta(\omega)$ denoting the \index{Dirac delta}Dirac delta function. As the cosine function is \index{symmetric}symmetric $\left(\cos(\omega_0 n)=\cos(-\omega_0 n)\right)$, only the phase shift $\phi$ determines the phase response of $X(e^{j\omega})$
\begin{equation}\label{eq:c3.4}
\angle X(e^{j\omega})=\begin{cases}\phi, & \omega=\omega_0,\\
-\phi, & \omega=-\omega_0
\end{cases}.
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%% FIGURE 3.1 %%%%%%%%%%%%%
\begin{figure}
	\center % 20 587 530 727
	\includegraphics{figures/figure3_1.eps}
	\vspace{-0.6cm}
	\caption{Visualization of window impact on a sinusoid $x(n)=\cos(\omega_0 n+\phi)$ in time and frequency domain with $\omega_0=0.1\cdot2\pi$ and $\phi=-\pi/8$ and a rectangular window with length $N_w=21$. The window \gls{DTFT}T $W(e^{j\omega})$ is shifted dependent on $\omega_0$ and multiplied by $e^{j\phi}$ and $e^{-j\phi}$, respectively, as shown in the phase response of $\text{\gls{DTFT}T}\left(x(n)w(n)\right)$}\label{Figure31}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\noindent The left column of Figure~\ref{Figure31} represents the sequence $x(n)$ along time $n$ followed by its \gls{DTFT}T representation with real $\Re\{X(e^{j\omega})\}$ and imaginary $\Im\{X(e^{j\omega})\}$ parts as well as magnitude $|X(e^{j\omega})|$ and phase $\angle X(e^{j\omega})$ response. We set $\omega=0.1\cdot2\pi$ and $\phi=-\pi/8$.\\

\noindent The result in \eqref{eq:c3.4} is valid for an observation range of $n$ within $n\in]-\infty,\infty[$. In practice only a subset of samples $n$ is available for analysis. This limitation can be represented by introducing an \index{analysis!window}analysis window function which is multiplied with the sequence $x(n)$. The modest \index{window!rectangular, uniform, boxcar}analysis window is the rectangular, also known as boxcar or uniform window which has the value of one within the range of $N_w$ and zero outside
\begin{equation}\label{eq:c3.5}
w(n)=\begin{cases}1, & |n|\leq\frac{N_w-1}{2},\\
0, & \text{else},
\end{cases}
\end{equation}
for odd $N_w$.
This window is \index{symmetric}symmetric $(w(n)=w(-n))$ and has the \index{zero!-phase}\index{phase!zero-phase}zero-phase property, yielding a real-valued \gls{DTFT}T of the analysis window
\begin{equation}\label{eq:c3.6}
W(e^{j\omega})=\sum_{n=-\infty}^{\infty}w(n)e^{-j\omega n}=\sum_{n=-(N_w-1)/{2}}^{(N_w-1)/{2}}1e^{-j\omega n}=\frac{\sin\left(\frac{N_w\omega}{2}\right)}{\sin\left(\frac{\omega}{2}\right)},
\end{equation}
also known as \index{Dirichlet kernel}\textit{Dirichlet kernel}. The middle column of Figure~\ref{Figure31} illustrates a \index{symmetric}symmetric rectangular window in time and frequency domain with length $N_w=21$ and a \gls{DTFT}T length of $N=31$ (continuous line). The real part is equal to the Dirichlet kernel and the imaginary part is equal to zero due to the symmetry of the window $w(n)$. The phase response represents the sign of $W(e^{j\omega})$ dependent on $\omega$ and is equal to zero within the mainlobe width.\\
% Analyzing $x(n)$ within a limited range of $n\in[-\frac{N_w-1}{2},\frac{N_w-1}{2}]$ is equivalent of multiplying $x(n)$ with the analysis window $w(n)$.

\noindent The product $x(n)w(n)$ corresponds to a convolution in the frequency domain according to
\begin{equation}\label{eq:c1.7}
X_w(e^{j\omega})=\text{\gls{DTFT}T}\left(x(n)w(n)\right)=\left\{X\ast W\right\}(e^{j\omega}).
\end{equation}

\begin{mdframed}
	\begin{lstlisting}[caption={Add formulars to your text}]
	\begin{equation}\label{eq:c1.7}
	X_w(e^{j\omega})=\text{\gls{DTFT}T}\left(x(n)w(n)\right)=\left\{X\ast W\right\}(e^{j\omega}).
	\end{equation}
	\end{lstlisting}
	By adding a label to your equation, you will be able to refer to the equation within the text!
\end{mdframed}

Plugging $X(e^{j\omega})$ and $W(e^{j\omega})$, derived in \eqref{eq:c3.3} and \eqref{eq:c3.6}, respectively, in \eqref{eq:c1.7} yields the following expression
\begin{equation}\label{eq:c3.8}
X_w(e^{j\omega})=\pi e^{j\phi}W(e^{j(\omega-\omega_0)})+\pi e^{-j\phi}W(e^{j(\omega+\omega_0)}).
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%% FIGURE 3.2 %%%%%%%%%%%%%
\begin{figure}
	\center % 20 587 530 727
	\includegraphics{figures/figure3_2.eps}
	% \includegraphics{Ch3/Figures/figure3_2.eps}
	\vspace{-0.6cm}
	\caption{Relation of sinusoidal periods and window length and its impact on amplitude and phase: (a) shows a sinusoid multiplied by a boxcar window with a length of one period $(m=1)$. The Dirichlet kernels do not interfere at $\omega=\omega_0$ and $\omega=-\omega_0$ which yields an unbiased phase estimate of $\angle X_w(e^{j\omega_0})=\phi$, (b) presents the more general case of a window length which does not correspond to an integer multiplier of the sinusoids period $(m=1.1)$. The amplitude as well as the phase do not approach the true value and thus, the outcome is biased.}\label{Figure32}
\end{figure}
%%%%%%%%%%%%%%%%

\begin{mdframed}
	\begin{lstlisting}[caption={Example on how to add a figure to your text}]
	\begin{figure}
		\center % 20 587 530 727
		\includegraphics{figures/figure3_2.eps}
		\vspace{-0.6cm}
		\caption{Relation of sinusoidal periods and window length and its impact on amplitude and phase: (a) shows a sinusoid multiplied by a boxcar window with a length of one period $(m=1)$. ...}\label{Figure32}
	\end{figure}
	\end{lstlisting}
	By adding a label to your figure, you will be able to refer to the figure within the text!
\end{mdframed}

This is a rather important observation as the Dirichlet kernels are shifted along the frequency axis to $\omega=\omega_0$ and $\omega=-\omega_0$. The multiplication by the constants $e^{j\phi}$ and $e^{-j\phi}$, respectively, yields a complex valued $X_w(e^{j\omega})$ as shown in the right column of Figure~\ref{Figure31}. The terms on the right-hand-side of \eqref{eq:c3.8} constructively add or eliminate each other, dependent on the value of $\phi$ and $\omega_0$, described as \index{leakage effect}\emph{leakage effect}.\\

\noindent The interaction between the Dirichlet kernels is minimized if the frequency $\omega_0$ fulfills the following requirement
\begin{equation}\label{eq:c3.9}
\omega_0=\frac{2{m}\pi}{N_w}, \quad m\in\mathbb{N}.
\end{equation}
with $m$ denoting the number of periods contained in one window length $N_w$. Figure~\ref{Figure32} illustrates the impact of $\omega$ on the resulting magnitude and phase response of one sinusoid $x(n)=\cos(\omega_0 n+\phi)$ with $\phi=-\pi/8$, multiplied by a symmetric window of length $N_w=31$. Setting $\omega_0=1\frac{2\pi}{N_w}$ leads to $m=1$ and for the phase response at frequency $\omega=\omega_0$ we obtain
\begin{equation}
X_w(e^{j\omega_0})=\pi e^{j\phi}W(e^{j(\omega_0-\omega_0)})+\pi e^{-j\phi}W(e^{j(\omega_0+\omega_0)})\\
\end{equation}
As the \gls{DTFT}T of the rectangular window $W(e^{j2\omega_0})=0$ for $m\in\mathbb{N}$, the phase response yields the true value of $\phi$ at frequency $\omega_0$.
\begin{align}\label{eq:c3.10}
	X_w(e^{j\omega_0})&=\pi e^{j\phi} CG\\
	\angle X_w(e^{j\omega_0})&=\phi,
\end{align}
with defining $CG=W(e^{j0})$ as the \index{coherent gain}{coherent gain} for the selected window\footnote{see }. The right column of Figure~\ref{Figure32} demonstrates the more general case of $m\notin\mathbb{N}$. The sidelobs of $W(e^{j\omega})$ interact with each other, resulting in a \index{bias}biased phase response at $\omega_0$. Note, that the peak's location of the magnitude response is not at $\omega_0$ due to the complex-valued superposition of both kernels. Therefore, any \index{peak-picking}peak-picking method for obtaining a sinusoidal phase would result in a biased outcome.\\

\noindent In order to reduce the unpleasant impact of the sidelobe level, the choice of the window type becomes of particular interest. Basically, their behavior can be categorized by two characteristics: \index{leakage effect}{spectral leakage} and {frequency resolution}. The frequency resolution is limited by the mainlobe width which corresponds to the ability to resolve two adjacent spectral lines. To increase the frequency resolution a window function with a small mainlobe width is preferred. As a smaller mainlobe width is on the expense of a reduced sidelobe level the choice of an appropriate window function is a trade-off between a high frequency resolution and a low sidelobe level. Another way of optimizing the window choice is to adjust the window length $N_w$ to fulfill the requirement in \eqref{eq:c3.9}. However, adapting $N_w$ needs knowledge of $\omega_0$ and is in general only possible for one single sinusoid. \\

\noindent Figure~\ref{Figure33} shows the influence of three prominent window types on the amplitude and phase response of the sequence $x_w(n)=\sin(\omega_0 n+\phi)w(n)$ with $\omega=3.1\cdot 2\pi/N_w$ and $\phi=-\pi/8$. So far, only the rectangular window was discussed. Its high frequency resolution ($6\,\text{dB}$ bandwidth of the mainlobe width: $\Delta\omega_{\text{MW}}=1.21\cdot 2\pi/N_w$) is at the cost of rather poor sidelobe suppression level of $-13\,\text{dB}$ for the strongest neighboring sidelobe (Fig.~\ref{Figure33} a)). The widely used Hamming window (b) consists of three shifted Dirichlet kernels with the purpose to minimize the sidelobe levels, achieving a suppression of $-42\,\text{dB}$ at the cost of a worse frequency resolution ($6\,\text{dB}$ bandwidth of mainlobe: $\Delta\omega_{\text{MW}}=1.81\cdot 2\pi/N_w$). The amplitude and phase response of the Hamming-windowed sinusoid reveal the advantage of a higher sidelobe suppression. The phase response at frequencies within the mainlobe width
is determined by the true phase value $\phi=-\pi/8$.
Compared to the rectangular window, the employment of a \index{window!Hamming}Hamming window results in a more robust phase estimation if an inaccurate frequency estimate of the sinusoid is given. Further, the magnitude's peak location is less shifted which yields a more accurate phase estimation when using \index{peak-picking}peak-picking. Finally, the \index{window!Blackman}{Blackman} window is presented in (c) with a sidelobe suppression of $-58\,\text{dB}$ and a $6\,\text{dB}$ mainlobe bandwidth of $\Delta\omega_{\text{MW}}=2.35\cdot 2\pi/N_w$. The neighboring phase values at $\omega_0$ are strongly influenced by the true phase value of $\phi$. However, both Hamming and Blackman windows deal with an increased mainlobe width. Once we extend our signal to multiple sinusoids, the mainlobe width plays a major role in selecting an appropriate window. If the mainlobe width contains more than one sinusoid then it is no longer possible to resolve the phase values of the sinusoids.

%%%%%%%%%%%%% FIGURE 3.3 %%%%%%%%%%%%%
\begin{figure}[t]
	\center % 20 587 530 727
	\includegraphics{figures/figure3_3.eps}
	% \includegraphics{Ch3/Figures/figure3_3.eps}
	% \vspace{-0.6cm}
	\caption{Illustration of different windows' impact on the magnitude and phase response of one sinusoid. The improved sidelobe suppression is at the cost of a higher mainlobe width resulting in a lower frequency resolution. For windows with higher sidelobe suppression, the phase response at frequency $\omega_0$ is increasingly dominated by the phase $\phi$ within the mainlobe width.}\label{Figure33}
\end{figure}
%%%%%%%%%%%%%%%%

%\input{Chapters/Captions/Figures/figure3_4}
\subsection{Example 2: discrete-time sinusoid in noise}~\\
So far, one sinusoid without additive noise was considered. For practical scenarios, the signal of interest is composed of multiple sinusoids corrupted with noise as shown in Figure~\ref{Figure34}. The problem to solve is the estimation of the sinusoidal phase ${\phi}_{h}$ given its amplitude and frequency denoted as $A_h$ and ${\omega}_h$, respectively. Following the harmonic model of a speech signal we assume that the sinusoidal frequency ${\omega}_h$ is constrained to be a multiple harmonic of a fundamental frequency, i.e. ${\omega}_h=h{\omega}_0$ with $h\in[1,\ldots,H]$ denoting the harmonic index
\begin{equation}\label{eq:c3.11}
x(n)=\sum_{h=1}^H{A_h\cos({\omega}_hn+{\phi}_h)}+d(n),
\end{equation}
with $\omega_h=h2\pi f_0/f_s$ and $d(n)$ as the additive noise. Application of a window function $w(n)$, having non zero values in the range of $[-(N_w-1)/2,(N_w-1)/2]$, yields the windowed signal $x_w(n)=x(n)w(n)$. Similar to \eqref{eq:c3.8}, the \gls{DTFT}T spectrum follows as
\begin{equation}\label{eq:c3.12}
X_w(e^{j\omega})=\sum_{h=1}^{H} A_h\pi \left( e^{j{\phi}_h}W(e^{j(\omega-{\omega}_h)}) + e^{-j{\phi}_h}W(e^{j(\omega+{\omega}_h)})\right)+D_w(e^{j\omega}),
\end{equation}
where $D_w(e^{j\omega})=\sum_{n=-(N_w-1)/2}^{(N_w-1)/2}{d(n)w(n)e^{-j{\omega}n}}$ and $W(e^{j\omega})$ is the window frequency response. To have a better insight into the phase estimation problem, we are interested in the effect of the neighboring harmonics $h\neq \bar{h}$ on the desired harmonic $\bar{h}$. We evaluate the frequency response of $X(e^{j\omega_{\bar{h}}})$ at the desired frequency ${\omega}_{\bar{h}}$
\begin{equation}\label{eq:c3.13}
\begin{split}
X_w(&e^{j\omega_{\bar{h}}})=A_{\bar{h}}\pi e^{j{\phi}_{\bar{h}}}\cdot CG+A_{\bar{h}}\pi e^{-j{\phi}_{\bar{h}}}W(e^{j2\omega_{\bar{h}}})\\
&+\sum_{h=1,h\neq{\bar{h}}}^H {A_h\pi e^{j{\phi}_h}W(e^{j(\omega_{\bar{h}}-\omega_h)})+A_h\pi e^{-j{\phi}_h}W(e^{j(\omega_{\bar{h}}+\omega_h)}})+D_w(e^{j\omega_{\bar{h}}}).
\end{split}
\end{equation}
The additive terms on the right-hand-side of Equation \eqref{eq:c3.13} show the interaction of the adjacent harmonics to the desired phase $\phi_{\bar{h}}$ as well as the impact of the additive noise. In the following we are interested in the influence of these terms on the desired phase by re-writing \eqref{eq:c3.13} according to
%%%%%%%%%%%%%%%%
%TODO \input{Chapters/Captions/Figures/figure3_5}

\begin{equation}\label{eq:c3.14}
X_w(e^{j\omega_{\bar{h}}})=X_r(e^{j\omega_{\bar{h}}})e_c
\end{equation}
where $X_r(e^{j\omega_{\bar{h}}})=A_{\bar{h}}\pi e^{j\phi_{\bar{h}}}$ and $e_c$ captures the phase estimation errors, given by
\begin{eqnarray}\label{eq:c3.15}
e_c&=&\sum_{h=1}^H\frac{A_h}{A_{\bar{h}}}\left( e^{j(\phi_h-\phi_{\bar{h}})}W(e^{j(\omega_{\bar{h}}-\omega_h)})+ e^{-j(\phi_h+\phi_{\bar{h}})}W(e^{j(\omega_{\bar{h}}+\omega_h)}) \right)\nonumber\\
&+& \frac{1}{\pi A_{\bar{h}}}e^{-j\phi_{\bar{h}}}D_w(e^{j\omega_{\bar{h}}})
\end{eqnarray}
In order to get more insight on the phase error term $e_c$, in the following we derive its phase mean and variance.\\
\textbf{First moment of }$\boldsymbol{e_c}$~\\
\noindent The mean value of the phase error term is given by
\begin{equation}\label{eq:c3.16}
\mathbb{E}_\phi(e_c)=\int_{-\pi}^\pi e_c p(\phi)d\phi
\end{equation}
with $p(\phi)$ denoting the phase distribution. Applying \eqref{eq:c3.16} to \eqref{eq:c3.15} the first moment of $e_c$ is given by
\begin{equation}\label{eq:c3.17}
\begin{split}
\mathbb{E}_\phi(e_c) &= \sum_{h=1}^H\frac{A_h}{A_{\bar{h}}} \mathbb{E}(e^{j(\phi_h-\phi_{\bar{h}})})W(e^{j(\omega_{\bar{h}}-\omega_h)})\\
&+ \sum_{h=1}^H\frac{A_h}{A_{\bar{h}}} \mathbb{E}(e^{-j(\phi_h+\phi_{\bar{h}})})W(e^{j(\omega_{\bar{h}}+\omega_h)})\\
&+ \frac{1}{\pi A_{\bar{h}}}\mathbb{E}(e^{-j\phi_{\bar{h}}})D_w(e^{j\omega_{\bar{h}}})
\end{split}
\end{equation}
\textbf{Second moment of }$\boldsymbol{e_c}$~\\
\noindent \index{phase!error variance}The second moment of the error term is given by
\begin{equation}\label{phasevar0}
\begin{split}
\mathbb{E}_\phi\left(e_c e_c^*\right)&=\!\sum_{h_1=1}^{H-1} \sum_{h_2=h_1+1}^H \! C_1(\bar{h},h_1,h_2) \mathbb{E}_\phi\!\left(\cos(\phi_{h_1} \!-\! \phi_{h_2} \!+\!\angle w_1(\bar{h},h_1,h_2))\right)\\
&+ \!\sum_{h_1=1}^{H-1} \sum_{h_2=h_1+1}^H\!C_2(\bar{h},h_1,h_2)\mathbb{E}_\phi\!\left(\cos(\phi_{h_1} \!-\! \phi_{h_2} \!-\!\angle w_2(\bar{h},h_1,h_2))\right)\\
&+ \!\sum_{h_1=1}^H \sum_{h_2=1}^H\!C_3(\bar{h},h_1,h_2)\mathbb{E}_\phi\!\left(\cos(\phi_{h_1} \!+\! \phi_{h_2} \!+\!\angle w_3(\bar{h},h_1,h_2))\right)\\
&+\sum_{h=1}^H C_4(\bar{h},h)\mathbb{E}_\phi\left(\cos(\phi_h-\phi_{\bar{h}}+\angle W(e^{j(\omega_{\bar{h}}-\omega_h)})D_w(e^{-j\omega_{\bar{h}}}))\right)\\
&+\sum_{h=1}^H C_5(\bar{h},h)\mathbb{E}_\phi\left(\cos(\phi_h+\phi_{\bar{h}}-\angle W(e^{j(\omega_{\bar{h}}+\omega_h)})D_w(e^{-j\omega_{\bar{h}}}))\right)\\
&+\frac{1}{\pi^2A_{\bar{h}}^2}|D_w(e^{j\omega_{\bar{h}}})|^2 + C_6(\bar{h}),
\end{split}
\end{equation}
\begin{mdframed}
	\begin{lstlisting}[caption={Add formulars to your text, splitted equations}]
	\begin{equation}\label{phasevar0}
	\begin{split}
	\mathbb{E}_\phi\left(e_c e_c^*\right)&=\!\sum_{h_1=1}^{H-1} \sum_{h_2=h_1+1}^H \! C_1(\bar{h},h_1,h_2) \mathbb{E}_\phi\!\left(\cos(\phi_{h_1} \!-\! \phi_{h_2} \!+\!\angle w_1(\bar{h},h_1,h_2))\right)\\
	&+ \!\sum_{h_1=1}^{H-1} \sum_{h_2=h_1+1}^H\!C_2(\bar{h},h_1,h_2)\mathbb{E}_\phi\!\left(\cos(\phi_{h_1} \!-\! \phi_{h_2} \!-\!\angle w_2(\bar{h},h_1,h_2))\right)\\
	&+ \!\sum_{h_1=1}^H \sum_{h_2=1}^H\!C_3(\bar{h},h_1,h_2)\mathbb{E}_\phi\!\left(\cos(\phi_{h_1} \!+\! \phi_{h_2} \!+\!\angle w_3(\bar{h},h_1,h_2))\right)\\
	&+\sum_{h=1}^H C_4(\bar{h},h)\mathbb{E}_\phi\left(\cos(\phi_h-\phi_{\bar{h}}+\angle W(e^{j(\omega_{\bar{h}}-\omega_h)})D_w(e^{-j\omega_{\bar{h}}}))\right)\\
	&+\sum_{h=1}^H C_5(\bar{h},h)\mathbb{E}_\phi\left(\cos(\phi_h+\phi_{\bar{h}}-\angle W(e^{j(\omega_{\bar{h}}+\omega_h)})D_w(e^{-j\omega_{\bar{h}}}))\right)\\
	&+\frac{1}{\pi^2A_{\bar{h}}^2}|D_w(e^{j\omega_{\bar{h}}})|^2 + C_6(\bar{h}),
	\end{split}
	\end{equation}
	with the abbreviations
	\begin{equation}
	\begin{split}
	w_1(\bar{h},h_1,h_2)&=W(e^{j(\omega_{\bar{h}}-\omega_{h_1})})W(e^{-j(\omega_{\bar{h}}-\omega_{h_2})}),\\
	w_2(\bar{h},h_1,h_2)&=W(e^{j(\omega_{\bar{h}}+\omega_{h_1})})W(e^{-j(\omega_{\bar{h}}+\omega_{h_2})}),\\
	w_3(\bar{h},h_1,h_2)&=W(e^{j(\omega_{\bar{h}}-\omega_{h_1})})W(e^{-j(\omega_{\bar{h}}+\omega_{h_2})}),
	\end{split}
	\end{equation}
	\end{lstlisting}
	By adding a label to your equation, you will be able to refer to the equation within the text!
\end{mdframed}
with the abbreviations
\begin{equation}
\begin{split}
w_1(\bar{h},h_1,h_2)&=W(e^{j(\omega_{\bar{h}}-\omega_{h_1})})W(e^{-j(\omega_{\bar{h}}-\omega_{h_2})}),\\
w_2(\bar{h},h_1,h_2)&=W(e^{j(\omega_{\bar{h}}+\omega_{h_1})})W(e^{-j(\omega_{\bar{h}}+\omega_{h_2})}),\\
w_3(\bar{h},h_1,h_2)&=W(e^{j(\omega_{\bar{h}}-\omega_{h_1})})W(e^{-j(\omega_{\bar{h}}+\omega_{h_2})}),
\end{split}
\end{equation}
and the phase independent constants
\begin{equation}
\begin{split}
C_1(\bar{h},h_1,h_2)&=2\frac{A_{h_1} A_{h_2}}{A^2_{\bar{h}}}|w_1(\bar{h},h_1,h_2)|,\\
C_2(\bar{h},h_1,h_2)&=2\frac{A_{h_1} A_{h_2}}{A^2_{\bar{h}}}|w_2(\bar{h},h_1,h_2)|,\\
C_3(\bar{h},h_1,h_2)&=2\frac{A_{h_1} A_{h_2}}{A^2_{\bar{h}}}|w_3(\bar{h},h_1,h_2)|,\\
C_4(\bar{h},h)&=2\frac{A_h}{\pi A^2_{\bar{h}}}|W(e^{j(\omega_{\bar{h}}-\omega_h)})||D_w(e^{j\omega_{\bar{h}}})|,\\
C_5(\bar{h},h)&=2\frac{A_h}{\pi A^2_{\bar{h}}}|W(e^{j(\omega_{\bar{h}}+\omega_h)})||D_w(e^{j\omega_{\bar{h}}})|,\\
C_6(\bar{h})&=\sum_{h=1}^H \frac{A_h^2}{A_{\bar{h}}^2}\left( |W(e^{j(\omega_{\bar{h}}+\omega_{{h}})})|^2+|W(e^{j(\omega_{\bar{h}}-\omega_{{h}})})|^2 \right).
\end{split}
\end{equation}
The second moment of the phase error in \eqref{phasevar0} provides useful insights on how the phase of the desired frequency $\omega_{\bar{h}}$ is a function of the chosen window, the additive noise and the neighboring harmonics. Subsequently the key factors are summarized as
\begin{itemize}
	\item $W(e^{j\omega})$ the magnitude and phase response of the \index{analysis!window}analysis window function
	\item $D_w(e^{j\omega})$ the magnitude and phase response of the additive noise
	\item $\frac{A_h}{A_{\bar{h}}}$ the amplitude ratio of the adjacent and desired harmonics
\end{itemize}
The impact of the selected window $W(e^{j\omega})$ can be considered for two cases: First, the harmonics are separated sufficiently which means there is no neighboring harmonic within the mainlobe width. The window's amplitude $W(e^{j(\omega_{\bar{h}}-\omega_h}))$ for $\bar{h}\neq h$ suppresses the neighboring harmonic by its sidelobe level (see Figure~\ref{Figure33}) which results in a low impact of the neighboring harmonics to $\hat{\phi}$. If the harmonics are not separated, i.e., the adjacent harmonic is located within the mainlobe width then the phase error gets larger. The adjacent harmonic is not attenuated by the sidelobe level of the window thus the phase estimation gets more \index{bias}biased. Figure~\ref{Figure2}
 illustrates the impact of adjacent harmonics for a Hamming and a Blackman window. The harmonics are sufficiently separated as the adjacent harmonics are outside of the mainlobe width. The Hamming window has a poorer sidelobe level which potentially yields a higher phase error. The larger
sidelobe level
of the Blackman window reduces the phase error but it also demands a longer window length $N_w$ in order to separate the harmonics. The drawback of a longer window length is a reduced time resolution in STFT spectral analysis. The phase estimation problem is therefore adequately addressed by choosing the best window that balances a good trade-off between a low phase variance and good enough time resolution.


\section{The fixed effects estimator}


\section{Assumptions}
\begin{itemize}
  \item[$\diamond$] \textbf{Assumption 1:} The wood decomposed by fungi is an ideal cylinder, and the fungi are distributed on the surface of the wood.
        \begin{itemize}
          \item[$\hookrightarrow$] \textbf{Justification:} Symmetrical the decomposition environment of the fungi as much as possible to facilitate the connection of this process with the Gauss theorem and Gauss surface in electromagnetic.
        \end{itemize}
  \item[$\diamond$] \textbf{Assumption 2:} The decomposition rate of the fungi is only related to its growth rate, tolerance to moisture and environment temperature.
        \begin{itemize}
          \item[$\hookrightarrow$] \textbf{Justification:} In the real natural environment, fungi are susceptible to natural disasters. We limit the determination of the factors affecting the decomposition rate of fungi in advance to facilitate subsequent analysis.
        \end{itemize}
  \item[$\diamond$] \textbf{Assumption 3:} After the part of Fungi Selection, we obtain five virtual fungal species, which are typical and representative.
        \begin{itemize}
          \item[$\hookrightarrow$] \textbf{Justification:} There are thousands of species of fungi, hence, it does not have much practical significance to study a specific species of fungi. The five fungi we set can be analyzed and studied more comprehensively.
        \end{itemize}
  \item[$\diamond$] \textbf{Assumption 4:} The Logistic model is also valid when studying microorganisms, such as fungi.
        \begin{itemize}
          \item[$\hookrightarrow$] \textbf{Justification:} Logistic model is generally used to model changes in common animals and plants populations. Since fungi are also biological, we will use analogy reasoning to apply it to the study of fungi.
        \end{itemize}
  \item[$\diamond$] \textbf{Assumption 5:} The study on the interactions of the two fungal species combination could show that of more fungal species combination.
        \begin{itemize}
          \item[$\hookrightarrow$] \textbf{Justification:} In a piece of natural land, although there are more than two types of fungi, the interaction of multiple fungi can be understood as the superimposed effect of multiple combinations.
        \end{itemize}
  \item[$\diamond$] \textbf{Assumption 6:} Fungi play a major role in the decomposition of organic matter in the certain carbon cycle.
        \begin{itemize}
          \item[$\hookrightarrow$] \textbf{Justification:} In biology, the decomposition process of organic matter is completed by microorganisms, including fungi, bacteria and so on. Since this article only studies fungi, this process is simplified.
        \end{itemize}
\end{itemize}


\section{Introduction}

\emph{Cryptography} is the science of secure communication and storage. What does ``secure'' mean in this definition? First, it means that, apart from the two communicating parties, there may be a third party trying to learn confidential information, to disrupt the communication, or to mislead the communicating parties. Second, ``secure'' means that a predefined set of goals cannot be achieved by any adversary with set capabilities, such as an ability to read or modify the communications, or limitations, such as having limited computational power.

Cryptography is often divided into design and cryptanalysis. \emph{Cryptographic design} is the design of secure communication systems, called \emph{cryptosystems}. \emph{Cryptanalysis} is \emph{breaking} the security of cryptosystems. In order to understand the security of a cryptosystem better, simplified versions of the cryptosystem are often analyzed. Of course, cryptography and cryptanalysis are not independent. Design of a cryptosystem usually follows alternating steps: design - cryptanalysis - design - cryptanalysis - ..., until the designers can not cryptanalyze the cryptosystem. After that, the cryptosystem is published and for others to analyze.

Modern cryptography is broadly split into private-key and public-key cryptography, also called symmetric-key and asymmetric-key cryptography.

\emph{Symmetric-key cryptography} assumes that the communicating parties have a shared private key. For example, they could meet physically and agree on the common secret key. In this case, the same shared key can be used both for encrypting and decrypting communications.

Symmetric-key cryptosystems are usually constructed from low-level, bitwise operations and functions with small domains. They are very efficient. However, their security is not mathematically proven and is not based on any simple mathematical problem.

\emph{Asymmetric-key cryptography} does not necessarily require pre-shared keys. The defining property, however, is that the key may consist of two parts - a public key and a private key. For example, the public key may be used for encrypting messages and may be openly published. The private key is then used for decrypting the messages and must be kept secret.
Asymmetric-key cryptography can also be used to establish a shared secret securely while using an insecure channel.

Asymmetric-key cryptosystems are usually constructed around mathematical structures from number theory or algebraic geometry. Most often these cryptosystems are relatively inefficient. However, their security is based on the hardness of solving a mathematical problem, such as factoring large integers. It means that, if the cryptosystem is cryptanalyzed and broken, then it would mean that the underlying mathematical problem is not hard and can be solved efficiently.

The Public-key cryptography is a very rich area which gives rise to many beautiful cryptosystems and protocols. It is a very active field and many challenging problems are continuously solved and new ones are identified. In this dissertation, however, I dive into symmetric-key cryptography and do not study public-key cryptosystems. As an exception, in \PartRef{wb}, I study white-box cryptography, which, in particular, aims to construct a public-key cryptosystem from a symmetric-key primitive.

In practice, a hybrid method is used. The public-key cryptography is used to establish a shared secret key between the communicating parties, and all the consequent communications are encrypted using fast symmetric-key cryptography.

\subsection{Authenticated Encryption}

The main goal of symmetric cryptography is to provide \emph{authenticated encryption}. Authenticated encryption is a cryptosystem providing \emph{confidentiality}, \emph{integrity}, and \emph{authenticity}.
\begin{itemize}
    \item \emph{Confidentiality} guarantees that any adversary with predefined capabilities cannot recover any information about the original messages (called plaintexts) from the encrypted messages (called ciphertexts).

    \item \emph{Integrity} guarantees that any adversary with predefined capabilities cannot modify a transmitted ciphertext without the change being noticed by the receiver.

    \item \emph{Authenticity} guarantees that the receiving party can be assured that the message was generated by the sender.
\end{itemize}

There are several ways to construct an authenticated encryption scheme.

\subsubsection{Authenticated Encryption from Block Ciphers}

Block ciphers are the classical and the most widely used symmetric-key primitives. Formally, a block cipher is a family of permutations, where the secret key selects one of the permutations. The domain of the permutation is the message space.

The most widely spread block cipher is the AES block cipher~\cite{AES}, also called Rijndael, designed by Vincent Rijmen and Joan Daemen. It was standardized in 2001 by the US standardization agency NIST.

I and my colleagues designed a block cipher called SPARX~\cite{OurSPARX}. The design process and analysis are described in \ChapRef{sparx}.

A plain block cipher can only encrypt fixed-width messages. For example, AES has a 128-bit block size. The bigger problem is that direct encryption of message blocks under the same key (i.e., by the same permutation) leaks information about the equality of message blocks: if the two plaintext blocks are equal, then the two ciphertext blocks are equal too, which contradicts the confidentiality requirement.
Another big problem is that authenticity is not guaranteed. The blocks can be removed arbitrarily without being noticed.

An \emph{authenticated block cipher mode} is a construction that uses a block cipher to create an authenticated encryption scheme. Such a construction often consists of two parts: an encryption scheme for confidentiality and a message authentication code (MAC) for integrity. For example, the Encrypt-then-MAC~\cite{EncryptMAC} is a very generic mode that can combine an arbitrary (secure) encryption scheme and an arbitrary (secure) message authentication code in order to create the authenticated encryption. More specific authenticated encryption modes (e.g. GCM~\cite{GCM}, OCB~\cite{OCB}) partially reuse the computations of the two components and achieve better performance. Another class of modes (e.g. SCT~\cite{SCT}, COPA~\cite{COPA}, POET~\cite{POET}) requires a \emph{tweakable block cipher}~\cite{TBC}. Tweakable block ciphers take as input an extra \emph{public} parameter called a \emph{tweak}, and different tweaks should produce indistinguishable block ciphers.

\subsubsection{Authenticated Encryption from Stream Ciphers}

The \emph{one-time pad} is one of the first modern encryption schemes. It is a very simple cipher, but it is famous for achieving \emph{perfect secrecy}. It means that \emph{no information} is revealed from a ciphertext, even for a computationally-unbounded adversary. This property is also called \emph{information-theoretic} security. The one-time pad accepts an $n$-bit plaintext and an $n$-bit key. It combines the key and the plaintext using the exclusive-or operation. The plaintext and the key bits at the same position are added modulo 2. The requirement, however, is that the key should be sampled uniformly at random and used to encrypt only one message. These restrictions are not very practical and are the price for \emph{perfect secrecy}. Indeed, Shannon~\cite{Shannon} proved that these restrictions are \emph{necessary} if perfect secrecy must be kept.

\emph{Stream ciphers}, similarly to block ciphers, discard the perfect secrecy requirement and aim at more practical cryptosystems. Unlike block ciphers, stream ciphers attempt to simulate the one-time pad by generating the required large key (a \emph{keystream}) from a small key on the fly. The keystream is then called \emph{pseudorandom}. The security is based on the requirement that any (computationally-bounded) adversary cannot distinguish the pseudorandom keystream from a purely random sequence.

As in block ciphers, stream ciphers can be combined with a message authentication code (MAC) to create an authenticated encryption (e.g. ChaCha20-Poly1305~\cite{chachapoly}). Authenticated modes for stream ciphers were studied in~\cite{streamae}. Another approach is to design an authenticated stream cipher from scratch (e.g. ACORN~\cite{ACORN}).

\subsubsection{Authenticated Encryption from Permutations via Sponge construction}

A (cryptographic) \emph{hash function} is a cryptographic primitive that maps a bit-string of arbitrary length into a fixed-length bit-string. For a secure hash function, it should not be computationally easy to invert it (\emph{preimage resistance}) or find two messages that have the same hash value (\emph{collision resistance}). Furthermore, given a fixed message it should be computationally difficult to find another distinct message that has the same hash value (\emph{second preimage resistance}). In general, hash functions are often modeled as \emph{random oracles}. These oracles always return a truly random element of the hash function's codomain, except that for repeated queries with the same message they always return the same hash value. Hash functions are used in a huge number of protocols and public-key constructions. Since hash functions are \emph{keyless}, it is not clear whether they belong to symmetric-key or asymmetric-key cryptography. In practice, hash functions used are made in the symmetric-key style: created from low-level binary operations, very efficient but with heuristic security. However, there exist hash functions from more algebraic constructions, but they are typically only used in theoretic studies due to their inefficiency.

The \emph{sponge} construction was first formally presented in~\cite{sponge-ecrypt}, though similar ideas had already been used before (e.g.~\cite{LEX,RadioGatun}). It was used to design the winner Keccak~\cite{sponge-keccak} of the SHA-3~\cite{sha3} hash function competition organized by NIST. The sponge construction uses a primitive called \emph{cryptographic permutation}. The state is divided into the \emph{rate} part and the \emph{capacity} part. The rate part is usually controlled by an adversary, while the capacity part is uncontrolled. The sponge \emph{absorbs} message blocks in-between calls to the permutation. Afterward, it \emph{squeezes} pseudorandom outputs (e.g., hash values) in-between calls to the permutation. The construction is illustrated in~\FigRef{sponge}. The sponge is a \emph{provably secure} construction: if the chosen cryptographic permutation is \emph{ideal} (e.g., a purely random permutation), then the construction is guaranteed to be secure up to some level.

\FigTex{sponge.tex}

The designers of Keccak further showed~\cite{sponge-duplex,sponge-monkey} that sponges can be used to construct authenticated encryption. The latter variant of the mode is called MonkeyDuplex. This mode and its variants were used in several encryption schemes (e.g.~\cite{NORX,Ketje,ASCON}). Since a sponge only requires a cryptographic permutation, it inspired cryptography designs called \emph{permutation-based cryptography}. A recent improvement is the Beetle mode~\cite{beetle}, which achieves better security bounds.

I and my colleagues designed a hash function family \texttt{Esch} and an authenticated encryption family \texttt{Schwaemm}. They are based on the recent sponge-based mode called Beetle and a cryptographic permutation family derived from our SPARX block cipher. These designs are submitted to the NIST Call for Lightweight Cryptography~\cite{NISTlight}. I describe the design process and analysis of these primitives in \ChapRef{sparkle}.

\subsubsection{The CAESAR Competition}
Recently, the CAESAR competition was organized~\cite{CAESAR}. Its name stands for ``Competition for Authenticated Encryption: Security, Applicability, and Robustness''. The competition started in 2014 when 53 authenticated encryption schemes were submitted. After 5 years of selection process consisting of 3 rounds, the committee selected 8 portfolio members, from which 4 are the preferred choice. The portfolio is split into 3 use cases:

\begin{enumerate}
    \item \emph{Lightweight applications (resource constrained environments).}
    The preferred choice is ASCON~\cite{ASCON} which is based on MonkeyDuplex sponge mode. The second choice is ACORN~\cite{ACORN}, a dedicated authenticated stream cipher.

    \item \emph{High-performance applications.}
    The following two choices are chosen without a preference. The first one is AEGIS-128~\cite{AEGIS}, a dedicated design using a reduced-round AES as a component. The second one is OCB~\cite{OCBcaesar}, a block cipher mode.

    \item \emph{Defense in depth.}
    The preferred choice is Deoxys-II~\cite{DEOXYS}, an authenticated encryption scheme based on a tweakable block cipher. The alternative choices are COLM~\cite{COLM}, AES-COPA~\cite{COPA}, and ELmD~\cite{ELMD}, block cipher modes.
\end{enumerate}

\subsection{Black, Gray and White-box Models}

Security of cryptosystems is most often analyzed in a \emph{game-based} setting. The game usually happens between a \emph{challenger} and an \emph{adversary}. The challenger possesses secret information, for example, a secret key. The adversary is allowed to ask specified queries to the challenger. The goal of the adversary is to recover the secret information or, at least, a part of it.

Consider an encryption scheme. The challenger flips a coin and decides whether he will use the encryption scheme or its \emph{ideal} equivalent. In the first case, the challenger chooses the secret key uniformly at random. In the second case, the encryption is performed in the best possible way while maintaining the interface and semantics of the encryption scheme. For example, for each plaintext, the ciphertext may be assigned uniformly at random. Note that the challenger is not necessarily an algorithm and usually is not computationally-bounded, unlike the adversary. The game continues. The adversary can ask the challenger to encrypt several plaintexts chosen by the adversary. The challenger performs the encryption (either using the encryption scheme or its idealized version) and gives ciphertexts to the adversary. It is said the adversary is given access to the \emph{encryption oracle}.
The adversary finally has to guess, what the outcome of the challenger's coin flip was. That is, the adversary has to decide, whether the encryption was done using the encryption scheme or using its idealized version. If the adversary succeeds with non-negligible probability, then it is said that the encryption scheme has an  \emph{adaptive chosen-plaintext distinguisher}. If the adversary accesses the encryption oracle only once, it is said that the scheme has a \emph{(non-adaptive) chosen-plaintext distinguisher}.

There are three major models in which cryptosystems are analyzed.

\subsubsection{The Black-box Model}
The \emph{black-box} model restricts the analysis to the ``functional'' side of cryptosystems. An adversary in this model is usually given access to encryption and/or decryption oracles. That is, the adversary is allowed to ask the challenger to encrypt and/or decrypt arbitrary messages. Any intermediate computations or events are not visible to the adversary, thus the name ``black-box''. This model is fundamental - any weakness in this model is inherited to the gray-box and white-box models.

\subsubsection{The Gray-box Model}
The \emph{gray-box} model studies the ``physical'' side of cryptosystems, more precisely, of their \emph{implementations} and the devices on which the implementation is deployed. Indeed, this side provides much more information to the adversary. The adversary may be allowed to measure the time of execution of a query, the power consumption of the device, the electromagnetic radiation. This information is usually referred to as \emph{side-channel} information.
The adversary may be \emph{active} - for example, introduce faults in the computations, by heating the device or tweaking the voltage. It is an interesting phenomenon that physical access to the device often enables much more efficient attacks on the cryptosystem. Cryptanalysis in this model is called \emph{side-channel} cryptanalysis.

Countermeasures against \emph{side-channel} attacks may be introduced both in the implementation code and on the physical side. Protections that can be added to the implementation are more generic and, therefore, more preferable. In practice, both methods are used to ensure maximum security. Importantly, implementations typically may use (pseudo)randomness in order to protect computations. Together with the noise and uncertainty of the physical observations, these properties allow the creation of sound countermeasures against side-channel attacks.

\subsubsection{The White-box Model}
The \emph{white-box} model considers the extreme case when the adversary has \emph{full} access to the implementation, in the form of compiled code or Boolean circuits. Typically, the implementation contains a secret key and the adversary's main goal is to recover it. The hardness of the key recovery is often called the \emph{weak white-box} requirement. Other goals may be considered, such as compressing the implementation, inverting the computed function, or removing hidden ``watermarks'' allowing the user possessing the implementation to be traced. The respective security properties are called \emph{unbreakability}, \emph{incompressibility}, \emph{one-wayness}, \emph{(traitor) traceability} (see~\cite{wbNotionsOld,wbNotions}).
Unbreakability together with one-wayness result in a public-key scheme, if the embedded secret key allows efficient decryption. Such implementation is also called a \emph{strong white-box}. Indeed, the implementation secure in the white-box model can be seen as a public key, and the embedded secret key can be seen as a private key. A white-box implementation of a common symmetric encryption scheme would then have a very efficient decryption code. However, it turns out to be a challenging, if not impossible problem.

The white-box model was first introduced by Chow~\etal{}~\cite{ChowAES,ChowDES} in 2002. The authors proposed rather efficient white-box implementations of the AES and DES block ciphers. Unfortunately, they were broken with practical attacks. All consequent attempts to fix the scheme failed as well. A secure white-box implementation of a block cipher remains an open problem today.

White-box implementations are closely related to the notion of \emph{cryptographic obfuscation}. Indeed, a basic implementation has to be obfuscated in order to hide the secret key. There is an active research direction related to \emph{indistinguishability obfuscation} (iO), which is widely believed to be ``the best possible'' obfuscation. iO has many applications in theory: it is known that many provably secure cryptographic primitives can be created from secure iO. Unfortunately, many recent iO candidates were broken. Furthermore, all constructions are very inefficient. For example, a recent framework 5Gen-c~\cite{5GEN} can be used to obfuscate only a single round of the AES block cipher. I remark though, that there is no established provable link between white-box and iO.

\subsubsection{The WhibOx Competition}

In 2017, the WhibOx competition~\cite{whibox} was organized. Any person or team in the world could submit a white-box AES-128 implementation in C code of size up to 50 megabytes, then the implementation was publicly available for analysis. The goal was to recover the secret key from the implementation.

Among 94 submissions, most implementations were broken in less than a day. Only 13 implementations required at least one day to be broken, and only 8 of them required at least two days. The winning implementation survived 28 days, and the following implementation only 12 days. The winning design was created by myself and Alex Biryukov. The implementation did not involve any new provable security techniques, but relied on many interesting obfuscation tricks, effectively slowing down the reverse-engineering effort. We were also first to successfully cryptanalyze the best 3 implementations besides ours. Our participation in the competition initiated the research that resulted in~\PartRef{wb} of this thesis.

\subsection{Cryptanalysis of Symmetric-key Primitives}

The framework for cryptanalysis is most developed for block ciphers. Indeed, block ciphers were used from the 1970s with the designs of the LUCIFER and DES block ciphers. Together with a proper mode, a block cipher can be used to construct an authenticated encryption scheme. Furthermore, block ciphers tend to have a reasonably simple structure. This simplicity attracts cryptanalysts, who try to break the cipher using both established and novel methods of cryptanalysis. Since the same low-level operations are used in most symmetric-key primitives, cryptanalysis methods for block ciphers are usually very generic and can be applied to other primitives, such as stream ciphers, hash functions, message authentication codes, and authenticated encryption.

What does it mean to break a cipher? In the scientific community, successful cryptanalysis means an algorithm that disproves a security claim of the designers. A typical security claim is that the secret key can not be recovered faster than exhaustive search over the whole key space. A block cipher with a fixed secret key should not be distinguishable from a truly random permutation. Even if the attack is impossible in practice, it only matters that it is faster than the generic attack. The reason is that such an attack shows a \emph{weakness} of the block cipher. Since block ciphers are not provably secure, any weakness should be avoided.

The \emph{complexity} of a cryptanalytic attack is measured by the time, memory and data complexities of the algorithm. The data complexity corresponds to the number of queries that it makes.

In the simplest form, cryptanalytic attacks lead to a \emph{distinguisher} from a random permutation. In most cases, such an attack can be extended into the secret key recovery. This is done by guessing a part of the secret key and decrypting a part of the ciphertext. Then the correctness of the key guess is verified by using the established distinguisher.

\subsubsection{Cryptanalysis Methods}

In \emph{differential} cryptanalysis, an adversary encrypts two plaintexts with a fixed \txor difference. By an analysis of the cipher's structure, the adversary predicts a difference between ciphertexts with high enough probability. More precisely, the cryptanalyst studies the evolution of the plaintexts difference through all computations, until the ciphertext difference. A transition through nonlinear components is usually probabilistic, and all transitions' probabilities accumulate in an approximation of the probability of observing a particular ciphertext difference.

In \emph{linear} cryptanalysis, an adversary receives many plaintext-ciphertext pairs generated by the analyzed block cipher. The cipher is approximated by \emph{linear} equations, i.e. equations involving only the \txor operation. As in differential cryptanalysis, approximations of nonlinear components induce a cost in the form of probability. As a result, the resulting equations linking the key, the plaintext and the ciphertext hold only with particular probability. If the adversary observes enough data, then correct equations may be established with high probability. In practice, only the ratio of plaintext-ciphertext pairs for which the equation is correct is computed. For a random permutation, this ratio will be close to 1/2. For a weak block cipher, this ratio may be distinguishable from 1/2 with high probability.

In \emph{integral} cryptanalysis, the \emph{algebraic degree} of a block cipher is studied. It corresponds to the degree of the multivariate polynomial representation of the cipher. If the algebraic degree is not high enough, the cryptanalyst can deduce a set of plaintexts, for which the corresponding ciphertexts \txor to zero, independently of the secret key. Evaluation of the algebraic degree of a cryptographic primitive is a challenging problem and usually, only upper bounds on the degree can be proved. In~\ChapRef{feistel} I describe a method to obtain such upper bounds for the particular block cipher structure, called a \emph{Feistel Network}. It is based on the joint work~\cite{OurFeistel} with my colleague Léo Perrin.

Integral cryptanalysis is one of the main tools for \emph{structural} cryptanalysis. This branch of cryptanalysis studies ways to distinguish \emph{structures} of cryptographic functions and further decompose the function into components of the structure. It means that only the structure of the function is known to the adversary, and its components are kept secret. The most common structures are the substitution-permutation-network (SPN) and the Feistel network (FN). My colleagues Léo Perrin and Alex Biryukov found an intriguing application of structural and decomposition cryptanalysis. They applied it to \emph{small} functions called S-Boxes, which are used to build cryptographic primitives. S-Boxes are usually represented by tables in specifications and the process of their generation may be kept undisclosed. Structural cryptanalysis allows distinguishing particular structures in an S-Box. Together with analysis of resistance against linear and differential attacks, these methods can often reveal secret criteria behind an S-Box of unexplained origin. This direction is called the \emph{reverse-engineering} of S-Boxes. I contributed to the work of my colleagues in reverse-engineering of the S-Box used in the latest Russian cryptographic standards, and reverse-engineering of an S-Box of a mathematical origin. These results are described in \ChapRef{kuz} and \ChapRef{apn} respectively.

A recent direction of cryptanalysis is the search for \emph{invariants} of the cryptographic primitives. \emph{Linear} invariants correspond to a critical flaw in the primitive and are usually easy to avoid. \emph{Nonlinear} invariants are much harder to find. Indeed, the ideas of invariant-based cryptanalysis appeared a long time ago, but the actual applications of the method appeared only recently. A special case of a nonlinear invariant is an \emph{invariant subspace}. Invariant subspace cryptanalysis was introduced in~\cite{InvSpacePrint} and was used to break the PRINTcipher, designed in 2010. Another class of nonlinear invariants is formed by \emph{quadratic} invariants. This class was used in~\cite{NonlinInv} to show a practical distinguisher of recently designed block ciphers Midori, SCREAM, and iSCREAM. In~\PartRef{ni} I describe invariant subspaces in NORX, a CAESAR third round candidate; I also show a theoretical study of generalization of quadratic invariants to higher degrees. This part is based on joint work~\cite{OurNORX} with Alex Biryukov and Vesselin Velichkov, and on joint work~\cite{OurNLI} with Christof Beierle and Alex Biryukov.


\chapter{Cryptographic schemes}

Cryptographic schemes conceived for \zclaim are defined in the following pages.
On the other hand, schemes taken from Sapling to which reference is made in this work were introduced in~\cref{ch:prelims} and will not be reproduced here.

\section{Commitment schemes}
We define a commitment scheme as in~\cite[Section 4.1.7]{hopwood2016zcash}.

\subsection{Nonce commitment scheme}
\label{app:nncm}
The nonce commitment scheme \nncm may be instantiated as a Windowed Pedersen commitment scheme as defined in~\cite[Section 5.4.7.2]{hopwood2016zcash} in a similar fashion to Sapling's note commitment scheme \ncm, since the homomorphic properties required when hiding the note value are not necessary.
It is defined as follows:
\begin{flalign*}
    &\nncm_{\rcn}(\nlock) := \wpc_{\rcn}(\nlock)&
\end{flalign*}

\section{Signature schemes}
We use the definition of a signature scheme in~\cite[Section 4.1.6]{hopwood2016zcash}.

\subsection{Minting signature}
\label{app:mas}
\mas may be instantiated as \redjj as defined in~\cite[Section 5.4.6]{hopwood2016zcash} without key re-randomisation and with generator
\begin{flalign*}
    &\mathcal{P}_{\G} = \fgh(\text{``\zclaimm''}, \text{``''})&
\end{flalign*}

\subsection{Vault signature}
\label{app:vaultsig}
\vaultsig may be instantiated as \redjj as defined in~\cite[Section 5.4.6]{hopwood2016zcash} without key re-randomisation and with generator
\begin{flalign*}
   &\mathcal{P}_{\G} = \divhash(\dvf)&
\end{flalign*}
where \dvf is the diversifier associated with the vault in the vault registry.

\section{SIGHASH transaction hashing}
\label{app:sighash}

We use the \sighash transaction hash as defined in~\cite{ZIP243}, not associated with an input and using the \sighash type \sighashall, to which we add two new fields:
\begin{itemize}
    \item $\hashmints :  \B^{[256]}$ is 0 if the transaction does not contain a Mint transfer, otherwise it is the \blakezclaim hash of the serialization of the Mint transfer (in its canonical transaction serialization format) with the personalisation field set to ``$\mathtt{ZclaimMintHash}$''.
    \item $\hashburns :  \B^{[256]}$ is 0 if the transaction does not contain a Burn transfer, otherwise it is the \blakezclaim hash of the serialization of the Burn transfer (in its canonical transaction serialization format) with the personalisation field set to ``$\mathtt{ZclaimBurnHash}$''.
\end{itemize}


% -----------------------------*- LaTeX -*------------------------------
\documentclass[12pt]{report}
\usepackage{scribe_hgen486}
\usepackage{hyperref}
\begin{document}

\scribe{Arjun Biddanda}		% required
\lecturenumber{3}			% required, must be a number
\lecturedate{January 12}		% required, omit year
\lecturer{Matthew Stephens}
\maketitle

% please leave this comment
\framebox[.95\textwidth]{\parbox{.93\textwidth}{ {{\bf Note:}} These
lecture notes are still rough, and have only have been mildly
proofread.  }}
\vspace*{.1in}

% feel free to delete content below this line
% ----------------------------------------------------------------------

\section{Introduction}

The primary theme of todays lecture is on likelihoods, likelihood
ratios, and their interpretation. The writeup here \textit{heavily}
samples from the excellent vignettes found here: \\
\url{http://stephens999.github.io/fiveMinuteStats/analysis/likelihood\_ratio\_simple\_models.html}.

\section{Example : DNA Barcoding of Poached Elephant Tusks}

Elephant ivory is a commonly poached item, and we would like to determine which particular environment the elephants are coming from. The elephants can either come from (1) the savannah or (2) the forest. We denote the following two models and data from an elephant tusk:

\begin{align*}
M_s &: \text{Tusk comes from a savannah elephant}\\
M_f &: \text{Tusk comes from a forest elephant}
\end{align*}

\begin{center}
	\begin{tabular}{ c | c | c | c }
		Marker & Allele & $f_S$ & $f_F$\\
		\hline
	  1 & 1 & 0.4 & 0.8\\
	  2 & 0 & 0.12 & 0.2\\
	  3 & 1 & 0.21 & 0.11\\
	  4 & 0 & 0.12 & 0.17\\
	  5 & 0 & 0.02 & 0.23\\
	  6 & 1 & 0.32 & 0.25\\
	\end{tabular}
\end{center}

$f_S$ and $f_F$ represent the allele frequency of the ``1'' allele in the savannah and forest elephant populations respectively. The likelihood of the model $M_S$ can be defined as the probability of the data being generated under the model.

\begin{align*}
	L(M_S) &= P(Data | M_S)\\
	&= \prod (f_S)^x\times(1-f_S)^{1-x}\\
	&= (0.4)(1-0.12)(0.21)(1-0.12)(1-0.02)(0.32)\\
	&= 0.020399\\
	L(M_F) &= P(Data | M_F)\\
	&= \prod (f_F)^x\times(1-f_F)^{1-x}\\
	&= (0.8)(1-0.2)(0.11)(1-0.17)(1-0.23)(0.25)\\
	&= 0.0112\\
	LR(M_S/M_F) &= \frac{L(M_S)}{L(M_F)}\\
	&= \frac{0.0204}{0.0112} \approx 1.8
\end{align*}

We have defined the likelihood ratio as the ratio of the likelihood of the model $M_S$ over the likelihood of $M_F$. We also note that the models $M_S$ and $M_F$ are \textit{fully-specified}, that is to say that there are no free parameters in the model. We will explore unspecified models later on in the lecture.

Two distinct questions arise as a result of calculating this likelihood ratio : (1) how to interpret the likelihood ratio and (2) when can we claim that we believe a model over another model given a likelihood ratio?

\subsection{Context of Individual Likelihoods}

The purpose of a likelihood ratio is to examine the evidence (data) in the context of one model against another. As a brief toy example let us consider a fair coin tossed 100 times and landing with 50 heads and 50 tails. What is the likelihood of the model that this coin is a fair coin? $L(M_{fair}) = \left(\frac{1}{2}\right)^{100}$. However this very small number is simply a probability, and actually getting any set of 100 tosses with a fair coin results in this likelihood! Thus we can see that likelihoods are important by providing a context under which they can be interpreted, by comparing two models against each other.

\subsection{Notational Things}
\begin{itemize}
\item We often work in ``log-space'' since the individual likelihoods may be quite small. The Log-Likelihood Ratio (LLR) is defined as $log(LR)$
\item In english we would say $P(Data | M_0)$ as ``The likelihood under the model $M_0$''
\item Sometimes semicolons are used to denote the data, and curly a ``L'' for the likelihood (i.e. $log(LR) = log\left( \frac{\mathcal{L}(M_S ; D)}{ \mathcal{L}(M_F ; D)}\right)$)
\end{itemize}

\section{Example : Continuous Measurement of Protein in Blood}
Suppose that we a protein that is measured in the blood and we call this random variable $X$. We want to see if this protein varies in concentration according to disease or non-disease status We wish to test the following two models :

\begin{align*}
M_n &: X \sim Gamma(0.5, 2)\\
M_d &: X \sim Gamma(1,2)\\
\end{align*}

Where $M_n$ is the model under a non-diseased state and $M_d$ is a model under the diseased state. If we observe the data as $X = 4.02$ the likelihood ratio can be detemined as: $$ LR = \frac{f_{X|M_n}}{ f_{x | M_d}}$$. However we would only like to compare the density around the measurement we have actually obtained, so we will use a quick trick and assume the precision of the measurement of $X$ to be $\pm 0.05$ making $X \in [4.015, 4.025]$. Now that we have discretized this measurement we can integrate the respective conditional densities over this range, making the likelihood ratio :
$$ LR = \frac{ \int^{4.025}_{4.015} f_{X | M_n} }{\int^{4.025}_{4.015} f_{X | M_d} }$$

There are a couple of caveats to this approach as well that translate broadly to the calculation of likelihood ratios :

% TODO : define well-behaved?
\begin{itemize}
	\item The density function must be well-behaved in the integration bounds
	\item The data must be held the same between the models that we are comparing (transforming one and not the other is illegal!)
	\item If a likelihood ratio is 0 we can certainly say that
\end{itemize}

\subsubsection{Different Support of Random Variables}

Likelihood Ratios still work properly when we have different support for the models. Suppose we have a random variable $X$ which corresponds to the roll of a standard 6-sided dice and the following two models:

\begin{align*}
M_6 &: \text{All the dice rolls are a 6}\\
M_{fair} &: \text{The dice is a fair dice}\\
\end{align*}

We can imagine two scenarios from this : (1) when the roll is a 6 and (2) when the roll is not a 6. When $X$ is a 6 we have a likelihood ratio of $LR = \frac{P(X | M_6)}{P(X | M_{fair})} = 1/(1/6) = 1/6$. However when $X \neq 6$ we can say that the likelihoo ratio is 0 since $P(X \neq 6 | M_6) = 0$ and this is the numerator of our likelihood ratio.

\section{Continuation of Disease Example}

Let us assume that $Z_i = 1$ if patient is diseased, else $Z_i = 0$.

\begin{align*}
P(Z_i = 1 | X_i = x) &= \frac{P(X_i = x | Z_i = 1) \cdot P(Z_i = 1)}{P(X_i = 1)}\\
P(Z_i = 0 | X_i = x) &= \frac{P(X_i = x | Z_i = 0) \cdot P(Z_i = 0)}{P(X_i = 0)}\\
\frac{P(Z_i = 1 | X_i = x)}{P(Z_i = 0 | X_i = x)} &= \frac{P(Z_i = 1)\cdot P(X_i = x | Z_i = 1)}{ P(Z_i = 0) \cdot P(X_i = x | Z_i = 0) }\\
Odds_{Posterior} &= Odds_{Prior} \times \text{Bayes Factor}\\
\end{align*}

When the model is fully-specified the Bayes Factor is equal to the Likelhood Ratio. However the role of the prior odds also plays a large role in our interpretation of the posterior odds. For instance if we believe that the disease is very rare, then we will have to have a much higher likelihood ratio in order to truly believe that we have the disease. It is very important to consider the prior odds of having the disease.

\section{Likelihood Functions and Partially Specified Models}

Let us review our elephant example now. But let us assume that we have sampled 100 elephants only from the savannah and look at their alleles at one particular marker. We obtain 40 samples that carry the ``1'' and 60 samples that carry the ``0'' allele. We then want to evaluate a particular model and its likelihood :
\begin{align*}
M_q &: \text{Allele frequency of the 1 allele is q}, q \in[0,1]\\
\mathcal{L}(M_q) &= P(Data | M_q) = q^{40}(1-q)^{60}\\
\end{align*}

One way in which we can compare two potentially different values of $q$ would be to look at their difference in log-likelihood units. We would then look at $log(\mathcal{L}(M_{q_1})) - log(\mathcal{L}(M_{q_2}))$. If we get a log-likelihood difference of 2 then we know that $LR = e^2 \approx 7.4$.
\end{document}